LMCache L2 plugin

What it is

The plugin makes the ADP card the second tier (L2) of the LMCache multiprocess server. LMCache keeps all its parts: its interface to vLLM, its index, its policies and its host-memory tier (L1, pinned host memory). Objects that leave L1, and objects that LMCache looks up after a restart, are on the card behind the storage gateway. LMCache loads the plugin through its own native_plugin adapter type. LMCache itself does not change.

Use it when LMCache is already part of your stack. Otherwise use the OffloadingSpec backend: it needs no host-memory tier (comparison).

How it stores data

• Objects and fragments. One LMCache chunk is one object. The plugin cuts it into fragments of fragment_bytes, and each fragment is one I/O. The plugin writes a small manifest after the card acknowledged every fragment. An object is on the card only when its manifest is there.
• Integrity. A load checks the manifest: size, geometry and key. By default it also checks a sample checksum of every fragment. A mismatch is a miss for that object only. The plugin removes a manifest that it cannot use, so the next store writes the object again.
• No double writes. Before a store, the plugin asks the card for the object. When the object is there, the plugin skips the store. A key is a hash of the tokens, so the same chunk is the same entry.
• Failures. A missing key or fragment is a miss. When the gateway stops to answer (wait_timeout_s), every answer is a miss until the plugin opens a new session (reopen_after_s). This is also how the plugin finds a restarted gateway, or a gateway that was not up at the start of the server.
• Backpressure. When the store backlog is above max_pending_store_mb, the plugin refuses new stores at once. A slow card then does not hold L1 objects. Loads have priority over stores.

Requirements

vLLM	0.28.0
LMCache	0.5.4 (in the vllm/vllm-openai:v0.28.0 image) or 0.5.5, in multiprocess (server) mode.
Image	vLLM 0.28.0 with the plugin compiled on it, delivered by Awide Labs on request
Gateway and card	as for the other integrations: ADP card setup, the gateway from Installation

Run it

The LMCache server takes the plugin as its L2 adapter:

lmcache server --host 127.0.0.1 --port 6555 --l1-size-gb 120 --eviction-policy LRU --chunk-size 256 \
  --l2-adapter '{"type":"native_plugin","module_path":"xkv_lmcache","class_name":"XkvL2Connector",
                 "adapter_params":{"port":5557,"fragment_bytes":131072,"affinity_chunks":1}}'

vLLM connects to the LMCache server:

--kv-transfer-config '{"kv_connector":"LMCacheMPConnector",
  "kv_connector_module_path":"lmcache.integration.vllm.lmcache_mp_connector",
  "kv_role":"kv_both","kv_connector_extra_config":{"lmcache.mp.port":6555}}'

Inside the vLLM container. With LMC_ENABLE=1, the main process of the container starts the LMCache server and waits until its port answers. The server and its L1 then restart with vLLM.

LMC_ENABLE=1 LMC_L1_GB=120 LMC_CHUNK=256 LMC_XKV_PORT=5557 LMC_XKV_FRAG=131072
# optional: LMC_XKV_VERIFY, LMC_XKV_COPY_THREADS, LMC_XKV_STATS_PATH=/tmp/xkv-stats.json,
#           LMC_XKV_PARAMS='{"batch_slots":2048}' (merged last); python3 -m xkv_lmcache_autostart --help lists them all

As a separate process. Then L1 stays after a restart of vLLM. Use the same image and start python3 -m xkv_lmcache_autostart --exec (without LMC_ENABLE). The process needs:

• the GPUs, because it copies L1 to the GPU through CUDA IPC
• the IPC namespace of the host, and the same /dev/shm as vLLM and the gateway
• the gateway port

docker run -d --name lmcache-server --gpus all --network host --ipc host \
  -e LMC_HOST=0.0.0.0 -e LMC_PORT=6555 -e LMC_L1_GB=120 -e LMC_CHUNK=256 \
  -e LMC_XKV_PORT=5557 -e LMC_XKV_HOST=<gateway address> -e LMC_XKV_FRAG=131072 \
  --entrypoint python3 <image> -m xkv_lmcache_autostart --exec
# vLLM: "kv_connector_extra_config": {"lmcache.mp.host": "<server address>", "lmcache.mp.port": 6555}

On Kubernetes, run it as a second container in the model pod, or as a pod with host network and host IPC on the node.

Settings

Set the plugin in the adapter_params of the L2 adapter, or with the LMC_XKV_* variables of the autostart. Each key and its default is in Configuration → LMCache plugin. Set fragment_bytes to the I/O size of the card: 131072 on a remote card, 261120 on a local card. Keep the L2 limit of LMCache (max_capacity_gb) at 0. The key eviction of the card manages the space on the card.

affinity_chunks (default 1, or the variable LMC_XKV_AFFINITY_CHUNKS of the autostart) sets how many LMCache chunks at the start of the prompt make the dialog mark. All chunks of one dialog go to one card (how a dialog finds its card).

What to expect

• Hits in L1 come back a little faster than from the card, but L1 takes host memory.
• Hits from the card go from the card to L1 and then to the GPU. LMCache completes the whole prefetch into L1 before it copies to the GPU. So a restore after a restart of the engine is slower than through the other integrations. This cost is inside LMCache, not in the plugin.
• Host memory. The data goes through two host buffers: the gateway ring and L1.

Limits:

• one LMCache server for each node, and one gateway for each server
• two servers must not write to one card
• Linux on x86_64 only

Monitoring

Every stats_interval_s, the plugin writes a summary line with the prefix [xkv-lmcache] to the log of the LMCache server. stats_path writes the full JSON: tasks for each operation, hits, bytes, latency percentiles, queue depths, ring use, integrity counters, session state and process memory. These counters show the result of the stores:

Counter	Meaning
store_skipped	The object was already on the card.
store_deduped	The plugin removed a duplicate store.
store_dropped	The backlog was full.
store_rejected_by_card	The card refused the write. When it grows, check the capacity of the card.