vLLM KV connector

How it plugs in

The connector uses the KV connector interface of vLLM (V1), with the interface for hybrid models with several KV cache groups. vLLM loads it by name from --kv-transfer-config. There is no fork and no patch of the engine. One code base runs on vLLM 0.26, 0.28 and 0.29. It finds the features of vLLM, not the version number.

It works on both sides of the engine:

• Scheduler side. For a new request, it asks the gateway which chunks of the prompt are on the card. It tells vLLM how many tokens it can load. After the allocation, it selects the blocks to read and the new blocks to write.
• Worker side, on every tensor-parallel rank. It registers the KV caches of the engine and sends the reads at the start of the forward pass. Each layer waits only for its own pages. The connector writes the new pages after each layer.

At start, a registry decides who writes and who reads each cache of the engine. When nobody can write a cache of the model, the connector refuses to start. It does not cache nothing in silence. Architecture follows a request through both sides.

Turn it on

vllm serve <model> \
  --enable-prefix-caching \
  --enable-prompt-tokens-details \
  --kv-transfer-config '{"kv_connector":"FusIOnXConnector",
    "kv_connector_module_path":"xkv_vllm_connector.fusionx_connector",
    "kv_role":"kv_both",
    "kv_connector_extra_config":{"xkv_gateway_port":5557,"chunk_size":128}}'

Set these variables in the environment of the vLLM process:

export PYTHONHASHSEED=0                 # required
export XKV_GATEWAY_HOSTNAME=localhost   # where the gateway runs; localhost is the default
export VLLM_USE_V2_MODEL_RUNNER=0       # vLLM 0.29 only

Setting	Why
PYTHONHASHSEED=0	A fixed hash seed. The block hashes of vLLM, and so the keys on the card, stay the same after a restart.
--enable-prefix-caching	The connector serves prefix-cache hits. Hybrid models need it explicitly. Without it, vLLM turns the Mamba cache off and the connector refuses to start.
--enable-prompt-tokens-details	Optional. Responses then report cached_tokens, which shows a restore.
VLLM_USE_V2_MODEL_RUNNER=0	vLLM 0.29 only. The connector works with the V1 model runner.

The names FusIOnXConnector, xkv_vllm_connector and xkv_gateway_port are the interface of the connector. They do not change between releases. The options for each model (--kv-cache-dtype, --max-num-batched-tokens, --watermark, chunk_size) are in Supported models.

Settings

All keys go into kv_connector_extra_config. Each key and its default is in Configuration → Connector. Most deployments set only three:

• xkv_gateway_port, the control port of the gateway, 5557 by default
• chunk_size, the tokens in one stored object, for each model (Supported models)
• slot_alignment_bytes: 4096 on a remote card, for models with a KV page that is not a multiple of 4 KiB (GLM-5.x)

Per-request control

This field in the request body keeps a request out of the external cache:

{ "kv_transfer_params": { "xkv_store": false } }

There is no lookup, no restore and no write for the request. A front end sets it for traffic of one request that nobody asks for again. Nothing from the prompt goes to the card. Clients must not set kv_transfer_params themselves.

Tenant isolation

The connector makes its keys only from the block hashes of vLLM, and these hashes carry the cache_salt of each request. When the API front end salts requests for each tenant, the scope of a tenant holds on the card as in GPU memory. No connector setting is necessary. Two optional switches make it stricter:

• require_cache_salt keeps unsalted requests out of the card.
• key_namespace separates deployments on one card. On a shared cache, use key_namespace_from_model: true.

The rules for the front end, a working front end and a test are in Tenant isolation.

NVLink fan-out

On MLA models such as GLM-5.x, every tensor-parallel rank needs the same latent page. With fan-out, each rank of an NVLink group copies only its share of a layer from host memory. The ranks exchange the rest over NVLink with NCCL just before the layer computes. A restore then copies each page once for each group from host memory, not once for each rank.

"kv_connector_extra_config": { "nvlink_fanout": true }

The nvlink_fanout_* keys in Configuration set the groups, the staging buffer and the smallest layer for fan-out. With auto, the groups are the GPUs that NVML reports as NVLink peers.

Fan-out needs:

• NVIDIA GPUs with NVLink peer access
• tensor parallelism above 1
• an MLA model with the same latent page on every rank

Hybrid models (also Kimi Linear and K3), models with attention only and the packed layout of DeepSeek-V4 do not qualify. In such a case fan-out stays off, the log shows one warning with the reason, and the restore runs as without fan-out. The staging buffer and the NCCL communicator are outside gpu_memory_utilization. With the defaults, lower --gpu-memory-utilization by about 0.005 on a 141 GB GPU. With NCCL 2.26 or newer, set NCCL_LAUNCH_ORDER_IMPLICIT=1 in the vLLM environment.

Logs to expect

At start, on rank 0:

XKV: <N> cache tensors registered for exchange
KV cache registry: <N> cache(s) in <G> group(s); writers: ... ; readers: ...
KV cache layout: <G> group(s)

While vLLM serves, its periodic stats line has External prefix cache hit rate. When the gateway stops, the log shows:

External KV cache turned off on this rank

After this line the engine gets zero hits until you restart the gateway and vLLM together (what happens then).