vLLM OffloadingSpec

What it is

vLLM has a KV offloading framework (vllm/v1/kv_offload) that does not depend on the model. vLLM itself handles all parts that depend on the model:

• the KV cache groups of hybrid models
• the page layouts and the padding
• sliding windows and the alignment of Mamba state
• which caches go to the backend

vLLM gives the backend a key and the bytes of a chunk. The ADP backend (XkvOffloadingSpec) stores these bytes on the card through the storage gateway. XkvOffloadingConnector is the offloading connector of vLLM with two additions:

• A failed load is computed again.
• A lost gateway pauses the external cache, not the engine.

Why we recommend it

• Fast restore. A long prompt comes back after a restart of vLLM in a fraction of the time of its prefill.
• Sharded reads over NVLink. The card stores one copy of a page that all ranks hold the same. One rank in each NVLink group copies a share of the page from host memory. The other ranks get it over NVLink. With two NVLink groups of four GPUs, each restore copies a quarter of the data from host memory.
• Every model of vLLM’s framework. A new architecture or a new vLLM version needs little or no work on our side. The backend finds the features of vLLM, not its version number.
• Safe failures. A load that the card cannot serve is computed again. A lost gateway pauses the external cache, and the backend opens the session again when the gateway comes back. The engine continues to serve.
• Metrics in Prometheus, from the scheduler and from every worker.

How it differs from the vLLM KV connector

	OffloadingSpec backend	vLLM KV connector
Load	the whole stored prefix before the step	layer by layer, while the model computes
Model support	vLLM’s framework	the code of the connector, for each architecture
Tensor parallelism	one object for each group for replicated pages, sharded reads over NVLink	NVLink fan-out for MLA models
Failed load	computed again	computed again
Lost gateway	pause, then a new session without a restart	the cache is off until vLLM restarts
vLLM versions	0.28, 0.29, 0.31	0.28, 0.29 (0.26 on request)

The load before the step costs the prefill time of the prompt tail. On a hybrid model on one GPU the vLLM KV connector restores a little faster.

Turn it on

The product image carries both integrations. Set the connector in --kv-transfer-config:

vllm serve <model> \
  --enable-prefix-caching \
  --enable-prompt-tokens-details \
  --kv-transfer-config '{"kv_connector":"XkvOffloadingConnector",
    "kv_connector_module_path":"xkv_offload.connector","kv_role":"kv_both",
    "kv_connector_extra_config":{"spec_name":"XkvOffloadingSpec","spec_module_path":"xkv_offload",
      "block_size":128,"xkv.port":5557,"xkv.fragment_bytes":131072,"xkv.shard_reads":"auto"}}'

with PYTHONHASHSEED=0 in the environment of the vLLM process.

Setting	Why
block_size	Tokens in one stored chunk. A multiple of the block size of vLLM. For hybrid models and DeepSeek-V4, leave it out and set "blocks_per_chunk": 1.
xkv.port	The port of the gateway.
xkv.fragment_bytes	The size of one object on the card. 131072 for a remote card, 261120 for a local card. Set the gateway fragment_size to 0 (on a local card the gateway keeps 261120, which these objects fit).
xkv.shard_reads	auto uses sharded reads over NVLink where they apply.
xkv.namespace	The key space of the deployment. Set it on a shared cache.
xkv.affinity_chunks	The chunks at the start of the prompt that make the dialog mark (default 1). All chunks of one dialog go to one card (how a dialog finds its card).
PYTHONHASHSEED=0	A fixed hash seed. The block hashes, and so the keys on the card, stay the same after a restart.

One image, both paths. The product image carries this backend and the vLLM KV connector. The environment variable KV_PATH of the vLLM container selects the path when the start script of the image builds the command: fusionx, the default, keeps a FusIOnXConnector configuration as it is, and offload translates it to this backend. A switch, and a switch back, is a change of KV_PATH and a restart of the vLLM container. Mind the different keys: see the note below.

From a configuration of the vLLM KV connector. The image can translate a FusIOnXConnector configuration. Set KV_PATH=offload in the environment of the vLLM container. The start script translates xkv_gateway_port to xkv.port, key_namespace to xkv.namespace and chunk_size to block_size. It names the settings that have no meaning here in one warning. The log shows the result: KV_PATH=offload --kv-transfer-config {...}. Two settings stop the start instead: clean_cache: true (start the gateway clean with --start-clean=1), and a key that the offload path does not know, for example key_namespace_from_model (unknown settings [...]). Set key_namespace instead.

Tensor parallelism

• Replicated pages (MLA, the DSA indexer, the packed block of DeepSeek-V4): one object for the whole group. Rank 0 writes it, and every rank reads it. All ranks are one client group of one gateway, and they send the same requests in the same order.
• Pages that ranks hold in parts (MiniMax M3 at tensor parallelism 8, Mamba state): one object holds all parts. Each part goes to the card once.
• Pages sharded by attention head above one GPU need xkv.tp_layout: per_rank and a gateway for each rank (xkv.port_per_rank: true). The backend refuses other cases at start and names the way out.
• Pipeline parallelism and context parallelism are refused at start.

Failures

Event	What happens
A store fails (card full, gateway lost)	The object is not on the card. A later lookup is a miss.
A load fails	vLLM computes the missing part of the prefix again (xkv.load_failure: recompute). With fail, the request ends with an error.
The gateway stops or restarts	The external cache pauses. Lookups are misses and requests continue. The scheduler probes the gateway after 2 s, then at longer intervals up to 60 s. When the gateway answers, every rank opens a new session at the same step.
The card loses its storage, the gateway answers	Loads fail and are computed again. Stores are lost. Both continue when the card comes back.
A rank of vLLM stops	vLLM stops. This is the executor of vLLM, not the card.

Settings

The settings of the backend take the xkv. prefix in kv_connector_extra_config. The full list with defaults is in Configuration → OffloadingSpec backend.

Metrics

vLLM exports the transfers (vllm:kv_offload_load_bytes_total, vllm:kv_offload_store_bytes_total) and the hits (vllm:external_prefix_cache_hits_total). The backend adds vllm:kv_offload_xkv_*:

Metric	What it tells you
load_objects, load_misses	Objects read back, and objects that the card did not return. More than 0 misses: the card was wiped or a session was lost.
store_objects, store_failed	Objects that the card took, and objects that it refused.
probe_keys, probe_found	Lookups of the scheduler.
session_reopens	New gateway sessions after a lost one.
card_state	0 serving, 1 paused or opening a new session, 2 a rank left its lockstep group: restart the engine (all ranks). Alert when the maximum stays above 0 for longer than 30 s.
requests_opted_out	Requests that the front end kept out of the card.

Per-request control and salts

"kv_transfer_params": {"xkv_store": false} in a request keeps it out of the card. There is no lookup and no write for it. vLLM gives the backend no cache_salt, so the backend cannot see an unsalted request. When your deployment requires a salt, the front end marks every unsalted request with xkv_store: false (Tenant isolation).

Limits

• One engine for each gateway. Two vLLM instances, or both integrations, cannot share a gateway.
• vLLM 0.28 does not give the KV cache layout to the backend. After a change of the attention backend on 0.28, wipe the card.
• CUDA images only. The ROCm and other images do not carry this backend.
• The load is not layer by layer. The framework of vLLM has no such interface. The cost is the prefill of the prompt tail, see above.