Architecture

Overview

Components

Component	Where it runs	What it does
ADP integration	inside the engine, in the scheduler process and in every tensor-parallel worker	Finds the blocks of a prompt that are on the card, moves pages in and out, and fences the transfers against the forward pass. For vLLM this is the OffloadingSpec backend (recommended) or the vLLM KV connector. For SGLang it is the HiCache backend.
SDK client	inside the engine	Makes the storage keys, sends requests in batches, and copies pages between GPU memory and the shared-memory ring.
Shared-memory ring	/dev/shm of the host	The transfer window between the engine and the gateway. A slot of the ring holds one transfer. The ring is not a cache.
Storage gateway	a separate process (container) on the GPU server, one for each engine	Collects the requests of all ranks of one engine and schedules reads and writes. Cuts large objects into fragments, spreads them over the databases of the card, and talks to the card.
ADP card	a PCIe slot of the GPU server, or a storage server reached over NVMe-oF	Keeps the objects as key-value pairs on the NVMe SSDs behind it.
Cluster service	storage servers and witness hosts, only in a card cluster	Keeps the card set and the health of each card in etcd for the gateways.

Control messages between the SDK and the gateway go over ZeroMQ (port 5557 by default). Data never goes over ZeroMQ. It goes through the ring.

A restore with the OffloadingSpec backend

1. Lookup. The scheduler of vLLM computes the block hashes of a new request. The backend asks the card which chunks it holds. The card answers, so hits are also found after a restart. vLLM counts the found tokens as computed and reserves GPU blocks for them.
2. Load. Before the step computes, the workers read the whole stored prefix in waves. The gateway reads the objects into the ring. Each rank copies its share into a staging slot in GPU memory, and from there into the KV blocks.
3. Sharded reads. For a page that all ranks hold the same, the card is read once for the group. One rank in each NVLink group copies each share from the ring, and the other ranks get it over NVLink.
4. Save. vLLM gives each full chunk to the backend after the step. Rank 0 writes a replicated page. The gateway writes it to the card. The step does not wait for the card.

A restore with the vLLM KV connector, step by step

1. Lookup. When vLLM schedules a request, the scheduler side of the connector asks the gateway which chunks of the prompt are on the card (EXIST). vLLM counts them as computed tokens and reserves GPU blocks for them.
2. Load. In the forward pass, every rank sends one read request for each layer (GET). The gateway reads the objects into the ring. The SDK copies them into the reserved GPU blocks on its own CUDA stream.
3. Fence for each layer. Before a layer computes, it waits only for its own pages (wait_for_layer_load). So the reads of later layers overlap with the computation of earlier layers. The engine reads some caches before any layer hook: the indexer of DeepSeek Sparse Attention and the recurrent state of Mamba-type layers. The connector restores these caches before the pass.
4. Save. After a layer computes, the SDK copies its new pages into the ring, and the gateway writes them asynchronously (PUT). On decode steps under full CUDA graphs, the writes of a step go out as one request (deferred write). The engine then keeps its fastest graphs.

The gateway gives the ranks of one engine one shared view. It waits until every rank sent a request, then runs it once. A restore of a long prompt is one request for each layer of each rank, not one for each page. The gateway checks the ring capacity for each request, so the ring does not limit the context length.

Keys and persistence

A stored object is one chunk of tokens. Its key comes from:

• the block hash of the chunk in the engine. This hash covers the tokens of the chunk and of every chunk before it. It already carries the cache_salt of the request, the LoRA adapter and the multimodal inputs.
• the layout. The OffloadingSpec backend adds the model, the KV data type, the rank layout and the byte layout of the page. The vLLM KV connector adds the layer.
• a deployment namespace, optionally (xkv.namespace, key_namespace).

No part of a key depends on the process that wrote it. So a model server after a restart finds everything that it stored before. This needs a fixed PYTHONHASHSEED. Without it, Python salts the block hashes for each process, and nothing is found again. Two requests with the same prefix share the stored chunks. Two tenants with different cache_salt values never share them. Tenant isolation tells how to set it up.

The chunk is the unit of a stored object and of a hit. A prompt of N tokens can hit at most floor(N / chunk_size) × chunk_size tokens. The engine computes the tail again.

Fragments

The card accepts objects up to 256 KiB. Over NVMe-oF, one command carries at most the maximum transfer size of the device (MDTS, usually 128 KiB). A KV page of a hybrid model is 2–3 MiB. So the integration or the gateway cuts a large object into fragments under several keys and puts it together again on read. The fragments of one object go to different databases, so the card reads them in parallel. The fragment size comes from the placement: 131072 for a remote card, 261120 for a local card. The OffloadingSpec backend and the SGLang backend cut the objects themselves (xkv.fragment_bytes, max_io_bytes). The vLLM KV connector uses the gateway’s fragment_size. See ADP card setup.

The shared-memory ring

The ring is a file in /dev/shm. The gateway makes it, and every rank maps it. Its size (shared_memory_size, 6 GiB by default, 10 GiB for 8-GPU servers) sets how many transfers can be in flight:

max_parallel_ios = (shared_memory_size / 2) / io_size

io_size is one object, and the gateway writes it to its log at start. When a request needs more slots than are free, it runs in batches. The gateway gives a free slot to a transfer. It takes the slot back when the page is in GPU memory. A larger ring helps only when one request needs more slots than the ring has. An example is one layer of a long prompt with a small chunk. The ring uses host memory while the gateway runs. Both containers must see the same /dev/shm.

Choosing a mode

Your situation	Mode	Details
One deployment on one GPU server	One card for the pod, local or remote	Installation
One deployment that must keep its cache when a card fails, or needs more capacity	Several cards behind one gateway, key_placement: dialog	Placement
One long restore must read as fast as possible	Several cards, key_placement: stripe	Placement
Replicas or several models on several GPU servers share one cache; the set of cards changes rarely	Shared cache (remote-xdp, the same list of cards in every gateway)	Shared cache
Cards and storage servers join and leave while the models serve; one place for the card set and its health	Card cluster (kv-spdk, etcd on three or more hosts, up to 8 cards)	Card cluster

Deployment topologies

Topology	Notes
One engine, one gateway, one card	The basic unit. One gateway serves the ranks of exactly one engine. A second engine on the same gateway gets a refusal.
Several engines on one server	One gateway for each engine, each with its own port and ring.
Several deployments on one or more cards	Each pod has its own gateway on the same cards. Each model configuration has its own namespace. See Shared cache.
Cards managed by a cluster service	The gateways read the card set from etcd and change it at run time. See Card cluster.
Card in a storage server	The GPU server connects the NVMe-oF namespace of the card over RDMA. The gateway uses it as a local device.

Several cards: failover and placement

A gateway can use several cards. When one card fails, the gateway takes it out of service and keeps the cache on the others; when the card comes back, the gateway uses it again, with no restart. The gateway setting key_placement selects how the blocks of a conversation go to the cards:

• dialog (default): all blocks of a conversation on one card. A failed card costs only its conversations, which are computed again in full.
• stripe: the blocks of a conversation over all cards. One long restore reads from every card at once.

Use dialog unless one long restore must be faster than one card can read: then use stripe, and accept that a failed card cuts most long conversations short until they are computed again.

Shared cache → placement tells how to choose and set it.

Card cluster

In cluster mode the gateways do not have a list of cards. The cluster service awide-cluster runs on the storage servers and on witness hosts. The hosts keep an etcd quorum, and the leader writes the card registry: the cards, their NVMe-oF addresses and their health. Each gateway reads the registry about once a second, connects to new cards, and stops to use a card that is sick or gone, while the models serve.

Card cluster tells how to set it up.

When the card is unavailable

The external cache never stops the generation:

• OffloadingSpec backend. When the gateway stops to answer, the backend pauses the external cache. Lookups are misses and requests continue. When the gateway answers again, every rank opens a new session at the same step, with no restart. A load that fails is computed again.
• vLLM KV connector. When the gateway stops to answer, the connector turns the external cache off and writes External KV cache turned off on this rank once. A model with attention only then serves from GPU memory. A hybrid model (Mamba-type layers) stops cleanly, because it must not serve a recurrent state that is restored only in part. To turn the external cache on again, restart the gateway and vLLM together (on Kubernetes, the whole pod).
• SGLang HiCache backend. A failed request is a miss. The backend stops to use the card for a cooldown time, then tries again.

The gateway takes a card that comes back into service by itself, with no restart:

Backend	When the card goes	When it comes back
local-xdp (local card)	After three answers in a row that the card or its storage service is gone, or a failed probe while no I/O comes (every 5 s), the card leaves service. A lookup is a miss, a write fails at once.	A background thread tries again after 1 s, then with the pause doubled up to 30 s. It takes the card back only when it is the same card: the same serial number and the storage identity that the gateway wrote on the card.
remote-xdp (remote card)	The health check (every second) takes out a card that does not answer Identify. Without io_timeout_ms, a failed command fails only its key. With io_timeout_ms set, io_timeout_strikes commands in a row that pass the deadline or fail in the transport take the card out; below that, only their keys fail.	The gateway looks for the card by its serial number every 5 s, also under a new device name after a restart of the target or a reconnect. It takes the namespace only when it answers with the same serial number and model.
file-system (a directory, for tests)	A directory that cannot be read or written leaves service.	A directory that comes back under its path with the label .xkv_storage_id of the gateway is used again. A directory with another label, or with files and no label, is another storage and stays out.
kv-spdk (card cluster)	The cluster reports the card sick for cluster_sick_polls polls, its listener is gone for cluster_absent_polls polls, or io_timeout_strikes commands in a row pass the io_timeout_ms deadline.	At the next poll that reports it healthy (Card cluster).

The container of a local-xdp gateway must see /etc/adp and /sys for the check of the serial number. The container of a remote-xdp gateway must see the host’s /dev and /sys/class/nvme, not a list of devices fixed at start. Without them the card stays out of service until the gateway restarts.

Sharded reads and NVLink fan-out

On MLA models with tensor parallelism, every rank needs the same latent page. Both vLLM integrations use NVLink to copy this page from host memory only once for each NVLink group:

• OffloadingSpec backend: sharded reads (xkv.shard_reads: auto). Each rank copies a share of each load wave from the ring. It gets the rest over NVLink from the staging slots of the other ranks. At start the ranks ask the driver and NVML which peers have NVLink, and they make the groups from that. On a server with two NVLink islands of four GPUs, the ring is copied twice instead of eight times.
• vLLM KV connector: NVLink fan-out (nvlink_fanout). Each rank of an NVLink group copies its share of a layer from host memory. The ranks exchange the rest over NVLink just before the layer computes. Settings: vLLM KV connector.