ADP KV cache 6.5.0: a card cluster with failover, SGLang and vLLM 0.31, faster restores

In 6.5.0 we changed how OffloadingSpec finds and reads a stored prefix. The gain shows on models whose KV cache has sliding-window or compressed groups: DeepSeek-V4 and V4.1, Mamba and Gated DeltaNet hybrids such as Qwen 3.8, and models with sliding-window attention such as Gemma. It is largest for long contexts and for many restores at the same time.

• A key space for each model. The key namespace comes from the model configuration. Replicas of one model share their cache, and two models on the same cards never read each other's objects.
• No accidental wipe. A gateway on shared cards refuses to clean them, at start and on a client request.
• One engine for each gateway. A second engine on the same gateway gets a clear refusal instead of somebody else's data.
• A key layout fingerprint. A pod that sees a different set of cards does not start, instead of losing its hits silently.

A new replica restores a 196,000-token prompt that another pod wrote in 1.40 s instead of 15.1 s of prefill, and the failure of one gateway does not touch the others. Shared cache on several cards.

In a shared cache every gateway has the list of cards in its configuration. A card cluster moves that list into a service. awide-cluster runs on every storage server and on witness hosts; together they keep an etcd quorum, and the leader writes there which cards exist, their NVMe-oF addresses and their health. A gateway in cluster mode reads this registry about once a second and connects to the cards with its own NVMe-oF initiator. The gateway:

• takes the slots of the cards from the full registry, so every gateway puts every card in the same place, whatever order the cards appeared in;
• bounds every etcd call (500 ms by default), so a slow etcd never stalls the cache;
• takes a card out only after it is reported sick several polls in a row, and back as soon as it is healthy;
• keeps working with the last known cards when etcd or the leader's metrics are unavailable.

Cards join and leave while the models serve. A card that leaves the cluster, because it is sick or its listener is gone, is taken out of service on every gateway: only its conversations are computed again, none is restored in part. When it comes back, the gateways use it again and its conversations come back from the cache, with no restart of the pods. A new card is connected on the fly at the next polls; it usually takes a free slot, and when its slot is taken the gateways move the other card alike and say so in the log. The cluster holds up to 8 cards and needs three or more hosts for etcd. Setting up a card cluster.

One cache can live on several cards in two ways: several cards behind one gateway, which also keeps the cache running when a card fails, and the card cluster above, where many gateways share the cards of many storage servers. In both, the gateway decides on which card each block of a conversation goes. In 6.5.0 this is a setting of each deployment, key_placement:

• dialog, the default: for fault tolerance. All blocks of a conversation go to one card, chosen by a tag of the prompt beginning. When a card fails or freezes, only its conversations are lost, and they are computed again in full, never restored in part; the conversations on the other cards come back whole, and the engine keeps serving.
• stripe: for speed. The blocks of a conversation spread over all cards, so one long restore reads from all of them at once. Every block keeps its description and its data on one card, so a striped cache survives the loss of a card and its return: the affected conversations are computed again and stored on the live cards, and after the card comes back everything is found again.

Both placements are part of the key layout that every gateway on the same cards must share. Choosing a placement, how it works.

• Local card. When the storage service of the card restarts or crashes, or the card leaves the PCI bus and comes back, the gateway takes the card out of service (a lookup is a miss, a write fails at once), checks that the card that comes back is the same one by its serial number and an identity stored on the card, and puts it back, with no gateway restart.
• Remote card. After a restart of the NVMe-oF target or a reconnect, the namespace often comes back under a new device name. The gateway finds it by its serial number and takes it only if it answers with the same serial number and model. The engine keeps serving all the time.
• The NVMe-oF target follows its store. Each target service is bound to the store service of its card: a restart of the store restarts the target after it.
• A directory as a backend carries a label, so a directory that comes back under its path is used again, and a different storage mounted in its place is not.

When a card is unavailable, operations and monitoring.

• SGLang 0.5.21 with the card as the L3 tier of its hierarchical cache, on stock SGLang, with tensor parallelism. After a restart of SGLang the first requests come from the card, and generation runs at 99.6–101.5 % of the speed of plain SGLang. SGLang HiCache backend.
• vLLM 0.31 joins 0.28 and 0.29, with MiniMax M3 and DeepSeek-V4.1-Flash among the verified models.
• GLM-5.x: use vLLM 0.28. It decodes GLM-5.x faster than 0.31.
• From vLLM 0.32 on, ADP works with vLLM through its KV offloading framework only (the OffloadingSpec backend, which we introduced earlier): vLLM retires the model runner that our KV connector needs. The KV connector stays supported on vLLM 0.26–0.29.

Supported models and engine versions.

• Prefix cache isolation between clients. Keys come from vLLM's and SGLang's salted block hashes, with a key namespace and switches to refuse or skip unsalted requests. A cache_salt keeps tenants apart on every path. All options are off by default. Tenant isolation.
• Full decode speed on servers with two NVLink islands, also on vLLM 0.29 and 0.31.

After the upgrade the cache starts cold and fills again from new traffic. No cleaning of the card is needed.