New ADP KV cache connector release: hybrid Mamba and MTP on vLLM 0.26

Inference no longer looks like a chat box. An agent opens a repository, reads files, calls tools, corrects itself, and keeps every step of that in its context: a session that starts at a few thousand tokens is still running an hour later at fifty thousand. A support copilot carries the whole ticket history into every turn. A document assistant is handed 200,000 tokens before it is asked its first question. These are long-lived, context-heavy sessions — and a production server runs many of them at once.

All of that context exists as a KV cache in GPU memory, sitting next to the weights in the scarcest space in the server. The model claims its share first, and whatever is left has to be divided among every session in flight. A single 200,000-token prompt can take the rest on its own. Eight coding agents, each with a different history, simply evict one another.

Eviction has exactly one price, and it is always the same: recompute. The next request that needed those pages pushes the entire prompt back through the model before it can emit a single token. On eight H200s and a 200,000-token prompt that is 44.5 seconds of prefill before the first word appears. Past a handful of concurrent agents the cache stops holding anything reusable at all and generation collapses to 2.7 tokens per second per agent — unusable for anything interactive.

And none of that work is new. Those keys and values were computed once already, correctly, and then thrown away because there was nowhere to put them. The longer the session, the more times it pays for the same tokens.

The ADP KV cache connector plugs into vLLM through its v1 connector interface — the hook the engine uses to hand KV pages to something outside itself. As a request generates, the connector writes those pages out to the Awide Data Processor, the PCIe accelerator that holds them (reached over NVMe-oF in the tests below). When a later request shares a prefix — the same document, the same system prompt, the same agent history — the pages are read back off the card and placed straight into the engine's cache.

The model never sees the difference: these are the same keys and values it wrote, compared byte for byte on the way back. What changes is the cost. Prefill stops being a matter of pushing every token through the attention stack and becomes a matter of reading it off a device.

The GPU cache stays exactly where it is and keeps doing what it is good at. The connector simply makes it the first tier instead of the only one.

What this release adds: HMA and MTP

This release moves the connector onto vLLM 0.26 and, with it, onto the model families that shipped this year. Hybrid Mamba (HMA) and Gated DeltaNet layers store their cache in a shape the previous version could not describe, sparse attention keeps an index the engine never hands to a connector, and multi-token prediction (MTP) conflicted with the connector outright. Each of them failed differently.

Four model families are new to the supported list, and three ceilings are gone — cache pages larger than the accelerator's transfer size, contexts above roughly 49,000 tokens, and the conflict with MTP. All three are now handled in the shared layer, so the next architecture is a configuration change rather than a rewrite.

Everything below is measured on the release build.

The hard part of KV offload is not the fast path — it is that every new model family lays its cache out differently. Sparse attention keeps an extra index the engine never hands over. Hybrid Mamba models carry a recurrent state instead of keys and values. Sliding-window layers expire pages that full-attention layers keep. All of it is now handled in the shared layer, not per model.

MTP runs alongside it

Multi-token prediction works with the connector enabled at a 55.3% acceptance rate — 2.76 of every five draft tokens accepted — with a 100% cache hit rate and no errors. The storage format is independent of speculation, so a cache written with MTP on reads back correctly on an engine running without it.

The measurement below is deliberately unkind to us. Between the cold and the warm column the vLLM engine is restarted — the GPU cache is wiped, so a hit can only come from the accelerator. Every prompt is independent, and each cold run is confirmed cold by the query counter reading the prompt's full length.

Coding agents are the worst case for a GPU-only cache. Each one carries a 40–50k-token history, and the histories are all different, so they evict one another. Past a handful of concurrent agents the GPU cache stops holding anything useful and generation collapses.

Below is a customer A/B on eight H200 PCIe cards. Same model, same traffic, same prompts — the only difference is whether the ADP connector is enabled.

At one and four agents the connector costs nothing and already runs ahead. At eight — the point where the GPU cache is full and the baseline falls off a cliff — it is more than six times faster. The share of requests below the customer's speed floor drops with it, to 42.2% against 50.6%.

A separate capacity sweep, from one to three hundred concurrent users on the same eight-GPU server. The 95th-percentile time to first token is lower with the connector enabled at all seventeen measured points — not on average, at every one of them.

At three hundred concurrent users the connector holds first-token latency at 33.9 seconds against 74.1, and it does so while returning zero failed requests. The baseline drops 1.64% of them at the same load.

A cache that is fast and occasionally wrong is worse than no cache at all. Every acceptance run compares each restored page against exactly what the engine wrote.