Documentation menu
Overview

ADP KV cache documentation

The ADP KV cache keeps the KV cache of vLLM and SGLang on the Awide Data Processor (ADP) card. It brings the cache back after an eviction from GPU memory or a restart of the model server. This section tells how it works, what it needs, and how to install and configure it.

What it does

A model server keeps the KV cache of each prompt in GPU memory, but only while there is space. Sometimes the server evicts a long conversation, the context of an agent or a shared system prompt. The next request with the same start must then compute the full prefill again. The same happens after a restart of the server. On a large model with a long context, this prefill takes tens of seconds.

The ADP KV cache gives the engine a second, much larger tier for this cache. The engine also writes each KV page that it computes to the ADP card. When a prompt comes back, the engine restores its prefix from the card and does not compute it again. The keys are the block hashes of the engine. So the cache stays after a restart of the model server, and all requests with the same prefix share it. A long prompt comes back in seconds instead of tens of seconds of prefill.

What you get:

  • Restore, not recompute. The time to first token of a long prompt that comes back is an order of magnitude shorter. The GPUs use that time for new work.
  • A cache that stays when GPU memory does not. Terabytes of KV cache on NVMe behind the card. The cache stays after restarts, upgrades of the model server and evictions from GPU memory.
  • One cache for many pods. Replicas on several servers can share one cache on a set of cards (Cluster modes).
  • No extra tier in host RAM. The integration uses only a shared-memory transfer window of a few GiB.
  • Stock engines. vLLM loads the integration with one command-line option, and SGLang through its storage backend interface. There is no fork of the engine.

Architecture

The integration runs inside the engine, next to each tensor-parallel rank. It gives KV pages to the storage gateway, a separate process on the same server, through a ring in shared memory. The gateway schedules the reads and writes, cuts large objects into fragments that the card accepts, and spreads them over the databases of the card. The ADP card keeps them on NVMe SSDs.

How the pieces fit
one GPU server · the ADP card local or remote
GPU SERVER vLLM or SGLang engine one process per tensor-parallel rank GPU 0 GPU 1 GPU 2 … GPU N ADP KV cache integration · OffloadingSpec backend KV pages, layer by layer Shared-memory ring /dev/shm on the host · a transfer window, not a cache read and written by slot Storage gateway schedules I/O, fragments large objects local · PCIe ADP card · in the GPU server PCIe Gen5 x16 slot, NVMe SSDs behind it Clients OpenAI-compatible API prompts OR: STORAGE SERVER ADP card · remote exported as an NVMe-oF target NVMe SSDs the capacity of the cache Either placement: the same integration and gateway, another gateway backend. NVMe-oF RDMA

The card can be in one of two places. Nothing above the gateway changes between them:

PlacementWhere the card isHow the gateway reaches itGateway backend
Locala PCIe slot of the GPU serverthe ADP software stack on the same hostlocal-xdp
Remotea storage serverNVMe over Fabrics on an RDMA network (RoCE v2 or InfiniBand)remote-xdp

A remote card keeps storage off the GPU servers, and the cache can be on hardware sized for it. A local card needs no fabric. Several pods can also share one cache on several remote cards (Cluster modes). Architecture follows a page through the system.

Compatibility at a glance

Supported
Inference enginesvLLM 0.28, 0.29 and 0.31 with the OffloadingSpec backend. vLLM 0.28 and 0.29 with the vLLM KV connector, and 0.26 on request. SGLang 0.5.21 with the HiCache backend.
GPUsthe NVIDIA GPUs that the engine image supports (compute capability 7.5 and newer in the upstream images)
CUDAthe CUDA of the engine base image: 13.0 for the default vllm/vllm-openai tags, 12.9 for the -cu129 tags
Model architecturesfull attention, sliding window, MLA, DeepSeek Sparse Attention, compressed sparse MLA, Mamba, Mamba-2, Gated DeltaNet and Kimi Delta Attention hybrids, MoE, multi-token prediction (MTP)
Parallelismtensor parallelism. Pipeline parallelism 1 for hybrid models.
HostLinux x86_64, Docker or Kubernetes

Supported models lists the models that we verified on hardware and the engine version for each. Requirements has all requirements.

Integration options

There are four ways to put the ADP card under an inference engine. They share the gateway and the card. They differ in how the engine gives its KV cache to the gateway. Choosing an integration compares them.

RecommendedvLLM OffloadingSpec backendThe ADP card behind the KV offloading framework of vLLM. Fast restore, sharded reads over NVLink, safe failures. AlternativevLLM KV connectorOur KV connector for vLLM. It restores layer by layer while the model computes. AlternativeLMCache L2 pluginFor LMCache deployments: the ADP card as the L2 tier under the host-memory tier of LMCache. SGLangSGLang HiCache backendThe ADP card as the L3 tier of the hierarchical cache of SGLang.

Next steps

  1. Check the requirements for the GPU server, the card and the network.
  2. Set up the ADP card, local or remote.
  3. Install the gateway and vLLM with the OffloadingSpec backend, and check a restore.
  4. Tune with the configuration reference and the notes for your model.
  5. For many pods on many cards, read Cluster modes.
  6. When several tenants use one deployment, set up tenant isolation.
  7. Measure the effect on your hardware: Reproducing the 8×H200 benchmark.

Planning a deployment?

Talk to our engineers about your models, context lengths, concurrency and where the ADP card should sit.

Talk to Awide Labs