Choosing an integration

All integrations store the KV cache on the card through the storage gateway. They differ in how the engine gives the KV cache to the gateway and gets it back.

The options

vLLM OffloadingSpec backend (recommended). A backend for the KV offloading framework of vLLM. vLLM handles the layout of each model, and the backend stores bytes under keys. It reads replicated pages once and shares them over NVLink. When the gateway is lost, it pauses and does not fail. Details.

vLLM KV connector (alternative). Our own connector for the KV connector interface of vLLM. It restores a prompt layer by layer while the model computes. Use it when you need that overlap, for example for hybrid models on one GPU, or on vLLM 0.26. Details.

LMCache L2 plugin (alternative). For deployments that already run LMCache. The ADP card becomes the L2 tier of the LMCache multiprocess server, below the host-memory tier of LMCache. LMCache itself does not change. Details.

SGLang HiCache backend. The ADP card as the L3 tier of the hierarchical cache of SGLang, with stock SGLang. Details.

Comparison

	OffloadingSpec backend	vLLM KV connector	LMCache L2 plugin	SGLang HiCache backend
Use	recommended for vLLM	alternative	alternative	SGLang
Plugs into	the offloading framework of vLLM	the KV connector interface of vLLM	the LMCache multiprocess server	the HiCache storage interface of SGLang
Engine side	XkvOffloadingConnector with XkvOffloadingSpec	FusIOnXConnector	LMCacheMPConnector	XkvHiCacheStorage
Models	every model of vLLM’s framework, verified list	the verified list	what LMCache supports	what HiCache supports
Restore path	card → GPU staging → GPU, the whole prefix before the step	card → ring → GPU, layer by layer	card → LMCache host tier → GPU	card → host pool (L2) → GPU
Host memory	the shared-memory ring	the shared-memory ring	the LMCache host tier and the ring	the L2 pool and the ring
Engine versions	vLLM 0.28, 0.29, 0.31	vLLM 0.28, 0.29 (0.26 on request)	vLLM 0.28 with LMCache 0.5.4 or 0.5.5	SGLang 0.5.21
Tensor parallelism	one object for replicated pages, sharded reads over NVLink	NVLink fan-out for MLA	as LMCache	through the client of rank 0
Delivered as	the product image and the client kit	the product image and the client kit	an image, on request	the image awideai-cuda-sglang

How to choose

• Start with the OffloadingSpec backend. It supports the most models and engine versions.
• Plan for the OffloadingSpec backend on new vLLM versions. The vLLM KV connector needs the V1 model runner of vLLM, which vLLM deprecated in 0.31 and plans to remove in 0.32. From vLLM 0.32 on, the OffloadingSpec backend is the only integration for vLLM. The connector stays supported on vLLM 0.28 and 0.29, and on 0.26 on request.
• Choose the vLLM KV connector when the overlap of the restore with the computation matters more. On a hybrid model on one GPU it restores faster. It is also the integration for vLLM 0.26 (on request: no product image or kit is published for it).
• Choose the LMCache plugin when LMCache is already part of your serving stack. A prompt that is still in the host tier of LMCache comes back a little faster than from the card. A prompt from the card pays for two copies, from the card to host memory and then to the GPU.
• Choose the SGLang backend when you serve with SGLang.

The gateway, the card setup and the gateway configuration are the same for all integrations. For many pods on many cards, see Cluster modes.