Operations & troubleshooting

What to monitor

Signal	Where	What it tells you
cached_tokens	usage.prompt_tokens_details of each API response (with --enable-prompt-tokens-details)	The hit of one request, from GPU memory and the card together.
vllm:external_prefix_cache_queries_total, vllm:external_prefix_cache_hits_total	vLLM /metrics	The hit rate of the card: hits divided by queries over a time window. vllm:prefix_cache_* is the GPU tier.
vllm:kv_offload_xkv_card_state	vLLM /metrics, OffloadingSpec backend	0 when the backend serves from the card. Above 0 for longer than 30 s, the external cache does not come back. Page on it.
vllm:kv_offload_xkv_load_misses, vllm:kv_offload_xkv_store_failed	vLLM /metrics, OffloadingSpec backend	Objects that the card did not return, and stores that it refused.
External KV cache turned off on this rank	vLLM log, vLLM KV connector	The gateway stopped to answer. The external cache stays off until you restart the gateway and vLLM together. Page on it.
sglang:storage_prefetch_hit_tokens_total	SGLang /metrics	Tokens that SGLang read from the card.
xkv_gateway_storage_group_active, xkv_gateway_worker_oldest_io_age_seconds	status file of the gateway (status_file)	Which cards the gateway uses, and an I/O that does not complete.
xkv_gateway_key_layout_info	status file of the gateway	The key layout fingerprint. On a shared cache, all gateways must show the same value.
Physical usage, CapacityUsage	adpcli system get_disk_usage, adpcli system get_status on the host of the card	How full the card is. Alert well before the maximum (capacity).
Aggregated status	adpcli system get_status	Driver, service, RAID, firmware, temperature, supercapacitors, on-board flash.
New kinds of errors	gateway log	Look at all errors other than the code: 135 misses below.
Restarts	the gateway and engine containers	With the vLLM KV connector, a restart of the gateway needs a restart of vLLM (Kubernetes).

Compare hit rates with what is possible. A prompt of N tokens can hit at most floor(N / chunk_size) × chunk_size tokens, and the first request of each new prefix is always a miss. Compare the rate with and without the card on the same traffic, not against 100 %.

No hits after a restart

A long prompt comes back with cached_tokens: 0 after a restart of vLLM. Check these items in this order:

1. PYTHONHASHSEED has the same value in the vLLM environment as before. Without it, each process hashes blocks differently.
2. The prompt is longer than one chunk and the same as before, also the chat template and the system prompt.
3. The stored layout did not change since the engine wrote it: model and revision, tensor-parallel size, chunk size, namespace, integration, fragment settings (the list).
4. The gateway kept the cache. It starts with --start-clean=0, and only vLLM restarted. When an operator or a runtime class renders the gateway command, look at the command of the running gateway.
5. The salt did not change, when the front end sets one. The same tenant must get the same cache_salt after the restart. With require_cache_salt: true, a request without a salt is never cached.
6. The card accepts writes. adpcli system get_status shows CapacityUsage (OK), and the physical usage grows while new prompts arrive.
7. The integration started. The OffloadingSpec backend writes spec built with its key prefix. The vLLM KV connector writes XKV: N cache tensors registered for exchange. The log has no line about a lost gateway.

The hit rate falls during service

Cause	How it shows	What to do
The card is full and key eviction is off	CapacityUsage (CRITICAL) and event 3002 in the log of the storage service. vLLM continues to serve, but caches nothing new.	Turn on key eviction. Wipe the card if it must be empty now.
The target of a remote card restarted	The NVMe device does not answer. The gateway takes the card out of service.	The GPU server reconnects within --ctrl-loss-tmo. Otherwise connect again.
The gateway stopped	card_state above 0 (OffloadingSpec), or External KV cache turned off on this rank (connector)	OffloadingSpec: the backend opens a new session when the gateway answers. Connector: restart the gateway and vLLM together.
A gateway started with a clean	The physical usage of the card falls to zero at the start of a pod.	Set --start-clean=0 in every place that starts the gateway.
A gateway with another card set on a shared cache	More than one value of xkv_gateway_key_layout_info	Set the same storage_devices, storage_slots, num_dbs and key_layout_fingerprint on all gateways.
New traffic, new prefixes	The rate comes back as the cache fills.	Nothing. Make the cache large enough for the working set.

Troubleshooting

Hybrid models stop with “Cannot get N free blocks”

ValueError: Cannot get 2 free blocks from the pool

Hybrid models (Mamba, Mamba-2, Gated DeltaNet, Kimi Delta Attention) need a small reserve of free KV blocks. vLLM uses it when it admits a request with a prefix from the card. Each Mamba-type group takes one block for the restored state, and the admission check of vLLM does not count it. The product image does not need it. With another vLLM image, run these models with --watermark=0.02. This value keeps int(W × num_gpu_blocks) blocks free at admission. The reserve must be at least the number of Mamba groups G:

int(W × num_gpu_blocks) ≥ G

0.02 covers three Mamba groups from a pool of 150 blocks. For a smaller pool or more groups, use W ≥ G / num_gpu_blocks. num_gpu_blocks is a label of vllm:cache_config_info on /metrics. The start log has one group K: MambaSpec line for each Mamba group. The reserve is about 2 % of the pool. Models with attention only, sliding window, MLA or DSA do not need it.

The gateway refuses the engine: VERSION_MISMATCH

The gateway log shows SDK versions mismatch. Application version: <A>, SDK Version: <B>, and the integration turns the external cache off. The gateway and the engine image (or the kit for it) come from different releases. Deploy both from the same release.

The gateway refuses a second engine

Each gateway serves one engine. A second engine on the same gateway gets a refusal. Give each engine its own gateway and port.

Failed to open KVDatabase

vLLM and the gateway do not see the same /dev/shm, and the gateway made the ring in its own. Run both with the IPC namespace of the host (ipc: host, hostIPC: true), and remove each volume at /dev/shm.

The gateway does not start on a shared card

What the log tells	Cause	What to do
a clean at start is refused with shared_storage	A clean at start would remove the data of other deployments.	Set start_clean: 0.
the key layout fingerprint is different	This gateway has another card set, other slots or another num_dbs.	Make the configuration the same as on the other gateways.
Failed to open device	The device path is not the card, or the card is not connected.	Find the card by its serial number in nvme list, and connect it.

A hybrid model refuses to start

• Without --enable-prefix-caching, vLLM turns the Mamba cache off, and the integration refuses to start.
• The integrations accept only --mamba-cache-mode align. Nemotron 3 Nano selects another mode unless you set it.
• Pipeline parallelism above 1 is refused for models with several groups for each role.

A sliding-window model refuses to start (vLLM KV connector)

Deferred write of a sliding-window group (Gemma 4, GLM-4.7-Flash) needs synchronous scheduling and pipeline parallelism

1. Add --no-async-scheduling, or set enable_deferred_write: false at some cost in decode speed. The start message names the group.

The integration refuses to start and names a cache

KV cache registry: 1 of 4 registered cache(s) have no writer the connector can reach; first of each kind: ...

The model has a cache that the integration cannot exchange. The integration refuses to start, so that it does not cache nothing in silence. Tell us the model and the line.

Out of memory on long prompts with GLM-5.x

Use --max-num-batched-tokens=8192. With 16384, the sparse MLA kernel runs out of memory on prompts from about 128,000 tokens.

Slow decode of GLM-5.x on vLLM 0.29 or 0.31

Run GLM-5.x on vLLM 0.28. On vLLM 0.31 with the product image, GLM-5.x decodes at about 82 % of the speed of 0.28 on repeated requests with a warm cache, and at the same speed without cache hits. Use the product image: the upstream images of vLLM 0.29 and 0.31 decode it much slower on servers with two NVLink islands (Supported models).

The gateway log says that the card service does not answer

local-xdp: the card service of XDP <N> does not answer (...) for <S> s

A gateway with a local card waits for the storage service of the card, adpstore@<N>. Check it with systemctl status adpstore@<N>. A service process that was stopped (SIGSTOP) answers again when it continues, and the gateway goes on without a restart. When the service hangs, restart it with systemctl restart adpstore@<N>. The gateway takes the card back by itself (when the card is unavailable).

Long pauses on the first requests of DeepSeek-V4-Flash and V4.1-Flash

The engine compiles kernels at the first use, which pauses generation for many seconds a few times. Send typical requests before the service starts, and keep VLLM_CACHE_ROOT on a persistent volume, so the compiled kernels stay after a restart.

The gateway log has many lines NVMe media error type: 2 code: 135

These are ordinary misses: the key is not on the card. Judge the card by its status and usage, not by the number of these lines (details).

Upgrades

1. Read the release notes. When a release changes the stored layout, the notes say so, and you must wipe the card.
2. Pull the new images by their versioned tags. Never deploy a moving tag.
3. Upgrade the gateway and the engine image together, because the gateway accepts only an SDK of its own release. A gateway of a new release refuses the clients of an older one (SDK versions mismatch in its log), and their integration stays off. So upgrade every vLLM and SGLang image and every kit that uses the gateway at the same time. Stop the engine, then the gateway. Start the gateway, then the engine.
4. Check a restore (Installation).

A new vLLM version or another PYTHONHASHSEED can change the block hashes. The cache then fills again from new traffic, and nothing wrong is ever read. The same holds when a release changes the key format of an integration: the cache starts cold, and the card evicts the old objects by itself, with no wipe. Gateways that share cards and use stripe placement need the gateway and the integration of the same release. Code that calls the SDK directly imports it as xkv_awide.

For maintenance of the host of a card, see ADP card setup.

Measuring the effect

• Measure under concurrency. Numbers from one request at a time are a lower bound. They say little about a serving deployment, because batching changes the throughput much more than a cache setting.
• Measure a restore after a restart. Hits from GPU memory and hits from the card look the same in cached_tokens. Restart vLLM (only vLLM) before the warm run to measure the card.
• Alternate the setups (A, B, A, B) with at least three runs each, in quiet periods. Run the cold case only on a newly started server, because a second run is already warm.
• Do not compare answers byte for byte. The engine does not give exactly the same output each time, with or without a cache. Kernels are tuned again at each start, and a cached prefix can take another numeric path than a computed one. Judge a deployment by cached_tokens, latency and the correctness of the answers.

Reproducing the 8×H200 benchmark applies these rules to a complete A/B test.