SGLang HiCache backend

What it is

SGLang keeps the KV cache in three tiers when its hierarchical cache (HiCache) is on:

Tier	Where	Who manages it
L1	GPU memory	SGLang
L2	pinned host memory	SGLang
L3	a storage backend	the backend, here the ADP card through the storage gateway

The ADP HiCache backend (XkvHiCacheStorage) is an L3 backend for stock SGLang. SGLang loads it through its dynamic storage backend. There is no fork and no patch of SGLang. SGLang copies pages from L3 to L2 before a request needs them, and writes new pages from L2 to L3 in the background.

The pages on the card stay there after a restart of SGLang. A long prompt that comes back after the restart is read from the card. It is not computed again.

How it plugs in

• One client for the server. The gateway has one shared-memory ring for each port. The gateway serves one client: a second client on the same gateway is refused, and the first keeps working. So TP rank 0 owns the only gateway client of the SGLang server. The other ranks send their keys to rank 0. They exchange pages with it through a memory area that SGLang’s ranks share. Nothing is left in /dev/shm when the processes stop.
• A replicated page is read once. For MLA models and DeepSeek-V4, all ranks hold the same KV page. Rank 0 writes it, and the backend reads it from the card once for all ranks. Pages that each rank holds alone (attention heads, Mamba state) are keyed by rank.
• Objects fit the I/O limit of the card. The card takes at most 128 KiB in one transfer over NVMe-oF. The backend cuts each page into objects of at most max_io_bytes (131072 by default). A page is a hit only when all its objects are on the card.
• One lookup for each page. The backend asks the card for the last object of each page, and asks once for all ranks. SGLang stops a prefetch that does not get an answer in about a second, so a fast lookup is necessary.
• The client opens before the first request. On a card, the gateway client needs about 10 s to open. The backend opens it in the background when SGLang has registered its host pools (open_at_start). The first request after a restart then finds the client open.

Pools that go to the card

SGLang 0.5.21 gives these pools to L3, by architecture:

Architecture	Example	Pools
MLA with the DSA indexer	GLM-5.x	kv, indexer, and with MTP also draft and draft_indexer
Hybrid GDN or KDA	Qwen 3.8, Kimi Linear	kv, mamba
Sparse attention with an indexer	MiniMax M3	kv, indexer
Compressed MLA	DeepSeek-V4	kv, swa and the compressed pools
Sliding window	Gemma 4	kv, swa

Turn it on

The image with SGLang 0.5.21 and the backend is nexus.awide.io/awide-docker-hosted/awideai-cuda-sglang:v0.5.21-6.5.0 (Installation). Run the storage gateway next to the SGLang server as for vLLM (Installation). Then start SGLang with HiCache and the backend:

python -m sglang.launch_server --model-path <model> --tp 8 \
  --enable-hierarchical-cache \
  --hicache-size 72 \
  --hicache-storage-backend dynamic \
  --hicache-storage-backend-extra-config '{"backend_name": "xkv", "module_path": "xkv_sglang_hicache",
    "class_name": "XkvHiCacheStorage", "interface_v1": 1, "gateway_port": 5557, "model_id": "glm-5.3-fp8",
    "hicache_storage_pass_prefix_keys": true, "affinity_chunks": 1}'

Setting	Why
--enable-hierarchical-cache	Turns on L2 and L3.
--hicache-size or --hicache-ratio	The size of the L2 pool on each rank. Make it large enough for the working set. A page that the card finds needs a free L2 slot. Without one, SGLang drops the hit.
--hicache-storage-backend dynamic	SGLang loads the backend by module and class name.
gateway_port	The port of the gateway of this server.
model_id	Part of every key. Give each model its own value. Set it when the same weights come from different paths.

The gateway settings are the same as for vLLM. On a remote card keep the gateway fragment_size at 0, because the backend already cuts objects to 128 KiB.

Settings

All keys go into --hicache-storage-backend-extra-config. SGLang’s own keys of this object (for example prefetch_threshold) go into the same object.

Key	Default	What it does
gateway_port	5557	The port of the gateway.
model_id	SGLang’s model name	Part of every key. Two models never share a page.
key_namespace	empty	A key space of the deployment, at most 64 bytes. Use it on a shared cache.
affinity_chunks	1	How many pages at the start of the prompt make the dialog mark. All pages of one dialog go to one card (how a dialog finds its card).
hicache_storage_pass_prefix_keys	false in SGLang	SGLang’s own key. Set it to true. SGLang then gives the backend the pages before each page, and the backend finds the dialog of the page.
max_io_bytes	131072	The largest object. Keep 131072 for an ADP card.
max_pages_per_request	128	The most objects in one gateway request.
objects_per_request	0	Objects in one request. 0 means 32 with several requests in flight.
max_inflight_requests	4	Gateway requests in flight.
open_at_start	auto	Open the gateway client in the background before the first request.
open_at_start_delay_s	2.0	Seconds without a new pool before the backend opens the client.
lookup_open_wait_s	0.5	How long a lookup waits for the client to open. After that time the lookup is a miss.
tp_exchange	auto	Tensor parallelism above 1 through the client of rank 0. off refuses tensor parallelism above 1.
shard_reads	auto	Read a replicated page once for all ranks. off reads it on each rank.
exchange_bytes	2 GiB	The pinned host memory of rank 0 for the pages of the other ranks.
exchange_timeout_s	40	How long rank 0 waits for a page exchange.
failure_cooldown_s, failure_cooldown_max_s	1, 60	After a failed request, the backend stops to use the card for this time. The time doubles after each failure in a row.
start_clean	false	Empty the card when the first client opens. For tests only.
trace	false	One log line for each call that SGLang makes into the backend.

Size the gateway’s shared_memory_size for max_inflight_requests × objects_per_request objects of 128 KiB.

Check it

1. The SGLang log has the line gateway client opened ahead before the first request.
2. Send a long prompt. Restart SGLang only, not the gateway. Send the prompt again.
3. The second response reports cached tokens. /metrics shows the pages from L3 in sglang:storage_prefetch_hit_tokens_total.

Failures

The backend never raises an error into SGLang. An error would stop a prefetch or backup thread of SGLang.

• When the gateway is down or does not answer, a read is a miss and a write is lost. The backend stops to use the card for the cooldown time. SGLang serves from L1 and L2 and computes the rest.
• When the gateway is not up at the start of SGLang, the backend opens the client at the first request after the cooldown.
• When rank 0 cannot get the shared memory for the page exchange, all L3 calls are misses. The log shows an error.
• /flush_cache of SGLang does not remove the data on the card.

Isolation

SGLang names a page by a chain hash of its tokens. The chain starts from the request’s extra_key and cache_salt. So the salt of a tenant is in every key on the card, as with vLLM. SGLang has no field like xkv_store: false to keep a request out of the card. The front end must salt every request (Tenant isolation).

Limits

• One SGLang server for each gateway. Give each server its own gateway and port.
• Pipeline parallelism 1, context parallelism 1, --dp-size 1, no DP attention. The backend refuses to start otherwise and names the dimension.
• SGLang refuses L3 with --dcp-size above 1, and HiCache with --enable-int8-mamba-checkpoint. The no_buffer strategy of Mamba needs --page-size 1.
• L3 is not layer by layer. SGLang copies a prefix from L3 to L2 before the request starts.