Shared cache on several cards

When to use

• Replicas of a model, or several models, on several GPU servers share one KV cache: a prompt that one pod wrote restores on another. Models and clients stay apart: each model has its own key namespace, and the cache_salt of a tenant is part of every key (Tenant isolation).
• The cache needs the capacity and the read bandwidth of more than one card, also for a single pod.
• The set of cards changes rarely. A change means a new list in every gateway and a restart of the pods.
• Needs: remote cards over NVMe-oF on RDMA, the same storage_devices, storage_slots and num_dbs in every gateway.

When not to use

• One deployment on one server with one card: use one card for the pod (Installation).
• Cards and storage servers come and go, and the pods must not restart for that: use a card cluster.

All modes side by side: Choosing a mode.

What it is

In the basic setup, one pod uses one card. In a shared cache, many pods use one set of cards:

• Every pod has its own storage gateway, as in the basic setup.
• Every gateway opens the same ADP cards over NVMe-oF. The cards can be in one storage server or in several.
• Every gateway puts a dialog on the same card. So a replica of a model on another pod or another server finds the KV cache that its neighbour wrote. It restores the prompt instead of computing it again.
• All blocks of one dialog stay on one card. When a card fails, the dialogs on the other cards restore in full.
• Different models and different deployments keep their keys apart with a key namespace.

The cards add up. The capacity of the cache is the sum of the cards. Different dialogs restore from different cards at the same time.

Architecture

The parts:

Part	Where it runs	What it does
Model pod	a GPU server	vLLM or SGLang with the ADP KV cache integration.
Storage gateway	a container in the same pod	Serves one inference engine. It puts each key on a card and in a database of that card.
NVMe-oF initiator	the kernel of each GPU server	Connects the NVMe-oF subsystem of each card. A card shows as a device /dev/nvmeXnY.
NVMe-oF target	each storage server, one service per card	Exports the card over RDMA. The subsystem of each card has its own serial number.
ADP card	each storage server	Keeps the objects on its NVMe SSDs.

How a dialog finds its card

The engine restores a prompt from its start, block after block. It stops at the first block that it does not find, and it computes all blocks after that block again. So the gateway keeps all blocks of one dialog on one card (the default placement, dialog; the other one is below):

1. The connector marks each block with the dialog that it belongs to: the hash of the start of the prompt.
2. The gateway hashes this dialog mark. The hash selects a slot of the key distribution. Each card has one or more slots, by its serial number. The gateway does not use the order of the devices in its configuration.
3. All blocks of the dialog go to the card of that slot. Inside the card, they spread over its databases.

All gateways with the same set of cards, the same storage_slots and the same num_dbs put a dialog on the same card. All layers and fragments of all blocks of a dialog go to one card. When a card leaves, only the dialogs on that card are lost. The dialogs on the other cards restore in full.

A gateway with a different set of cards finds only part of the cache. It does not read wrong data, but the loss of hits is silent. The key layout fingerprint prevents this. It is a short hash of the cards, the slots and num_dbs. Each gateway writes it to its log at start. Give the same value to every gateway in key_layout_fingerprint. A gateway with a different layout then refuses to start.

Spare slots make a change of the card set cheaper. With one slot for each card, a fifth card moves most keys of four cards to other cards. With more slots than cards (storage_slots, for example 16 for six cards), a new card takes only its share of the dialogs.

The connector setting xkv.affinity_chunks sets how much of the start of the prompt makes the dialog mark. The default is 1: the first chunk of the prompt. Many dialogs often start with the same long system prompt. With 1, all these dialogs go to one card. Then increase xkv.affinity_chunks to the number of chunks of the common start plus one. The chunks of the common start can then go to other cards, so one dialog uses at most xkv.affinity_chunks + 1 cards.

Placement of conversations on the cards

Which one. dialog is the default: for fault tolerance, because a failed card costs only its own conversations. stripe is for the speed of one long restore, which then reads from all cards at once; a failed card cuts most long conversations short until they are computed again.

The gateway setting key_placement selects how the blocks of a conversation go to the cards. It applies to every gateway with more than one card: several cards behind one gateway, a shared cache and a card cluster. Choose it for each deployment:

	dialog (default)	stripe
Use it for	fault tolerance	the read speed of one long restore
One conversation	all its blocks on one card, chosen by the dialog mark	its blocks over all cards; each block, with its description, on one card
One long restore	reads from one card	reads from all cards at the same time
One of N cards fails	about one conversation in N is a miss and is computed again in full; the others restore in full	most long conversations miss from their first block on that card and are computed again from there
The engine	continues to serve	continues to serve
The card comes back	its conversations are hits again	its blocks are found again

Both placements spread many conversations evenly over the cards, skip a card that is out of service and use it again when it comes back. The keys are the same in both: a gateway that changes the placement looks for them on other cards, so the cache starts cold, and it never reads wrong data.

Set it on every gateway on the same cards:

key_placement: stripe        # or dialog, the default; the same on every gateway

• The placement is part of the key layout. With key_layout_fingerprint set, a gateway with the other placement refuses to start. The status file has xkv_gateway_key_placement_info{placement="..."}.
• stripe needs the gateway and the integration of the same release.
• With dialog, xkv.affinity_chunks (above) moves only the common start of many prompts to other cards. A very high value gives every chunk its own mark and spreads a conversation as stripe does.
• SGLang pages have no fragments. With stripe each page goes to a card by itself.

Isolation between deployments

Two levels keep data apart:

Level	What separates it	Who sets it
Models and deployments	the key namespace, a string in every key	the connector setting of each deployment
Tenants of one model	the cache_salt of each request, in vLLM’s or SGLang’s block hash	the API front end (Tenant isolation)

Give each model configuration its own namespace, and give all replicas of it the same namespace. Replicas then share the cache, and models do not. The gateway, the shared cards and the cluster service do not change the salt. They only route the key that the engine made.

When a card fails

The cache is a cache, not storage. When a card fails, the dialogs on that card are misses. The engine computes them again one time, and the gateway writes them to a card that works. The dialogs on the other cards restore in full. With N cards, the failure of one card affects about one dialog in N. The engine does not stop and does not read wrong data.

Event	What happens
A card stops answering, or its commands fail	The health check of the gateway (every second) takes out a card that does not answer. With io_timeout_ms set, io_timeout_strikes commands in a row that pass the deadline or fail in the transport also take it out. Below that, only the keys of those commands fail. Without io_timeout_ms, the commands to a card that hangs wait for the kernel timeout. Its dialogs are misses, and their new writes go to the other cards.
The card comes back	The gateway checks each card every second and uses the card again when it answers. Under the same device name, it uses the card again at once, or after 10 s when an I/O error took the card out. Under a new device name, the gateway finds the card by its serial number and model in 5 s or less, and takes it only when it answers Identify with the same serial number and model; another card behind the same node stays unused. You do not restart the pods. The dialogs on the card are hits again.
A card is not available when the gateway starts	The gateway starts with the other cards. Its log has is not available at startup, and xkv_gateway_storage_group_active is 0 for that card. The gateway uses the card when the card answers.
The gateway of one pod stops	Only that pod loses its external cache. The other pods are not affected.
A storage server stops	All its cards are out. The dialogs on the other servers stay available.

There is no replication. The engine computes a dialog of a failed card again, and the gateway writes it to a card that works.

Compared with other modes

	One card per pod	Shared cache	Card cluster
Cards of a gateway	one device	a list of devices, the same in every gateway	read from the cluster service, changed at run time
Gateway backend	remote-xdp	remote-xdp	kv-spdk
Replicas share the cache	no	yes	yes
A card fails	that pod has no external cache	the dialogs on that card only	the dialogs on that card only, the cluster removes it
Add a card	restart the pod	restart all pods with the new list	no restart
Card health	the gateway alone	each gateway alone	the leader of the cluster, from the card and ping exporters
Extra parts	none	none	a cluster service on three or more hosts

Which mode to choose: Choosing a mode.

Advantages and limits

Advantages:

• A request that goes to another replica restores its prefix from the cards.
• Capacity and read bandwidth grow with each card.
• A card failure costs only the dialogs on that card. All other dialogs restore in full.
• No central metadata server: each gateway computes the card of a key by itself.

Limits:

• No QoS. All deployments share the bandwidth and the capacity of the cards. One deployment can evict the objects of another and make its restores slower.
• One engine for each gateway. A second engine on the same gateway gets a clear refusal.
• No replication. The engine computes the dialogs of a failed card again.
• One card for each dialog. One dialog restores from one card. A very busy dialog uses the bandwidth of one card. With key_placement: stripe one dialog reads from all cards, at the price of more recomputation when a card fails.
• A new namespace is a new cache. When you set or change the namespace, the deployment finds nothing that it wrote before. Set the namespace when the deployment first joins the shared cards.
• A change of the card set moves dialogs. Until the cache fills again, the moved dialogs are misses.

Requirements

• An ADP card for each storage server slot, set up as a remote card (ADP card setup).
• An RDMA network between every GPU server and every storage server. Plan it for the sum of all restores at the same time.
• On each storage server, the NVMe-oF target from the current ADP software stack. It gives each card a fixed namespace identity and accepts many hosts.
• On each GPU server, nvme-cli and the nvme_rdma kernel module.
• In each pod, the storage gateway and the inference engine of the same release.

Set it up

1. Prepare the storage servers

Install the NVMe-oF target for each card. Give each storage server its own box ID, so that the serial numbers of all cards are different:

# storage server 1: cards 0 and 1, serial numbers L0P0 and L0P1
./awide_ai_spdk_target.py install --targets=192.0.2.10:4420,192.0.2.10:4421 --xdp-ids 0,1 \
  --awideai-box-id 0 --io-size 131072 --start-service
# storage server 2: serial numbers L1P0 and L1P1
./awide_ai_spdk_target.py install --targets=192.0.2.11:4420,192.0.2.11:4421 --xdp-ids 0,1 \
  --awideai-box-id 1 --io-size 131072 --start-service

The installer writes /etc/awide/spdk/config/target_<N>.conf for each card. Check three values there:

Key	Value	Why
SPDK_SER_NUM	unique on all storage servers	The gateways place keys by this serial number.
NS_UUID, NS_NGUID	set	A fixed namespace identity. Without it, the identity changes at each start of the target, and every initiator loses the device.
IO_SIZE	131072	The largest transfer. The fragment size of the connector must not be larger.

The target accepts any host. To limit the hosts, use a storage network that only the GPU servers can reach.

Turn on key eviction on every card.

2. Connect every GPU server to every card

modprobe nvme_rdma
nvme connect -t rdma -a 192.0.2.10 -s 4420 -n <NQN of card L0P0> --ctrl-loss-tmo=600 --fast_io_fail_tmo=2
nvme connect -t rdma -a 192.0.2.10 -s 4421 -n <NQN of card L0P1> --ctrl-loss-tmo=600 --fast_io_fail_tmo=2
nvme connect -t rdma -a 192.0.2.11 -s 4420 -n <NQN of card L1P0> --ctrl-loss-tmo=600 --fast_io_fail_tmo=2
nvme connect -t rdma -a 192.0.2.11 -s 4421 -n <NQN of card L1P1> --ctrl-loss-tmo=600 --fast_io_fail_tmo=2
nvme list        # the SN column shows the serial number of each card

Option	Why
--ctrl-loss-tmo=600	The kernel keeps the controller for 10 minutes while the target is away. After that time it removes the controller, and you must connect again.
--fast_io_fail_tmo=2	Commands to a card that reconnects fail after 2 seconds. They do not wait for the full controller loss timeout.

Make the connections persistent as your distribution does it. The device names can be different on each server and can change after a reconnect or a reboot: a discovery controller (/etc/nvme/discovery.conf, the nvmf-autoconnect service) takes a controller number while it runs, so a card can come up as nvme1n1 instead of nvme0n1. So:

• Connect each card directly by its subsystem NQN, as above, and start these connections before nvmf-autoconnect.
• Give the gateway the stable path of each card, /dev/disk/by-id/nvme-<model>_<serial number>, not /dev/nvmeXnY.
• After a reboot, check in nvme list which device has which serial number before you start the gateways.
• Run these connections at boot before the kubelet or the gateway containers start, with the same --fast_io_fail_tmo=2, and retry each one until the target answers.

3. Configure every gateway

Every gateway on the shared cards gets the same list of cards, the same distribution and the same fragment settings:

# gateway.yaml: the same on every pod, except the port
storage_backend: remote-xdp
storage_devices:            # every card, in any order
  - /dev/nvme0n1
  - /dev/nvme1n1
  - /dev/nvme2n1
  - /dev/nvme3n1
storage_slots: 16           # more slots than cards, the same on every gateway
num_dbs: 128                # the same on every gateway
shared_storage: true        # nobody can clean the shared cards
start_clean: 0              # never clean at start
key_layout_fingerprint: "" # first start: empty, then the value from the log
fragment_size: 0            # OffloadingSpec and SGLang backends; 131072 for the vLLM KV connector
fragment_spread_across_dbs: true
io_timeout_ms: 5000         # the deadline of one I/O command
io_timeout_strikes: 3       # commands in a row past it before the card leaves service
status_file: /run/akv/status.prom
port: 5557                  # unique on each GPU server
bind_address: 127.0.0.1
shared_memory_size: 10737418240   # 10 GiB, in bytes: a unit is not read in YAML
thread_pool_size: 64

Key	Why
storage_devices	The same cards in every gateway. The order does not matter, because the gateway uses the serial numbers. Use the /dev/disk/by-id/ paths, which do not change after a reboot.
key_placement	dialog (default) or stripe, the same on every gateway (placement).
storage_slots	The number of slots of the key distribution. Use more slots than cards. Then a new card moves only its share of the dialogs.
shared_storage: true	A clean removes every key on a card, also the keys of other deployments. With this setting the gateway never cleans. It refuses to start with start_clean: 1 and answers CLEAN_REFUSED to a clean request of a client.
key_layout_fingerprint	The first gateway writes Key layout fingerprint <16 hex digits> to its log. Put this value in the configuration of all gateways. A gateway with another layout then refuses to start.
io_timeout_ms, io_timeout_strikes	The deadline of one I/O command, and how many commands in a row may pass it before the gateway takes the card out of service. Measure the command latency under load first. Use a value between 5000 and 10000, below sequence_timeout_sec.
port	The shared-memory ring of the gateway has the name of its port. Two gateways on one server need two ports.

The status file has the health of each card (xkv_gateway_storage_group_active{group="<serial number>"}) and the fingerprint (xkv_gateway_key_layout_info{fingerprint="…"}). Give it to Prometheus with a node exporter text-file collector or a sidecar.

4. Configure the integration in every pod

Give each model configuration its own namespace. Give the same namespace to all replicas of it. Use the port of the gateway in the same pod.

vLLM with the OffloadingSpec backend (recommended):

--kv-transfer-config '{"kv_connector":"XkvOffloadingConnector",
  "kv_connector_module_path":"xkv_offload.connector","kv_role":"kv_both",
  "kv_connector_extra_config":{"spec_name":"XkvOffloadingSpec","spec_module_path":"xkv_offload",
    "block_size":128,"xkv.port":5557,"xkv.namespace":"glm-5.3-fp8-tp8",
    "xkv.fragment_bytes":131072,"xkv.shard_reads":"auto","xkv.affinity_chunks":1}}'

A good namespace is the model name with its parallelism, for example glm-5.3-fp8-tp8.

vLLM with the vLLM KV connector: set "key_namespace_from_model": true. The connector then makes the namespace from the model, its revision, the tensor-parallel size, chunk_size, the KV data type and the layout.

SGLang: set model_id (and optionally key_namespace) in the configuration of the HiCache backend (SGLang HiCache backend). Also set "hicache_storage_pass_prefix_keys": true, so that all pages of one dialog go to one card. Set affinity_chunks as xkv.affinity_chunks of vLLM.

LMCache: set affinity_chunks in the adapter_params of the plugin, or LMC_XKV_AFFINITY_CHUNKS (LMCache L2 plugin).

5. Run it on Kubernetes

The pod is the pod of Kubernetes with these changes:

• The gateway container gets every card. Mount the host’s /dev and /sys/class/nvme in it. The gateway then finds a card again when the card comes back under a new name.
• The gateway port is unique on each node. Use one port for each pod on a node, and give the same port to the integration.
• The rendered gateway command has --start-clean=0. Some operators and runtime classes use 1 as the default. Each new pod then removes all data on the cards. Check the command of a running pod after each change of the platform.
• The pod lands only on nodes that connect all cards. Use a node label and a node selector.

Verify it

1. All gateways see all cards. The log of each gateway has Group Health: active for each serial number. In the status file, xkv_gateway_storage_groups_active is the number of cards.

2. All gateways have the same layout. The fingerprint is the same in all status files. Alert on more than one value:

   count(count by (fingerprint) (xkv_gateway_key_layout_info)) > 1

3. Replicas share. Send a long prompt to replica 1. Send the same prompt to replica 2. The response of replica 2 has cached_tokens near the prompt length.

4. Models do not share. Send the same prompt to a deployment with another namespace. Its response has cached_tokens equal to 0.

5. Nobody can clean. A clean request through any gateway gets CLEAN_REFUSED. The gateway writes that nothing was deleted.

Operate it

• Do not clean the cache at the start of a gateway. Keep start_clean: 0 in every manifest, chart and runtime class. To empty the shared cards, stop every gateway on them. Then do one clean from one gateway that runs without shared_storage (wiping the card).
• Identify cards by serial number. A device name like /dev/nvme1n1 can change after a reconnect. Scripts and node labels must find the card by the serial number of its subsystem.
• Stop the storage service correctly before maintenance. The storage service saves the RAID metadata of the card only at a correct stop. Stop the NVMe-oF target and then adpstore@<N> with systemctl stop. Do not copy, move or remove /etc/adp while the service runs (maintenance).
• Change the card set on all gateways together. Put the new list and the new fingerprint in all pods, then restart them. Until the cache fills again, the moved keys are misses.
• Watch the health of the cards. Alert on xkv_gateway_storage_group_active equal to 0 and on a growing xkv_gateway_worker_oldest_io_age_seconds. The label group of the metric is the serial number of the card. The gateway writes moved from active to inactive to its log when it takes a card out. It writes moved from inactive to active when the card comes back. It writes reopened as /dev/... when the card comes back under a new device name.