Configuration

Where settings live

Component	Where you set it	Reference
Storage gateway	command-line flags, or a YAML file with --config	Storage gateway
OffloadingSpec backend	xkv.* keys in kv_connector_extra_config	OffloadingSpec backend
vLLM KV connector	kv_connector_extra_config of --kv-transfer-config, and kv_transfer_params for each request	Connector
SGLang HiCache backend	--hicache-storage-backend-extra-config	SGLang HiCache backend
vLLM process	environment variables	Environment variables
vLLM	its command line	vLLM options
ADP storage service	/etc/adp/<N>/adpstore.ini on the host of the card	ADP storage service
LMCache plugin	adapter_params of the L2 adapter	LMCache plugin
Cluster service	/etc/adp/awide-cluster on the storage servers	Card cluster

Installation and Kubernetes have complete deployments.

Storage gateway

The gateway takes its settings from the command line, from a YAML file (--config <file>), or from both. A flag on the command line overrides the file. The flag is the key with dashes: shared_memory_size becomes --shared-memory-size. On the command line, sizes take B, KiB, MiB, GiB, TiB (1024-based) or KB, MB, GB, TB (1000-based). A number without a unit is bytes. In the YAML file a size is a plain number of bytes: a unit there (10GiB) stops the gateway at start with bad conversion.

# gateway.yaml: a remote card on an 8-GPU server, OffloadingSpec backend
storage_backend: remote-xdp
storage_devices:
  - /dev/nvme1n1
fragment_size: 0
port: 5557
bind_address: 127.0.0.1
shared_memory_size: 10737418240   # 10 GiB: in the YAML file, sizes are in bytes
thread_pool_size: 64
num_dbs: 128
start_clean: 0
io_timeout_ms: 5000
io_timeout_strikes: 3
log_severity: info

Storage and keys

Key	Default	What it does
storage_backend	remote-xdp	Where the objects go. remote-xdp: an ADP card over NVMe-oF, through the kernel initiator. local-xdp: an ADP card in this server. kv-spdk: ADP cards over NVMe-oF through the initiator of the gateway, required for a card cluster. file-system: a directory, to try the software without a card. loopback: stores nothing, for tests.
storage_devices	—	The connected NVMe-oF namespaces of the cards, for example /dev/nvme1n1, or their stable /dev/disk/by-id/ paths. Required for remote-xdp, and for kv-spdk without cluster_hostname. The gateway places keys by the serial numbers of the cards, not by the order of the list.
storage_slots	0 (one for each device)	The slots of the key distribution, from the number of devices to 64. With more slots than devices, a new device moves only its share of the keys. All gateways on the same cards need the same value.
key_placement	dialog	How keys go to the cards when the gateway has more than one. dialog: all blocks of a conversation on one card, for fault tolerance. stripe: the blocks of a conversation over all cards, for the read speed of one long restore. All gateways on the same cards need the same value; it is part of the key layout fingerprint. ⚠️ A change starts a new cache (placement). stripe needs the gateway and the integration of the same release.
key_layout_fingerprint	empty (no check)	The 16 hex digits that the gateway writes at start (Key layout fingerprint …). A gateway with another set of cards, slots or num_dbs refuses to start. Use it on a shared cache.
shared_storage	false	Other gateways use the same cards. The gateway then never cleans: it refuses to start with start_clean: 1 and answers CLEAN_REFUSED to a clean request of a client.
num_dbs	128	The number of databases on the card. All gateways on the same cards need the same value.
start_clean	0	1 removes the databases of the gateway at start, so the cache is empty. Use it once and on purpose. Keep 0 while key eviction is on (wiping the card).
fragment_size	0 (off), 261120 on local-xdp	The gateway cuts larger objects into fragments. Keep 0 with the OffloadingSpec and SGLang backends, which cut objects themselves. For the vLLM KV connector: 131072 on a remote card, 261120 on a local card. On local-xdp, 0 also gives 261120, because the card takes no larger object; objects of xkv.fragment_bytes: 261120 are not cut again. ⚠️ A change requires a clean cache.
fragment_spread_across_dbs	true	Put the fragments of one object in different databases, so the card reads them in parallel. ⚠️ A change requires a clean cache.

Cluster mode

These keys apply only with kv-spdk and cluster_hostname (Card cluster).

Key	Default	What it does
cluster_hostname	empty	Cluster hosts, separated by commas. The first host that answers gives the etcd members. Do not set storage_devices with it.
cluster_etcd_timeout_ms	500	The deadline of one etcd call. When etcd does not answer, the gateway keeps the last card set.
cluster_sick_polls	3	Sick polls in a row before the gateway stops to use a card.
cluster_absent_polls	3	Answered polls in a row without a card before the gateway removes its controller.

In cluster mode the gateway takes the slots of the key distribution from the full list of cards of the cluster (8 slots), not from the cards that it reaches now. So all gateways put each card in the same slot, in any order of connection, and a card that is away keeps its slot. A card that joins the cluster usually takes a free slot. When the slot that its serial number selects is taken, the gateways move the cards alike and write Cluster slot plan changed to the log; the keys under a moved slot are then misses. Card cluster.

Network and performance

Key	Default	What it does
port	5557	The control port (ZeroMQ). The ring file in /dev/shm has the port in its name, so gateways on one host need different ports.
bind_address	*	The address of the port: an IPv4 address, an interface name or *. The port answers lookups and deletes without authentication. Bind it to 127.0.0.1 when the engine shares the network namespace of the gateway (one pod, or host networking). Keep it off tenant networks.
shared_memory_size	6GiB (at least 512MiB)	The ring in /dev/shm. The ranks of the engine share it. It is a transfer window, not a cache. 10 GiB is a good value for 8-GPU servers.
thread_pool_size	4	Worker threads. Each one runs one device command at a time, so this is also the most commands in flight. The main control of restore speed: 64 on 8-GPU servers.
core_list	all cores	The CPU cores of the workers, for example "0-7,16-23".
scheduler_max_parallel_reqs	2	Read requests that run at the same time. Keep 2.
scheduler_overlap_trigger_ratio	0.8 (1.0 on local-xdp when storage_backend is in the YAML file; with --storage-backend on the command line it stays 0.8)	The share of the previous read that must be complete before the next read can start.
sequence_timeout_sec	30 (at most 33)	How long a request from some ranks of an engine can wait for the other ranks before it fails.

Failures and monitoring

Key	Default	What it does
io_timeout_ms	0 (off)	The deadline of one I/O command, in ms, on the remote-xdp, kv-spdk and block-spdk backends. When it passes, the command is aborted and its key fails (a miss or an error). The gateway takes the card out of service after io_timeout_strikes such commands in a row; the health check brings it back. 0 turns it off. Recommended: 5000, with io_timeout_strikes: 3, so a short stall of the network or the target does not take the card out. It must be below sequence_timeout_sec. With remote-xdp (a remote card through the kernel initiator), also connect the card with --fast_io_fail_tmo=2 on the GPU server.
io_timeout_strikes	3	With io_timeout_ms set: how many commands of one card in a row may pass the deadline, or fail for a reason of the card or its transport, before the card is taken out of service. Any command that the card answers starts the count again. Below the limit only the keys of those commands fail. The health check is a separate signal. 1 takes the card out at the first one. remote-xdp, kv-spdk and block-spdk.
status_file	empty (off)	A file in the Prometheus text format that the gateway writes every second. It has the health of each card, the age of the oldest I/O and the key layout fingerprint. It also has the placement (xkv_gateway_key_placement_info), the keys that the card did not hold (xkv_gateway_store_key_missing_total) and, in cluster mode, the slot plan of the cards (xkv_gateway_cluster_slot_plan_info, xkv_gateway_cluster_slots_moved).
log_dir	empty	A directory for log files. Empty means standard output only.
log_severity	info	perf, trace, debug, info, notice, warning, error or fatal.

OffloadingSpec backend

xkv.<key> entries (or a nested xkv object) in kv_connector_extra_config. The environment variable XKV_OFFLOAD_PARAMS, a JSON object, is merged last (OffloadingSpec backend). vLLM’s own keys block_size and blocks_per_chunk go in the same object, without the prefix.

Key (xkv. prefix)	Default	What it does
port	5557	The gateway.
namespace	empty	The key space of the deployment. Use one value for each model configuration on a shared card. ⚠️ Setting or changing it starts a new cache.
fragment_bytes	131072	The size of an object on the card. 131072 for a remote card (at most the I/O size of the target), 261120 for a local card.
shard_reads	auto	Sharded reads over NVLink for pages that all ranks hold the same. on refuses to start without them, off turns them off.
tp_layout	auto	auto or replicated: one object for the tensor-parallel group. per_rank: one object for each rank, with port_per_rank: true.
port_per_rank	false	A gateway for each rank, for per_rank.
affinity_chunks	1	The chunks at the start of the prompt that make the dialog mark. All chunks of one dialog go to one card.
packed_block	auto	How to store the packed block of DeepSeek-V4: auto, replica or parts.
load_failure	recompute	What vLLM does with a request whose load failed: recompute or fail.
require_cache_salt	false	Only writes a warning at start. vLLM gives the backend no salt, so the front end must mark unsalted requests with xkv_store: false.
device_staging	auto	auto: staging slots in GPU memory and one host copy. off: two copies through pinned host memory.
staging_bytes, staging_min_slots	2 GiB, 4	The pool of staging slots.
pipeline_depth	2	Waves of transfers in flight.
max_objects_per_wave	0	0 is the pool divided by pipeline_depth.
get_inflight, batch_slots	8, 1024	Read requests in flight, and ring slots in one request.
sync_lookup, sync_lookup_timeout_ms	auto, 100	Look up the keys of a new request in the scheduler step, instead of a wait of one step. auto turns it on where vLLM allows it; true stops the start where it cannot.
probe_windows	auto	Probe only the chunks of a window or recurrent group that vLLM can use, not every chunk up to the end of the prompt. off probes every key.
trust_own_stores	false	Count own stores as present without a question to the card.
probe_batch_keys, absent_ttl_s, max_index_entries	4096, 60, 1000000	The index of what the card holds.
suspend_probe_s, suspend_probe_max_s, reopen_timeout_s	2, 60, 300	After a lost gateway: the first probe, the longest pause between probes, and the time limit for a new session on all ranks.
op_timeout_s, acquire_timeout_s	120, 30	Time limits of an operation and of a slot.
draft_groups	auto	For a hybrid model with MTP: auto keeps the Mamba groups out of the draft groups, which is necessary for hits after a restart. vllm keeps the decision of vLLM.
flush_order	auto	auto completes pending stores before model runner V2 uses their blocks again. vllm keeps the order of vLLM.
timing	short	Log lines with the timing: short (one line for each load job), true, all or false.
log_level	info	The log level of the backend.

For each request, in the request body: "kv_transfer_params": {"xkv_store": false} keeps the request out of the card.

Connector

The keys of kv_connector_extra_config for the vLLM KV connector. A boolean key also takes 1 and 0, "true" and "false", "yes" and "no", "on" and "off". Other values stop the connector at start.

Key	Default	What it does
xkv_gateway_port	5557	The control port of the gateway.
chunk_size	the block size of the vLLM scheduler	Tokens in one stored object. It must be a multiple of the block size of every KV cache group. For sliding-window models, the window must be at least twice chunk_size and a multiple of it. ⚠️ A change starts a new cache.
slot_alignment_bytes	0	Round up each transfer to this many bytes. 4096 on a remote card for models with a page that is not a multiple of 4 KiB (GLM-5.x). ⚠️ A change requires a clean cache.
enable_deferred_write	true	Send the writes of decode steps under full CUDA graphs as one request for each step. false makes the connector ask vLLM for piecewise graphs, which decode slower.
layer_callbacks	auto	Whether the connector uses the layer hooks of the engine for this model. auto decides from the model and checks it against the registered layers.
aggregated_get_submission, aggregated_put_completion	true	Send reads and complete writes in batches.
synchronous_layer_get	false	Every layer waits for its own read before it computes, also where the read path does not need it. For debugging.
verify_all_layers	false	Confirm a cached prefix on every layer, not on one layer for each group.
clean_cache	false	Clean the cache when the connector starts. For tests.
log_level	INFO	Ranks other than 0 log from NOTICE unless you set a level.
num_copy_thread_block_get, num_copy_threads_per_block_get, num_copy_thread_block_put, num_copy_threads_per_block_put	SDK defaults	Launch parameters of the copy kernels.
nvlink_fanout	false	Fan out restores of MLA models over NVLink (details).
nvlink_fanout_groups	auto	The NVLink groups of ranks: auto from NVML, or explicit.
nvlink_fanout_staging_mb	256	The staging buffer of each rank, in MiB.
nvlink_fanout_min_layer_bytes	4194304	Every rank copies smaller layers whole.
key_namespace	empty	A deployment name (up to 64 bytes) in every key. ⚠️ Setting, changing or clearing it starts a new cache.
key_namespace_from_model	false	Make the namespace from the model, its revision, the tensor-parallel size, chunk_size, the KV data type and the layout. Use it on a shared cache.
require_cache_salt	false	Requests without a cache_salt do not read or write the external cache.
cache_rules	—	An override of who writes a cache, by layer class. Not for production.

For each request, in the request body: "kv_transfer_params": {"xkv_store": false} keeps the request out of the external cache: no lookup, no restore and no write.

Environment variables

Variable	Where	What it does
PYTHONHASHSEED=0	vLLM	Required. A fixed hash seed. The block hashes, and so the keys on the card, stay the same after a restart.
XKV_GATEWAY_HOSTNAME	vLLM	The host of the control port of the gateway. localhost by default. In Docker Compose, the service name of the gateway.
KV_PATH	vLLM product image	offload translates a FusIOnXConnector configuration to the OffloadingSpec backend at start. fusionx keeps it. One image carries both paths; fusionx is the default.
XKV_OFFLOAD_PARAMS	vLLM, OffloadingSpec backend	A JSON object with xkv.* settings, merged last.
VLLM_USE_V2_MODEL_RUNNER=0	vLLM 0.29, vLLM KV connector	Required on 0.29 for the connector, which works with the V1 model runner. The OffloadingSpec backend does not need it.
NCCL_LAUNCH_ORDER_IMPLICIT=1	vLLM KV connector	Recommended with NVLink fan-out (NCCL 2.26 or newer).
XKV_EXIST_REOPEN_COOLDOWN_S	vLLM KV connector	Seconds between attempts to open the lookup connection again after a timeout. 30 by default.
XKV_STEP_METRICS_EVERY	vLLM KV connector	Report the timing of the connector every this many steps. 500 by default.
XKV_SYNCHRONOUS_LAYER_GET	vLLM KV connector	Overrides synchronous_layer_get.
XKV_NVTX	vLLM KV connector	NVTX ranges around the work of the connector, for profiling.

vLLM options the integrations depend on

Option	When	Why
--enable-prefix-caching	always	The integrations serve prefix-cache hits. Hybrid models refuse to start without it.
--enable-prompt-tokens-details	recommended	Responses report cached_tokens.
--mamba-cache-mode align	Mamba, GDN and KDA hybrids	The only Mamba cache mode that the integrations accept. vLLM 0.29 and newer select it with prefix caching.
--watermark=0.02	Mamba, GDN and KDA hybrids, only on an image without the Mamba align patch	The product image and an image built from the client kit carry the patch and do not need it; the kit build log says VLLM_MAMBA_FIX_SKIPPED when the patch does not apply. Without the patch, it keeps the few free blocks that a hybrid needs when vLLM restores a prefix on a nearly full KV cache (details).
--kv-cache-dtype fp8	DeepSeek Sparse Attention models, MiniMax M3	One page size for each group.
--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'	DeepSeek Sparse Attention models on vLLM 0.29, vLLM KV connector	Under piecewise graphs the engine does not write their KV.
--max-num-batched-tokens=8192	GLM-5.x	Prevents an out-of-memory error in the sparse MLA kernel on prompts from about 128,000 tokens.
--no-async-scheduling	full + sliding-window models (Gemma 4, GLM-4.7-Flash) with deferred write, vLLM KV connector	Deferred write of a sliding-window group needs synchronous scheduling and pipeline parallelism 1. The other way is enable_deferred_write: false. The connector tells this at start.
--gpu-memory-utilization	NVLink fan-out, vLLM KV connector	About 0.005 lower on a 141 GB GPU, for the fan-out buffers.

ADP storage service

/etc/adp/<N>/adpstore.ini on the host of card N. The installer writes it. Apply a change with the gateway and the engine stopped: systemctl restart adpstore@<N>. On a storage server the target restarts after it by itself (ADP card setup).

Section	Key	Value	Why
[db_settings]	key_eviction	true	Required. The card gets space back by itself. Without it, a full card refuses every write. vLLM continues to serve, but nothing new goes to the cache, and the hit rate falls to zero. It is off unless you set it.
[db_settings]	product_type	0	Key-value mode, which the gateway uses. 1 is block mode.
[qos]	aio_read_io_size, aio_write_io_size	as delivered (131072)	The I/O size of the service. It matches the 128 KiB transfer of the NVMe-oF target.
[media], [raid], [data], [compression]	—	as delivered	The installer writes them for this card and its SSDs.
[debug]	—	as delivered	Internal. Do not change.

LMCache plugin

adapter_params of the native_plugin L2 adapter in lmcache server --l2-adapter '{…}' (LMCache L2 plugin). All keys are optional. The plugin refuses an unknown key and lists the accepted keys.

Key	Default	What it does
port, gateway_host	5557, localhost	The gateway.
fragment_bytes	131072	The I/O size and the fragment width: at least 4096 and at most the transfer limit of the device. 261120 on a local card. A change makes everything stored a miss.
verify	sample	off, sample (first and last 4 KiB of every fragment) or full.
batch_slots, get_inflight, put_inflight, exist_batch_keys	1024, 4, 2, 8192	The request size in ring slots, and the requests in flight. A request never takes more than half the ring.
copy_threads, copy_chunk_frags	8, 32	Copy and checksum workers, and the size of their jobs in fragments.
max_pending_store_mb	16384	Above this store backlog, the plugin refuses new stores. 0 means no limit.
wait_timeout_s, reopen_after_s	30, 30	How long the gateway can be silent before the plugin stops the session, and the time before a new session (0 = never).
namespace	empty	In every key: separate deployments on one card. A change makes everything stored a miss.
affinity_chunks	1	The chunks at the start of the prompt that make the dialog mark. All chunks of one dialog go to one card.
start_clean	false	Clean the card when the plugin opens. For tests only.
prefault	true	Touch the ring pages at open.
stats_interval_s, stats_path, log_task_min_objects, log_level	60, empty, 64, info	A summary line in the server log every stats_interval_s, and JSON statistics in a file.

Keep the L2 limit of LMCache (max_capacity_gb) at 0. The key eviction of the card manages the space on the card.

Settings that require a clean cache

The layout of the stored data comes from the configuration. The card does not record it. After one of these changes, wipe the card (how):

Change	What happens to the old objects
fragment_size, fragment_spread_across_dbs (gateway)	The gateway does not read them back correctly and does not report them as misses. A wipe is mandatory.
slot_alignment_bytes (connector)	The object size changes. The old objects are not usable.
chunk_size, block_size, key_namespace, xkv.namespace, xkv.fragment_bytes	Never found again. They use space until the card evicts them.
The integration (vLLM KV connector or OffloadingSpec backend)	Never found again: the two integrations use different keys.
key_placement (gateway)	Looked for on other cards: a cold cache, never wrong data. The card evicts them.
The tensor-parallel size, the model or its revision	Never found again.
The attention backend, on vLLM 0.28 with the OffloadingSpec backend	vLLM 0.28 does not give the byte layout to the backend. A wipe is mandatory.
A release with release notes that say that the stored layout changed	As the release notes say. Wipe.

A change of PYTHONHASHSEED or of the vLLM version can also change the block hashes. The cache then fills again, and nothing wrong is ever read. On a shared cache you cannot wipe the cards while other deployments use them. Change the namespace instead, and let the card evict the old objects.