Installation - ADP KV Cache Docs

This page installs the recommended integration, the vLLM OffloadingSpec backend, with Docker Compose on one GPU server. Kubernetes uses the same parts. The vLLM KV connector, the LMCache plugin and the SGLang backend have their own pages.

1. Prepare the card

Set up the card as ADP card setup tells. Make sure that adpcli system get_status reports Aggregated status is OK and that key eviction is on. Then find the device for the gateway:

Placement	Device for the gateway
Local card	`/dev/adp0` (card 0), and the log directory `/var/log/adp`
Remote card	the NVMe-oF namespace on the GPU server, for example `/dev/nvme1n1`. `nvme list` shows the serial number of the card subsystem in the SN column.

2. Get the software

The images and client kits are in the Awide Labs registry. We give an account on request. Contact us to get one.

docker login nexus.awide.io

Artifact	Coordinates
vLLM with the ADP integrations	`nexus.awide.io/awide-docker-hosted/awideai-cuda-vllm-openai:<vLLM version>-6.5.0`
Storage gateway (remote card, local card, card cluster)	`nexus.awide.io/awide-docker-hosted/akv-gateway:6.5.0`
Client kit for your own vLLM image	`https://nexus.awide.io/repository/awide-ai-bundles/awideai/akv-kit-cuda-<vLLM version>-6.5.0.tar.gz`
SGLang with the HiCache backend	`nexus.awide.io/awide-docker-hosted/awideai-cuda-sglang:v0.5.21-6.5.0`
vLLM KV connector alone, as a Python package	`akv-vllm-connector==6.5.0` (on request). It installs the module `xkv_vllm_connector`.

The current release is 6.5.0: awideai-cuda-vllm-openai:v0.28.0-6.5.0, v0.29.0-6.5.0 and v0.31.0-6.5.0, awideai-cuda-sglang:v0.5.21-6.5.0, akv-gateway:6.5.0, and the kits akv-kit-cuda-v0.28.0-6.5.0.tar.gz, akv-kit-cuda-v0.29.0-6.5.0.tar.gz and akv-kit-cuda-v0.31.0-6.5.0.tar.gz.

Use the vLLM version that Supported models gives for your model. Use the gateway and the vLLM image (or kit) of the same release. At the handshake, the gateway refuses an SDK of another release (SDK versions mismatch in its log), and the integration then stays off. One gateway image serves a remote card, a card in the GPU server and a card cluster. The --storage-backend setting selects the card.

The product image is the upstream vllm/vllm-openai image of that version with the SDK and the integrations added. It starts the OpenAI-compatible server like the upstream image.

Your own vLLM image. When you run your own vLLM image, for example a CUDA 12.9 variant or a build with your patches, use the client kit. It compiles the SDK against the Python, PyTorch and CUDA of that image and installs the integrations on top. It does not change the packages of the image:

curl -fu <user> -O https://nexus.awide.io/repository/awide-ai-bundles/awideai/akv-kit-cuda-v0.28.0-6.5.0.tar.gz
tar xzf akv-kit-cuda-v0.28.0-6.5.0.tar.gz && cd akv-kit-cuda-v0.28.0-6.5.0
./build.sh --base-image vllm/vllm-openai:v0.28.0-cu129 --tag my-registry/vllm-openai-awideai:6.5.0

3. Write the compose file

The gateway and vLLM run as two containers. The gateway makes the shared-memory ring as a file in /dev/shm. So both containers must use the /dev/shm of the host: ipc: host on both, and nothing mounted over /dev/shm. vLLM finds the control port of the gateway by its service name (XKV_GATEWAY_HOSTNAME).

The file below is for a remote card and GLM-5.3 FP8 on eight GPUs. Comments mark the values that depend on the model. Supported models has the values for the other models.

services:
  gateway:
    image: nexus.awide.io/awide-docker-hosted/akv-gateway:6.5.0
    ipc: host                          # the ring is a file in the host's /dev/shm
    cap_add: [SYS_ADMIN, IPC_LOCK]     # SYS_ADMIN: NVMe pass-through commands to the card
    ulimits:
      memlock: -1
    devices:
      - /dev/nvme1n1:/dev/nvme1n1      # the connected ADP namespace (nvme list)
    command:
      - --storage-backend=remote-xdp
      - --storage-devices=/dev/nvme1n1
      - --fragment-size=0              # the integration cuts objects to the I/O size
      - --port=5557
      - --shared-memory-size=10GiB
      - --thread-pool-size=64
      - --num-dbs=128
      - --start-clean=0                # keep the cache across restarts
      - --io-timeout-ms=5000           # a card that hangs: abort its commands after 5 s
      - --io-timeout-strikes=3         # and take it out after 3 in a row
    restart: unless-stopped

  vllm:
    image: nexus.awide.io/awide-docker-hosted/awideai-cuda-vllm-openai:v0.28.0-6.5.0
    ipc: host
    cap_add: [IPC_LOCK]
    ulimits:
      memlock: -1
    depends_on: [gateway]
    ports:
      - "8000:8000"
    deploy:
      resources:
        reservations:
          devices:
            - { driver: nvidia, count: all, capabilities: [gpu] }
    environment:
      PYTHONHASHSEED: "0"              # required: otherwise nothing is found after a restart
      XKV_GATEWAY_HOSTNAME: gateway    # where the SDK finds the gateway
    volumes:
      - /models:/models:ro
    command:
      - --model=/models/GLM-5.3-FP8
      - --tensor-parallel-size=8
      - --enable-prefix-caching
      - --enable-prompt-tokens-details
      - --kv-cache-dtype=fp8                 # model-specific
      - --max-num-batched-tokens=8192        # model-specific
      - '--kv-transfer-config={"kv_connector":"XkvOffloadingConnector","kv_connector_module_path":"xkv_offload.connector","kv_role":"kv_both","kv_connector_extra_config":{"spec_name":"XkvOffloadingSpec","spec_module_path":"xkv_offload","block_size":128,"xkv.port":5557,"xkv.fragment_bytes":131072,"xkv.shard_reads":"auto"}}'
    restart: unless-stopped

Keep the JSON of --kv-transfer-config (and of --speculative-config for MTP) in single quotes as above. Add the options of the model itself as usual: tool-call and reasoning parsers, --max-model-len.

For a local card, change the gateway, the network of vLLM and one connector value. The gateway runs in the network of the host, because it talks to the storage service of the card over the host’s loopback. It needs the device of the card, the directory /etc/adp (the card mapping, for the check of the serial number) and the log directory of the card. vLLM then also runs in the network of the host: on the vllm service, set network_mode: host, remove ports (vLLM listens on port 8000 of the host) and set XKV_GATEWAY_HOSTNAME: 127.0.0.1. Set "xkv.fragment_bytes":261120 in --kv-transfer-config.

  gateway:
    image: nexus.awide.io/awide-docker-hosted/akv-gateway:6.5.0
    network_mode: host                 # the storage service of the card is on the host's loopback
    ipc: host
    cap_add: [IPC_LOCK]
    ulimits:
      memlock: -1
    devices:
      - /dev/adp0:/dev/adp0
    volumes:
      - /etc/adp:/etc/adp:ro           # the card mapping: the check of the serial number
      - /var/log/adp:/var/log/adp
    command:
      - --storage-backend=local-xdp
      - --bind-address=127.0.0.1       # the port is on the host network: keep it on loopback
      - --port=5557
      - --shared-memory-size=10GiB
      - --thread-pool-size=64
      - --num-dbs=128
      - --start-clean=0
    restart: "no"                      # the unit of the model starts it after the card

On a host with a local card, let a systemd unit start the gateway and vLLM, not the restart policy of Docker. With restart: unless-stopped, Docker starts the containers at boot without waiting for the storage service of the card, and stops them at shutdown in no order against it. So for a local card:

set restart: "no" on the gateway and on vLLM;
give vLLM time to stop: under load it can need more than a minute after the stop signal. On the vllm service set stop_grace_period: 120s or more;
create the unit of the model with autostart.py model-unit. It starts the compose project when the card is ready and stops it before the card (Autostart after a reboot of the server).

Each gateway setting can also come from a YAML file (--config /etc/akv/gateway.yaml). A flag on the command line overrides the file. In the YAML file, sizes are plain numbers of bytes (shared_memory_size: 10737418240). With a remote card, connect it with --fast_io_fail_tmo=2 (ADP card setup), so that the deadline of --io-timeout-ms can work. --io-timeout-ms does not apply to a local card. Configuration has all settings.

4. Start

docker compose up -d
docker compose logs -f gateway vllm
curl -f http://127.0.0.1:8000/health

The first start of a large model takes some time, because vLLM loads the weights and captures the CUDA graphs. The vLLM log then shows the key prefix of the backend (spec built) and, above one GPU, the NVLink groups of the sharded reads (sharded reads: links ... -> groups ...).

5. Check a restore

Only a hit from the card proves a restore. Send a long prompt and restart only vLLM. The gateway and the card keep the cache. Then send the prompt again.

MODEL=/models/GLM-5.3-FP8
PROMPT=$(python3 -c "print(' '.join(f'Line {i}: the ADP card keeps this prompt.' for i in range(3000)))")
ask() {
  curl -s http://127.0.0.1:8000/v1/completions -H 'Content-Type: application/json' \
    -d "$(jq -n --arg m "$MODEL" --arg p "$PROMPT" '{model: $m, prompt: $p, max_tokens: 8}')" \
    | jq '.usage | {prompt_tokens, cached: .prompt_tokens_details.cached_tokens}'
}

ask                                # cold: cached is 0, the prompt is computed and written to the card
docker compose restart vllm        # GPU memory is empty now, the card is not
until curl -sf http://127.0.0.1:8000/health; do sleep 10; done
time ask                           # warm: cached is close to prompt_tokens, and the answer comes much faster

cached_tokens counts whole chunks only. With block_size 128, a prompt of 30,100 tokens can hit at most 30,080. A prompt shorter than one chunk is never stored. /metrics shows the hits from the card and the transfers:

vllm:external_prefix_cache_queries_total   vllm:external_prefix_cache_hits_total
vllm:kv_offload_load_bytes_total           vllm:kv_offload_xkv_card_state

When cached_tokens stays at 0 after the restart, see Operations & troubleshooting. To compare with vLLM without the card, follow Reproducing the 8×H200 benchmark.

Stop and upgrade

docker compose down keeps the cache on the card. With --start-clean=0, the next start finds it. For an upgrade, see Operations → Upgrades.

Planning a deployment?

Talk to our engineers about your models, context lengths, concurrency and where the ADP card should sit.

Talk to Awide Labs