Kubernetes

On Kubernetes, the gateway runs as a second container in the model pod. All of Installation applies. This page tells how the two containers share memory and how they restart. Any operator or chart that makes the pod below works the same way.

What the pod needs

Requirement	How
The gateway next to the engine	A second container in the same pod: one gateway for each engine.
One /dev/shm for both	hostIPC: true on the pod, and no emptyDir or other volume at /dev/shm in either container. With a private /dev/shm, vLLM shows Failed to open KVDatabase, and the gateway reports the memory as allocated.
A unique gateway port on each node	The ring is a file in the /dev/shm of the node, with the gateway port in its name. Two pods on one node need different --port values, and the integration needs the same port (xkv.port).
Access to the card	The gateway container is privileged, so it can open the device of the card and send it pass-through commands. A local card is /dev/adp0. A remote card is the connected NVMe-oF namespace. Connect a remote card on the node, outside the pod (ADP card setup).
The device after a reconnect	Mount the /dev and /sys/class/nvme of the host in the gateway container. A remote card can come back under a new device name, and the gateway then finds it by its serial number.
The control port on loopback	The containers of one pod share the network namespace. Bind the gateway to 127.0.0.1 (--bind-address=127.0.0.1). The SDK connects to localhost by default.
No clean at start	--start-clean=0 in the command of the gateway.
A fixed hash seed	PYTHONHASHSEED=0 in the vLLM container.
Access to the registry	An image pull secret for nexus.awide.io.
Scheduling	The pod must go to a node with the card (local) or the connected namespace (remote). Use a node selector or an affinity on a label that you set on these nodes.

Example

kubectl create secret docker-registry awide-registry \
  --docker-server=nexus.awide.io --docker-username=<user> --docker-password=<token>

apiVersion: apps/v1
kind: Deployment
metadata:
  name: glm53-adp
spec:
  replicas: 1
  selector:
    matchLabels: { app: glm53-adp }
  template:
    metadata:
      labels: { app: glm53-adp }
    spec:
      hostIPC: true                              # the ring is in the node's /dev/shm
      nodeSelector:
        example.com/adp-card: "true"             # a label that you put on the nodes with the card
      imagePullSecrets:
        - name: awide-registry
      containers:
        - name: gateway
          image: nexus.awide.io/awide-docker-hosted/akv-gateway:6.5.0
          args:
            - --storage-backend=remote-xdp
            - --storage-devices=/dev/nvme1n1
            - --fragment-size=0
            - --bind-address=127.0.0.1
            - --port=5557
            - --shared-memory-size=10GiB
            - --thread-pool-size=64
            - --num-dbs=128
            - --start-clean=0
            - --io-timeout-ms=5000               # the deadline of one I/O command
            - --io-timeout-strikes=3             # late commands in a row before the card leaves service
          securityContext:
            privileged: true                     # the card's device and NVMe pass-through
          volumeMounts:
            - { name: host-dev, mountPath: /dev }
            - { name: sys-nvme, mountPath: /sys/class/nvme, readOnly: true }
        - name: vllm
          image: nexus.awide.io/awide-docker-hosted/awideai-cuda-vllm-openai:v0.28.0-6.5.0
          args:
            - --model=/models/GLM-5.3-FP8
            - --tensor-parallel-size=8
            - --enable-prefix-caching
            - --enable-prompt-tokens-details
            - --kv-cache-dtype=fp8
            - --max-num-batched-tokens=8192
            - '--kv-transfer-config={"kv_connector":"XkvOffloadingConnector","kv_connector_module_path":"xkv_offload.connector","kv_role":"kv_both","kv_connector_extra_config":{"spec_name":"XkvOffloadingSpec","spec_module_path":"xkv_offload","block_size":128,"xkv.port":5557,"xkv.fragment_bytes":131072,"xkv.shard_reads":"auto"}}'
          env:
            - { name: PYTHONHASHSEED, value: "0" }
          ports:
            - containerPort: 8000
          readinessProbe:
            httpGet: { path: /health, port: 8000 }
            periodSeconds: 10
          resources:
            limits:
              nvidia.com/gpu: 8
          volumeMounts:
            - { name: models, mountPath: /models, readOnly: true }
      volumes:
        - name: models
          hostPath: { path: /models }
        - name: host-dev
          hostPath: { path: /dev }
        - name: sys-nvme
          hostPath: { path: /sys/class/nvme }

For a local card, the gateway container takes the same image. Use --storage-backend=local-xdp instead of the NVMe-oF device, set hostNetwork: true on the pod (the gateway talks to the storage service of the card over the node’s loopback), add hostPath volumes for /etc/adp (read-only: the card mapping, for the check of the serial number) and /var/log/adp, and set "xkv.fragment_bytes":261120. Keep --bind-address=127.0.0.1: with hostNetwork, the port is on the network of the node. The gateway and vLLM images always come from the same release.

Check the shared memory once after the first start:

kubectl exec deploy/glm53-adp -c gateway -- sh -c 'echo ok > /dev/shm/adp-shm-check'
kubectl exec deploy/glm53-adp -c vllm -- cat /dev/shm/adp-shm-check     # must print "ok"
kubectl exec deploy/glm53-adp -c gateway -- rm /dev/shm/adp-shm-check

Restarts

The gateway and the engine are one unit.

• OffloadingSpec backend. When the gateway container restarts, the backend pauses the external cache. It opens a new session when the gateway answers again. You do not need to restart the pod. vllm:kv_offload_xkv_card_state above 0 for longer than 30 s shows that the cache does not come back.

• vLLM KV connector. When the gateway container restarts, recreate the pod (kubectl delete pod …), so that vLLM connects to the new gateway. Until then, vLLM serves from GPU memory and logs External KV cache turned off on this rank. To automate this, alert on the restart count of the gateway container or on this log line.

• When only the vLLM container restarts and the gateway runs, nothing else is necessary. This is the restore path of Installation.

• A reboot of the node. The first start after a reboot is cold: the weights come from disk and the kernels compile again. With a large model it can take many times longer than a restart. Give the startup probe of the engine (startupProbe) enough time for it, and keep the compile caches (VLLM_CACHE_ROOT, and the Torch Inductor and Triton caches) on a disk volume, not in /dev/shm. Connect the remote cards before the pod starts (ADP card setup).

Several pods and several cards

Each pod has its own gateway, also when many pods use the same cards. The setup is in Shared cache and Card cluster.

Security

The control port of the gateway answers lookups and deletes without authentication. /metrics and the KV events of vLLM show the behaviour of the cache. Keep the gateway on loopback as above. Do not expose /metrics, the KV events or the gateway port outside the pod. For a service with several tenants, see Tenant isolation.