Card cluster

When to use

• Cards and storage servers join and leave while the models serve, and the pods must not restart for that.
• Many gateways on many storage servers need one view of the cards and their health, kept in one place.
• Needs: three or more hosts for etcd (storage servers or witness hosts), the kv-spdk backend with hugepages and the RDMA devices in each pod. Up to 8 cards.

When not to use

• The set of cards changes rarely: a shared cache needs no cluster service and no etcd.
• One deployment on one server: use one card for the pod (Installation).

All modes side by side: Choosing a mode.

What it is

In a shared cache, each gateway has a fixed list of cards in its configuration. A card cluster moves this list into a service:

• The cluster service (awide-cluster) runs on every storage server and on witness hosts.
• The hosts keep an etcd quorum. The leader of the quorum writes there which storage servers and cards exist, their NVMe-oF addresses and their health.
• Each gateway in cluster mode reads the card set from etcd about once a second. It connects a new card and removes a card that is not healthy, while the model pods continue to serve.

The rest is the same as in a shared cache:

• one gateway for each pod
• the same key distribution in every gateway, with all blocks of one dialog on one card
• a key namespace for each deployment
• when a card fails, misses only for the dialogs on that card

Architecture

Part	Where it runs	What it does
Cluster service	every storage server and witness host	Asks the card CLI for the cards and their NVMe-oF addresses every 6 s. Writes them to etcd under /targets/<host>/. Serves the list of etcd members on port 5000.
etcd member	the same hosts, started by the cluster service	Keeps the cluster state. Three members tolerate one failure, five members tolerate two.
Leader	one cluster service, chosen by etcd	Reads the card exporters and the ping exporters through Prometheus. Writes the health of each card: 0 healthy, 1 sick, 2 unknown. Keeps the size of the quorum: it removes a member only after the member is bad for BAD_MEMBER_STRIKES cycles in a row (3 by default) and on a fresh check right before the removal. A new leader does not mark a node sick for a missing metric in its first LEADER_WARMUP_CYCLES cycles (3 by default).
Card exporter, ping exporter	every storage server	Report the state of each card service and the packet loss between the data addresses.
Witness exporter	a witness host	Makes a host without cards the third etcd member. Gateways never connect to it. A host with an ADP card does not need it.
Gateway in cluster mode	each model pod	Reads the etcd members from a cluster host, then the card set and the health from etcd. Connects to the cards with its own NVMe-oF initiator (the kv-spdk backend).

What happens on a failure

The cluster changes the card set only when the cause is clear. A problem of the control path does not remove cards.

Event	What the gateways do
etcd does not answer, or the quorum is lost	They keep the last card set and its health. Each call to etcd has a deadline (cluster_etcd_timeout_ms, 500 ms), so the poll does not stop.
The etcd of one host stops, or loses its data	The quorum continues on the other members. The cluster service of that host replaces its etcd member through the other members and starts etcd clean, and the host is a member again within seconds. A gateway that asked that host logs Cluster stale, then Cluster answered again.
The leader loses its card metrics	The leader writes no new health, or writes unknown. An unknown card keeps the health that it had.
A card is sick	The gateways remove it from the key distribution after cluster_sick_polls sick polls in a row (3 by default). Its dialogs are misses. The dialogs on the other cards restore in full.
The card is healthy again	The gateways use it again at once and find its dialogs again. A gateway also finds a card that comes back under a new device name, by its serial number.
A card disappears from etcd	The gateways remove its controller after cluster_absent_polls answered polls in a row (3 by default).
A storage server stops	The NVMe-oF keep-alive takes its cards out of service. The keys on other servers stay available. With three etcd members, the quorum continues.
A new card or server joins	The cluster service writes it to etcd. The gateways connect it at the next polls. The slot of each card comes from the full list of cards of the cluster, so every gateway puts the new card in the same slot (details).

Cards in and out at run time

What the gateways do on their own, with no restart of the pods and no change of their configuration:

Change in the cluster	What the gateways do
A card gets sick, or its NVMe-oF listener goes away	They stop to use it after cluster_sick_polls or cluster_absent_polls polls. Its conversations are misses; the others restore in full.
The card is healthy again, or its listener comes back	They use it again at the next poll, in the same slot. Its conversations are hits again.
A storage server is removed with clustercli remove_node, or added with clustercli add_node	They remove or connect its cards at the next polls.
A card that the cluster never had joins	They connect it at the next polls. It takes a free slot. When its slot is taken, the gateways move the other card alike and log Cluster slot plan changed; the keys under the moved slot are misses.

To take a card out on purpose, stop its storage service correctly (Operate it): the leader marks it sick, and the gateways stop to use it. Start it again, and the gateways take it back. At most 8 cards have a slot; a ninth card is logged as an error and not used. The status file of each gateway has the slot plan (xkv_gateway_cluster_slot_plan_info, the same on every gateway of one cluster) and the count of moved slots (xkv_gateway_cluster_slots_moved).

The three modes side by side: Shared cache → compared with other modes.

Advantages and limits

Advantages:

• Cards and storage servers join and leave while the models serve.
• One place has the card set and the health of every card. All gateways get the same view.
• A failure of etcd or of the monitoring does not remove cards from the gateways.
• All blocks of one dialog stay on one card. A sick card costs only its dialogs, also when the card set changes.

Limits:

• The kv-spdk backend only. The gateway runs its own NVMe-oF initiator. It needs hugepages and the RDMA devices in the pod.
• At most 8 cards in the key distribution of a cluster.
• Three or more hosts for etcd.
• No QoS and no replication, as in a shared cache.

Requirements

On each storage server and witness host:

• Docker.
• The ADP software stack with its monitoring: a Prometheus container and the card exporter on port 7891. A witness host runs the witness exporter on port 7891.
• Open ports between the hosts: 5000 (cluster API), 2379 and 2380 (etcd), 9090 (Prometheus), 9427 (ping exporter), 7891 (card exporter).
• Host names that every host and every gateway can resolve.
• The images quay.io/coreos/etcd:v3.5.17 and awideai/ping-exporter:1.0. On hosts without registry access, copy them with docker save and docker load.

On each GPU server:

• An RDMA network to every storage server, and the RDMA devices (/dev/infiniband).
• Hugepages of 2 MiB for the gateway.

Set it up

1. Prepare the storage servers

Install the NVMe-oF target of every card as for a shared cache, with a unique box ID on each server (Shared cache, step 1). Turn on key eviction on every card.

2. Install the cluster service on every host

Use the bundle from Awide Labs. It is not in the registry: we send it on request.

sudo ./install-awide-cluster.sh

The bundle installs the images, the clustercli tool, the systemd unit awide-cluster, the settings in /etc/adp/awide-cluster and the Prometheus configuration. The service can also run as a container, from the image that the bundle installs:

docker run -d --name awide-cluster --restart always --network host \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /etc/adp:/etc/adp -v /var/lib/etcd:/var/lib/etcd -v /var/log/adp:/var/log/adp \
  -v /srv/docker/awide/monitoring:/srv/docker/awide/monitoring \
  awideai/awide-cluster:1.0

On a witness host, run witness_exporter.py from the bundle on port 7891 instead of a card exporter.

3. Create the cluster

On the first host:

clustercli create_cluster --name kv-cache

On every other host:

clustercli add_node -n <first host>

The leader adds one host to the etcd quorum in each cycle of 6 s.

4. Check the cluster

clustercli get_cluster
docker exec etcd etcdctl member list

What	Expected
each storage server	enabled: 1
each card	health_xdp: 0 and the serial number of its subsystem
the quorum	"etcd_quorum": {"members": 3, "tolerated_failures": 1}
etcdctl member list	three members started

5. Configure the gateways

# gateway.yaml: cluster mode
storage_backend: kv-spdk
cluster_hostname: storage-1,storage-2,witness-1   # asked in this order
cluster_etcd_timeout_ms: 500
cluster_sick_polls: 3
cluster_absent_polls: 3
shared_storage: true
start_clean: 0
num_dbs: 128              # the same on every gateway
io_timeout_ms: 5000       # the deadline of one I/O command
io_timeout_strikes: 3     # commands in a row past it before the card leaves service
port: 5557                # unique on each GPU server
bind_address: 127.0.0.1
shared_memory_size: 10737418240   # 10 GiB, in bytes: a unit is not read in YAML
thread_pool_size: 4
core_list: "100-104"      # thread_pool_size + 1 cores
status_file: /run/akv/status.prom

Key	Why
cluster_hostname	Cluster hosts, separated by commas. The first host that answers gives the etcd members. Do not set storage_devices with it.
cluster_etcd_timeout_ms	The deadline of one etcd call. When etcd does not answer in time, the gateway keeps the last card set.
cluster_sick_polls	Sick polls in a row before the gateway stops to use a card.
cluster_absent_polls	Answered polls in a row without a card before the gateway removes its controller.
core_list	The CPU cores of the NVMe-oF initiator of the gateway: thread_pool_size plus one.

The key layout fingerprint changes when a card joins or leaves. Compare it between gateways only while the cluster is stable.

6. Run the gateway on Kubernetes

The pod is the pod of Kubernetes with these changes in the gateway container:

spec:
  hostNetwork: true                 # the gateway reaches the cards over the RDMA network
  hostIPC: true
  hostAliases:                      # when DNS does not resolve the cluster hosts
    - { ip: 192.0.2.10, hostnames: [storage-1] }
    - { ip: 192.0.2.11, hostnames: [storage-2] }
    - { ip: 192.0.2.12, hostnames: [witness-1] }
  containers:
    - name: gateway
      securityContext:
        privileged: true
      resources:
        requests: { hugepages-2Mi: 3Gi, memory: 1Gi }
        limits:   { hugepages-2Mi: 3Gi }
      volumeMounts:
        - { name: hugepages, mountPath: /dev/hugepages }
        - { name: infiniband, mountPath: /dev/infiniband }
  volumes:
    - { name: hugepages, emptyDir: { medium: HugePages-2Mi } }
    - { name: infiniband, hostPath: { path: /dev/infiniband } }

7. Configure the integration

The connector settings are the same as for a shared cache. Set a namespace for each model configuration. Use the port of the gateway in the same pod (Shared cache, step 4).

Verify it

1. The gateway log has Using cluster mode and Initializing NVMeSpdkBackend with <N> storage devices, where N is the number of cards in clustercli get_cluster.
2. In the status file, xkv_gateway_storage_groups_active is the number of healthy cards.
3. Send a long prompt, restart the inference engine, and send the prompt again. The second response has cached_tokens near the prompt length.

Monitor it

Signal	Where	What it tells you
health_xdp, etcd_quorum	clustercli get_cluster	The health of each card, and how many failures the quorum tolerates.
xdp_service_status	card exporter, port 7891	The state of the storage service of each card.
ping_loss_percent	ping exporter, port 9427	Packet loss between the data addresses of the hosts.
xkv_gateway_storage_group_active{group="<serial>"}	status file of each gateway	Which cards the gateway uses now.
Cluster stale and answered again	gateway log	etcd did not answer, and the gateway kept the last card set.
xkv_gateway_cluster_slot_plan_info{fingerprint="…"}	status file of each gateway	The slot plan of the cards. Alert when the gateways of one cluster show more than one value.
Cluster slot plan changed	gateway log	The cards of the cluster changed, and a card moved to another slot.

Operate it

• Remove a host with clustercli remove_node -n <host> on any host of the cluster. A host that left while it was down keeps a disabled record. clustercli add_node on it joins it again.
• Do not clean the cache at the start of a gateway. Keep start_clean: 0. With shared_storage: true the gateway refuses a clean.
• Stop the storage service correctly before maintenance of a card. Stop the NVMe-oF target of the card and then adpstore@<N> with systemctl stop (maintenance). The leader then marks the card sick, and the gateways stop to use it.
• Keep three etcd members healthy. When tolerated_failures is 0, the next failure stops the cluster service. The gateways then keep the last card set, but they do not see new changes. When the hosts come back, the quorum comes back by itself with the same members.