ADP card setup

What runs on the host of the card

The ADP software stack manages the card on the server that holds it. For a local card this is the GPU server. For a remote card it is a storage server. The stack has these parts:

• a driver (adp.ko)
• one storage service for each card (adpstore@<N>)
• the adpcli tool
• for a remote card, one NVMe-oF target service for each card (awide-spdk-target-<N>)

ADP card requirements tells what each part is and what the host needs for it.

Install the software stack

Awide Labs delivers the ADP software stack as an installer for the kernel of the host. ADP card requirements tells what the host needs before you start: slot, SSDs, packages, kernel.

1. Install the card in a PCIe Gen5 x16 slot. Install the NVMe SSDs for the cache in the same server, on the NUMA node of the card.

2. Install the prerequisite packages.

3. Run the installer with the SSDs of the card in the command. It installs the driver, the storage service and the command-line tool. It puts the SSDs into one RAID 0 array behind the card and reserves hugepages:

   sudo ./xdp-installer install --level=0 --resources="/dev/nvme1n1 /dev/nvme2n1" --vhc=0 \
     --enable-set-hugepages --hugepage-distribution-percentage=50 --assume-yes

   When another service of the host also uses hugepages, a database for example, add --start-before <service> and --hugepages-headroom <pages> to this command. The storage service then starts before that service at boot and has a margin of pages (other users of hugepages).

4. Remote card only: install the NVMe-oF target on the address that the GPU servers connect to, and start it. Use the 128 KiB I/O size of the integrations. On a storage server with several cards, give one address and port for each card. When there are several storage servers, give each server its own box ID:

   ./awide_ai_spdk_target.py install --targets=192.0.2.10:4420 --xdp-ids 0 --awideai-box-id 0 \
     --io-size 131072 --start-service

   The target gets the serial number L<box ID>P<index> and a fixed namespace identity. The serial number identifies the card on the GPU servers. With a fixed identity, the initiators keep the device after a restart of the target.

   The listen address of the target must stay after a reboot of the host. When the network configuration of the host does not have it yet, add --persist-address=<interface>:<address>/<prefix> to the install command. The installer then writes a netplan file for the address, which takes effect at the next boot. To see what the next boot starts, run ./awide_ai_spdk_target.py check.

5. On a storage server, set the CPU and I/O tuning profile:

   sudo tuned-adm profile throughput-performance

6. Make sure that the card is up (next section). Then configure the storage service.

7. Before the first reboot of the host, check that everything starts again by itself: sudo python3 autostart.py check (Autostart after a reboot of the server).

Check the card

systemctl is-active adpstore@0          # the storage service of card 0: "active"
adpcli system get_xdp_list              # the cards on this host, their device paths and mode
adpcli system get_status                # every component, and the aggregated status
adpcli system get_disk_usage -x 0       # physical usage, valid objects, databases
adpcli raid get_raid_info -x 0          # the SSD array behind card 0
systemctl is-active awide-spdk-target-0   # remote card only: the NVMe-oF target of card 0

A healthy card reports every component OK:

Components status:
	Driver (OK): Driver is working properly
	Service (OK): Adpstore service is working properly and system is active
	RAID (OK): RAID Array is functioning properly, raidLevel: RAID0, protection: No, performance: Normal
	CapacityUsage (OK): Capacity usage is below defined thresholds
	Firmware (OK): Firmware is working properly
	Temperature (OK): Temperature is at NORMAL level
	SuperCapacitors (OK): Super Capacitors are working properly
	OnboardFlash (OK): Onboard flash component is working properly
Aggregated status is OK

The usage report shows how full the card is and how many objects and databases it holds. After a gateway connects, the number of databases is the num_dbs of the gateway (128 by default):

Physical usage [%]       : 12.40
Physical capacity        : 863.10 GB / 6.96 TB
Number of valid objects  : 10051238 / 1680330742
Number of databases      : 128 / 4096 (0 deleting)

Configure the storage service

The storage service of card N reads /etc/adp/<N>/adpstore.ini, which the installer writes. Check two keys on every card that holds the KV cache. Configuration → ADP storage service has the full list.

[db_settings]
key_eviction = true
product_type = 0                        # 0 = KV, 1 = Block

Apply a change with the model server and the gateway stopped:

systemctl restart adpstore@0
journalctl -u adpstore@0 -o cat | grep -i "key eviction"    # "Key eviction Enabled"
systemctl is-active awide-spdk-target-0                   # remote card: the target restarted after the service

The target of a card is bound to its storage service. A restart of adpstore@<N> restarts awide-spdk-target-<N> after it, and the target opens the databases of the card again.

Then connect a remote card again on the GPU servers.

Capacity and key eviction

The storage service reports usage in three zones. adpcli system get_status shows them as CapacityUsage:

Physical usage	Status	Effect
below the soft threshold (90 % by default)	OK	—
between the soft and the hard threshold (100 %)	warning	none on writes
above the maximum	CRITICAL	every write fails until there is free space. The service logs event 3002.

With key_eviction = true, the card starts to get space back before it reaches the maximum. The storage service sets the point where eviction starts. It is not a setting in the INI file. The soft and hard thresholds (adpcli system get_usage_soft_threshold, get_usage_hard_threshold) change only the reported status. Do not change them to start eviction earlier.

Watch the physical usage (adpcli system get_disk_usage) as the fill level of a cache. Alert well before the maximum. When a card fills faster than it gets space back, the hit rate falls. vLLM shows no error.

Wiping the card

A wipe empties the cache. After a change of the stored layout it is necessary (settings that require a clean cache). In other cases it only costs a refill. The gateway does it at start with --start-clean=1: it removes its databases. The storage service refuses this while key eviction is on (Key eviction is ENABLED, delete DB not supported). So do a wipe in this order:

1. Stop vLLM and the gateway. On a shared cache, stop every gateway on the card.
2. Set key_eviction = false and restart the storage service. For a remote card, also restart the target and connect the GPU servers again.
3. Start one gateway once with --start-clean=1 and without shared_storage. Wait until its log reports the databases removed. adpcli system get_disk_usage -x 0 then shows 0 objects.
4. Set the gateway back to --start-clean=0.
5. Set key_eviction = true and restart the storage service. For a remote card, also restart the target and connect again.
6. Start the gateways and the engines.

Autostart after a reboot of the server

After a reboot of the host, the card, its storage service and the model next to it come back by themselves, in this order:

driver adp ─▶ hugepages ─▶ storage service adpstore@N ─▶ NVMe-oF target or gateway ─▶ model
                             ("up and ready")

At shutdown the order is the other way: the model stops first, then the gateway, then the storage service. The storage service closes its databases on a live card, so the KV cache on the card is still there after the reboot. A long prompt that was cached before the reboot is read back from the card and is not computed again.

The installer sets up most of this:

Step	What does it
Driver at boot	/usr/lib/modules-load.d/adp.conf from the installer and the packages. The unit of the storage service also loads the driver as its first step.
Hugepages	adp_hugepages.service reserves the pages of the storage service early at boot, before any adpstore@N (--enable-set-hugepages).
Storage service	adpstore.target starts adpstore@0..N-1. A storage service that fails restarts by itself (up to 3 times in 20 minutes).
NVMe-oF target	awide-spdk-target-N starts after its adpstore@N and follows its restarts.

Two things depend on your host: other services that use hugepages, and the model.

Other users of hugepages

The storage service needs all the pages reserved for it. Another service of the host can also use hugepages, for example a database with huge_pages = try. When it starts before the storage service at boot, it takes some of the reserved pages, and the storage service stops at start (CONFIG_AllocBuffFromHugePages in its log). Two settings prevent this:

• Start the storage service before that service. The storage service then takes its pages first. At shutdown the other service stops before it.
• Reserve some pages more than the storage service needs (a headroom). This covers a restart of the other service while the storage service is down. 256 pages (512 MB with 2 MB pages) are enough for a database with a small shared buffer.

At installation, add the flags to the installer command:

sudo ./xdp-installer install ... --enable-set-hugepages --hugepages-headroom 256 --start-before postgresql

On a host that is already installed, use autostart.py, a script of the card stack installer (adp/installer/utils/autostart.py). Without --go it only shows what it would do:

sudo python3 autostart.py apply --start-before postgresql --hugepages-headroom 256
sudo python3 autostart.py apply --start-before postgresql --hugepages-headroom 256 --go

Setting	Default	What it does
--start-before <services>	empty	The services that the storage service starts before (comma list). Written to /etc/systemd/system/adpstore@.service.d/50-autostart.conf.
--start-after <services>	empty	The services that the storage service starts after, for example a storage service that the host needs first.
--hugepages-headroom <pages>	0	Pages reserved on top of the need of the storage service, on the NUMA node of the card. Written to /etc/adp/adp_hugepages.sh; the former file stays next to it. It takes effect at the next boot.

--start-before '' and --start-after '' together remove the drop-in. An uninstall of the card stack also removes it. When the host reserves hugepages itself (vm.nr_hugepages in sysctl or hugepages= on the kernel command line), make sure that the number covers the storage service and the other users together.

The model on a host with a local card

The model must start when the card is ready and stop before the storage service. Run it from a systemd unit, not with the restart policy of Docker: Docker starts its containers at boot without any order against adpstore@N. autostart.py model-unit writes this unit for a model that runs with Docker Compose:

python3 autostart.py model-unit --name model-a --workdir /srv/model-a \
    --compose-file docker-compose.yaml --env-file .env --card 0 --out /etc/systemd/system/model-a.service
sudo systemctl daemon-reload && sudo systemctl enable model-a.service

The unit waits until the card reports “up and ready” (up to 600 s, --ready-timeout), because the storage service can restart a few times at boot before it is ready. Then it runs docker compose up -d, and at shutdown docker compose stop. It starts after adpstore@N and stops before it, but it does not stop the model when the storage service restarts: the gateway takes the card back by itself. When the gateway runs as its own unit, add --gateway-unit <unit>. The model then also starts after the gateway and stops before it. --after <units> adds more units to wait for. The compose file for this unit is in Installation.

Check before and after a reboot

Before a reboot:

sudo python3 autostart.py check

It changes nothing. It checks that something loads the driver at boot, that adpstore.target and adp_hugepages.service are enabled, and how many pages are reserved. It also finds the processes that hold hugepages now, and checks that the storage service starts before their services. autostart check OK and exit code 0 mean that nothing is in the way. Otherwise it prints each problem and what to do.

After a reboot:

systemctl --failed                                    # nothing of the card stack
lsmod | grep -w adp                                   # the driver
grep HugePages_Total /proc/meminfo                    # the reserved pages
adpcli system is_ready -x 0                           # "up and ready"
systemctl status model-a                              # active (exited): the containers were started
systemd-analyze critical-chain model-a.service        # the order and the time of each step

What to expect

On a host with an A100 GPU, one card and a 27B model, two planned reboots with nothing done by hand gave:

From the reboot command
about 3.5 min	the host answers over SSH. Most of this time is the firmware of the server.
a few seconds later	the driver is loaded, the pages are reserved, the card is “up and ready”
about 5 min	the model answers requests

The KV cache on the card survived both reboots. A prompt of about 100k tokens, cached before the reboot, got its answer in about 7 s after it, with 98 % of its tokens read from the card. The same prompt computed from the start takes about 51 s. The first two rows depend on the firmware of the server; the last one adds the start of the model.

Maintenance of the card host

The storage service writes the RAID metadata of the card to disk only at a correct stop. At the next start, the service reads this metadata and then removes the file. When the service stops in another way, or when someone changes /etc/adp while the service runs, the card can lose its RAID metadata. The cache on the card is then lost.

To service the host of a card:

1. On a remote card, the gateways take the card out of service when it stops. On a local card, stop the gateway and the engine first.
2. Stop the NVMe-oF target of the card: systemctl stop awide-spdk-target-<N>.
3. Stop the storage service: systemctl stop adpstore@<N>. Make sure that systemctl is-active adpstore@<N> shows inactive. This also stops the target, if step 2 did not.
4. Do the maintenance. Copy or move files in /etc/adp only now. To go back to an earlier copy, move the directory back. Do not remove it and unpack an archive over it.
5. Start the storage service, then the target. A start of the service does not start the target: start both. Check the card with adpcli system get_status.
6. On the GPU servers, make sure that the card is connected again. Identify it by its serial number. The gateway takes the card back by itself: a local card after its storage service returns, a remote card when its namespace comes back, also under a new device name (when a card comes back).

Planned reboot of the host

When the model runs from its own unit (the model on a host with a local card), a reboot needs no manual stops. Run sudo systemctl reboot: the units stop the model, then the gateway, then the storage service. Check the host after the reboot as Check before and after a reboot shows. When the model and the gateway are not in units, stop them first as in step 1 above.

Remote card: check the fabric

Check the RDMA path between the storage server and the GPU server before you connect the card. A link below its rate, or a port without a subnet manager, shows later only as slow restores or a failed nvme discover. On both servers:

ibv_devices                                        # the RDMA devices, mlx5_0 for example
ibv_devinfo -d mlx5_0 | grep -E 'state|active_mtu|link_layer'
#   state PORT_ACTIVE · link_layer InfiniBand (or Ethernet for RoCE) · active_mtu 4096
ibstat mlx5_0 | grep -E 'State|Rate'               # InfiniBand: State Active, Physical state LinkUp, Rate 100 (EDR)
ip -br addr                                        # the address of the port: IPoIB (ib0, …) or the RoCE interface

Then measure the bandwidth with ib_read_bw from perftest, over RDMA CM (-R), as NVMe-oF connects:

ib_read_bw -d mlx5_0 -R --report_gbits                 # storage server: waits for the client
ib_read_bw -d mlx5_0 -R --report_gbits 192.0.2.10      # GPU server: close to the line rate, above 90 Gb/s on 100 Gb/s

On InfiniBand, a port in PORT_INIT instead of PORT_ACTIVE has no subnet manager on the fabric. Start one: the one of the switch, or opensm on one host. The target listens on an address (--targets of the target installer), and the GPU server connects to it (-a of nvme connect). On InfiniBand this is the IPoIB address of the port of the storage server. On Ethernet it is the address of its RoCE interface.

Remote card: connect the GPU server

The NVMe-oF target service on the storage server exports one key-value namespace for each card over RDMA. On the GPU server, connect it with the kernel initiator:

modprobe nvme_rdma
nvme discover -t rdma -a 192.0.2.10 -s 4420                 # lists the subsystem NQN of the card
nvme connect  -t rdma -a 192.0.2.10 -s 4420 -n <subsystem NQN> --ctrl-loss-tmo=600 --fast_io_fail_tmo=2
nvme list          # the ADP namespace, with the serial number of the card subsystem (for example L0P0)
nvme list-subsys   # ... rdma traddr=192.0.2.10,trsvcid=4420 live

Option	Why
--ctrl-loss-tmo=600	The kernel keeps the controller for 10 minutes while the target is away. After that time it removes the controller, and you must connect again.
--fast_io_fail_tmo=2	Commands to a card that reconnects fail after 2 seconds and do not wait for the full timeout. Use it with the io_timeout_ms setting of the gateway.

The namespace is a key-value namespace, not a block device. nvme list shows its size as 0 B, and its SMART log is empty. Both are normal. Do not partition or format it. Its device path (for example /dev/nvme1n1) goes to the gateway as --storage-devices. The gateway container needs this device and SYS_ADMIN for the NVMe pass-through commands (Installation).

Make the connection persistent as your distribution does it. The device name can change after a reconnect or a reboot. A discovery controller (/etc/nvme/discovery.conf and the nvmf-autoconnect service of nvme-cli) takes a controller number while it runs, so the card can come up as nvme1n1 instead of nvme0n1. Connect the card directly by its subsystem NQN, as above, and start this connection before nvmf-autoconnect. Give the gateway the stable path /dev/disk/by-id/nvme-<model>_<serial number>, not /dev/nvmeXnY. After a reboot, check in nvme list which device has which serial number before you start the gateway. Run this connection at boot before the kubelet or the gateway container starts, with the same --fast_io_fail_tmo=2, and retry it until the target answers.

After a restart of the target

The current target keeps a fixed namespace identity, so the GPU servers reconnect by themselves within --ctrl-loss-tmo. The gateway finds a device that comes back under a new name by its serial number. When the controller was removed, connect again on the GPU server:

nvme disconnect -n <subsystem NQN>
nvme connect -t rdma -a 192.0.2.10 -s 4420 -n <subsystem NQN> --ctrl-loss-tmo=600 --fast_io_fail_tmo=2
nvme list          # the device path can be different from before

A running gateway finds the card under its new name by itself. For the next start, put the new path in its configuration, or use the /dev/disk/by-id/ path, which does not change.

Transfer size and fragments

Placement	Limit	Object size
Local card	an object of at most 256 KiB	261120 (255 KiB)
Remote card	one NVMe command of at most MDTS (128 KiB with the target’s --io-size 131072), a multiple of 4 KiB	131072

The OffloadingSpec backend (xkv.fragment_bytes) and the SGLang backend (max_io_bytes) cut objects to this size themselves. Keep the gateway fragment_size at 0 for them. The vLLM KV connector needs the gateway fragment_size set to the value of the table. The gateway logs the device limit at start (# Max I/O Size). A fragment size that does not fit the device is an error in the log (Fragment size … does not fit the storage device); the gateway keeps serving and holds the objects to the device limit. Some models have a KV page that is not a multiple of 4 KiB (GLM-5.x). With the vLLM KV connector on a remote card, they also need slot_alignment_bytes: 4096. A change of the fragment size requires a wipe.

Durability

The card holds a cache. The system does not lose anything important when the card loses part of it. After an abrupt restart of the storage service or the target, a write that the card acknowledged can be missing. The integration then reads that chunk as a miss, and the engine computes it again. Supercapacitors on the card protect its data against a power loss. Monitor them with adpcli system get_status (SuperCapacitors (OK)).

Reading the gateway log

NVMe media error type: 2 code: 135 is the answer of the card to a lookup of a key that it does not hold. This is an ordinary cache miss, and the gateway logs it at ERROR level. Thousands of these lines during a run are normal. The card also uses the same code for other refusals, for example a full card. Judge the card by adpcli system get_status and get_disk_usage, not by the count of these lines.