ADP card

Hardware

The card

	Requirement
Slot	PCIe Gen5 x16 (the card is a PCIe Gen5 x16 device).
Form factor	Low-profile, half-height half-length (HHHL, 6.6” × 2.536”). Tall and short brackets for full-height and low-profile slots.
Power	Under 25 W typical, 45 W at most, +12 V through the PCIe slot — no auxiliary power cable.
Cooling	Operating temperature 10–52 °C at 250 LFM of airflow across the card. Storage 5–35 °C, below 90 % humidity, non-condensing.
Power-loss protection	On-board supercapacitors. No battery and nothing to install.
Servers	Any standard x86_64 server with a free slot: Dell, HPE, Lenovo, Supermicro, Quanta, Wiwynn, Inspur, Sugon, Fujitsu, Hitachi, Tyan, MiTAC, Intel, Cisco, AIC.

GPU servers often control their fans by the GPU temperature. Make sure that the slot gets the airflow above also when the GPUs are idle.

The SSDs behind the card

The card stores the cache on NVMe SSDs in the same server. At installation the SSDs are assigned to the card and combined into one array behind it.

	Requirement
Interface	NVMe (PCIe Gen3, Gen4 or Gen5). The card also drives SAS and SATA SSDs. For the KV cache, use NVMe.
Type and vendor	TLC or QLC flash from any vendor (Samsung, WD, Micron, Intel, Kioxia, Hynix, Seagate, …).
Array	RAID 0 across the SSDs of one card. The content is a cache that can always be recomputed, so there is no need to pay for redundancy.
Capacity	Up to 128 TB of raw SSD capacity per card. The usable cache is about 90 % of the raw capacity: two 3.84 TB SSDs give about 7 TB.
Dedicated	The SSDs belong to the card: no file system, no partitions, not the boot disk. Keep the operating system on a disk of its own.
Placement	On the same NUMA node (CPU socket) as the card.

How much capacity. Plan from the prefixes that you want to keep, not from the request rate. Take the KV size of one token of your model. Multiply it by the tokens of all long prompts, conversations and agent contexts that must come back without a prefill. The usage report of the card (adpcli system get_disk_usage) shows the stored objects and their average size once a model has run for a while. When the card is full, it evicts objects by itself, and the hit rate is what the capacity allows (capacity and key eviction).

Where the card goes

Placement	Hardware
Local — in the GPU server	A free PCIe Gen5 x16 slot in the GPU server and room for the SSDs there. No network.
Remote — in a storage server	A storage server with the card and its SSDs (below), and an RDMA network between it and the GPU server. The GPU server needs only the network card.

A remote card keeps storage off the GPU servers, and the cache can be on hardware sized for it. Several GPU servers can share remote cards (Cluster modes). A local card is the simplest setup, with nothing between the gateway and the card.

Several cards in one server

Put one card per NUMA node: in a two-socket server, a card on each socket, each with its own SSDs on that socket. Every card has its own storage service. In a remote setup it also has its own NVMe-oF target and network address. Several engines on one GPU server can each use a separate card, or all engines can share the cards (Shared cache).

Storage server for a remote card

	Requirement
CPU	x86_64. The NVMe-oF target service polls its CPU cores all the time. Give every card a set of dedicated cores on its NUMA node. These cores show as 100 % busy, which is normal.
Memory	Enough for the operating system plus the hugepages the installer reserves for the target service. When another service of the host also uses hugepages, for example a database, the storage service must start before it, or the reservation needs a headroom for it (other users of hugepages).
Network	An RDMA-capable NIC per card, on the card’s NUMA node (below).
Disks	The SSDs of the cards, and a separate boot disk.
Remote management	IPMI or another out-of-band console is strongly recommended: the target runs on bare metal.

Network for a remote card

	Requirement
Transport	NVMe over Fabrics on RDMA: RoCE v2 over Ethernet, or InfiniBand.
NIC	An RDMA NIC on the storage server and on the GPU server. On the storage server, on the same NUMA node as its card.
Speed	100 Gb/s or faster per card (InfiniBand EDR or faster). The restore speed of a long prompt is capped by the slower of the link and the SSDs.
MTU	RoCE v2: jumbo frames, 9000, end to end, on NICs and switch ports. InfiniBand: the link MTU that the subnet manager sets, 4096 where possible (active_mtu in ibv_devinfo). There are no jumbo frames to configure.
Addressing	One IP address and port (for example 4420) for the target of each card. On InfiniBand, this is the IPoIB address of the port of the storage server. The GPU servers must reach it. Tenants must not.

A slower link works, but restores are slower. For RoCE, configure the switches as your NIC vendor recommends for RDMA traffic. For InfiniBand, a subnet manager must run on the fabric: the one of the switch, or opensm on one host. Each port with NVMe-oF needs an IPoIB interface with an address, because RDMA connections use the IP address also on InfiniBand. Check the fabric before connecting the card.

Software

The ADP software stack

All software of the card comes in one package from Awide Labs, the ADP software stack. You install it on the host of the card: the GPU server for a local card, the storage server for a remote card.

Component	What it is	On the host as
Driver	The kernel module of the card. Built for a specific kernel version.	kernel module adp.ko (a DKMS package builds it for the running kernel)
Firmware	The card’s firmware, delivered with the stack.	reported by adpcli system get_status
Storage service	Keeps the key-value databases on the card. One instance for each card. Starts at boot.	adpstore@<N> (systemd), config /etc/adp/<N>/adpstore.ini
Command-line tool	Status, usage, the SSD array, maintenance.	adpcli
NVMe-oF target service (remote card only)	Exports the key-value namespace of the card over RDMA. One instance for each card, with a fixed namespace identity and a unique serial number. Bound to the storage service of its card: a stop or a restart of adpstore@<N> stops or restarts the target with it.	awide-spdk-target-<N> (systemd), port 4420
Installers	Install the parts above and make the SSD array. The target installer installs the NVMe-oF target.	xdp-installer, awide_ai_spdk_target.py

On the GPU server, the integrations and the gateway come as container images (Installation). With a local card, the gateway container talks to the storage service on the same host. With a remote card, no part of the stack runs on the GPU server. Only the NVMe-oF initiator of the kernel runs there.

Use the ADP software stack of the same release as the gateway image. The gateway of a card in the GPU server talks to the storage service of the card by the names of this stack: adpstore, /dev/adp<N>, /etc/adp and /var/log/adp.

Operating system and kernel

Host	Requirement
Host of the card	Linux x86_64. The driver is built for the exact kernel of the host. Tell Awide Labs your distribution and kernel version, and you get the stack for it. A kernel update needs a stack for the new kernel, so pin the kernel on these hosts.
GPU server, remote card	Linux x86_64 with the NVMe-oF RDMA initiator in the kernel (nvme_rdma). All current distribution kernels have it.

Packages to install first

On the host of the card, the installers need these packages. The names are as in RHEL-family distributions. The Debian family has equivalents.

Package	Version	Why
nvme-cli, pciutils, libpciaccess	any	Finding and managing the card and its SSDs.
libaio, numactl, numactl-devel	any	I/O and NUMA placement of the storage service.
ledmon	0.96 or newer	Drive LEDs of the SSD array.
fuse3, fuse3-devel, libuuid-devel, ncurses-devel	any	Required by the stack’s services and tools.
python3, python3-pip	3.8 or newer	The installers. The target installer also needs the pip modules meson and pyelftools.
libibverbs, libibverbs-devel, librdmacm-devel	any	RDMA for the NVMe-oF target (remote card).
infiniband-diags, perftest	any	Checking the fabric (remote card): ibstat, ib_read_bw.
tuned	any	The throughput-performance profile of a storage server.

The target installer also installs the dependencies of the NVMe-oF target with RDMA support itself.

On the GPU server with a remote card:

Package or setting	Why
nvme-cli	nvme discover, nvme connect, nvme list.
kernel modules nvme_rdma, rdma_ucm	The NVMe-oF initiator over RDMA.
nvme_core.multipath=N (kernel parameter)	The gateway sends NVMe pass-through commands to the namespace device. Native NVMe multipath must be off.
the RDMA user-space stack of your NIC (rdma-core or the vendor’s driver package)	RDMA on the NIC.
kernel module ib_ipoib (InfiniBand)	The IP address the RDMA connection is set up by.
infiniband-diags, perftest	Checking the fabric: ibstat, ib_read_bw.

Installing and checking it

ADP card setup installs the stack and checks the card and its services. It also sets the two keys of the storage service that the KV cache needs: key eviction on, and key-value mode. Every key is listed in Configuration → ADP storage service.