GPU server

Hardware

	Requirement
Server	x86_64 with NVIDIA GPUs that the engine supports.
GPUs	The upstream vLLM images support compute capability 7.5 and newer: Turing, Ampere, Ada, Hopper, Blackwell. Some models need more from vLLM itself. DeepSeek Sparse Attention models run only on Hopper and Blackwell.
GPU memory	What the model and its KV cache need without the integration, plus a small reserve. The buffers of the integration are allocated after vLLM sets the size of its KV cache (resources).
ADP card	Local placement: a free PCIe Gen5 x16 slot and the SSDs behind it, see ADP card hardware. Remote placement: only the network card below.
Network (remote card)	An RDMA NIC (RoCE v2 or InfiniBand), 100 Gb/s or faster. Jumbo frames (MTU 9000) on RoCE, an IPoIB address on InfiniBand. See network.

Operating system and software

	Requirement
OS	Linux x86_64
NVIDIA driver	A driver for the CUDA of the engine image. Use R580 or newer for the CUDA 13.0 images (the default vllm/vllm-openai tags). Use R575 or newer for the CUDA 12.9 images (-cu129 tags).
Containers	Docker Engine 25 or newer with Compose v2 and the NVIDIA Container Toolkit (nvidia runtime), or Kubernetes with the NVIDIA device plugin.
Local card	The ADP software stack on this server (ADP card software).
Remote card	nvme-cli. The kernel modules nvme_rdma and rdma_ucm (and ib_ipoib on InfiniBand). The RDMA user-space stack of your NIC. NVMe native multipath off (kernel parameter nvme_core.multipath=N).
Card cluster	Hugepages of 2 MiB for the gateway (3 GiB for each gateway) and the RDMA devices in /dev/infiniband.

The container image has all other parts: CUDA, PyTorch, Python, the engine, the integrations and the SDK. The integrations go into the vLLM image without a change of its packages.

Resources

Resource	What to plan for
/dev/shm	At least the ring of every gateway on the host (shared_memory_size, 6 GiB by default, 10 GiB for 8-GPU servers), plus what the engine uses. The gateway and the engine must share the /dev/shm of the host.
Host RAM	The ring above plus the needs of the engine. The vLLM integrations keep no cache in host memory. The SGLang backend uses the L2 pool of SGLang.
CPU	Some cores for the gateway. Each worker thread (thread_pool_size, 4 by default, 64 on 8-GPU servers) runs one device command at a time.
Disk	About 50 GB for the images, plus the model weights.
GPU memory reserve	The OffloadingSpec backend keeps staging slots in GPU memory (xkv.staging_bytes, 2 GiB by default). NVLink fan-out of the vLLM KV connector takes a staging buffer of 256 MiB and an NCCL communicator on each GPU, outside gpu_memory_utilization. Lower --gpu-memory-utilization by about 0.005 on a 141 GB GPU for the defaults.

Software from Awide Labs

Artifact	What it is
Product image	vLLM with the SDK and the integrations, for vLLM 0.28.0, 0.29.0 and 0.31.0
SGLang image	SGLang 0.5.21 with the SDK and the HiCache backend: awideai-cuda-sglang:v0.5.21-6.5.0
Gateway image	The storage gateway for a remote card, a card in the GPU server and a card cluster (kv-spdk backend).
Client kit	SDK and integration sources with a Dockerfile, to build on your own vLLM image
Cluster service bundle	The cluster service for the storage servers of a card cluster

The images and kits are in the Awide Labs registry nexus.awide.io. We give an account on request. The cluster service bundle is not in the registry: we send it on request. The gateway and the engine image (or kit) must come from the same release, because the gateway refuses an SDK of another release. Installation has the exact names.