Reproducing the benchmarks

AI Key-Value Caching

vLLM with the ADP KV cache connector, compared against the same vLLM without it. The reference setup uses eight H200 GPUs and one remote ADP card.

What you reproduce

Every test runs twice on the same hardware, model and load: once with the ADP KV cache connector and once with plain vLLM. Only the connector and its gateway differ between the two.

Test	What it shows	Published reference
1. Long-context restore	Time to first token on a returning long prompt after a vLLM restart: restored from the card vs recomputed	GLM-5.3, 196,000 tokens: 2.12 s from the card (release 6.1.1) vs 47.7 s of prefill (overview); MiniMax M3: 3.8 s vs 18.3 s (release notes)
2. Multi-turn agent load	Throughput, time to first token and generation rate at 1 to 32 concurrent clients, once conversation histories exceed GPU memory	Pass criteria only: this uses a public synthetic workload rather than our recorded agent traces
3. DRAM tier (optional)	The connector alone vs LMCache with 400 GiB of host RAM in front of the card	Restore: 1.86 s vs 1.68 s from RAM, and 2.26 s vs 2.91 s after a restart (DRAM tier comparison)

Reference setup

Part	Reference setup
GPU server	8 × NVIDIA H200 NVL (141 GB each) in two 4-GPU NVLink domains; x86_64 Linux, NVIDIA driver R580+, Docker with the NVIDIA Container Toolkit
Storage server	A separate x86_64 host with the ADP card and its SSDs, on the same fabric as the GPU server (requirements)
Cache tier	One ADP card in a PCIe Gen5 x16 slot; two 3.84 TB datacenter NVMe SSDs behind it in RAID 0 (~7 TB usable cache)
Fabric	NVMe-oF over RDMA on InfiniBand EDR, 100 Gb/s per port. RoCE v2 at 100 Gb/s is configured the same way.
Software	Connector and gateway release 6.1.1, ADP software stack 2.4.5, vLLM 0.28.0 (GLM-5.3, MiniMax M3) and 0.29.0 (DeepSeek-V4-Flash, Kimi-Linear)
Main model	GLM-5.3 FP8, tensor parallelism 8, multi-token prediction (1 speculative token), 256,000-token context

You can run the same tests on other hardware for a functional check. To match our numbers, keep the GPU model and count, the model and tensor parallelism, the link speed, and the SSD count: long-prompt restores are bounded by whichever is slower, the network link or the SSDs.

Build it

1. Storage server. Install the card, its SSDs, the ADP software stack and the NVMe-oF target, then check that the card is healthy and key eviction is on: ADP card setup.
2. Fabric. Verify the RDMA path between the two servers end to end: Remote card: check the fabric.
3. GPU server. Connect the card’s namespace and note its device path: Remote card: connect the GPU server.
4. Images and models. Pull the release images (Installation) and place the model weights under /models on the GPU server.
5. Compose files. Create two compose files in separate directories, such as ~/bench/adp and ~/bench/gpu-only. Run only one at a time; both listen on port 8000.

Variant with ADP

~/bench/adp/compose.yaml — the deployment from Installation configured for the reference setup (multi-token prediction, NVLink fan-out, full context length, and GLM-5.3 parsers):

services:
  gateway:
    image: nexus.awide.io/awide-docker-hosted/akv-gateway:6.1.1
    ipc: host
    cap_add: [SYS_ADMIN, IPC_LOCK]
    ulimits:
      memlock: -1
    devices:
      - /dev/nvme1n1:/dev/nvme1n1      # the connected ADP namespace (nvme list)
    command:
      - --storage-backend=remote-xdp
      - --storage-devices=/dev/nvme1n1
      - --fragment-size=131072
      - --port=5557
      - --shared-memory-size=10GiB
      - --thread-pool-size=64
      - --num-dbs=128
      - --start-clean=0
    restart: unless-stopped

  vllm:
    image: nexus.awide.io/awide-docker-hosted/awideai-cuda-vllm-openai:v0.28.0-6.1.1
    ipc: host
    cap_add: [IPC_LOCK]
    ulimits:
      memlock: -1
    depends_on: [gateway]
    ports:
      - "8000:8000"
    deploy:
      resources:
        reservations:
          devices:
            - { driver: nvidia, count: all, capabilities: [gpu] }
    environment:
      PYTHONHASHSEED: "0"
      XKV_GATEWAY_HOSTNAME: gateway
      NCCL_LAUNCH_ORDER_IMPLICIT: "1"
    volumes:
      - /models:/models:ro
    command:
      - --model=/models/GLM-5.3-FP8
      - --served-model-name=glm-5.3
      - --tensor-parallel-size=8
      - --max-model-len=256000
      - --gpu-memory-utilization=0.92
      - --max-num-seqs=256
      - --max-num-batched-tokens=8192
      - --kv-cache-dtype=fp8
      - --enable-prefix-caching
      - --enable-prompt-tokens-details
      - '--speculative-config={"method":"mtp","num_speculative_tokens":1}'
      - --tool-call-parser=glm47
      - --enable-auto-tool-choice
      - --reasoning-parser=glm45
      - '--default-chat-template-kwargs={"enable_thinking":false}'   # test 2 counts answer tokens only
      - '--kv-transfer-config={"kv_connector":"FusIOnXConnector","kv_connector_module_path":"xkv_vllm_connector.fusionx_connector","kv_role":"kv_both","kv_connector_extra_config":{"xkv_gateway_port":5557,"chunk_size":128,"slot_alignment_bytes":4096,"nvlink_fanout":true}}'
    restart: unless-stopped

Variant without ADP

~/bench/gpu-only/compose.yaml — the same image and vLLM options, without --kv-transfer-config and without the gateway:

services:
  vllm:
    image: nexus.awide.io/awide-docker-hosted/awideai-cuda-vllm-openai:v0.28.0-6.1.1
    ipc: host
    ports:
      - "8000:8000"
    deploy:
      resources:
        reservations:
          devices:
            - { driver: nvidia, count: all, capabilities: [gpu] }
    environment:
      PYTHONHASHSEED: "0"
    volumes:
      - /models:/models:ro
    command:
      - --model=/models/GLM-5.3-FP8
      - --served-model-name=glm-5.3
      - --tensor-parallel-size=8
      - --max-model-len=256000
      - --gpu-memory-utilization=0.92
      - --max-num-seqs=256
      - --max-num-batched-tokens=8192
      - --kv-cache-dtype=fp8
      - --enable-prefix-caching
      - --enable-prompt-tokens-details
      - '--speculative-config={"method":"mtp","num_speculative_tokens":1}'
      - --tool-call-parser=glm47
      - --enable-auto-tool-choice
      - --reasoning-parser=glm45
      - '--default-chat-template-kwargs={"enable_thinking":false}'
    restart: unless-stopped

Start each variant once and check the GPU KV cache size vLLM logs at startup. It must match in both variants, or the test will measure a difference in GPU memory rather than the card. On the reference setup it is about 545,000 tokens:

GPU KV cache size: 545,536 tokens, Maximum concurrency for 256,000 tokens per request: 2.13x

The ADP variant also logs that the connector registered (XKV: <N> cache tensors registered for exchange), and the gateway logs that the card is open; see Installation.

Test 1: long-context restore

Send a long prompt once, restart vLLM to clear GPU memory, and send the same prompt again. Without the connector, the second request recomputes the entire prompt; with the connector, it restores the KV cache from the card. Restarting only vLLM leaves the gateway and card running, ensuring any hit comes from the card.

Define these shell helpers on the GPU server (requires curl, jq 1.6+, and Python 3):

URL=http://127.0.0.1:8000
MODEL=glm-5.3            # the --served-model-name
LINES=14580              # about 196,500 tokens on GLM-5.3; adjust for another tokenizer

new_prompt() {           # generates a fresh prompt with a unique run tag
  python3 - "$LINES" > prompt.txt <<'EOF'
import sys, time
n = int(sys.argv[1])
print(f"Run {time.time_ns()}.", " ".join(f"Line {i}: the ADP card keeps this prompt." for i in range(n)))
EOF
  jq -n --arg m "$MODEL" --rawfile p prompt.txt '{model: $m, prompt: $p, max_tokens: 1, temperature: 0}' > request.json
}
ask() {                  # prints response time, prompt length, and cached tokens
  curl -s -o response.json -w 'time %{time_total} s  ' "$URL/v1/completions" \
    -H 'Content-Type: application/json' --data-binary @request.json
  jq -c '.usage | {prompt_tokens, cached: .prompt_tokens_details.cached_tokens}' response.json
}
ready() {                # waits for health check, then sends one short warmup request
  until curl -sf "$URL/health" >/dev/null; do sleep 10; done
  curl -s "$URL/v1/completions" -H 'Content-Type: application/json' \
    -d "{\"model\":\"$MODEL\",\"prompt\":\"Hello\",\"max_tokens\":1}" >/dev/null
}

With max_tokens: 1, total response time equals the time to first token. Check prompt_tokens on the first run and adjust LINES if needed so it stays below --max-model-len.

Without ADP — three recomputes on an empty GPU cache:

cd ~/bench/gpu-only && docker compose up -d && ready
new_prompt
for i in 1 2 3; do
  ask                                     # recomputed: cached is 0
  docker compose restart vllm && ready    # clears GPU memory
done
docker compose down

With ADP — one cold request that writes the prompt to the card, followed by three restores:

cd ~/bench/adp && docker compose up -d && ready
new_prompt
ask                                       # cold: computed and written to the card
for i in 1 2 3; do
  docker compose restart vllm && ready    # restarts vLLM only; gateway and card keep the cache
  ask                                     # restored from the card
done
docker compose down                       # cache remains on the card

Pass criteria.

• After every restart, cached is at least 99% of prompt_tokens. Because hits are rounded down to whole chunks (chunk_size: 128), a 196,531-token prompt hits at most 196,480 tokens.
• Restore time is at most one-fifth of recompute time (or 5 s, whichever is larger).
• On the reference setup, GLM-5.3 restores a 196,000-token prompt in about 2.1 s, compared to about 47 s to recompute.

If the first run after a fresh start is slower while kernels compile and tune, repeat it once and use the second measurement.

Other models

To run the same test on the other three published models, update both compose files:

Model	vLLM image	--max-model-len	Prompt	Additional changes	Published reference
GLM-5.3 FP8	v0.28.0-6.1.1	256000	~196,000 tokens	— (as above)	2.12 s (release 6.1.1)
DeepSeek-V4-Flash	v0.29.0-6.1.1	262144	~194,000 tokens	chunk_size: 512; VLLM_USE_V2_MODEL_RUNNER=0; warm up before measuring (DeepGEMM compiles kernels on first use)	0.97 s vs 15.61 s (presentation)
MiniMax M3 MXFP8	v0.28.0-6.1.1	262144	~142,700 tokens	--kv-cache-dtype=fp8; default chunk_size	3.8 s vs 18.3 s (release notes)
Kimi-Linear-48B-A3B	v0.29.0-6.1.1	262144	~147,000 tokens	VLLM_USE_V2_MODEL_RUNNER=0, --watermark=0.02, chunk_size a multiple of 256	1.46 s vs 3.28 s (presentation)

For models other than GLM-5.x, remove slot_alignment_bytes, nvlink_fanout, --speculative-config, the GLM parsers, and --max-num-batched-tokens; see Supported models for full per-model flags. The gateway configuration does not change. You do not need to wipe the card when switching models: keys do not collide across models, though they share the card’s capacity.

Test 2: multi-turn agent load

Coding agents send long conversations that grow with each turn: a system prompt, tool definitions, repository context, and a sequence of tool results and replies. While all active histories fit in GPU memory, vLLM serves them directly and the connector adds nothing. Once they exceed GPU memory, plain vLLM evicts histories between turns and must recompute them on the next turn. This test steps concurrency up into that regime.

Workload

The test uses vLLM’s built-in multi-turn benchmark (/vllm-workspace/benchmarks/multi_turn in the vLLM image). It generates synthetic conversations and replays them through the OpenAI-compatible API. Each client keeps two conversations active at a time, sending the full conversation history on every turn.

The configuration below matches the shape of our recorded agent traces: a 12,000-token prefix shared across all conversations, 12,000–20,000 tokens of unique context per conversation, 6–8 user turns of 1,000–2,000 new tokens each, and 400-token replies. Requests carry 30,000 to 45,000 tokens of history (about 33,000 on average).

Prepare the workload directory on the GPU server:

mkdir -p ~/bench/load && cd ~/bench/load
for book in 1184 2600 135; do       # public-domain text used to build conversations
  curl -fLo pg$book.txt https://www.gutenberg.org/ebooks/$book.txt.utf-8
done
cat > agents.json <<'EOF'
{
  "filetype": "generate_conversations",
  "num_conversations": 96,
  "text_files": ["run.txt", "pg1184.txt", "pg2600.txt", "pg135.txt"],
  "print_stats": true,
  "prompt_input": {
    "num_turns": {"distribution": "uniform", "min": 12, "max": 16},
    "common_prefix_num_tokens": {"distribution": "constant", "value": 12000},
    "prefix_num_tokens": {"distribution": "uniform", "min": 12000, "max": 20000},
    "num_tokens": {"distribution": "uniform", "min": 1000, "max": 2000}
  },
  "prompt_output": {
    "num_tokens": {"distribution": "constant", "value": 400}
  }
}
EOF
docker build -t adp-bench-client - <<'EOF'
FROM vllm/vllm-openai:v0.28.0
RUN pip install --no-cache-dir -r /vllm-workspace/benchmarks/multi_turn/requirements.txt
ENTRYPOINT ["python3", "/vllm-workspace/benchmarks/multi_turn/benchmark_serving_multi_turn.py"]
EOF

num_turns counts both user and assistant messages, so 12–16 messages equals 6–8 user turns. Before each concurrency level, run.txt writes a unique tag at the start of the shared prefix so nothing carries over in cache from earlier levels or runs. If the generator stops with Not enough input text, add another book to text_files.

Run the concurrency ladder

A ladder runs six concurrency levels on one running server: 1, 2, 4, 8, 16, and 32 clients. Each level generates three conversations per client while keeping two active at a time (~17 requests per client). On the reference setup, active histories fill the GPU KV cache at 8 clients, and reach 2× and 4× the GPU cache size at 16 and 32 clients.

URL=http://127.0.0.1:8000
ladder() {               # usage: ladder <variant> <run>, e.g. ladder adp 1
  mkdir -p results
  for p in ${LEVELS:-1 2 4 8 16 32}; do
    echo "Benchmark $1, run $2, $p clients." > run.txt
    jq ".num_conversations = $((3 * p))" agents.json > level.json
    curl -s "$URL/metrics" > "results/$1-$2-L$p.pre.prom"
    docker run --rm --network host -v "$PWD":/work -w /work -v /models:/models:ro adp-bench-client \
      --model /models/GLM-5.3-FP8 --served-model-name glm-5.3 --url "$URL" \
      --input-file level.json --seed 520052 \
      --num-clients $p --max-active-conversations $((2 * p)) \
      --limit-min-tokens 400 --limit-max-tokens 400 \
      --request-timeout-sec 3600 --no-early-stop \
      --stats-json-output "results/$1-$2-L$p.json" | tee "results/$1-$2-L$p.log"
    curl -s "$URL/metrics" > "results/$1-$2-L$p.post.prom"
    sleep 15
  done
}

Alternate between the two variants on a freshly started server, running each at least twice:

cd ~/bench/adp      && docker compose up -d && ready && cd ~/bench/load && ladder adp 1
cd ~/bench/adp      && docker compose down
cd ~/bench/gpu-only && docker compose up -d && ready && cd ~/bench/load && ladder gpu-only 1
cd ~/bench/gpu-only && docker compose down
# repeat for run 2

At 16 and 32 clients, the run without the connector takes several times longer because it must recompute evicted histories.

Read the results

This script prints total output throughput, median time to first token (TTFT), median generation rate per request, and prefix-cache hit rates (GPU and card) for each run and level:

cd ~/bench/load
python3 - results/*-L*.json <<'EOF'
import json, re, sys, statistics as st

def counters(path):
    s = {}
    for line in open(path):
        m = re.match(r'(vllm:(?:external_)?prefix_cache_(?:queries|hits)_total)\{[^}]*\} (\S+)', line)
        if m:
            s[m[1]] = s.get(m[1], 0) + float(m[2])
    return s

def hit_rate(pre, post, tier):
    q = post.get(f'vllm:{tier}prefix_cache_queries_total', 0) - pre.get(f'vllm:{tier}prefix_cache_queries_total', 0)
    h = post.get(f'vllm:{tier}prefix_cache_hits_total', 0) - pre.get(f'vllm:{tier}prefix_cache_hits_total', 0)
    return f'{100 * h / q:.0f} %' if q else '-'

def level(f):
    name, lvl = f.rsplit('-L', 1)
    return name, int(lvl[:-5])

print(f"{'variant-run':16} {'clients':>7} {'requests':>8} {'output tok/s':>12} {'TTFT p50 s':>10} "
      f"{'gen tok/s':>9} {'GPU hit':>8} {'card hit':>8}")
for f in sorted(sys.argv[1:], key=level):
    r = json.load(open(f))
    t0 = min(x['start_time_ms'] for x in r)
    t1 = max(x['start_time_ms'] + x['latency_ms'] for x in r)
    out = sum(x['output_num_tokens'] for x in r) / ((t1 - t0) / 1000)
    pre, post = counters(f[:-5] + '.pre.prom'), counters(f[:-5] + '.post.prom')
    name, lvl = level(f)
    print(f"{name.split('/')[-1]:16} {lvl:7} {len(r):8} {out:12.1f} "
          f"{st.median(x['ttft_ms'] for x in r) / 1000:10.2f} {1000 / st.median(x['tpot_ms'] for x in r):9.1f} "
          f"{hit_rate(pre, post, ''):>8} {hit_rate(pre, post, 'external_'):>8}")
EOF

Pass criteria.

• 1–4 clients. All histories fit in GPU memory. Both variants perform similarly, with the ADP variant within 10% of plain vLLM in throughput and generation rate.
• 8 clients. The GPU cache fills up and the GPU hit rate on plain vLLM begins to drop.
• 16 and 32 clients. Without ADP, the GPU hit rate drops and evicted histories are recomputed, increasing TTFT and slowing generation. With ADP, card hits replace the lost GPU hits, keeping throughput and generation rate higher and TTFT lower.
• Both variants. Every level finishes with num_failures=0, and the ADP vLLM log never prints External KV cache turned off on this rank.

Compare generation rate against your own SLA (our published comparisons use 20 tok/s per request, or 50 ms/token). Because this workload uses synthetic text rather than recorded agent traces, compare the ratio between the two variants at each concurrency level rather than absolute numbers.

Test 3: against a DRAM tier (optional)

A common alternative to an NVMe cache tier is host RAM. This test compares the ADP connector alone against LMCache configured with 400 GiB of pinned host RAM (L1) in front of the same ADP card (L2, via the LMCache plugin).

	Connector alone	LMCache with a DRAM tier
vLLM image	Product image (as in the ADP variant)	vLLM 0.28.0 with LMCache 0.5.5 (multiprocess mode) and the ADP L2 plugin, provided by Awide Labs on request
KV transfer	FusIOnXConnector, as above	LMCacheMPConnector with the LMCache server inside the vLLM container (setup)
Host RAM for KV cache	10 GiB shared-memory ring	400 GiB L1 (LMC_L1_GB=400) plus the 10 GiB ring
Gateway and card	Unchanged	Unchanged, with LMC_XKV_HOST=gateway and LMC_XKV_FRAG=131072

Run three checks:

1. Restore on a running engine. Send the test 1 prompt once, reset the GPU prefix cache, and send the prompt again (repeat three times). Resetting the cache requires VLLM_SERVER_DEV_MODE=1 in the vLLM environment (development only; do not enable in production):

   until [ "$(curl -s -X POST "$URL/reset_prefix_cache" | jq .success)" = true ]; do sleep 1; done
   ask

   This clears GPU memory while leaving the prompt in LMCache’s L1 and on the card.

2. Restore after a vLLM restart. Run test 1. Because LMCache’s L1 runs inside the vLLM container, it clears on restart and LMCache restores the prompt from the card.

3. Under load. Run the test 2 ladder at 8, 16, and 32 clients: LEVELS="8 16 32" ladder lmcache 1.

In our GLM-5.3 comparison, a warm restore took 1.68 s from RAM vs 1.86 s from the card; a restore after a vLLM restart took 2.26 s with the connector alone vs 2.91 s through LMCache, with matching generation rates under load (DRAM tier comparison).

Report the results

To compare numbers or ask about a run, send us:

• the results directory from test 2 and the output of test 1;
• both compose files;
• the GPU KV cache size startup line from both variants, plus the connector startup lines and gateway log from the ADP variant;
• from the storage server: pliocli system get_status and pliocli system get_disk_usage -x 0; from the GPU server: nvme list-subsys and ib_read_bw output (check the fabric).

If a restore returns zero cached tokens after a restart, see Operations & troubleshooting and Measuring the effect.

Storage acceleration

Raw block-device benchmarks with fio, comparing the ADP card in block mode (the FSA bundle) against Linux software RAID10 on the same four NVMe drives, with no filesystem. The reference numbers below come from the storage presentation.

The ADP software comes in two separate bundles:

• KV bundle (ADP card setup): places the SSDs in a RAID 0 array behind the card and exposes a key-value store to the gateway (used in AI Key-Value Caching) or to KV engines such as RocksDB (XDP-Rocks) and MongoDB (XDP-WiredTiger).
• Block (FSA) bundle: exposes a single block device while the card handles inline compression, write aggregation, and RAID parity in hardware.

This section uses only the block bundle.

What this comparison shows

Every workload runs twice on the same server, the same four drives, and the same fio job: once on Linux mdraid RAID10 and once on the ADP card in RAID5 mode. Test buffers are set to 67% compressibility to match typical database pages.

Workload	Job	mdraid RAID10	ADP RAID5	Gain
Random read, 8 KB	32 jobs × depth 32 (QD 1024)	16,851 MB/s	19,589 MB/s	+16%
Mixed random (70% read / 30% write), 8 KB	128 jobs × depth 8 (QD 1024)	3,162 MB/s	13,224 MB/s	4.2×
Random write, 8 KB	8 jobs × depth 8 (QD 64)	1,082 MB/s	6,934 MB/s	6.4×
Sequential read, 1 MB	8 jobs × depth 4 (QD 32)	26,192 MB/s	29,510 MB/s	+13%
Sequential write, 1 MB	8 jobs × depth 4 (QD 32)	5,066 MB/s	7,400 MB/s	+46%

(Note: Slide 8 of the presentation rounds the 5,066 MB/s sequential write baseline to 5.1 GB/s; another run at 4 jobs × depth 4 reached 5,074 MB/s. In both cases ADP is +46% faster.)

Read throughput is similar on both arrays, while ADP uses about half the host system CPU at the same IOPS. The large gains are on writes and mixed workloads under concurrency: at queue depth 1024, the mixed 8 KB workload reaches 1,653,000 IOPS on ADP vs 395,000 IOPS on RAID10, and at queue depth 256 its p99 write latency is 122 µs vs 3,522 µs (29× lower). At queue depth 1, a single unqueued read is faster on RAID10 (70 µs vs 111 µs).

Usable capacity. Four 3.84 TB drives provide 15.4 TB of raw flash:

• mdraid RAID10 mirrors every block, leaving 7.7 TB usable (50% of raw).
• ADP RAID5 uses parity instead of mirroring (11.5 TB before compression, 75% of raw) and compresses data inline, exposing 15.4 TB to the host — 2× the usable capacity of RAID10 with parity protection. Actual effective capacity depends on how well your data compresses.

Reference setup

Part	Reference setup
Server	Dell PowerEdge R750, Ubuntu 24.04
CPU	2 × Intel Xeon Gold 6346 at 3.10 GHz
Memory	512 GB
Drives	4 × Samsung PM9A3 (3.84 TB, PCIe Gen4), used by one array at a time
Baseline	Linux mdraid RAID10 (mdadm)
ADP	Block (FSA) bundle, RAID5 (--level=5), card index 1 on this host
Tool	fio, direct I/O, raw block device (no filesystem), 67% compressible buffers
Suite	63 matched configurations per array (126 runs), selected from 800+ total runs

You can run these jobs on other drives to verify the behavior. To match the bandwidth numbers in the table above, use four PCIe Gen4 drives of the same class: with four drives, the ceiling is the drives rather than the card (which supports up to 48 drives and 56 GB/s).

1. Prepare the drives

List the NVMe drives and discard all blocks on the four test drives so both arrays start from the same clean state. Stop any existing array using these drives first, or blkdiscard will fail.

nvme list
sudo blkdiscard /dev/nvme0n1
sudo blkdiscard /dev/nvme1n1
sudo blkdiscard /dev/nvme2n1
sudo blkdiscard /dev/nvme3n1

Replace /dev/nvme0n1 … /dev/nvme3n1 with the device names from nvme list, and use the same four drives for both tests.

2. Test Linux software RAID10

Create the RAID10 array:

sudo mdadm --create --verbose /dev/md126 --level=10 --raid-devices=4 \
  /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --force --assume-clean

--assume-clean skips the initial mirror resync. This is safe here only because the drives were just discarded and fio writes across the span.

Verify that the array is active ([UUUU]) before running fio with DEV=/dev/md126:

cat /proc/mdstat
lsblk /dev/md126

3. Switch to the ADP block bundle

When the RAID10 runs finish, stop and remove the mdraid array, then discard the four drives again before installing the ADP block bundle:

sudo mdadm --stop /dev/md126
sudo mdadm --zero-superblock /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1
sudo blkdiscard /dev/nvme0n1
sudo blkdiscard /dev/nvme1n1
sudo blkdiscard /dev/nvme2n1
sudo blkdiscard /dev/nvme3n1

Install the block (FSA) bundle (provided by Awide Labs for your host kernel). Pass the card index from pliocli system get_xdp_list (--xdp=1 on the reference host, or 0 on a single-card host) and --level=5 for RAID5:

sudo ./xdp-installer install --xdp=1 --level=5 \
  --resources="/dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1" \
  --assume-yes --enable-set-hugepages

Verify that the service and RAID5 array are healthy, and find the new ADP block device in lsblk:

systemctl is-active pliostore@1
pliocli system get_status            # Aggregated status is OK; RAID reports RAID5
pliocli raid get_raid_info -x 1
lsblk -d -o NAME,SIZE,MODEL

In block mode, /etc/pliops/1/pliostore.ini has product_type = 1 (whereas the KV bundle sets product_type = 0). Do not mix both modes on the same card.

When you finish testing and want to release the drives, uninstall the block bundle before running blkdiscard on the drives again.

4. Run fio

Use the helper below for both arrays. Set DEV to /dev/md126 (for RAID10) or to the ADP block device, and set OUT to results/raid10 or results/adp.

--size=3800000000000 (~3.5 TiB) sets the working span; --runtime=60 stops each run at 60 seconds.

DEV=/dev/md126
OUT=results/raid10
mkdir -p "$OUT/iostat"

run() {                              # usage: run <name> <fio options...>
  local name=$1
  shift
  iostat -xm 1 > "$OUT/iostat/$name.iostat" &
  local mon=$!
  sudo -E fio \
    --random_generator=tausworthe64 --per_job_logs=0 \
    --buffer_compress_chunk=4096 --buffer_compress_percentage=67 --refill_buffers=1 \
    --direct=1 --group_reporting=1 --randrepeat=0 --norandommap=1 \
    --ioengine=libaio \
    "$@" \
    --filename="$DEV" --name="$name" \
    --output-format=json --output="$OUT/$name.json"
  kill "$mon"
}

# Precondition once per array before the measured runs
run precondition --readwrite=write --bs=128k --numjobs=8 --iodepth=8 --size=1g --runtime=60

# The five headline workloads from the table above
run randread-8k-j32-d32  --readwrite=randread  --bs=8k  --numjobs=32  --iodepth=32 --size=3800000000000 --runtime=60
run randrw-8k-j128-d8    --readwrite=randrw    --bs=8k  --numjobs=128 --iodepth=8  --rwmixread=70 --size=3800000000000 --runtime=60
run randwrite-8k-j8-d8   --readwrite=randwrite --bs=8k  --numjobs=8   --iodepth=8  --size=3800000000000 --runtime=60
run read-1m-j8-d4        --readwrite=read      --bs=1m  --numjobs=8   --iodepth=4  --size=3800000000000 --runtime=60
run write-1m-j8-d4       --readwrite=write     --bs=1m  --numjobs=8   --iodepth=4  --size=3800000000000 --runtime=60

The rest of the 63-run suite sweeps block sizes (8k, 16k, 1m) and queue depths (numjobs × iodepth from 1 to 1024; single-job QD 1 runs use --ioengine=sync).

While each job runs, iostat logs per-drive activity once per second. All four drives should show balanced load. On random writes, mdraid RAID10 shows higher write wait times (w_await), whereas ADP aggregates small random writes into full sequential stripes.

5. Read the storage results

The script below parses the fio JSON files and prints total bandwidth (decimal MB/s, read + write), total IOPS (kIOPS), and p99 completion latency (µs):

python3 - results/*/*.json <<'EOF'
import json, sys
print(f"{'array/job':36} {'MB/s':>8} {'kIOPS':>8} {'read p99 us':>12} {'write p99 us':>12}")
for f in sorted(sys.argv[1:]):
    j = json.load(open(f))["jobs"][0]
    arr = f.split("/")[-2]
    def mb(s):
        return s.get("bw_bytes", 0) / 1e6
    def p99(s):
        p = (s.get("clat_ns") or s.get("lat_ns") or {}).get("percentile") or {}
        v = p.get("99.000000")
        return f"{v / 1000:.0f}" if v else "-"
    r, w = j["read"], j["write"]
    label = f"{arr}/{j['jobname']}"
    print(f"{label:36} {mb(r) + mb(w):8.0f} {(r['iops'] + w['iops']) / 1000:8.1f} "
          f"{p99(r):>12} {p99(w):>12}")
EOF

Pass criteria (on four reference-class Gen4 NVMe drives):

• Random write, 8 KB (8 × 8, QD 64): ADP bandwidth is at least 4× RAID10 (published: 6.4×, 6,934 vs 1,082 MB/s; mean latency 71 µs vs 458 µs).
• Mixed 70/30, 8 KB (128 × 8, QD 1024): ADP bandwidth is at least 3× RAID10 (published: 4.2×, 13,224 vs 3,162 MB/s).
• Random read, 8 KB (32 × 32, QD 1024) and sequential read, 1 MB (8 × 4, QD 32): ADP is within 10% of RAID10 or faster (published: +16% and +13%), with roughly half the system CPU on random reads at matching IOPS.
• Sequential write, 1 MB (8 × 4, QD 32): ADP is ahead of RAID10 (published: +46%, 7,400 vs 5,066 MB/s).
• Queue depth 1: RAID10 can be faster on a single unqueued read (70 µs vs 111 µs); this is expected.
• Both arrays: Every fio job exits with status 0.

Report the storage results

To compare with the presentation or ask about a run, send us:

• both results directories (results/raid10 and results/adp, including JSON and iostat files);
• the run commands and DEV paths used;
• nvme list, cat /proc/mdstat (for RAID10), and pliocli system get_status plus pliocli raid get_raid_info -x 1 (for ADP).