Supported models

Verified models

The models below are supported. The columns show the vLLM version for each integration.

Model	Attention architecture	OffloadingSpec backend	vLLM KV connector	MTP
GLM-5.3 FP8	Sparse MLA (DSA)	0.28.0, 0.31.0	0.28.0	yes
MiniMax M3 MXFP8	Sparse attention with indexer, MoE	0.31.0	0.28.0	—
DeepSeek-V4.1-Flash	Compressed sparse MLA + sliding window	0.31.0	—	—
DeepSeek-V4-Flash	Compressed sparse MLA + sliding window	0.31.0	0.29.0	—
Kimi K3	Kimi Delta Attention + MLA	—	0.29.0	—
Kimi-Linear-48B-A3B	Kimi Delta Attention + MLA	—	0.29.0	—
GLM-5.2 FP8	Sparse MLA (DSA)	—	0.26	yes
GLM-4.7-Flash	MLA, AWQ	—	0.26	—
DeepSeek-V2-Lite	MLA	—	0.26	—
Gemma 4 12B	Full + sliding-window attention	—	0.26	yes
Qwen 3.8 27B	Linear + full attention	—	0.26	yes
Qwen 3.6 35B-A3B	Gated DeltaNet, MoE	—	0.26	yes
Qwen 3.6 27B	Mamba hybrid, three cache groups	—	0.26	—
Nemotron 3 Nano 30B-A3B	Mamba-2 + sparse attention	—	0.26	—

A dash means that we did not verify the model with that integration on hardware. It does not mean that the model does not work. The OffloadingSpec backend leaves the layout of the model to vLLM, so a model of vLLM’s offloading framework usually works without change. DeepSeek-V4.1-Flash runs only with the OffloadingSpec backend.

Talk to us about a model that is not on the list.

Engine versions

Engine	OffloadingSpec backend	vLLM KV connector	Notes
vLLM 0.26	—	yes	The connector only. On request: no product image or kit is published for it.
vLLM 0.28.0	yes	yes	Recommended for GLM-5.x.
vLLM 0.29.0	yes	yes	The connector needs VLLM_USE_V2_MODEL_RUNNER=0.
vLLM 0.31.0	yes	—	Required for DeepSeek-V4.1-Flash. Not for GLM-5.x (see below).
SGLang 0.5.21	—	—	The SGLang HiCache backend. Verified with Qwen 3.8 and GLM-5.3 FP8.

What each architecture needs

Architecture	Examples	What the deployment needs
Full attention, MLA	DeepSeek-V2-Lite, GLM-4.7-Flash	chunk_size (connector) or block_size (OffloadingSpec) a multiple of the block size of vLLM.
Full + sliding-window attention	Gemma 4	Connector: the window at least twice chunk_size and a multiple of it. With deferred write (the default), --no-async-scheduling and pipeline parallelism 1, or enable_deferred_write: false. The connector refuses to start otherwise and tells which.
Mamba, Mamba-2, Gated DeltaNet, Kimi Delta Attention hybrids	Qwen 3.6, Qwen 3.8, Nemotron 3 Nano, Kimi-Linear, Kimi K3	--enable-prefix-caching, --mamba-cache-mode align (vLLM 0.29 and newer set it with prefix caching), --watermark=0.02 only on an image without the Mamba align patch, not on the product image (why), pipeline parallelism 1. OffloadingSpec: no block_size, set "blocks_per_chunk": 1.
DeepSeek Sparse Attention	GLM-5.2, GLM-5.3	--kv-cache-dtype fp8. A GPU that the sparse MLA kernels of vLLM support (Hopper or Blackwell). Connector on vLLM 0.29: FULL_DECODE_ONLY CUDA graphs.
Compressed sparse MLA	DeepSeek-V4-Flash, DeepSeek-V4.1-Flash	OffloadingSpec: "blocks_per_chunk": 1, because the KV cache groups have different block sizes. Connector: deferred write on (the default).
Sparse attention with indexer	MiniMax M3	--kv-cache-dtype fp8, for one page size in each group.

Multi-token prediction (MTP) works with every architecture that has an MTP head.

Settings for each model

OffloadingSpec backend

The xkv.* keys and block_size or blocks_per_chunk go in kv_connector_extra_config (OffloadingSpec backend). The vLLM options go on the vLLM command line, next to the options of the model itself (tool-call and reasoning parsers, context length).

Model	Backend	vLLM
GLM-5.3 FP8	"block_size": 128, "xkv.shard_reads": "auto"	vLLM 0.28.0, --kv-cache-dtype fp8, --max-num-batched-tokens=8192. MTP: '{"method":"mtp","num_speculative_tokens":1}'.
MiniMax M3 MXFP8	defaults	vLLM 0.31.0, --kv-cache-dtype fp8
DeepSeek-V4.1-Flash	"blocks_per_chunk": 1	vLLM 0.31.0. The first start compiles kernels for a long time: keep VLLM_CACHE_ROOT on a persistent volume and give the startup probe enough time.
DeepSeek-V4-Flash	"blocks_per_chunk": 1	vLLM 0.31.0
Hybrid models	"blocks_per_chunk": 1	--mamba-cache-mode align

vLLM KV connector

The connector keys go in kv_connector_extra_config (vLLM KV connector).

Model	Connector	vLLM
DeepSeek-V4-Flash	chunk_size: 512	vLLM 0.29.0, VLLM_USE_V2_MODEL_RUNNER=0. DeepGEMM compiles kernels at the first requests, which can pause them for 15–23 s. Warm up with typical requests and keep VLLM_CACHE_ROOT on a persistent volume.
Kimi K3	chunk_size a multiple of the attention block of vLLM (768 tokens at TP=8, 1536 with an fp8 KV cache). The gateway needs fragment_size, because the page is 864 KiB at TP=8.	vLLM 0.29.0, VLLM_USE_V2_MODEL_RUNNER=0, --additional-config '{"kda_prefill_backend": "triton"}', no --speculative-config
Kimi-Linear-48B-A3B	chunk_size a multiple of 256 at TP=8. The gateway needs fragment_size, because the page is 288 KiB at TP=8.	vLLM 0.29.0, VLLM_USE_V2_MODEL_RUNNER=0
GLM-5.3 FP8	chunk_size: 128. Remote card: slot_alignment_bytes: 4096.	vLLM 0.28.0, --kv-cache-dtype fp8, --max-num-batched-tokens=8192. MTP: '{"method":"mtp","num_speculative_tokens":1}'.
MiniMax M3 MXFP8	—	vLLM 0.28.0, --kv-cache-dtype fp8
GLM-5.2 FP8	chunk_size: 128. Remote card: slot_alignment_bytes: 4096.	vLLM 0.26 or 0.28.0, --kv-cache-dtype fp8, --max-num-batched-tokens=8192. MTP: '{"method":"mtp","num_speculative_tokens":1}'.
GLM-4.7-Flash	chunk_size: 256	—
DeepSeek-V2-Lite	—	—
Gemma 4 12B	chunk_size so that the sliding window is at least twice it and a multiple of it	--no-async-scheduling. MTP: '{"method":"gemma4_mtp","model":"google/gemma-4-12B-it-assistant","num_speculative_tokens":1}'.
Qwen 3.8 27B	— (vLLM selects the block)	--mamba-cache-mode align. MTP: '{"method":"qwen3_5_mtp","num_speculative_tokens":1}'.
Qwen 3.6 35B-A3B	—	--mamba-cache-mode align. MTP: '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'.
Qwen 3.6 27B	—	--mamba-cache-mode align
Nemotron 3 Nano 30B-A3B	—	--mamba-cache-mode align (required, because this model selects another mode by itself)

Hybrid models need --watermark=0.02 only on an image without the Mamba align patch (why); the product image carries the patch. MTP values go to --speculative-config, in single quotes on a container command line. A dash in the connector column means that the defaults are correct. Then chunk_size is the block size of vLLM. vLLM makes this block larger for hybrid models, so that a page holds one recurrent state.

Choosing the chunk size

chunk_size (connector) and block_size (OffloadingSpec) are the number of tokens in one stored object. Larger chunks give fewer and larger I/Os for each restore. Smaller chunks give a finer hit. A prompt of N tokens can hit at most floor(N / chunk_size) × chunk_size tokens. A prompt shorter than one chunk is never stored. The value must be a multiple of the block size of every KV cache group, and the integration checks this at start. Keep the value from the tables unless you measured another one. A change of the value starts a new cache.