Tenant isolation

Why it matters

Prefix caching shares computed KV blocks between all requests that start the same way, whoever sent them. A hit skips the prefill, and the prefill is most of the time to first token of a long prompt. On GLM-5.3, a prompt of about 3,000 tokens answers in 672 ms from the start and in 92 ms from the cache. So any client can find out if someone already sent a given text: guess, measure the time, repeat. Block by block, a client can then reconstruct the prompt of another tenant. In vLLM this is CVE-2025-46570. An external KV cache makes the window larger, because the cache now stays after GPU memory and restarts.

For a single team or application there is nobody to hide prompts from and nothing to do. For a service shared by several customers, or by users with private data, isolate the cache as described here.

How isolation reaches the card

vLLM accepts a cache_salt field in the request body (chat completions, completions, the Responses API, pooling and the Anthropic-compatible /v1/messages). vLLM mixes the salt into the hash of the first block of the prompt. Every later block hashes its parent, so the salt changes every block hash of the request. Requests with different salts never share a block. Requests with the same salt share as before. SGLang does the same with its cache_salt and extra_key fields.

The ADP integrations do not hash tokens themselves. They make every key on the card from the salted block hashes of the engine. This is true in every cache group, also for Mamba state, MTP layers and the sparse-attention indexer. So the scope of a tenant holds on the card as in GPU memory, with no setting of the integration. The gateway, a shared cache and a card cluster do not see or change the salt. They only route the key.

Isolation is only as good as the component that sets the salt. vLLM does not authenticate anyone, accepts whatever salt the client sends and treats an empty salt as none. That component is the API front end in front of vLLM.

1. Choose the scope

A salt splits the cache. Two scopes can have a prefix in common, for example a long shared system prompt or the same document. The engine then computes and stores it once for each scope, not once in total. Inside a scope nothing changes. Put the boundary where trust ends:

Deployment	Salt	Why
One team or one application, one trust domain	none	There is nobody to hide prompts from. A salt only costs hits.
Shared public system prompt, few-shot examples, a documentation assistant	none or per tenant	The shared prefix is the point of the cache, and there is nothing secret in it.
An API or SaaS serving several customers from one deployment	per tenant, at least	Customers must not learn the prompts of other customers. Colleagues still share.
Users of one organization with private data: HR, medical, legal, personal mail	per user	The boundary runs between people, not between companies.
One-off prompts with secrets, such as credentials pasted into a chat	per session or request	No reuse at all — use it narrowly, or keep such requests out of the external cache with xkv_store: false.

A client may narrow its scope, never widen it.

2. Set the salt in the front end

Six rules for the API front end:

1. Set the salt on every request, from the authenticated identity. Remove each cache_salt that the client sent. Also remove the fields that go to the engine or its integrations without a filter. These are kv_transfer_params, vllm_xargs and the multimodal uuid values of the client, which go into the same hash.
2. Make the salt unguessable. Derive it with HMAC-SHA256 from a secret of at least 32 bytes that only the front end holds. A salt equal to the tenant name or account id can be guessed, and a guessed salt reopens the channel.
3. Match the scope to the trust boundary (above).
4. Expose only the endpoints that carry the salt. vLLM must not be reachable around the front end, and endpoints without a salt field stay closed.
5. Close the internal surfaces — next step.
6. Keep the salt stable. Derived from the secret and the identity, the same tenant gets the same salt after a restart, and the ADP card keeps serving that scope. Rotating the secret empties the cache for everyone: the right tool after a leak, the wrong one on a schedule.

A front end in Python with FastAPI, for chat and completion requests, streaming included. The same few lines go into any proxy you already run.

import base64, hashlib, hmac, os

import httpx
from fastapi import FastAPI, HTTPException, Request
from fastapi.responses import JSONResponse, StreamingResponse
from starlette.background import BackgroundTask

SECRET = bytes.fromhex(os.environ["CACHE_SALT_SECRET"])   # 32+ random bytes, only the front end has them
DEPLOYMENT = "prod"                                        # part of the salt: two deployments never share
STRIP = ("cache_salt", "kv_transfer_params", "vllm_xargs") # never taken from the client

app = FastAPI()
vllm = httpx.AsyncClient(base_url="http://vllm:8000", timeout=None)


def cache_salt(scope: str, scope_id: str) -> str:
    """The same scope always gets the same salt; nobody without SECRET can compute it."""
    message = f"v1|{DEPLOYMENT}|{scope}|{scope_id}".encode()
    digest = hmac.new(SECRET, message, hashlib.sha256).digest()
    return base64.urlsafe_b64encode(digest).rstrip(b"=").decode()


def authenticate(request: Request) -> str:
    """Your existing authentication: API key -> tenant id."""
    tenant = lookup_tenant(request.headers.get("authorization", ""))
    if tenant is None:
        raise HTTPException(status_code=401)
    return tenant


def salted(body: dict, tenant: str) -> dict:
    for field in STRIP:
        body.pop(field, None)
    for message in body.get("messages", []):               # client-supplied multimodal ids feed the hash too
        if isinstance(message.get("content"), list):
            for part in message["content"]:
                if isinstance(part, dict):
                    part.pop("uuid", None)
    body["cache_salt"] = cache_salt("tenant", tenant)      # or ("user", user_id) for a per-user scope
    return body


async def forward(request: Request, path: str):
    body = salted(await request.json(), authenticate(request))
    if body.get("stream"):
        upstream = await vllm.send(vllm.build_request("POST", path, json=body), stream=True)
        return StreamingResponse(upstream.aiter_raw(), status_code=upstream.status_code,
                                 media_type=upstream.headers.get("content-type"),
                                 background=BackgroundTask(upstream.aclose))
    upstream = await vllm.post(path, json=body)
    return JSONResponse(upstream.json(), status_code=upstream.status_code)


@app.post("/v1/chat/completions")
async def chat_completions(request: Request):
    return await forward(request, "/v1/chat/completions")


@app.post("/v1/completions")
async def completions(request: Request):
    return await forward(request, "/v1/completions")

Generate the secret once (openssl rand -hex 32), keep it in your secret store, and give it to the front end only.

3. Close the internal surfaces

Surface	Why	What to do
vLLM’s API port	Requests that bypass the front end carry no salt, or one the client chose.	Reachable from the front end only.
/metrics, /load	The API key does not cover them. They count hits for all tenants.	Internal network only.
KV events stream	Carries tokens and salts in the clear.	Internal network only, or off.
The gateway’s control port	Answers lookups and deletes without authentication.	--bind-address=127.0.0.1 when vLLM shares its network namespace. Never on a tenant network.
The NVMe-oF fabric of a remote card	Storage traffic.	A storage network, not reachable from tenants.

4. Set the switches of the integration

Isolation works without a setting of the integration. Some switches make it stricter. They are off by default.

vLLM OffloadingSpec backend. vLLM does not give the salt to the backend, so the backend cannot see an unsalted request. The front end must mark every request that it leaves unsalted:

Setting	Use it when
"kv_transfer_params": {"xkv_store": false} in a request	The request has no salt, or it is traffic of one request that nobody matches again. There is no lookup and no write for it. Only the front end sets this field.
xkv.namespace	Several deployments share a card. Use one value for each model configuration. Setting or changing it starts a new cache.
xkv.require_cache_salt: true	Only a reminder: the backend writes one warning at start. The front end must still mark unsalted requests.

vLLM KV connector.

Setting	Use it when
require_cache_salt: true	Your front end salts every request. A request without a salt, or with an empty salt, then does not read or write the external cache. The connector writes one warning for the first such request.
key_namespace: "<deployment>"	Several deployments share a card and can have the same block hashes, for example two models of the same geometry. Up to 64 bytes. Setting or changing it starts a new cache.
"kv_transfer_params": {"xkv_store": false} in a request	Traffic of one request that nobody matches again. There is no lookup, no restore and no write. Nothing from the prompt goes to the card. Only the front end sets this field.

With these switches off, the storage keys of the connector are the same as in a deployment without isolation, so an upgrade needs no wipe.

--kv-transfer-config '{"kv_connector":"FusIOnXConnector",
  "kv_connector_module_path":"xkv_vllm_connector.fusionx_connector","kv_role":"kv_both",
  "kv_connector_extra_config":{"xkv_gateway_port":5557,"chunk_size":128,
    "require_cache_salt":true,"key_namespace":"prod-glm53"}}'

SGLang HiCache backend. SGLang has no field to keep a request out of the card. The front end must salt every request. Set model_id and, on a shared card, key_namespace (SGLang HiCache backend).

5. Verify it

Send the same long prompt through the front end as two tenants, before and after a restart of vLLM. Start vLLM with --enable-prompt-tokens-details so responses report cached_tokens.

FRONT=http://front-end.internal:8080
PROMPT=$(python3 -c "print(' '.join(f'Line {i}: a private document of tenant A.' for i in range(3000)))")
ask() {   # ask <api key>: cached prompt tokens of one request
  curl -s "$FRONT/v1/completions" -H "Authorization: Bearer $1" -H 'Content-Type: application/json' \
    -d "$(jq -n --arg p "$PROMPT" '{model: "glm-5.3", prompt: $p, max_tokens: 4}')" \
    | jq '.usage.prompt_tokens_details.cached_tokens'
}

ask "$KEY_TENANT_A"     # 0: the first time anyone sends it
ask "$KEY_TENANT_A"     # close to the prompt length: same tenant, same salt
ask "$KEY_TENANT_B"     # 0: another tenant, another key space
# restart vLLM (not the gateway), wait for /health, then:
ask "$KEY_TENANT_A"     # close to the prompt length: restored from the ADP card
ask "$KEY_TENANT_B"     # still 0

A non-zero answer for tenant B means a request reached vLLM without the front end’s salt, or with a salt shared between the tenants.

The full study: Isolating the KV cache between tenants.

What a salt does not cover

• Inside a scope everything is shared by design, including the cached-token counters. Make the scope no wider than the trust boundary.
• Load and eviction. Tenants still share GPU memory and the card’s capacity. A tenant can notice that the system is busy or push others’ blocks out of the cache, but cannot learn what their prompts contain.
• Other channels. Speculative decoding leaks through the number of tokens accepted per step, streaming through packet sizes and timing, a cache-aware scheduler through queue order. These need their own measures.
• The operator. Whoever runs the hosts, holds the front end’s secret or reads the KV events stream is outside this model.