Amazon SageMaker AI added KV cache-aware routing to real-time LLM endpoints on November 17, 2026. The new KV_AWARE strategy reads live KV cache state from vLLM model containers, routes each request toward the instance that already computed the longest matching prefix, and falls back to load-aware routing when every warm node is busy or caches miss.
If you serve multi-turn agents, long RAG system prompts, or shared tool schemas on SageMaker instead of only on Bedrock, this is the first managed router that treats cache eviction as first-class—complementing, not replacing, prefix-aware routing.
Why prefix-aware routing was not enough
AWS introduced prefix-aware routing earlier in its inference series. That mode hashes the first N bytes (or characters on the OpenAI-compatible path) of each request and sticks conversations to one instance as long as the hash is stable.
That breaks down in two common agent patterns AWS names in its November 17 post:
- Eviction under memory pressure — the router still sends matching prefixes to an instance that already dropped those KV entries from GPU RAM.
- Growing chats — until the transcript exceeds PrefixLength, every new user turn changes the hash, so multi-turn threads bounce across instances and lose warm caches.
KV-aware routing fixes both by indexing token spans the container reports as cached, not just the first fixed slice of the HTTP body.
How KV_AWARE works (managed index)
SageMaker KV-aware routing sends requests to the instance with the longest cached prefix match
Image: AWS
The AWS design has three moving parts:
| Piece | Role |
|---|---|
| Container events | With SM_VLLM_KV_EVENTS_CONFIG, vLLM publishes store/evict events over ZeroMQ inside the instance |
| Managed cache index | SageMaker keeps one-way truncated hashes → instance mappings; no prompts or tensors stored in the index |
| Router | Tokenizes the incoming prompt, finds candidates with the longest cached span, skips overloaded instances, else load-aware fallback |
Routing activates only with at least two instances (or two copies of an inference component). A single-instance endpoint accepts KV_AWARE in config but always sends traffic to that lone worker.
For dynamic LoRA endpoints, AWS says KV-aware choice runs among instances that already loaded the adapter.
Routing strategies at a glance
| Strategy | Best for |
|---|---|
| RANDOM | Non-LLM or interchangeable requests |
| LEAST_OUTSTANDING_REQUESTS | Variable decode times, keep GPUs even |
| PREFIX_AWARE | Stable shared prefix, short prompts, any prefix-caching container |
| KV_AWARE (new) | vLLM DLC, eviction, long histories, partial shared context |
AWS measured 1.3–1.9 ms added latency for prefix-aware in its earlier post; KV-aware adds a few milliseconds on short prompts (mostly tokenization), scaling with prompt length.
Enable it: environment and endpoint config
AWS documents a two-step enablement on the same blog page:
1. Model environment (KV events on):
"Environment": {
"SM_VLLM_MODEL": "<model-id-or-/opt/ml/model>",
"SM_VLLM_KV_EVENTS_CONFIG": "{\"enable_kv_cache_events\": true, \"publisher\": \"zmq\", \"endpoint\": \"tcp://*:5557\", \"topic\": \"kv-events\", \"replay_endpoint\": \"tcp://*:5658\"}"
}
2. Endpoint RoutingConfig:
"RoutingConfig": {
"RoutingStrategy": "KV_AWARE"
}
Requirements at launch: GPU real-time endpoints, OpenAI-compatible Completions or Chat payloads, Boto3 ≥ 1.43.31 (or AWS CLI 2.35.6+) if you want detailed observability metrics such as KV hit rate via PromQL.
Not supported: InvokeEndpointWithBidirectionalStream, non-null cache_salt fields (rejected with HTTP 400), and mismatched model fields versus SM_VLLM_MODEL.
When builders should switch
Choose KV_AWARE when:
- Chat histories grow every turn and you care about TTFT, not only tokens/sec at saturation.
- Many tenants share overlapping RAG corpora or tool instructions but not identical first-256-byte prefixes.
- You already run multi-instance p5 pools where prefix-aware metrics show cache hit cliffs after load spikes.
Stay on PREFIX_AWARE when:
- Your container is not the supported vLLM DLC yet.
- Prompts are short and prefix-stable with plenty of GPU headroom—AWS says both strategies behave similarly when nothing evicts.
For Bedrock-only teams, compare this with AgentCore memory patterns and AWS agent registry discovery—routing here is inference-plane, not tool metadata.
Benchmark claims (vendor, workload-specific)
SageMaker response showing cached prompt tokens and KV cache hit rate in Insights
Image: AWS
On Llama 3.1 70B Instruct, seven ml.p5.48xlarge instances, 20–100 simulated users, and prompts up to ~64K tokens, AWS reports versus random routing:
- Up to 97% lower p50 time to first token
- Up to 234% higher throughput
- KV hit rate from 3–48% up to ~80–93%
Gains shrink on short prompts where little prefill can be skipped. Run your own A/B with cached_tokens per response before promising SLAs to product teams.
Errors and edge cases AWS documents
KV-aware routing introduces failure modes beyond standard inference errors. AWS lists these explicitly:
modelmismatch — if the client names a model ID the container does not serve, SageMaker returns ModelError.cache_saltrejection — requests with a non-null top-levelcache_saltfield get HTTP 400 (KV-aware mode does not implement the prefix-isolation salt path).- Scale events — scale-in drops index entries for removed instances; scale-out starts with cold caches, so expect TTFT spikes until sessions warm.
- CUDA 13 images — production variants may need
InferenceAmiVersionsuch asal2023-ami-sagemaker-inference-gpu-4-1or containers fail to start.
None of these replace application-level retries. Agent frameworks that automatically replay failed tool calls should dedupe side effects when a routed request lands on a cold instance after autoscaling.
CloudFormation and quotas
AWS notes you can configure KV_AWARE through CloudFormation the same way as other routing strategies on production variants. Before you cut traffic over:
- Confirm service quotas for your instance type × instance count (p5 pools exhaust quotas fast).
- Stage a duplicate endpoint with 10% shadow traffic if your org requires change-advisory boards.
- Document rollback to PREFIX_AWARE—only the environment variable and strategy enum change, so rollback is minutes, not a model rebuild.
For teams comparing self-hosted vLLM against SageMaker, pair this launch with vLLM throughput claims on Rubin only as vendor context; your TTFT win here is routing, not a new silicon generation.
Cost and capacity planning
KV-aware routing does not reduce per-token model FLOPs. It reduces repeated prefill work when caches hit. Economic gains show up as:
- Higher effective throughput on the same GPU count (AWS cites up to 234% in its benchmark scenario).
- Delayed scale-out because each instance serves more multi-turn sessions before saturating.
It can also concentrate load on the warmest instance until load-aware fallback kicks in—watch GPU memory headroom so hot instances do not enter thrashing eviction loops that defeat the router.
If you run agents with 100K-token system prompts, combine KV_AWARE with long-context sparse attention research on the model side and prompt compression on the application side. Routing cannot fix quadratic attention inside the layer.
Choosing between Bedrock, SageMaker, and DIY vLLM
| Host | When KV_AWARE matters |
|---|---|
| SageMaker real-time | You already pay for multi-instance GPU endpoints and OpenAI-compatible clients |
| Bedrock | Managed models with different routing internals—this launch is not a Bedrock feature |
| EKS/self-managed vLLM | You would implement custom consistent hashing or sticky sessions yourself |
Organizations standardizing on AWS agent registry metadata for governance often keep inference on SageMaker for VPC-only endpoints. KV_AWARE is a reason to consolidate chat traffic on those endpoints instead of one-off GPU boxes per team.
Verify routing in production
CloudWatch latency comparison of RANDOM versus KV_AWARE SageMaker endpoints
Image: AWS
AWS suggests three checks:
SM_VLLM_ENABLE_PROMPT_TOKENS_DETAILS=true→ readcached_tokenson follow-up turns.EnableDetailedObservabilityon the endpoint → PromQL KV cache hit rate and TTFT in CloudWatch.- Side-by-side endpoint with PREFIX_AWARE or RANDOM on identical traffic mirrors.
Cached tokens come from vLLM's automatic prefix caching, which reuses KV blocks from earlier requests and speeds up prefill, not decoding.
KV-aware routing ignores prefix-isolation headers from the prefix-aware article—multi-tenant designs must enforce isolation above the router (separate endpoints or strict auth), as with any shared GPU pool.
How this fits the wider agent inference stack
Long-context agents stress both attention kernels and serving economics. NVIDIA's sparse attention co-design guidance attacks compute inside the model; KV_AWARE attacks where repeated prefill runs. Teams shipping Together prefix caches for agents or Baseten frontier inference guides should add managed cache indices to the comparison matrix when AWS is the host.
If you route tools through MCP on AgentCore, remember: tool latency plus TTFT sets perceived agent speed—routing wins show up in second-turn chats, not cold-start demos.
Step-by-step rollout for a typical agent endpoint
- Clone an existing prefix-aware endpoint to a staging name with two instances minimum.
- Bake
SM_VLLM_KV_EVENTS_CONFIGinto the model; redeploy cannot rely on endpoint-only changes. - Flip
RoutingStrategytoKV_AWAREon the staging variant; run synthetic multi-turn scripts that grow history past PrefixLength each turn. - Compare
cached_tokensand p50 TTFT against staging PREFIX_AWARE for 30+ minutes under synthetic load. - Promote to production during a low-traffic window; keep PREFIX_AWARE endpoint id for instant rollback.
Document client behavior: mobile apps that retry on slow first tokens can accidentally stampede new instances after deploy—use jittered backoff.
What AWS did not ship yet
The November 17 post is explicit about launch scope:
- No first-party dashboard tile for KV routing decisions—operators rely on
cached_tokensand CloudWatch PromQL. - No bidirectional streaming path.
- Container allowlist—only specified vLLM DLC builds.
If your team runs custom vLLM forks, plan to stay on PREFIX_AWARE until AWS expands support or you contribute compatibility upstream.
Security and tenancy notes
KV indices store hashes, not prompts, but co-tenant risk remains: two customers hitting the same endpoint with guessable prefixes could theoretically probe cache warmth (a side channel class AWS does not discuss in the launch post). For regulated workloads, prefer dedicated endpoints per customer or per environment, the same guidance as for Snowflake Cortex OAuth patterns on AgentCore.
Reading gets you the idea. Building it with feedback gets you the skill.
