explainx.ai350K+ learn from us
TrendingAI News TodayPathwaysSkills
Pricing
explainx.ai

Upskill in AI — 16 free pathways, live workshops & bootcamps, and 50+ courses from practitioners. Plus the skills, tools, and MCP servers to practice on.

follow us

follow on google

Add explainx.ai as a preferred source

corporate training

support@explainx.ai

get started

Find your pathTake Free Evaluation

community

Join the community

learn

mind: share how you thinkpathways — start freeworkshopsbootcampscoursescompare Explainxcertificationsmock testsexplainx universitycorporate traininglearn skills & mcp

discover

skillsmcp serversexplainx mcpestacktoolsmdx readeragentsllmsdesignsdictionarypeopleagi trackerfelony benchranks

company

aboutvisionmissionteaminstructorsteach on explainxpartnershipscommunityhackathonscareers

content

daily AI newsstate of AI — live resultsblogreleasespromptsgeneratorsresource libraryfor LLMsexplainx.ai kids

solutions

all solutionsdeveloper upskillingmarketing upskillingproduct manager upskillingleadership upskilling

newsletter · weekly

Get AI news, tools, and insights in your inbox.

supportcontactprivacytermsdata rightshow we create contentsubmission guidelines

© 2026 AISOLO Technologies Pvt Ltd

explainx.ai

On this page

  • Why prefix-aware routing was not enough
  • How KV_AWARE works (managed index)
  • Routing strategies at a glance
  • Enable it: environment and endpoint config
  • When builders should switch
  • Benchmark claims (vendor, workload-specific)
  • Errors and edge cases AWS documents
  • CloudFormation and quotas
  • Cost and capacity planning
  • Choosing between Bedrock, SageMaker, and DIY vLLM
  • Verify routing in production
  • How this fits the wider agent inference stack
  • Step-by-step rollout for a typical agent endpoint
  • What AWS did not ship yet
  • Security and tenancy notes
  • Related reading
← Back to blog

explainx / blog

SageMaker KV Cache-Aware Routing: What KV_AWARE Changes for Multi-Turn LLM Endpoints

AWS, SageMaker, LLM Inference, vLLM, Agents

Part of Microsoft, Apple and Amazon AI

AWS added KV cache-aware routing to SageMaker AI real-time endpoints Nov 17, 2026. Route requests to the instance holding the longest KV match on vLLM DLCs.

Nov 17, 2026·9 min read·Yash Thakker
add explainx.ai
go deep
SageMaker KV Cache-Aware Routing: What KV_AWARE Changes for Multi-Turn LLM Endpoints

Amazon SageMaker AI added KV cache-aware routing to real-time LLM endpoints on November 17, 2026. The new KV_AWARE strategy reads live KV cache state from vLLM model containers, routes each request toward the instance that already computed the longest matching prefix, and falls back to load-aware routing when every warm node is busy or caches miss.

If you serve multi-turn agents, long RAG system prompts, or shared tool schemas on SageMaker instead of only on Bedrock, this is the first managed router that treats cache eviction as first-class—complementing, not replacing, prefix-aware routing.

Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

Why prefix-aware routing was not enough

AWS introduced prefix-aware routing earlier in its inference series. That mode hashes the first N bytes (or characters on the OpenAI-compatible path) of each request and sticks conversations to one instance as long as the hash is stable.

That breaks down in two common agent patterns AWS names in its November 17 post:

  1. Eviction under memory pressure — the router still sends matching prefixes to an instance that already dropped those KV entries from GPU RAM.
  2. Growing chats — until the transcript exceeds PrefixLength, every new user turn changes the hash, so multi-turn threads bounce across instances and lose warm caches.

KV-aware routing fixes both by indexing token spans the container reports as cached, not just the first fixed slice of the HTTP body.

How KV_AWARE works (managed index)

SageMaker KV-aware routing sends requests to the instance with the longest cached prefix matchSageMaker KV-aware routing sends requests to the instance with the longest cached prefix match Image: AWS

The AWS design has three moving parts:

table · 2 cols
PieceRole
Container eventsWith SM_VLLM_KV_EVENTS_CONFIG, vLLM publishes store/evict events over ZeroMQ inside the instance
Managed cache indexSageMaker keeps one-way truncated hashes → instance mappings; no prompts or tensors stored in the index
RouterTokenizes the incoming prompt, finds candidates with the longest cached span, skips overloaded instances, else load-aware fallback

Routing activates only with at least two instances (or two copies of an inference component). A single-instance endpoint accepts KV_AWARE in config but always sends traffic to that lone worker.

For dynamic LoRA endpoints, AWS says KV-aware choice runs among instances that already loaded the adapter.

Routing strategies at a glance

table · 2 cols
StrategyBest for
RANDOMNon-LLM or interchangeable requests
LEAST_OUTSTANDING_REQUESTSVariable decode times, keep GPUs even
PREFIX_AWAREStable shared prefix, short prompts, any prefix-caching container
KV_AWARE (new)vLLM DLC, eviction, long histories, partial shared context

AWS measured 1.3–1.9 ms added latency for prefix-aware in its earlier post; KV-aware adds a few milliseconds on short prompts (mostly tokenization), scaling with prompt length.

Enable it: environment and endpoint config

AWS documents a two-step enablement on the same blog page:

1. Model environment (KV events on):

json
"Environment": {
  "SM_VLLM_MODEL": "<model-id-or-/opt/ml/model>",
  "SM_VLLM_KV_EVENTS_CONFIG": "{\"enable_kv_cache_events\": true, \"publisher\": \"zmq\", \"endpoint\": \"tcp://*:5557\", \"topic\": \"kv-events\", \"replay_endpoint\": \"tcp://*:5658\"}"
}

2. Endpoint RoutingConfig:

json
"RoutingConfig": {
  "RoutingStrategy": "KV_AWARE"
}

Requirements at launch: GPU real-time endpoints, OpenAI-compatible Completions or Chat payloads, Boto3 ≥ 1.43.31 (or AWS CLI 2.35.6+) if you want detailed observability metrics such as KV hit rate via PromQL.

Not supported: InvokeEndpointWithBidirectionalStream, non-null cache_salt fields (rejected with HTTP 400), and mismatched model fields versus SM_VLLM_MODEL.

When builders should switch

Choose KV_AWARE when:

  • Chat histories grow every turn and you care about TTFT, not only tokens/sec at saturation.
  • Many tenants share overlapping RAG corpora or tool instructions but not identical first-256-byte prefixes.
  • You already run multi-instance p5 pools where prefix-aware metrics show cache hit cliffs after load spikes.

Stay on PREFIX_AWARE when:

  • Your container is not the supported vLLM DLC yet.
  • Prompts are short and prefix-stable with plenty of GPU headroom—AWS says both strategies behave similarly when nothing evicts.

For Bedrock-only teams, compare this with AgentCore memory patterns and AWS agent registry discovery—routing here is inference-plane, not tool metadata.

Benchmark claims (vendor, workload-specific)

SageMaker response showing cached prompt tokens and KV cache hit rate in InsightsSageMaker response showing cached prompt tokens and KV cache hit rate in Insights Image: AWS

On Llama 3.1 70B Instruct, seven ml.p5.48xlarge instances, 20–100 simulated users, and prompts up to ~64K tokens, AWS reports versus random routing:

  • Up to 97% lower p50 time to first token
  • Up to 234% higher throughput
  • KV hit rate from 3–48% up to ~80–93%

Gains shrink on short prompts where little prefill can be skipped. Run your own A/B with cached_tokens per response before promising SLAs to product teams.

Errors and edge cases AWS documents

KV-aware routing introduces failure modes beyond standard inference errors. AWS lists these explicitly:

  • model mismatch — if the client names a model ID the container does not serve, SageMaker returns ModelError.
  • cache_salt rejection — requests with a non-null top-level cache_salt field get HTTP 400 (KV-aware mode does not implement the prefix-isolation salt path).
  • Scale events — scale-in drops index entries for removed instances; scale-out starts with cold caches, so expect TTFT spikes until sessions warm.
  • CUDA 13 images — production variants may need InferenceAmiVersion such as al2023-ami-sagemaker-inference-gpu-4-1 or containers fail to start.

None of these replace application-level retries. Agent frameworks that automatically replay failed tool calls should dedupe side effects when a routed request lands on a cold instance after autoscaling.

CloudFormation and quotas

AWS notes you can configure KV_AWARE through CloudFormation the same way as other routing strategies on production variants. Before you cut traffic over:

  1. Confirm service quotas for your instance type × instance count (p5 pools exhaust quotas fast).
  2. Stage a duplicate endpoint with 10% shadow traffic if your org requires change-advisory boards.
  3. Document rollback to PREFIX_AWARE—only the environment variable and strategy enum change, so rollback is minutes, not a model rebuild.

For teams comparing self-hosted vLLM against SageMaker, pair this launch with vLLM throughput claims on Rubin only as vendor context; your TTFT win here is routing, not a new silicon generation.

Cost and capacity planning

KV-aware routing does not reduce per-token model FLOPs. It reduces repeated prefill work when caches hit. Economic gains show up as:

  • Higher effective throughput on the same GPU count (AWS cites up to 234% in its benchmark scenario).
  • Delayed scale-out because each instance serves more multi-turn sessions before saturating.

It can also concentrate load on the warmest instance until load-aware fallback kicks in—watch GPU memory headroom so hot instances do not enter thrashing eviction loops that defeat the router.

If you run agents with 100K-token system prompts, combine KV_AWARE with long-context sparse attention research on the model side and prompt compression on the application side. Routing cannot fix quadratic attention inside the layer.

Choosing between Bedrock, SageMaker, and DIY vLLM

table · 2 cols
HostWhen KV_AWARE matters
SageMaker real-timeYou already pay for multi-instance GPU endpoints and OpenAI-compatible clients
BedrockManaged models with different routing internals—this launch is not a Bedrock feature
EKS/self-managed vLLMYou would implement custom consistent hashing or sticky sessions yourself

Organizations standardizing on AWS agent registry metadata for governance often keep inference on SageMaker for VPC-only endpoints. KV_AWARE is a reason to consolidate chat traffic on those endpoints instead of one-off GPU boxes per team.

Verify routing in production

CloudWatch latency comparison of RANDOM versus KV_AWARE SageMaker endpointsCloudWatch latency comparison of RANDOM versus KV_AWARE SageMaker endpoints Image: AWS

AWS suggests three checks:

  1. SM_VLLM_ENABLE_PROMPT_TOKENS_DETAILS=true → read cached_tokens on follow-up turns.
  2. EnableDetailedObservability on the endpoint → PromQL KV cache hit rate and TTFT in CloudWatch.
  3. Side-by-side endpoint with PREFIX_AWARE or RANDOM on identical traffic mirrors.

Cached tokens come from vLLM's automatic prefix caching, which reuses KV blocks from earlier requests and speeds up prefill, not decoding.

KV-aware routing ignores prefix-isolation headers from the prefix-aware article—multi-tenant designs must enforce isolation above the router (separate endpoints or strict auth), as with any shared GPU pool.

How this fits the wider agent inference stack

Long-context agents stress both attention kernels and serving economics. NVIDIA's sparse attention co-design guidance attacks compute inside the model; KV_AWARE attacks where repeated prefill runs. Teams shipping Together prefix caches for agents or Baseten frontier inference guides should add managed cache indices to the comparison matrix when AWS is the host.

If you route tools through MCP on AgentCore, remember: tool latency plus TTFT sets perceived agent speed—routing wins show up in second-turn chats, not cold-start demos.

Step-by-step rollout for a typical agent endpoint

  1. Clone an existing prefix-aware endpoint to a staging name with two instances minimum.
  2. Bake SM_VLLM_KV_EVENTS_CONFIG into the model; redeploy cannot rely on endpoint-only changes.
  3. Flip RoutingStrategy to KV_AWARE on the staging variant; run synthetic multi-turn scripts that grow history past PrefixLength each turn.
  4. Compare cached_tokens and p50 TTFT against staging PREFIX_AWARE for 30+ minutes under synthetic load.
  5. Promote to production during a low-traffic window; keep PREFIX_AWARE endpoint id for instant rollback.

Document client behavior: mobile apps that retry on slow first tokens can accidentally stampede new instances after deploy—use jittered backoff.

What AWS did not ship yet

The November 17 post is explicit about launch scope:

  • No first-party dashboard tile for KV routing decisions—operators rely on cached_tokens and CloudWatch PromQL.
  • No bidirectional streaming path.
  • Container allowlist—only specified vLLM DLC builds.

If your team runs custom vLLM forks, plan to stay on PREFIX_AWARE until AWS expands support or you contribute compatibility upstream.

Security and tenancy notes

KV indices store hashes, not prompts, but co-tenant risk remains: two customers hitting the same endpoint with guessable prefixes could theoretically probe cache warmth (a side channel class AWS does not discuss in the launch post). For regulated workloads, prefer dedicated endpoints per customer or per environment, the same guidance as for Snowflake Cortex OAuth patterns on AgentCore.

Reading gets you the idea. Building it with feedback gets you the skill.

Related reading

  • SageMaker MIG multi-instance GPU endpoints
  • Bedrock AgentCore multi-user memory
  • AWS Agent Registry runtime discovery
  • AWS responsible AI agentic roadmap
  • Together Ollama prefix cache for agents
  • Baseten efficient frontier LLM inference
Spotted something out of date? Let us know.
Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

  • ✓ Free
  • ✓ One email a day, no spam
  • ✓ Unsubscribe anytime
Yash Thakker

Written by

Yash Thakker

Yash is an AI expert with over 300K learners. Join his workshops →

View Yash Thakker in People in AI →

Related posts

Oct 19, 2026

vLLM Push-Based and Bi-Directional KV Transfer: What AWS Measured on P5en

Two AWS posts describe new KV cache transfer modes for vLLM prefill-decode disaggregation. One pushes KV from prefill to decode, the other lets prefill reuse decode-side context across turns. Here is what changed, the numbers, and the fine print.

Nov 13, 2026

OpenFold3 on SageMaker JumpStart: Async Protein Folding in AWS

On November 13, 2026 AWS published a SageMaker JumpStart guide for OpenFold3-preview — the OpenFold Consortium structure predictor — with asynchronous inference, optional private MSAs, multi-GPU parallelism, and scale-to-zero to cut idle GPU bills. explainx.ai walks through what changed and how it compares to BioNeMo paths we already cover.

Nov 12, 2026

Amazon Bedrock: Four Request Settings That Change Your Inference Bill

On November 12, 2026 AWS published measured tests for four Amazon Bedrock inference choices that change token counts and rates even when your prompt stays the same—cache duration, Flex service tier, adaptive thinking effort, and maxTokens. This guide translates the tables into decisions for agents, eval harnesses, and multi-tenant apps.