Route Coding Agents on Bedrock with LiteLLM and NVIDIA NeMo Switchyard

AWS published an end-to-end sample on November 13, 2026 that routes coding agent traffic on Amazon Bedrock with NVIDIA NeMo Switchyard inside LiteLLM, then benchmarks cost per solved task instead of token list price. The post extends the same week’s Bedrock cost-per-task guide with runnable infrastructure: Fargate, private subnets, WAF, and an OpenAI-compatible endpoint your agent already understands.

Primary source: Route coding agent requests to the right model on Amazon Bedrock with NVIDIA NeMo Switchyard. Sample repo: aws-samples/sample-route-coding-agent-requests-Amazon-Bedrock-nemo-switchyard (MIT-0, sample code for non-production evaluation).

TL;DR

Why cost per solved task matters here

Coding agents such as Claude Code and OpenAI Codex CLI loop: read files, edit, run tests, retry. AWS's Part 2 post already argued list price per token hides verbosity, tokenizer differences, and extra turns. The Switchyard sample makes that argument measurable: deploy routing, run the same agent harness against routed and single-model LiteLLM groups, divide dollars by green tests.

Failed attempts still spend money. Routing that cuts retries can beat a cheaper per-token model — the same lesson as the illustrative 65% savings table in Part 2.

Architecture AWS describes

1. Agent sends OpenAI-compatible POST /v1/chat/completions with "model": "switchyard" and a gateway key.
2. Route 53 resolves your hostname to the ALB (or you CNAME to LoadBalancerDnsName with your own cert).
3. AWS WAF applies IP reputation, known-bad-inputs, and 2,000 requests / 5 min / IP before traffic hits the task.
4. The ALB forwards only /v1/chat/completions, /v1/models, and /health/liveliness; other paths return 404 to block the LiteLLM admin UI.
5. LiteLLM validates the key, invokes Switchyard, calls Bedrock Converse or ConverseStream through a VPC interface endpoint.
6. Responses stream back; x-litellm-model-name exposes which backend answered while JSON model stays switchyard.

Secrets Manager holds the gateway secret; the task prefixes sk- at startup. Profiles live in profiles/stage/litellm.yaml and switchyard.toml, injected via environment variables at container launch.

Model routing gateway selecting efficient vs capable Bedrock coding models

Stage routing defaults

From profiles/stage/switchyard.toml in the sample:

• algorithm = "stage"
• picker = "efficient_first" — inconclusive signals stay on the efficient candidate (Nemotron for a new task with no tool history)
• confidence_threshold = 0.5
• recent_window = 3 tool results inspected
• only_on_wrong_signal_escalation = true

Stage requires exactly two candidates in the LiteLLM group, listed capable first, efficient second in YAML. The smoke test checks that plain requests pick Nemotron and that a recorded failure history escalates to Opus.

Random routing (routingProfile=random) uses algorithm = "random" with weights for controlled A/B tests.

AWS notes Task and Composite Switchyard strategies need extra judge calls this LiteLLM integration does not support yet.

Deploy and test (outline)

Prereqs from AWS: non-prod account with Bedrock access to global Claude Opus 5.5 and Nemotron 3 Super 120B in a tested Region (post uses us-east-1), Node 20+, Docker BuildKit, Python 3.11+, CDK bootstrapped.

High-level commands from the blog:

git clone https://github.com/aws-samples/sample-route-coding-agent-requests-Amazon-Bedrock-nemo-switchyard.git
cd sample-route-coding-agent-requests-Amazon-Bedrock-nemo-switchyard
npm ci
scripts/preflight.sh
npx cdk deploy -c allowedCidr="$ALLOWED_CIDR" -c domainName="$GATEWAY_DOMAIN" -c hostedZone="$HOSTED_ZONE"

Local trial uses ROUTING_PROFILE=stage docker compose up — still bills Bedrock.

Point your agent's OpenAI base URL at the gateway, API key at the secret, model at switchyard. Stage needs full conversation history with tool messages on each request.

For enterprise identity patterns, pair this with Bedrock spend caps per IAM principal and Codex plus AgentCore Gateway SSO.

Benchmark tasks AWS ships

The sample repo includes bench/run.py for single-task coding loops and bench/multiturn.py for longer agentic conversations. AWS documents four starter tasks:

A solved task means unit tests pass within the default six model turns (raise to 12 for calc). The runner saves transcripts under bench/results/ and prints RUNS, SOLVED, ERRORS, COST/TASK, COST/SOLVED.

Cost per solved task equals estimated spend on all runs in a model group divided by runs that ended green. AWS warns that a group with lower cost per attempt but fewer solves can lose on cost per solved task because failed attempts still bill.

Pricing comes from a PRICES table inside bench/run.py, keyed off x-litellm-model-name and token counts. Missing price rows mark runs unpriced rather than zero — a honesty detail FinOps teams should copy.

bench/multiturn.py adds scripted seven-turn chats mixing simple checks with design, debug, and coding prompts, with --max-conv-cost (default $1) capping each conversation. That better mimics agents that chatter before editing.

Security warnings AWS repeats

The blog is explicit about runner risk:

• Model-written Python executes on your machine during benchmarks.
• Test subprocesses lack AWS creds in env, but are not OS sandboxes.
• Generated code can read files your user can read and use the network.
• Use disposable dev machines without long-lived AWS keys on disk.
• Protect bench/results/ — they store prompts, completions, and patches.

That mirrors why enterprises route agents through AgentCore isolated runtimes instead of laptops.

Gateway hardening in the sample:

• Secrets Manager master key with sk- prefix at runtime only.
• WAF rate limits and reputation lists; body rules omitted so agent payloads are not falsely blocked.
• ALB path allowlist blocks LiteLLM admin UI routes.
• allowedCidr required — stack refuses open CIDR synthesis.

AWS nudges teams toward HTTPS only; plain HTTP opt-in logs a warning every synth because keys and source cross the wire unencrypted.

IAM and residency details

The sample derives Bedrock invoke policies from litellm.yaml model strings. Global inference profiles use wildcard regions; AWS notes data may process outside your home Region. Swap to geographic profiles if contracts require it.

Interface VPC endpoints keep inference traffic off the public internet even when the ALB faces your office IP.

How to read benchmark output without fooling yourself

AWS lists interpretation rules worth quoting for any agent eval:

1. Compare solve counts before celebrating low cost per task.
2. Break out hard tasks separately — Stage only escalates after failure signals, so it may burn cash replaying failed histories.
3. Increase --repeats until solve-rate gaps stabilize; consider Fisher's exact test on small samples.
4. Match --max-turns to production agent limits.
5. Watch max_tokens truncation — truncated replies cheapen measured cost but poison later turns.
6. Re-benchmark when you change candidates, thresholds, or profiles.

Those bullets are vendor advice, but they are also the difference between a demo win and a production routing policy.

Switchyard beyond this sample

We first explained Switchyard in NVIDIA's eight-node DGX Spark cluster post, where a classifier routed between MiniCPM and Qwen replicas. The AWS sample moves the same library into enterprise Bedrock behind LiteLLM — closer to how most teams already centralize model keys.

LiteLLM also ships its own Switchyard stage router, which AWS mentions as an alternative auto-router strategy. Evaluate both if you already run LiteLLM without CDK.

Safety and compliance notes

AWS repeats: sample only, work with security and legal, deploy in a non-production account, and treat switchyard-server 0.3.0 as demo/eval quality. Prompts, tool output, and source code cross the gateway; allowedPlainHttp=true is for short tests only.

Global inference profiles may process outside your home Region. If data residency matters, swap model IDs per AWS guidance in the post.

Related reading

• Bedrock cost per task Part 2
• Self-service LiteLLM gateway on Bedrock
• AgentCore runtime next generation
• DGX Spark Switchyard cluster
• SGLang on AWS Trainium
• Loop engineering for coding agents

When to pick Random routing instead of Stage

AWS's Random profile exists for A/B tests, not production savings. Use it when you need a controlled 50/50 split (or other weights) to compare solve rates without history-based escalation. Stage fits production agent loops where early turns are cheap reads and late turns follow tool failures.

If your agent emits sparse tool history — for example, single-shot refactors — Stage may never escalate because only_on_wrong_signal_escalation waits for failure signals. In that regime, measure whether you should start on the capable model or adjust confidence_threshold downward with care.

Cost accounting tie-in for platform teams

Pair Switchyard measurements with CUR 2.0 Bedrock tagging so gateway spend rolls up per team. Tag the CloudFormation stack, map x-litellm-model-name values to cost centers, and reconcile bench estimates weekly until PRICES matches your negotiated rates.

For coding agents already on Codex plus Bedrock, the SSO AgentCore gateway guide shows identity patterns; Switchyard sits downstream of identity, at the model pick layer.

Cleanup and ongoing operations

AWS ends the walkthrough with cdk destroy, noting the master key secret is deleted without recovery and the bootstrap ECR image may remain. Keep CloudWatch log retention in mind if compliance requires longer inference audit trails than the sample's default 30-day log group retention. Rotate gateway keys after any plain-HTTP trial. If you change profiles/stage/switchyard.toml, rerun npm run validate:profile before deploy so LiteLLM does not fail at container start — a failure mode that is easy to miss when only the CDK stack changes.

AWS blog steps, version pins, and repository layout are accurate as of November 13, 2026.