SGLang-Neuron: Run SGLang Serving on AWS Trainium With the Same API

Amazon on November 12, 2026 made SGLang-Neuron a public Deep Learning Container path: the same SGLang server and HTTP APIs you already use, with Trainium execution behind an out-of-tree plugin and native PyTorch via TorchNeuron. If you have been comparing vLLM versus SGLang or following the SGLang consortium summit, this is AWS betting that familiar orchestration matters more than asking teams to learn a second inference control plane.

The post is not a benchmark sheet. It is a shipping integration: DLC tag, environment variable, launch flags, and a curl smoke test against gpt-oss-120b.

TL;DR

Architecture: what stays in SGLang versus the plugin

SGLang-Neuron architecture diagram showing SGLang core connected to AWS Trainium plugin graph runner Image: Amazon Web Services

AWS separates responsibilities cleanly:

• SGLang core still owns the API server, request lifecycle, RadixAttention, and prefix-cache bookkeeping.
• SGLang-Neuron plugin registers through SGLang entry points and implements Trainium-specific model code, attention, mixture-of-experts routing, and the compiled graph runner.
• TorchNeuron exposes Trainium as a first-class PyTorch device so the hot path stays in familiar tensors rather than a bespoke rewrite.

That split mirrors how teams already run GPU SGLang while swapping vendor kernels underneath. It also aligns with explainx.ai’s earlier TorchNeuron claimed-vs-verified thread: AWS keeps insisting PyTorch portability is the point.

PD disaggregation on Trainium

Disaggregated prefill-decode is no longer a GPU-only pattern. AWS documents:

1. A prefill server computes prompt KV activations.
2. NIXL moves KV tensors to a decode server over Elastic Fabric Adapter.
3. Each side keeps SGLang schedulers; the plugin owns Trainium execution and transfer plumbing.

If you operate multi-tenant inference, PD split is how you right-size memory for long prompts without starving decode batching. Trainium economics only matter if the transfer path is stable—EFA is AWS’s answer on their network fabric.

Quickstart commands from the AWS walkthrough

The blog’s three-step flow is copy-paste friendly:

1. Start the DLC

docker run --rm -it \
--privileged --ipc=host \
-v /path/to/work:/work \
public.ecr.aws/neuron/pytorch-inference-sglang-neuron:v0.1.0-neuron-py314-sdk-2.33.0-ubuntu26.04 \
/bin/bash

2. Launch the server

SGLANG_PLATFORM=neuron \
python -m sglang.launch_server \
--model-path openai/gpt-oss-120b \
--tp-size 8 \
--ep-size 8 \
--kv-cache-dtype fp8_e4m3 \
--context-length 10240 \
--chunked-prefill-size 8192 \
--max-running-requests 64 \
--max-total-tokens 655360 \
--mem-fraction-static 0.8 \
--host 127.0.0.1 \
--port 30000

3. Hit the OpenAI-compatible route

curl -s http://127.0.0.1:30000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "openai/gpt-oss-120b",
"messages": [
{"role": "user", "content": "Explain why the sky appears blue."}
],
"temperature": 0,
"max_tokens": 128
}'

Before you scale this to production, read the plugin repo’s supported models and validated configurations—AWS points to that matrix instead of implying every Hugging Face repo works day one.

How this fits AWS’s broader agent and RL stack

November has been dense for Bedrock and SageMaker agents—AgentCore runtime refreshes, cost-per-task guidance, and SageMaker RL walkthroughs. SGLang-Neuron targets the rollout generation slice: fast batched completions for RL environments without standing up a separate inference codebase on Trainium.

Teams already routing traffic through Bedrock inference profiles may still self-host SGLang when they need custom speculative decoding, RadixAttention prefix reuse, or open-weight checkpoints Bedrock does not list.

What people are asking

Custom Trainium silicon path for SGLang inference lets AWS shops keep one serving API

Cost and capacity planning notes

Trainium slots compete with GPU reservations inside the same AWS account governance stories you may already track—Bedrock spend caps and neocloud oversight reporting. SGLang-Neuron does not magically dodge capacity planning; it gives you another silicon knob with a familiar scheduler.

When estimating RL budgets, pair container experiments with SageMaker’s published Wordle RL economics post cited above—orders of magnitude beat guessing from blog prose alone.

Neuron Kernel Interface and torch.compile in practice

AWS is explicit that not every operator hand-rolls a custom kernel. The plugin path combines:

• Standard PyTorch ops for much of the model graph.
• NKI (Neuron Kernel Interface) kernels where Annapurna Labs needs Trainium-specific performance.
• torch.compile to fuse and schedule the resulting graph for serving.

For teams migrating from CUDA SGLang, the mental model is not “rewrite every layer in Neuron C.” It is closer to compile the same module definitions once the plugin supplies Trainium implementations for attention and MoE dispatch. That is why the announcement pairs with TorchNeuron documentation instead of a brand-new inference DSL.

You should still expect first-run compile latency and artifact caching behavior to differ from GPU servers. Budget time in CI to warm graphs before you attach production traffic.

Flag glossary for the documented gpt-oss launch

The sample launch_server command packs several production knobs:

None of these numbers are universal. They tell you what AWS validated for one MoE checkpoint, not the right settings for a 7B dense model or a vision-language stack. Treat them as a profiling baseline: sweep --chunked-prefill-size and KV dtype when your p95 prompt length differs from the blog’s smoke test.

When to pick SGLANG_PLATFORM=neuron versus GPU SGLang

Observability and failure modes

Watch the same signals you would on GPU SGLang, plus Trainium-specific compile stalls:

1. Queue depth versus --max-running-requests — saturation shows up before OOM.
2. Prefix cache hit rate — RadixAttention wins should still appear in logs.
3. PD disaggregation lag — if enabled, monitor NIXL transfer time separately from decode tokens/sec.
4. Graph recompile events — unexpected deploys or dynamic shapes can trigger new compile paths.

Pair CloudWatch with your existing Bedrock CUR metering patterns if you bill inference to internal customers.

Security and tenancy notes

The walkthrough container uses --privileged and host IPC. That is appropriate for a single-tenant benchmark box, not a multi-tenant SaaS pattern. For production:

• Run on dedicated Trainium hosts with locked-down security groups.
• Mount model weights read-only.
• Strip credentials from the shell environment before running local scans—same discipline as Codex Security CLI guidance recommends for agent hosts.

Roadmap signals from the ecosystem

RadixArk’s plugin model implies faster iteration than monolithic forks: SGLang core can ship scheduler features while Annapurna updates Trainium kernels. Watch the plugin repository for:

• Additional open-weight cards.
• Quantization modes beyond the FP8 KV example.
• Feature parity with GPU-only SGLang flags you rely on (speculative decoding, structured outputs, etc.).

explainx.ai will update this post when AWS publishes throughput tables or expands the DLC beyond the v0.1.0 tag named November 12.

Limitations AWS does not hide

• Model coverage is gated by plugin support matrices—not every MoE or multimodal architecture ships day one.
• --privileged Docker in the sample is a dev convenience; production hardening is on you.
• Performance tuning (chunked prefill sizes, mem fraction, KV dtype) still requires profiling; the sample flags are starting points only.

Related on explainx.ai

• vLLM vs SGLang comparison
• SGLang consortium summit
• TorchNeuron no-rewrite claims
• Bedrock inference profile guide
• AgentCore runtime generation
• SageMaker RL on gpt-oss
• Bedrock per-principal spend caps

Getting help and reporting issues

AWS asks teams to try workloads, open issues on the plugin repository, and share feedback on missing models. That is standard for day-zero DLCs, but it matters operationally: your production SLO should include a fallback route to GPU SGLang until the neuron platform marks your checkpoint as supported. Document that fallback in internal runbooks the same way you would document multi-inference-profile routing on Bedrock.

If you are an AWS partner building managed inference, treat SGLANG_PLATFORM=neuron as a feature flag per tenant—not a global flip—so you can roll back one customer without draining an entire region. Keep a pinned GPU container tag in the same runbook so rollback is one environment variable change, not a rebuild.

Container tags, SDK versions, and supported model lists follow the AWS ML announcement as of November 12, 2026.