Amazon SageMaker HyperPod now has a Global Inference Gateway: one managed endpoint that routes requests across every HyperPod cluster you register, in one AWS Region or several. AWS published the launch on November 20, 2026. The gateway authenticates each call with SigV4 and IAM, scores clusters on live load, and carries traffic over AWS PrivateLink. Inside each cluster, the existing HyperPod Inference Gateway still picks the GPU pod, unchanged from the architecture AWS described earlier this year.
If you already run HyperPod inference in more than one cluster, this is the piece that removes custom cross-Region DNS, load balancers, and manual failover runbooks.
TL;DR
| Question | Answer |
|---|---|
| What shipped? | Global Inference Gateway for SageMaker HyperPod |
| When? | November 20, 2026, per AWS Machine Learning Blog |
| What problem does it solve? | Idle GPUs in another cluster or Region while requests queue on a saturated cluster |
| Client surface | One regional gateway endpoint; OpenAI-compatible requests signed with SigV4 |
| Cluster tier | Unchanged per-cluster Inference Gateway picks the pod |
| Default routing weights | KV headroom 0.20, queue depth 0.25, Region proximity 0.40, error rate 0.15 |
| Regions | Thirteen AWS Regions listed in the launch post |
| Security | IAM plus PrivateLink; AWS says prompts and completions are not logged |
How does the Global Inference Gateway fit HyperPod?
HyperPod teams spread models across multiple clusters and Regions when a single rack is not enough. Before this gateway, AWS's own example counted twenty client endpoints for four models on five clusters in two Regions (one endpoint per model per cluster). The per-cluster HyperPod Inference Gateway collapses that to five cluster endpoints, but each application still chooses a cluster.
The Global Inference Gateway is the second routing tier. AWS positions it immediately after the September 2026 per-cluster gateway in the same launch narrative. Our part 2 review of SageMaker inference launches already indexed the per-cluster gateway as launch 23; this post is the cross-cluster extension AWS teased in that architecture story.
Do not confuse it with SageMaker cross-Region inference routing for ordinary SageMaker AI endpoints. That router, which became generally available November 19, 2026, targets standard endpoints and inference components. The Global Inference Gateway only applies to HyperPod clusters where you enable spec.globalRouting in InferenceGatewayConfig.
What happens on each request?
AWS describes four steps in its architecture figure (see the launch post figure):
- The client signs the request with SigV4 and calls the gateway's regional endpoint.
- The gateway authenticates the caller and scores every healthy registered cluster.
- AWS PrivateLink carries the request to the chosen cluster, same Region or not.
- The cluster's Inference Gateway routes to a model pod the way it does today.
Each cluster publishes load back to the gateway through a service-managed metrics publisher with its own IAM role. AWS operates the routing fleet that terminates TLS, enforces rate limits, and health-checks clusters.
How does cluster scoring work?
For every request the gateway combines four signals. Default weights, from the launch post:
| Signal | Default weight | What it measures |
|---|---|---|
| KVHeadroom | 0.20 | Available KV cache and GPU memory |
| QueueDepth | 0.25 | Backlog on the cluster |
| RegionProximity | 0.40 | Distance between cluster and gateway Region |
| ErrorRate | 0.15 | Recent failed requests |
Defaults favor nearby Regions but still shift load toward clusters with spare capacity and away from failing ones. If resilience matters more than latency, AWS's example raises ErrorRate to 0.7 and lowers the others:
{"Weights": {"KVHeadroom": 0.1, "QueueDepth": 0.1, "RegionProximity": 0.1, "ErrorRate": 0.7}}
Pass weights in CreateGlobalInferenceGateway and change them later with UpdateGlobalInferenceGateway. In-flight requests finish under the old weights while new requests pick up updates.
This design rhymes with KV-cache-aware routing on single-cluster SageMaker endpoints, but the gateway adds geography as a first-class signal. Teams running managed NVIDIA Dynamo on HyperPod still benefit from pod-level schedulers; the global tier only decides which cluster should receive the request.
GPU cluster grid illustrating multi-cluster SageMaker HyperPod capacity
Setup path: publish, group, call
AWS breaks onboarding into two decisions: which clusters to publish for global routing, and which published clusters to place behind one gateway.
Publish a cluster (once per cluster): set spec.globalRouting in the cluster's InferenceGatewayConfig. The HyperPod Inference controller creates a Network Load Balancer, a VPC endpoint service, and uploads the listener certificate to your TLS bucket. Resources stay in your account.
Create the gateway (once per gateway): call CreateGlobalInferenceGateway with the cluster list. SageMaker returns one endpoint. Listing a cluster before it is published fails with an error that tells you what to fix.
AWS also documents three starting points:
| You are here | What to do | Result |
|---|---|---|
| Inference Gateway already on several clusters | Update the add-on, publish each cluster, create the gateway | One endpoint; per-cluster URLs keep working during migration |
| HyperPod Inference without the gateway | Enable the gateway per cluster, add schedulers, then publish | Collapse per-model URLs first |
| Two Regions, no inference yet | Stand up HyperPod Inference in each Region, enable gateways, publish | Cross-Region endpoint from day one |
Our HyperPod enhanced observability post covers how to find a bad GPU during training; for inference, combine that telemetry with gateway scoring so you know whether a cluster is unhealthy before the global tier marks it down.
Operating the gateway day to day
Add a cluster: prepare the cluster, extend the execution role's resource list, then AddInferenceGateway in the gateway's Region even when the new cluster lives elsewhere. Routing starts once the cluster is healthy; other clusters and clients stay up.
Detach a cluster: RemoveInferenceGateway stops new requests, drains in-flight work, and removes the private endpoint. A gateway must keep at least one cluster.
Delete the gateway: DeleteGlobalInferenceGateway stops the global endpoint while per-cluster endpoints continue serving. Withdraw each cluster with globalRouting.enabled: false before deleting roles.
| Event | Effect on traffic |
|---|---|
| Cluster fails | Traffic moves to next-best cluster; gradual return after recovery |
| Change weights | New requests use new weights immediately |
| Gateway creation fails | Gateway enters FAILED; ListInferenceGateways shows per-cluster status |
| No healthy cluster | HTTP 503 |
AWS documents JWT forwarding when every cluster in the gateway enforces JSON Web Tokens. Pass --customer-identity-token without a Bearer prefix; unsigned tokens get HTTP 403. Enable JWT checks on all clusters that serve a model or on none, so a request does not succeed in one Region and fail in another.
Security and what AWS does not log
Every request is signed with SigV4 and authorized with IAM. AWS states the gateway does not log or store prompts or completions. Connectivity to your clusters stays on PrivateLink.
The execution role is scoped: describe only listed clusters, reach only their Amazon EKS APIs, read only routing certificates under the gig/ prefix of your TLS bucket with a HyperPod Inference tag. That is tighter than handing a shared API key to a home-grown router.
For agent workloads that already use Amazon Bedrock AgentCore, the permission model should feel familiar: SigV4 on every hop, explicit IAM actions, no anonymous inference.
What does a named customer say?
AWS quotes Jinhyung Park, VP (AI Platform) at Bagelcode, on a beta that routed across two Regions through one endpoint and kept serving when one cluster could not host the model. That is the only customer paragraph in the launch post. Treat it as early evidence, not a benchmark.
Regions and how this compares to Bedrock routing
The gateway is available in thirteen AWS Regions listed in the post (Americas, Europe, and Asia Pacific). That is a subset of all AWS commercial Regions, so confirm your deployment Regions before you promise global failover.
SageMaker teams mixing HyperPod with Bedrock should keep the boundaries straight: Bedrock is a managed model API; HyperPod is your own weights on your clusters. The Global Inference Gateway is for the second case when you already pay for reserved HyperPod GPU capacity and want utilization across clusters.
Interconnect fabric showing logical links between inference Regions
Best practices AWS highlights
AWS's guidance is operational, not cosmetic:
- Plan gateway groups before publishing. Every cluster behind a gateway should expose the same
modelNameif they share traffic. - Do not publish unused clusters. A published cluster that is not in any gateway still bills its Network Load Balancer.
- Cover every model in schedulers. A model missing from
InferenceGatewayConfigreturns HTTP 404 through the gateway. - Use
--contexton kubectl. Multi-cluster edits go wrong quickly without an explicit kube context. - Keep latency-sensitive workloads in one Region by creating a gateway that lists only local clusters.
What should you do Monday?
- Inventory how many HyperPod cluster endpoints your clients track today.
- If you have two or more clusters, pilot publish plus one gateway in a non-production Region.
- Set routing weights to match your SLO: proximity-first for chat, error-rate-heavy for batch.
- Wire CloudWatch or your existing HyperPod observability dashboards so queue depth and KV headroom match what the gateway sees.
- Document IAM actions for application teams before you cut over DNS.
Related reading
- SageMaker inference 2026 launches, part 2
- Cross-Region inference routing for SageMaker AI
- What is MCP?
- NVIDIA Dynamo and KV routing context
- AWS Bedrock cost per task baselines
- Together Storage Accelerator for GPU clusters
Gateway behavior, IAM actions and Region list follow AWS's November 20, 2026 Machine Learning Blog post and may change in service documentation.
