explainx.ai350K+ learn from us
TrendingAI News TodayExplainx for Business
Pricing
explainx.ai

Upskill in AI — 16 free pathways, live workshops & bootcamps, and 50+ courses from practitioners. Plus the skills, tools, and MCP servers to practice on.

follow us

follow on google

Add explainx.ai as a preferred source

corporate training

support@explainx.ai

get started

Find your pathTake Free Evaluation

community

Join the community

learn

mind: share how you thinkpathways — start freeworkshopsbootcampscoursescompare Explainxcertificationsmock testsexplainx universitycorporate traininglearn skills & mcp

discover

skillsmcp serversexplainx mcpestacktoolsmdx readeragentsllmsdesignsdictionarypeopleagi trackerfelony benchranks

company

aboutvisionmissionteaminstructorsteach on explainxpartnershipscommunityhackathonscareers

content

daily AI newsstate of AI — live resultsblogreleasespromptsgeneratorsresource libraryfor LLMsexplainx.ai kids

solutions

all solutionsdeveloper upskillingmarketing upskillingproduct manager upskillingleadership upskilling

newsletter · weekly

Get AI news, tools, and insights in your inbox.

supportcontactprivacytermsdata rightshow we create contentsubmission guidelines

© 2026 AISOLO Technologies Pvt Ltd

explainx.ai

On this page

  • TL;DR
  • How does the Global Inference Gateway fit HyperPod?
  • What happens on each request?
  • How does cluster scoring work?
  • Setup path: publish, group, call
  • Operating the gateway day to day
  • Security and what AWS does not log
  • What does a named customer say?
  • Regions and how this compares to Bedrock routing
  • Best practices AWS highlights
  • What should you do Monday?
  • Related reading
← Back to blog

explainx / blog

SageMaker HyperPod Global Inference Gateway: One Endpoint Across Regions and Clusters

Amazon SageMaker, AWS, HyperPod, LLM Inference, MLOps

Part of Microsoft, Apple and Amazon AI

AWS launched the SageMaker HyperPod Global Inference Gateway on Nov 20, 2026: one SigV4 endpoint, PrivateLink routing, cluster scoring, and automatic failover across Regions.

Nov 20, 2026·8 min read·Yash Thakker
add explainx.ai
go deep
SageMaker HyperPod Global Inference Gateway: One Endpoint Across Regions and Clusters

Amazon SageMaker HyperPod now has a Global Inference Gateway: one managed endpoint that routes requests across every HyperPod cluster you register, in one AWS Region or several. AWS published the launch on November 20, 2026. The gateway authenticates each call with SigV4 and IAM, scores clusters on live load, and carries traffic over AWS PrivateLink. Inside each cluster, the existing HyperPod Inference Gateway still picks the GPU pod, unchanged from the architecture AWS described earlier this year.

If you already run HyperPod inference in more than one cluster, this is the piece that removes custom cross-Region DNS, load balancers, and manual failover runbooks.

TL;DR

table · 2 cols
QuestionAnswer
What shipped?Global Inference Gateway for SageMaker HyperPod
When?November 20, 2026, per AWS Machine Learning Blog
What problem does it solve?Idle GPUs in another cluster or Region while requests queue on a saturated cluster
Client surfaceOne regional gateway endpoint; OpenAI-compatible requests signed with SigV4
Cluster tierUnchanged per-cluster Inference Gateway picks the pod
Default routing weightsKV headroom 0.20, queue depth 0.25, Region proximity 0.40, error rate 0.15
RegionsThirteen AWS Regions listed in the launch post
SecurityIAM plus PrivateLink; AWS says prompts and completions are not logged
Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

How does the Global Inference Gateway fit HyperPod?

HyperPod teams spread models across multiple clusters and Regions when a single rack is not enough. Before this gateway, AWS's own example counted twenty client endpoints for four models on five clusters in two Regions (one endpoint per model per cluster). The per-cluster HyperPod Inference Gateway collapses that to five cluster endpoints, but each application still chooses a cluster.

The Global Inference Gateway is the second routing tier. AWS positions it immediately after the September 2026 per-cluster gateway in the same launch narrative. Our part 2 review of SageMaker inference launches already indexed the per-cluster gateway as launch 23; this post is the cross-cluster extension AWS teased in that architecture story.

Do not confuse it with SageMaker cross-Region inference routing for ordinary SageMaker AI endpoints. That router, which became generally available November 19, 2026, targets standard endpoints and inference components. The Global Inference Gateway only applies to HyperPod clusters where you enable spec.globalRouting in InferenceGatewayConfig.

What happens on each request?

AWS describes four steps in its architecture figure (see the launch post figure):

  1. The client signs the request with SigV4 and calls the gateway's regional endpoint.
  2. The gateway authenticates the caller and scores every healthy registered cluster.
  3. AWS PrivateLink carries the request to the chosen cluster, same Region or not.
  4. The cluster's Inference Gateway routes to a model pod the way it does today.

Each cluster publishes load back to the gateway through a service-managed metrics publisher with its own IAM role. AWS operates the routing fleet that terminates TLS, enforces rate limits, and health-checks clusters.

How does cluster scoring work?

For every request the gateway combines four signals. Default weights, from the launch post:

table · 3 cols
SignalDefault weightWhat it measures
KVHeadroom0.20Available KV cache and GPU memory
QueueDepth0.25Backlog on the cluster
RegionProximity0.40Distance between cluster and gateway Region
ErrorRate0.15Recent failed requests

Defaults favor nearby Regions but still shift load toward clusters with spare capacity and away from failing ones. If resilience matters more than latency, AWS's example raises ErrorRate to 0.7 and lowers the others:

json
{"Weights": {"KVHeadroom": 0.1, "QueueDepth": 0.1, "RegionProximity": 0.1, "ErrorRate": 0.7}}

Pass weights in CreateGlobalInferenceGateway and change them later with UpdateGlobalInferenceGateway. In-flight requests finish under the old weights while new requests pick up updates.

This design rhymes with KV-cache-aware routing on single-cluster SageMaker endpoints, but the gateway adds geography as a first-class signal. Teams running managed NVIDIA Dynamo on HyperPod still benefit from pod-level schedulers; the global tier only decides which cluster should receive the request.

GPU cluster grid illustrating multi-cluster SageMaker HyperPod capacityGPU cluster grid illustrating multi-cluster SageMaker HyperPod capacity

Setup path: publish, group, call

AWS breaks onboarding into two decisions: which clusters to publish for global routing, and which published clusters to place behind one gateway.

Publish a cluster (once per cluster): set spec.globalRouting in the cluster's InferenceGatewayConfig. The HyperPod Inference controller creates a Network Load Balancer, a VPC endpoint service, and uploads the listener certificate to your TLS bucket. Resources stay in your account.

Create the gateway (once per gateway): call CreateGlobalInferenceGateway with the cluster list. SageMaker returns one endpoint. Listing a cluster before it is published fails with an error that tells you what to fix.

AWS also documents three starting points:

table · 3 cols
You are hereWhat to doResult
Inference Gateway already on several clustersUpdate the add-on, publish each cluster, create the gatewayOne endpoint; per-cluster URLs keep working during migration
HyperPod Inference without the gatewayEnable the gateway per cluster, add schedulers, then publishCollapse per-model URLs first
Two Regions, no inference yetStand up HyperPod Inference in each Region, enable gateways, publishCross-Region endpoint from day one

Our HyperPod enhanced observability post covers how to find a bad GPU during training; for inference, combine that telemetry with gateway scoring so you know whether a cluster is unhealthy before the global tier marks it down.

Operating the gateway day to day

Add a cluster: prepare the cluster, extend the execution role's resource list, then AddInferenceGateway in the gateway's Region even when the new cluster lives elsewhere. Routing starts once the cluster is healthy; other clusters and clients stay up.

Detach a cluster: RemoveInferenceGateway stops new requests, drains in-flight work, and removes the private endpoint. A gateway must keep at least one cluster.

Delete the gateway: DeleteGlobalInferenceGateway stops the global endpoint while per-cluster endpoints continue serving. Withdraw each cluster with globalRouting.enabled: false before deleting roles.

table · 2 cols
EventEffect on traffic
Cluster failsTraffic moves to next-best cluster; gradual return after recovery
Change weightsNew requests use new weights immediately
Gateway creation failsGateway enters FAILED; ListInferenceGateways shows per-cluster status
No healthy clusterHTTP 503

AWS documents JWT forwarding when every cluster in the gateway enforces JSON Web Tokens. Pass --customer-identity-token without a Bearer prefix; unsigned tokens get HTTP 403. Enable JWT checks on all clusters that serve a model or on none, so a request does not succeed in one Region and fail in another.

Security and what AWS does not log

Every request is signed with SigV4 and authorized with IAM. AWS states the gateway does not log or store prompts or completions. Connectivity to your clusters stays on PrivateLink.

The execution role is scoped: describe only listed clusters, reach only their Amazon EKS APIs, read only routing certificates under the gig/ prefix of your TLS bucket with a HyperPod Inference tag. That is tighter than handing a shared API key to a home-grown router.

For agent workloads that already use Amazon Bedrock AgentCore, the permission model should feel familiar: SigV4 on every hop, explicit IAM actions, no anonymous inference.

What does a named customer say?

AWS quotes Jinhyung Park, VP (AI Platform) at Bagelcode, on a beta that routed across two Regions through one endpoint and kept serving when one cluster could not host the model. That is the only customer paragraph in the launch post. Treat it as early evidence, not a benchmark.

Regions and how this compares to Bedrock routing

The gateway is available in thirteen AWS Regions listed in the post (Americas, Europe, and Asia Pacific). That is a subset of all AWS commercial Regions, so confirm your deployment Regions before you promise global failover.

SageMaker teams mixing HyperPod with Bedrock should keep the boundaries straight: Bedrock is a managed model API; HyperPod is your own weights on your clusters. The Global Inference Gateway is for the second case when you already pay for reserved HyperPod GPU capacity and want utilization across clusters.

Interconnect fabric showing logical links between inference RegionsInterconnect fabric showing logical links between inference Regions

Best practices AWS highlights

AWS's guidance is operational, not cosmetic:

  • Plan gateway groups before publishing. Every cluster behind a gateway should expose the same modelName if they share traffic.
  • Do not publish unused clusters. A published cluster that is not in any gateway still bills its Network Load Balancer.
  • Cover every model in schedulers. A model missing from InferenceGatewayConfig returns HTTP 404 through the gateway.
  • Use --context on kubectl. Multi-cluster edits go wrong quickly without an explicit kube context.
  • Keep latency-sensitive workloads in one Region by creating a gateway that lists only local clusters.

What should you do Monday?

  1. Inventory how many HyperPod cluster endpoints your clients track today.
  2. If you have two or more clusters, pilot publish plus one gateway in a non-production Region.
  3. Set routing weights to match your SLO: proximity-first for chat, error-rate-heavy for batch.
  4. Wire CloudWatch or your existing HyperPod observability dashboards so queue depth and KV headroom match what the gateway sees.
  5. Document IAM actions for application teams before you cut over DNS.

Related reading

  • SageMaker inference 2026 launches, part 2
  • Cross-Region inference routing for SageMaker AI
  • What is MCP?
  • NVIDIA Dynamo and KV routing context
  • AWS Bedrock cost per task baselines
  • Together Storage Accelerator for GPU clusters

Gateway behavior, IAM actions and Region list follow AWS's November 20, 2026 Machine Learning Blog post and may change in service documentation.

Spotted something out of date? Let us know.
Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

  • ✓ Free
  • ✓ One email a day, no spam
  • ✓ Unsubscribe anytime
Yash Thakker

Written by

Yash Thakker

Yash is an AI expert with over 300K learners. Join his workshops →

View Yash Thakker in People in AI →

Related posts

Nov 20, 2026

Amazon SageMaker Inference 2026 Launches, Part 2: 13 Features From MIG to In-Place Updates

AWS published part 2 of its 2026 SageMaker inference review on November 19. It adds 13 launches in five stages: choose a configuration, fit more models per GPU, serve very large models, route across Regions, and keep endpoints tuned. Here is what each does and which ones matter for your bill.

Oct 27, 2026

SageMaker SOCI-Indexed Images: Training Containers Start in Seconds, Not Minutes

On October 27, 2026 AWS announced Seekable OCI (SOCI) indexing for SageMaker AI training and processing jobs. It lazy-loads container images so jobs start in seconds. We cover the numbers, the two ways to adopt it, and when it will not help.

Oct 26, 2026

One Model per Camera: AWS Shows a SageMaker SDK v3 Pipeline for Edge Vision Fleets

A fleet of factory cameras rarely shares one model. An AWS Machine Learning blog post from October 26, 2026 shows a SageMaker SDK v3 reference pipeline that trains an anomaly detector and a classifier per camera, gates them on metrics suited to imbalanced defect data, compiles for the edge with Neo, and paces thousands of retrains.