NVIDIA has shown how it uses AI agents to run its own GPU clusters. In a November 4, 2026 post on the NVIDIA Technical Blog, the company describes the AI Factory Operations Agent. It is a set of cooperating sub-agents that help site reliability engineers (SREs) find why a training job slowed down or failed in a DGX SuperPOD.
The pitch is simple. Today an SRE checks separate dashboards for compute, network, hardware and scheduling. NVIDIA's agents do that correlation behind one chat window. The orchestration framework is released as experimental open source.
TL;DR
| Question | Answer |
|---|---|
| What is it? | A multi-agent operations assistant for GPU clusters, used by NVIDIA SREs on DGX SuperPOD |
| Who wrote the post? | Kavya G Y and Rohan Adwankar of NVIDIA |
| What is the runtime? | The NVIDIA NemoClaw blueprint, with NVIDIA OpenShell for sandboxing |
| Which models? | Locally served NVIDIA Nemotron open models, fully on-premises capable |
| Where does it run internally? | DGX H100, DGX B200 and DGX GB300 production clusters |
| How do agents reach the cluster? | OpenClaw plugins and MCP servers, each exposing a small set of approved tools |
| Is it open source? | The orchestration framework is, but it is labelled experimental and may change |
| Is there accuracy data? | No. NVIDIA says it measures RCA accuracy and SRE agreement but publishes no figures |
The problem: one slowdown, many dashboards
Large GPU clusters fail in layers. A throughput drop can come from a flaky PCIe link, a thermal throttle, a bad NIC, a scheduler placement, or a software bug in the job. Each layer has its own tool. NVIDIA's argument is that the root cause often sits between those tools, where no single dashboard shows it.
This is the same gap NVIDIA has attacked from other directions in the last two weeks. Attribution uses deterministic rules to name the cause of a failed workload. Minerva builds an observability knowledge graph to catch silent slowdowns. The Operations Agent is the conversational layer: a place where an engineer asks a question in plain language and gets evidence from several systems.
The architecture: NemoClaw at the centre
NVIDIA says the developer example is built around the NemoClaw blueprint. At the centre is a user-facing agent runtime that receives SRE queries through a chat interface and decides which domain agent and tools to call. You can use the built-in web UI or plug in an existing coding-agent interface through the API.
Agents connect to infrastructure through OpenClaw plugins and MCP servers. Each MCP server exposes "a focused set of approved operational tools". Because of that design, a team can add a new agent or a new integration without changing the agents already in place.
Deployment is a Kubernetes control-plane application. Helm installs the UI, the OpenClaw gateway, the MCP tool servers, the inference endpoint and the OpenShell runtime as services. When an agent needs to run shell commands, OpenShell creates a per-session Kubernetes sandbox rather than running them in the main gateway pod. The sandbox receives the agent workspace, selected skills and a command policy, and NVIDIA says its lifecycle does not modify existing services or production workloads.
If you want background on the sandbox layer, see our coverage of NVIDIA OpenShell and what developers said about it on Hacker News, and of NVIDIA Sentry for hardware-level agent monitoring.
The six domain agents
| Agent | What NVIDIA says it does |
|---|---|
| Job Recovery | Detects training job failures and goodput loss; does multi-layer root cause analysis across scheduler, application, hardware and network; recommends recovery steps once a cause is confirmed |
| Hardware | Triages device-level faults: PCIe errors, ECC errors, GPU issues, NVMe failures and thermal anomalies, down to the component |
| Observability | Queries Prometheus history, telemetry and Grafana Loki logs, and builds dashboards on demand in the web UI |
| Research | Searches preloaded documentation, including NVIDIA Mission Control manuals and DGX SuperPOD reference architectures; you can upload your own runbooks |
| Kubernetes | Gives read-only awareness of pods, services, deployments, GPU resources, node status and scheduling state |
| Cluster Management | Calls NVIDIA Base Command Manager APIs through diagnostic MCP tools for advisory management and troubleshooting |
On top of these sit shared MCP tools for web search, code execution and Grafana dashboards. NVIDIA says you can also attach your own MCP servers for the schedulers, observability stacks and cluster managers you already run.
The Job Recovery agent is the one most likely to matter to ML teams. It targets the question every training run eventually asks: why did throughput fall, and is it safe to restart?
Security by layers, and a read-only start
The most useful part of the post for builders is the security model. NVIDIA enforces limits in several layers instead of trusting the model to behave.
- Read-only credentials. Kubernetes inspection runs through a server-side run_kubectl tool that calls a pinned kubectl binary against registered read-only credentials.
- No credentials in the sandbox. OpenShell gets neither kubectl nor Kubernetes credentials.
- RBAC as the ceiling. Kubernetes service accounts and role-based access control define what the tool can inspect. The post prints a check showing the agent can get pods and configmaps, but cannot get secrets, create pods or update pods.
- Kernel-level isolation for tenants. In multi-tenant clusters, scoped agents, tools, sandboxes and permissions limit what each tenant sees. Tenant SREs only see their own namespaces, jobs and nodes, while central operations keep wider visibility. NVIDIA says OpenShell enforces this by running each tenant's agents in isolated sandboxes with kernel-level restrictions.
The same checks run through the access path the runtime uses, so the output shows what the agent can really do in the cluster, not what a prompt promises. For a deeper look at Kubernetes permissions, the upstream RBAC documentation is the reference.
Five lessons NVIDIA reports from production
NVIDIA lists five lessons from running the system.
- Clear agent boundaries. Overlapping ownership between agents creates noise rather than signal.
- Read-only first. Accurate investigation builds SRE trust before any automation is allowed to act.
- Cross-layer correlation. Tying workload, hardware and network signals together finds root causes that isolated tools miss.
- Precise MCP tools. Clear tool definitions improve routing and reduce wrong tool selection.
- Transparent reasoning. SREs need to see tool calls, evidence and conclusions before they act.
Lesson four is worth copying in any MCP project. A vague tool description is a routing bug. Short, specific tool names and descriptions with narrow scope are cheaper to fix than a larger model.
What NVIDIA does not say
This is a vendor engineering post, so a reader should note the gaps.
- No accuracy numbers. NVIDIA says it measures root-cause analysis accuracy and tracks SRE agreement, but gives neither. There is no false-positive rate, no time-to-diagnosis figure and no comparison with the SRE workflow before agents.
- One demo. The post links a video of an incident injected into a DGX cluster for internal demonstration. That shows the flow, not field performance.
- Which Nemotron model. The post says "locally served NVIDIA Nemotron open models" without naming sizes or versions in the text we read.
- Experimental status. The orchestration framework may change between releases. NVIDIA customers can request access to NVIDIA-developed cluster operations agents through their account manager, which suggests the strongest agents are not all in the open repository.
None of this means the system is weak. It means the evidence for the headline benefit, less manual effort, is NVIDIA's word so far.
How to try it
NVIDIA points to the AI Factory Operations Agent developer example and an open-source repository on GitHub. Based on the post, a sensible evaluation path is:
- Start in a lab cluster, not production. Deploy the Helm chart into a control-plane namespace on a Kubernetes cluster you can break.
- Keep credentials read-only. Mirror NVIDIA's pattern: a service account with get and list on pods and configmaps, no secrets, no write verbs. Run the auth checks yourself.
- Serve a local model. Use a Nemotron model on a dedicated GPU, or on an RTX PRO Workstation or DGX Spark for small setups, as NVIDIA suggests.
- Load your own runbooks into the Research agent so answers cite your procedures.
- Add one MCP server at a time for your scheduler or monitoring stack, and test routing on a list of real past incidents before trusting it on live ones.
- Isolate the control plane. NVIDIA's preferred model is a dedicated GPU-accelerated control plane separate from the production cluster, so the agent does not consume training GPUs or disrupt workloads.
Where it sits among NVIDIA's agent releases
NVIDIA has published a steady run of agent blueprints on the NemoClaw and OpenShell stack. Fox, the factory manager agent blueprint, applies the same idea to Siemens manufacturing lines. NeMo Compass turns agent failures into tests and harness fixes. The pattern is consistent: a domain-specific agent set, MCP tools with tight scope, local open models and a sandbox layer. The Operations Agent is the data-center instance of that template.
For anyone building internal agents, this is the useful takeaway. Most of NVIDIA's lessons are not about the model. They are about boundaries, permissions, tool design and showing evidence.
Why it matters
AI factories are getting large enough that human triage does not scale. If agents can narrow a cause from dozens of signals to a short, evidence-backed list, SREs spend more time on fixes. If they get it wrong with write access, the damage lands on expensive training runs. NVIDIA's decision to start read-only and enforce limits outside the model is the right pattern whatever you think of the agent's accuracy.
The open question is evidence. Until NVIDIA or independent teams publish accuracy and time-saved numbers, treat the system as a well-designed reference architecture rather than a proven productivity gain.
Details are accurate as of November 4, 2026, from NVIDIA's technical blog post. The framework is experimental and may change.