NVIDIA Details AI Agents for DGX SuperPOD Cluster Management
NVIDIA shares insights on using the AI Factory Operations Agent and Model Context Protocol to streamline internal DGX SuperPOD operations and reduce SRE manual work.
Managing large GPU clusters involves handling complex layers spanning compute, networking, hardware, and job scheduling. Site reliability engineers typically must check multiple different dashboards when training performance drops to determine the underlying issue. To minimize manual troubleshooting and maintain cluster uptime, NVIDIA built the AI Factory Operations Agent to handle operations across its internal DGX SuperPOD infrastructure.
By leveraging specialized sub-agents that communicate via the Model Context Protocol, the system correlates metrics across various layers. Instead of forcing engineers to inspect software and hardware separately, the agents detect failures, trace root causes, and present recommendations through a single chat interface. Engineers can converse with the agents using natural language, jump between different investigations, and audit every automated action.
Architecture and Security Controls
The framework is built around the NVIDIA NemoClaw blueprint and uses locally served NVIDIA Nemotron open models for inference. At the core, a user-facing agent runtime receives queries from engineers and delegates tasks to specific domain agents and tools. Communication happens through OpenClaw plugins and MCP servers, where each server exposes a curated set of approved operational tools. Developers looking to experiment with powerful language models can easily try top AI models cheaply through one API at https://apixoai.online.
Running as a Kubernetes control-plane application, Helm handles the deployment of the user interface, gateways, tool servers, and the NVIDIA OpenShell runtime. For executing shell commands, OpenShell provisions isolated per-session Kubernetes sandboxes rather than running commands inside the main gateway pod. Security is enforced through multiple layers, including server-side tool execution with pinned binaries and strict role-based access control.
Specialized Domain Agents
Internal deployments utilize several domain-specific agents, each targeting a distinct operational area:
- Job Recovery Agent: Identifies training failures and performance drops, carrying out multi-layer root cause analysis across the scheduler, application, and hardware layers.
- Hardware Agent: Triages hardware issues like PCIe errors, thermal anomalies, and NVMe failures down to specific components.
- Observability Agent: Queries Prometheus telemetry and Grafana Loki logs to dynamically generate troubleshooting dashboards.
- Research Agent: Searches preloaded documentation, reference architectures, and custom site runbooks.
- Kubernetes and Cluster Management Agents: Provide operational awareness of cluster resources, node status, and administrative Base Command Manager tools.
What it means for developers
For engineering teams scaling their own infrastructure, the architecture demonstrates a modular pattern for combining open models with domain-specific tools via standardized protocols like MCP. The implementation highlights several key production takeaways: maintaining clear boundaries between agents to prevent conflicting signals, prioritizing read-only investigation phases to build trust before granting automation privileges, and enforcing multi-layer correlation to catch issues that isolated tools miss. Organizations can isolate control planes from production workloads to ensure that reasoning and orchestration tasks do not interfere with active training jobs.
Source: How NVIDIA Uses AI Agents to Streamline AI Factory Operations — NVIDIA Technical Blog. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
