NVIDIA Open-Sources NVIDIA Attribution to Diagnose Distributed AI Workload Failures
NVIDIA has open-sourced NVIDIA Attribution, a tool that correlates cluster logs and telemetry to identify root causes of distributed AI training failures.
When scaling distributed AI workloads across massive GPU clusters, identifying why a job failed can turn into a complex diagnostic challenge. A localized hardware fault on a single GPU frequently triggers a wave of secondary symptoms, such as NVIDIA Collective Communications Library (NCCL) timeouts across hundreds of dependent nodes. To address this issue, NVIDIA has open-sourced NVIDIA Attribution, a workload-centric system engineered to isolate the root cause of failures in large-scale AI infrastructure.
Having operated in production at NVIDIA for years, the framework processes millions of workloads weekly across hundreds of thousands of GPUs. In one deployment on a large RDMA-backed cluster, NVIDIA Attribution correlated connection resets, port errors, and scheduler events to pinpoint failure patterns linked to specific host and switch ports. This enabled engineering teams to target precise fabric repairs and congestion tuning rather than manually searching through unorganized log files.
How NVIDIA Attribution Pinpoints Root Causes
Standard monitoring tools report where a workload halted, but rarely explain where the failure originated. NVIDIA Attribution solves this by aggregating logs, telemetry, metrics, and scheduler states into a single structured diagnosis. The system runs through a four-stage process: scoping the specific workload attempt, collecting observations from cluster sources, extracting and correlating events through deterministic rules, and returning a structured result.
In a multi-rank training scenario, for example, a GPU might experience a hard fault such as an XID 149 NVLink signal integrity issue on one node, followed by cascading NCCL communication errors across other ranks. The system's rules engine evaluates the timing and interaction between these events, demoting downstream NCCL errors to secondary symptoms while elevating the NVLink signal integrity fault on the affected node as the primary root cause. If telemetry is incomplete, the diagnosis records the ambiguity directly rather than returning a false verdict.
Architecture and Customization
The platform consists of two main components: the NVIDIA Attribution Service and the NVIDIA Attribution Blueprint. The Attribution Service operates as a self-hosted HTTP REST API wrapper around the core rules engine. The Attribution Blueprint serves as a Python reference orchestrator that discovers workload contexts, collects telemetry, calls the service, and outputs structured records.
The Blueprint includes execution-context connectors for Slurm, Kueue, and static inputs, alongside telemetry connectors for Loki, Kubernetes pod logs, local files, and PromQL metrics. Output records can be written to SQLite or stdout and reviewed through a built-in read-only dashboard. Developers can customize the system by writing declarative TOML rules or Python extractors. A quickstart workflow is available on GitHub under the dsx-ai-factory/attribution repository, using Docker and the uv package manager to test rules locally against sample workload evidence.
What it means for developers
For AI infrastructure engineers and platform developers, NVIDIA Attribution converts messy log streams into machine-readable JSON outputs containing fields like terminal_state and sub_domain. Instead of writing custom log parsers or manually investigating cascading network timeouts, downstream automated systems—such as incident management, fleet monitoring, or automated job recovery tools—can directly consume these structured verdicts to trigger node repairs or reschedule jobs.
As engineering teams build complex training pipelines and multi-model applications, maintaining operational efficiency is critical. While tools like NVIDIA Attribution handle low-level infrastructure failures, developers building applications on top of modern LLMs can try top AI models cheaply through one API at https://apixoai.online.
By open-sourcing the tool, NVIDIA allows the broader ecosystem to encode custom failure patterns once and apply them across heterogeneous clusters. Key contributors to the project include NVIDIA engineers Aaron Blakeman, Ajay Hotchandani, Alex Shaposhnikov, Ashwin Poojary, and Pasha Shamis.
Source: Diagnose Distributed AI Workload Failures with NVIDIA Attribution — NVIDIA Technical Blog. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
