NVIDIA Standardizes Kubernetes Infrastructure for Dynamo Inference via AICR Recipes
NVIDIA AI Cluster Runtime introduces validated recipes to automate Kubernetes infrastructure, GPU allocation, and routing for NVIDIA Dynamo inference endpoints.
Deploying distributed AI model inference on Kubernetes often presents complex setup challenges. While orchestrators can handle container lifecycles, ensuring proper alignment between GPU drivers, scheduling constraints, dynamic resource allocations, and telemetry stacks remains a major bottleneck. To streamline this process, NVIDIA has detailed the integration of NVIDIA Dynamo within the NVIDIA AI Cluster Runtime (AICR), a component of the NVIDIA DSX AI Factory Platform designed to deliver pre-tested Kubernetes infrastructure blueprints for high-performance workloads.
Navigating underlying infrastructure for distributed inference
Running a distributed model architecture through Dynamo begins with defining a DynamoGraphDeployment Custom Resource Definition (CRD), which now uses the v1beta1 API specification. However, sustaining production-grade inference requires a fully compatible foundation beneath the application tier. Clusters must properly expose hardware using mechanisms like the Dynamic Resource Allocation (DRA) driver, manage pod placement semantics using specialized schedulers, and route metrics to Prometheus.
To remove manual dependency resolution, AICR packages Dynamo 1.4.2 alongside pre-validated infrastructure tiers. A sample recipe tailored for Amazon EKS environments running NVIDIA GB200 and GB300 NVL72 hardware locks in compatible versions across the entire stack:
- GPU Management: NVIDIA GPU Operator v26.7.1, Node Feature Discovery 0.19.0, and NVIDIA DRA GPU Driver 0.5.0.
- Scheduling & Health: KAI Scheduler v0.16.9, Grove v0.1.0-alpha.13, NVSentinel v1.25.0, and NodeWright Operator v0.19.0.
- Networking & Ingress: Agent Gateway v1.5.0 and associated CRDs.
- Observability: Kube Prometheus Stack 84.4.0 and Prometheus Adapter 5.3.0.
Notably, Dynamo 1.4 transitions away from requiring NATS by default for service discovery, relying instead on native Kubernetes resources, though NATS remains available as an opt-in component.
Evaluating request routing and KV-cache architectures
Agentic AI workloads heavily depend on maintaining key-value (KV) cache context across multi-turn interactions. When requests hit workers missing the required system prompts or conversation history in GPU memory, latency spikes due to prefix reprocessing. Dynamo addresses this by decoupling the request control plane from the KV-cache event plane.
In the default AICR configuration for Dynamo 1.4.2, requests move across a TCP-based data plane, while worker nodes broadcast local cache states over a ZMQ-backed event plane. Worker engines running vLLM publish events on designated endpoints (such as tcp://*:5557), allowing routers to make token-aware dispatching decisions. Operators can choose between two distinct routing architectures:
- Dynamo Frontend Router: Running with
DYN_ROUTER_MODE=kv, the frontend tokenizes incoming traffic, scores worker nodes based on prefix cache overlap and active load, and forwards requests directly over TCP. This provides a lightweight, self-contained deployment model. - Gateway API & Endpoint Picker (EPP): Integrates Kubernetes ingress proxies like
agentgatewaywith the Gateway API Inference Extension (GAIE). The gateway queries the Dynamo EPP to select an optimal pod from anInferencePool, after which traffic reaches a worker-side Frontend sidecar executing indirectrouter mode.
Looking ahead to Dynamo 1.5, NVIDIA plans to replace the Go/CGO bridge in the EPP path with a native Rust implementation using the ext_proc interface, reducing per-request processing overhead while retaining full KV-aware routing.
What it means for developers
For platform engineering teams, automated cluster runtime validation eliminates the friction of manually verifying node health, GPU availability, and driver alignment before launching LLM services. Instead of spending cycles debugging lower-level container runtime errors or gang scheduling failures, engineers receive a verified foundation that guarantees uniform behavior across environments.
This standardized approach allows developer teams to focus their efforts entirely on prompt engineering, model tuning, disaggregated serving optimization (separating prefill and decode tasks), and application logic. For developers looking to experiment with and evaluate top AI models cheaply through one API without managing backend Kubernetes clusters, platforms like https://apixoai.online provide streamlined access.
The AICR deployment and validation workflow
The AICR deployment model follows a structured multi-step lifecycle to guarantee cluster readiness before any model serving workloads are provisioned:
- Cluster Inspection: The CLI captures the current cluster configuration using
aicr snapshot, targeting nodes marked withnvidia.com/gpu.present=true. - Recipe Generation: Running
aicr recipeyields a declarative blueprint detailing exact Helm chart versions, node selectors, tolerations, and validation checks. - Bundle Rendering & Application: The
aicr bundlecommand converts the recipe into manifests compatible with Helmfile, ArgoCD, or Flux. - Live Validation: Before submitting a
DynamoGraphDeployment, operators executeaicr validate --phase deployment --phase conformance. This runs automated cluster jobs to test GPU allocation, driver responsiveness, Prometheus metric collection, and ingress routing.
Only after these verification checks pass is the infrastructure cleared for production application traffic.
Source: Build a Validated Kubernetes Runtime for NVIDIA Dynamo — NVIDIA Technical Blog. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
