How Developers Use Frontier AI Agents to Build NVIDIA Omniverse Simulations

Similarly, Tae Kim, who leads NVIDIA Omniverse engineering and product, directed Astra to construct "Robo Olympics." This experimental project used sports videos and natural-language prompts to program a simulated Unitree G1 humanoid robot. Powered by the Newton Physics Engine, the open-source NVIDIA Warp framework, and ovrtx rendering, the robot successfully cleared a physical hurdle in 64 out of 100 simulation trials, providing immediate feedback to fine-tune its control policies.

For industrial applications, Jens Jebens, a senior product manager for OpenUSD, used Astra to import a car suspension model from PTC Onshape into NVIDIA Isaac Sim. The AI agent assessed the physical space and designed a custom wrench tool capable of reaching the suspension bolts, proving the disassembly process could succeed in simulation. Additionally, Chirag Majithia from the Isaac engineering applications team turned stereo camera captures into editable OpenUSD environments. Majithia's workflow integrated PyCuSFM, FoundationStereo, and nvblox, using USD Content Agents to define how reconstructed objects like doors and drawers physically interact.

Driving Simulation and Digital Twins

Autonomous vehicle development also benefits heavily from agent-led simulation. Doyub Kim, a simulation technology manager, directed Astra to build "Zero to Alpamayo," a virtual recreation of San Francisco's Market Street. The agent mapped out the workflow, integrating asset creation, simulated traffic, and RTX sensor feedback. By using Cosmos3-Nano to alter environmental variables like weather and lighting, Kim compared how different driving models reacted to identical road scenarios under varying conditions.

Validating these virtual environments requires checking how well simulated sensors match physical ones. Ashley Reid, an RTX sensor validation specialist, guided Astra and Claude Fable 5 agents through a three-day iterative process to compare ovrtx camera and LiDAR outputs with real-world datasets. The AI agents constructed two new digital twins and optimized two existing ones by measuring discrepancies in geometry, materials, and missing objects, using sensor metrics as the standard for acceptance.

Beyond earthbound applications, Nic Johns, an engineering director, prompted Astra to assemble NASA assets into an interactive, browser-based OpenUSD model of the International Space Station. With a single prompt, Johns established the telemetry-connected station, and used a subsequent prompt to rotate the scene to Earth's daytime side, utilizing ovrtx, ovstage, and ovstream for real-time web rendering.

What it means for developers

This shift toward agent-guided simulation represents a fundamental change in how software and physical systems are designed. Instead of spending days manually configuring scene hierarchies, writing boilerplate integration code, or troubleshooting rendering pipelines, developers can act as high-level directors. AI agents handle the low-level heavy lifting of connecting various GPU-accelerated libraries, while human developers focus on iterative refinement and validation.

To build these agentic workflows successfully, developers often need to experiment with multiple LLMs to see which model handles specific spatial reasoning, code generation, or translation tasks best. Developers can try top AI models cheaply through one API at https://apixoai.online, simplifying the process of testing and deploying different frontier models without managing multiple platform accounts. As AI agents continue to mature, their integration with frameworks like NVIDIA Omniverse will make high-fidelity physical simulations accessible to a broader range of engineers and researchers.

Source: Into the Omniverse: How Developers Turn Ideas Into Simulations With Frontier AI Agents — NVIDIA blog. Written by the Apixo team from that report.