NVIDIA Omniverse is empowering developers to dramatically accelerate the creation of complex simulations by integrating advanced AI models and its powerful libraries. These tools allow for the rapid transformation of conceptual ideas into functional applications, ideal for scenario exploration, failure analysis, and design refinement.
Developers can now interact with AI agents using natural language commands, review simulation outcomes, and implement adjustments on the fly. NVIDIA Omniverse libraries provide GPU-accelerated physics, rendering, and sensor simulation capabilities, further streamlining the development process.
AI Agents in Action: Simulating Real-World Scenarios
Humanoid Robot Simulator for Warehouse Environments
Automating warehouse operations demands interactive simulation environments. Frank DeLise, Omniverse Product Manager at NVIDIA, leveraged the Astra model to build an interactive warehouse simulation featuring a humanoid robot. This simulator allows for first-person and third-person exploration of the robot’s behavior, controlled through physics-based interactions.
DeLise instructed Astra to integrate NVIDIA Omniverse libraries for physics (ovphysx), scene updates (ovstage), rendering (ovrtx), and user interface (ovui). Astra then generated the animation and application code, seamlessly combining these functionalities.
Integrating Autonomous Driving into Test Workflows
Changes to a scene, sensors, or driving models can significantly impact the results of autonomous vehicle simulations. Doob Kim, Simulation Technology Manager at NVIDIA, tasked Astra with creating a reusable simulation environment called “Zero to Alpamayo,” based on San Francisco’s Market Street.
Kim guided the process, integrating asset creation, traffic simulation, Omniverse RTX sensor simulation, and the Alpamayo driving model. The resulting prototype serves as a testbed for comparing different models and tracking the effects of scene or sensor modifications on driving behavior. A separate Cosmos3-Nano experiment altered weather and lighting conditions in recorded simulation videos, enabling a comparison of driving model responses to identical scenarios under varying environmental factors.
Leveraging Sensor Discrepancies for Digital Twin Creation and Refinement
A critical aspect of testing robots and autonomous vehicles is ensuring simulated sensors accurately reflect real-world performance. Ashley Reed from NVIDIA’s RTX Sensor Validation team utilized Astra and Claude Fable 5 agents to compare output from ovrtx cameras and LiDAR against recorded data.
The agents were instrumental in creating two digital twins from scratch and enhancing two existing ones. Over three days, Reed managed an iterative process where the agents measured discrepancies, generated or modified OpenUSD scenes, and validated the results. Adjustments focused on missing objects, geometry, and materials, with acceptance criteria based on camera and LiDAR metrics.
Robo Olympics: Testing Robot Skills Through Simulation
Training robots for new movements requires verifying their functional capabilities within physical constraints. Tae Kim, Head of Engineering and Product at NVIDIA Omniverse, used sports videos and natural language instructions to guide Astra in creating the “Robo Olympics” project. This experimental endeavor tests simulated Unitree G1 humanoid robots performing athletic maneuvers.
Under Kim’s direction, Astra developed and refined controllers through physical trials. The Newton Physics Engine modeled robot behavior, the NVIDIA Warp framework accelerated computations, and ovrtx rendered scenes and virtual camera imagery. In one test, a robot successfully navigated an obstacle in 64 out of 100 simulation attempts, providing Kim with feedback to improve the robot’s timing and control.
Testing Robotic Disassembly with CAD and Simulation
Before a robot can disassemble a product, developers must ensure its tools can access and remove components. Jens Jebens, Senior Product Manager for OpenUSD at NVIDIA, tasked Astra with modeling a car’s suspension in PTC Onshape and configuring it within NVIDIA Isaac Sim.
With Astra’s assistance, Jebens explored CAD and robotic tooling solutions informed by simulation. The agent measured accessible space and designed a wrench that enabled the robot to reach suspension bolts. Jebens reported successful removal of a suspension component in simulation, directly linking design and tooling decisions to disassembly outcomes.
The International Space Station on the Web
Transforming 3D models into functional applications requires integrating assets, real-time data, and a user interface. Nick Jones, Head of Engineering at NVIDIA, requested Astra to assemble NASA assets into an OpenUSD model of the International Space Station (ISS) with telemetry data.
Jones created the application with a single prompt and then used a subsequent prompt to reposition the scene to Earth’s daytime side, making the planet visible. The workflow utilized Blender for asset preparation and Omniverse libraries for rendering (ovrtx), scene execution (ovstage), and streaming (ovstream). The application delivers 3D models and operational data within a web browser, with Jones guiding its development through prompts and corrections.
Converting Captured Rooms into Test Environments
Digitally reconstructed rooms require editable objects and accurate physical behavior before developers can test interactions. Chirag Majithia from NVIDIA’s Isaac Engineering Applications team directed Astra to convert stereo camera captures into an editable OpenUSD environment.
The workflow combined PyCuSFM, FoundationStereo, and nvblox for reconstruction, with user review guiding object selection and placement. Astra compiled generated assets and those created in Blender, using USD Content Agents to configure object interactions within the simulation. Isaac Sim tests facilitated the review of collisions and contacts for doors and drawers. This approach merges captured geometry with interaction testing, simplifying the inspection of object gaps and behaviors.
Have a simulation idea? Explore NVIDIA Omniverse libraries to start building with an AI agent.
Source: NVIDIA
