Skip to content

SIMULATION_INTEGRATION.md

Latest commit

 

History

History
87 lines (63 loc) · 3.4 KB

SIMULATION_INTEGRATION.md

File metadata and controls

87 lines (63 loc) · 3.4 KB

🤖 Simulation Integration Guide

This guide details how to take the "Digital Twin" environments generated by World Labs and import them into physics-based simulators: NVIDIA Isaac Sim and MuJoCo.

🟢 NVIDIA Isaac Sim Integration

NVIDIA Isaac Sim is built on the Universal Scene Description (USD) standard, which makes it the most direct target for World Labs exports (which are natively USD-compatible).

Prerequisites

Step-by-Step

  1. Download the Asset:

    • Go to the World Labs URL generated by the script.
    • Click "Download" and select USD format.

  2. Launch Isaac Sim:

    • Open the Omniverse Launcher -> Library -> Isaac Sim -> Launch.

  3. Import the World:

    • In the content browser, navigate to your downloaded USD file.
    • Drag and drop the file into the viewport or use File > Open.
    • Tip: If the units look wrong, check the "Layer Scale" in the stage properties. World Labs defaults to meters (1.0).

  4. Add Physics (Collision Check):

    • By default, the mesh is visual. To make it collidable:
      • Select the root mesh in the Stage / World panel.
      • Go to Property tab -> Add -> Physics -> Collider Preset.
      • Choose Triangle Mesh Approximation for complex terrain (like the construction site) or Convex Decomposition for faster performance on simpler objects.

  5. Add a Robot:

    • Go to Create -> Isaac -> Robots -> Franka Panda (or your desired robot).
    • It will spawn at (0,0,0). Use the Gizmo to move it to a valid starting position (e.g., on top of the kitchen floor).

🔴 MuJoCo Integration

MuJoCo (Multi-Joint dynamics with Contact) uses the MJCF (XML) format. Integrating mesh-based worlds requires converting the visual assets into a format MuJoCo can reference (OBJ/STL/GLB) and wrapping them in an MJCF file.

Prerequisites

  • Python 3.10+
  • mujoco library (pip install mujoco)
  • A generated World Labs environment (downloaded .glb or .obj file).

Step-by-Step

  1. Download the Asset:

    • Go to the World Labs URL.
    • Click "Download" and select GLB or OBJ.

  2. Prepare the Mesh:

    • Place the downloaded kitchen.glb (example) in your project folder, e.g., assets/kitchen.glb.

  3. Create the Scene MJCF (scene.xml): You need to define a worldbody that includes your mesh as a geom.

    <mujoco model="world_labs_kitchen">
  <asset>
    <!-- Load the mesh file -->
    <mesh name="kitchen_mesh" file="assets/kitchen.glb" scale="1 1 1"/>
  </asset>

  <worldbody>
    <light pos="0 0 3" dir="0 0 -1" directional="true"/>
    <geom name="floor" type="plane" size="10 10 0.1" rgba=".8 .8 .8 1"/>
    
    <!-- Instantiate the mesh as a collision/visual object -->
    <body name="kitchen_env" pos="0 0 0">
       <!-- Visualize -->
       <geom type="mesh" mesh="kitchen_mesh" rgba="1 1 1 1" contype="1" conaffinity="1"/> 
    </body>
  </worldbody>
</mujoco>

  4. Add a Robot (e.g., Franka Emika):

    • You can include an external MJCF file for the robot using the <include> tag or merging the XMLs.
    • See the provided script scripts/mujoco_demo.py for a programmatic example.