diff --git a/docs/pages/example_workflows/locomanipulation/step_2_teleoperation.rst b/docs/pages/example_workflows/locomanipulation/step_2_teleoperation.rst index b23625315..07bae8766 100644 --- a/docs/pages/example_workflows/locomanipulation/step_2_teleoperation.rst +++ b/docs/pages/example_workflows/locomanipulation/step_2_teleoperation.rst @@ -6,48 +6,58 @@ This workflow covers collecting demonstrations for the G1 loco-manipulation task Step 1: Start the CloudXR Runtime ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -On the host machine, configure the firewall to allow CloudXR traffic. The required ports depend on the client type. +#. On the host machine, configure the firewall to allow CloudXR traffic. The required ports depend on the client type. -.. code-block:: bash - - sudo ufw allow 49100/tcp # Signaling - sudo ufw allow 47998/udp # Media stream - sudo ufw allow 48322/tcp # Proxy (HTTPS mode only) + .. code-block:: bash + sudo ufw allow 49100/tcp # Signaling + sudo ufw allow 47998/udp # Media stream + sudo ufw allow 48322/tcp # Proxy (HTTPS mode only) -Start the CloudXR runtime from the Arena Docker container: +#. Start the CloudXR runtime from the Arena Docker container: -:docker_run_default: + :docker_run_default: -.. code-block:: bash + .. code-block:: bash - python -m isaacteleop.cloudxr + python -m isaacteleop.cloudxr Step 2: Start Arena Teleop ^^^^^^^^^^^^^^^^^^^^^^^^^^ -In another terminal, start the Arena Docker container and launch the teleop session to verify the pipeline: +#. In another terminal, start the Arena Docker container: -:docker_run_default: + :docker_run_default: -.. code-block:: bash +#. Run the following command to activate IsaacTeleop CloudXR environment settings: - source ~/.cloudxr/run/cloudxr.env - python isaaclab_arena/scripts/imitation_learning/teleop.py \ - --viz kit \ - --device cpu \ - galileo_g1_locomanip_pick_and_place \ - --teleop_device openxr + .. code-block:: bash -Start the session from the **XR** tab in the application window. + source ~/.cloudxr/run/cloudxr.env -.. figure:: ../../../images/locomanip_arena_server.png - :width: 100% - :alt: Arena teleop with XR running (stereoscopic view and OpenXR settings) - :align: center + .. important:: + **Order matters.** In the terminal where you will run Arena, ``source ~/.cloudxr/run/cloudxr.env`` *after* the CloudXR runtime from Step 1 is already running, + and *before* you start the Arena app. The Arena app must inherit the IsaacTeleop CloudXR environment variables. + +#. Run the teleop script: + + .. code-block:: bash + + python isaaclab_arena/scripts/imitation_learning/teleop.py \ + --viz kit \ + --device cpu \ + galileo_g1_locomanip_pick_and_place \ + --teleop_device openxr + +#. In the running application, start the session from the **XR** tab in the application window. - Arena teleop session with XR running. Stereoscopic view (left) and OpenXR settings in the XR tab (right). + .. figure:: ../../../images/locomanip_arena_server.png + :width: 100% + :alt: Arena teleop with XR running (stereoscopic view and OpenXR settings) + :align: center + + Arena teleop session with XR running. Stereoscopic view (left) and OpenXR settings in the XR tab (right). Step 3: Connect from Meta Quest 3 @@ -59,7 +69,6 @@ A strong wireless connection is essential for a high-quality streaming experienc #. Open the browser on your headset and navigate to ``_. - #. Enter the IP address of your Isaac Lab host machine in the **Server IP** field. #. Click the **Click https://:48322/ to accept cert** link that appears on the page. @@ -70,9 +79,9 @@ A strong wireless connection is essential for a high-quality streaming experienc #. **Teleoperation Controls**: -* **Left joystick**: Move the body forward/backward/left/right. -* **Right joystick**: Squat (down), rotate torso (left/right). -* **Controllers**: Move end-effector (EE) targets for the arms. + * **Left joystick**: Move the body forward/backward/left/right. + * **Right joystick**: Squat (down), rotate torso (left/right). + * **Controllers**: Move end-effector (EE) targets for the arms. .. note:: @@ -86,11 +95,20 @@ A strong wireless connection is essential for a high-quality streaming experienc Reducing render resolution from 1 (default) to 0.2. +Once you have verified the teleoperation pipeline, exit VR from the Quest 3 headset, and stop the Arena teleop app. Step 4: Record with Quest 3 ^^^^^^^^^^^^^^^^^^^^^^^^^^^ -#. **Recording**: When ready to collect data, run the recording script from the Arena container: +.. note:: + + Run the following command to activate IsaacTeleop CloudXR environment settings again if you are starting the recording app from a different terminal. + + .. code-block:: bash + + source ~/.cloudxr/run/cloudxr.env + +#. Run the recording script from the Arena container: .. code-block:: bash @@ -107,6 +125,10 @@ Step 4: Record with Quest 3 galileo_g1_locomanip_pick_and_place \ --teleop_device openxr +#. In the running application, start the session from the XR tab in the application window. + +#. Follow Step 3 to connect the Quest 3 headset again. + #. Complete the task for each demo. Reset between demos. The script saves successful runs to the HDF5 file above. .. hint:: diff --git a/docs/pages/example_workflows/sequential_static_manipulation/step_2_teleoperation.rst b/docs/pages/example_workflows/sequential_static_manipulation/step_2_teleoperation.rst index 4f3a22e52..3f7771806 100644 --- a/docs/pages/example_workflows/sequential_static_manipulation/step_2_teleoperation.rst +++ b/docs/pages/example_workflows/sequential_static_manipulation/step_2_teleoperation.rst @@ -6,98 +6,103 @@ This workflow covers collecting demonstrations using Isaac Teleop with an XR dev Step 1: Start the CloudXR Runtime ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - .. tab-set:: .. tab-item:: Meta Quest 3 / Pico 4 Ultra :selected: - On the host machine, configure the firewall to allow CloudXR traffic. - - .. code-block:: bash + #. On the host machine, configure the firewall to allow CloudXR traffic. - sudo ufw allow 49100/tcp # Signaling - sudo ufw allow 47998/udp # Media stream - sudo ufw allow 48322/tcp # Proxy (HTTPS mode only) + .. code-block:: bash + sudo ufw allow 49100/tcp # Signaling + sudo ufw allow 47998/udp # Media stream + sudo ufw allow 48322/tcp # Proxy (HTTPS mode only) - Start the CloudXR runtime from the Arena Docker container: + #. Start the CloudXR runtime from the Arena Docker container: - :docker_run_default: + :docker_run_default: - Create a CloudXR config to enable hand tracking: + #. Create a CloudXR config to enable hand tracking: - .. code-block:: bash + .. code-block:: bash - echo "NV_CXR_ENABLE_PUSH_DEVICES=0" > handtracking.env + echo "NV_CXR_ENABLE_PUSH_DEVICES=0" > handtracking.env + #. Start the CloudXR runtime with the customized config file: - Start the CloudXR runtime with the customized config file: + .. code-block:: bash - .. code-block:: bash - - python -m isaacteleop.cloudxr --cloudxr-env-config=handtracking.env + python -m isaacteleop.cloudxr --cloudxr-env-config=handtracking.env .. tab-item:: Apple Vision Pro - On the host machine, configure the firewall to allow CloudXR traffic. - - .. code-block:: bash + #. On the host machine, configure the firewall to allow CloudXR traffic. - # Signaling (use one based on connection mode) - sudo ufw allow 48010/tcp # Standard mode - sudo ufw allow 48322/tcp # Secure mode - # Video - sudo ufw allow 47998/udp - sudo ufw allow 48005/udp - sudo ufw allow 48008/udp - sudo ufw allow 48012/udp - # Input - sudo ufw allow 47999/udp - # Audio - sudo ufw allow 48000/udp - sudo ufw allow 48002/udp + .. code-block:: bash - Start the CloudXR runtime from the Arena Docker container: + # Signaling (use one based on connection mode) + sudo ufw allow 48010/tcp # Standard mode + sudo ufw allow 48322/tcp # Secure mode + # Video + sudo ufw allow 47998/udp + sudo ufw allow 48005/udp + sudo ufw allow 48008/udp + sudo ufw allow 48012/udp + # Input + sudo ufw allow 47999/udp + # Audio + sudo ufw allow 48000/udp + sudo ufw allow 48002/udp - :docker_run_default: + #. Start the CloudXR runtime from the Arena Docker container: - Create a customized config file with the following content: + :docker_run_default: - .. code-block:: bash + #. Create a customized config file with the following content: - printf '%s\n' 'NV_DEVICE_PROFILE=auto-native' 'NV_CXR_ENABLE_PUSH_DEVICES=0' > avp.env + .. code-block:: bash + printf '%s\n' 'NV_DEVICE_PROFILE=auto-native' 'NV_CXR_ENABLE_PUSH_DEVICES=0' > avp.env - Start the CloudXR runtime with the customized config file: + #. Start the CloudXR runtime with the customized config file: - .. code-block:: bash + .. code-block:: bash - python -m isaacteleop.cloudxr --cloudxr-env-config=avp.env + python -m isaacteleop.cloudxr --cloudxr-env-config=avp.env Step 2: Start Recording ^^^^^^^^^^^^^^^^^^^^^^^ -In another terminal, start the Arena Docker container: +#. In another terminal, start the Arena Docker container: + + :docker_run_default: + +#. Run the following command to activate IsaacTeleop CloudXR environment settings: + + .. code-block:: bash + + source ~/.cloudxr/run/cloudxr.env -:docker_run_default: + .. important:: + **Order matters.** In the terminal where you will run Arena, ``source ~/.cloudxr/run/cloudxr.env`` *after* the CloudXR runtime from Step 1 is already running, + and *before* you start the Arena app. The Arena app must inherit the IsaacTeleop CloudXR environment variables. -Run the recording script: +#. Run the recording script: -.. code-block:: bash + .. code-block:: bash - source ~/.cloudxr/run/cloudxr.env - python isaaclab_arena/scripts/imitation_learning/record_demos.py \ - --device cpu \ - --viz kit \ - --dataset_file $DATASET_DIR/ranch_bottle_into_fridge_recorded.hdf5 \ - --num_demos 10 \ - --num_success_steps 10 \ - put_item_in_fridge_and_close_door \ - --object ranch_dressing_hope_robolab \ - --embodiment gr1_pink \ - --teleop_device openxr + python isaaclab_arena/scripts/imitation_learning/record_demos.py \ + --device cpu \ + --viz kit \ + --dataset_file $DATASET_DIR/ranch_bottle_into_fridge_recorded.hdf5 \ + --num_demos 10 \ + --num_success_steps 10 \ + put_item_in_fridge_and_close_door \ + --object ranch_dressing_hope_robolab \ + --embodiment gr1_pink \ + --teleop_device openxr Step 3: Connect XR Device and Record diff --git a/docs/pages/example_workflows/static_manipulation/step_2_teleoperation.rst b/docs/pages/example_workflows/static_manipulation/step_2_teleoperation.rst index 7662c28bf..21b08902a 100644 --- a/docs/pages/example_workflows/static_manipulation/step_2_teleoperation.rst +++ b/docs/pages/example_workflows/static_manipulation/step_2_teleoperation.rst @@ -11,90 +11,96 @@ Step 1: Start the CloudXR Runtime .. tab-item:: Meta Quest 3 / Pico 4 Ultra :selected: - On the host machine, configure the firewall to allow CloudXR traffic. + #. On the host machine, configure the firewall to allow CloudXR traffic. - .. code-block:: bash + .. code-block:: bash - sudo ufw allow 49100/tcp # Signaling - sudo ufw allow 47998/udp # Media stream - sudo ufw allow 48322/tcp # Proxy (HTTPS mode only) + sudo ufw allow 49100/tcp # Signaling + sudo ufw allow 47998/udp # Media stream + sudo ufw allow 48322/tcp # Proxy (HTTPS mode only) + #. Start the CloudXR runtime from the Arena Docker container: - Start the CloudXR runtime from the Arena Docker container: + :docker_run_default: - :docker_run_default: + #. Create a CloudXR config to enable hand tracking: - Create a CloudXR config to enable hand tracking: + .. code-block:: bash - .. code-block:: bash + echo "NV_CXR_ENABLE_PUSH_DEVICES=0" > handtracking.env - echo "NV_CXR_ENABLE_PUSH_DEVICES=0" > handtracking.env + #. Start the CloudXR runtime with the customized config file: + .. code-block:: bash - Start the CloudXR runtime with the customized config file: - - .. code-block:: bash - - python -m isaacteleop.cloudxr --cloudxr-env-config=handtracking.env + python -m isaacteleop.cloudxr --cloudxr-env-config=handtracking.env .. tab-item:: Apple Vision Pro - On the host machine, configure the firewall to allow CloudXR traffic. + #. On the host machine, configure the firewall to allow CloudXR traffic. - .. code-block:: bash + .. code-block:: bash - # Signaling (use one based on connection mode) - sudo ufw allow 48010/tcp # Standard mode - sudo ufw allow 48322/tcp # Secure mode - # Video - sudo ufw allow 47998/udp - sudo ufw allow 48005/udp - sudo ufw allow 48008/udp - sudo ufw allow 48012/udp - # Input - sudo ufw allow 47999/udp - # Audio - sudo ufw allow 48000/udp - sudo ufw allow 48002/udp + # Signaling (use one based on connection mode) + sudo ufw allow 48010/tcp # Standard mode + sudo ufw allow 48322/tcp # Secure mode + # Video + sudo ufw allow 47998/udp + sudo ufw allow 48005/udp + sudo ufw allow 48008/udp + sudo ufw allow 48012/udp + # Input + sudo ufw allow 47999/udp + # Audio + sudo ufw allow 48000/udp + sudo ufw allow 48002/udp - Start the CloudXR runtime from the Arena Docker container: + #. Start the CloudXR runtime from the Arena Docker container: - :docker_run_default: + :docker_run_default: - Create a customized config file with the following content: + #. Create a customized config file with the following content: - .. code-block:: bash + .. code-block:: bash - printf '%s\n' 'NV_DEVICE_PROFILE=auto-native' 'NV_CXR_ENABLE_PUSH_DEVICES=0' > avp.env + printf '%s\n' 'NV_DEVICE_PROFILE=auto-native' 'NV_CXR_ENABLE_PUSH_DEVICES=0' > avp.env + #. Start the CloudXR runtime with the customized config file: - Start the CloudXR runtime with the customized config file: + .. code-block:: bash - .. code-block:: bash - - python -m isaacteleop.cloudxr --cloudxr-env-config=avp.env + python -m isaacteleop.cloudxr --cloudxr-env-config=avp.env Step 2: Start Recording ^^^^^^^^^^^^^^^^^^^^^^^ -In another terminal, start the Arena Docker container: +#. In another terminal, start the Arena Docker container: + + :docker_run_default: + +#. Run the following command to activate IsaacTeleop CloudXR environment settings: + + .. code-block:: bash + + source ~/.cloudxr/run/cloudxr.env -:docker_run_default: + .. important:: + **Order matters.** In the terminal where you will run Arena, ``source ~/.cloudxr/run/cloudxr.env`` *after* the CloudXR runtime from Step 1 is already running, + and *before* you start the Arena app. The Arena app must inherit the IsaacTeleop CloudXR environment variables. -Run the recording script: +#. Run the recording script: -.. code-block:: bash + .. code-block:: bash - source ~/.cloudxr/run/cloudxr.env - python isaaclab_arena/scripts/imitation_learning/record_demos.py \ - --device cpu \ - --viz kit \ - --dataset_file $DATASET_DIR/arena_gr1_manipulation_dataset_recorded.hdf5 \ - --num_demos 10 \ - --num_success_steps 2 \ - gr1_open_microwave \ - --teleop_device openxr + python isaaclab_arena/scripts/imitation_learning/record_demos.py \ + --device cpu \ + --viz kit \ + --dataset_file $DATASET_DIR/arena_gr1_manipulation_dataset_recorded.hdf5 \ + --num_demos 10 \ + --num_success_steps 2 \ + gr1_open_microwave \ + --teleop_device openxr Step 3: Connect XR Device and Record