kuka-allegro-lab

Isaac Lab (Newton branch) RL training for dexterous manipulation with a Kuka IIWA14 arm + Allegro Hand.

Phase 1 of a two-phase pipeline:

• Phase 1 (this repo): Train RL policies on RTX Pro 6000 workstations using Isaac Lab's Newton physics engine
• Phase 2 (later): Fine-tune Cosmos Policy on a DGX Spark using demonstration data collected here

---

What this repo provides

Component	Description
Isaac-KukaAllegro-Lift-v0	Pick a cube at least 4 cm off the table
Isaac-KukaAllegro-Reorient-v0	Lift + rotate cube to a goal orientation
MDP (rewards, obs, terminations)	Ported from dexsuite/kuka_allegro and inhand/mdp, Newton-aware
Demo collection	Camera + proprioception recordings in Cosmos Policy format
Docker Compose	Newton-ready container for RTX Pro 6000 workstations

---

Prerequisites

Requirement	Version
Python	3.11
PyTorch	2.7.0
CUDA driver	≥ 525 (for Isaac Sim 5.1+)
OS	Ubuntu 22.04 (x86_64)
GPU	RTX Pro 6000 or similar NVIDIA GPU

---

Quick start

1. Install

# Clone this repo
git clone https://github.com/ynashed/kuka-allegro-lab.git
cd kuka-allegro-lab

# Set up Python 3.11 environment
conda create -n kuka-allegro python=3.11
conda activate kuka-allegro

# Clone Newton branch of Isaac Lab + install extension
bash scripts/install.sh

Pinning Newton commit: Before running install.sh, edit NEWTON_COMMIT in the script to a specific commit hash from feature/newton. Run git log --oneline -5 in the cloned IsaacLab/ to find a known-good hash.

2. Train

# Lift task, 64 parallel envs
bash scripts/train.sh --task Isaac-KukaAllegro-Lift-v0 --num_envs 64

# Reorient task, 128 envs
bash scripts/train.sh --task Isaac-KukaAllegro-Reorient-v0 --num_envs 128

Checkpoints are saved to logs/rsl_rl/<experiment_name>/.

3. Collect demos (Phase 2 data)

# Uses the most recent checkpoint automatically
bash scripts/collect_demos.sh --num_demos 100

# Or specify a checkpoint
bash scripts/collect_demos.sh \
    --checkpoint logs/rsl_rl/kuka_allegro_lift/model_5000.pt \
    --num_demos 100

Demo format per episode (data/demo_N/):

File	Shape	Description
primary_{t:04d}.jpg	224×224 RGB	Third-person camera
wrist_{t:04d}.jpg	224×224 RGB	Wrist-mounted camera
proprio_{t:04d}.npy	(30,)	kuka_q(7) + allegro_q(16) + ee_pos(3) + ee_quat_xyzw(4)
action_{t:04d}.npy	(23,)	kuka_dq(7) + allegro_dq(16)

This format matches NVlabs/cosmos-policy expected inputs directly.

---

Docker (RTX Pro 6000 workstation)

# First run: clones Isaac Lab and installs extension inside the container
docker compose -f docker/docker-compose.yaml up

# Subsequent runs: drops straight to bash
docker compose -f docker/docker-compose.yaml run kuka-allegro-lab bash

Image tag: Update docker/docker-compose.yaml with a Newton-compatible nvcr.io/nvidia/isaac-lab digest once a stable Newton image is published.

---

Repository structure

kuka-allegro-lab/
├── .github/workflows/lint.yaml        # ruff lint CI
├── docker/docker-compose.yaml         # Newton container
├── scripts/
│   ├── install.sh                     # Clone Newton branch + install extension
│   ├── train.sh                       # Launch RSL-RL training
│   └── collect_demos.sh               # Record demos for Cosmos Policy
├── source/kuka_allegro/               # pip-installable Isaac Lab extension
│   ├── kuka_allegro/
│   │   ├── tasks/manipulation/
│   │   │   ├── kuka_allegro_env_cfg.py   # Manager-based env (Lift + Reorient)
│   │   │   └── mdp/
│   │   │       ├── rewards.py            # finger_contact, lift, goal_dist
│   │   │       ├── observations.py       # fingertip_pos, goal_pose, cameras
│   │   │       └── terminations.py       # out_of_reach
│   │   └── assets/robots.py             # Re-exports KukaAllegroRobotCfg
│   └── pyproject.toml
├── data/                              # Demo recordings (gitignored by pattern)
└── README.md

---

Newton branch: key differences from Isaac Lab main

Property	Main branch	Newton (feature/newton)
Physics engine	PhysX	Newton (NVIDIA Warp)
Quaternion convention	wxyz	xyzw
Data arrays	PyTorch tensors	Warp arrays → call wp.to_torch()
Python	3.10	3.11 required
PyTorch	2.5	2.7.0
Status	Stable	Beta 2 (target: Isaac Lab 3.0)

All MDP functions in source/kuka_allegro/kuka_allegro/tasks/manipulation/mdp/ guard articulation data reads with wp.to_torch() from the start.

---

Tasks registered

import gymnasium as gym
import kuka_allegro  # triggers gym registration

env = gym.make("Isaac-KukaAllegro-Lift-v0")
env = gym.make("Isaac-KukaAllegro-Reorient-v0")

Action space: 23-dim joint position deltas — 7 Kuka arm + 16 Allegro hand Observation space: 72-dim proprioception — arm pos/vel (14) + hand pos/vel (32) + fingertip positions (12) + object pose (7) + goal pose (7)

---

Isaac Lab reference files

The following files in feature/newton were used as primary references:

Reference	Path
Starting point	isaaclab_tasks/manager_based/manipulation/dexsuite/config/kuka_allegro/dexsuite_kuka_allegro_env_cfg.py
Reward/obs MDP	isaaclab_tasks/manager_based/manipulation/inhand/mdp/
Asset config	isaaclab_assets/robots/kuka_allegro.py
Newton API reference	isaaclab_tasks/direct/cartpole/

---

Milestones

• scripts/install.sh runs clean — Newton branch cloned, extension installed
• python train.py --task Isaac-KukaAllegro-Lift-v0 --num_envs 64 — env creates, rewards fire, episodes terminate
• scripts/collect_demos.sh with 10 episodes — data format matches Cosmos Policy spec
• Lift task converges (cube lifted consistently)
• Reorient task converges
• 100+ demos collected for Phase 2

---

Risks

Risk	Mitigation
Newton Beta 2 breaking changes	Commit pinned in install.sh; update deliberately
No dexterous Newton example yet	Ported from dexsuite_kuka_allegro; verify under Newton backend
Warp array / torch tensor mismatch	wp.to_torch() guards on all articulation reads
Isaac Sim 5.1 not yet on workstations	Check version; Newton can run headless without Omniverse