From f0fea1ea2260fc6f6bbc21d1157186feb0be2d20 Mon Sep 17 00:00:00 2001 From: "-T.K.-" Date: Sat, 27 Jun 2026 19:46:52 -0700 Subject: [PATCH 1/4] how-to: calibrate the Prime eRob arms New how-to mirroring the Lite "Calibrate the zero pose" page, for the eRob (ZeroErr) EtherCAT arms. Covers what differs from Lite: CSP has no MIT zero-torque, so limp is done by writing the drive's internal loop gains; the 0x2383 "Bus Regulation of PID" gate must be on first or gain writes are ignored; damped-limp (kp=0, ki=0, keep kd) per joint; and the offset is applied by folding it into the per-joint PDO factor/offset (fold_calibration.py) rather than a software offset, since ethercat_driver is upstream. Includes the confirmed 0-9 bus-split table and the common gotchas. Added to the How-to sidebar after calibrate_zero_pose. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_019z5ubj27Me94u163g7E3ec --- docs/how_to/calibrate_prime_erob.md | 135 ++++++++++++++++++++++++++++ sidebars.ts | 1 + 2 files changed, 136 insertions(+) create mode 100644 docs/how_to/calibrate_prime_erob.md diff --git a/docs/how_to/calibrate_prime_erob.md b/docs/how_to/calibrate_prime_erob.md new file mode 100644 index 0000000..b16263a --- /dev/null +++ b/docs/how_to/calibrate_prime_erob.md @@ -0,0 +1,135 @@ +# Calibrate the Prime eRob arms + +Per-physical-robot recipe: regenerate `bar_bringup_prime/config/prime_calibration.yaml` +so each eRob (ZeroErr) joint's absolute encoder zero maps to the URDF's joint zero. +Same idea as [Calibrate the zero pose](./calibrate_zero_pose.md) for Lite — but the +eRob is a stiff CiA402 servo on EtherCAT, not a backdrivable MIT motor, so the +"make it limp" and "apply the offset" steps work differently. + +## What the calibration does + +The eRob output encoder is absolute (it keeps its zero across power cycles) with a +factory zero; the URDF defines a different joint zero. The offset bridges them, +exactly as on Lite: + +``` +joint_pos = direction * (raw_pos - homing_offset) +``` + +`direction` (plus/minus 1) is a wiring fact; `homing_offset` is per-physical-robot. +The full derivation — `homing_offset = 0.5 * ((min - lower) + (max - upper)) * direction`, +averaged over both mechanical stops — is on the +[Calibration math](../concepts/calibration_math.md) page. + +The Lite/eRob difference is **where the offset is applied**. Lite owns its hardware +interface, so it subtracts `homing_offset` in `read()`/`write()`. The eRob runs on the +third-party `ethercat_driver`, which we do not fork — so `scripts/fold_calibration.py` +folds the offset into each joint's PDO `factor`/`offset` in a generated per-slave +config, and `real.launch.py` hands that to the xacro at launch via the +`erob_config_dir` argument. No drive-NVM writes; the calibration stays in one +git-tracked YAML. + +## How the eRob differs from Lite + +- **No MIT zero-torque.** The eRob runs CSP (Cyclic Synchronous Position); it has no + stiffness/damping command interface. To hand-sweep, we turn it into a **velocity + damper** by writing its internal loop gains. +- **The PID gate — `0x2383`.** Object `0x2383` ("Bus Regulation of PID", default 0 = + off) gates whether bus-written loop gains are applied. **It must be set to 1 first**, + or writes to the gain objects are silently ignored and the joint stays stiff. This is + the single most common reason "the limp does nothing". +- **Damped-limp, one joint at a time.** Set position-loop gain `0x2382:01` to 0 (no + stiffness) and velocity-loop integral `0x2381:02` to 0 (no spring-back), but **keep** + the velocity-loop gain `0x2381:01` (damping) so gravity gives a slow controlled + descent instead of freefall. On a heavy arm, limp one joint while the others stay + held — `scripts/erob_limp_joint.sh` does exactly this (snapshot, gate on, kp/ki to 0, + sweep, restore on exit). + +## Prerequisites + +- The arms are **supported** (jig, table, or a helper). Damping limits the *speed* of a + fall, not the position — the shoulder still holds the whole arm. +- The IgH EtherCAT master is running and the drives are powered. See + [First real-hardware bringup](./first_real_bringup.md). +- e-stop in reach. + +## Step 1 — Bring up the chain + +Bring the eRob drives to CSP Operation-Enabled. On the full robot this is +`calibrate.launch.py`; while only a subset is wired, bring up just those drives (the +full `real.launch.py` expects all 13 eRob plus the Sito wrists, so it will not activate +on a partial chain). Bring-up is slow — roughly 6 s per drive, serial, in the IgH/eRob +handshake — so a 10-drive chain takes about 70 s. That is a fixed cost of the eRob +bring-up (not config-tunable: DC, PDO size, and `control_frequency` were all ruled out). + +Activate the broadcaster so `/joint_states` flows: + +```bash +ros2 control load_controller joint_state_broadcaster --set-state active +``` + +## Step 2 — Start the tracker + +```bash +ros2 run bar_bringup_prime calibrate_erob \ + --output ~/prime_calibration.yaml --topic /joint_states +``` + +`calibrate_erob` discovers the eRob joints from the URDF `ec_module`s, reads each +joint's `lower`/`upper` from the kinematic limits, and tracks per-joint `min`/`max` of +`/joint_states` with a live readout. (Pass `--prior ` to preserve already-swept +joints — e.g. calibrate the right arm without re-sweeping the left.) + +## Step 3 — Damped-limp and sweep each joint + +For each ring position, support the joint, then: + +```bash +ros2 run bar_bringup_prime erob_limp_joint # e.g. 4 +``` + +The joint goes damped-limp (the read-back prints `Kp=0`, confirming the `0x2383` gate +worked). Sweep it firmly to **both** mechanical stops — watch the tracker's `sweep` +grow past about 0.5 rad so it is not skipped — return near neutral, and press Enter to +re-hold. If a joint is too stiff to move, pass a smaller damping value as a second +argument. Work distal to proximal; do the shoulder last. + +When every joint is swept, **Ctrl-C the tracker** — it writes the YAML and flags any +joint with too small a sweep (skipped, prior kept) or `abs(homing_offset) > pi` (a +`direction` sign flip — set `-1` and re-sweep that joint). + +## Step 4 — Apply and verify + +Copy the reviewed file over `bar_bringup_prime/config/prime_calibration.yaml`. +`real.launch.py` folds it into per-joint configs automatically at launch. To verify the +sign end-to-end, fold it, re-launch, and move one joint to a known stop — at the stop, +`/joint_states` should read that joint's URDF limit. + +A good cross-check on a symmetric robot: the two arms' offsets should mirror each other +(a joint whose limits are mirrored, like `shoulder_roll`, gets a sign-flipped offset). + +## eRob bus-split (hardware-confirmed) + +Both arms sit on one EtherCAT ring, 0-based and contiguous. The fifth arm eRob is +`wrist_yaw` (not `wrist_roll`); `wrist_roll` and `wrist_pitch` are the small Sito (CAN) +wrists, not on the eRob chain. + +| pos | joint | pos | joint | +|---|---|---|---| +| 0 | left_shoulder_pitch | 5 | right_shoulder_pitch | +| 1 | left_shoulder_roll | 6 | right_shoulder_roll | +| 2 | left_shoulder_yaw | 7 | right_shoulder_yaw | +| 3 | left_elbow_pitch | 8 | right_elbow_pitch | +| 4 | left_wrist_yaw | 9 | right_wrist_yaw | + +Waist (positions 10-12) is inferred and not yet verified. + +## Gotchas + +- **"Limp does nothing / joint stays stiff"** — `0x2383` was not set to 1, so the + drive ignored the gain writes. `erob_limp_joint.sh` sets it first and the read-back + shows `Kp=0` when it took. +- **DC is required.** The eRob faults (status `4616`) in free-run CSP; keep DC enabled. + Damped-limp keeps the drive in CSP/Operation-Enabled the whole time, so this is fine. +- **`wrist_roll`/`wrist_pitch`** are Sito joints — calibrate them with the Sito (CAN) + procedure, not this one. diff --git a/sidebars.ts b/sidebars.ts index 9ae0c22..d79fa53 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -43,6 +43,7 @@ const sidebars: SidebarsConfig = { 'how_to/index', 'how_to/first_real_bringup', 'how_to/calibrate_zero_pose', + 'how_to/calibrate_prime_erob', 'how_to/probe_can_bus', 'how_to/switch_controllers_manually', 'how_to/mit_slider_gui', From 376843ddd84ee6ce2ffbc49fe5be7e29b95e1b29 Mon Sep 17 00:00:00 2001 From: "-T.K.-" Date: Sun, 28 Jun 2026 00:59:40 -0700 Subject: [PATCH 2/4] how-to: extend Prime calibration for Sito wrists + control_frequency - Cover the 4 Sito wrists: same calibrate_erob tool + prime_calibration.yaml, backends:=can isolated sweep, and the flip-a-direction-without-resweeping formula (offset_new = offset + (lower+upper)). - Correct the Fault 4616 guidance: control_frequency MUST equal the controller_manager update_rate (ethercat_driver syncs DC from the CM loop, no separate thread). Separated from the fixed ~70s staged activation. - Drop the waist note (the 3 waist joints are fixed this version). - Fold the bring-up + tracker into one calibrate.launch.py invocation; /joint_states -> /prime/joint_states. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_019z5ubj27Me94u163g7E3ec --- docs/how_to/calibrate_prime_erob.md | 116 ++++++++++++++++++++-------- 1 file changed, 84 insertions(+), 32 deletions(-) diff --git a/docs/how_to/calibrate_prime_erob.md b/docs/how_to/calibrate_prime_erob.md index b16263a..4e76200 100644 --- a/docs/how_to/calibrate_prime_erob.md +++ b/docs/how_to/calibrate_prime_erob.md @@ -1,10 +1,14 @@ -# Calibrate the Prime eRob arms +# Calibrate the Prime arms (eRob + Sito) Per-physical-robot recipe: regenerate `bar_bringup_prime/config/prime_calibration.yaml` -so each eRob (ZeroErr) joint's absolute encoder zero maps to the URDF's joint zero. -Same idea as [Calibrate the zero pose](./calibrate_zero_pose.md) for Lite — but the -eRob is a stiff CiA402 servo on EtherCAT, not a backdrivable MIT motor, so the -"make it limp" and "apply the offset" steps work differently. +so each joint's encoder zero maps to the URDF's joint zero. One file holds all 14 +joints — the 10 eRob (ZeroErr, EtherCAT) arm joints and the 4 Sito (CAN) wrists — and +the **same** `calibrate_erob` sweep tool calibrates both. Same idea as +[Calibrate the zero pose](./calibrate_zero_pose.md) for Lite. + +The eRob is the involved case — a stiff CiA402 servo on EtherCAT, so "make it limp" +works differently — and is covered first. The Sito wrists are backdrivable MIT motors, +limp by default; see [Calibrate the Sito wrists](#calibrate-the-sito-wrists) below. ## What the calibration does @@ -53,34 +57,32 @@ git-tracked YAML. [First real-hardware bringup](./first_real_bringup.md). - e-stop in reach. -## Step 1 — Bring up the chain - -Bring the eRob drives to CSP Operation-Enabled. On the full robot this is -`calibrate.launch.py`; while only a subset is wired, bring up just those drives (the -full `real.launch.py` expects all 13 eRob plus the Sito wrists, so it will not activate -on a partial chain). Bring-up is slow — roughly 6 s per drive, serial, in the IgH/eRob -handshake — so a 10-drive chain takes about 70 s. That is a fixed cost of the eRob -bring-up (not config-tunable: DC, PDO size, and `control_frequency` were all ruled out). - -Activate the broadcaster so `/joint_states` flows: +## Step 1 — Bring up the chain and the tracker -```bash -ros2 control load_controller joint_state_broadcaster --set-state active -``` +Bring the eRob drives to CSP Operation-Enabled with `calibrate.launch.py`. Pass +`backends:=ec` to bring up only the EtherCAT arms (no Sito); the default `backends:=all` +brings up everything. Staged activation is slow — roughly 6-7 s per drive, serial, in +the IgH/eRob handshake — so a 10-drive chain takes about 70 s. That **activation** cost +is fixed (not tunable: it is the per-slave handshake). Note this is separate from +`control_frequency`, which *does* matter — but for steady-state DC sync, not activation +time (see [Gotchas](#gotchas)). -## Step 2 — Start the tracker +The launch also spawns the `joint_state_broadcaster` (so `/prime/joint_states` flows) +and the `calibrate_erob` tracker: ```bash -ros2 run bar_bringup_prime calibrate_erob \ - --output ~/prime_calibration.yaml --topic /joint_states +ros2 launch bar_bringup_prime calibrate.launch.py backends:=ec \ + output:=~/prime_calibration.yaml \ + prior:=$(ros2 pkg prefix bar_bringup_prime)/share/bar_bringup_prime/config/prime_calibration.yaml ``` -`calibrate_erob` discovers the eRob joints from the URDF `ec_module`s, reads each -joint's `lower`/`upper` from the kinematic limits, and tracks per-joint `min`/`max` of -`/joint_states` with a live readout. (Pass `--prior ` to preserve already-swept -joints — e.g. calibrate the right arm without re-sweeping the left.) +`prior:=...` carries already-calibrated joints (the Sito wrists, or the other arm) +through unchanged, so a partial run still writes a complete file. `calibrate_erob` +discovers the eRob joints from the URDF `ec_module`s, reads each joint's `lower`/`upper` +from the kinematic limits, and tracks per-joint `min`/`max` of `/prime/joint_states` +with a live readout. -## Step 3 — Damped-limp and sweep each joint +## Step 2 — Damped-limp and sweep each joint For each ring position, support the joint, then: @@ -98,7 +100,7 @@ When every joint is swept, **Ctrl-C the tracker** — it writes the YAML and fla joint with too small a sweep (skipped, prior kept) or `abs(homing_offset) > pi` (a `direction` sign flip — set `-1` and re-sweep that joint). -## Step 4 — Apply and verify +## Step 3 — Apply and verify Copy the reviewed file over `bar_bringup_prime/config/prime_calibration.yaml`. `real.launch.py` folds it into per-joint configs automatically at launch. To verify the @@ -108,6 +110,44 @@ sign end-to-end, fold it, re-launch, and move one joint to a known stop — at t A good cross-check on a symmetric robot: the two arms' offsets should mirror each other (a joint whose limits are mirrored, like `shoulder_roll`, gets a sign-flipped offset). +## Calibrate the Sito wrists + +The 4 Sito wrists (`left`/`right` `wrist_roll` + `wrist_pitch`) use the **same** +`calibrate_erob` tool and the **same** `prime_calibration.yaml` — it discovers them from +the `bar_sito/SitoSystem` block (by `can_id`) alongside the eRob. Two differences: + +- **Limp is free.** A Sito is an MIT motor; with no command controller active its gains + default to zero, so it is backdrivable out of the box — no gain-gate dance. +- **Sweep on an isolated CAN loop.** Run `backends:=can` so only the Sito come up: the + eRob's ~70 s activation and DC handshake do not interfere, and there are no stiff arm + joints to fight. Seed the eRob offsets through unchanged with `prior:=...` so the + single-bus run still writes a complete 14-joint file. + +```bash +ros2 launch bar_bringup_prime calibrate.launch.py backends:=can \ + output:=~/prime_calibration.yaml \ + prior:=$(ros2 pkg prefix bar_bringup_prime)/share/bar_bringup_prime/config/prime_calibration.yaml +``` + +Hand-sweep each wrist to both stops (the live readout tracks them), Ctrl-C, review, copy +over `prime_calibration.yaml`, and `colcon build --packages-select bar_bringup_prime`. +Unlike the eRob, the Sito read `direction` **and** `homing_offset` straight from this +file (`SitoSystem::load_calibration`) — so a flipped wrist is a one-line edit. + +### Flipping a direction without re-sweeping + +If a joint tracks backwards in viz, flip its `direction` (`+1` to `-1`) and recompute the +offset with the URDF limits: + +``` +offset_new = offset + (lower + upper) +``` + +For a symmetric joint (`lower = -upper`, e.g. the wrists) the offset is unchanged; for an +asymmetric one (e.g. `shoulder_roll`, `elbow_pitch`) it shifts. Then +`colcon build --packages-select bar_bringup_prime` — no source rebuild; the eRob fold and +`SitoSystem` both re-read the file at launch. This works for any joint, eRob or Sito. + ## eRob bus-split (hardware-confirmed) Both arms sit on one EtherCAT ring, 0-based and contiguous. The fifth arm eRob is @@ -122,14 +162,26 @@ wrists, not on the eRob chain. | 3 | left_elbow_pitch | 8 | right_elbow_pitch | | 4 | left_wrist_yaw | 9 | right_wrist_yaw | -Waist (positions 10-12) is inferred and not yet verified. +There is no waist this version — the 3 former waist joints are `fixed` in the URDF, so +the ring is exactly these 10 drives. ## Gotchas - **"Limp does nothing / joint stays stiff"** — `0x2383` was not set to 1, so the drive ignored the gain writes. `erob_limp_joint.sh` sets it first and the read-back shows `Kp=0` when it took. -- **DC is required.** The eRob faults (status `4616`) in free-run CSP; keep DC enabled. - Damped-limp keeps the drive in CSP/Operation-Enabled the whole time, so this is fine. -- **`wrist_roll`/`wrist_pitch`** are Sito joints — calibrate them with the Sito (CAN) - procedure, not this one. +- **`control_frequency` must equal the controller_manager `update_rate`.** + `ethercat_driver` sends PDOs and syncs the DC clock from the CM `update()` loop (there + is no separate EtherCAT thread), so the DC SYNC0 cycle — set by `control_frequency` — + must match the loop rate. Mismatched (e.g. `control_frequency=1000` against a 50 Hz + loop) the drives see SYNC0 firing far more often than frames arrive and fault in steady + state (status `4616`/`520`, domain WC collapses). `real.launch.py` derives + `control_frequency` from the controllers YAML `update_rate` so they cannot diverge; if + you set it by hand, keep them equal. This is the usual cause of "eRob hold for a moment, + then fault". +- **Keep DC enabled.** The eRob also faults (`4616`) in free-run CSP with DC disabled — + do not turn DC off to "fix" timing; match `control_frequency` instead. Damped-limp + keeps the drive in CSP/Operation-Enabled the whole time, so calibration is unaffected. +- **`wrist_roll`/`wrist_pitch` are Sito** — calibrate them with the same tool via + [Calibrate the Sito wrists](#calibrate-the-sito-wrists) (`backends:=can`), not the eRob + limp procedure. From 4fd4f8cf380a33c343057bbc533de36e2301c192 Mon Sep 17 00:00:00 2001 From: "-T.K.-" Date: Sun, 28 Jun 2026 19:36:52 -0700 Subject: [PATCH 3/4] docs: Prime hybrid actuation + bringup reference New concepts/prime_hybrid_actuation.md documenting the BAR Prime bringup: the 14-DoF two-bus topology (10 eRob CiA402 over EtherCAT + 4 Sito MIT over SocketCAN), the one-controller_manager / two-ros2_control-block architecture, the control_frequency==update_rate DC rule and sequenced spawners, both impedance realizations with exact SI->actuator equations (eRob MIT-in-CSP loop gains kp/kd -> 0x2382/0x2381 gated by 0x2383; Sito MIT kp/688.58, kd/1.125), the transmission-efficiency caveat + torque_efficiency knob, the per-mode PD table, the impedance manager and its "no SDO until /prime/joint_states" 0xA000 rule, and both performance fixes (parallel-SDO mode switch 3.6s->0.46s; 1 kHz startup bring-up loop 70s->13.6s). Adds three troubleshooting entries and a Concepts sidebar entry. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_019z5ubj27Me94u163g7E3ec --- docs/concepts/prime_hybrid_actuation.md | 277 ++++++++++++++++++++++++ docs/reference/troubleshooting.md | 43 ++++ sidebars.ts | 1 + 3 files changed, 321 insertions(+) create mode 100644 docs/concepts/prime_hybrid_actuation.md diff --git a/docs/concepts/prime_hybrid_actuation.md b/docs/concepts/prime_hybrid_actuation.md new file mode 100644 index 0000000..8cc7969 --- /dev/null +++ b/docs/concepts/prime_hybrid_actuation.md @@ -0,0 +1,277 @@ +--- +title: Prime hybrid actuation +--- + +# Prime hybrid actuation + +BAR Prime is the bimanual humanoid. The **waist is dropped** this version +(rigid torso, no waist drives), leaving **14 actuated DoF** — 7 per arm. +What makes Prime different from Lite is that those 14 joints live on **two +different buses, driven by two different actuator families**, yet present a +single uniform [MIT command surface](./mit_command_surface.md) to one +`controller_manager`. This page is the "why" layer for the whole Prime +bringup: the topology, how each family realizes impedance, the exact +SI-to-actuator conversions, the per-mode PD values, and the DC-sync, +switch-latency, and startup behaviors we had to solve. + +## Topology — 14 DoF on two buses + +Each arm has the same 7 joints. Five are **eRob** (ZeroErr CiA402 servos on +EtherCAT, through the IgH master); two — the distal wrist roll/pitch — are +**Sito** (TA40-50 MIT motors on SocketCAN). + +| Joint (per arm) | Family | Bus | Address | +|---|---|---|---| +| `shoulder_pitch` | eRob | EtherCAT (master 0) | ring pos 0 / 5 | +| `shoulder_roll` | eRob | EtherCAT | ring pos 1 / 6 | +| `shoulder_yaw` | eRob | EtherCAT | ring pos 2 / 7 | +| `elbow_pitch` | eRob | EtherCAT | ring pos 3 / 8 | +| `wrist_roll` | Sito | SocketCAN `can2` | id 22 (L) / 38 (R) | +| `wrist_pitch` | Sito | SocketCAN `can2` | id 23 (L) / 39 (R) | +| `wrist_yaw` | eRob | EtherCAT | ring pos 4 / 9 | + +So the EtherCAT ring carries **10 eRob** (positions 0-9, left arm 0-4, right +arm 5-9) and the CAN bus carries **4 Sito** wrists (ids `0x16/0x17/0x26/0x27`). +All Prime eRob are **50:1** gear. The single source of truth for this mapping +is `bar_bringup_prime/config/prime_hardware.yaml` (`buses`, `joints.all_joints`, +`joints.erob_slaves`, `joints.mit_joints`); the per-joint bus assignment is +emitted by `bar_description_prime/urdf/prime.ros2_control.xacro`. + +Note the **kinematic order** in `all_joints` (shoulder pitch/roll/yaw, elbow, +wrist roll/pitch/yaw) is not the **ring order** — `wrist_yaw` is eRob position +4/9 even though it sits distal of the two Sito wrists. Controllers bind to the +flat 14-joint `all_joints` list regardless of which bus carries each joint. + +### Which eRob model, and why it matters a little + +Two eRob models appear on Prime, distinguished only by their motor torque +constant (read each joint's model off its label; see the eRob manual §25.2): + +| Model / version | `Kt` (Nm/mA, at motor) | +|---|---| +| eRob70 V4_MC2 | `0.132e-3` (the code default) | +| eRob80 V5_MC2 | `0.134e-3` (≈ 1.5% higher) | + +The 1.5% `Kt` difference is negligible in practice; the **gear ratio** would +be the real lever, but every Prime eRob is 50:1, so a single conversion +serves all of them. `bar_bringup_prime/config/prime_hardware.yaml` can still +override `Kt`/gear per joint (`joints.erob_kt` / `joints.erob_gear`) if a model +ever differs. + +## One controller_manager, two ros2_control blocks + +`real.launch.py` expands the xacro with `use_fake_hardware:=false use_sim:=false`, +which emits **two concurrent `ros2_control` blocks** — `PrimeEtherCATSide` +(`ethercat_driver/EthercatDriver`, the 10 eRob) and `PrimeSitoCAN` +(`bar_sito/SitoSystem`, the 4 Sito wrists). One `controller_manager` runs them +together and exposes a flat 14-joint list to the mode controllers. The sim path +(`mujoco.launch.py`) collapses both into one `MujocoSystem` block but presents +the identical 14 joints, so the shared controllers run unchanged. + +Two bringup details are load-bearing: + +- **`control_frequency` must equal the CM `update_rate` (50 Hz).** The eRob DC + SYNC0 cycle is driven from the controller_manager's `update()` loop, not a + separate thread. If `control_frequency` is higher than `update_rate`, SYNC0 + fires more often than process-data frames arrive, the distributed clock can't + lock, and the drives fault. `real.launch.py` reads `update_rate` out of the + controllers YAML and derives `control_frequency` from it so they cannot + diverge. +- **Spawners are sequenced** (`joint_state_broadcaster` → `zero_torque_controller` + → the inactive mode controllers). The eRob activation takes tens of seconds + (see [Startup time](#startup-time-70-s--136-s)); running the spawners + concurrently makes them contend on the spawner's hard-coded 20 s file lock, + time out, and collide. Chaining them on process-exit means each acquires the + lock alone. + +## Two actuator families, two ways to realize impedance + +Both families expose the same five MIT command interfaces +([position, velocity, effort, stiffness, damping](./mit_command_surface.md)), +but they implement the `τ = Kp·(q_cmd − q) + Kd·(q̇_cmd − q̇) + τ_ff` law very +differently — and that difference is the heart of the Prime control story. + +### Sito: native MIT over CAN + +The Sito firmware runs the MIT law directly. The host sends the desired +position, velocity, and feedforward torque in a command frame, and `Kp`/`Kd` +in a separate gains frame. Conversions (in +`bar_devices/bar_sito/include/bar_sito/sito_protocol.hpp`, TA40-50): + +``` +position_counts = q_cmd · 65536 / (2π) # 16-bit encoder, MOTOR side +velocity_counts = q̇_cmd · 65536 / (2π) +ff_current[mA] = τ_ff / (Kt · gear) # Kt = 9e-5 Nm/mA, gear = 51 +Kp_sent = kp / 688.58 # Nm/rad per firmware Kp unit +Kd_sent = kd / 1.125 # Nm·s/rad per firmware Kd unit +``` + +The `688.58` / `1.125` divisors are a **measured** calibration of the firmware's +per-unit physical effect at the joint output. An earlier version of the code +used `500` / `0.5`, which silently ran the wrists about **1.38x too stiff** and +**2.25x over-damped**; the measured divisors fix that. (Because that section of +the source derivation is iterative, reconfirm with a torque measurement if you +need precision.) + +### eRob: MIT-in-CSP (impedance emulated by loop gains) + +The eRob has no per-tick stiffness/damping interface. It runs **CSP** (Cyclic +Synchronous Position, mode 8): the controller commands a target position every +tick, and the joint's *impedance* is the drive's **internal position and +velocity loop gains**. A cascaded position→velocity loop linearizes to +`kp = Kpos · Kvel` and `kd = Kvel`, which is why stiffness is expressed through +the loop gains. + +The gains are manufacturer CoE objects, reachable only by **acyclic SDO** (they +are not PDO-mappable, so they cannot be a per-tick command interface): + +| Object | Role | +|---|---| +| `0x2382:01` | position loop gain (`pos_reg`) | +| `0x2381:01` | velocity loop gain (`vel_reg`) | +| `0x2381:02` | velocity loop integral (held at 0 for clean impedance) | +| `0x2383` | "Bus Regulation of PID" gate — `1` = use bus-written gains, `0` = factory | + +The SI-to-register conversion (`bar_bringup_prime/scripts/erob_impedance_manager.py`, +`erob_gains()`), validated exactly against the ZeroErr derivation: + +``` +vel_reg (0x2381:01) = kd / 0.1063 ≈ 9.41 · kd # cD = 0.1063 Nm·s/rad per LSB +pos_reg (0x2382:01) = (kp / kd) · 51.47 # cP = 0.01943, ratio only +integral (0x2381:02) = 0 +``` + +Consequences worth remembering: + +- **`kd > 0` is required for any stiffness.** Because `pos_reg` is proportional + to `kp/kd`, a `kd` of 0 collapses both registers to 0 (true limp / zero + torque). +- **Only `vel_reg` depends on `Kt` and gear.** The `kp/kd` ratio that sets + `pos_reg` is model-independent, so the eRob80 vs eRob70 difference touches + only the damping register. +- **Feedback is gear-independent.** Position and velocity are read from the + 19-bit *output* encoder, so a wrong gear/model would only mis-scale the + *impedance magnitude*, never the position tracking. + +#### The realized-vs-nominal caveat (transmission efficiency) + +Load tests showed the eRob realizes roughly **0.7x** the nominal stiffness. The +clean explanation is that the manual's output-torque relation is +`joint Kt = motor Kt · gear · transmission_efficiency`, and the conversion above +omits the efficiency term (harmonic drives are ~0.7-0.85 efficient). The +manager exposes an optional `torque_efficiency` parameter (default `1.0` = off); +setting it to the measured realized/nominal ratio compensates the loss. It is +applied to **`vel_reg` only**, which provably corrects *both* the realized `kp` +and `kd` while leaving `pos_reg` (the ratio) untouched. + +## Per-mode PD values + +The mode controllers ([five-mode FSM](./five_mode_fsm.md)) set these per mode. +The eRob per-mode gains live in `erob_impedance_manager`'s `mode_kp`/`mode_kd`; +the Sito gains come from the active controller's `stiffness`/`damping`. + +| Mode | `kp` (Nm/rad) | `kd` (Nm·s/rad) | Notes | +|---|---|---|---| +| ZERO_TORQUE | 0 | 0 | True limp on both families | +| DAMPING | 0 | 6 | Uniform across eRob + Sito (compliant fail-safe) | +| STANDBY | 20 | 2 | Position hold; ramps in over the entry trajectory | +| LOCOMOTION | 20 | 2 (eRob, fixed) | Sito follow the policy's per-tick gains | +| REMOTE | 20 | 2 (eRob, fixed) | Sito follow the per-tick remote command | + +Key asymmetry: **the eRob impedance is per-*mode* only.** SDO is acyclic and +slow, so the manager cannot retrack a per-tick varying stiffness — it writes a +fixed impedance on each mode change. In LOCOMOTION/REMOTE the four Sito wrists +honor the policy's per-tick `Kp`/`Kd`, but the ten eRob arm joints hold the +fixed mode impedance and track position in CSP. + +## The eRob impedance manager + +`bar_bringup_prime/scripts/erob_impedance_manager.py` is the bridge between the +mode FSM and the eRob loop gains. It subscribes to `/control_mode` and, on each +transition, converts that mode's `(kp, kd)` to loop-gain registers and writes +them over the EtherLab `ethercat download` CLI. (It uses the CLI, not the +in-process `ethercat_manager` SDO service, because the conda `libethercat` is +version-mismatched against the running kernel master.) + +The one rule that keeps bringup healthy: + +> **No SDO touches the bus until `/prime/joint_states` is flowing.** The +> joint_state_broadcaster only activates once every slave is fully OP, so that +> topic is the "hardware is up" signal. Mailbox SDO traffic during the staged +> DC activation disrupts the cyclic exchange and faults a slave with `0xA000` +> (EtherCAT communication error). The manager therefore stays entirely off the +> bus until the robot is up, then arms the gate (`0x2383=1`) and writes the +> first mode's gains. It resets the gate to `0` on exit so the next bringup +> activates with factory (stiff) gains. + +Per-joint `Kt`/gear and the scalar `torque_efficiency` are parameters, wired +from `prime_hardware.yaml`. + +## Two things we solved + +### PD-switch latency (3.6 s → 0.46 s) + +**Symptom:** switching modes (e.g. DAMPING ↔ STANDBY) propagated visibly across +the arms — the left shoulder responded immediately, the right arm seconds later. + +**Mechanism:** in OP each CoE SDO transfer is **cycle-gated** — the master +advances the SDO state machine roughly once per 50 Hz cycle, so one object takes +~120 ms (~6 cycles). A mode switch writes 3 objects to 10 slaves = **30 SDOs**, +and the manager did them strictly in ring order, so the total was ~3.6 s with +the tail-of-ring (right arm) carrying the full cumulative delay. + +**Fix:** the IgH master pipelines outstanding mailbox transfers across slaves, +so issuing the per-slave writes **concurrently** (one worker per slave) collapses +the wall-clock from sum-of-slaves to roughly one slave's time — measured +**~0.46 s, all slaves within ~40 ms** of each other. Controlled by the +`parallel_sdo` parameter (default on); set it false to reproduce the sequential +baseline. + +### Startup time (70 s → 13.6 s) + +**Symptom:** bringup took ~70 s and logged ~7 transient `0xA000` fault/recover +cycles before stabilizing. + +**Mechanism:** a slave reaches EtherCAT OP only after the master's **DC drift +compensation converges**, which is cycle-count bound. The ICube +`ethercat_driver`'s `on_activate` bring-up loop paced its `update()` at +`control_frequency` (50 Hz), so convergence took ~7 s per slave (the domain +working-counter climbed one slave at a time, ~7 s apart). The `0xA000` faults +were collateral: each newly-joining slave briefly glitched the domain and +starved the others' output-watchdog. + +**Fix:** a small patch runs the bring-up loop at **1 kHz**, independent of +`control_frequency`. DC converges ~5x faster and the watchdog stays fed, so +bringup drops to **~13.6 s with zero faults** — and there is **no steady-state +change** (DC SYNC0 still runs at `control_frequency`, and the CM read/write loop +takes over at `update_rate` once activation returns). The patch is a local +modification to the ICube `ethercat_driver_ros2` that `bar.repos` pins +(`ethercat_driver/src/ethercat_driver.cpp`); to survive a fresh `vcs import` it +must land in a Berkeley fork that `bar.repos` then pins (TODO). + +## Fault / status reference + +The CiA402 statusword and eRob error codes you will actually see on Prime: + +| Code | Meaning | +|---|---| +| statusword `5687` | Operation Enabled (healthy, running) | +| statusword `4616` | Fault state — on Prime almost always the DC-sync / comms family | +| error `0xA000` | EtherCAT communication error (see startup + impedance-manager notes above) | +| error `0x8500` | Position error exceeds limit | +| error `0x8400` | Velocity error exceeds limit | +| error `0x8130` | CAN heartbeat error | + +Read the live error code with `ethercat upload -pN 0x603F 0` and the stored +history with `0x1003`. Symptom-first entries are on the +[Troubleshooting](../reference/troubleshooting.md) page. + +## See also + +- [Five-mode FSM](./five_mode_fsm.md) — the mode controllers and transition gating. +- [MIT command surface](./mit_command_surface.md) — the shared 5-interface model. +- [Calibrate the Prime arms](../how_to/calibrate_prime_erob.md) — software + calibration (direction + homing offset), single-source in + `prime_calibration.yaml`, folded into the eRob configs at launch and read by + `SitoSystem` for the wrists. +- [First real-hardware bringup](../how_to/first_real_bringup.md). diff --git a/docs/reference/troubleshooting.md b/docs/reference/troubleshooting.md index ccb12e4..95222a3 100644 --- a/docs/reference/troubleshooting.md +++ b/docs/reference/troubleshooting.md @@ -164,8 +164,51 @@ controller is misbehaving, deactivate it and replace with `zero_torque`. Otherwise raise `txqueuelen` as a workaround while you diagnose. +## Prime eRob bringup takes ~70 s with repeated `0xA000` faults + +**Diagnosis**: the eRob reach EtherCAT OP one at a time (~7 s each), and +`0xA000` (EtherCAT communication error) faults cycle until the domain is +complete. + +**Why**: a slave reaches OP only after the master's DC drift compensation +converges, which is cycle-count bound. The ICube `ethercat_driver` `on_activate` +bring-up loop paced its `update()` at the 50 Hz control rate, so convergence +took ~7 s per slave; the faults are collateral (each joining slave briefly +starves the others' output watchdog). + +**Fix**: a local patch to the ICube `ethercat_driver_ros2` (pinned by `bar.repos`) +runs the bring-up loop at 1 kHz, independent of `control_frequency`. Bringup drops +to ~13.6 s with zero faults, no steady-state change. See [Prime hybrid actuation](../concepts/prime_hybrid_actuation.md). + +## Prime mode switch propagates slowly across the arm (one joint at a time) + +**Diagnosis**: the right arm gets its new stiffness/damping seconds after the +left on a mode change. + +**Why**: the eRob loop gains are written by acyclic SDO, and in OP each transfer +is cycle-gated (~120 ms). Writing 3 objects to 10 slaves in ring order is ~3.6 s, +right arm last. + +**Fix**: keep `parallel_sdo` enabled (the default) on `erob_impedance_manager` — +concurrent per-slave writes pipeline through the IgH master to ~0.46 s, all +slaves within ~40 ms. See [Prime hybrid actuation](../concepts/prime_hybrid_actuation.md). + +## Prime eRob faults `4616` immediately on enable + +**Diagnosis**: CiA402 Fault state (statusword `4616`) right after activation; on +Prime this is the DC-sync / comms family. + +**Why**: usually `control_frequency` does not equal the controller_manager +`update_rate`. SYNC0 is driven from the CM loop, so a mismatch means the +distributed clock never locks. + +**Fix**: `real.launch.py` derives `control_frequency` from the controllers YAML +`update_rate` so they cannot diverge; if you set it by hand, keep them equal +(50 Hz). Read the live error with `ethercat upload -pN 0x603F 0`. + ## Cross-references +- **Hybrid actuation, PD conversion, eRob/Sito impedance**: [Prime hybrid actuation](../concepts/prime_hybrid_actuation.md) - **Boot-time bringup checks**: [First real-hardware bringup → Common boot-time failures](../how_to/first_real_bringup.md#common-boot-time-failures) - **Calibration drift**: [Calibrate the zero pose](../how_to/calibrate_zero_pose.md) - **Bus / qdisc nitty-gritty**: [Diagnose ENOBUFS](../how_to/diagnose_enobufs.md) diff --git a/sidebars.ts b/sidebars.ts index d79fa53..648cf3a 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -67,6 +67,7 @@ const sidebars: SidebarsConfig = { 'concepts/workspace_and_environment', 'concepts/five_mode_fsm', 'concepts/mit_command_surface', + 'concepts/prime_hybrid_actuation', 'concepts/calibration_math', 'concepts/safety_pipeline', 'concepts/frozen_schemas', From 04c149a84139121356ad993d5b2f852590a173fc Mon Sep 17 00:00:00 2001 From: "-T.K.-" Date: Sun, 28 Jun 2026 21:06:58 -0700 Subject: [PATCH 4/4] docs: Prime description moved to external prime_description repo Reflect the bar_description_prime -> prime_description migration (external T-K-233/Prime-Description, prime_dummy variant, like lite_description): packages reference + heading anchor, the prime_hybrid_actuation ros2_control path, the launch_args cross-link, the installation package list, and the stale hardware_specs placeholder note (Prime is now external + 14-DoF hardware-validated). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_019z5ubj27Me94u163g7E3ec --- docs/concepts/prime_hybrid_actuation.md | 2 +- docs/getting_started/installation.md | 1 - docs/reference/hardware_specs.md | 12 +++++++----- docs/reference/launch_args.md | 2 +- docs/reference/packages.md | 4 ++-- 5 files changed, 11 insertions(+), 10 deletions(-) diff --git a/docs/concepts/prime_hybrid_actuation.md b/docs/concepts/prime_hybrid_actuation.md index 8cc7969..49ba085 100644 --- a/docs/concepts/prime_hybrid_actuation.md +++ b/docs/concepts/prime_hybrid_actuation.md @@ -35,7 +35,7 @@ arm 5-9) and the CAN bus carries **4 Sito** wrists (ids `0x16/0x17/0x26/0x27`). All Prime eRob are **50:1** gear. The single source of truth for this mapping is `bar_bringup_prime/config/prime_hardware.yaml` (`buses`, `joints.all_joints`, `joints.erob_slaves`, `joints.mit_joints`); the per-joint bus assignment is -emitted by `bar_description_prime/urdf/prime.ros2_control.xacro`. +emitted by `prime_description` (`robots/prime_dummy/xacro/prime_dummy.ros2_control.xacro`). Note the **kinematic order** in `all_joints` (shoulder pitch/roll/yaw, elbow, wrist roll/pitch/yaw) is not the **ring order** — `wrist_yaw` is eRob position diff --git a/docs/getting_started/installation.md b/docs/getting_started/installation.md index f1b172e..8b7db3b 100644 --- a/docs/getting_started/installation.md +++ b/docs/getting_started/installation.md @@ -235,7 +235,6 @@ bar_bringup_prime bar_cli bar_common bar_controllers -bar_description_prime bar_robstride bar_sito bar_socketcan diff --git a/docs/reference/hardware_specs.md b/docs/reference/hardware_specs.md index bb51b10..a1eb346 100644 --- a/docs/reference/hardware_specs.md +++ b/docs/reference/hardware_specs.md @@ -193,11 +193,13 @@ Bimanual with EtherCAT-driven eRob actuators in the arms and SocketCAN-driven Sito actuators for auxiliary joints, running concurrently in the same `controller_manager`. -:::info[Prime URDF is not yet imported] -The Prime mechanical CAD has not been finalized at the time of writing. -`bar_description_prime` is a placeholder package; `bar_lite_controllers.yaml` -binds 17 real joints, but `bar_prime_controllers.yaml` is still -`["__placeholder__"]`. Joint specs below are projections, not commitments. +:::info[Prime description is external + hardware-validated] +The Prime description now lives in the external CAD-generated +[`prime_description`](https://github.com/T-K-233/Prime-Description) repo — bar +deploys the `prime_dummy` variant via `bar.repos` — with the **waist dropped** +(rigid torso, **14 actuated DoF**). `bar_prime_controllers.yaml` binds the real +14-joint set (no longer a placeholder). The joint specs below were early +projections; cross-check against `prime_description` + `prime_hardware.yaml`. ::: ### Projected joint topology diff --git a/docs/reference/launch_args.md b/docs/reference/launch_args.md index 1ce53ba..58d2b24 100644 --- a/docs/reference/launch_args.md +++ b/docs/reference/launch_args.md @@ -115,7 +115,7 @@ Implicit: - The xacro is invoked with `use_sim:=true`. **`use_sim` wins over `use_fake_hardware`** in the xacro's `` selector — see the - decision tree in [Packages](packages.md#lite_description-external--bar_description_prime). + decision tree in [Packages](packages.md#lite_description--prime_description-external). - Every node runs with `use_sim_time:=true`. Time advances at MuJoCo's pace via `/clock`. - `bar_bringup_lite/config/sim_overrides.yaml` is layered on top of diff --git a/docs/reference/packages.md b/docs/reference/packages.md index 2edcc97..1c28fc2 100644 --- a/docs/reference/packages.md +++ b/docs/reference/packages.md @@ -75,14 +75,14 @@ wire conventions only** — no participant/transport. The `lite_sdk2` SDK builds its publisher/subscriber layer on top. See [Talk to bar_ros2 from Python](../how_to/talk_to_bar_ros2_from_python.md). -### `lite_description` (external) / `bar_description_prime` +### `lite_description` / `prime_description` (external) URDF / xacro / meshes / `` blocks. **Lite's description is no longer in `bar_ros2`** — it lives in the external, CAD-generated [`lite_description`](https://github.com/Berkeley-Humanoids/Lite-Description) repo (bar deploys the `lite_dummy` variant), pulled in via `bar.repos`. It is **asset-only**: the RViz inspector (`view_lite.launch.py` + `view_lite.rviz`) now -lives in `bar_bringup_lite`. `bar_description_prime` is still an in-tree stub. +lives in `bar_bringup_lite`. Prime's description likewise lives in the external [`prime_description`](https://github.com/T-K-233/Prime-Description) repo (bar deploys the `prime_dummy` variant, which also carries the hybrid eRob+Sito ``). Layout (Lite shown, inside the `lite_description` repo):