Request for an Example of the 57D Egocentric Motion Input

[Zhangshy1019]

Hi, thank you for releasing this great project!

I am currently trying to use the Egocentric Motion input, but I am a bit confused about the format of the 57D action representation.

Could you please provide a simple example of the Egocentric Motion input (e.g., a sequence of 57D actions) or explain how each dimension is defined?

In particular, I would like to understand:

What coordinate system is used (camera coordinate, world coordinate, or another convention)?
What is the scale/unit of the translation values (meters, centimeters, normalized values, etc.)?
Are the rotation values represented as Euler angles, axis-angle, quaternions, or another format?
Is any normalization applied to the 57D action before feeding it into the model?

Having a concrete example or a description of the preprocessing would be extremely helpful for reproducing the results.

Thank you very much for your help!

[qianlim]

Hi, while we work on providing an example, here are some details on how to get the 57D hand pose feature.

action = [camera | R_wrist | R_fingers | L_wrist | L_fingers]

Camera pose: 3 (xyz) + 6 (rot6d) = 9D
R wrist pose: 3 (xyz) + 6 (rot6d) = 9D
R fingertips: 5 joints × 3 (xyz) = 15D
L wrist pose: 3 (xyz) + 6 (rot6d) = 9D
L fingertips: 5 joints × 3 (xyz) = 15D
──────────────────────────────────────────
Total: 9 + 9 + 15 + 9 + 15 = 57D

We use the standard 21-joint hand skeleton. Fingertip joint indices: (4, 8, 12, 16, 20), i.e. thumb tip through pinky tip. index 0 = wrist.

Unit: meters.

Rotation representation: 6D rotation, i.e. first two columns of a 3×3 rotation matrix (Zhou et al. CVPR 2019), see this paper.

How to get the features:

• Camera (ego motion): get world-frame camera-to-world pose from data, compute frame-to-frame delta inv(cam_t) @ cam_{t+1}. Why? This encodes head/viewpoint motion in camera's own coordinates at t.
• Wrists: raw wrist pose in camera frame → lifted to world via cam_c2w → compute frame-to-frame delta inv(wrist_world_t) @ wrist_world_{t+1}. Why? world-lift strips out camera ego-motion so wrist signal = pure hand trajectory in space.
• Fingertips: positions kept in camera frame, projected into per-frame wrist-local via inv(wrist_cam_space) @ finger_cam_space. World-lift is not applied here because finger and wrist share camera frame, so wrist-local result is frame-invariant.