Can mrcal calculate the extrinsics of multiple objects using a single camera from multiple views?

[zumpchke]

I have multiple "AprilTag" fiducial markers in a scene. I want to recover the pose of these markers. Simulation of scene:

I will independently calibrate the camera.

Then move it amongst the scene and record AprilTag corners as a 2x2 chessboard and pass in --skip-intrinsics. There will likely be overlapping views of multiple AprilTags.

I understand that for each frame, I will get the camera pose (relative to the first camera position/frame). Is there a way to get the poses of the calibration objects in the frame of the first calibration object (Apriltag id=0)?

[dkogan]

Let's make sure I understand. One moving camera, you have intrinsics, you don't know the poses of the motion. You're observing N apriltags that are stationary. You don't know their relative pose. Yes?

If you're using the mrcal-calibrate-cameras tool, today you can treat the 8 apriltag corners as 8 completely independent points or you can use just ONE of the tags as a small "chessboard". You cannot do both. Solving this as a "calibration" problem should work just fine, but that's honestly overkill, and constrains you. If you're only looking at ONE apriltag, you can solve for the relative pose by solving the "pnp problem". Opencv can do this with SolvePnp() and opengv has many other methods. With more than one apriltag, you can use the mrcal Python API. Set indices_frame_camintrinsics_camextrinsics appropriately. You should play with it.

[zumpchke]

Thanks for the response!

Let's make sure I understand. One moving camera, you have intrinsics, you don't know the poses of the motion. You're observing N apriltags that are stationary. You don't know their relative pose. Yes?

Yes, that's correct.

To make it concrete, let's say I have these 3 views:

I'm trying to figure out: with the 8 apriltag corners in 3 views, is there a way to feed that data into mrcal-calibrate-cameras to recover the AprilTag poses or would I have to dig into the Python API?

Previously, I was using a moving checkerboard to calibrate multiple cameras (#39), and I had figured out a way to provide the appropriate data, but this seems slightly different and I can't quite wrap my head around if it's possible.

I understood the point about the solvePnP: The problem is that I'll eventually have N apriltags that can't be seen in a single frame, so I was hoping the wizardry of mrcal could help me handle multiple views.

[dkogan]

mrcal-calibrate-cameras is not flexible-enough to do this, but the Python API is. I don't recall ever doing this exact thing, but I THINK it should work. This API isn't super complex, but there are a lot of knobs, and probably the documentation isn't too detailed.

There's one call: https://mrcal.secretsauce.net/mrcal-python-api-reference.html#-optimize

You have 2 "chessboards" with 4 corners each. If you have 2 tags and 3 cameras observing all of these, the observations_board array will have shape (6,2,2,3). This is the pixel coordinates. Each of the 6 observations has metadata in indices_frame_camintrinsics_camextrinsics: this has shape (6,3). Let's say you put the reference at the first camera pose. Then indices_frame_camintrinsics_camextrinsics[:,0] is the index of the chessboard poses: [0,1,0,1,0,1]. indices_frame_camintrinsics_camextrinsics[:,1] is the index of the intrinsics: all 0. indices_frame_camintrinsics_camextrinsics[:,2] is the index of the camera poses: [-1,-1,0,0,1,1]. Let me know if it doesn't work. I think it should

[dkogan]

One more thing you should be aware of: mrcal.optimize() takes estimates for the final solution as input. You can't make these up completely, it won't work. mrcal-calibrate-cameras takes care of this for you. The seeding routines are https://mrcal.secretsauce.net/mrcal-python-api-reference.html#-estimate_monocular_calobject_poses_Rt_tocam and https://mrcal.secretsauce.net/mrcal-python-api-reference.html#-estimate_joint_frame_poses These may or may not be applicable to your multiple-chessboard-moving-camera scenario. Read the docstrings closely.

[zumpchke]

Thanks for the sketch! I'm giving it a go. However, seeing this following your recommendation on indices_frame_camintrinsics_camextrinsics

Traceback (most recent call last):
  File "/repos/mrcal-experiment/main.py", line 127, in <module>
    main()
  File "/repos/mrcal-experiment/main.py", line 111, in main
    stats = mrcal.optimize(intrinsics=intrinsics_data,
RuntimeError: mrcal-pywrap.c:1102 optimize_validate_args(): iframe MUST be monotonically increasing in indices_frame_camintrinsics_camextrinsics. Instead row 2 of indices_frame_camintrinsics_camextrinsics has iframe=0 after previously seeing iframe=1

[dkogan]

Oh yeah. I vaguely remember making this assumption. Reorder your observations. iframe [0,0,0,1,1,1], icamextrinsics [-1,0,1,-1,0,1]

[zumpchke]

Thank you.

Now hitting this issue with mrcal 2.5.2

Traceback (most recent call last):
  File "/repos/mrcal-experiment/main.py", line 129, in <module>
    main()
  File "/repos/mrcal-experiment/main.py", line 113, in main
    stats = mrcal.optimize(intrinsics=intrinsics_data,
RuntimeError: mrcal-pywrap.c:1123 optimize_validate_args(): iframe in indices_frame_camintrinsics_camextrinsics must cover ALL frames. Instead the last row of indices_frame_camintrinsics_camextrinsics has iframe=1, but Nframes=6

https://gist.github.com/zumpchke/8970cb92370da1975c307a6da5489805
I can't see where I'm giving Nframes=6?

[zumpchke]

It looks like indices_frame_camera had to be re-sorted too! Seems to complete with no errors, but expected an RMS <1px for a simulated camera system.

[zumpchke]

Scrubbed some issues, and it's looking pretty good! Will hold this if I have more questions surrounding recovering the poses of the AprilTags

[dkogan]

Excellent. Looks like the seeding routines worked also?

[zumpchke]

Mostly. I had to make this change in calibration.py:

        #if np.count_nonzero(i) < 6:
        #    raise SolvePnPerror_toofew(f"Insufficient observations; need at least 6; got {np.count_nonzero(i)} instead. Cannot estimate initial extrinsics for {what}")

The results are good for my simulated environment. With more photos, I am able to recover the pose of tag 1 wrt to tag 0 (red box's transform calculated from mrcal)

For seeding, I couldn't figure out what to put in Rt_cam_ref that goes into estimate_joint_frame_poses, so I made it the identity. Will this cause problems?

[dkogan]

A bad seed would make the solve not converge. Since you're getting reasonable-looking results, it would appear that using the identity transform as a seed was good-enough in this case. The full logic exists, but it isn't pulled out into a nice function all by itself. See

mrcal/mrcal/calibration.py

Line 1398 in a750c88

def seed_stereographic( imagersizes,

and

mrcal/mrcal/calibration.py

Line 925 in a750c88

def _estimate_camera_poses( # shape (Nobservations,4,3)

[zumpchke]

Looking at this again.

I'm curious what is the difference between board observations and points in the context of the structure from motion problem?

- observations_board: array of dims (Nobservations_board,
                                     calibration_object_height_n,
                                     calibration_object_width_n,
                                     3).
  Each slice is an (x,y,weight) tuple where (x,y) are the observed pixel
  coordinates of the corners in the calibration object, and "weight" is the
  relative weight of this point observation. Most of the weights are expected to
  be 1.0, which implies that the noise on that observation has the nominal
  standard deviation of observed_pixel_uncertainty (in addition to the overall
  assumption of gaussian noise, independent on x,y). weight<0 indicates that
  this is an outlier. This is respected on input (even if
  !do_apply_outlier_rejection). New outliers are marked with weight<0 on output.
  Subpixel interpolation is assumed, so these contain 64-bit floating point
  values, like all the other data. The frame and camera that produced these
  observations are given in the indices_frame_camintrinsics_camextrinsics

- observations_point: array of dims (Nobservations_point, 3). Each slice is an
  (x,y,weight) tuple where (x,y) are the pixel coordinates of the observed
  point, and "weight" is the relative weight of this point observation. Most of
  the weights are expected to be 1.0, which implies that the noise on the
  observation is gaussian, independent on x,y, and has the nominal standard
  deviation of observed_pixel_uncertainty. weight<0 indicates that this is an
  outlier. This is respected on input (even if !do_apply_outlier_rejection). At
  this time, no new outliers are detected for point observations. Subpixel
  interpolation is assumed, so these contain 64-bit floating point values, like
  all the other data. The point index and camera that produced these
  observations are given in the indices_point_camera_points array.

Can I mix board observations and points?

[dkogan]

You can use both. This isn't something people do a lot, so MAYBE you'll hit bugs, but I'm fairly certain it'll work fine. Today, the uncertainty propagation definitely will NOT work when you do this, but that's something I'm actively working on. Let me know if you run into trouble

[dkogan]

Can I ask what you're doing? I'm going to be doing something similar to what you're describing (multiple apriltags as 2x2 chesboards) in the near future