Extrinsics seem a bit off

[CSchott]

I am using your Tracking for an AR App. I calculate the extrinsics with the calculateExtrinsics Method in the Marker Class. I then calculate a view-matrix to set the Cameraposition in a Volume Rendering Process on a backend server. When I overlay the taken picture of my Camera and the generated picture of the volume rendering, the rendered object is never positioned at the position of the marker.
Have you experienced similar problems? I am literally out of ideas right now.
I am happy with every suggestion you can give me.
Thank you in advance.

[jsmith613]

Can you show me an example of something being rendered?

[CSchott]

Hello and thank you very much for your quick reply.

This is the current state I have. As you can see the object is not rendered on the Marker itself, what would be expected.
I need to transfer the RVecs and TVecs to a view Matrix. This transformation could be incorrect.
I am trying the transformation given in here
However if I use it just like suggested there, I get some weird looking artefacts rather than a good looking rendered object.

`Matrix rotMat;
{
Mat rVec = data.getRVec();
Mat rMat = new Mat( 3, 3, CvType.CV_64F );
Calib3d.Rodrigues( rVec, rMat );
rotMat = new Matrix( rMat );
}

            // Generate translation vector from tVec
            Matrix tVec = new Matrix( data.getTVec() );

            Matrix viewMat;
            {
                Matrix compMat = new Matrix( 4, 4, new double[]{
                        // @formatter:off
                        rotMat.get( 0, 0 ), rotMat.get( 0, 1 ), rotMat.get( 0, 2 ), tVec.get( 0, 0 ),
                        rotMat.get( 1, 0 ), rotMat.get( 1, 1 ), rotMat.get( 1, 2 ), tVec.get( 1, 0 ),
                        -rotMat.get( 2, 0 ), -rotMat.get( 2, 1 ), -rotMat.get( 2, 2 ), -tVec.get( 2, 0 ),
                        0.0, 0.0, 0.0, 1.0
                      
                } );
                viewMat =  Matrix.transpose(compMat);`

This results in the shown Image.
I thank you very much for your help.

[CSchott]

I tested my issue some more. I found out, that the main problem seems to be the TVec. I am currently not using the RVec and it seems, that the error gets, when the distance to the marker is increased.

Distant
Near

But I am not sure how to resolve this, since the TVec is retrieved in the solvePnP method.
The position of the marker seems to be found correctly.

[jsmith613]

Well I'm not exactly sure what it's doing, but the algorithm on that wordpress page multiplies the viewMatrix by some inverse matrix. He explains it as: "We then apply an inverse matrix, as OpenGL’s Y and Z coordinates run in the opposite direction to OpenCV." Sounds like this could be your problem.

[CSchott]

Thank you for your reply.
I found out, that the problem seems to be the solvePnP itself or the calculate extrinsics. The X and Y Values of the TVec seem to be wrong. They get bigger when the Z Value gets higher.
For example: I only moved the camera away from the marker. The resulting TVecs were

(2.451448134824027)
(1.2727915154079479)
(6.332965784104173)

(4.625307053467822)
(2.9585316223655993)
(13.495538522635183)

(6.6035148163498)
(3.6465885749382143)
(20.11573099565936)

[CSchott]

I tested some more and made a little progress.
I tried to project a 3D cube with the draw3dCube function into the image. Additionally I printed the index of the drawn marker points (stored in the collection).

As you can see the indices of the marker points change. This seem to cause a flipping of the axis as can be seen here:

This seem to be a problem but I don't believe it causes the problems with the translation vector.
What bothers me is, that the z axis is always pointed into the lower right corner of the display. If I rotate the camera the z value always stays pointed towards the same corner.
Because of this I think it is impossible to align the front side of the cube with the backside, which should be the case, when the camera is directly above the marker.

Did you encounter any problems like this before?