Questions about Codebase and Evaluation

[shibowing]

Khronos Build Issue and Integration Questions

Hi all,

We’ve been following the Khronos project and are excited about its direction. As we’re developing a SLAM system with a similar focus, we’d like to compare results and explore integration. However, we’ve encountered an issue when building the latest version.

During compilation, we get:

error: ‘SemanticIntegratorPtr’ has not been declared
This occurs in projective_integrator.h, introduced in commit 5bf72d9. We couldn’t locate the definition of SemanticIntegratorPtr. Is this missing or defined elsewhere?

As a temporary workaround, we removed the parameter from the constructor and related calls. The build succeeded, but running on real datasets resulted in:

Warning: Cannot perform object reconstruction without semantic integrator
No input topics in RViz (only Dynamic Images)
Malformed bounding boxes near the ground

Could you clarify the expected inputs for Khronos' real dataset pipeline? Are RGB, depth, and semantic images sufficient?

We’re also exploring some modifications and would appreciate your input:

Semantic Point Cloud Input
– Can we replace depth + semantic images with a semantic point cloud? If so, where should this change go?
External Pose Injection
– How can we bypass VIO and use precomputed poses?
Thanks in advance for your help—we’re excited to build on your work!

Best regards,

[shibowing]

Here is the result from our side. Screenshots: input topics are missing, only Dynamic Images are available. Tiny boxes observed near ground

[yunzc]

@shibowing we looked into this and @nathanhhughes just pushed a fix for the error: ‘SemanticIntegratorPtr’ has not been declared error. Can you give this a try?

Khronos Build Issue and Integration Questions

Hi all,

We’ve been following the Khronos project and are excited about its direction. As we’re developing a SLAM system with a similar focus, we’d like to compare results and explore integration. However, we’ve encountered an issue when building the latest version.

During compilation, we get:

error: ‘SemanticIntegratorPtr’ has not been declared This occurs in projective_integrator.h, introduced in commit 5bf72d9. We couldn’t locate the definition of SemanticIntegratorPtr. Is this missing or defined elsewhere?

As a temporary workaround, we removed the parameter from the constructor and related calls. The build succeeded, but running on real datasets resulted in:

Warning: Cannot perform object reconstruction without semantic integrator No input topics in RViz (only Dynamic Images) Malformed bounding boxes near the ground

Could you clarify the expected inputs for Khronos' real dataset pipeline? Are RGB, depth, and semantic images sufficient?

We’re also exploring some modifications and would appreciate your input:

Semantic Point Cloud Input – Can we replace depth + semantic images with a semantic point cloud? If so, where should this change go?

Currently we have only tested with depth and semantic / instance images. We do plan on supporting a semantic point cloud, but that is a work in progress.

External Pose Injection – How can we bypass VIO and use precomputed poses? Thanks in advance for your help—we’re excited to build on your work!

This should already be supported. Khronos is just listening to the tf, so make sure the frames are defined properly!

Best regards,

[zl3466]

Thank you very much for the prompt reply! We just gave it a try and the repo did build without error. The scene graph nodes are being visualized properly as well for real data replays.

However, we are still observing tiny boxes near ground. Is this expected?

[zl3466]

Hi all,
To follow up on our previous idea about using our own precomputed poses instead of kimera_vio, we are having some issues running khronos with our own real dataset.

Our current input rosbag contains the following topics, and we have moidified the input topic names in jackal_khronos.launch file accordingly.

topics:      /aft_mapped_to_init               492 msgs    : nav_msgs/Odometry          
             /aft_mapped_to_init_imu           492 msgs    : nav_msgs/Odometry          
             /camera/camera_info               490 msgs    : sensor_msgs/CameraInfo     
             /camera/depth/camera_info         490 msgs    : sensor_msgs/CameraInfo     
             /camera/depth/image_raw           490 msgs    : sensor_msgs/Image          
             /camera/image                     490 msgs    : sensor_msgs/Image          
             /camera/image/compressed          490 msgs    : sensor_msgs/CompressedImage
             /imu/data                        9830 msgs    : sensor_msgs/Imu            
             /integrated_to_init              9814 msgs    : nav_msgs/Odometry          
             /rosout                             7 msgs    : rosgraph_msgs/Log          
             /rosout_agg                         7 msgs    : rosgraph_msgs/Log          
             /velodyne_cloud_registered_imu    492 msgs    : sensor_msgs/PointCloud2

Since our rgb images are from a 360 camera, we do not have a valid distortion coefficient for camera_info messages, and therefore cannot use kimera_vio_ros for odometry generation. We thus skipped launching kimera_vio_ros and tried to use our own prepared odometry in /aft_mapped_to_init_imu topic with frame_id sensor_init by setting up some new tf nodes like this:

    <arg name="robot_frame" default="sensor"/>
    <arg name="odom_frame" default="sensor_init"/>
    <arg name="map_frame" default="map"/>
    <arg name="world_frame" default="world"/>
    <arg name="sensor_frame" default="camera_link"/>

    <!-- Static transforms -->
    <node pkg="tf2_ros" type="static_transform_publisher" name="robot_to_camera_tf" 
        args="0 0 0 0 0 0 1 sensor camera_link"/>
    <node pkg="tf2_ros" type="static_transform_publisher" name="robot_to_lidar_tf" 
        args="0 0 0 0 0 0 1 sensor lidar_link"/>
    <node pkg="tf2_ros" type="static_transform_publisher" name="world_to_map_tf" 
        args="0 0 0 0 0 0 1 world map"/>
    <!-- Convert odometry to TF using hydra_ros odom_to_tf -->
    <node pkg="hydra_ros" type="odom_to_tf" name="odom_to_tf_broadcaster">
        <remap from="odom" to="$(arg odom_topic)"/>
        <param name="parent_frame" value="$(arg odom_frame)"/>
        <param name="child_frame" value="$(arg robot_frame)"/>
    </node>

With these, both khronos and semantic_inference can run without error, and we can see the pose trajectory being visualized in rviz, but the mesh reconstruction does not seem to be working properly. The visualizer only shows some tiny pieces of mesh triangles at the beginning, and fail to continue reconstruction over the course of the replay.

ours_no_kimera-2025-07-11_19.09.54.mp4

Could you please help us learn if this is caused by mis-configured frame tf, inaccurate camera_info, or something else we missed in the pipeline? Thanks a lot in advance!

[Schmluk]

Hi @shibowing,

It looks like Khronos is getting the poses correctly, otherwise no mesh would show up at all. There are several things that can go wrong here:

• The extrinsics are wrong (getting the wrong sensor poses, not sure this is the case here though)
• The intrinsics are wrong: Khronos is using camera intrinsics either specified through configs or read from ROS data, if they are wrong of course the reconstruction will look ad.
• The sensor model is wrong: This seems most likely since our default camera model is a pinhole model that likely will not work well with 360deg images. You can find all our currently supported sensor models here : https://github.com/MIT-SPARK/Hydra/blob/main/include/hydra/input/camera.h Adding a 360deg model that is maybe similar to that for LiDAR might be the most promising first thing to check.

Hope this helps!

[adrienzhh]

Hi all,

I see that your Hydra supports semantic point cloud input. If we want to integrate Khronos to work with point clouds, what steps would it take? In this case, do we still need to provide camera extrinsics/intrinsics?

Here are the topics we provide:

/semantic_pointcloud
/rgb_image
/semantic_image

Thanks!