[Performance] Oak-D camera not able to reach desired publish rates

[mstiehm-NF]

I have been trying to use the Oak-D Wide and Oak-D Lite in RGBD mode for use in vslam using the PR here

I have noticed that the camera is not able to publish at the requested rates with either the Depth or RGBD pipelines.

My RGBD yaml is:

/oak:
  ros__parameters:
    camera:
      i_calibration_dump: true
      i_enable_imu: true
      i_enable_ir: true
      i_pipeline_type: RGBD
      i_publish_tf_from_calibration: true
      i_tf_imu_from_descr: 'true'
      i_usb_speed: SUPER_PLUS
    stereo:
      i_publish_topic: true
      i_reverse_stereo_socket_order: true
      i_publish_synced_rect_pair: true
      i_publish_left_rect: true
      i_publish_right_rect: true
      i_update_ros_base_time_on_ros_msg: false
      i_align_depth: true
      i_resolution: '400P'
    left:
      i_resolution: '400P'
      i_fps: 120.0
    right:
      i_resolution: '400P'
      i_fps: 120.0
    rgb:
      i_resolution: '1080P'
      i_fps: 30.0
      i_publish_topic: true
    use_sim_time: false

All the topics, including the IMU are not able to reach the desired rates.
The USB is in SUPER mode.

Any help here would be appreciated.

[Serafadam]

Hi, I think this might be bandwidth related, by default in RGBD mode Stereo is scaled up for better alignment with RGB frame (and its FPS is based on left/right streams). This can be overridden with i_height/i_width params, or you can shrink down rgb frames using ISP scaling. If that does not improve things, you can also try using low_bandwidth mode for the sensor streams. More information on latency can be found here.

[mstiehm-NF]

I have modified my yaml to the following:

/oak:
  ros__parameters:
    camera:
      i_calibration_dump: true
      i_enable_imu: true
      i_enable_ir: true
      i_pipeline_type: RGBD
      i_publish_tf_from_calibration: true
      i_tf_imu_from_descr: 'true'
      i_usb_speed: SUPER_PLUS
    stereo:
      i_publish_topic: true
      i_reverse_stereo_socket_order: true
      i_publish_synced_rect_pair: true
      i_publish_left_rect: true
      i_publish_right_rect: true
      i_update_ros_base_time_on_ros_msg: false
      i_align_depth: true
      i_resolution: '400P'
      i_low_bandwidth: true
      i_low_bandwidth_quality: 1
    imu:
      i_acc_cov: 0.0
      i_acc_freq: 250
      i_batch_report_threshold: 5
      i_enable_rotation: true
      i_get_base_device_timestamp: false
      i_gyro_cov: 0.0
      i_gyro_freq: 200
      i_mag_cov: 0.0
      i_max_batch_reports: 10
      i_message_type: IMU
      i_rot_cov: 0.0
      i_rot_freq: 200
      i_sync_method: LINEAR_INTERPOLATE_ACCEL
      i_update_ros_base_time_on_ros_msg: false
    left:
      i_height: 400
      i_width: 640
      i_fps: 117.0
      i_low_bandwidth: true
      i_low_bandwidth_quality: 1
    right:
      i_height: 400
      i_width: 640
      i_fps: 117.0
      i_low_bandwidth: true
      i_low_bandwidth_quality: 1
    rgb:
      i_resolution: '1080P'
      i_isp_den: 3
      i_isp_num: 2
      i_output_isp: true
      i_fps: 30.0
      i_low_bandwidth: true
      i_low_bandwidth_quality: 1
      i_publish_topic: true
    use_sim_time: false

And I am still only able to get max 30hz from my left and right images and about 30hz on my IMU.
I am at least getting the 30hz on the rgb image though.
Is there any other way to get the full 90hz out of the images and 200hz from the IMU?

[Serafadam]

Hi, sorry for the delay, as mentioned this is probably related to bandwidth, for rectified stereo streams you need to set those parameters separately, for example i_right_rect_low_bandwidth_quality: 50 (full yaml file can be found here)

[mstiehm-NF]

Okay, I'll try that now. Are the IMU noise values on your Isaac PR what I should be using for the BMI270?

I was also able to achieve better results (60fps on the rect streams) by turning the nn off. Still only 60hz on the IMU data though. I'm wondering if it's a processing bottleneck, not just a pure bandwidth problem.

Thanks!
~Matt