RAM usage keeps accumlating

[BarzelS]

Description:
RAM memory usage keeps accumlating in a rate of about 20MB per second

Running the Kimera-VIO-ROS with D435i
Normal movement around the 2 room and back to the starting point on the table.

Command:

roscore
roslaunch kimera_vio_ros kimera_vio_ros_realsense_IR.launch
rosbag play ~/d435i_recs/my_bag.bag
rviz

Console output:

-----------                        #	Hz	(avg     +- std    )	[min,max]
Data Provider [ms]            	      0	
IMU Preintegration Timing [us]	   1745	11.5143	(133.979 +- 14.3228)	[86,2735]
Mesher [ms]                   	    284	1.87226	(21.0669 +- 5.96944)	[3,40]
VioBackEnd [ms]               	    284	1.87236	(49.3134 +- 13.4856)	[2,139]
VioFrontEnd [ms]              	   1745	11.4843	(56.1456 +- 119.395)	[5,410]
I0510 15:20:52.095813 24526 KimeraVioRos.cpp:99] Statistics

Additional files:

Please give also the following information:
SparkVio branch, tag or commit used
GTSAM version used: master branch
OpenGV version used: master branch
OpenCV version used: 3.4.2
Operating system and version (e.g. Ubuntu 16.04 or Windows 10): ubuntu 18.04
Did you change the source code? (yes / no): No, only testlogger.cpp in order to allign with gtsam

Hardware:
Both on Jetson TX2 and on Dell G5

[ToniRV]

Hi @shirbarzel,

Thanks for the heads-up.
Just to be sure, are you using the OpenCV visualizer or Rviz?

I was also seeing a memory leak a while ago and we figured out it was OpenCV's 3D viz module that was not freeing memory appropriately.

[BarzelS]

Hi @shirbarzel,

Thanks for the heads-up.
Just to be sure, are you using the OpenCV visualizer or Rviz?

I was also seeing a memory leak a while ago and we figured out it was OpenCV's 3D viz module that was not freeing memory appropriately.

Hi @ToniRV, thanks for the replay,

Actually the high RAM usage I mentioned was without running Rviz in the background:

[ToniRV]

So you also have the --visualize flag set to 0 or false?
Meaning, you are not seeing any kind of visualization (OpenCV)?

[BarzelS]

Yes, the default value of the visualize flag in the launch file is set to false(src/Kimera-VIO-ROS/launch/kimera_vio_ros.launch)

[violetteavi]

+1. Running on some simulated data (60Hz, 1024x768), my kimera_vio_ros_node consistently accumulated ~2GB of RAM in 12 seconds. I am using RViz to visualize the feature tracks and transforms. When I shut down RViz and run Kimera, the load drops from 2.5GB -> 1.7GB. Seems like a memory leak on the order of image size every frame.

[violetteavi]

I think I found the reason why my use case is leaking so horribly-- Kimera throttles its output to 30Hz, so half of my frames are getting effectively leaked as the queue backs up.

Changing the publishing rate to 120Hz kept the RAM usage to a manageable 400MB throughout

[BarzelS]

I think I found the reason why my use case is leaking so horribly-- Kimera throttles its output to 30Hz, so half of my frames are getting effectively leaked as the queue backs up.

Changing the publishing rate to 120Hz kept the RAM usage to a manageable 400MB throughout

Hi @violetteavi, thanks for the replay
I've changed the param in line 363(Kimera-VIO-ROS/src/RosOnlineDataProvider.cpp)
to :
ros::Rate rate(120);

But still got huge memory usage accumulating:

[violetteavi]

My next guess is that the processing speed on your device is not quite fast enough to process the D435i, leading to a backlog of image frames in the input queues. Are you experiencing any time delay between your input images and Kimera's output? This is pretty easy to check by viewing the RealSense image alongside the debug image /kimera_vio_ros/feature_tracks/image_raw.

[violetteavi]

If the debug image looks like it's behind the RealSense image, that means Kimera isn't keeping up with the image stream. You can a) decrease the RealSense framerate to 15Hz, b) tune the Kimera parameters to decrease quality in favor of speed, or c) use a faster processor. I recommend a), because it is simple. You just need to change d435i's rs_camera.launch and the framerates in the LeftCameraParams.yaml and RightCameraParams.yaml.

To check to see if this will actually work, you can also artificially double your computer's processing power by running the bag at half speed

rosbag play ~/d435i_recs/my_bag.bag -r 0.5

[BarzelS]

My next guess is that the processing speed on your device is not quite fast enough to process the D435i, leading to a backlog of image frames in the input queues. Are you experiencing any time delay between your input images and Kimera's output? This is pretty easy to check by viewing the RealSense image alongside the debug image /kimera_vio_ros/feature_tracks/image_raw.

1. I'm not experiencing any time delay between my input and the kimera output, the debug image appears pretty stable.

BTW: I turned on verbosity flag in the launch file and I got this output:

----------- # Hz (avg +- std ) [min,max]
Data Provider [ms] 0
IMU Preintegration Timing [us] 4434 30.0094 (27.7161 +- 6.52376) [20,179]
VioBackEnd [ms] 734 4.9707 (1.73978 +- 0.4) [0,10]
VioFrontEnd [ms] 4434 29.9895 (10.5474 +- 15.1792) [3,80]
I0512 19:26:40.121872 10296 StereoVisionFrontEnd.cpp:83] ------------------- Processing frame k = 4436--------------------
W0512 19:26:40.121872 10229 PipelineModule.h:220] Module: Data Provider - No Input received.
I0512 19:26:40.121922 10296 StereoVisionFrontEnd.cpp:107] Current IMU Preintegration frequency: 344828 Hz. (29 us).
I0512 19:26:40.125460 10296 Tracker.cpp:123] Optical Flow Timing [ms]: 2
W0512 19:26:40.158215 10229 PipelineModule.h:220] Module: Data Provider - No Input received.
I0512 19:26:40.166760 10296 FeatureDetector.cpp:180] Number of points detected : 800
I0512 19:26:40.166776 10296 FeatureDetector.cpp:192] Need n corners: 87
I0512 19:26:40.166806 10296 NonMaximumSuppression.cpp:87] Running RangeTree: 4
I0512 19:26:40.167212 10296 NonMaximumSuppression.cpp:105] Non Maximum Suppression Timing [ms]: 0
I0512 19:26:40.173795 10296 FeatureDetector.cpp:236] Corner Sub Pixel Refinement Timing [ms]: 6
I0512 19:26:40.178676 10296 StereoVisionFrontEnd.cpp:146] Keyframe 4436 with: 791 smart measurements
I0512 19:26:40.178715 10296 ThreadsafeQueue.h:270] Queue with id: Frame Rate Frontend output ROS is getting full, size: 4426
I0512 19:26:40.178722 10296 ThreadsafeQueue.h:270] Queue with id: Keyframe Rate Frontend output ROS is getting full, size: 732
I0512 19:26:40.178730 10296 StereoVisionFrontEnd.cpp:83] ------------------- Processing frame k = 4437--------------------
I0512 19:26:40.178771 10296 StereoVisionFrontEnd.cpp:107] Current IMU Preintegration frequency: 285714 Hz. (35 us).
I0512 19:26:40.178915 10298 RegularVioBackEnd.cpp:212] Tracker has a LOW_DISPARITY status.
I0512 19:26:40.180821 10298 VioBackEnd.cpp:1181] Backend: Update IMU Bias.
I0512 19:26:40.180860 10298 ThreadsafeQueue.h:270] Queue with id: Backend output ROS is getting full, size: 732
I0512 19:26:40.182718 10296 Tracker.cpp:123] Optical Flow Timing [ms]: 3
I0512 19:26:40.184615 10296 ThreadsafeQueue.h:270] Queue with id: Frame Rate Frontend output ROS is getting full, size: 4427
I0512 19:26:40.192183 10296 StereoVisionFrontEnd.cpp:83] ------------------- Processing frame k = 4438--------------------

I saw in the other issue that you mentioned the bold lines, what does it mean?
What can I do in order to fix it?

[violetteavi]

! Wow! I'm pretty sure Backend output ROS is where Kimera-VIO puts keyframes that it has processed. If you have 732 frames in that queue, Kimera is running fast enough but you have a serious publishing backlog. I'm not sure how that would happen except if your publishing rate is super slow-- you did change it to 120Hz, right?

[BarzelS]

! Wow! I'm pretty sure Backend output ROS is where Kimera-VIO puts keyframes that it has processed. If you have 732 frames in that queue, Kimera is running fast enough but you have a serious publishing backlog. I'm not sure how that would happen except if your publishing rate is super slow-- you did change it to 120Hz, right?

Yes changed line 363 in src/Kimera-VIO-ROS/src/RosOnlineDataProvider.cpp to:

ros::Rate rate(120);

[BarzelS]

I tried valgrind and thats the result:

my_executable_heap_profiling.pdf

[violetteavi]

This confirms what I suspected-- 90% RAM usage is from the creation of the stereo image packet.

Pair this with the queue logs you posted, and I think the images are getting processed all the way through and just not published. Fortunately, the latest PR introduced asynchronous publishing from VIO callbacks. Can you pull VIO and VIO-ROS master, rebuild, and try it again?