Memory "leak"

[acristoffers]

Description

We observed that after hours of operation, the node was killed by OOM-killer. I then tried to figure out where the problem is, and a MWE could reproduce it, showing it was not a fault in our code. I tried a simple node to see if the problem was in the RMW, but just a node does not reproduce the problem, only when the rosbag2_transport::Recorder class is used.

Expected Behavior

The node should free all memory it no longer uses.

Actual Behavior

The node keeps climbing on memory usage. Valgrind was not able to find the problem, but running a simple node we can replicate the issue.

To Reproduce

It may take a bit of retrying, but:

#include <rosbag2_cpp/writer.hpp>
#include <rosbag2_storage/storage_options.hpp>
#include <rosbag2_transport/record_options.hpp>
#include <rosbag2_transport/recorder.hpp>

using namespace std::chrono_literals;

static const rosbag2_storage::StorageOptions STORAGE_OPTIONS{
    .uri                    = "/tmp/bag_test",
    .storage_id             = "mcap",
    .max_bagfile_duration   = 10,
};

static const rosbag2_transport::RecordOptions RECORD_OPTIONS{
    .all_topics                   = true,
    .all_services                 = true,
    .rmw_serialization_format     = "cdr",
    .topic_polling_interval       = 100ms,
};

class MemSubNode : public rosbag2_transport::Recorder {
   public:
    explicit MemSubNode(const rclcpp::NodeOptions& options = rclcpp::NodeOptions())
        : rosbag2_transport::Recorder(std::make_shared<rosbag2_cpp::Writer>(),
                                      STORAGE_OPTIONS,
                                      RECORD_OPTIONS,
                                      "vb_mem_sub",
                                      options) {
        record();
    }
};

int main(int argc, char* argv[]) {
    rclcpp::init(argc, argv);
    rclcpp::spin(std::make_shared<MemSubNode>());
    rclcpp::shutdown();
}

pub_big.py:

import rclpy
import os
import sys
from rclpy.node import Node
from std_msgs.msg import ByteMultiArray

class BigPub(Node):
    def __init__(self):
        super().__init__("big_pub")
        self.pub = self.create_publisher(ByteMultiArray, "/big_topic", 10)
        for i in range(100):
            pub = self.create_publisher(ByteMultiArray, f"/many/t{i}", 1)
            msg = ByteMultiArray()
            msg.data = bytearray(os.urandom(16))
            pub.publish(msg)

        msg = ByteMultiArray()

        for _ in range(512):
            msg.data = bytearray(os.urandom(int(sys.argv[1]) * 1024**2))
            self.pub.publish(msg)
        self.get_logger().info("Published big message")
        exit(0)

rclpy.init()
rclpy.spin(BigPub())

pub_small.py:

import rclpy
import os
import sys
from rclpy.node import Node
from std_msgs.msg import ByteMultiArray

class BigPub(Node):
    def __init__(self):
        super().__init__("small_pub")
        self.pub = self.create_publisher(ByteMultiArray, f"/small_topic", 1)

        n = int(sys.argv[1])
        msg = ByteMultiArray()
        for _ in range(n):
            msg.data = bytearray(os.urandom(64))
            self.pub.publish(msg)
        self.get_logger().info("Published small message")
        exit(0)

rclpy.init()
rclpy.spin(BigPub())

Compile the node. I called it vb_mem_sub, and then in one terminal I run it and in another I watch its memory usage with (or you can just use [h]top):

while true; do
      ps --no-headers eo comm,rss,%mem --pid $(pidof vb_mem_sub)
      sleep 1
done

In another terminal, run python3 pub_small.py 1000000 and in another run python3 pub_big.py 10; python3 pub_big.py 1. You may need to stop and relaunch the python nodes a few times, but at some point you will get an unreasonable climb in memory. The increase will stop once both python nodes are killed, but the claimed memory will never be released.

System (please complete the following information)

• OS: Ubuntu 24.04.4 LTS
• ROS 2 Distro: Jazzy
• Install Method: APT
• Version: 0.26.9-1noble.20260126.180134

Additional context

The problem was identified in a robot running for about 10 hours. The scripts try to replicate the way I found to trigger the problem faster. I can reproduce it 100% of the time by launching the pub_small.py script, then launching our nodes and stopping them, which made me assume that the creation and deletion of topics would trigger the issue, which seems to be the case. Of course, in the real robot the climb is much slower and the script is meant to make it faster and more visible, but after ~10h it caused many of our robots to stop because of the OOM kill. With our entire system running, this way of triggering it makes the memory skyrocket very quickly. It seems like messages are not being released, but then Valgrind found no leak, so the memory seems to still be accessible, like a vector that was supposed to be cleared but never got or something like that.

[MichaelOrlov]

I spent some time reproducing and narrowing this down with the Codex Agent gpt-5.4.
Here is a summary of what we found.

Summary

We can reproduce the reported recorder memory growth, but the evidence so far does not point to a classic unreachable-memory leak inside rosbag2_transport::Recorder itself.

The strongest current conclusion is:

• the symptom is real runtime RSS growth, not just a spike at bag close
• it is not MCAP-specific
• it is not primarily caused by rosbag2 write cache
• it is not primarily caused by bag splitting
• it is dominated by large-message traffic
• it is RMW-dependent
• with Fast DDS, the main suspect is large fragmented message buffering in the Fast DDS receive path, especially shared-memory transport
• with Cyclone DDS, I could not reproduce the same staircase RSS growth

Reproducer shape

We used a workload close to the one described here:

• recorder:
  • storage.storage_id = mcap
  • record.all_topics = true
  • record.all_services = true
  • storage.max_bagfile_duration = 10 for split-on runs
  • also tested storage.max_bagfile_duration = 0
  • also tested storage.max_cache_size = 0
• publishers:
  • many small messages on /small_topic
  • large std_msgs/msg/ByteMultiArray bursts on /big_topic
  • optional transient /many/t* topic churn

We also reduced the scenario step by step to isolate the effect.

What we observed

1. This is not just a shutdown spike

A generic sustained workload did not show linear growth:

• recorder wrote about ~602 MB over ~30s
• recorder RSS stayed around ~40 MB

But the issue-shaped workload did show real in-process RSS increase while the recorder kept running.

2. Split enabled vs disabled

With split enabled (max_bagfile_duration=10), recorder RSS ratcheted upward over 5 cycles:

• baseline: ~36 MB
• cycle 1: ~228-229 MB
• cycle 2: ~229-232 MB
• cycle 3: ~231 MB
• cycle 4: ~240-241 MB
• cycle 5: ~251-252 MB

With split disabled (max_bagfile_duration=0), growth was worse:

• baseline: ~36 MB
• cycle 1: ~255 MB
• cycle 2: ~276 MB
• cycle 3: ~277 MB
• cycle 4: ~297 MB
• cycle 5: ~298 MB

Conclusion:

• split is not the primary cause
• this is not just a bag-close or split-close spike

3. MCAP vs sqlite3

Using the same workload with split disabled:

mcap

• cycle 1: ~255 MB
• cycle 2: ~276 MB
• cycle 3: ~277 MB
• cycle 4: ~297 MB
• cycle 5: ~298 MB

sqlite3

• cycle 1: ~254 MB
• cycle 2: ~311 MB
• cycle 3: ~326 MB
• cycle 4: ~408 MB
• cycle 5: ~398-408 MB
• transient peak: ~531 MB

Conclusion:

• this is not MCAP-specific
• sqlite3 was actually worse in this reproducer

4. all_topics=true makes it worse, but is not the whole problem

With mcap, split disabled, explicit recording of only [/small_topic,/big_topic]:

• cycle 1: ~223 MB
• cycle 2: ~236 MB
• cycle 3: ~250 MB
• cycle 4: ~244 MB
• cycle 5: ~253 MB

Compared to all_topics=true, this is better, but still clearly elevated.

Conclusion:

• topic discovery/churn overhead contributes
• but it is not the main root cause

5. Transient /many/t* topic churn contributes only modestly

With explicit topics fixed to [/small_topic,/big_topic] and removing transient /many/t* topics:

• cycle 1: ~215 MB
• cycle 2: ~229-230 MB
• cycle 3: ~226-227 MB
• cycle 4: ~240-241 MB
• cycle 5: ~238-239 MB

Conclusion:

• transient topic churn contributes some overhead
• but the core retained RSS still happens without it

6. Disabling rosbag2 cache does not fix it

With the same explicit-topic, no-churn setup, but storage.max_cache_size:=0:

• cycle 1: ~247 MB
• cycle 2: ~319 MB
• cycle 3: ~409 MB
• cycle 4: ~420 MB
• cycle 5: ~423 MB

Conclusion:

• rosbag2 write cache is not the primary cause
• disabling cache made this reproducer worse

7. Large messages are the key trigger

This was the clearest isolating test.

/small_topic only:

• RSS stayed around ~40-41 MB over 5 cycles

/big_topic only:

• cycle 1: ~262-263 MB
• cycle 2: ~344-345 MB
• cycle 3: ~354-356 MB
• cycle 4: ~355-356 MB
• cycle 5: ~385-386 MB
• transient peaks above 400 MB

Conclusion:

• the problem is dominated by the large-message path
• small messages alone did not reproduce the growth

AddressSanitizer / LeakSanitizer results

We built an ASan/LSan-instrumented recorder and ran the strongest reproducer:

• /big_topic only
• mcap
• no split
• storage.max_cache_size:=0

LeakSanitizer did not report anything resembling the hundreds of MB of observed RSS growth.

The only actual leak reported was tiny:

• 319 bytes leaked in 4 allocations

The stack pointed into generic ROS 2 type support / rcutils paths, not rosbag2 recorder write path.

There was also a lot of unrelated new-delete-type-mismatch noise from rcl / rcutils teardown under ASan, but nothing indicating a large recorder leak.

Conclusion:

• We did not find evidence of a classic unreachable heap leak in recorder matching the RSS growth

Heap profiling results

Heaptrack

We ran heaptrack, but its summary for this run was too collapsed to isolate the large-message owner cleanly.

Valgrind massif

massif was much more useful.

For the same /big_topic-only reproducer, peak heap was around:

• ~121 MB

The peak snapshot was dominated by:

• eprosima::fastrtps::rtps::TopicPayloadPool::do_get_payload(...)
• called via:
  • RTPSReader::reserveCache(...)
  • StatefulReader::processDataFragMsg(...)
  • MessageReceiver::process_data_fragment_message_without_security(...)
  • SharedMemChannelResource::perform_listen_operation(...)

This is a Fast DDS receive-side path for fragmented large messages, specifically involving shared-memory transport.

Also important: after that peak, massif showed heap dropping sharply, which is not what I would expect from a large unreachable-memory leak.

Conclusion:

• the peak allocations we found were in Fast DDS receive buffering, not rosbag2 writer code
  cc: @MiguelCompany @mjcarroll

RMW comparison

This was the strongest differentiator.

Fast DDS, SHM enabled

/big_topic only, no split, no cache:

• cycle 1: ~262-263 MB
• cycle 2: ~344-345 MB
• cycle 3: ~354-356 MB
• cycle 4: ~355-356 MB
• cycle 5: ~385-386 MB

Cyclone DDS

Same test, same recorder config:

• cycle 1: 118656 kB
• cycle 2: 118732 kB
• cycle 3: 118780 kB
• cycle 4: 118840 kB
• cycle 5: 118884 kB

This is essentially flat.

Fast DDS with SHM disabled

I reran Fast DDS with:

• RMW_IMPLEMENTATION=rmw_fastrtps_cpp
• FASTDDS_BUILTIN_TRANSPORTS=UDPv4

Same /big_topic-only, no split, no cache test:

• cycle 1: 189244 kB
• cycle 2: 189252 kB
• cycle 3: 189252 kB
• cycle 4: 189268 kB
• cycle 5: 189268 kB

This is also essentially flat.

Conclusion:

• the staircase growth reproduces with Fast DDS + SHM
• it does not reproduce the same way with Cyclone DDS
• it also does not reproduce the same way with Fast DDS SHM disabled
• this strongly points at Fast DDS shared-memory handling of large fragmented messages rather than rosbag2 Recorder ownership itself

Current conclusion

At this point I do not think the main problem is a classic memory leak in rosbag2_transport::Recorder.

The current evidence points much more strongly to:

• large fragmented /big_topic messages
• Fast DDS receive-side buffering / payload pool behavior
• especially Fast DDS shared memory transport
• plus possible allocator retention / fragmentation effects visible as recorder RSS staying high

In other words:

• rosbag2 Recorder is the workload that exposes the issue
• but the dominant retained memory I could attribute in profiling was in the middleware receive path, not the rosbag2 cache or writer code

Caveats

A couple of runs with Cyclone DDS and Fast DDS-without-SHM had slightly lower message counts than ideal, likely due to slower startup or delivery. But both still recorded large real bags and, importantly, both stayed flat in RSS instead of showing the Fast DDS SHM staircase.

Practical takeaway

If someone needs an immediate mitigation for this exact workload, the most effective knobs I found were:

• use rmw_cyclonedds_cpp, or
• keep Fast DDS but disable SHM transport

Those changes affected the memory behavior far more than:

• switching mcap vs sqlite3
• changing split on/off
• changing rosbag2 cache on/off
• reducing topic discovery churn

Suggested next step

This now looks like it may belong more to Fast DDS / rmw_fastrtps_cpp investigation than to rosbag2 Recorder itself.

The most useful next follow-up would be a minimal non-rosbag2 reproducer that:

• subscribes to the same large ByteMultiArray topic
• does little or nothing with the data
• compares Fast DDS SHM on/off and Cyclone DDS

If that shows the same staircase, it would confirm this is fundamentally middleware-side rather than rosbag2-side.

[MichaelOrlov]

cc: @MiguelCompany @mjcarroll This memory leak appears to be an issue in the Fast DDS SHM pass for large, fragmented messages.
Please see the comprehensive analysis above. ^^^

[MichaelOrlov]

• The minimal reproducer without Rosbag2 recorder can be found in the [don't merge!] min reproducer for fast dds shm memory leak #2378

[acristoffers]

Thank you for your response @MichaelOrlov, but I think we are seeing different issues as I use CycloneDDS without shared memory. After your response, I tried to find a way to create a better MWE to show what I see, but the problem is a bit random/not deterministic, so it's hard to create a perfect MWE.

What I have seen is that it requires the Recorder class subscribing to all topics (it never triggers if you have just one subscriber, no matter the load), and it also triggers more often if there are already many topics in the system, like having my simulator open or the entirety of our system running. I'm now convinced it is not a problem with this package, but a weird interplay between the class, the DDS (the problem is there with fastdds too) and the allocator.

I decided to investigate the allocator because, as I said, no tool reports any leak, so assuming the memory management inside all the code is correct, the allocator becomes a possible problem. And alas, changing the allocator solves the ever-growing memory problem. I tried MTAlloc, and with it there is no growth, but it gives the memory back really slowly. Then I tried jemalloc and during the burst it allocated 1.5GB memory which it never releases, but at least it does not seem to ever allocate again on subsequent bursts. 1.5GB is a lot, but if it stays constant it may be acceptable. I'm now trying mimalloc, and it is the best... except when not. Sometimes it allocates only ~500MB during burst and then releases it after. Sometimes it allocates 3GB and quickly releases it after. Sometimes it allocates 3GB and releases as slowly as MTAlloc, so I'm still experimenting with it.

I'll close the issue as there is nothing actionable on your side, but it's good that it exists so if someone has the same problem, they can at least try different allocators to see if this solves their problem too. If I find anything else of relevance during my testing I'll leave a comment here too.

Thank you again for your time.