From ed3031b825039f60e1ac63b113fd5f3d59f5d5c0 Mon Sep 17 00:00:00 2001 From: Michel Hidalgo Date: Tue, 24 Feb 2026 01:23:44 -0300 Subject: [PATCH] Address peer review comments Signed-off-by: Michel Hidalgo --- _posts/rep-0157:2026.md | 138 ++++++++++++++++++++++++++-------------- 1 file changed, 90 insertions(+), 48 deletions(-) diff --git a/_posts/rep-0157:2026.md b/_posts/rep-0157:2026.md index 01b8a92..3e1cb62 100644 --- a/_posts/rep-0157:2026.md +++ b/_posts/rep-0157:2026.md @@ -39,9 +39,11 @@ Adoption therefore remains low, all the while `rclcpp` intra-process communicati The proposed re-design hinges on one central idea: `rosidl` messages as language-specific views to language-agnostic contiguous memory layouts, bounded on write (ie. publish), unbounded on read (ie. subscription). -Three (3) abstractions are introduced to generated code for `rosidl` messages (including request-response message pairs for services): views, storage, and shapes. +## Messaging changes -## Views +Three (3) abstractions are introduced to generated code for `rosidl` messages (including request-response message pairs for services): views, storage, and constraints. + +### Views Language-specific data interfaces or _views_ rather than data structures. Instead of relying on language-specific memory management and layout, these data views provide semantically meaningful, idiomatic access to otherwise raw memory. @@ -51,12 +53,12 @@ Data views may own the memory they point to (e.g. when default constructed) or s ![Message view over non-contiguous (e.g. heap allocated) storage](/assets/reps/rep-0157/non_contiguous_view.png) -## Storage +### Storage For each message member, the underlying blob of memory is wrapped by a `rosidl_memory_t` data structure: ```c -typedef struct rosidl_memory { +typedef struct rosidl_memory_s { void *address; int attributes; } rosidl_memory_t; @@ -88,64 +90,93 @@ struct sensor_msgs::msg::Image::Storage { }; ``` -## Shapes +### Constraints -To help data storage allocation when memory is externally managed, language-specific shape data structures matching their corresponding message layout are introduced. +To help data storage allocation when memory is externally managed, language-specific constraint data structures matching their corresponding message layout are introduced. These data structures constrain variable-length members to a given size and thus partially characterize the memory footprint of the message (as fixed-size members' contribution is implied and the underlying implementation may still pad and align as need be). A `size_t` value is used for each string and POD sequence member, whereas sequences of `size_t` values are used for sequence of string members (and eventually tensor members). -For message members and sequences thereof, the corresponding shape data structures are used in liue of `size_t`. +For message members and sequences thereof, the corresponding constraints are used in lieu of `size_t`. E.g.: ```c++ -typedef struct rosidl_sized { - size_t size; // as an aggregate for better readability +typedef struct rosidl_sized_s { + size_t size; // as an aggregate for better readability } rosidl_sized_t; -struct sensor_msgs::msg::Image::Shape { - std_msgs::msg::Header::Shape header; +struct sensor_msgs::msg::Image::Constraints { + std_msgs::msg::Header::Constraints header; + rosidl_sized_t encoding; rosidl_sized_t data; }; ``` -In this re-design, message shapes are not communicated explicitly. +In this re-design, message constraints are not communicated explicitly. Sizes of variable-length members must be encoded within the chosen in-memory representation. -A binary compatible and descriptive serialization format is a valid option and the chosen one for the reference implementation put associated to this REP. +A binary compatible and descriptive serialization format is a valid option for such an in-memory representation and the reference implementation associated to this REP goes down this path. ## Middleware changes -One (1) change is introduced to `rmw` APIs. -Message shapes may be provided to publishers and subscriptions on construction, through options. -Implementations can then optimize (and optionally check) for bounded messages, upon loan or else: +Two (2) changes are introduced to `rmw` APIs. + +Message constraints may be provided to publishers and subscriptions on construction, through options: ```c typedef struct RMW_PUBLIC_TYPE rmw_subscription_options_s { // ... - void * message_shape; // defaults to NULL + void * message_constraints; // defaults to NULL // ... } rmw_subscription_options_t; typedef struct RMW_PUBLIC_TYPE rmw_publisher_options_s { // ... - void * message_shape; // defaults to NULL + void * message_constraints; // defaults to NULL // ... } rmw_publisher_options_t; ``` -`rmw` implementations may then use this additional information to optimize message transport. -TBD: should device specifics be communicated at this point too? +These constraints apply to all messages affected by the corresponding publisher or subscription. `rmw` implementations can use this information to optimize (and potentially check) for bounded messages, in terms of data transport and resource allocation. -# Rationale +Message constraints may also be provided to publishers and subscriptions upon message loan: + +```c +rmw_ret_t +rmw_borrow_loaned_message_with_constraints( + const rmw_publisher_t * publisher, + const rosidl_message_type_support_t * type_support, + const void * message_constraints, + void ** ros_message); + +rmw_ret_t +rmw_take_loaned_message_with_constraints( + const rmw_subscription_t * subscription, + const void * message_constraints, + void ** loaned_message, + bool * taken, + rmw_message_info_t * message_info); +``` + +These constraints take precedence over publisher and subscription wide constraints, if any. +`rmw` implementations may reject such loan specific constraints if in violation with the latter, effectively introducing a bounded form of dynamically sized messages. + +# Remarks + +Language-specific views, storage, and constraints necessarily imply language-specific type support code. +While choosing a single language to implement reusable type support remains an option (an option exercised by the `rosidl_generator_py` package, wrapping C messages and type support) it is hypothesized that language-specific implementations will result in simpler generated (and generating) code. +As an example, consider native message views in C, C++, and Python, with type support code in C, C++, and either CPython aware C++ or thinly wrapped Python, respectively. +Each code path is free to make the most adequate choice for the corresponding runtime. -Middlewares that feature zero-copy data transport can implement message loaning APIs and rely on message type support to determine the size of the allocation and delegate message construction. +# Rationale -ROS nodes that publish data can leverage these APIs by specifying the shape of the message they intend to publish. -This is reasonable in many cases e.g. for sensor drivers with preconfigured resolution. +Middlewares that feature zero-copy data transport often implement message loaning APIs and rely on message type support to determine the size of the allocation and delegate message construction. +This design allows ROS nodes that publish data to fully leverage these APIs by specifying constraints for the messages they intend to publish. +This is reasonable in many cases e.g. for sensor drivers with a finite set of resolution settings, often preconfigured to a single, fixed setting on bring up. By constraining the size of variable-length members, memory locality can be achieved. -ROS nodes that subscribe data need not know about any shape, as the resulting memory layout remains structurally consistent with the message layout. -Furthermore, since the underlying memory layout can stay the same regardless of the nature of the view, this design affords cross-language zero-copy data transport by construction. +At the same time, ROS nodes that subscribe data need not know about constraints, as the resulting memory layout remains structurally consistent with the message layout. +Moreover, since the underlying memory layout can stay the same regardless of the nature of the view, this design affords cross-language zero-copy data transport out of the box, -Language-specific views, storage, and shapes imply language-specific type support code. -While choosing one language for a reusable implementation that can be bound by the rest remains an option -- an option exercised by the `rosidl_generator_py` package -- it is hypothesized that fully decoupling these abstractions just above the serialization format will result in simpler generated (and generating) code. +As a second order effect, constraints also communicate resource needs, effectively enabling preallocation. +Given message types and constraints, as well as QoS profiles, a middleware can reason about resource requirements for both publishers and subscriptions, allocate those resources early on, and spare the corresponding on-deterministic calls during steady state operation. +This is a staple of real-time systems. # Backwards Compatibility @@ -171,11 +202,11 @@ public: LoanedImagePublisher() : Node("loaned_image_publisher") { - sensor_msgs::msg::Image::Shape vga_image_shape; - vga_image_shape.encoding.size = 4; - vga_image_shape.data.size = 640 * 480; + sensor_msgs::msg::Image::constraints vga_image_constraints; + vga_image_constraints.encoding.size = 4; + vga_image_constraints.data.size = 640 * 480; rclcpp::PublisherOptions options; - options.shape = &vga_image_shape; + options.constraints = &vga_image_constraints; publisher_ = this->create_publisher("image", 10, options); timer_ = this->create_wall_timer( std::chrono::milliseconds(100), @@ -253,13 +284,13 @@ For the reference implementation, [XCDRv1](https://www.omg.org/spec/DDS-XTypes/1 - It ensures over-the-wire compatibility. At the time of writing of this REP, most RMW implementations, including all Tier 1 RMW implementations, use XCDRv1 as serialization format. Messages exchanged by processes using the proposed re-design and the original design will be mutually intelligible. - TBD: should we be using XCDRv2? Append-only message extension would be easier to support. - It is feature complete. The IDL specification that XCDRv1 was designed to support is a superset of that of `rosidl`, including mechanisms for messages to evolve over time. TBD is chosen as the target middleware, featuring both shared-memory and network transports to put the messaging system to test in relevant operating conditions. -Message runtime APIs are devised so as to be functionally equivalent (or approximately so) to those of the original `rosidl` design. Member-based access is kept, relying on language-specific forms of the data descriptor pattern. +Message runtime APIs are devised so as to be functionally equivalent (or approximately so) to those of the original `rosidl` design. +Method-based access is introduced and member-based access is kept, relying on language-specific forms of the data descriptor pattern. This is trivial in Python, where `@property` is a thing, but a bit less so in C++: ```c++ @@ -268,7 +299,7 @@ struct rosidl_runtime_cpp::Property { // ... operator T() const { - return *reinterpret_cast(buffer->data); + return *reinterpret_cast(buffer->data); } T& operator=(const T& value) { @@ -277,6 +308,10 @@ struct rosidl_runtime_cpp::Property { return storage; } + const T& operator() const { return *reinterpret_cast(buffer->data); } + + T& operator() { return *reinterpret_cast(buffer->data); } + private: rosidl_memory_t buffer; }; @@ -284,30 +319,37 @@ struct rosidl_runtime_cpp::Property { struct sensor_msgs::msg::Image { std_msgs::msg::Header header; rosidl_runtime_cpp::Property width; + // ... }; sensor_msgs::msg::Image message; +// member-based access message.width = 640; +// method-based access +message.width() = 640; ``` and significantly harder in C: ```c -struct rosidl_runtime_c__uint32_property_t { +typedef struct rosidl_runtime_c__uint32_property_s { uint32_t value; -}; +} rosidl_runtime_c__uint32_property_t; -struct sensor_msgs__msg__Image { +typedef struct sensor_msgs__msg__Image_s { std_msgs__msg__Header header; rosidl_runtime_c__uint32_property_t *width; // ... /*implementation-defined*/ __impl; -}; +} sensor_msgs__msg__Image; sensor_msgs__msg__Image message; sensor_msgs__msg__Image_init(&message); +// member-based access message.width->value = 640; +// method-based access +sensor_msgs__msg__Image__set_width(&message, 640); ``` where `__impl` may be a type-erased reference or a nested `struct` where `rosidl_memory_t` instances may be stored and handled by the associated message functions. @@ -319,11 +361,11 @@ C++ is not, and thus `std::array`, `std::vector`, `std::string`, and `std::wstri Deviating from standard practice, message type support APIs are cast into a uniform set through virtual tables, allowing for message size computations and construction and casting in place as well as standard (de)serialization in any implementation: ```c -/// Computes message size given its type and shape. +/// Computes message size given its type and some constraints. rcutils_ret_t rosidl_typesupport_get_expected_message_size( rosidl_message_type_support_t * type_support, - void * shape, size_t * size); + void * constraints, size_t * size); /// Computes a given message size. rcutils_ret_t @@ -331,22 +373,22 @@ rosidl_typesupport_get_message_size( rosidl_message_type_support_t * type_support, void * message, size_t * size); -/// Constructs a message of a given type and shape at the given storage. +/// Constructs a message given its type and some constraints at a given storage. /** * Can be understood as pre-serialization procedure. * Storage size is assumed to be adequate, as returned by * rosidl_typesupport_get_expected_message_size() for the - * same type and shape. + * same type and constraints. * Message members are default initialized. * Message lifetime is to be managed by the caller. */ rcutils_ret_t rosidl_typesupport_construct_message_at( rosidl_memory_t * storage, - rosidl_message_type_support_t * type_support, - void * shape, void ** message); + rosidl_message_type_support_t * type_support, + void * constraints, void ** message); -/// Casts a memory blobin storage into a message of a given type. +/// Casts a memory blob in storage into a message of a given type. /** * Can be understood as a zero-copy deserialization procedure. * Variable-length message members details are retrieved from the blob. @@ -356,7 +398,7 @@ rosidl_typesupport_construct_message_at( rcutils_ret_t rosidl_typesupport_cast_message_at( rosidl_memory_t * storage, - rosidl_message_type_support_t * type_support, + rosidl_message_type_support_t * type_support, void ** message); /// Deserializes a message of a given type from storage. @@ -368,7 +410,7 @@ rosidl_typesupport_cast_message_at( rcutils_ret_t rosidl_typesupport_deserialize_message_from( rosidl_memory_t * storage, - rosidl_message_type_support_t * type_support, + rosidl_message_type_support_t * type_support, void ** message); /// Serializes a message of a given type into storage.