A way to consume multiple messages as a single collection (e.g. List<T>) #887
Replies: 2 comments 1 reply
-
|
I converted the original issue into a conversation. A new issue can be created if we come up with more precise actionable items. The current API provides a good isolation from the underlying protocol, as messages arrive in a "chunk", within 1 delivery frame. A chunk can contain 1 to several thousands messages, depending on the ingress. We could choose to pack all the messages of a chunk in a list, so the batch size would vary. To enforce a setting like a batch size, the list of messages would need to span several chunks in some cases.
I don't think making it "closer to the network" would make a big difference, except making the code base more complex. Obviously we would trade latency for throughput with this approach. A first prototype could be implemented as a regular No users requested this feature in over 4 years, but we would consider contributions if they are aligned with the current library design. |
Beta Was this translation helpful? Give feedback.
-
|
I would like to provide a concrete use case to highlight why the batch pull functionality is critical for us. Use Case: Metering Service Throughput & Database IOPS Analysis
Conclusion: |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Is your feature request related to a problem? Please describe.
I am using RabbitMQ Stream for high-throughput scenarios, where the bottleneck often shifts from network I/O to the processing logic, specifically when interacting with downstream systems like databases (ClickHouse, Elasticsearch) or calling external APIs.
The current
ConsumerBuilderonly supports a single-message callback viamessageHandler(BiConsumer<Context, Message> messageHandler).This forces the application to process messages one by one. In scenarios where I need to perform bulk inserts or batch processing to optimize IOPS and transaction overhead, the current API is limiting.
Describe the solution you'd like
I would like to request native support for batch consumption in the Java Client.
Ideally, the
ConsumerBuildercould offer a mechanism to receive a batch of messages.For example, an API similar to this:
Describe alternatives you've considered
Currently, we have to implement a manual "Accumulator/Buffer" pattern on the client side:
Why this alternative is not ideal:
Additional context
Many other streaming clients (like Kafka's
pollreturning a generic collection, or Spring's batch listeners) support this out of the box. Since RabbitMQ Stream is designed for high throughput, adding batch consumption support would significantly enhance its capability in "Big Data" or "Data Pipeline" use cases.Beta Was this translation helpful? Give feedback.
All reactions