-
Notifications
You must be signed in to change notification settings - Fork 9
Conflation queues
Peak loads of 10K or 100K messages per second are not unusual in high frequency applications such as trading systems. If the consumer of the messages cannot keep up during such peak loads, input queues run full and the producer is blocked or drops messages in the best case.
Even if the consumer survives, blocking the producer or dropping messages is a bad idea because the most recent (and most important) messages are lost. A better approach is to drop old messages that have now become redundant and replace them with newer more up-to-date information. Of course this strategy can only be applied to certain types of messages such as price updates. For other message types like order updates and trades, message loss must be prevented at all costs. Fortunately, price updates usually occur in much higher numbers and frequencies compared to orders and trades.
Conflation queues efficiently solve this problem. Old messages in the queue are replaced when a new message of the same kind arrives before the old one is consumed. A producer enqueues messages by providing a conflation key along with the message. In the case of price updates, the ticker of a stock or a currency pair symbol can be used as conflation key. Existing messages with the same key already residing in the queue will be conflated, which means replaced in the simplest case. More complex conflation queue variants support merging of the old and new message when conflation occurs.
It is important to note that an entry preserves its position in the queue when it is conflated. This guarantees that the message will eventually be consumed. It is also important to understand that under normal load, conflation rarely occurs and all messages are processed by the consumer. Messages are only conflated in peak load situations to protect the system and to guarantee that the application still operates in an orderly fashion.
The nobark library implements different variants of conflation queues. All variants are efficient, lock free and zero garbage and hence suitable for low latency applications. All queues can operate in single or multi producer/consumer mode --- implemented by a normal (non-conflating) backing queue. For best performance we recommend queues from the agrona library as backing queues.
See this article published on linkedin.