Other EventStore implementations besides DBAL, e.g. RabbitMQ Streams

[robinlehrmann]

Hi,

I have found out that RabbitMQ has a new feature called RabbitMQ Streams:

https://www.cloudamqp.com/blog/rabbitmq-streams-and-replay-features-part-1-when-to-use-rabbitmq-streams.html
https://www.rabbitmq.com/docs/streams

Those streams are persistent are they feels like a good fit for an alternative solutions to a classic database stream store apporach.

Do you think it would make sense to provide a Store implementation for that?

I know I can built this by my self with the \Patchlevel\EventSourcing\Store\StreamStore. But I thought I already share the idea with you to hear you opinion if this would be a good idea :)

[DanielBadura]

It could be that it is possible to use it as an event store, not sure though. In general it sounds like it: Append only log, replays etc. but we need some query capabilities which I don't know if rabbitmq streams are providing them. For example query only events for a specific aggregate, or only some events. Also, a transactional commit is needed which I believe is not given.

So it might sound on the first glance as this might be an option, but I think some pieces are missing to use it as an event store.
Is this just an interesting topic for you or do you think about moving the event store somewhere else? If so, why do you want to migrate the store? And why RabbitMQ, because it's already running in the stack?

[robinlehrmann]

@DanielBadura

Is this just an interesting topic for you or do you think about moving the event store somewhere else?

Yes, it was just an interesting topic, if this would make sense because it felt like a good fit for the event store at first glance.

But with those functionalities:

but we need some query capabilities which I don't know if rabbitmq streams are providing them. For example query only events for a specific aggregate, or only some events. Also, a transactional commit is needed which I believe is not given.

I think the database still might be the better option here.

If so, why do you want to migrate the store? And why RabbitMQ, because it's already running in the stack?

I don't really have plans to migrate it. I thought I share this with you, as it could be also interesting for you as well and yes, RabbitMQ is already in our stack.

[DanielBadura]

We already thought about adding more store option besides dbal, e.g. for kurrent which is designed for event sourcing. This would then most probably be released as an extra package. We are open to any contributions regarding other store's which make sense. We just must make sure, that all the features are working we currently have.

[HellPat]

A tool that many have already in their infrastructure is also Redis.

They describe the Event Sourcing Use-Case for Redis Streams in their official docs.

https://redis.io/docs/latest/develop/data-types/streams/

I don't need it and I did not test it, just wanted to leave the comment in case you're not aware of that.

[DavidBadura]

I somehow doubt that the Redis documentation is talking about DDD event sourcing when it says:

Event sourcing (e.g., tracking user actions, clicks, etc.)

Instead, it is more likely referring to “a stream used as a source” for things like statistics. Redis Streams is more of an alternative to Kafka. That is also why articles about Kafka and event sourcing are quite relevant here.

Requirements for an event sourcing database (event store):

• Complete history (possible forever)
• Optimistic concurrency control (to prevent concurrent writes to the same aggregate)
• Atomic append (either all events are written or none)
• Loading a stream (by aggregate ID, tags, from playhead, in the correct order)
• Rebuild capability (for projections)

Kafka and Redis are not designed for that. They serve a different use case: distributing infrastructure events to different services. If you ignore the requirement for optimistic concurrency control in particular, then you can also use these systems as an event store for event sourcing.

(Relational) databases, on the other hand, provide all of the requirements above and much more, which is why they are commonly used as event stores for event sourcing.