Crash resilience

[rasmus]

If the application using EventFlow shuts down unexpectedly, e.g. due to a power failure, EventFlow will lose consistency between what is committed to the event store and what is applied to read models and subscribers as any in-memory updates will be lost.

While any commands published (passed to ICommandBus.PublishAsync) but hasn’t yet resulted in any committed events to the event store are also lost, they are considered out-of-scope for this issue, as EventFlow doesn’t lose consistency.

Possible solutions

• Transaction log: Any events committed to the event store could be written to a log and removed when read models and subscribers have been successfully updated
• others... ?

Mitigation

Steps can be made to minimize the impact that an unexpected shutdown has

• Always drain servers for activity, e.g. web requests, before doing a application shutdown or upgrade
• Use asynchronous subscribers, i.e., let Hangfire (or what ever scheduling you have configured for EventFlow) handle subscriber updates

[wgtmpeters]

My team is also looking into situation's like this and
I believe it would be pretty hard to ensure for every possible crash situation.
Would using asynchronous subscribers not loose the order in which the subscribers are called?
Right now we are sure that the ReadModel Subscribers have been called before any others.

Transaction log you mean that each subscriber has to ACK that it has processed the event?
And the DomainEventPublisher somehow retries after recovery for events that have not been processed by all registered Subscribers?

Has this never been an issue in production @rasmus ?

[rasmus]

@wgtmpeters We haven't detected any abnormalities in our production setup that could be traced back to this. However, our production setup is rather stable and not prone to unexpected host crashes or application termination and we carefully drain all servers of web requests before upgrading.

I do think the issue is important, as it will allow me to sleep better at night 😄 (and it makes sense)

For the transaction log, I thought merely to write one entry per committed batch of events and when the DomainEventPublisher finishes successfully, then remove the entry. In case of a crash, the events could be picked up and retied. Events would be pushed to read models and subscribers more than once in case of a crash, which would need to be handled.

Even-though our servers in production is stable, our network infrastructure isn't and in addition we use RabbitMQ quite a lot. This means that our applications are (usually) written to handle a "at least once event", which is what I thought could be done for read models and subscribers.

[wgtmpeters]

Persistence of the event and the transaction log would then to have be in the same logical commit right?

And you would need to extend the existing eventstore implementations to also be able to handle this. Could be as simple as a flag indicator per event if It has been published.

But how to handle this the eventstore eventstore package?
As a side Q: Do you consider the Eventstore. Eventstore package production ready because er are looking at using it.

[wgtmpeters]

Are there still plans to do something with this issue? We are going to create something for this as well. When we have a concept ready I wil update this issue. We will try and come up with a solution that fits into eventflow core

[rasmus]

@wgtmpeters Currently no plans, any suggestions and/or PRs would be greatly appreciated

[bedej]

@wgtmpeters @rasmus Any progress on a fix for this? I'm currently trying to pick a CQRS+ES framework, and while this one is pretty feature rich and actively developed, this particular issue is a scary one for me.

If using MSSQL (which I am), would wrapping each command in a transaction scope ensure the read/write models stay consistent? Not sure of the viability of this given the code structure, but the possibility is mentioned here in the docs (with a recommendation not to do it!) https://eventflow.readthedocs.io/FAQ.html

[wgtmpeters]

we are working on it internally but have it is not ready yet.
At the moment we are working on being able to recover from crashing anywhere within the process of the DomainEventPublisher.
So each handle (readmodel update, subscriber update, saga update) will be a small transaction which is flagged as handled somewhere for each DomainEvent. That way when processing stops somewhere some other instance of your service can pick up where it stopped processing.
For now we see no use for wrapping the commands because those already have a retry mechanisme depending on how you send them.

For us it is either based on a event from kafka, which will retry when the command fails because the commit on the kafka message is not done.
Or the command is coming from some subscriber (which is already in a transaction as mentioned before) , so that will also be retried eventually.

Some things to consider in this way of working is that you should be able to handle a single command being sent multiple times.

[rasmus]

@wgtmpeters will it be possible to map that functionality to the core package or is it a new DomainEventPublisher that's installed?

[wgtmpeters]

@rasmus so far it seems more likely that we have to replace the existing one.
It also is not only in the DomainEventPublisher but also the IDispatchToSagas and IDispatchToSubscribers implementation.

We will first create a fully working POC and then we will see if we can clean it up.

[sitepodmatt]

Catch up subscriptions solution is quite elegant
https://eventstore.org/blog/20130306/getting-started-part-3-subscriptions/

[rasmus]

@sitepodmatt we still need some mechanism for stamping the domain events that have been successfully handled by subscribers and as the handling is asynchronous, the completion might not be trivial. E.g. event 1, 3, might have been successfully been handled, but event 2 took longer and a crash happen in the mean time.