Projections on External Stores

[nvivo]

This is just for some to expose some ideas I have.

Currently, there are some examples to use the projection as an in-memory store. You can receive events and store in a List<T> or a DataTable for example. While this works for many scenarios, the original idea of the framework is to support persistence on external stores in an easy way, and that will probably require some other base classes to make things easier.

Once an initial SQL Server store is implemented, I plan to add something that can automatically persist on SQL tables using a generic statement that is fluent and kinda error-free.

A projection to store products on a table could be written as:

// WARNING: this is not implemented yet
public class ProductList : SqlProjection
{
    protected override string TableName => "Products";
    protected override string StreamIDColumn => "StreamID"; // default
    protected override string StreamSequenceColumn => "StreamSequence"; // default

    public ActiveProducts()
    {
        OnEvent<ProductCreated>(pe =>
        {
            var e = pe.DomainEvent;

            await Project(e.ProductId, new
            {
                e.Name,
                e.Description,
                e.Price
            });
        });

        OnEvent<ProductPriceChanged>(pe =>
        {
            var e = pe.DomainEvent;

            await Project(e.ProductId, new {
                Price = e.NewPrice;
            });
    }
}

The project method would be defined as:

// core implementation
protected Task Project(object key, Dictionary<string, object> data, string streamId, int streamSequence);

// uses stream id / sequence data from current event being processed
protected Task Project(object key, Dictionary<string, object> data);

// serializes "data" object into a dictionary using Json.NET
protected Task Project(object key, object data);

The Project method would work as both insert or update, and it looks like the easiest way to achieve that would be something like:

var affected = ExecuteNonQuery("update table (...) values (...)");

if (affected == 0)
    ExecuteNonQuery("insert into table (...) (...)");

A Delete method would probably be required as well.

Some things I'm considering:

• I think it's better if the user creates the tables beforehand. Projections can truncate the table during rebuild, but I believe the structure of tables shouldn't be changed while the application is running.
• The implementation could also try to insert first and update on some exception. It could also use some "key caching" mechanism to detect if a key was used before and decide the best approach. In any case, the logic should always work by inserting or updating a record.
• Need some thinking on how to handle errors like field lengths or wrong data types. Initially, I believe any projection errors should be logged by some instrumentation infrastructure in a way that is easy to identify, but it's not clear yet how that would work.
• The Project method could be called something else. Ex: Merge, InsertOrUpdate, ???
• A very similar class could be build for NoSql stores without issues with new fields and deep objects. In that case, probably each store would need it's own projection class.

If anyone have any thought on this, feel free to comment.

[DamianReeves]

As noted in the discussions, for this to work we would need to support a concept of a projection storage provider. Overall, I think it is a great idea.

[promontis]

Great idea! One quick question though... do you have some guidelines on when to use an external storage versus an in-memory representation of a projection?