Feature request: log=livefanout

[oscni]

Is your feature request related to a problem? Please describe.
long lived shapes combined with lots of updated makes it almost unusable, a simpler solution than log compaction should be possible

Describe the solution you'd like
In general:

1. New clients connecting should always mean a new snapshot/sql query of the latest materialized view from the db
2. Keep a list of current offset of all clients. Remove logs with older offset than what the oldest client has
3. When TCP disconnects a client update the list of current offset of all clients and same as in step 2 remove logs with older offset than what the oldest client has

[balegas]

Thanks for reaching out.

What kind of data are you syncing and do you need the full history? Do these high-frequency updates need to go through Postgres, or are they more ephemeral/event-like (e.g. cursor positions, presence, live metrics)?

[oscni]

Can be basically any kind of user or application data, but not prescene such as cursor positions.
Don't need full history. Only latest materialized view + all future updates.

[balegas]

Most applications don't have that high amount of updates. So, practically, log compaction addresses any disk usage concerns. We have support for subset queries that allow you to get a snapshot and start tailing changes at snapshot time. It's fully supported in Tanstack DB

For high cardinality of events you can use durable streams as sync via the database is too heavy in any case.

Does this make sense?

[oscni]

Seems to work using subset queries without using an actual subset :) But I guess we still have the issue that the electric log will grow forever. Should i close this and create a new feature request to have retention policies like kafka. Time, Size, Compactation and my favorite oldest client offset would be some options.

Thanks for a great project by the way!