feat: Write transaction fragments directly to storage to reduce consumer memory footprint#3783

Merged

alco merged 33 commits intomainfrom

alco/write-txn-fragments-to-storage

Feb 26, 2026

Commits on Feb 23, 2026

Clarify the explanatory comment on Xid.after_snapshot?()
alco
committed
Add write_unit to Consumer.State to support direct fragment-to-storage streaming
alco
committed
Buffer transaction fragments as soon as received without trying to build a complete transaction first
alco
committed
Refactor whole transaction handling in Consumer since we know it can be at most 1 txn at a time
alco
committed
Write transaction fragments directly to storage when write_unit=txn_fragment
alco
committed
Clearer var naming
alco
committed
Implement new storage functions for direct txn fragment writing
alco
committed
Update PureFileStorage.WriteLoop to allow writing log lines and advancing txn boundary independently
alco
committed
Minor cleanup of PureFileStorage
alco
committed
In ShapeLogCollector make sure the FlushTracker is only invoked when processing txn fragments with commit
alco
committed
Include full-txn and txn-fragment storage calls in test assertions
alco
committed
Update shape consumer tests to cover direct fragment-to-storage streaming
alco
committed
Test new PureFileStorage functions
alco
committed
Add changeset
alco
committed
Add a timeout in Support.StorageTracer.collect_traced_calls()
alco
committed
Split notifying the materializer and clients of new changes into separate functions
alco
committed
Fix typo
alco
committed
Use commit fragment offset for txn_offset_mapping and client notifications

alco
and
claude
committed
Add supports_txn_fragment_streaming? callback to Storage behaviour

alco
and
claude
committed
Downgrade write_unit to :txn when storage lacks fragment streaming support

alco
and
claude
committed
Fix code formatting
alco
committed
Defer Materializer subscriber notifications until commit fragment

alco
and
claude
committed
Do not signal txn commit to storage if it there hasn't been any preceding writes for this txn
alco
committed
Add test for event ordering across uncommitted fragments

alco
and
claude
committed
Fix filtering of empty materializer events and incorrect assumption about ordering
alco
committed
fix(sync-service): Fix inconsistent return type in write_txn_fragment_to_storage

alco
and
claude
committed
fix(sync-service): Use shape-specific offset for fragment commit notifications

alco
and
claude
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Write transaction fragments directly to storage to reduce consumer memory footprint#3783

feat: Write transaction fragments directly to storage to reduce consumer memory footprint#3783
alco merged 33 commits intomainfrom
alco/write-txn-fragments-to-storage

Commits on Feb 23, 2026

Clarify the explanatory comment on Xid.after_snapshot?()

Add write_unit to Consumer.State to support direct fragment-to-storage streaming

Buffer transaction fragments as soon as received without trying to build a complete transaction first

Refactor whole transaction handling in Consumer since we know it can be at most 1 txn at a time

Write transaction fragments directly to storage when write_unit=txn_fragment

Clearer var naming

Implement new storage functions for direct txn fragment writing

Update PureFileStorage.WriteLoop to allow writing log lines and advancing txn boundary independently

Minor cleanup of PureFileStorage

In ShapeLogCollector make sure the FlushTracker is only invoked when processing txn fragments with commit

Include full-txn and txn-fragment storage calls in test assertions

Update shape consumer tests to cover direct fragment-to-storage streaming

Test new PureFileStorage functions

Add changeset

Add a timeout in Support.StorageTracer.collect_traced_calls()

Split notifying the materializer and clients of new changes into separate functions

Fix typo

Use commit fragment offset for txn_offset_mapping and client notifications

Add supports_txn_fragment_streaming? callback to Storage behaviour

Downgrade write_unit to :txn when storage lacks fragment streaming support

Fix code formatting

Defer Materializer subscriber notifications until commit fragment

Do not signal txn commit to storage if it there hasn't been any preceding writes for this txn

Add test for event ordering across uncommitted fragments

Fix filtering of empty materializer events and incorrect assumption about ordering

fix(sync-service): Fix inconsistent return type in write_txn_fragment_to_storage

fix(sync-service): Use shape-specific offset for fragment commit notifications

Commits on Feb 24, 2026

Initialize consumer with write_unit=txn when its shape is part of a shape-with-subqueries hierarchy

Pass only stack_id across ShapeCache calls

Enrich the config passed from ShapeCache to Consumer.start_link with subquery info

Document new Storage behaviour callbacks for fragment streaming in changeset

Add unit tests for write_unit initialization based on subquery shape config

Disable txn_fragment write_unit when subqueries are enabled for the stack