Batch WAL operations in stream mode

[tsg]

Description

This is the implementation for the plan in #770

Before this PR, we were batching write operations against the target in snapshot mode, but not in stream mode. This adds batching also in stream mode, as a performance optimisation.

Related Issue(s)

• Closes WAL streaming batch writer is slow for high-churn tables during catch-up #769
• Relates to Add plan for coalescing operations during streaming #770

Type of Change

Please select the relevant option(s):

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
• 📚 Documentation update
• 🔧 Refactoring (no functional changes)
• ⚡ Performance improvement
• 🧪 Test coverage improvement
• 🔨 Build/CI changes
• 🧹 Code cleanup

Testing

• Unit tests added/updated
• Integration tests added/updated
• Manual testing performed
• All existing tests pass

Checklist

• Code follows project style guidelines
• Self-review completed
• Code is well-commented
• Documentation updated where necessary

[tsg]

This means we can still have the same type of issue with UPDATE ... WHERE ... statements that updates a lot of rows. Hopefully less common than the commit case.

[github-actions]

Merging this branch will increase overall coverage

Impacted Packages	Coverage Δ	🤖
github.com/xataio/pgstream/pkg/stream/integration	0.00% (ø)
github.com/xataio/pgstream/pkg/wal/processor/postgres	80.57% (+1.68%)	👍

---

Coverage by file

Changed files (no unit tests)

Changed File	Coverage Δ	Total	Covered	Missed	🤖
github.com/xataio/pgstream/pkg/wal/processor/postgres/instrumented_wal_adapter.go	0.00% (ø)	10 (+3)	0	10 (+3)
github.com/xataio/pgstream/pkg/wal/processor/postgres/postgres_batch_writer.go	74.80% (-4.47%)	127 (+45)	95 (+30)	32 (+15)	👎
github.com/xataio/pgstream/pkg/wal/processor/postgres/postgres_wal_adapter.go	73.91% (+13.91%)	46 (+16)	34 (+16)	12	🎉
github.com/xataio/pgstream/pkg/wal/processor/postgres/postgres_wal_dml_adapter_bulk.go	91.34% (+91.34%)	127 (+127)	116 (+116)	11 (+11)	🌟
github.com/xataio/pgstream/pkg/wal/processor/postgres/postgres_wal_message.go	85.71% (+85.71%)	14 (+14)	12 (+12)	2 (+2)	🌟
github.com/xataio/pgstream/pkg/wal/processor/postgres/postgres_writer.go	37.14% (ø)	35	13	22

Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code.

Changed unit test files

• github.com/xataio/pgstream/pkg/stream/integration/pg_pg_batch_coalesce_integration_test.go
• github.com/xataio/pgstream/pkg/wal/processor/postgres/helper_test.go
• github.com/xataio/pgstream/pkg/wal/processor/postgres/postgres_batch_writer_test.go
• github.com/xataio/pgstream/pkg/wal/processor/postgres/postgres_wal_adapter_test.go
• github.com/xataio/pgstream/pkg/wal/processor/postgres/postgres_wal_dml_adapter_bulk_test.go
• github.com/xataio/pgstream/pkg/wal/processor/postgres/postgres_wal_message_test.go