feat(consumoor): add kafka-to-clickhouse consumoor pipeline#747
Open
feat(consumoor): add kafka-to-clickhouse consumoor pipeline#747
Conversation
…inator Replaces at-most-once Kafka offset handling with a centralized commit coordinator that defers offset commits until ClickHouse writes succeed. - Add FlushAll to Writer interface for coordinator-driven flushes - Blocking Write() propagates CH backpressure to Kafka consumption - Non-destructive flush: batches preserved on CH failure for retry - Disable Kafka auto-commit; coordinator owns flush+mark+commit cycle - Add commitInterval config (default 5s) for commit cycle timing - Add commits_total, commit_errors_total, flush_all_duration_seconds metrics
Cover happy path, empty pending no-op, FlushAll failure with retry, failure ordering preservation, multi-topic/partition highwater, Stop final flush, and interval-triggered commits.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new
xatu consumoorcommand and service implementation underpkg/consumoorthat consumes KafkaDecoratedEventmessages and writes flattened rows directly to ClickHouse.Registers flattener implementations for beacon, canonical, execution, consensus, MEV relay, node record, and libp2p events, including conditional routing and validator fan-out behavior.
Includes consumoor config, metrics, metadata extraction, and routing/writing infrastructure.
Changes
consumoorCLI command, config loading/overrides, and an example config file.