Open
Conversation
Add HorizonType enum and HorizonLocation message to coordinator.proto for tracking HEAD and FILL slot positions in the Horizon module. Includes RPC methods GetHorizonLocation and UpsertHorizonLocation.
…ions Implement server-side handlers for GetHorizonLocation and UpsertHorizonLocation RPC methods in the coordinator service. - Add horizon_location PostgreSQL table migration - Create horizon persistence package with Location struct - Add Marshal/Unmarshal methods for proto conversion - Implement GetHorizonLocationByNetworkIDAndType query - Add UpsertHorizonLocation with ON CONFLICT handling
- Create pkg/cldata/beacon.go with BeaconClient interface for block fetching - Create shared BeaconBlockDeriver in pkg/cldata/deriver/beacon_block.go - Refactor to use Iterator and ContextProvider interfaces - Add adapters in pkg/cannon/deriver/beacon/eth/v2/adapters.go: - BeaconClientAdapter wraps BeaconNode for cldata.BeaconClient - IteratorAdapter wraps BackfillingCheckpoint for cldata/iterator.Iterator - ContextProviderAdapter wraps metadata for cldata.ContextProvider - Update cannon.go to use shared BeaconBlockDeriver with adapters - Update Position.LookAheads to LookAheadEpochs for epoch-based preloading
Create shared ProposerSlashingDeriver in pkg/cldata/deriver using Iterator, BeaconClient, and ContextProvider interfaces. Update Cannon to use the shared implementation with adapters for code reuse between Cannon and upcoming Horizon module.
Remove v1/ and v2/ directories now that all derivers are in pkg/cldata/: - Remove pkg/cannon/deriver/beacon/eth/v1/ (beacon_blob, beacon_committee, beacon_validators, proposer_duty) - Remove pkg/cannon/deriver/beacon/eth/v2/ (attester_slashing, beacon_block, bls_to_execution_change, deposit, elaborated_attestation, execution_transaction, proposer_slashing, voluntary_exit, withdrawal) - Move adapters.go to pkg/cannon/deriver/adapters.go (used to bridge Cannon types to shared cldata interfaces) - Update event_deriver.go to reference only cldata shared derivers - Simplify config.go with unified DeriverConfig type This cleanup removes ~4,900 lines of code that was duplicated in the shared pkg/cldata package.
- Create pkg/horizon/ethereum/ directory with BeaconNodePool implementation - BeaconNodeConfig accepts name, address, and optional headers per node - BeaconNodePool manages multiple beacon nodes with health checking - Health checks run at configurable interval using ethpandaops/beacon library - Metrics track connection status per node (xatu_horizon_beacon_node_status) - Additional metrics: blocks fetched, cache hits/misses, fetch errors - Shared services (metadata, duties) initialized from first healthy node - Block cache shared across all nodes with singleflight deduplication - Config and override support for beacon node URLs and headers
Create pkg/horizon/subscription/block.go with BlockSubscription struct that: - Subscribes to /eth/v1/events?topics=block SSE stream on each beacon node - Leverages ethpandaops/beacon library for SSE connection management and reconnection - Parses block event payload (slot, block root, execution_optimistic flag) - Emits parsed BlockEvent to channel for processing - Tracks SSE metrics: events_total, connection_status, reconnects, processing_delay
Add TTL-based deduplication cache for block events by block root.
Add coordinator to manage HEAD and FILL iterators running in separate goroutines without blocking each other: - HEAD iterator has priority and processes real-time SSE block events - FILL iterator runs independently for consistency catch-up - Both iterators skip slots already processed by the other - HEAD checks both head_slot and fill_slot before processing - FILL checks both fill_slot and head_slot before processing - Separate location markers in coordinator (head_slot, fill_slot)
Add validation to ensure at least one output sink is configured. All other acceptance criteria validations were already implemented: - At least one beacon node URL (ethereum/config.go) - Coordinator address (coordinator/config.go) - LAG distance positive (iterator/fill.go - sets default) - TTL positive duration (cache/dedup.go - sets default) - Clear error messages for all validation failures
- Add xatu-horizon service to docker-compose.yml with "horizon" profile - Create xatu-horizon.yaml config file for local development - Add Horizon output handler to xatu-server.yaml to route events to ClickHouse - Horizon connects to local beacon node(s) and sends events to xatu-server
Add Kurtosis configuration for E2E testing Horizon with all consensus clients: - horizon-test.yaml: ethereum-package config with Lighthouse, Prysm, Teku, Lodestar, Nimbus, Grandine - xatu-horizon.yaml: Horizon config for connecting to all 6 beacon nodes - xatu-server.yaml: xatu-server config for E2E test event routing - README.md: Documentation for running E2E tests with validation queries
Add automated E2E test script for Horizon module: - scripts/e2e-horizon-test.sh orchestrates full E2E test - Spins up Kurtosis network with all 6 consensus clients - Starts Xatu stack via docker-compose - Connects networks and generates Horizon config dynamically - Waits for data collection (~2 epochs by default) - Runs validation queries against ClickHouse - Reports pass/fail status and cleans up resources Options: - --quick: Run 1 epoch instead of 2 (~8 min vs ~15 min) - --skip-build: Use existing Docker image - --skip-cleanup: Keep resources for debugging Also updated deploy/kurtosis/README.md with: - Quick start section for automated testing - Manual test procedure for step-by-step execution
Add comprehensive SQL validation queries for Horizon E2E testing: - scripts/e2e-horizon-validate.sql with 8 validation queries - Document expected results in deploy/kurtosis/README.md Queries cover: - Duplicate block detection (deduplication validation) - Slot gap analysis (FILL iterator validation) - Module name verification (HORIZON events only) - Events per deriver type (all 13 derivers) - Slot coverage summary - Block latency analysis - Multi-node event attribution - Recent blocks sanity check - Validation summary for automated pass/fail
…terators This commit introduces several key features: 1. Adds `HORIZON_XATU_COORDINATOR_AUTHORIZATION` override for setting coordinator authorization secrets. 2. Implements a `BlockEventBroadcaster` to deduplicate block events before distribution. 3. Refactors block event consumption to use a `DualIterator` which multiplexes HEAD and FILL logic, replacing direct HeadIterator usage. 4. Adds configuration for iterator settings and a startup timeout for beacon nodes in `horizon.yaml`. 5. Adds reorg tracking and rollback logic for block-based derivers to maintain consistency after chain reorganizations. 6. Updates E2E tests to configure Kurtosis devnet with custom genesis time and disables future forks for stability.
Add the admin database and cryo table for tracking cryo extraction progress. This matches the platform repo's migration 049.
Replace verbose individual deriver implementations with a declarative registry pattern that eliminates boilerplate code. Changes: - Add registry.go for declarative deriver specifications - Add generic.go as universal deriver handling run loop, backoff, tracing - Add event_builder.go for common event creation logic - Add factory.go for creating derivers from registry - Add extractors/ package with 13 focused extractor functions - Update horizon.go to use factory pattern (~110 -> 35 lines) - Update cannon.go to use factory pattern (~280 -> 55 lines) - Remove 13 verbose deriver files (~5500 lines deleted) Code reduction: ~75-80% for simple derivers, ~60-70% for complex ones.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Horizon is a
HEADversion ofCannon