Feat/persistent block archive

ARCHITECTURE.md

@@ -67,7 +67,7 @@ Pure CPU, no I/O. Takes raw protobuf bytes, returns structured types.
 - **Block header verification**: secp256k1 signature recovery, check the recovered witness address against the active SR set, validate the txTrieRoot. **On the dual-engine question**: java-tron has an `SM2` codepath (China's GM/T 0003 national standard) alongside `ECKey`/secp256k1, and from a distance one might assume both are in use. They are not. Investigation 2026-05-09 (java-tron #6588, PR #6627, `MiscConfig.java`): `isECKeyCryptoEngine` is hard-true on mainnet, no mainnet SR signs with SM2, and core devs have proposed deleting the unused module. lightcycle therefore implements only secp256k1 sigverify, and the codec's `WitnessAddressMismatch` branch represents a real bug or an attacker — not a benign engine mismatch.
 - **Transaction decoding**: TRON has 40+ contract types. The four high-volume ones (`Transfer`, `TransferAsset`/TRC-10, `TriggerSmartContract` — TRC-20/DEX/USDT all flow through here, `CreateSmartContract`) get fully-decoded payload variants on `DecodedContract`; everything else lands in `Other { kind, raw }` with the original `Any.value` bytes preserved so consumers can decode against the matching java-tron protobuf message themselves.
 - **TransactionInfo decoding**: the side channel java-tron exposes via `GetTransactionInfoByBlockNum` / `GetTransactionInfoById`. The block proto carries the request (signed tx); `TransactionInfo` carries the result (success/fail, energy/bandwidth, emitted logs, internal sub-calls). We surface logs raw, internal txs as `InternalTx`, and the energy/bandwidth breakdown as `ResourceCost` — consumers reconstructing token flow, computing TRX cost, or replaying internal calls don't need a second round-trip per tx.
-- **Event log decoding**: the universal token events (`TRC-20 Transfer`/`Approval`, `TRC-721 Transfer`) are recognized by topic-0 hash + topic-count and ship in v0.1 — they cover every standard token contract without any registry. Custom contract events (governance, oracles, anything not following the standard signatures) require an ABI registry and land behind a future `decode_event(log, abi)` entry point. ABI registry shape is pluggable: file-based, HTTP, or on-chain via `getContract`.
+- **Event log decoding**: two complementary surfaces. The universal token events (`TRC-20 Transfer`/`Approval`, `TRC-721 Transfer`) ship in `events.rs` and are recognized by topic-0 hash + topic-count alone — every standard token contract decodes without any setup. Arbitrary contract events go through `abi.rs`: operators register Solidity-style signatures (e.g. `"Swap(address indexed sender, uint256 amount0In, uint256 amount1In, uint256 amount0Out, uint256 amount1Out, address indexed to)"`); the registry computes `topic[0] = keccak256(canonical_signature)` and decodes matching logs into a structured `DecodedEvent` with positional + named param access. v0.1 supports the static-type subset (`address`, `bool`, `uintN`/`intN`, `bytesN`); dynamic types (`string`, `bytes`, `T[]`) and tuples surface a typed `AbiParseError::UnsupportedType` so consumers know the boundary.
 - **Address ergonomics**: `Address::to_base58check` / `from_base58check` for the human-facing `T...` form; raw 21-byte forms remain the wire/storage canonical.
 
 ### lightcycle-relayer
@@ -83,7 +83,7 @@ The orchestrator. Owns the canonical head pointer and the live block buffer.
 
 gRPC server speaking the [`sf.firehose.v2`](https://github.com/streamingfast/proto) protocol used by Substreams. Multiplexes one upstream block stream to many downstream consumers via a `tokio::sync::broadcast` hub — slow subscribers get `RESOURCE_EXHAUSTED` rather than back-pressuring the engine, which is correct semantics for a relayer.
 
-Three Firehose v2 services ship. **`Stream.Blocks`** runs live-tail by default; when an in-memory block cache is attached (the CLI wires one when `--firehose-listen` is set), requests with `start_block_num` or `cursor` walk the cache from the requested height to its tip and then transition into the live broadcast (with dedup so a block emitted from the cache isn't re-emitted from live). Backfill beyond the in-memory cache window requires a persistent block archive — that lands later. `stop_block_num`, `final_blocks_only`, and `transforms` are rejected with `FailedPrecondition`. **`Fetch.Block`** does point-in-time block lookup by `BlockNumber` reference; the service delegates to a `BlockOracle`. The CLI composes a `CachingBlockOracle` (read-through over the same in-memory cache the relayer feeds) wrapping a `GrpcBlockOracle` (dedicated `GrpcSource` connection — separate from the relayer's source so request-driven fetches don't serialize behind the live tail). `BlockHashAndNumber` and `Cursor` references plus `transforms` are rejected with `FailedPrecondition`. **`EndpointInfo.Info`** reports chain identity for orchestrator sanity-check.
+Three Firehose v2 services ship. **`Stream.Blocks`** runs live-tail by default; when an in-memory block cache is attached (the CLI wires one when `--firehose-listen` is set), requests with `start_block_num` or `cursor` walk the cache from the requested height to its tip and then transition into the live broadcast (with dedup so a block emitted from the cache isn't re-emitted from live). When a persistent block archive is also attached (`--archive-path`), backfill walks the archive first for any heights below the cache window, then chains into the cache walk — extending resume capability past the in-memory horizon. `stop_block_num`, `final_blocks_only`, and `transforms` are rejected with `FailedPrecondition`. **`Fetch.Block`** does point-in-time block lookup by `BlockNumber` reference; the service delegates to a `BlockOracle`. The CLI composes a `CachingBlockOracle` (read-through over the same in-memory cache the relayer feeds) wrapping a `GrpcBlockOracle` (dedicated `GrpcSource` connection — separate from the relayer's source so request-driven fetches don't serialize behind the live tail); the archive (when configured) is checked first to short-circuit upstream RPC for any height past the cache window. `BlockHashAndNumber` and `Cursor` references plus `transforms` are rejected with `FailedPrecondition`. **`EndpointInfo.Info`** reports chain identity for orchestrator sanity-check.
 
 `Response.metadata` is fully populated (num, id, parent_num, parent_id, lib_num=0 for now, time). `Response.block` carries a `google.protobuf.Any` whose `type_url` is `type.googleapis.com/sf.tron.type.v1.Block` and whose value is the prost-encoded `sf.tron.type.v1.Block` — header, transactions, and contract payloads (typed for the four high-volume contract kinds: `Transfer`, `TransferAsset`, `TriggerSmartContract`, `CreateSmartContract`; raw bytes plus wire kind tag for everything else, so consumers can decode locally against java-tron's protobuf for the long-tail governance/admin contracts).
 
@@ -98,8 +98,9 @@ The proto schema lives at `proto/sf/tron/type/v1/block.proto` and is compiled in
 Local persistence + the consistency-horizon SLO. Per ADR-0021, this crate is constitutionally bound to chain-finality as the only legal cross-replica consistency source — no Raft / Paxos / custom-quorum shims (the chain's SR consensus already solves the verification problem and the Das Sarma round-complexity floor makes any locally-engineered alternative provably worse).
 
 - **Consistency-horizon SLO** (landed): `ConsistencyHorizonObserver` records `seen_at` per block id when the relayer first surfaces a `tier=Seen` block, then closes the loop on the `tier=Finalized` transition by observing the elapsed wall-clock time into `lightcycle_store_block_seen_to_finalized_seconds`. **Target: p99 ≤ 5s under healthy operation; alert if >5s sustained for >5 min.** Exported via the standard prometheus exporter (`lightcycle relay --metrics-listen ...`).
-- **Block cache** (planned): recent N blocks for reorg replay, in-memory with spill to `redb`.
-- **Cursor store** (planned): per-consumer cursor checkpoints (optional, mostly for ops dashboards).
+- **Block cache** (landed): bounded in-memory `BlockCache<T>` indexed by both height and block id, generic over `T` so the crate stays free of `lightcycle-relayer` deps. The relayer writes on every `Output::New`/`Output::Undo`; firehose `Fetch.Block` and `Stream.Blocks` backfill read from it. Eviction is "drop the lowest-height entry first" — height-LRU is the right policy because consumer demand drops sharply with block age.
+- **Block archive** (landed): redb-backed `BlockArchive` that catches blocks past the chain's solidified-head threshold. Same opaque-bytes API shape as the cursor store; the firehose layer encodes `pb::Block` bytes into it via the archiver task. Stream.Blocks backfill walks archive → cache → live; Fetch.Block hits the archive on cache miss. Operator-facing CLI: `--archive-path` + `--archive-retention-blocks`. Append-only on the happy path (only past-finality blocks land here, by design).
+- **Cursor store** (landed): per-consumer cursor checkpoints in a redb-backed map. Per ADR-0021 explicitly **not** a cross-replica primitive — each replica tracks the consumers attached to it.
 - **SR set checkpoints** (planned): trusted starting point + maintenance-period diffs, so cold restarts don't have to re-derive the whole history.
 
 Future cross-replica work must implement `lightcycle_store::ConsistencySource`. The blessed implementation is `FinalityFromChain` (snapshots the relayer's view of the chain's solidified head). Reviewers proposing alternatives should be redirected to ADR-0021.

Cargo.toml

@@ -69,7 +69,6 @@ thiserror = "1.0"
 # Serialization
 serde = { version = "1.0", features = ["derive"] }
 serde_json = "1.0"
-bincode = "1.3"
 hex = "0.4"
 bs58 = { version = "0.5", default-features = false, features = ["alloc", "check"] }
 

README.md

@@ -48,11 +48,15 @@ This brings up `java-tron` (chain peer), `lightcycle` (relayer), Prometheus + Gr
 ## CLI
 
 ```bash
-# Live ingest pipeline: decode + verify + reorg + serve Firehose v2 over gRPC
+# Live ingest pipeline: decode + verify + reorg + serve Firehose v2 over gRPC,
+# with persistent block archive + healthcheck for k8s / orchestrator deployment.
 lightcycle relay \
   --grpc-url http://localhost:50051 \
   --firehose-listen 0.0.0.0:13042 \
-  --metrics-listen 0.0.0.0:9529
+  --metrics-listen 0.0.0.0:9529 \
+  --health-listen 0.0.0.0:9530 \
+  --archive-path /var/lib/lightcycle/blocks.redb \
+  --archive-retention-blocks 0
 
 # Lightweight HTTP-RPC head poller (no decode, no firehose): for the
 # kulen-side comparison dashboard or a Grafana liveness panel.
@@ -65,14 +69,23 @@ lightcycle inspect --grpc-url http://localhost:50051 --block 60123456
 `relay` is the flagship subcommand. With `--firehose-listen` set, the
 Firehose v2 server exposes:
 
-- **`Stream.Blocks`** — live tail with optional in-cache backfill via
-  `start_block_num` or `cursor` (cache window defaults to ~1h of mainnet
-  blocks; tune with `--block-cache-capacity`).
-- **`Fetch.Block`** — point-in-time lookup by height, read-through over
-  the same in-memory cache the relayer feeds (cache hit short-circuits
-  the upstream RPC).
+- **`Stream.Blocks`** — live tail with backfill via `start_block_num`
+  or `cursor`. Backfill walks the in-memory `BlockCache` first; with
+  `--archive-path` set it falls through to the persistent archive,
+  letting consumers resume past the in-memory window. Tune with
+  `--block-cache-capacity` (default 1024 ≈ 1h of mainnet at 3s slots)
+  and `--archive-retention-blocks` (default 0 = keep everything).
+- **`Fetch.Block`** — point-in-time lookup by height. Cache hit
+  short-circuits the upstream RPC; archive hit short-circuits the
+  upstream RPC for any height past the cache window.
 - **`EndpointInfo.Info`** — chain identity for orchestrator sanity-check.
 
+`--health-listen` exposes `/healthz` (200 if alive) and `/readyz`
+(200 once the relayer has observed the chain's solidified head). Both
+are kubelet-probe-compatible. Bound separately from `--metrics-listen`
+so a misconfigured Prometheus scrape can't black-hole the readiness
+signal.
+
 ## Benchmarking
 
 See [`BENCHMARKS.md`](./BENCHMARKS.md) for the methodology, baselines, and harness. Headline expectations on modern hardware: