shadowfork: dbfork mutation engine + LevelDB/RocksDB e2e

.github/workflows/ci.yml

@@ -80,6 +80,57 @@ jobs:
           go install golang.org/x/vuln/cmd/govulncheck@latest
           govulncheck ./...
 
+  proto-drift:
+    name: Proto-binding drift
+    runs-on: ubuntu-latest
+    # Regenerates internal/dbfork/proto/pb/ via the project's gen
+    # script and fails if the result differs from what's committed.
+    # Catches two failure modes:
+    #   1. Upstream .proto edit (via the git subtree pull workflow)
+    #      without running the gen script — committed bindings would
+    #      silently lag the proto definitions.
+    #   2. Hand-edit of a *.pb.go file (they look like normal Go and
+    #      tempt operators to "just tweak" — they're machine-
+    #      generated and will be clobbered by the next regen).
+    #
+    # protoc-gen-go version is pinned via go.mod's `tool` directive
+    # (Go 1.24+ pattern). `go install tool` here installs ALL tools
+    # declared in go.mod — currently just protoc-gen-go @ the
+    # matching google.golang.org/protobuf runtime version. Single
+    # source of truth; mismatched generator vs runtime can't happen.
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
+        with:
+          go-version: "1.25"
+          cache: true
+      - name: Install protoc
+        uses: arduino/setup-protoc@v3
+        with:
+          # Pinned to the EXACT 35.0 that generated the committed
+          # internal/dbfork/proto/pb/*.pb.go files (their header reads
+          # `protoc v7.35.0`). The drift job regenerates and diffs the
+          # committed bytes, including that header line — so a wildcard
+          # `35.x` would resolve to the latest 35.minor and false-report
+          # drift the day 35.1 ships, despite no .proto change. Bump
+          # this deliberately alongside a regenerate-and-commit.
+          version: "35.0"
+          repo-token: ${{ secrets.GITHUB_TOKEN }}
+      - name: Install pinned tools (protoc-gen-go via go.mod)
+        run: go install tool
+      - name: Regenerate proto bindings
+        run: bash scripts/gen-dbfork-protos.sh
+      - name: Fail on drift
+        # `git diff --exit-code` returns 1 if any file differs. We dump
+        # the diff first so the failure log shows what changed without
+        # the developer needing to re-run locally.
+        run: |
+          if ! git diff --exit-code internal/dbfork/proto/pb/; then
+            echo "::error::Committed .pb.go files drifted from regeneration."
+            echo "::error::Run 'bash scripts/gen-dbfork-protos.sh' locally + commit the result."
+            exit 1
+          fi
+
   build-matrix:
     name: Cross-compile
     runs-on: ubuntu-latest

.github/workflows/dbfork-equivalence.yml

@@ -0,0 +1,215 @@
+name: dbfork Go-vs-Java Equivalence
+
+# Phase 1 release gate: runs TestEquivalence_GoVsJava, which stands
+# up two scratch copies of a Nile snapshot, applies the canonical
+# fork.conf via Go (dbfork) and Java (tron-docker/tools/toolkit
+# DbFork), and diffs all 8 stores. A pass means the Go port produces
+# byte-identical state to the reference implementation.
+#
+# Why a separate workflow:
+#   - The fixture is multi-GB (Nile lite snapshot, ~45 GB compressed).
+#     Too expensive to run on every PR; this fires on schedule + on
+#     explicit dispatch + on PRs that touch internal/dbfork/**.
+#   - Builds the java toolkit on demand. Requires JDK 11+ and the
+#     `tron-docker` repo as a peer checkout.
+#
+# When a release-bound PR lands in internal/dbfork, this workflow
+# acts as the final byte-equivalence gate before the PR can merge.
+
+on:
+  push:
+    branches: [master, develop]
+    paths:
+      - 'internal/dbfork/**'
+      - 'internal/snapshot/**'
+      - '.github/workflows/dbfork-equivalence.yml'
+  pull_request:
+    paths:
+      - 'internal/dbfork/**'
+      - 'internal/snapshot/**'
+      - '.github/workflows/dbfork-equivalence.yml'
+  schedule:
+    # Weekly catch — surfaces snapshot-format drift (java DbFork
+    # upstream changes, Nile snapshot publisher output rotation)
+    # even when no dbfork code has changed.
+    - cron: "0 6 * * 0"
+  workflow_dispatch: {}
+
+permissions:
+  contents: read
+
+jobs:
+  equivalence:
+    runs-on: ubuntu-latest
+    timeout-minutes: 90  # snapshot download is the long pole (~30-45 min)
+
+    steps:
+      - uses: actions/checkout@v4
+
+      # GitHub-hosted runners ship ~14 GB of preinstalled tools we
+      # don't need (Android SDK, .NET, CodeQL packages, etc.) and have
+      # ~84 GB total disk. The Nile lite snapshot is ~45 GB compressed
+      # / ~90 GB extracted, so we MUST reclaim disk before trond's
+      # pre-download free-space check (the first run's "Error [DISK_
+      # SPACE_ERROR]: need ~91.57 GB free, have 88.36 GB" failure).
+      # This action reclaims ~30-40 GB by removing unused tools.
+      - name: Free runner disk space for the Nile fixture
+        uses: jlumbroso/free-disk-space@v1.3.1
+        with:
+          tool-cache: true
+          android: true
+          dotnet: true
+          haskell: true
+          large-packages: true
+          # docker engine cleanup is the slow pass; we don't run docker
+          # in this workflow but disabling saves several minutes.
+          docker-images: false
+          swap-storage: false
+
+      - uses: actions/setup-go@v5
+        with:
+          go-version: "1.25"
+          cache: true
+
+      - uses: actions/setup-java@v4
+        with:
+          distribution: temurin
+          java-version: "11"
+
+      - name: Checkout tron-docker for java toolkit
+        uses: actions/checkout@v4
+        with:
+          repository: tronprotocol/tron-docker
+          path: tron-docker
+          # Pinned to a SHA for a reproducible reference implementation:
+          # a floating `main` could change DbFork's output mid-flight and
+          # turn the equivalence gate red (or, worse, green against a
+          # changed reference) without any change to this repo. Bump
+          # deliberately when adopting a new java DbFork behaviour.
+          ref: d89d353b06d1f5016d91b02654508eeecdf5a904  # tron-docker main @ 2026-05-27
+
+      - name: Build java DbFork toolkit
+        # tron-docker's tools/ is a multi-project gradle build. The
+        # wrapper lives at tools/gradlew/, and toolkit is referenced
+        # as the `:toolkit` subproject. Per the toolkit README's
+        # "Build The Toolkit" section. shadowJar (fat jar) bundles
+        # the dependency closure so the equivalence test can launch
+        # via `java -jar` without a classpath dance.
+        run: |
+          cd tron-docker/tools/gradlew
+          ./gradlew :toolkit:shadowJar
+          ls -la ../toolkit/build/libs/
+
+      - name: Resolve toolkit jar path
+        id: jar
+        # The toolkit build.gradle sets archiveBaseName='Toolkit' +
+        # archiveClassifier='' with no version, so shadowJar emits a
+        # single fat jar named exactly `Toolkit.jar` (NOT the shadow-
+        # plugin default `Toolkit-<ver>-all.jar`). An earlier glob of
+        # `Toolkit*-all.jar` matched nothing, left this output empty,
+        # and the empty DBFORK_JAVA_TOOLKIT resolved to a directory ->
+        # the test SKIPPED and the job passed vacuously. Fail loudly if
+        # the artifact name ever changes again.
+        run: |
+          set -euo pipefail
+          JAR=tron-docker/tools/toolkit/build/libs/Toolkit.jar
+          if [ ! -f "$JAR" ]; then
+            echo "::error::toolkit jar not found at $JAR — did the shadowJar artifact name change?"
+            ls -la tron-docker/tools/toolkit/build/libs/ || true
+            exit 1
+          fi
+          echo "path=$JAR" >> "$GITHUB_OUTPUT"
+          echo "Found toolkit jar at: $JAR"
+
+      - name: Compute weekly cache key
+        id: weekkey
+        # ISO week-of-year — stable across all runs in a given week,
+        # so the cache primary key actually hits. github.run_id rotates
+        # per run and would force a fresh write every time, bloating
+        # the 10 GB per-repo cache budget without ever restoring.
+        run: echo "yyyyww=$(date -u +%Y%V)" >> "$GITHUB_OUTPUT"
+
+      - name: Cache Nile fixture
+        id: fixture-cache
+        uses: actions/cache@v4
+        with:
+          path: ./nile-fixture
+          # Refresh weekly. Snapshot rotates daily but byte content
+          # doesn't matter for equivalence — both Go and Java see the
+          # SAME copy, so any captured snapshot is fine for diffing.
+          key: nile-fixture-${{ steps.weekkey.outputs.yyyyww }}
+          restore-keys: |
+            nile-fixture-
+
+      - name: Build trond + download Nile fixture (if cache miss)
+        if: steps.fixture-cache.outputs.cache-hit != 'true'
+        run: |
+          set -euo pipefail
+          go build -o bin/trond ./
+          ./bin/trond snapshot download \
+            --network nile --type lite \
+            --to ./nile-fixture
+          # snapshot download streams gunzip|tar straight to disk and
+          # never persists the .tgz, so there is nothing to clean up
+          # here. Guard that the extracted layout the test expects
+          # actually materialised, so a future download-format change
+          # fails here instead of as a confusing SKIP downstream.
+          DB=./nile-fixture/output-directory/database
+          test -d "$DB" \
+            || { echo "::error::fixture missing output-directory/database after download"; exit 1; }
+          # Prune to the 8 dbfork stores. A full Nile snapshot is ~90 GB,
+          # dominated by block/trans/pbft-sign-data which dbfork never
+          # touches — java DbFork's initStore() and Go's Apply open only
+          # these 8 stores. The test makes TWO scratch copies, so keeping
+          # the bulk overflowed the runner disk ("no space left on
+          # device"). Pruning frees ~40+ GB and is what gets cached, so
+          # cache-hit runs are already lean.
+          KEEP="witness witness_schedule account properties asset-issue-v2 account-asset contract storage-row"
+          for d in "$DB"/*/; do
+            name=$(basename "$d")
+            case " $KEEP " in
+              *" $name "*) : ;;          # keep the 8 dbfork stores
+              *) rm -rf "$d" ;;          # drop everything else
+            esac
+          done
+          echo "Pruned fixture to dbfork stores; remaining:"
+          ls -1 "$DB"
+          df -h . | tail -1
+
+      - name: Run equivalence test
+        env:
+          DBFORK_NILE_FIXTURE: ${{ github.workspace }}/nile-fixture/output-directory
+          DBFORK_JAVA_TOOLKIT: ${{ github.workspace }}/${{ steps.jar.outputs.path }}
+          DBFORK_FORK_CONF: ${{ github.workspace }}/tron-docker/tools/toolkit/src/main/resources/fork.conf
+          DBFORK_JAVA_HEAP: "4g"
+        run: |
+          set -euo pipefail
+          # TestEquivalence_GoVsJava SKIPs (does not fail) when its env
+          # prereqs are missing — that keeps local `go test ./...` green
+          # for devs without the toolkit. But in THIS workflow a skip
+          # means the release gate didn't actually run, which previously
+          # passed vacuously. Capture the output and hard-fail on SKIP so
+          # the gate can never be silently hollow again.
+          go test -v -timeout 30m -run TestEquivalence_GoVsJava ./internal/dbfork/ 2>&1 | tee equivalence.out
+          if grep -q -- "--- SKIP: TestEquivalence_GoVsJava" equivalence.out; then
+            echo "::error::equivalence test SKIPPED — env prereqs (fixture / toolkit jar / fork.conf) not satisfied; the gate did not run"
+            exit 1
+          fi
+          # Also assert the per-store diff actually executed (diffStore
+          # logs "<store>: N keys on Go, M keys on Java"); its absence
+          # means the run was hollow even if it didn't formally SKIP.
+          if ! grep -q "keys on Go" equivalence.out; then
+            echo "::error::equivalence test produced no per-store diff output — run was hollow"
+            exit 1
+          fi
+
+      - name: Upload diff log on failure
+        if: failure()
+        uses: actions/upload-artifact@v4
+        with:
+          name: equivalence-failure-log
+          path: |
+            equivalence.out
+            tron-docker/tools/toolkit/build/logs/
+          retention-days: 14
+          if-no-files-found: ignore

.gitignore

@@ -35,3 +35,12 @@ Thumbs.db
 # Local Claude Code workspace state
 .claude/
 CLAUDE.md
+
+# Shadow-fork PoC artifacts — generated by scripts/poc-shadow-fork.sh.
+# .shadow-fork-witness.env contains a freshly-generated secp256k1
+# private key and MUST never be committed; the data dir and templated
+# conf/intent files are similarly local-only.
+.shadow-fork-witness.env
+shadow-fork-data/
+shadow-fork.conf
+shadow-fork-intent.yaml

AGENTS.md

@@ -301,7 +301,85 @@ trond network destroy pn --confirm pn -o json
 
 ---
 
-## Workflow 5 — Build java-tron from source
+## Workflow 5 — Shadow-fork testing on a real snapshot
+
+Use when the user wants to test a hard-fork proposal, a witness-set
+transition, or a contract-state migration against realistic
+mainnet/Nile state — without affecting the real network. Takes a
+chain DB snapshot, surgically replaces the witness set + funds test
+accounts + tweaks DynamicProperties timestamps, then launches a
+private network from the modified state.
+
+The mutation engine is a Go port of java-tron's `DbFork` toolkit
+(equivalence test runs in CI against a real Nile snapshot to prove
+byte-equivalence). Both HOCON and YAML fork.conf formats accepted.
+
+```bash
+# 1. Pull a Nile snapshot. Mainnet works too; Nile is lighter and
+#    faster for the demo path.
+trond snapshot download --network nile --type lite \
+    --to /srv/shadow-fork -o json
+
+# 2. The node owning the data dir MUST be stopped before mutate
+#    (LevelDB locks are exclusive). If you just downloaded a fresh
+#    snapshot, no node is running — skip this. Otherwise:
+trond stop <node> -o json
+# Output: {"name":"<node>","previous_state":"running","new_state":"stopped",...}
+
+# 3. Apply the fork.conf — replace witnesses, fund accounts, set
+#    TRC10/TRC20 balances, adjust timestamps. The mutation is
+#    in-place and idempotent (re-runs against the same conf
+#    produce the same on-disk state).
+trond shadow-fork mutate \
+    -d /srv/shadow-fork/output-directory \
+    -c fork.conf -o json
+# Output: {"witnesses_written":1, "active_witnesses":1,
+#          "accounts_modified":1, "trc20_slots_updated":0,
+#          "properties_updated":3, "duration_ms":...}
+
+# 4. Launch a private shadow-fork node off the mutated dir. The
+#    intent MUST use `network: nile` (the snapshot's genesis hash
+#    must match the base config) AND isolate via:
+#    - network_overrides.need_sync_check: false
+#    - config_overrides."seed.node.ip.list": []
+#    - config_overrides."node.p2p.version": 99999  (foreign chain)
+trond apply --intent shadow-fork-intent.yaml --auto-approve --wait -o json
+
+# 5. Verify the shadow chain is producing blocks. Single-witness
+#    setups produce a block every 3s (every slot is that witness's).
+trond status <node> -o json
+# Output: latest_block_number climbs each invocation.
+```
+
+See `knowledge/shadow-fork-poc.md` + `examples/shadow-fork/` for
+the full walkthrough including the witness-keypair generation step
+(tronpy) + the canonical fork.conf template. `scripts/poc-shadow-fork.sh`
+orchestrates the whole flow.
+
+### Key invariants
+
+- **fork.conf is the contract**. Per-section semantics live in
+  `internal/dbfork/{witnesses,accounts,trc20,properties}.go` and
+  mirror java DbFork verbatim. Operators reading the conf in either
+  HOCON or YAML get identical Config shapes.
+- **Genesis hash must match**. The shadow-fork intent's `network:`
+  determines the base config's genesis block. A Nile snapshot
+  requires `network: nile`; a mainnet snapshot requires
+  `network: mainnet`. Mismatch crash-loops the container with
+  `Genesis block modify, please delete database directory`.
+- **The node must be stopped** before mutate. `shadow_fork_mutate`
+  fails loudly with `resource temporarily unavailable` if a
+  process holds the LevelDB lock — but the failure is mid-mutation
+  and may leave partial writes. Always `trond stop` first.
+- **Single-witness setups produce blocks but lack finality**. The
+  per-block SR confirmation count stays at 1 (well below the 19/27
+  threshold). Good enough to verify mutations applied; not
+  sufficient for testing finality-dependent contract logic. Real
+  shadow-fork production parity uses 27 witnesses.
+
+---
+
+## Workflow 6 — Build java-tron from source
 
 Use when the user wants a custom build: a fork, an unreleased patch,
 a wired-in profiler. The build pipeline is content-addressed by
@@ -534,7 +612,7 @@ Configure once in your client. Example for Claude Desktop
 }
 ```
 
-The server registers 19 tools (read-only unless marked):
+The server registers 20 tools (read-only unless marked):
 
 - **inspection** (3): `list`, `status`, `inspect`
 - **diagnostic** (4): `doctor`, `version`, `health`, `diagnose`
@@ -546,7 +624,10 @@ The server registers 19 tools (read-only unless marked):
 - **build** (3): `build_list`, `build_inspect`, `build_prune`
   (destructive — dry-run by default; `confirm=true` actually deletes).
   Build *execution* is NOT exposed via MCP; call the CLI directly
-  for that (see Workflow 5).
+  for that (see Workflow 6).
+- **shadow-fork** (1): `shadow_fork_mutate` (destructive — mutates a
+  halted java-tron data dir in place; see Workflow 5 for the full
+  recipe).
 
 Destructive tools carry the MCP `destructiveHint` annotation so MCP
 clients prompt the user. The server's `Instructions` field