From d51b7426e11129ff836dbaee6d0cfb6b6fecfd66 Mon Sep 17 00:00:00 2001 From: dkijania Date: Wed, 27 May 2026 15:56:49 +0200 Subject: [PATCH 1/5] docs(archive-node): add Backfilling Missing Blocks page and split storage/restore Adds a new node-operators/archive-node/backfilling-missing-blocks page covering mina-missing-blocks-auditor (detection, exit-code bits, structured output), mina-archive-blocks (precomputed + extensional restore), and the missing-blocks-guardian script (audit / single-run / daemon modes), plus a hard-fork section spelling out the two preconditions the auditor needs (first canonical post-fork block + at least one block at or above the target tip) and pointers to o1Labs-published precomputed-block / dump buckets alongside the recommendation to self-host them for decentralization. Refactors archive-redundancy.mdx to focus purely on storage: keeps the multi-daemon / multi-archive redundancy guidance and block-data backup methods, adds a Back up the archive database subsection (pg_dump scheduling plus the mina-archive-dumps bucket), and replaces the old detection + restore sections with a redirect to the new page. The two pages now cross-link in both directions so storage and recovery have a clean split. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../archive-node/archive-redundancy.mdx | 36 ++-- .../backfilling-missing-blocks.mdx | 176 +++++++++++++++ sidebars.js | 1 + static/llms-full.txt | 202 ++++++++++++++++-- static/llms.txt | 1 + 5 files changed, 384 insertions(+), 32 deletions(-) create mode 100644 docs/node-operators/archive-node/backfilling-missing-blocks.mdx diff --git a/docs/node-operators/archive-node/archive-redundancy.mdx b/docs/node-operators/archive-node/archive-redundancy.mdx index a76ecd96f..af132e4df 100644 --- a/docs/node-operators/archive-node/archive-redundancy.mdx +++ b/docs/node-operators/archive-node/archive-redundancy.mdx @@ -19,7 +19,7 @@ Archive data is critical for applications that require historical lookup. On the protocol side, archive data is important for disaster recovery to reconstruct a certain state. A single [archive node](/node-operators/archive-node/getting-started) set up might not be sufficient. -If the daemon that sends blocks to the archive process or if the archive process itself fails for some reason, there can be missing blocks in the database. To minimize the risk of archive data loss, employ redundancy techniques. +If the daemon that sends blocks to the archive process or if the archive process itself fails for some reason, there can be missing blocks in the database. To minimize the risk of archive data loss, employ the redundancy techniques described on this page; to detect and repair gaps once they have already formed, see [Backfilling Missing Blocks](./backfilling-missing-blocks). A single archive node setup has a daemon sending blocks to an archive process that writes them to the database. @@ -89,31 +89,35 @@ If only the end hash is provided, then the tool generates blocks starting with t Provide the flag `--all-blocks` to write out all blocks contained in the database. -## Identify missing blocks +### Back up the archive database -To determine any missing blocks in an archive database, use the `mina-missing-block-auditor` tool. +In addition to retaining per-block files, you can snapshot the entire PostgreSQL archive database. A database dump is the fastest way to seed a fresh archive without replaying every block. -The tool outputs a list of state hashes of all the blocks in the database that are missing a parent. You can use this list to monitor the archive database for any missing blocks. To specify the URI of the PostgreSQL database, use the `--archive-uri` flag. +Use `pg_dump` to create a snapshot: -## Restore blocks +```bash +pg_dump -U -Fp | gzip > archive-$(date +%F).sql.gz +``` -When you have block data (precomputed or extensional) available from [Back up block data](/node-operators/archive-node/archive-redundancy#back-up-block-data), you can restore missing blocks in an archive database using the tool `mina-archive-blocks`. +Schedule this regularly (a nightly cron is typical) and publish the dumps to object storage that your downstream consumers can reach. -1. Restore precomputed blocks from: +O1Labs publishes nightly dumps of its public-network archives to the [mina-archive-dumps](https://console.cloud.google.com/storage/browser/mina-archive-dumps) GCS bucket as a convenience: - - [Upload block data to Google Cloud Storage](/node-operators/archive-node/archive-redundancy#upload-block-data-to-google-cloud-storage) +```bash +# Mainnet +curl -O https://storage.googleapis.com/mina-archive-dumps/mainnet-archive-dump-$(date +%F)_0000.sql.tar.gz +# Devnet — swap "mainnet" for "devnet". +``` - - [Save block data from logs](/node-operators/archive-node/archive-redundancy#save-block-data-from-logs) +:::tip Mirror the dumps you depend on -``` - mina-archive-blocks --precomputed --archive-uri FILES -``` +Hosting your own dumps (and your own per-block files) is what keeps the network from converging on a single provider. The o1Labs bucket is a convenience, not a guarantee. -2. For extensional blocks: (Generated from option [3](#generate-block-data-from-another-archive-database)) +::: -``` - mina-archive-blocks --extensional --archive-uri FILES -``` +## Restoring missing blocks + +Detecting gaps and restoring blocks from the backups described above is covered on a dedicated page: [Backfilling Missing Blocks](./backfilling-missing-blocks). That guide walks through `mina-missing-blocks-auditor`, `mina-archive-blocks`, and the missing-blocks guardian script. ## Staking ledgers diff --git a/docs/node-operators/archive-node/backfilling-missing-blocks.mdx b/docs/node-operators/archive-node/backfilling-missing-blocks.mdx new file mode 100644 index 000000000..4e292c12f --- /dev/null +++ b/docs/node-operators/archive-node/backfilling-missing-blocks.mdx @@ -0,0 +1,176 @@ +--- +title: Backfilling Missing Blocks +sidebar_label: Backfilling Missing Blocks +description: How to detect and repair gaps in a Mina archive database using archive dumps, precomputed blocks, mina-archive-blocks, and the mina-missing-blocks-auditor. +keywords: + - mina archive node + - missing blocks + - mina-missing-blocks-auditor + - mina-archive-blocks + - precomputed blocks + - archive dump + - backfill + - hardfork +--- + +# Backfilling Missing Blocks + +A Mina archive database can develop gaps whenever the daemon cannot reach the `mina-archive` RPC endpoint as a new block is added to its transition frontier. The daemon ships each new breadcrumb to the archive process in a fire-and-forget loop: on a failed RPC it retries up to 5 times back-to-back (no backoff), logs a warning, and moves on to the next block. Crucially, **the daemon does not persist missed blocks or queue them for later delivery** — once the 5 retries are exhausted, that block is gone from the daemon's perspective and only blocks added *after* the archive reconnects will be archived. + +This means even a short archive-process restart, a transient network blip beyond a few hundred milliseconds, or any RPC error during block ingestion can leave a permanent gap. The `k`-block transition-frontier retention used for consensus does *not* re-ship blocks to the archive on reconnect. + +Because the archive stores only incremental changes, even a single missing block breaks chain continuity and makes downstream queries (balances, Rosetta, zkApp event/action lookups, replayer runs) unreliable. + +This page explains how to: + +- detect gaps with [`mina-missing-blocks-auditor`](#detect-gaps-with-mina-missing-blocks-auditor), +- obtain the block data needed to repair them (see [Sources of block data](#sources-of-block-data)), +- write that data back into PostgreSQL with [`mina-archive-blocks`](#restore-blocks-with-mina-archive-blocks), +- automate the whole loop with the [missing blocks guardian](#automated-backfill-the-missing-blocks-guardian), +- and apply this workflow to the [hard-fork case](#backfilling-around-a-hard-fork), which has additional requirements. + +## Detect gaps with mina-missing-blocks-auditor + +`mina-missing-blocks-auditor` is the canonical tool for inspecting an archive database. It is shipped in the `mina-archive` Debian package and Docker image alongside `mina-archive-blocks` and `mina-extract-blocks`. + +Run it against your archive database: + +```bash +mina-missing-blocks-auditor \ + --archive-uri postgres://:@:/ +``` + +The auditor performs four independent checks and encodes the results in its exit code (a bitfield, `0` means a healthy database): + +| Bit | Check | What it means | +| --- | --- | --- | +| 0 | Missing blocks | One or more blocks have no parent row in `blocks` (orphan parents). Genesis and the first post-hard-fork block are excluded. | +| 1 | Pending blocks below tip | Pending blocks exist at a height below the highest canonical block — a sign that finalization stalled or rows were lost. | +| 2 | Chain length | The reconstructed canonical chain is shorter than `max(height)` of canonical blocks. | +| 3 | Chain status | A block reachable from the canonical tip has a `chain_status` other than `canonical`. | + +For every missing block the auditor logs a structured entry that names the orphan, its expected parent, and the size of the gap: + +```text +Block has no parent in archive db + block_id: 12345 + state_hash: 3N... + height: 387200 + parent_hash: 3N... + parent_height: 387199 + missing_blocks_gap: 1 +``` + +The `parent_hash` and `parent_height` are exactly what you need to locate the missing block file. This is how the [guardian script](#automated-backfill-the-missing-blocks-guardian) drives its download loop. + +The auditor is read-only — it never modifies the database, so it is safe to run on a live archive node. + +## Sources of block data + +To fill a gap you need the block(s) that the auditor reports as missing in one of two formats: + +- **Precomputed blocks** — the full block JSON the daemon serializes, named `--.json`. Use these whenever possible. +- **Extensional blocks** — a slimmer JSON generated from an existing archive database via `mina-extract-blocks`. Sufficient to repair an archive but lacks the SNARK and other fields that are not stored in PostgreSQL. + +How to *store* these files (uploading blocks from a daemon, saving them from logs, extracting them from another archive, scheduling database dumps) is covered on [Archive Redundancy](./archive-redundancy#back-up-block-data). This page assumes the files already exist somewhere and only documents *where to read them from* for a backfill. + +The two public sources that o1Labs operates as a convenience are: + +| Bucket | Contents | Use for | +| --- | --- | --- | +| [`gs://mina-archive-dumps`](https://console.cloud.google.com/storage/browser/mina-archive-dumps) | Nightly archive-database snapshots (`-archive-dump-_0000.sql.tar.gz`) | Bootstrapping a fresh archive — apply the dump first, then backfill anything produced after the snapshot. | +| [`gs://mina_network_block_data`](https://console.cloud.google.com/storage/browser/mina_network_block_data) | Per-block precomputed JSON files (`--.json`) | The block-by-block input the auditor + guardian walk through to close gaps. | + +:::tip Decentralization + +These o1Labs buckets are a convenience, not a guarantee. The network is healthier when operators publish their own dumps and precomputed-block mirrors — see [Archive Redundancy](./archive-redundancy) for how to set that up. + +::: + +If you mirror the data to your own bucket, point the [guardian](#automated-backfill-the-missing-blocks-guardian) at your URL via the `PRECOMPUTED_BLOCKS_URL` environment variable. + +## Restore blocks with mina-archive-blocks + +Once you have the missing block files, `mina-archive-blocks` writes them into PostgreSQL. Pass `--precomputed` or `--extensional` to match the file format: + +```bash +# Precomputed blocks (from the daemon or the public bucket) +mina-archive-blocks \ + --precomputed \ + --archive-uri postgres://:@:/ \ + mainnet-387200-3NK....json mainnet-387201-3NL....json + +# Extensional blocks (produced by mina-extract-blocks) +mina-archive-blocks \ + --extensional \ + --archive-uri postgres://:@:/ \ + 3NK....json +``` + +Useful flags: + +- `--successful-files ` — append successfully imported filenames to a log. +- `--failed-files ` — append filenames that failed (parse error, schema mismatch, missing dependency). +- `--log-successful false` — suppress per-block success logs when importing many files. + +Re-run `mina-missing-blocks-auditor` after the import to confirm the gap is closed. + +## Automated backfill: the missing blocks guardian + +For continuous archives, the [`missing-blocks-guardian.sh`](https://github.com/MinaProtocol/mina/blob/master/scripts/archive/missing-blocks-guardian.sh) script bundles the audit → download → restore loop. It is included in the `mina-archive` Docker image and is what the [Docker Compose example](./docker-compose) wires up out of the box. + +The guardian has three modes: + +| Subcommand | Behavior | +| --- | --- | +| `audit` | Run the auditor once and exit. Returns whether the database is healthy without modifying anything. Use this from monitoring/alerting (e.g. a Kubernetes liveness check or a cron-driven Prometheus exporter). | +| `single-run` | Audit, then walk back from each orphan parent, download the corresponding precomputed block from `PRECOMPUTED_BLOCKS_URL`, import it with `mina-archive-blocks`, and repeat until the database is clean. Exits when done. Right for one-off repairs and bootstrapping. | +| `daemon` | The same recovery loop as `single-run`, but kept running forever. Re-audits every `TIMEOUT` seconds (default 600); when a repair completes it sleeps `6 × TIMEOUT` before checking again. Right for production archive nodes. | + +Required environment variables: + +| Variable | Purpose | +| --- | --- | +| `DB_USERNAME`, `PGPASSWORD`, `DB_HOST`, `DB_PORT`, `DB_NAME` | PostgreSQL connection for the archive database. | +| `PRECOMPUTED_BLOCKS_URL` | Base URL of the precomputed-blocks bucket (e.g. `https://storage.googleapis.com/mina_network_block_data` or your own mirror). | +| `MINA_NETWORK` | Network name used as a filename prefix (`mainnet`, `devnet`, ...). | + +Optional overrides (`MISSING_BLOCKS_AUDITOR`, `ARCHIVE_BLOCKS`, `BLOCKS_FORMAT`, `TIMEOUT`) are documented in the script's `--help` output. + +A minimal invocation: + +```bash +export DB_USERNAME=postgres DB_HOST=localhost DB_PORT=5432 DB_NAME=archive +export PGPASSWORD=postgres +export PRECOMPUTED_BLOCKS_URL=https://storage.googleapis.com/mina_network_block_data +export MINA_NETWORK=mainnet + +./missing-blocks-guardian.sh single-run +``` + +## Backfilling around a hard fork + +A hard fork creates a fresh chain that starts at its own *first canonical block* — the block whose `global_slot_since_hard_fork = 0`. Before the new chain begins producing pending blocks, an archive database upgraded from the previous network has a discontinuity that the auditor cannot repair by itself. + +The missing-blocks workflow **does work** across a hard fork, but it relies on the auditor's gap-only model. The auditor fills gaps *between existing blocks* — it does not invent new endpoints. For the workflow to succeed, the archive database must satisfy two conditions: + +1. **The first canonical block of the new fork is already in the database.** This is the post-fork genesis-equivalent. It is typically inserted either by running the upgrade script that comes with the new release, or by importing a post-fork archive dump that already contains it. Without this row, the auditor has nothing on the new chain to anchor a backfill against. +2. **At least one block at or above your target tip height (canonical or pending) is also already in the database.** The auditor walks *backwards* from existing blocks toward their missing parents; if the high end of the new chain is absent, there is no orphan for the auditor to report and no parent-hash for the guardian to download. + +With both endpoints present, the guardian's `single-run` (or `daemon`) mode will pull every precomputed block in between from `PRECOMPUTED_BLOCKS_URL` and stitch the new chain together. + +For hard-fork-specific precomputed-blocks distributions, o1Labs typically publishes a dedicated bucket alongside the release (for example, `gs://mesa-hf-precomputed-blocks` for the Mesa upgrade). Check the network-upgrade documentation for the bucket name and version pin matching your fork, set `PRECOMPUTED_BLOCKS_URL` and `MINA_NETWORK` to that bucket and network name, and run the guardian as above. + +:::caution What this workflow does *not* do + +- It does not upgrade your archive schema. Schema changes that accompany a hard fork (for example, Berkeley → Mesa) must be applied with the upgrade scripts shipped in the matching `mina-archive` release before any backfill is attempted. See [Mesa Archive Upgrade](../../network-upgrades/mesa/archive-upgrade) for the canonical procedure. +- It does not migrate historical (pre-fork) data into the new schema — that is handled separately by the migration tooling for the network upgrade. +- It does not validate the contents of imported blocks against a running daemon. Use [`mina-replayer`](./archive-redundancy) afterwards if you need ledger-level verification. + +::: + +## Related + +- [Archive Redundancy](./archive-redundancy) — the companion page on how to *store* precomputed blocks, extensional blocks, and database dumps so a backfill source always exists. This page covers what to do once those backups are in place. +- [Docker Compose example](./docker-compose) — a ready-made stack that runs the guardian alongside a bootstrap-from-dump postgres and a mina-archive process. +- [Archive Nodes Getting Started](./getting-started) — initial setup of the archive database and `mina-archive` process. diff --git a/sidebars.js b/sidebars.js index df6506e5e..afb4392b7 100644 --- a/sidebars.js +++ b/sidebars.js @@ -1528,6 +1528,7 @@ module.exports = { items: [ 'node-operators/archive-node/getting-started', 'node-operators/archive-node/archive-redundancy', + 'node-operators/archive-node/backfilling-missing-blocks', 'node-operators/archive-node/docker-compose', ], }, diff --git a/static/llms-full.txt b/static/llms-full.txt index bfab5df49..341b8516e 100644 --- a/static/llms-full.txt +++ b/static/llms-full.txt @@ -5167,7 +5167,7 @@ Archive data is critical for applications that require historical lookup. On the protocol side, archive data is important for disaster recovery to reconstruct a certain state. A single [archive node](/node-operators/archive-node/getting-started) set up might not be sufficient. -If the daemon that sends blocks to the archive process or if the archive process itself fails for some reason, there can be missing blocks in the database. To minimize the risk of archive data loss, employ redundancy techniques. +If the daemon that sends blocks to the archive process or if the archive process itself fails for some reason, there can be missing blocks in the database. To minimize the risk of archive data loss, employ the redundancy techniques described on this page; to detect and repair gaps once they have already formed, see [Backfilling Missing Blocks](./backfilling-missing-blocks). A single archive node setup has a daemon sending blocks to an archive process that writes them to the database. @@ -5237,31 +5237,35 @@ If only the end hash is provided, then the tool generates blocks starting with t Provide the flag `--all-blocks` to write out all blocks contained in the database. -## Identify missing blocks +### Back up the archive database -To determine any missing blocks in an archive database, use the `mina-missing-block-auditor` tool. +In addition to retaining per-block files, you can snapshot the entire PostgreSQL archive database. A database dump is the fastest way to seed a fresh archive without replaying every block. -The tool outputs a list of state hashes of all the blocks in the database that are missing a parent. You can use this list to monitor the archive database for any missing blocks. To specify the URI of the PostgreSQL database, use the `--archive-uri` flag. +Use `pg_dump` to create a snapshot: -## Restore blocks +```bash +pg_dump -U -Fp | gzip > archive-$(date +%F).sql.gz +``` -When you have block data (precomputed or extensional) available from [Back up block data](/node-operators/archive-node/archive-redundancy#back-up-block-data), you can restore missing blocks in an archive database using the tool `mina-archive-blocks`. +Schedule this regularly (a nightly cron is typical) and publish the dumps to object storage that your downstream consumers can reach. -1. Restore precomputed blocks from: +O1Labs publishes nightly dumps of its public-network archives to the [mina-archive-dumps](https://console.cloud.google.com/storage/browser/mina-archive-dumps) GCS bucket as a convenience: - - [Upload block data to Google Cloud Storage](/node-operators/archive-node/archive-redundancy#upload-block-data-to-google-cloud-storage) +```bash +# Mainnet +curl -O https://storage.googleapis.com/mina-archive-dumps/mainnet-archive-dump-$(date +%F)_0000.sql.tar.gz +# Devnet — swap "mainnet" for "devnet". +``` - - [Save block data from logs](/node-operators/archive-node/archive-redundancy#save-block-data-from-logs) +:::tip Mirror the dumps you depend on -``` - mina-archive-blocks --precomputed --archive-uri FILES -``` +Hosting your own dumps (and your own per-block files) is what keeps the network from converging on a single provider. The o1Labs bucket is a convenience, not a guarantee. -2. For extensional blocks: (Generated from option [3](#generate-block-data-from-another-archive-database)) +::: -``` - mina-archive-blocks --extensional --archive-uri FILES -``` +## Restoring missing blocks + +Detecting gaps and restoring blocks from the backups described above is covered on a dedicated page: [Backfilling Missing Blocks](./backfilling-missing-blocks). That guide walks through `mina-missing-blocks-auditor`, `mina-archive-blocks`, and the missing-blocks guardian script. ## Staking ledgers @@ -5275,6 +5279,172 @@ Epoch ledger transition happens once every 14 days (given slot-time = 3mins and The window to backup a staking ledger is ~27 days considering "next" staking ledger is finalized after k (currently 290) blocks in the current epoch and therefore is available for the rest of the current epoch and the entire next epoch. +--- +url: /node-operators/archive-node/backfilling-missing-blocks +--- + +# Backfilling Missing Blocks + +A Mina archive database can develop gaps whenever the daemon cannot reach the `mina-archive` RPC endpoint as a new block is added to its transition frontier. The daemon ships each new breadcrumb to the archive process in a fire-and-forget loop: on a failed RPC it retries up to 5 times back-to-back (no backoff), logs a warning, and moves on to the next block. Crucially, **the daemon does not persist missed blocks or queue them for later delivery** — once the 5 retries are exhausted, that block is gone from the daemon's perspective and only blocks added *after* the archive reconnects will be archived. + +This means even a short archive-process restart, a transient network blip beyond a few hundred milliseconds, or any RPC error during block ingestion can leave a permanent gap. The `k`-block transition-frontier retention used for consensus does *not* re-ship blocks to the archive on reconnect. + +Because the archive stores only incremental changes, even a single missing block breaks chain continuity and makes downstream queries (balances, Rosetta, zkApp event/action lookups, replayer runs) unreliable. + +This page explains how to: + +- detect gaps with [`mina-missing-blocks-auditor`](#detect-gaps-with-mina-missing-blocks-auditor), +- obtain the block data needed to repair them (see [Sources of block data](#sources-of-block-data)), +- write that data back into PostgreSQL with [`mina-archive-blocks`](#restore-blocks-with-mina-archive-blocks), +- automate the whole loop with the [missing blocks guardian](#automated-backfill-the-missing-blocks-guardian), +- and apply this workflow to the [hard-fork case](#backfilling-around-a-hard-fork), which has additional requirements. + +## Detect gaps with mina-missing-blocks-auditor + +`mina-missing-blocks-auditor` is the canonical tool for inspecting an archive database. It is shipped in the `mina-archive` Debian package and Docker image alongside `mina-archive-blocks` and `mina-extract-blocks`. + +Run it against your archive database: + +```bash +mina-missing-blocks-auditor \ + --archive-uri postgres://:@:/ +``` + +The auditor performs four independent checks and encodes the results in its exit code (a bitfield, `0` means a healthy database): + +| Bit | Check | What it means | +| --- | --- | --- | +| 0 | Missing blocks | One or more blocks have no parent row in `blocks` (orphan parents). Genesis and the first post-hard-fork block are excluded. | +| 1 | Pending blocks below tip | Pending blocks exist at a height below the highest canonical block — a sign that finalization stalled or rows were lost. | +| 2 | Chain length | The reconstructed canonical chain is shorter than `max(height)` of canonical blocks. | +| 3 | Chain status | A block reachable from the canonical tip has a `chain_status` other than `canonical`. | + +For every missing block the auditor logs a structured entry that names the orphan, its expected parent, and the size of the gap: + +```text +Block has no parent in archive db + block_id: 12345 + state_hash: 3N... + height: 387200 + parent_hash: 3N... + parent_height: 387199 + missing_blocks_gap: 1 +``` + +The `parent_hash` and `parent_height` are exactly what you need to locate the missing block file. This is how the [guardian script](#automated-backfill-the-missing-blocks-guardian) drives its download loop. + +The auditor is read-only — it never modifies the database, so it is safe to run on a live archive node. + +## Sources of block data + +To fill a gap you need the block(s) that the auditor reports as missing in one of two formats: + +- **Precomputed blocks** — the full block JSON the daemon serializes, named `--.json`. Use these whenever possible. +- **Extensional blocks** — a slimmer JSON generated from an existing archive database via `mina-extract-blocks`. Sufficient to repair an archive but lacks the SNARK and other fields that are not stored in PostgreSQL. + +How to *store* these files (uploading blocks from a daemon, saving them from logs, extracting them from another archive, scheduling database dumps) is covered on [Archive Redundancy](./archive-redundancy#back-up-block-data). This page assumes the files already exist somewhere and only documents *where to read them from* for a backfill. + +The two public sources that o1Labs operates as a convenience are: + +| Bucket | Contents | Use for | +| --- | --- | --- | +| [`gs://mina-archive-dumps`](https://console.cloud.google.com/storage/browser/mina-archive-dumps) | Nightly archive-database snapshots (`-archive-dump-_0000.sql.tar.gz`) | Bootstrapping a fresh archive — apply the dump first, then backfill anything produced after the snapshot. | +| [`gs://mina_network_block_data`](https://console.cloud.google.com/storage/browser/mina_network_block_data) | Per-block precomputed JSON files (`--.json`) | The block-by-block input the auditor + guardian walk through to close gaps. | + +:::tip Decentralization + +These o1Labs buckets are a convenience, not a guarantee. The network is healthier when operators publish their own dumps and precomputed-block mirrors — see [Archive Redundancy](./archive-redundancy) for how to set that up. + +::: + +If you mirror the data to your own bucket, point the [guardian](#automated-backfill-the-missing-blocks-guardian) at your URL via the `PRECOMPUTED_BLOCKS_URL` environment variable. + +## Restore blocks with mina-archive-blocks + +Once you have the missing block files, `mina-archive-blocks` writes them into PostgreSQL. Pass `--precomputed` or `--extensional` to match the file format: + +```bash +# Precomputed blocks (from the daemon or the public bucket) +mina-archive-blocks \ + --precomputed \ + --archive-uri postgres://:@:/ \ + mainnet-387200-3NK....json mainnet-387201-3NL....json + +# Extensional blocks (produced by mina-extract-blocks) +mina-archive-blocks \ + --extensional \ + --archive-uri postgres://:@:/ \ + 3NK....json +``` + +Useful flags: + +- `--successful-files ` — append successfully imported filenames to a log. +- `--failed-files ` — append filenames that failed (parse error, schema mismatch, missing dependency). +- `--log-successful false` — suppress per-block success logs when importing many files. + +Re-run `mina-missing-blocks-auditor` after the import to confirm the gap is closed. + +## Automated backfill: the missing blocks guardian + +For continuous archives, the [`missing-blocks-guardian.sh`](https://github.com/MinaProtocol/mina/blob/master/scripts/archive/missing-blocks-guardian.sh) script bundles the audit → download → restore loop. It is included in the `mina-archive` Docker image and is what the [Docker Compose example](./docker-compose) wires up out of the box. + +The guardian has three modes: + +| Subcommand | Behavior | +| --- | --- | +| `audit` | Run the auditor once and exit. Returns whether the database is healthy without modifying anything. Use this from monitoring/alerting (e.g. a Kubernetes liveness check or a cron-driven Prometheus exporter). | +| `single-run` | Audit, then walk back from each orphan parent, download the corresponding precomputed block from `PRECOMPUTED_BLOCKS_URL`, import it with `mina-archive-blocks`, and repeat until the database is clean. Exits when done. Right for one-off repairs and bootstrapping. | +| `daemon` | The same recovery loop as `single-run`, but kept running forever. Re-audits every `TIMEOUT` seconds (default 600); when a repair completes it sleeps `6 × TIMEOUT` before checking again. Right for production archive nodes. | + +Required environment variables: + +| Variable | Purpose | +| --- | --- | +| `DB_USERNAME`, `PGPASSWORD`, `DB_HOST`, `DB_PORT`, `DB_NAME` | PostgreSQL connection for the archive database. | +| `PRECOMPUTED_BLOCKS_URL` | Base URL of the precomputed-blocks bucket (e.g. `https://storage.googleapis.com/mina_network_block_data` or your own mirror). | +| `MINA_NETWORK` | Network name used as a filename prefix (`mainnet`, `devnet`, ...). | + +Optional overrides (`MISSING_BLOCKS_AUDITOR`, `ARCHIVE_BLOCKS`, `BLOCKS_FORMAT`, `TIMEOUT`) are documented in the script's `--help` output. + +A minimal invocation: + +```bash +export DB_USERNAME=postgres DB_HOST=localhost DB_PORT=5432 DB_NAME=archive +export PGPASSWORD=postgres +export PRECOMPUTED_BLOCKS_URL=https://storage.googleapis.com/mina_network_block_data +export MINA_NETWORK=mainnet + +./missing-blocks-guardian.sh single-run +``` + +## Backfilling around a hard fork + +A hard fork creates a fresh chain that starts at its own *first canonical block* — the block whose `global_slot_since_hard_fork = 0`. Before the new chain begins producing pending blocks, an archive database upgraded from the previous network has a discontinuity that the auditor cannot repair by itself. + +The missing-blocks workflow **does work** across a hard fork, but it relies on the auditor's gap-only model. The auditor fills gaps *between existing blocks* — it does not invent new endpoints. For the workflow to succeed, the archive database must satisfy two conditions: + +1. **The first canonical block of the new fork is already in the database.** This is the post-fork genesis-equivalent. It is typically inserted either by running the upgrade script that comes with the new release, or by importing a post-fork archive dump that already contains it. Without this row, the auditor has nothing on the new chain to anchor a backfill against. +2. **At least one block at or above your target tip height (canonical or pending) is also already in the database.** The auditor walks *backwards* from existing blocks toward their missing parents; if the high end of the new chain is absent, there is no orphan for the auditor to report and no parent-hash for the guardian to download. + +With both endpoints present, the guardian's `single-run` (or `daemon`) mode will pull every precomputed block in between from `PRECOMPUTED_BLOCKS_URL` and stitch the new chain together. + +For hard-fork-specific precomputed-blocks distributions, o1Labs typically publishes a dedicated bucket alongside the release (for example, `gs://mesa-hf-precomputed-blocks` for the Mesa upgrade). Check the network-upgrade documentation for the bucket name and version pin matching your fork, set `PRECOMPUTED_BLOCKS_URL` and `MINA_NETWORK` to that bucket and network name, and run the guardian as above. + +:::caution What this workflow does *not* do + +- It does not upgrade your archive schema. Schema changes that accompany a hard fork (for example, Berkeley → Mesa) must be applied with the upgrade scripts shipped in the matching `mina-archive` release before any backfill is attempted. See [Mesa Archive Upgrade](../../network-upgrades/mesa/archive-upgrade) for the canonical procedure. +- It does not migrate historical (pre-fork) data into the new schema — that is handled separately by the migration tooling for the network upgrade. +- It does not validate the contents of imported blocks against a running daemon. Use [`mina-replayer`](./archive-redundancy) afterwards if you need ledger-level verification. + +::: + +## Related + +- [Archive Redundancy](./archive-redundancy) — the companion page on how to *store* precomputed blocks, extensional blocks, and database dumps so a backfill source always exists. This page covers what to do once those backups are in place. +- [Docker Compose example](./docker-compose) — a ready-made stack that runs the guardian alongside a bootstrap-from-dump postgres and a mina-archive process. +- [Archive Nodes Getting Started](./getting-started) — initial setup of the archive database and `mina-archive` process. + --- url: /node-operators/archive-node/docker-compose --- diff --git a/static/llms.txt b/static/llms.txt index a14913161..29e03cea4 100644 --- a/static/llms.txt +++ b/static/llms.txt @@ -109,6 +109,7 @@ This file is auto-generated from `sidebars.js` and the frontmatter of each docum - [Archive Node](https://docs.minaprotocol.com/node-operators/archive-node): What are Archive nodes. - [Archive Nodes Getting Started](https://docs.minaprotocol.com/node-operators/archive-node/getting-started): Mina archive nodes maintain historical information about the network, block, and transactions. A zkApp can retrieve events and actions from one or more Mina archive nodes. - [Archive Redundancy](https://docs.minaprotocol.com/node-operators/archive-node/archive-redundancy): Strategies for archive node redundancy, backup, and data recovery. +- [Backfilling Missing Blocks](https://docs.minaprotocol.com/node-operators/archive-node/backfilling-missing-blocks): How to detect and repair gaps in a Mina archive database using archive dumps, precomputed blocks, mina-archive-blocks, and the mina-missing-blocks-auditor. - [Docker Compose Archive](https://docs.minaprotocol.com/node-operators/archive-node/docker-compose): Example of how to run a Mina archive node using Docker Compose. - [Seed Peers](https://docs.minaprotocol.com/node-operators/seed-peers): What are seed peers - [Seed Peers Getting Started](https://docs.minaprotocol.com/node-operators/seed-peers/getting-started): How to run a seed peer. Seed Peers help connect to the network. From 9c07b1754fcc2289266dfc58ada0508555592e4e Mon Sep 17 00:00:00 2001 From: dkijania Date: Wed, 27 May 2026 18:32:41 +0200 Subject: [PATCH 2/5] docs(archive-node): lead with mina-missing-blocks-guardian, slim hard-fork section Restructures the backfilling page to advertise the guardian first (installed as mina-missing-blocks-guardian from the mina-archive Debian package and Docker image) and demotes mina-missing-blocks-auditor + mina-archive-blocks into an "individual tools" section for setups where the guardian's defaults are not a good fit. Rewrites the hard-fork subsection to reflect what actually happens: the chain is continuous across the fork, but operators commonly miss the fork window or join late and need to patch in post-fork blocks. The only extra precondition for the guardian to work is that the first canonical post-fork block (global_slot_since_hard_fork = 0) is already in the database. Drops the hard-fork-specific bucket paragraph and the caution block per review. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../backfilling-missing-blocks.mdx | 160 ++++++++---------- static/llms-full.txt | 160 ++++++++---------- 2 files changed, 150 insertions(+), 170 deletions(-) diff --git a/docs/node-operators/archive-node/backfilling-missing-blocks.mdx b/docs/node-operators/archive-node/backfilling-missing-blocks.mdx index 4e292c12f..a5d7add57 100644 --- a/docs/node-operators/archive-node/backfilling-missing-blocks.mdx +++ b/docs/node-operators/archive-node/backfilling-missing-blocks.mdx @@ -15,55 +15,50 @@ keywords: # Backfilling Missing Blocks -A Mina archive database can develop gaps whenever the daemon cannot reach the `mina-archive` RPC endpoint as a new block is added to its transition frontier. The daemon ships each new breadcrumb to the archive process in a fire-and-forget loop: on a failed RPC it retries up to 5 times back-to-back (no backoff), logs a warning, and moves on to the next block. Crucially, **the daemon does not persist missed blocks or queue them for later delivery** — once the 5 retries are exhausted, that block is gone from the daemon's perspective and only blocks added *after* the archive reconnects will be archived. - -This means even a short archive-process restart, a transient network blip beyond a few hundred milliseconds, or any RPC error during block ingestion can leave a permanent gap. The `k`-block transition-frontier retention used for consensus does *not* re-ship blocks to the archive on reconnect. +A Mina archive database can develop gaps whenever the daemon or the `mina-archive` process is unavailable while a new block is being added — for example a process restart, a network blip between the daemon and the archive, or the archive process being briefly down. The daemon does not persist missed blocks for later delivery, so any block that lands during the outage will be missing from the database until you backfill it from another source. Because the archive stores only incremental changes, even a single missing block breaks chain continuity and makes downstream queries (balances, Rosetta, zkApp event/action lookups, replayer runs) unreliable. This page explains how to: -- detect gaps with [`mina-missing-blocks-auditor`](#detect-gaps-with-mina-missing-blocks-auditor), -- obtain the block data needed to repair them (see [Sources of block data](#sources-of-block-data)), -- write that data back into PostgreSQL with [`mina-archive-blocks`](#restore-blocks-with-mina-archive-blocks), -- automate the whole loop with the [missing blocks guardian](#automated-backfill-the-missing-blocks-guardian), -- and apply this workflow to the [hard-fork case](#backfilling-around-a-hard-fork), which has additional requirements. +- detect and repair gaps automatically with the [missing-blocks-guardian](#recommended-the-missing-blocks-guardian) script (recommended for most setups), +- obtain the block data the repair pulls from (see [Sources of block data](#sources-of-block-data)), +- and reach for the [individual tools](#individual-tools) when the guardian's defaults are not a good fit. -## Detect gaps with mina-missing-blocks-auditor +## Recommended: the missing-blocks-guardian -`mina-missing-blocks-auditor` is the canonical tool for inspecting an archive database. It is shipped in the `mina-archive` Debian package and Docker image alongside `mina-archive-blocks` and `mina-extract-blocks`. +For almost every archive setup, the right tool is `mina-missing-blocks-guardian`. It is shipped in the `mina-archive` Debian package and Docker image, and wraps the auditor and the block-importer into a single loop — detect gap → download the missing precomputed block → import it → re-check. The [Docker Compose example](./docker-compose) wires it up out of the box. -Run it against your archive database: +The guardian has three modes: -```bash -mina-missing-blocks-auditor \ - --archive-uri postgres://:@:/ -``` +| Subcommand | Behavior | +| --- | --- | +| `audit` | Run the check once and exit. Returns whether the database is healthy without modifying anything. Use this from monitoring / alerting (e.g. a Kubernetes liveness check or a cron-driven Prometheus exporter). | +| `single-run` | Detect gaps, then walk back from each orphan parent, download the corresponding precomputed block from `PRECOMPUTED_BLOCKS_URL`, import it, and repeat until the database is clean. Exits when done. Right for one-off repairs and bootstrapping. | +| `daemon` | The same recovery loop as `single-run`, but kept running forever. Re-audits every `TIMEOUT` seconds (default 600); when a repair completes it sleeps `6 × TIMEOUT` before checking again. Right for production archive nodes. | -The auditor performs four independent checks and encodes the results in its exit code (a bitfield, `0` means a healthy database): +Required environment variables: -| Bit | Check | What it means | -| --- | --- | --- | -| 0 | Missing blocks | One or more blocks have no parent row in `blocks` (orphan parents). Genesis and the first post-hard-fork block are excluded. | -| 1 | Pending blocks below tip | Pending blocks exist at a height below the highest canonical block — a sign that finalization stalled or rows were lost. | -| 2 | Chain length | The reconstructed canonical chain is shorter than `max(height)` of canonical blocks. | -| 3 | Chain status | A block reachable from the canonical tip has a `chain_status` other than `canonical`. | +| Variable | Purpose | +| --- | --- | +| `DB_USERNAME`, `PGPASSWORD`, `DB_HOST`, `DB_PORT`, `DB_NAME` | PostgreSQL connection for the archive database. | +| `PRECOMPUTED_BLOCKS_URL` | Base URL of the precomputed-blocks bucket (e.g. `https://storage.googleapis.com/mina_network_block_data` or your own mirror). | +| `MINA_NETWORK` | Network name used as a filename prefix (`mainnet`, `devnet`, ...). | -For every missing block the auditor logs a structured entry that names the orphan, its expected parent, and the size of the gap: +Optional overrides (`MISSING_BLOCKS_AUDITOR`, `ARCHIVE_BLOCKS`, `BLOCKS_FORMAT`, `TIMEOUT`) are documented in the script's `--help` output. -```text -Block has no parent in archive db - block_id: 12345 - state_hash: 3N... - height: 387200 - parent_hash: 3N... - parent_height: 387199 - missing_blocks_gap: 1 -``` +A minimal one-shot repair: -The `parent_hash` and `parent_height` are exactly what you need to locate the missing block file. This is how the [guardian script](#automated-backfill-the-missing-blocks-guardian) drives its download loop. +```bash +export DB_USERNAME=postgres DB_HOST=localhost DB_PORT=5432 DB_NAME=archive +export PGPASSWORD=postgres +export PRECOMPUTED_BLOCKS_URL=https://storage.googleapis.com/mina_network_block_data +export MINA_NETWORK=mainnet + +mina-missing-blocks-guardian single-run +``` -The auditor is read-only — it never modifies the database, so it is safe to run on a live archive node. +Swap `single-run` for `daemon` to run the loop continuously alongside your archive node, or `audit` to plug it into health-checking. ## Sources of block data @@ -87,9 +82,50 @@ These o1Labs buckets are a convenience, not a guarantee. The network is healthie ::: -If you mirror the data to your own bucket, point the [guardian](#automated-backfill-the-missing-blocks-guardian) at your URL via the `PRECOMPUTED_BLOCKS_URL` environment variable. +If you mirror the data to your own bucket, point the guardian at your URL via the `PRECOMPUTED_BLOCKS_URL` environment variable. -## Restore blocks with mina-archive-blocks +## Individual tools + +The guardian script is a thin wrapper around two underlying binaries shipped in the `mina-archive` Debian package and Docker image: `mina-missing-blocks-auditor` (detect) and `mina-archive-blocks` (restore). Most operators never need to invoke them directly — but the details are useful when: + +- you want to plug detection into an existing monitoring pipeline rather than run the guardian's loop, +- you are importing extensional blocks produced by `mina-extract-blocks` instead of precomputed blocks from a bucket, +- you are debugging a repair that the guardian could not complete on its own, +- or your setup deviates enough from the assumed shape that the guardian's defaults do not apply (custom block-name layout, non-Postgres staging, etc.). + +### Detecting gaps with mina-missing-blocks-auditor + +`mina-missing-blocks-auditor` reports on the health of the archive database. It is read-only and safe to run against a live node. + +```bash +mina-missing-blocks-auditor \ + --archive-uri postgres://:@:/ +``` + +It performs four independent checks and encodes the results in its exit code (a bitfield, `0` means a healthy database): + +| Bit | Check | What it means | +| --- | --- | --- | +| 0 | Missing blocks | One or more blocks have no parent row in `blocks` (orphan parents). Genesis and the first post-hard-fork block are excluded. | +| 1 | Pending blocks below tip | Pending blocks exist at a height below the highest canonical block — a sign that finalization stalled or rows were lost. | +| 2 | Chain length | The reconstructed canonical chain is shorter than `max(height)` of canonical blocks. | +| 3 | Chain status | A block reachable from the canonical tip has a `chain_status` other than `canonical`. | + +For every missing block the auditor logs a structured entry that names the orphan, its expected parent, and the size of the gap: + +```text +Block has no parent in archive db + block_id: 12345 + state_hash: 3N... + height: 387200 + parent_hash: 3N... + parent_height: 387199 + missing_blocks_gap: 1 +``` + +The `parent_hash` and `parent_height` are exactly what you need to locate the missing block file — this is how the guardian drives its download loop. + +### Restoring blocks with mina-archive-blocks Once you have the missing block files, `mina-archive-blocks` writes them into PostgreSQL. Pass `--precomputed` or `--extensional` to match the file format: @@ -115,59 +151,13 @@ Useful flags: Re-run `mina-missing-blocks-auditor` after the import to confirm the gap is closed. -## Automated backfill: the missing blocks guardian - -For continuous archives, the [`missing-blocks-guardian.sh`](https://github.com/MinaProtocol/mina/blob/master/scripts/archive/missing-blocks-guardian.sh) script bundles the audit → download → restore loop. It is included in the `mina-archive` Docker image and is what the [Docker Compose example](./docker-compose) wires up out of the box. - -The guardian has three modes: - -| Subcommand | Behavior | -| --- | --- | -| `audit` | Run the auditor once and exit. Returns whether the database is healthy without modifying anything. Use this from monitoring/alerting (e.g. a Kubernetes liveness check or a cron-driven Prometheus exporter). | -| `single-run` | Audit, then walk back from each orphan parent, download the corresponding precomputed block from `PRECOMPUTED_BLOCKS_URL`, import it with `mina-archive-blocks`, and repeat until the database is clean. Exits when done. Right for one-off repairs and bootstrapping. | -| `daemon` | The same recovery loop as `single-run`, but kept running forever. Re-audits every `TIMEOUT` seconds (default 600); when a repair completes it sleeps `6 × TIMEOUT` before checking again. Right for production archive nodes. | - -Required environment variables: - -| Variable | Purpose | -| --- | --- | -| `DB_USERNAME`, `PGPASSWORD`, `DB_HOST`, `DB_PORT`, `DB_NAME` | PostgreSQL connection for the archive database. | -| `PRECOMPUTED_BLOCKS_URL` | Base URL of the precomputed-blocks bucket (e.g. `https://storage.googleapis.com/mina_network_block_data` or your own mirror). | -| `MINA_NETWORK` | Network name used as a filename prefix (`mainnet`, `devnet`, ...). | - -Optional overrides (`MISSING_BLOCKS_AUDITOR`, `ARCHIVE_BLOCKS`, `BLOCKS_FORMAT`, `TIMEOUT`) are documented in the script's `--help` output. - -A minimal invocation: - -```bash -export DB_USERNAME=postgres DB_HOST=localhost DB_PORT=5432 DB_NAME=archive -export PGPASSWORD=postgres -export PRECOMPUTED_BLOCKS_URL=https://storage.googleapis.com/mina_network_block_data -export MINA_NETWORK=mainnet - -./missing-blocks-guardian.sh single-run -``` - ## Backfilling around a hard fork -A hard fork creates a fresh chain that starts at its own *first canonical block* — the block whose `global_slot_since_hard_fork = 0`. Before the new chain begins producing pending blocks, an archive database upgraded from the previous network has a discontinuity that the auditor cannot repair by itself. +The chain itself is continuous across a hard fork — but it is a common case that an archive operator is not online at the moment the fork happens, or joins the network later, and therefore has to patch in the post-fork blocks they missed. Backfilling around a hard fork uses exactly the same workflow as any other gap, with one extra precondition: -The missing-blocks workflow **does work** across a hard fork, but it relies on the auditor's gap-only model. The auditor fills gaps *between existing blocks* — it does not invent new endpoints. For the workflow to succeed, the archive database must satisfy two conditions: +- **The first canonical block of the new fork must already be in the database.** This is the block whose `global_slot_since_hard_fork = 0`. It is typically inserted either by running the upgrade script that comes with the new release, or by importing a post-fork archive dump that already contains it. Without this row, the auditor has no anchor on the new chain to walk back from, and the guardian cannot stitch the post-fork blocks together. -1. **The first canonical block of the new fork is already in the database.** This is the post-fork genesis-equivalent. It is typically inserted either by running the upgrade script that comes with the new release, or by importing a post-fork archive dump that already contains it. Without this row, the auditor has nothing on the new chain to anchor a backfill against. -2. **At least one block at or above your target tip height (canonical or pending) is also already in the database.** The auditor walks *backwards* from existing blocks toward their missing parents; if the high end of the new chain is absent, there is no orphan for the auditor to report and no parent-hash for the guardian to download. - -With both endpoints present, the guardian's `single-run` (or `daemon`) mode will pull every precomputed block in between from `PRECOMPUTED_BLOCKS_URL` and stitch the new chain together. - -For hard-fork-specific precomputed-blocks distributions, o1Labs typically publishes a dedicated bucket alongside the release (for example, `gs://mesa-hf-precomputed-blocks` for the Mesa upgrade). Check the network-upgrade documentation for the bucket name and version pin matching your fork, set `PRECOMPUTED_BLOCKS_URL` and `MINA_NETWORK` to that bucket and network name, and run the guardian as above. - -:::caution What this workflow does *not* do - -- It does not upgrade your archive schema. Schema changes that accompany a hard fork (for example, Berkeley → Mesa) must be applied with the upgrade scripts shipped in the matching `mina-archive` release before any backfill is attempted. See [Mesa Archive Upgrade](../../network-upgrades/mesa/archive-upgrade) for the canonical procedure. -- It does not migrate historical (pre-fork) data into the new schema — that is handled separately by the migration tooling for the network upgrade. -- It does not validate the contents of imported blocks against a running daemon. Use [`mina-replayer`](./archive-redundancy) afterwards if you need ledger-level verification. - -::: +With the fork block present, run `mina-missing-blocks-guardian single-run` (or `daemon`) as in the [recommended flow](#recommended-the-missing-blocks-guardian) and it will pull every precomputed block in between from `PRECOMPUTED_BLOCKS_URL` until the new chain is contiguous. ## Related diff --git a/static/llms-full.txt b/static/llms-full.txt index 341b8516e..e547d9485 100644 --- a/static/llms-full.txt +++ b/static/llms-full.txt @@ -5285,55 +5285,50 @@ url: /node-operators/archive-node/backfilling-missing-blocks # Backfilling Missing Blocks -A Mina archive database can develop gaps whenever the daemon cannot reach the `mina-archive` RPC endpoint as a new block is added to its transition frontier. The daemon ships each new breadcrumb to the archive process in a fire-and-forget loop: on a failed RPC it retries up to 5 times back-to-back (no backoff), logs a warning, and moves on to the next block. Crucially, **the daemon does not persist missed blocks or queue them for later delivery** — once the 5 retries are exhausted, that block is gone from the daemon's perspective and only blocks added *after* the archive reconnects will be archived. - -This means even a short archive-process restart, a transient network blip beyond a few hundred milliseconds, or any RPC error during block ingestion can leave a permanent gap. The `k`-block transition-frontier retention used for consensus does *not* re-ship blocks to the archive on reconnect. +A Mina archive database can develop gaps whenever the daemon or the `mina-archive` process is unavailable while a new block is being added — for example a process restart, a network blip between the daemon and the archive, or the archive process being briefly down. The daemon does not persist missed blocks for later delivery, so any block that lands during the outage will be missing from the database until you backfill it from another source. Because the archive stores only incremental changes, even a single missing block breaks chain continuity and makes downstream queries (balances, Rosetta, zkApp event/action lookups, replayer runs) unreliable. This page explains how to: -- detect gaps with [`mina-missing-blocks-auditor`](#detect-gaps-with-mina-missing-blocks-auditor), -- obtain the block data needed to repair them (see [Sources of block data](#sources-of-block-data)), -- write that data back into PostgreSQL with [`mina-archive-blocks`](#restore-blocks-with-mina-archive-blocks), -- automate the whole loop with the [missing blocks guardian](#automated-backfill-the-missing-blocks-guardian), -- and apply this workflow to the [hard-fork case](#backfilling-around-a-hard-fork), which has additional requirements. +- detect and repair gaps automatically with the [missing-blocks-guardian](#recommended-the-missing-blocks-guardian) script (recommended for most setups), +- obtain the block data the repair pulls from (see [Sources of block data](#sources-of-block-data)), +- and reach for the [individual tools](#individual-tools) when the guardian's defaults are not a good fit. -## Detect gaps with mina-missing-blocks-auditor +## Recommended: the missing-blocks-guardian -`mina-missing-blocks-auditor` is the canonical tool for inspecting an archive database. It is shipped in the `mina-archive` Debian package and Docker image alongside `mina-archive-blocks` and `mina-extract-blocks`. +For almost every archive setup, the right tool is `mina-missing-blocks-guardian`. It is shipped in the `mina-archive` Debian package and Docker image, and wraps the auditor and the block-importer into a single loop — detect gap → download the missing precomputed block → import it → re-check. The [Docker Compose example](./docker-compose) wires it up out of the box. -Run it against your archive database: +The guardian has three modes: -```bash -mina-missing-blocks-auditor \ - --archive-uri postgres://:@:/ -``` +| Subcommand | Behavior | +| --- | --- | +| `audit` | Run the check once and exit. Returns whether the database is healthy without modifying anything. Use this from monitoring / alerting (e.g. a Kubernetes liveness check or a cron-driven Prometheus exporter). | +| `single-run` | Detect gaps, then walk back from each orphan parent, download the corresponding precomputed block from `PRECOMPUTED_BLOCKS_URL`, import it, and repeat until the database is clean. Exits when done. Right for one-off repairs and bootstrapping. | +| `daemon` | The same recovery loop as `single-run`, but kept running forever. Re-audits every `TIMEOUT` seconds (default 600); when a repair completes it sleeps `6 × TIMEOUT` before checking again. Right for production archive nodes. | -The auditor performs four independent checks and encodes the results in its exit code (a bitfield, `0` means a healthy database): +Required environment variables: -| Bit | Check | What it means | -| --- | --- | --- | -| 0 | Missing blocks | One or more blocks have no parent row in `blocks` (orphan parents). Genesis and the first post-hard-fork block are excluded. | -| 1 | Pending blocks below tip | Pending blocks exist at a height below the highest canonical block — a sign that finalization stalled or rows were lost. | -| 2 | Chain length | The reconstructed canonical chain is shorter than `max(height)` of canonical blocks. | -| 3 | Chain status | A block reachable from the canonical tip has a `chain_status` other than `canonical`. | +| Variable | Purpose | +| --- | --- | +| `DB_USERNAME`, `PGPASSWORD`, `DB_HOST`, `DB_PORT`, `DB_NAME` | PostgreSQL connection for the archive database. | +| `PRECOMPUTED_BLOCKS_URL` | Base URL of the precomputed-blocks bucket (e.g. `https://storage.googleapis.com/mina_network_block_data` or your own mirror). | +| `MINA_NETWORK` | Network name used as a filename prefix (`mainnet`, `devnet`, ...). | -For every missing block the auditor logs a structured entry that names the orphan, its expected parent, and the size of the gap: +Optional overrides (`MISSING_BLOCKS_AUDITOR`, `ARCHIVE_BLOCKS`, `BLOCKS_FORMAT`, `TIMEOUT`) are documented in the script's `--help` output. -```text -Block has no parent in archive db - block_id: 12345 - state_hash: 3N... - height: 387200 - parent_hash: 3N... - parent_height: 387199 - missing_blocks_gap: 1 -``` +A minimal one-shot repair: -The `parent_hash` and `parent_height` are exactly what you need to locate the missing block file. This is how the [guardian script](#automated-backfill-the-missing-blocks-guardian) drives its download loop. +```bash +export DB_USERNAME=postgres DB_HOST=localhost DB_PORT=5432 DB_NAME=archive +export PGPASSWORD=postgres +export PRECOMPUTED_BLOCKS_URL=https://storage.googleapis.com/mina_network_block_data +export MINA_NETWORK=mainnet + +mina-missing-blocks-guardian single-run +``` -The auditor is read-only — it never modifies the database, so it is safe to run on a live archive node. +Swap `single-run` for `daemon` to run the loop continuously alongside your archive node, or `audit` to plug it into health-checking. ## Sources of block data @@ -5357,9 +5352,50 @@ These o1Labs buckets are a convenience, not a guarantee. The network is healthie ::: -If you mirror the data to your own bucket, point the [guardian](#automated-backfill-the-missing-blocks-guardian) at your URL via the `PRECOMPUTED_BLOCKS_URL` environment variable. +If you mirror the data to your own bucket, point the guardian at your URL via the `PRECOMPUTED_BLOCKS_URL` environment variable. -## Restore blocks with mina-archive-blocks +## Individual tools + +The guardian script is a thin wrapper around two underlying binaries shipped in the `mina-archive` Debian package and Docker image: `mina-missing-blocks-auditor` (detect) and `mina-archive-blocks` (restore). Most operators never need to invoke them directly — but the details are useful when: + +- you want to plug detection into an existing monitoring pipeline rather than run the guardian's loop, +- you are importing extensional blocks produced by `mina-extract-blocks` instead of precomputed blocks from a bucket, +- you are debugging a repair that the guardian could not complete on its own, +- or your setup deviates enough from the assumed shape that the guardian's defaults do not apply (custom block-name layout, non-Postgres staging, etc.). + +### Detecting gaps with mina-missing-blocks-auditor + +`mina-missing-blocks-auditor` reports on the health of the archive database. It is read-only and safe to run against a live node. + +```bash +mina-missing-blocks-auditor \ + --archive-uri postgres://:@:/ +``` + +It performs four independent checks and encodes the results in its exit code (a bitfield, `0` means a healthy database): + +| Bit | Check | What it means | +| --- | --- | --- | +| 0 | Missing blocks | One or more blocks have no parent row in `blocks` (orphan parents). Genesis and the first post-hard-fork block are excluded. | +| 1 | Pending blocks below tip | Pending blocks exist at a height below the highest canonical block — a sign that finalization stalled or rows were lost. | +| 2 | Chain length | The reconstructed canonical chain is shorter than `max(height)` of canonical blocks. | +| 3 | Chain status | A block reachable from the canonical tip has a `chain_status` other than `canonical`. | + +For every missing block the auditor logs a structured entry that names the orphan, its expected parent, and the size of the gap: + +```text +Block has no parent in archive db + block_id: 12345 + state_hash: 3N... + height: 387200 + parent_hash: 3N... + parent_height: 387199 + missing_blocks_gap: 1 +``` + +The `parent_hash` and `parent_height` are exactly what you need to locate the missing block file — this is how the guardian drives its download loop. + +### Restoring blocks with mina-archive-blocks Once you have the missing block files, `mina-archive-blocks` writes them into PostgreSQL. Pass `--precomputed` or `--extensional` to match the file format: @@ -5385,59 +5421,13 @@ Useful flags: Re-run `mina-missing-blocks-auditor` after the import to confirm the gap is closed. -## Automated backfill: the missing blocks guardian - -For continuous archives, the [`missing-blocks-guardian.sh`](https://github.com/MinaProtocol/mina/blob/master/scripts/archive/missing-blocks-guardian.sh) script bundles the audit → download → restore loop. It is included in the `mina-archive` Docker image and is what the [Docker Compose example](./docker-compose) wires up out of the box. - -The guardian has three modes: - -| Subcommand | Behavior | -| --- | --- | -| `audit` | Run the auditor once and exit. Returns whether the database is healthy without modifying anything. Use this from monitoring/alerting (e.g. a Kubernetes liveness check or a cron-driven Prometheus exporter). | -| `single-run` | Audit, then walk back from each orphan parent, download the corresponding precomputed block from `PRECOMPUTED_BLOCKS_URL`, import it with `mina-archive-blocks`, and repeat until the database is clean. Exits when done. Right for one-off repairs and bootstrapping. | -| `daemon` | The same recovery loop as `single-run`, but kept running forever. Re-audits every `TIMEOUT` seconds (default 600); when a repair completes it sleeps `6 × TIMEOUT` before checking again. Right for production archive nodes. | - -Required environment variables: - -| Variable | Purpose | -| --- | --- | -| `DB_USERNAME`, `PGPASSWORD`, `DB_HOST`, `DB_PORT`, `DB_NAME` | PostgreSQL connection for the archive database. | -| `PRECOMPUTED_BLOCKS_URL` | Base URL of the precomputed-blocks bucket (e.g. `https://storage.googleapis.com/mina_network_block_data` or your own mirror). | -| `MINA_NETWORK` | Network name used as a filename prefix (`mainnet`, `devnet`, ...). | - -Optional overrides (`MISSING_BLOCKS_AUDITOR`, `ARCHIVE_BLOCKS`, `BLOCKS_FORMAT`, `TIMEOUT`) are documented in the script's `--help` output. - -A minimal invocation: - -```bash -export DB_USERNAME=postgres DB_HOST=localhost DB_PORT=5432 DB_NAME=archive -export PGPASSWORD=postgres -export PRECOMPUTED_BLOCKS_URL=https://storage.googleapis.com/mina_network_block_data -export MINA_NETWORK=mainnet - -./missing-blocks-guardian.sh single-run -``` - ## Backfilling around a hard fork -A hard fork creates a fresh chain that starts at its own *first canonical block* — the block whose `global_slot_since_hard_fork = 0`. Before the new chain begins producing pending blocks, an archive database upgraded from the previous network has a discontinuity that the auditor cannot repair by itself. +The chain itself is continuous across a hard fork — but it is a common case that an archive operator is not online at the moment the fork happens, or joins the network later, and therefore has to patch in the post-fork blocks they missed. Backfilling around a hard fork uses exactly the same workflow as any other gap, with one extra precondition: -The missing-blocks workflow **does work** across a hard fork, but it relies on the auditor's gap-only model. The auditor fills gaps *between existing blocks* — it does not invent new endpoints. For the workflow to succeed, the archive database must satisfy two conditions: +- **The first canonical block of the new fork must already be in the database.** This is the block whose `global_slot_since_hard_fork = 0`. It is typically inserted either by running the upgrade script that comes with the new release, or by importing a post-fork archive dump that already contains it. Without this row, the auditor has no anchor on the new chain to walk back from, and the guardian cannot stitch the post-fork blocks together. -1. **The first canonical block of the new fork is already in the database.** This is the post-fork genesis-equivalent. It is typically inserted either by running the upgrade script that comes with the new release, or by importing a post-fork archive dump that already contains it. Without this row, the auditor has nothing on the new chain to anchor a backfill against. -2. **At least one block at or above your target tip height (canonical or pending) is also already in the database.** The auditor walks *backwards* from existing blocks toward their missing parents; if the high end of the new chain is absent, there is no orphan for the auditor to report and no parent-hash for the guardian to download. - -With both endpoints present, the guardian's `single-run` (or `daemon`) mode will pull every precomputed block in between from `PRECOMPUTED_BLOCKS_URL` and stitch the new chain together. - -For hard-fork-specific precomputed-blocks distributions, o1Labs typically publishes a dedicated bucket alongside the release (for example, `gs://mesa-hf-precomputed-blocks` for the Mesa upgrade). Check the network-upgrade documentation for the bucket name and version pin matching your fork, set `PRECOMPUTED_BLOCKS_URL` and `MINA_NETWORK` to that bucket and network name, and run the guardian as above. - -:::caution What this workflow does *not* do - -- It does not upgrade your archive schema. Schema changes that accompany a hard fork (for example, Berkeley → Mesa) must be applied with the upgrade scripts shipped in the matching `mina-archive` release before any backfill is attempted. See [Mesa Archive Upgrade](../../network-upgrades/mesa/archive-upgrade) for the canonical procedure. -- It does not migrate historical (pre-fork) data into the new schema — that is handled separately by the migration tooling for the network upgrade. -- It does not validate the contents of imported blocks against a running daemon. Use [`mina-replayer`](./archive-redundancy) afterwards if you need ledger-level verification. - -::: +With the fork block present, run `mina-missing-blocks-guardian single-run` (or `daemon`) as in the [recommended flow](#recommended-the-missing-blocks-guardian) and it will pull every precomputed block in between from `PRECOMPUTED_BLOCKS_URL` until the new chain is contiguous. ## Related From 88355a9aafc3b9b906e115dc89da5745225098d8 Mon Sep 17 00:00:00 2001 From: dkijania Date: Wed, 27 May 2026 19:10:27 +0200 Subject: [PATCH 3/5] docs(archive-node): use public storage.googleapis.com URLs for o1Labs buckets Replaces the auth-walled cloud-console "storage/browser" links with the public storage.googleapis.com bucket URLs that already appear in the curl examples on the same pages, so the bucket references are consistent and work without a Google account. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/node-operators/archive-node/archive-redundancy.mdx | 2 +- .../archive-node/backfilling-missing-blocks.mdx | 4 ++-- static/llms-full.txt | 6 +++--- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/node-operators/archive-node/archive-redundancy.mdx b/docs/node-operators/archive-node/archive-redundancy.mdx index af132e4df..5f73baa4d 100644 --- a/docs/node-operators/archive-node/archive-redundancy.mdx +++ b/docs/node-operators/archive-node/archive-redundancy.mdx @@ -101,7 +101,7 @@ pg_dump -U -Fp | gzip > archive-$(date +%F).sql.gz Schedule this regularly (a nightly cron is typical) and publish the dumps to object storage that your downstream consumers can reach. -O1Labs publishes nightly dumps of its public-network archives to the [mina-archive-dumps](https://console.cloud.google.com/storage/browser/mina-archive-dumps) GCS bucket as a convenience: +O1Labs publishes nightly dumps of its public-network archives to the [mina-archive-dumps](https://storage.googleapis.com/mina-archive-dumps/) GCS bucket as a convenience: ```bash # Mainnet diff --git a/docs/node-operators/archive-node/backfilling-missing-blocks.mdx b/docs/node-operators/archive-node/backfilling-missing-blocks.mdx index a5d7add57..d6d5fe7ba 100644 --- a/docs/node-operators/archive-node/backfilling-missing-blocks.mdx +++ b/docs/node-operators/archive-node/backfilling-missing-blocks.mdx @@ -73,8 +73,8 @@ The two public sources that o1Labs operates as a convenience are: | Bucket | Contents | Use for | | --- | --- | --- | -| [`gs://mina-archive-dumps`](https://console.cloud.google.com/storage/browser/mina-archive-dumps) | Nightly archive-database snapshots (`-archive-dump-_0000.sql.tar.gz`) | Bootstrapping a fresh archive — apply the dump first, then backfill anything produced after the snapshot. | -| [`gs://mina_network_block_data`](https://console.cloud.google.com/storage/browser/mina_network_block_data) | Per-block precomputed JSON files (`--.json`) | The block-by-block input the auditor + guardian walk through to close gaps. | +| [`gs://mina-archive-dumps`](https://storage.googleapis.com/mina-archive-dumps/) | Nightly archive-database snapshots (`-archive-dump-_0000.sql.tar.gz`) | Bootstrapping a fresh archive — apply the dump first, then backfill anything produced after the snapshot. | +| [`gs://mina_network_block_data`](https://storage.googleapis.com/mina_network_block_data/) | Per-block precomputed JSON files (`--.json`) | The block-by-block input the auditor + guardian walk through to close gaps. | :::tip Decentralization diff --git a/static/llms-full.txt b/static/llms-full.txt index e547d9485..13bd075f4 100644 --- a/static/llms-full.txt +++ b/static/llms-full.txt @@ -5249,7 +5249,7 @@ pg_dump -U -Fp | gzip > archive-$(date +%F).sql.gz Schedule this regularly (a nightly cron is typical) and publish the dumps to object storage that your downstream consumers can reach. -O1Labs publishes nightly dumps of its public-network archives to the [mina-archive-dumps](https://console.cloud.google.com/storage/browser/mina-archive-dumps) GCS bucket as a convenience: +O1Labs publishes nightly dumps of its public-network archives to the [mina-archive-dumps](https://storage.googleapis.com/mina-archive-dumps/) GCS bucket as a convenience: ```bash # Mainnet @@ -5343,8 +5343,8 @@ The two public sources that o1Labs operates as a convenience are: | Bucket | Contents | Use for | | --- | --- | --- | -| [`gs://mina-archive-dumps`](https://console.cloud.google.com/storage/browser/mina-archive-dumps) | Nightly archive-database snapshots (`-archive-dump-_0000.sql.tar.gz`) | Bootstrapping a fresh archive — apply the dump first, then backfill anything produced after the snapshot. | -| [`gs://mina_network_block_data`](https://console.cloud.google.com/storage/browser/mina_network_block_data) | Per-block precomputed JSON files (`--.json`) | The block-by-block input the auditor + guardian walk through to close gaps. | +| [`gs://mina-archive-dumps`](https://storage.googleapis.com/mina-archive-dumps/) | Nightly archive-database snapshots (`-archive-dump-_0000.sql.tar.gz`) | Bootstrapping a fresh archive — apply the dump first, then backfill anything produced after the snapshot. | +| [`gs://mina_network_block_data`](https://storage.googleapis.com/mina_network_block_data/) | Per-block precomputed JSON files (`--.json`) | The block-by-block input the auditor + guardian walk through to close gaps. | :::tip Decentralization From f107a8a5a4cb2c01eef04c4dfb19a911eaa028fe Mon Sep 17 00:00:00 2001 From: Dariusz Kijania Date: Wed, 27 May 2026 21:13:24 +0200 Subject: [PATCH 4/5] Update docs/node-operators/archive-node/archive-redundancy.mdx Co-authored-by: SanabriaRusso --- docs/node-operators/archive-node/archive-redundancy.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/node-operators/archive-node/archive-redundancy.mdx b/docs/node-operators/archive-node/archive-redundancy.mdx index 5f73baa4d..960838097 100644 --- a/docs/node-operators/archive-node/archive-redundancy.mdx +++ b/docs/node-operators/archive-node/archive-redundancy.mdx @@ -101,7 +101,7 @@ pg_dump -U -Fp | gzip > archive-$(date +%F).sql.gz Schedule this regularly (a nightly cron is typical) and publish the dumps to object storage that your downstream consumers can reach. -O1Labs publishes nightly dumps of its public-network archives to the [mina-archive-dumps](https://storage.googleapis.com/mina-archive-dumps/) GCS bucket as a convenience: +o1Labs publishes nightly dumps of its public-network archives to the [mina-archive-dumps](https://storage.googleapis.com/mina-archive-dumps/) GCS bucket as a convenience: ```bash # Mainnet From a51ac10af2dcd6f6f30e63c4ba942c93bb4f440d Mon Sep 17 00:00:00 2001 From: dkijania Date: Wed, 27 May 2026 21:22:21 +0200 Subject: [PATCH 5/5] regenerate llms-full.txt after o1Labs casing fix Co-Authored-By: Claude Opus 4.7 (1M context) --- static/llms-full.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/static/llms-full.txt b/static/llms-full.txt index 13bd075f4..ba345473b 100644 --- a/static/llms-full.txt +++ b/static/llms-full.txt @@ -5249,7 +5249,7 @@ pg_dump -U -Fp | gzip > archive-$(date +%F).sql.gz Schedule this regularly (a nightly cron is typical) and publish the dumps to object storage that your downstream consumers can reach. -O1Labs publishes nightly dumps of its public-network archives to the [mina-archive-dumps](https://storage.googleapis.com/mina-archive-dumps/) GCS bucket as a convenience: +o1Labs publishes nightly dumps of its public-network archives to the [mina-archive-dumps](https://storage.googleapis.com/mina-archive-dumps/) GCS bucket as a convenience: ```bash # Mainnet