From 1d9718c598c8a9114e995cf33835263ba93c9f62 Mon Sep 17 00:00:00 2001 From: Sameh Abouel-saad Date: Thu, 2 Apr 2026 00:32:21 +0200 Subject: [PATCH] docs: update all project documentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - README.md: Node v16→v20, src/types described as auto-generated, added Makefile targets and docs/typeChanges.md to project layout, restored architecture diagram - docs/readme.md: added links to typeChanges.md and release_process.md - docs/development.md: added sections for type generation workflow, GraphQL schema modification, and DB reset; restored log screenshots - docs/production.md: documented all env vars for indexer and processor, added architecture overview, improved resync section - docs/release_process.md: documented make version-bump, fixed chart paths (indexer/chart.yaml→indexer/chart/Chart.yaml), added CI auto-publish note - indexer/readme.md: documented START_HEIGHT, reformatted container list as table - indexer/development.md: fixed title (types.json→typesBundle.json), clarified typesBundle is frozen with link to typeChanges.md - indexer/chart/README.md: fixed typo (missing space in helm command) - processor-chart/README.md: fixed wrong path (cd tfchain/graphql/processor-chart→cd processor-chart) - scripts/readme.md: fixed typo (Sripts→Scripts), fixed wrong script name (restart-db.sh→reset-db.sh), added missing scripts (seed-versions.sh, merge-versions.js, init-db.sh) Refs: #119, #120 Co-Authored-By: Claude Opus 4.6 (1M context) --- README.md | 46 +++++---- docs/advanced-development.md | 195 +++++++++++++++++++++++++++++++++++ docs/development.md | 134 +++++++++++++++++++----- docs/production.md | 60 +++++++---- docs/readme.md | 20 +++- docs/release_process.md | 36 ++++++- indexer/chart/README.md | 12 +-- indexer/development.md | 14 ++- indexer/readme.md | 37 ++++--- package.json | 4 +- processor-chart/README.md | 16 ++- scripts/readme.md | 26 ++++- 12 files changed, 485 insertions(+), 115 deletions(-) create mode 100644 docs/advanced-development.md diff --git a/README.md b/README.md index b94e23d..84d0bfb 100644 --- a/README.md +++ b/README.md @@ -1,37 +1,45 @@ -# Tfchain graphql +# TFChain GraphQL -[Subsquid](https://docs.subsquid.io) is used to index and provide a graphql interface on top of tfchain. +[Subsquid](https://docs.subsquid.io) is used to index and provide a GraphQL interface on top of TFChain. ## Concept The substrate events are processed in a multi-step pipeline: - Tfchain => Squid Indexer => Indexer GraphQL gateway => Squid Processor => Database => Query Node GraphQL endpoint + TFChain => Squid Indexer => Indexer GraphQL gateway => Squid Processor => Database => Query Node GraphQL endpoint ![Bird eye overview](https://gblobscdn.gitbook.com/assets%2F-MdI-MAyz-csivC8mmdb%2Fsync%2Fe587479ff22ad79886861487b2734b6556302d10.png?alt=media) ## Prerequisites -* Node v16x +* Node v20+ * Docker -* Docker-compose +* Docker Compose ## Running -see [docs](./docs/readme.md) +See [docs](./docs/readme.md) ## Project layout -- `indexer` - docker-compose setup for the indexer -- `db` - Processor db migration files -- `scripts` - Scripts for generating initial state and development scripts -- `src` - Source - - `mappings` - Mapper functions for the indexer data - - `model` - Generated models from the `schema.graphql` file - - `types` - Type files that require manual edit if the schema changes / or chain types change - - `processor.ts` - Processor entrypoint -- `typegen` - Where the declaration files are generated from (used for development) - - `tfchainVersions.jsonl` - Generated tfchain runtime versions and their data - - `typegen.json` - Typegen config - - `typesBundle.json` - Typegen bundle config -- `schema.graphql` - The graphql schema file, changes to this file will results in changes to the models (`src/models`) \ No newline at end of file +- `indexer/` - Docker Compose setup for the indexer (archive) +- `db/` - Processor database migration files +- `scripts/` - Utility scripts (see [scripts/readme.md](./scripts/readme.md)) +- `src/` - Processor source code + - `mappings/` - Event handler functions that map chain events to database entities + - `model/` - Generated TypeORM models from `schema.graphql` + - `types/` - Auto-generated type definitions (do not edit manually — run `make typegen`) + - `processor.ts` - Processor entrypoint: event subscription and dispatch +- `typegen/` - Type generation infrastructure + - `tfchainVersions.jsonl` - Append-only log of runtime metadata from all TFChain networks + - `typegen.json` - Typegen config: which events to generate types for + - `typesBundle.json` - Frozen pre-V14 type mappings (do not edit for new runtime versions) +- `docs/` - Documentation + - [typeChanges.md](./docs/typeChanges.md) - How to handle type changes on chain (adding new runtime versions, resync guidance) + - [development.md](./docs/development.md) - Local development setup + - [production.md](./docs/production.md) - Production deployment + - [release_process.md](./docs/release_process.md) - Release workflow +- `schema.graphql` - GraphQL schema — changes here regenerate `src/model/` via `yarn codegen` +- `Makefile` - Common tasks: `typegen`, `typegen-add`, `typegen-seed`, `version-bump` +- `processor-chart/` - Helm chart for processor + query node deployment +- `indexer/chart/` - Helm chart for indexer stack deployment diff --git a/docs/advanced-development.md b/docs/advanced-development.md new file mode 100644 index 0000000..ae17256 --- /dev/null +++ b/docs/advanced-development.md @@ -0,0 +1,195 @@ +# Advanced Development Guide + +This document covers the internal architecture of the tfchain_graphql indexer/processor stack. For the day-to-day workflow (adding new runtime versions, resyncing), see [typeChanges.md](./typeChanges.md). + +## 1. Pre-V14 Metadata and the typesBundle + +Substrate runtime metadata comes in different versions. TFChain has two eras: + +- **V12 (pre-V14)**: metadata is NOT self-describing for custom types. The `typesBundle.json` file provides type definitions that tell the decoder how to interpret SCALE-encoded data. +- **V14+**: metadata is self-describing. All type information is embedded in the metadata itself. The typesBundle is not used. + +The boundary differs per network: + +| Network | Pre-V14 specs | First V14 spec | +|---------|--------------|----------------| +| Devnet | 49-67 | 101 | +| QAnet | 61-67 | 104 | +| Testnet | 9-70 | 113 | +| Mainnet | 31-70 | 113 | + +The typesBundle uses `minmax` ranges to define which type definitions apply at which spec versions. For example, `[50, None]` means "from spec 50 onwards until overridden by a later entry." When a type changes (e.g., a new field is added), a new `minmax` entry is added with the updated definition. + +**Event hashes** for pre-V14 specs are computed from the typesBundle type definitions combined with the metadata. This is important: the same Rust struct can produce different event hashes depending on how the typesBundle defines it (field names, field order). + +## 2. The Indexer-Processor Contract + +The data flow from chain to GraphQL API is: + +``` +Chain (SCALE-encoded events) + -> Indexer (substrate-ingest) decodes SCALE using typesBundle -> stores decoded JSON in CockroachDB + -> Gateway serves stored JSON + -> Processor's decodeEvent() reads args[fieldName] from the JSON + -> Processor stores entities in PostgreSQL + -> Query node serves GraphQL API from PostgreSQL +``` + +The processor does NOT decode raw SCALE bytes. It reads pre-decoded JSON from the gateway. The `decodeEvent` method in `@subsquid/substrate-processor` iterates over the event's field definitions and reads `args[fieldName]` from the stored JSON object. + +This means **field names must match** between: +- The typesBundle that the indexer used to decode and store the JSON +- The type definitions the processor expects (derived from the current typesBundle + metadata) + +If the typesBundle is updated but the indexer is not resynced, the stored JSON has field names from the old typesBundle while the processor expects names from the new one. This mismatch causes assertion failures during decoding. + +**Rule: always resync the indexer after changing the typesBundle.** Alternatively, add workaround patches in the mapping handlers to fix the field names before decoding (see topic 4). + +## 3. Cross-Network Metadata Differences + +A given spec version number typically represents the same runtime binary across all networks. This was verified by comparing metadata hex from the firesquid indexers on all 4 networks: + +```graphql +# Query on each network's firesquid +{ metadata(where: {specVersion_eq: 49}) { hex } } +``` + +All shared pre-V14 specs have identical metadata across all networks that deployed them. + +**Exceptions**: specs 125 and 134 have different WASM on devnet vs other networks. Devnet received release candidate builds that were later revised before deployment to qa/test/main. These are V14 specs, so the typesBundle is not involved. Verified that all tracked event hashes are identical despite the metadata differences (the changes are in non-event types). + +The JSONL merge script (`scripts/merge-versions.js`) deduplicates by specVersion, keeping the first entry encountered. Since devnet is seeded first, devnet's metadata wins for conflicts. + +## 4. The `dedicatedFarm:` Colon Bug + +The typesBundle historically had a typo: `"dedicatedFarm:"` (trailing colon) in the Farm struct definition at `[63, None]`. + +- **Introduced**: commit `478ee70` +- **Fixed**: commit `980dd11` +- **Grid deployment updated**: commit `119b5dc` (June 2024) +- **Indexers resynced**: never + +Because the indexers were never resynced, all network indexer snapshots contain decoded JSON with `"dedicatedFarm:"` (with colon) as the field key for pre-V14 Farm events. The current typesBundle (without colon) produces a different event hash, and the processor expects `"dedicatedFarm"` (no colon) as the field name. + +When the processor reads `args["dedicatedFarm"]`, it gets `undefined` because the stored key is `"dedicatedFarm:"`. The SCALE JSON codec then asserts `typeof undefined == "boolean"` and crashes. + +**Workaround** (in the `isV63` branches of `farmStored` and `farmUpdated`): + +```typescript +(item.event.args as any).dedicatedFarm = false +``` + +This adds the expected field name to the args object before decoding. The value `false` is safe because `farmStored` hardcodes `dedicatedFarm = false` anyway. + +**Proper fix**: resync all indexers with the corrected typesBundle so the stored JSON has correct field names. After resync, the workaround can be removed. + +**Note**: the mainnet `grid_deployment` repo still has the old typesBundle with this colon bug. Devnet, qanet, and testnet have the corrected version. + +## 5. How Typegen Assigns Version Labels + +Typegen reads the JSONL file in order (sorted by specVersion). For each tracked event, it: + +1. Computes the event hash at each specVersion by decoding the metadata (using the typesBundle for pre-V14) +2. Compares the hash to the previous entry's hash +3. If the hash changed, generates a new `isVxx`/`asVxx` accessor named after the specVersion +4. If the hash is the same as the previous entry, skips it (consecutive hash dedup) + +This means: +- The JSONL order determines which specVersion gets the "canonical" label for each hash +- The same hash can produce accessors at multiple specs if it appears, changes, then reverts (e.g., Twin hash oscillates at specs 125-127 due to devnet metadata differences) +- Typegen **cannot** handle two JSONL entries with the same specVersion (duplicate TypeScript method names would cause compilation errors) + +The `isVxx` runtime check is `getEventHash(eventName) === 'hash'`. It checks the current block's runtime hash, not the spec version. So `isV9` can match blocks at any spec version as long as the event hash is the same. + +## 6. Network Deployment History + +| Network | Genesis spec | Genesis block | Pre-V14 range | Notes | +|---------|-------------|---------------|---------------|-------| +| Testnet | 9 | 0 | 9-70 | Oldest continuous chain. Has all historical specs. | +| Mainnet | 31 | 0 | 31-70 | Started later than testnet. | +| QAnet | 61 | 0 | 61-67 | Reset. Started from spec 61. | +| Devnet | 49 | 0 | 49-67 | Reset multiple times. Current chain starts at spec 49. Has RC specs 63-67 that are devnet-only. | + +**Spec reuse after resets**: devnet was reset multiple times. Spec numbers 1-48 existed on the old devnet but are gone. The current devnet starts at spec 49. Git history may show commits with the same spec number from different eras. The deployed version is always the commit that bumps `spec_version` in `substrate-node/runtime/src/lib.rs`. + +**Git commit pattern**: developers add features in commits while the runtime still has the old spec version, then a separate commit bumps the spec. The bump commit is what gets built and deployed. When tracing types at a spec version, look at the commit that set `spec_version: XX`, not earlier commits that may show intermediate states. + +**Verifying deployed runtime**: compare metadata hex from firesquid indexers across networks. If the hex matches, the same WASM was deployed. If it differs, the networks have different code at that spec (typically devnet RC vs production release). + +## 7. Debugging Event Decode Failures + +### Symptoms + +- `AssertionError: typeof value == "boolean"` (or "string", "number") +- `AssertionError: The expression evaluated to a falsy value` in codec-json.js +- Processor crash loop at a specific block + +### Step 1: Identify the spec version + +Enable `SQD_DEBUG=sqd:processor:mapping` on the processor (avoid `sqd:processor:*` which floods logs with serialization errors from the node-fetch URLSearchParams bug). Look for the specId in debug output or check which block the processor is stuck on: + +```bash +docker logs processor-container 2>&1 | grep "last processed block" +``` + +Then query the indexer explorer for the spec at that block: +```graphql +{ blocks(where: {height_eq: XXXXX}) { spec { specVersion } } } +``` + +### Step 2: Determine pre-V14 or V14 + +- Devnet: spec < 101 is pre-V14 +- Testnet/Mainnet: spec < 113 is pre-V14 +- QAnet: spec < 104 is pre-V14 + +For V14 failures, the issue is in the auto-generated types. For pre-V14, the typesBundle is involved. + +### Step 3: For pre-V14 failures + +1. Check the typesBundle `minmax` range covering this spec +2. Verify the type definition matches the Rust source at that spec +3. **Check what the indexer stored**: query the firesquid gateway for the event: + ```graphql + { events(where: {name_eq: "TfgridModule.FarmStored", block: {height_eq: XXXXX}}) { args } } + ``` +4. Compare the stored field names with what the processor expects +5. If field names don't match, the indexer was built with a different typesBundle + +### Step 4: Compare metadata across networks + +Query each network's firesquid: + +```graphql +{ metadata(where: {specVersion_eq: XX}) { hex } } +``` + +Compare the hex values directly. Same hex = same WASM. Different hex = different code at the same spec (typically devnet RC). + +### Step 5: Compare event hashes + +Save metadata to temporary JSONL files and run typegen on each to see what event hashes they produce: + +```bash +# Create single-entry JSONL from indexer metadata +# Run typegen with the tracked events +# Compare the getEventHash lines in the output +``` + +### Step 6: Check production code + +If production works but your branch doesn't: + +```bash +diff <(git show origin/production-branch:src/types/events.ts | grep getEventHash | sort) \ + <(grep getEventHash src/types/events.ts | sort) +``` + +Look for removed hash branches or missing workaround patches in the mapping handlers. + +### Common pitfalls + +- **Mixed-version DB contamination**: running two different processor versions against the same PostgreSQL database creates mixed entity ID formats. Always do a full DB reset when switching versions. +- **CockroachDB snapshot extraction**: never use `--strip-components` when extracting indexer snapshots. The tar archive has SST files at root level. +- **Startup race**: on first start, the processor may fail with "relation does not exist" if migrations haven't completed. Docker restart policy recovers this automatically. +- **Shared Docker network DNS**: when both compose stacks share a network, all service names must be unique across both stacks. The indexer uses `cockroachdb` (not `db`) to avoid collision with the processor's PostgreSQL `db` service. diff --git a/docs/development.md b/docs/development.md index 3cd60df..a27c310 100644 --- a/docs/development.md +++ b/docs/development.md @@ -1,74 +1,152 @@ -# Developing on tfchain graphql +# Developing on TFChain GraphQL -Install: +## Install -``` +```bash yarn yarn build ``` ## Local Network -### Run tfchain +### Run TFChain -see https://github.com/threefoldtech/tfchain +See https://github.com/threefoldtech/tfchain ### Run Indexer -Check `indexer/.env` and adjust the websocket endpoint to your local tfchain address. +Check `indexer/.env` and adjust the websocket endpoint to your local TFChain address. -``` +```bash cd indexer -docker-compose up -d +docker compose up -d ``` -Indexer services should now be started, you can check if it's syncing properly by streaming the logs for the indexer: +Indexer services should now be started. Check if it's syncing properly by streaming the ingest logs: -``` +```bash docker logs indexer-ingest-1 -f ``` -You should be able to follow tfchain blocks processing: +You should see TFChain blocks being processed: -![image](https://user-images.githubusercontent.com/73958772/209998096-3d5381d9-97ee-438d-824d-d92d997b42aa.png) +![Indexer logs](https://user-images.githubusercontent.com/73958772/209998096-3d5381d9-97ee-438d-824d-d92d997b42aa.png) -### Run processor +### Run Processor (local, outside Docker) -Check `.env` and adjust the websocket endpoint to your local tfchain address. +Check `.env` and adjust the websocket endpoint and indexer URL to your local setup. -``` +Start the local PostgreSQL container and run the processor: + +```bash yarn build yarn db:up yarn process ``` -You should be able to follow tfchain blocks processing: +You should see TFChain blocks being processed by the processor: + +![Processor logs](https://user-images.githubusercontent.com/73958772/210000023-c575d91a-382e-4fdc-85b3-199a135b493f.png) + +If you make changes, stop the containers before restarting: + +```bash +docker compose down +``` + +### Run GraphQL UI -![image](https://user-images.githubusercontent.com/73958772/210000023-c575d91a-382e-4fdc-85b3-199a135b493f.png) +At this step, running `docker ps` should show the indexer containers running: +![Docker containers](https://user-images.githubusercontent.com/42457449/258668686-cd331bd6-ed80-47ea-87a5-16f88d969025.png) -If you make some changes, don't forget to turn down container before tuning it on again. +Start the query node: +```bash +yarn api ``` -docker-compose down + +Now you can use the GraphQL playground at http://localhost:4000/graphql + +## Adding New Runtime Versions + +When TFChain has a new spec version with type changes, see [typeChanges.md](./typeChanges.md) for the full workflow. The short version: + +```bash +# Point at your local chain (or a remote network via WS_URL=wss://...) +make typegen-add + +# Check what changed in src/types/, add handler branches if needed +yarn build ``` -### Run graphql UI +## Modifying the GraphQL Schema + +If you need to add new entities or fields to the GraphQL API: + +1. Edit `schema.graphql` +2. Regenerate models and create a migration: + ```bash + yarn codegen + yarn build + yarn db:create-migration + ``` +3. Add or update event handlers in `src/mappings/` +4. Register new events in `src/processor.ts` if needed +5. Test locally: + ```bash + yarn db:up + yarn db:migrate + yarn process + ``` + +## Resetting the Processor Database -At this step, by running +### Option A: Volume wipe (recommended) +The simplest and safest approach. Wipes the entire PostgreSQL data directory: + +```bash +# Stop processor and DB +docker compose down processor db + +# Wipe postgres data +rm -rf /path/to/postgres-data/* + +# Restart - processor will run migrations and start from block 0 +docker compose up -d ``` -docker ps + +### Option B: Script (local dev) + +```bash +./scripts/reset-db.sh +yarn process ``` -it should display such list of running containers: +## Debugging -![image](https://user-images.githubusercontent.com/42457449/258668686-cd331bd6-ed80-47ea-87a5-16f88d969025.png) +### Debug logging -Make sure indexer and processor are both listening to tfchain to be able to browse. +Use `SQD_DEBUG=sqd:processor:mapping` for event processing visibility. Avoid `sqd:processor:*` which floods logs with serialization errors from a known node-fetch bug. -``` -yarn api +```bash +# In .env or docker-compose.yml environment: +SQD_DEBUG=sqd:processor:mapping ``` -Now you can use the UI (http://localhost:4000/graphql) and run some tests. +### Check processor progress + +```bash +# Current height +docker exec db-container psql -U postgres -d tfgrid-graphql \ + -c 'SELECT height FROM squid_processor.status;' + +# Entity counts +docker exec db-container psql -U postgres -d tfgrid-graphql -c " + SELECT 'farms' AS entity, count(*) FROM farm + UNION ALL SELECT 'nodes', count(*) FROM node + UNION ALL SELECT 'twins', count(*) FROM twin + UNION ALL SELECT 'contracts', count(*) FROM node_contract; +" +``` diff --git a/docs/production.md b/docs/production.md index 7e6cc61..a8e4eb7 100644 --- a/docs/production.md +++ b/docs/production.md @@ -1,42 +1,66 @@ -# Production setup +# Production Setup ## Requirements -- Tfchain network url. (e.g. `wss://tfchain.dev.grid.tf/ws`) +- TFChain network WebSocket URL (e.g., `wss://tfchain.dev.grid.tf/ws`) - Docker -- Docker-compose +- Docker Compose -## Run the setup +## Architecture -### Indexer +The production stack has two independent layers: -Configure the `.env` file in `./indexer` +1. **Indexer (archive)** — ingests raw blocks from a TFChain node into CockroachDB. Provides a GraphQL gateway for the processor to query block data. +2. **Processor + Query Node** — reads events from the indexer, maps them to domain entities (nodes, farms, contracts, etc.), stores in PostgreSQL, and serves the public GraphQL API. -Set the `WS_URL` to a tfchain network url. +## Run the Setup + +### 1. Indexer + +Configure `indexer/.env`: + +| Variable | Description | Default | +|----------|-------------|---------| +| `WS_ENDPOINT` | TFChain node WebSocket URL | `ws://localhost:9944` | +| `START_HEIGHT` | Block height to start ingesting from. `0` = genesis (full history). Set higher only for testing or partial deployments — the processor will miss events before this height. | `0` | ```bash cd indexer -docker-compose up -d +docker compose up -d ``` -### Processor +See [indexer/readme.md](../indexer/readme.md) for details on the indexer stack containers. -Configure the `.env` file in the root of this repository. +### 2. Processor + Query Node -Set the `WS_URL` to a tfchain network url. +Configure `.env` in the project root: + +| Variable | Description | Default | +|----------|-------------|---------| +| `DB_NAME` | PostgreSQL database name | `tfgrid-graphql` | +| `DB_USER` | PostgreSQL user | `postgres` | +| `DB_PASS` | PostgreSQL password | `postgres` | +| `DB_PORT` | PostgreSQL port | `5432` | +| `INDEXER_ENDPOINT_URL` | Indexer GraphQL gateway URL | `http://localhost:8888/graphql` | +| `WS_URL` | TFChain node WebSocket URL (used for RPC calls) | `ws://localhost:9944` | +| `TYPEORM_LOGGING` | TypeORM log level | `error` | ```bash -docker-compose up -d +docker compose up -d ``` -Now the graphql endpoint is available at `http://localhost:4000/graphql` +The GraphQL endpoint is now available at `http://localhost:4000/graphql` -## Releasing +## Reprocessing -See [release process](./release_process.md) +If a mapping bug is fixed and data needs remapping, reset the processor database and reindex: -## Notes +```bash +docker compose down processor query-node +./scripts/reset-db.sh +docker compose up -d +``` -### Reprocessing of indexer data +This drops and recreates the processor's PostgreSQL database and reindexes from block 0 against the existing indexer data. The indexer does not need to be restarted — it stores raw blocks independently. -Reprocessing of indexer data (running the processor) would be usefull in situation where bugs are found and the data that is mapped would need remapping. This can be done by deleting the processor data and running it from 0 again with the code changes. \ No newline at end of file +See [typeChanges.md](./typeChanges.md) for more detail on when a resync is needed. diff --git a/docs/readme.md b/docs/readme.md index ecaa8cd..63e3396 100644 --- a/docs/readme.md +++ b/docs/readme.md @@ -1,9 +1,21 @@ -# TFchain Graphql Documentation +# TFChain GraphQL Documentation ## Development -see [development](./development.md) +See [development](./development.md) -## Running in Production mode +## Running in Production -see [production](./production.md) \ No newline at end of file +See [production](./production.md) + +## Handling Type Changes + +See [typeChanges](./typeChanges.md) — how to add new runtime versions, resync the processor, and how version detection works. + +## Advanced Topics + +See [advanced-development](./advanced-development.md) — pre-V14 metadata internals, indexer-processor contract, cross-network metadata, debugging decode failures. + +## Releasing + +See [release_process](./release_process.md) diff --git a/docs/release_process.md b/docs/release_process.md index 4e609ff..53ee24c 100644 --- a/docs/release_process.md +++ b/docs/release_process.md @@ -1,8 +1,34 @@ # Release Process -## Release a new version +## Automated Version Bump -1. Update the version in `indexer/chart.yaml` and `processor-chart/chart.yaml` -2. Update the version `package.json` and `package-lock.json` (by doing `npm install` or `yarn`) -3. Commit and these changes and tag the commit with the new version (e.g. `v0.0.1`) -4. Create a release on github release page from the tag. The release title should be the version number and the release description should contain the changelog. \ No newline at end of file +The Makefile automates version bumping across `package.json` and both Helm charts: + +```bash +make version-bump type=patch # 2.12.3 → 2.12.4 +make version-bump type=minor # 2.12.3 → 2.13.0 +make version-bump type=major # 2.12.3 → 3.0.0 +``` + +This will: +1. Checkout the default branch and pull latest +2. Create a new branch `{default-branch}-bump-version-to-{version}` +3. Update versions in `package.json`, `indexer/chart/Chart.yaml`, and `processor-chart/Chart.yaml` +4. Commit the changes + +Then create a PR, get it merged, and tag the release. + +## Manual Version Bump + +If you prefer to do it manually: + +1. Update the version in `package.json` +2. Update `appVersion` in `indexer/chart/Chart.yaml` and `processor-chart/Chart.yaml` +3. Commit and tag the commit with the new version (e.g., `v2.13.0`) + +## Creating a Release + +1. Create a release on the GitHub releases page from the tag +2. The release title should be the version number +3. The release description should contain the changelog +4. The `publish_container_images.yml` workflow will automatically build and push container images on tagged releases diff --git a/indexer/chart/README.md b/indexer/chart/README.md index dd273d8..3a6e637 100644 --- a/indexer/chart/README.md +++ b/indexer/chart/README.md @@ -1,17 +1,17 @@ -# Indexer stack chart +# Indexer Stack Chart -## Install chart with helm +## Install chart with Helm -Create PersistentVolumeClaims for the database if wanted and reference the name in your values file in the `volume.existingpersistentVolumeClaim` property. +Create PersistentVolumeClaims for the database if needed and reference the name in your values file in the `volume.existingpersistentVolumeClaim` property. ```sh -helm installtfchainindexer [-f yourvaluesfile.yaml] . +helm install tfchainindexer [-f yourvaluesfile.yaml] . ``` -If the indexer cannot reach the database, you can set `db_url` to the db-service cluster ip. +If the indexer cannot reach the database, you can set `db_url` to the db-service cluster IP: ```sh kubectl get svc ``` -NOTE: take note of the IP assigned the db-service. Use this IP in `values.yaml` for the db_endpoint, ws_endpoint and indexer_status_service_url. Until DNS resolution works you can update these via 'helm upgrade'. +Take note of the IP assigned to the db-service. Use this IP in `values.yaml` for `db_endpoint`, `ws_endpoint`, and `indexer_status_service_url`. You can update these via `helm upgrade` if the IPs change. diff --git a/indexer/development.md b/indexer/development.md index 48a9936..48ca8e2 100644 --- a/indexer/development.md +++ b/indexer/development.md @@ -1,11 +1,17 @@ -# Tfchain hydra indexer development +# Indexer Development -## Modifying the types.json +## typesBundle.json -When the typesBundle.json file has been modified, it needs to be updated in the helm chart as well. +The `typesBundle.json` file maps Substrate custom types for pre-V14 metadata (specVersions before ~v100). This file is **frozen** — TFChain now uses Polkadot SDK v1.1.0 with V14+ self-describing metadata, so new runtime versions do not require changes to this file. + +See [docs/typeChanges.md](../docs/typeChanges.md#notes-on-typesbundlejson) for details. + +## Updating the Helm Chart ConfigMap + +If `typesBundle.json` ever needs updating (unlikely — only for newly discovered pre-V14 type gaps), regenerate the Helm chart configmap: ```sh kubectl create configmap indexer-config --from-file=./typesBundle.json --dry-run=client --output=yaml > chart/templates/indexer-config.yaml ``` -Don't forget to update the chart version afterwards. +Then update the chart version in `chart/Chart.yaml`. diff --git a/indexer/readme.md b/indexer/readme.md index 87cb8cc..69a7e15 100644 --- a/indexer/readme.md +++ b/indexer/readme.md @@ -1,35 +1,40 @@ -# Setting up an indexer +# Setting Up an Indexer -## Docker compose +## Configuration -First check `.env` file, this contains 2 variables: +The `.env` file contains the indexer options: ``` WS_ENDPOINT=ws://localhost:9944 START_HEIGHT=0 ``` -Set the `WS_ENDPOINT` to a URL you want to connect to. Eg `wss://tfchain.dev.grid.tf` or `ws://localhost:9944` +| Variable | Description | +|----------|-------------| +| `WS_ENDPOINT` | TFChain node WebSocket URL. e.g., `wss://tfchain.dev.grid.tf` or `ws://localhost:9944` | +| `START_HEIGHT` | Block height to start ingesting from. `0` = genesis (full history). Setting this higher skips earlier blocks — the processor will miss any events before this height. Only use non-zero values for testing or partial deployments. | -Start indexer: +## Docker Compose +Start the indexer stack: + +``` +docker compose up -d ``` -docker-compose up -d -``` Stop: ``` -docker-compose down +docker compose down ``` -### Docker Compose setup - -This stack consists of 4 containers: +### Stack Components -- Cockcroach db: database for storing the indexer data -- subsquid/substrate-ingest: the ingester program -- subsquid/substrate-gateway: a gateway to the ingested data (graphql endpoint) -- subsquid/substrate-explorer: a web interface to explore the raw ingester data and it's status +| Container | Image | Role | +|-----------|-------|------| +| db | `cockroachdb/cockroach` | Database for storing raw indexed block data | +| ingest | `subsquid/substrate-ingest` | Connects to the TFChain node and ingests blocks into the database | +| gateway | `subsquid/substrate-gateway` | GraphQL gateway over ingested data — the processor queries this | +| explorer | `subsquid/substrate-explorer` | Web UI to browse raw ingested data and check sync status | -Note on cockroach db: "The --insecure flag used in this tutorial is intended for non-production testing only. To run CockroachDB in production, use a secure cluster instead." See [official docs](https://www.cockroachlabs.com/docs/stable/deploy-cockroachdb-on-premises-insecure) \ No newline at end of file +**Note on CockroachDB:** The `--insecure` flag is used for non-production/testing only. For production, use a secure cluster. See [CockroachDB docs](https://www.cockroachlabs.com/docs/stable/deploy-cockroachdb-on-premises-insecure). diff --git a/package.json b/package.json index 3d92133..0fa8539 100644 --- a/package.json +++ b/package.json @@ -10,8 +10,8 @@ "build": "rm -rf lib && tsc", "db:create-migration": "npx squid-typeorm-migration generate", "db:migrate": "npx squid-typeorm-migration apply", - "db:init": "node init-countries.js", - "db:up": "docker-compose up -d db", + "db:init": "node scripts/init-countries.js", + "db:up": "docker compose up -d db", "process": "node db/init.js && node scripts/init-countries.js && node -r dotenv/config lib/processor.js", "api": "squid-graphql-server", "codegen": "npx squid-typeorm-codegen", diff --git a/processor-chart/README.md b/processor-chart/README.md index 518e50b..def8a10 100644 --- a/processor-chart/README.md +++ b/processor-chart/README.md @@ -1,20 +1,18 @@ -# Processor chart +# Processor Chart -## Install chart with helm +## Install chart with Helm -Create PersistentVolumeClaims for the database if wanted and reference the name in your values file in the `volume.existingpersistentVolumeClaim` property. +Create PersistentVolumeClaims for the database if needed and reference the name in your values file in the `volume.existingpersistentVolumeClaim` property. ```sh -cd tfchain/graphql/processor-chart -helm install tfchainprocessor [-f yourvaluesfile.yaml] . +cd processor-chart +helm install tfchainprocessor [-f yourvaluesfile.yaml] . ``` -If the processor cannot reach the database, you can set `db_url` to the db-service cluster ip. +If the processor cannot reach the database, you can set `db_url` to the db-service cluster IP: ```sh kubectl get svc ``` -## NOTES - -take note of the IP assigned the db-service. Use this IP in `values.yaml` for the db url. +Take note of the IP assigned to the db-service. Use this IP in `values.yaml` for the db URL. diff --git a/scripts/readme.md b/scripts/readme.md index 42f42c4..9f08a97 100644 --- a/scripts/readme.md +++ b/scripts/readme.md @@ -1,5 +1,23 @@ -# Sripts +# Scripts -- init-countries.js: Initialize countries and cities collection with data from open country and city apis. This script is ran everytime the processor is restarted. -- restart-db.sh: clean restart db (also removes migrations folder) -- restart.sh: restart processor script \ No newline at end of file +## Processor Scripts + +| Script | Description | +|--------|-------------| +| `init-countries.js` | Initialize countries/cities collection from open APIs. Runs automatically on every processor start (via `yarn process`). | +| `reset-db.sh` | Drop and recreate the processor database. Use this to reindex from scratch after mapping changes. | +| `restart.sh` | Restart the processor. | +| `init-db.sh` | Initialize database setup. | + +## Type Generation Scripts + +| Script | Description | +|--------|-------------| +| `seed-versions.sh` | Discover all specVersions from all TFChain networks (devnet, qanet, testnet, mainnet) and populate `typegen/tfchainVersions.jsonl`. Used for initial seeding or recovery. | +| `merge-versions.js` | Append-only merger for JSONL files. Merges new specVersions into the existing version log, sorted by specVersion. Used by `make typegen-add`. | + +## Data Files + +| File | Description | +|------|-------------| +| `countries.json` | Static country/city data used by `init-countries.js`. |