Skip to content

typeChanges.md

Latest commit

 

History

History
330 lines (243 loc) · 13 KB

typeChanges.md

File metadata and controls

330 lines (243 loc) · 13 KB

How to handle type changes on chain

Overview

When a new runtime version adds or modifies events/types, we need to update the processor's type definitions so it can decode them. This is now largely automated via the tfchainVersions.jsonl append-only log and Subsquid's typegen tool.

Key concepts

  • typegen/tfchainVersions.jsonl is an append-only log of runtime metadata from all Ledger Chain networks. It is committed to git and is the single source of truth for type generation. Never delete or recreate this file — only append.

  • typegen/typesBundle.json is a frozen historical artifact that helps the metadata explorer decode pre-V14 runtime metadata (specVersions before ~v100). Ledger Chain now uses Polkadot SDK v1.1.0 with V14+ self-describing metadata, so this file does not need editing for new runtime versions. See Notes on typesBundle.json below.

  • src/types/ is fully auto-generated by typegen from the JSONL file. Do not edit these files manually.

How version detection works

Subsquid's typegen generates event classes with isVxx/asVxx getters. Each getter compares the event hash at the current block's runtime against a known hash. The hash is derived from the event's SCALE-encoded type structure.

  • If two specVersions have the same type structure for an event, they produce the same hash. Typegen generates one getter that covers both.
  • If the type structure changes (fields added/removed/reordered), the hash changes. Typegen generates a new getter for that specVersion.
  • The isVxx check does NOT mean "is this block at specVersion xx". It means "does the current runtime's event hash match what was recorded at specVersion xx". A single isV63 getter can match blocks at v63, v64, v65, etc. — as long as the type didn't change.

Network deployment order matters

Ledger Chain follows the deployment pipeline: devnet → qanet → testnet → mainnet. The JSONL file should be seeded in this order so that typegen assigns version labels matching the earliest deployment. The seed-versions.sh script handles this automatically.

A specVersion number represents the same runtime binary on all networks — v63 on devnet is the same code as v63 on mainnet. The difference is which networks deployed it and when.

History and devnet resets

Devnet has been reset multiple times (#21, #55, #499). The current devnet chain starts at v49, losing earlier history (v9-v48). Testnet has the oldest continuous history starting from v9. Mainnet starts from v31.

Some specVersions existed briefly on old devnet incarnations and are not present in the JSONL (e.g., v122 was an RC that the current devnet skips v121→v123). This doesn't affect indexing because the release version (v123) has the same event type hashes as the RC — typegen generates equivalent decoders from v123's metadata. The same applies to any short-lived RC: the release version that follows will have the same or updated types.

Deployment order

Always deploy the processor before the runtime upgrade goes live.

The processor is forward-compatible — having isV158 branches while the chain is still on v157 is harmless (the check simply returns false). But if the runtime upgrade goes live first, the processor encounters events with unrecognized hashes, silently skips them, and those blocks are never re-visited. At that point a full processor resync is required.

The recommended order:

  1. Deploy the updated processor (with new version branches)
  2. Deploy the runtime upgrade to the chain
  3. The processor immediately handles the new events — no missed blocks

This applies per network. For the typical pipeline (devnet → qanet → testnet → mainnet), update the processor for each network before its runtime upgrade.

Adding a new runtime version

If a type change was made or new events were added on chain, follow these steps.

1: Run a local chain

See ./development

Make sure you increment the specVersion before you compile and run tfchain.

2: Add new specVersion and regenerate types

Run the following command to discover the new specVersion from your local chain, merge it into the master JSONL, and regenerate all type definitions:

make typegen-add

This does three things:

  1. Runs the metadata explorer against ws://localhost:9944 (or $WS_URL)
  2. Merges any new specVersions into typegen/tfchainVersions.jsonl
  3. Runs typegen to regenerate src/types/ with all version-aware decoders

You can also point at a remote network:

WS_URL=wss://tfchain.test.grid.tf make typegen-add

3: Determine what changed

After step 2, check what typegen generated. There are three scenarios:

Scenario A: Updated event that was already tracked

The event is already in typegen/typegen.json and already has a handler in src/mappings/. Typegen will detect the hash change and generate a new isVxx/asVxx getter automatically. You only need to:

  1. Update the mapping handler to add a branch for the new version:
if (event.isV158) {
    data = event.asV158
    // handle new fields
} else if (event.isV123) {
    data = event.asV123
}
  1. Update schema.graphql if new fields need to be exposed, then:
npm run codegen
npm run build
npm run db:create-migration

Scenario B: New event to track

The event is new — it didn't exist before on chain and the processor never indexed it. You need to:

  1. Add the event to typegen/typegen.json:
"events": [
    ...
    "SomeModule.NewEvent"
]
  1. Re-run typegen:
make typegen

  1. Register the event in src/processor.ts:

    • Add .addEvent('SomeModule.NewEvent', eventOptions)
    • Add a case to the handleEvents switch statement

  2. Create a mapping handler in src/mappings/

  3. Update schema.graphql if needed, then codegen + migration as above.

Scenario C: Old event that was never tracked, now needs indexing

The event already exists on chain (possibly across multiple specVersions) but the processor never tracked it. This is like Scenario B, but the event may have multiple historical type shapes. You need to:

  1. Add the event to typegen/typegen.json (same as Scenario B)

  2. Re-run typegen:

make typegen

  1. Check what typegen generated — it will create isVxx/asVxx getters for every specVersion where the event's hash changed. You need to handle all of them in your mapping, since the processor will encounter blocks from all historical versions when indexing from genesis.

  2. Register in src/processor.ts and create the mapping handler with branches for all generated versions.

  3. Update schema.graphql, codegen, migration as needed.

4: Do I need to resync?

The indexer (archive) never needs resyncing for type changes — it stores raw blocks regardless of what the processor tracks.

The processor depends on the scenario and timing:

  • Scenario A (updated existing event): No resync needed if the processor is updated before (or at the same time as) the runtime goes live. If the runtime upgrade goes live first, the processor will encounter events with an unrecognized hash — they silently fall through the if/else chain and are skipped. Those blocks won't be re-visited. In that case, resync from the block where the runtime upgrade happened.
  • Scenario B (new event): Same rule — if the processor is updated before the runtime goes live, no resync. If the runtime goes live first and events are emitted before the processor update, those events are missed. Resync from the block where the event first appeared.
  • Scenario C (old event, need full history): Resync always required — drop the processor's database and reindex from genesis (or from the block where the event first appeared).

To resync the processor, stop it, reset the database, and restart:

# Stop the processor, then:
./scripts/reset-db.sh
npm run process

This drops and recreates the database, re-runs migrations, and the processor reindexes from block 0. There is no partial resync — the processor writes into application tables assuming fresh state, so rewinding to a specific block would conflict with existing data.

The indexer (archive) never needs resyncing for processor-level changes — it stores raw blocks independently.

5: Build and test

npm run build
npm run process

6: Commit

Commit both the updated typegen/tfchainVersions.jsonl (the new specVersion entry) and the regenerated src/types/ files. The JSONL file is the source of truth — the type files can always be regenerated from it.

Recovering or reseeding the JSONL

If the JSONL file is accidentally deleted or corrupted, you can rebuild it from scratch by running the metadata explorer against all networks. This is a one-time recovery operation, not part of the normal workflow.

rm typegen/tfchainVersions.jsonl
touch typegen/tfchainVersions.jsonl
make typegen-seed
make typegen

This takes several minutes per network as the explorer walks the chain to discover all runtime upgrades. Devnet is the slowest (~5-10 minutes).

The seed order matters — seed-versions.sh runs devnet first to ensure version labels match the earliest deployment in the pipeline.

Notes on typesBundle.json

typegen/typesBundle.json maps Substrate custom types for pre-V14 metadata. It covers historical runtime versions (v9 through ~v100) and is frozen.

Why it exists: Older Substrate versions (pre-V14) used metadata that wasn't fully self-describing. The types bundle tells the metadata explorer and typegen how to decode those custom pallet types. Without it, the explorer can't read metadata from blocks at those old specVersions.

Why it's frozen: Ledger Chain upgraded to Polkadot SDK v1.1.0, which uses V14+ self-describing metadata. Any new specVersion includes a complete type registry in its metadata — no external bundle needed. The types bundle only matters for the old pre-V14 blocks, which will never change.

When to edit it: Only if a missing pre-V14 type definition is discovered (unlikely — the current file has been in use since the indexer's inception and covers all historical types). For any new runtime version, the answer is always: you don't need to touch this file.

Architecture notes

Files and their roles

File Role Edited by
typegen/tfchainVersions.jsonl Append-only metadata log merge-versions.js (via make typegen-add)
typegen/typesBundle.json Pre-V14 type mappings Frozen — do not edit
typegen/typegen.json Events to track Developer (when adding new events)
src/types/events.ts Event decoder classes Typegen (auto-generated)
src/types/vXX.ts Type interfaces per version Typegen (auto-generated)
src/types/support.ts Base interfaces (Chain, Event) Typegen (auto-generated)
src/mappings/*.ts Business logic handlers Developer
src/processor.ts Event registration + dispatch Developer
schema.graphql GraphQL/DB schema Developer

What was changed from the previous manual workflow

Previously, src/types/ was hand-maintained: types were generated into src/typesDevelopment/ from a local chain node, then manually copied and merged into src/types/. This was necessary because the JSONL file was deleted and recreated each time from a single node (capturing only one specVersion), and devnet resets meant no single network had the full history.

The new workflow treats the JSONL as a persistent, append-only log that accumulates metadata from all networks. Typegen reads the full log and generates complete, version-aware types automatically. The typesDevelopment/ directory is no longer needed.

This also uncovered two issues in the hand-maintained types:

  1. Phantom hashes from wrong field ordering. Several event versions (e.g., TwinStored v49, FarmStored v63, NodeUpdated v63) had hashes computed from interfaces where fields were listed in a different order than the actual Rust struct. These hashes never matched at runtime on any network. The code worked because the if/else chain would fall through to a later version check that had the correct hash. The auto-generated types have correct hashes from actual chain metadata.

  2. Mislabeled version numbers from manual editing. Some version labels referenced short-lived release candidates (e.g., v122 for ServiceContracts, v134 for NodeExtraFeeSet) whose metadata is no longer available — either because devnet was reset and the RC blocks are gone, or because the RC was never deployed to the current devnet. The hand-maintained types for these versions were written from the release code (v123, v140) but labeled with the RC number. The hashes in the hand-maintained code were actually the release hashes, not the RC hashes. The auto-generated types now use the correct version labels (v123, v140) matching the actual metadata.