Skip to content

fix(data-api): order broad extrinsics/chain-events feeds by observed_at#4955

Merged
JSONbored merged 2 commits into
mainfrom
fix/extrinsics-timescale-ordering
Jul 12, 2026
Merged

fix(data-api): order broad extrinsics/chain-events feeds by observed_at#4955
JSONbored merged 2 commits into
mainfrom
fix/extrinsics-timescale-ordering

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

  • GET /api/v1/extrinsics was silently returning an empty feed (HTTP 200, extrinsic_count:0, generated_at:null) for the unfiltered/default view and for broad filters (success=true, call_module=SubtensorModule) — confirmed in production during a live-wiring verification sweep. Narrow filters (a specific block, success=false, a specific signer) worked correctly, which pointed at a query-shape problem rather than a data or decode problem.
  • Root cause: extrinsics/chain_events are TimescaleDB hypertables partitioned on observed_at, but the recent-feed queries ordered by block_number DESC, extrinsic_index DESC — a column with zero chunk-exclusion value for the planner. EXPLAIN ANALYZE against production showed a full cross-chunk Parallel Append + Sort, decompressing 15 historical compressed chunks, taking 6041ms — well past this route's 3000ms statement_timeout. tryPostgresTier treats any non-2xx as "fall back to D1"; D1's extrinsics table was fully dropped in the 2026-07-11 chain-data retirement (Fully retire D1 for chain data — delete tables, prune logic, ingest endpoints, and the D1 fallback (gated on #4771) #4772), so the fallback silently resolved to an empty-but-200 response instead of surfacing an error anywhere.
  • Fix: reorder the recent-feed queries (/api/v1/extrinsics, /api/v1/sudo, /api/v1/governance/config-changes, /api/v1/chain-events) to ORDER BY observed_at DESC, block_number DESC, ..., which lets TimescaleDB's ChunkAppend prune straight to the live head chunk. Confirmed live against production Postgres: the identical queries dropped from 6041ms to 0.5–75ms. observed_at is the genuine on-chain block timestamp (verified against block chore(deps): bump astral-sh/setup-uv from d0d8abe699bfb85fec6de9f7adb5ae17292296ff to d0cc045d04ccac9d8b7881df0226f9e82c39688e #1 → 2023-03-20, not an ingestion clock), so ordering by it produces the exact same "most recent first" semantics as block_number DESC — block production is strictly monotonic with wall-clock time. Cursors for the affected routes are now 3-tuples (observed_at, block_number, position); an old 2-part cursor just decodes to null (cursor ignored, page restarts) rather than erroring — no live callers depend on cursor stability across this change.
  • Also fixes scripts/fetch-subnet-hyperparams.py: every hyperparameters row served in production had block_number:0 because getattr(s.substrate, "block_number", 0) silently fell back to the default — SubstrateInterface has no block_number attribute (it's the method get_block_number()). Now reuses the metagraph snapshot's own per-info .block field (already fetched in the same run, zero extra RPC calls), matching fetch-metagraph-native.py's existing convention. Verified live: 129/129 subnets now report the real chain height (~8605812) instead of 0.

Test plan

  • npm run lint / npm run format:check clean
  • npm test — 256/256 files, 7579/7579 tests green (full suite, twice)
  • npm run validate / validate:contract-drift / validate:schemas / validate:api / validate:openapi all green
  • Live-verified the root cause and the fix directly against production Postgres via EXPLAIN (ANALYZE, BUFFERS, TIMING) (SSH to the indexer box) — before: 6041ms unfiltered scan; after: 36–75ms for unfiltered, success=true, and call_module=SubtensorModule cases
  • Live-ran the fixed fetch-subnet-hyperparams.py end-to-end against production finney (bittensor 10.4.0, the CI-pinned version) — 129/129 subnets, all with a real non-zero block_number
  • Updated + added regression tests in tests/data-api.test.mjs for the new 3-tuple cursor shape on all four affected routes
  • Schema contract updated (ChainEventsFeedArtifact.next_cursor pattern/length/description) and regenerated (npm run buildopenapi.json, types.d.ts, packages/contract/index.d.ts, schemas/api-components.schema.json)

GET /api/v1/extrinsics was returning HTTP 200 with an empty feed
(extrinsic_count:0, generated_at:null) for the unfiltered/default view and
for broad filters like success=true or call_module=SubtensorModule, while
narrow filters worked fine.

Root cause: extrinsics/chain_events are TimescaleDB hypertables partitioned
on observed_at, but the recent-feed queries ordered by
`block_number DESC, extrinsic_index DESC` -- a column with no chunk-exclusion
information, forcing a full cross-chunk sort (measured 6+ seconds live,
including decompressing 15 historical compressed chunks). That blew past
this route's 3000ms statement_timeout, which tryPostgresTier treats as "fall
back to D1" -- and D1's extrinsics table was fully dropped in the 2026-07-11
chain-data retirement (#4772), so the fallback silently resolved to an
empty-but-200 response instead of erroring.

Reordering the recent-feed queries (extrinsics, sudo, governance/
config-changes, chain-events) to `observed_at DESC, block_number DESC, ...`
lets ChunkAppend prune straight to the live head chunk -- confirmed live
against production Postgres: the same queries dropped from 6041ms to
0.5-75ms. observed_at is the genuine on-chain block timestamp (verified
against block #1: 2023-03-20, not an ingestion clock), so this produces the
identical ordering as block_number DESC; block_number/extrinsic_index only
break ties among extrinsics sharing one block's timestamp. Cursors for the
affected routes are now 3-tuples (observed_at, block_number, position) to
match; an old 2-part cursor decodes to null (page restarts) rather than
crashing.

Also fixes scripts/fetch-subnet-hyperparams.py: every hyperparameters row
served in production had block_number:0, because
`getattr(s.substrate, "block_number", 0)` always fell back to 0 --
SubstrateInterface has no `block_number` attribute (it's the method
get_block_number()). Reuses the metagraph snapshot's own per-info `.block`
(already fetched in the same run) instead of an extra RPC call per netuid,
matching fetch-metagraph-native.py's existing convention. Verified live:
129/129 subnets now report the real chain height.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 12, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
metagraphed-data-api dcca3d0 Jul 12 2026, 02:38 PM

@superagent-security

Copy link
Copy Markdown

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.94%. Comparing base (259928a) to head (dcca3d0).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4955   +/-   ##
=======================================
  Coverage   97.94%   97.94%           
=======================================
  Files         163      163           
  Lines       19418    19420    +2     
  Branches     7376     7376           
=======================================
+ Hits        19018    19020    +2     
  Misses         53       53           
  Partials      347      347           
Files with missing lines Coverage Δ
src/openapi-sample.mjs 100.00% <100.00%> (ø)
workers/data-api.mjs 99.79% <100.00%> (ø)
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@JSONbored JSONbored self-assigned this Jul 12, 2026
@gittensory-orb gittensory-orb Bot added the gittensor:bug Gittensor-scored bug fix or unsolicited PR — scores a 0.05x multiplier. label Jul 12, 2026
@gittensory-orb

gittensory-orb Bot commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Warning

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏸️ Gittensory review result - manual review recommended

Review updated: 2026-07-12 14:43:23 UTC

11 files · 1 AI reviewer · 2 blockers · readiness 93/100 · CI green · unstable

⏸️ Suggested Action - Manual Review

  • No linked issue detected — If this PR is intended to solve an issue, link it explicitly in the PR body.
  • Maintainer requires a linked issue — Link the relevant issue (for example Closes #123) before opening the PR.

Review summary
This PR reorders four hypertable-backed feed queries to sort/seek by observed_at first instead of block_number, fixing a real production timeout (chunk exclusion) that was silently masked as an empty-but-200 response via D1 fallback. Cursor arity bumps from 2 to 3 parts consistently across the extrinsics/sudo/config-changes/chain-events routes, with matching schema/openapi/types regeneration and updated tests reflecting the new cursor format. The fetch-subnet-hyperparams.py change is a distinct but justified fix (block_number attribute never existed on SubstrateInterface, was always 0) bundled into this PR.

Blockers

  • workers/data-api.mjs:2500 and workers/data-api.mjs:2575 now emit three-part next_cursor values for /api/v1/extrinsics, /api/v1/sudo, and /api/v1/governance/config-changes, but the visible schema/artifact updates only change ChainEventsFeedArtifact, so the extrinsics-derived API contract remains stale for the new observed_at.block_number.extrinsic_index cursor shape.
Nits — 5 non-blocking
  • The PR bundles an unrelated fix (scripts/fetch-subnet-hyperparams.py block_number sourcing) with the observed_at ordering fix, which could have been a separate PR for cleaner history.
  • workers/data-api.mjs: the cursor arity `3` is a repeated magic number across three call sites (lines ~2474, 2543, 4895); a named constant would make the 2-tuple→3-tuple contract clearer.
  • chain-events legacy `?before` cursor still does a plain block_number inequality without the chunk-exclusion benefit (acknowledged in the comment) — fine as a documented limitation, not a defect.
  • Test coverage is reasonable but thin relative to diff size (0.22 ratio); consider a direct test asserting an old 2-part cursor is ignored per the PR description's stated behavior.
  • Extract the cursor arity (3) into a shared constant near decodeCursor to self-document the 2-tuple→3-tuple migration across all four routes.

Concerns raised — review before merging

  • No linked issue detected — If this PR is intended to solve an issue, link it explicitly in the PR body.
  • Maintainer requires a linked issue — Link the relevant issue (for example Closes #123) before opening the PR.
Signal Result Evidence
Code review ❌ 2 blockers 1 reviewer
Linked issue ⚠️ Missing No linked issue or no-issue rationale found.
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (no linked issue context).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 44 registered-repo PR(s), 36 merged, 476 issue(s).
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 44 PR(s), 476 issue(s).
Gate result ❌ Blocking Repo-configured hard blocker found.
Review context
  • Author: JSONbored
  • Role context: owner (maintainer lane)
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: not available
  • Official Gittensor activity: 44 PR(s), 476 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Treat this as maintainer-lane context rather than normal contributor-lane activity.
  • Explain no-issue PR.
  • Link the issue being solved, or explicitly explain why this is a no-issue PR.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
[BETA] Chat with Gittensory

Ask Gittensory a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @gittensory ask &lt;question&gt; answers contribution-quality Q&A with source citations and freshness.
  • @gittensory chat &lt;question&gt; answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @gittensory mention with a real question is routed to the closest matching read-only command automatically -- no exact syntax required.

Full command reference: https://gittensory.aethereal.dev/docs/gittensory-commands

Visual preview
Route Viewport Before (production) After (this PR's preview) Diff
/ desktop before / after /
/ mobile before / (mobile) after / (mobile)

Click any thumbnail to open the full-size screenshot. Before = production · After = this PR's preview deploy.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

@gittensory-orb gittensory-orb Bot added the manual-review Gittensor contributor context label Jul 12, 2026
…tract change

public/metagraph/openapi.json and packages/contract/index.d.ts changed
(ChainEventsFeedArtifact.next_cursor's pattern/maxLength widened to a
3-part keyset cursor) -- validate:client-sdk-sync requires the published
client version to move in the same PR as a contract change.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 12, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
metagraphed-registry-sync-api dcca3d0 Jul 12 2026, 02:38 PM

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 12, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
metagraphed-ui dcca3d0 Commit Preview URL

Branch Preview URL
Jul 12 2026, 02:39 PM

@JSONbored JSONbored merged commit 71814ee into main Jul 12, 2026
17 checks passed
@JSONbored JSONbored deleted the fix/extrinsics-timescale-ordering branch July 12, 2026 14:47
@github-project-automation github-project-automation Bot moved this from Backlog to Done in Block-Explorer Completion Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix or unsolicited PR — scores a 0.05x multiplier. manual-review Gittensor contributor context

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant