Skip to content

fix(observability): remove dangling Orb metric refs, fix duplicate panel id, make github-prs $scope dynamic#5302

Merged
JSONbored merged 1 commit into
mainfrom
claude/observability-phase-c-dashboards
Jul 12, 2026
Merged

fix(observability): remove dangling Orb metric refs, fix duplicate panel id, make github-prs $scope dynamic#5302
JSONbored merged 1 commit into
mainfrom
claude/observability-phase-c-dashboards

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

Phase C of the observability audit (dashboard/config correctness sweep, follow-up to #5247/#5259/#5267).

  • Dangling Orb metricsgittensory.json had 4 panels referencing
    gittensory_orb_events_recorded_total/gittensory_orb_installs_total, both retired along with the
    per-instance Orb App in feat(orb): fleet calibration collector + analytics (retire per-instance App; feed from review_audit) #1256 (incr("gittensory_orb_events_recorded_total") was literally deleted
    in that commit) but never cleaned out of the dashboard — they've rendered a permanent 0 ever since.
    Removed the 2 panels with no live equivalent in the new fleet-collector architecture (there's no
    "recorded" step distinct from "exported" anymore), stripped the dangling series from the 2 mixed
    panels that partially still work, and fixed the test that had locked in the broken names as expected
    content.
  • Duplicate panel idgittensory.json reused id 158 for two unrelated panels ("Maintenance
    Admission Deferrals (total)" and "Orb Relay Registration: Streak vs Drain Progress"); renumbered.
  • Stale commentprometheus/rules/alerts.yml's gittensory-http group header claimed the
    status-label + duration-histogram metrics were still "being added"; both are fully live in
    src/server.ts today.
  • github-prs.json's $scope — was a hardcoded 5-value custom list (org:JSONbored + 4 named
    repos). Converted to a query-backed variable built from whatever repos are actually tracked in the
    local reporting DB, same pattern as the other dashboards' $repo/$provider variables.
  • grafana-github-datasource read-only gap — documented (with an empirical Grafana API check: a
    PUT with "readOnly": true in the payload is silently ignored) why this one datasource can't be
    locked read-only like the other 4 without moving it to file-provisioning, which would reintroduce
    the exact boot-crash risk its API-based setup was built to avoid.
  • Alertmanager no-op receiver — verified only, no code change needed: the live deployment already
    has this wired (a docker-compose.override.yml + alertmanager.local pointing the Discord receiver
    at a real webhook, predating this session). Confirmed via the Alertmanager API that a real production
    alert has been routing to discord for hours with zero errors, and a manual test alert routed
    identically.
  • Deep-audited gpu-metrics.json for the same bug classes (dangling refs, hardcoded values) — clean,
    no findings.

Test plan

  • npm run typecheck — clean
  • node scripts/validate-observability-configs.mjs — dashboards/alerts valid
  • npm run test:ci — full local gate green
  • New/updated tests: dangling-metric-ref regression, panel-id-uniqueness regression, $scope
    dynamic-variable shape + a real sqlite3-CLI empirical narrowing test
  • Live-verified the Alertmanager Discord routing via its API (real + synthetic alerts both routed
    to discord with no delivery errors logged)

…nel id, make github-prs $scope dynamic

Phase C of the observability audit: 4 gittensory.json panels referenced
gittensory_orb_events_recorded_total/gittensory_orb_installs_total, both
retired along with the per-instance Orb App in #1256 but never cleaned out
of the dashboard -- they've rendered a permanent 0 ever since, and the locked-in
test asserted the broken names as expected content. Removed the 2 panels with
no live equivalent (there's no "recorded" step distinct from "exported" in the
current fleet-collector architecture) and stripped the dangling series from
the 2 mixed panels that partially still work.

Also: renumbered a duplicate panel id (158 used by two different panels),
fixed a stale alerts.yml comment claiming status-label/latency-histogram
metrics are still "being added" (both are fully live), converted
github-prs.json's $scope from a hardcoded 5-value custom list to a
query-backed variable built from whatever repos are actually tracked in the
local reporting DB, and documented (with an empirical Grafana-API check) why
the grafana-github-datasource can't be locked read-only like the other 4
datasources without reintroducing the boot-crash risk its API-based
provisioning was built to avoid.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@JSONbored JSONbored self-assigned this Jul 12, 2026
@codecov

codecov Bot commented Jul 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.37%. Comparing base (d56f4ec) to head (03fe1ac).
⚠️ Report is 4 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #5302   +/-   ##
=======================================
  Coverage   94.37%   94.37%           
=======================================
  Files         474      474           
  Lines       40128    40128           
  Branches    14631    14631           
=======================================
  Hits        37869    37869           
  Misses       1583     1583           
  Partials      676      676           
Flag Coverage Δ
shard-1 46.42% <ø> (ø)
shard-2 34.70% <ø> (ø)
shard-3 32.07% <ø> (ø)
shard-4 32.10% <ø> (ø)
shard-5 33.68% <ø> (ø)
shard-6 44.83% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant