Skip to content

feat(observability): consolidate the 3 AI-usage dashboards into one, filterable by provider/feature/model#5267

Merged
JSONbored merged 1 commit into
mainfrom
claude/ai-usage-dashboard-consolidation
Jul 12, 2026
Merged

feat(observability): consolidate the 3 AI-usage dashboards into one, filterable by provider/feature/model#5267
JSONbored merged 1 commit into
mainfrom
claude/ai-usage-dashboard-consolidation

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

Phase B2 of the Grafana/AI-usage observability audit (follow-up to #5247/#5259, which fixed the
underlying ai_usage_events instrumentation this dashboard depends on).

  • claude-usage.json, codex-usage.json, and orb-ai-usage.json overlapped heavily — the same
    ai_usage_events/Prometheus queries sliced 3 different ways (confirmed one exact duplicate: the
    "Cumulative AI cost by provider" panel existed byte-identical in both claude-usage.json and the
    infra dashboard gittensory.json).
  • orb-ai-usage.json's SQL panels had no time-window scoping at all — every panel was a silent
    all-time aggregate regardless of what the dashboard's time picker showed.
  • No provider/feature/model filtering existed anywhere, despite ai_usage_events already carrying
    all three columns.

Replaces all 3 with grafana/dashboards/ai-usage.json:

  • Dynamic $provider/$feature/$model template variables (query-backed against
    ai_usage_events, never hardcoded — a self-hoster's own configured providers/features/models show
    up automatically, matching the existing $repo variable convention from the maintainer dashboard).
  • Every durable-log panel now properly time-scoped (unixepoch(created_at) >= $__from ... < $__to).
  • New "tokens by feature over time" and "cost share by feature" panels so an operator can see what's
    using the most tokens and spot spikes at a glance.
  • A "live Prometheus counters" section (real-time signal — the durable log lags a completed call).
  • Claude Code's own OTEL session telemetry kept in its own clearly-labeled, non-joinable section
    (structurally different data: session-scoped, no repo/PR attribution) rather than force-merged.

Also removes the now-duplicate "AI Usage & Cost" row from gittensory.json's infra dashboard, and
updates resource-hub.json's dashboard catalog (which was also missing gpu-metrics.json entirely).

Test plan

  • npm run typecheck — clean
  • node scripts/validate-observability-configs.mjs — dashboards valid
  • npm run test:ci — full local gate green
  • New test/unit/selfhost-grafana-ai-usage-dashboard.test.ts: template-variable shape, every
    panel scoped by provider/feature/model + time window (except the deliberately-unscoped
    "missing usage" panel, which has its own dedicated test), byte-for-byte carried-over Prometheus
    exprs, real sqlite3-CLI empirical narrowing tests (provider/feature/model filters actually
    narrow counts; "All" still sums everything; the missing-usage panel finds NULL-provider rows
    regardless of which provider/model they'd have had).
  • Updated test/unit/selfhost-grafana-dashboard.test.ts: regression test proving the duplicated
    AI panels are gone from gittensory.json.

…filterable by provider/feature/model

claude-usage.json, codex-usage.json, and orb-ai-usage.json overlapped heavily
(the same ai_usage_events/Prometheus queries sliced 3 different ways), and
none of orb-ai-usage.json's SQL panels had any time-window scoping at all
(every panel was a silent all-time aggregate regardless of the picker).
Replaces all 3 with grafana/dashboards/ai-usage.json: dynamic $provider/
$feature/$model template variables (never hardcoded, so a self-hoster's own
providers/features show up automatically), every durable-log panel properly
time-scoped, new tokens-by-feature and cost-share-by-feature views for
spotting spikes, live Prometheus counters, and Claude Code's own OTEL
telemetry kept in a clearly separate, non-joinable section.

Also removes the now-duplicate "AI Usage & Cost" row from gittensory.json's
infra dashboard, and updates resource-hub.json's dashboard catalog (which was
also missing gpu-metrics.json entirely).
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.37%. Comparing base (864c52d) to head (f2a5ccd).
⚠️ Report is 2 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #5267   +/-   ##
=======================================
  Coverage   94.37%   94.37%           
=======================================
  Files         474      474           
  Lines       40114    40114           
  Branches    14627    14627           
=======================================
  Hits        37856    37856           
  Misses       1583     1583           
  Partials      675      675           
Flag Coverage Δ
shard-1 46.27% <ø> (-0.15%) ⬇️
shard-2 34.73% <ø> (+0.01%) ⬆️
shard-3 32.05% <ø> (-0.05%) ⬇️
shard-4 31.82% <ø> (-0.26%) ⬇️
shard-5 33.59% <ø> (-0.07%) ⬇️
shard-6 45.06% <ø> (+0.30%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@gittensory-orb gittensory-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 12, 2026
@JSONbored JSONbored self-assigned this Jul 12, 2026
@JSONbored JSONbored merged commit 3d4e1a3 into main Jul 12, 2026
19 checks passed
@JSONbored JSONbored deleted the claude/ai-usage-dashboard-consolidation branch July 12, 2026 12:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant