feat(observability): consolidate the 3 AI-usage dashboards into one, filterable by provider/feature/model#5267
Merged
Conversation
…filterable by provider/feature/model claude-usage.json, codex-usage.json, and orb-ai-usage.json overlapped heavily (the same ai_usage_events/Prometheus queries sliced 3 different ways), and none of orb-ai-usage.json's SQL panels had any time-window scoping at all (every panel was a silent all-time aggregate regardless of the picker). Replaces all 3 with grafana/dashboards/ai-usage.json: dynamic $provider/ $feature/$model template variables (never hardcoded, so a self-hoster's own providers/features show up automatically), every durable-log panel properly time-scoped, new tokens-by-feature and cost-share-by-feature views for spotting spikes, live Prometheus counters, and Claude Code's own OTEL telemetry kept in a clearly separate, non-joinable section. Also removes the now-duplicate "AI Usage & Cost" row from gittensory.json's infra dashboard, and updates resource-hub.json's dashboard catalog (which was also missing gpu-metrics.json entirely).
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #5267 +/- ##
=======================================
Coverage 94.37% 94.37%
=======================================
Files 474 474
Lines 40114 40114
Branches 14627 14627
=======================================
Hits 37856 37856
Misses 1583 1583
Partials 675 675
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
This was referenced Jul 12, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase B2 of the Grafana/AI-usage observability audit (follow-up to #5247/#5259, which fixed the
underlying
ai_usage_eventsinstrumentation this dashboard depends on).claude-usage.json,codex-usage.json, andorb-ai-usage.jsonoverlapped heavily — the sameai_usage_events/Prometheus queries sliced 3 different ways (confirmed one exact duplicate: the"Cumulative AI cost by provider" panel existed byte-identical in both
claude-usage.jsonand theinfra dashboard
gittensory.json).orb-ai-usage.json's SQL panels had no time-window scoping at all — every panel was a silentall-time aggregate regardless of what the dashboard's time picker showed.
ai_usage_eventsalready carryingall three columns.
Replaces all 3 with
grafana/dashboards/ai-usage.json:$provider/$feature/$modeltemplate variables (query-backed againstai_usage_events, never hardcoded — a self-hoster's own configured providers/features/models showup automatically, matching the existing
$repovariable convention from the maintainer dashboard).unixepoch(created_at) >= $__from ... < $__to).using the most tokens and spot spikes at a glance.
(structurally different data: session-scoped, no repo/PR attribution) rather than force-merged.
Also removes the now-duplicate "AI Usage & Cost" row from
gittensory.json's infra dashboard, andupdates
resource-hub.json's dashboard catalog (which was also missinggpu-metrics.jsonentirely).Test plan
npm run typecheck— cleannode scripts/validate-observability-configs.mjs— dashboards validnpm run test:ci— full local gate greentest/unit/selfhost-grafana-ai-usage-dashboard.test.ts: template-variable shape, everypanel scoped by provider/feature/model + time window (except the deliberately-unscoped
"missing usage" panel, which has its own dedicated test), byte-for-byte carried-over Prometheus
exprs, real sqlite3-CLI empirical narrowing tests (provider/feature/model filters actually
narrow counts; "All" still sums everything; the missing-usage panel finds NULL-provider rows
regardless of which provider/model they'd have had).
test/unit/selfhost-grafana-dashboard.test.ts: regression test proving the duplicatedAI panels are gone from
gittensory.json.