feat: premade grafana dashboard#4398
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
| "type": "prometheus", | ||
| "uid": "${datasource}" | ||
| }, | ||
| "expr": "sum by(taskname) (stats.timers.sentry.taskworker.worker.execution_duration.upper.90{host=~\"$host\"})", |
There was a problem hiding this comment.
Bug: Two queries in sentry-overview.json use an incorrect metric suffix (.upper.90 instead of _upper_90), which will cause them to return no data.
Severity: MEDIUM
Suggested Fix
In sentry-overview.json, replace the incorrect metric suffix .upper.90 with the correct suffix _upper_90 in the queries for panel 5 (refId "B") and panel 10 (refId "B").
Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.
Location: grafana-dashboards/sentry-overview.json#L568
Potential issue: In `sentry-overview.json`, two Grafana panel queries use the metric
suffix `.upper.90` for p90 percentiles. The correct StatsD naming convention is
`_upper_90`. This discrepancy will cause the queries for
`stats.timers.sentry.taskworker.worker.execution_duration.upper.90` and
`stats.timers.sentry.events.since_received.upper.90` to fail silently, as the metrics
with a dot separator do not exist. As a result, the p90 percentile lines will be missing
from the "Task Execution Duration by Task" and "Event Pipeline Latency by Stage" charts,
providing an incomplete view of performance metrics.
Also affects:
grafana-dashboards/sentry-overview.json:998~998
Did we get this right? 👍 / 👎 to inform future reviews.
|
This would be a really useful addition to self-hosted, thanks! |
|
Very appreciated! Thank you! |
| }, | ||
| "expr": "sum (stats.timers.snuba.consumer.arroyo.consumer.backpressure.time.upper_90{host=~\"$host\"})", | ||
| "refId": "A", | ||
| "legendFormat": "backpressure p90" |
There was a problem hiding this comment.
Mixed rates and latency same axis
Medium Severity
The Arroyo Backpressure & Strategies panel applies the ops unit to every series, but one query reads stats.timers.snuba.consumer.arroyo.consumer.backpressure.time.upper_90 (a duration) alongside rate(...) backpressure event counters. Duration and events-per-second are combined on one axis, so the p90 series is mislabeled and hard to read against the counter rates.
Reviewed by Cursor Bugbot for commit 4ab5fbb. Configure here.
| }, | ||
| "expr": "sum by(stage) (stats.timers.sentry.events.since_received.upper{host=~\"$host\"})", | ||
| "refId": "A", | ||
| "legendFormat": "max {{stage}}" |
There was a problem hiding this comment.
Sum adds latency percentiles
Medium Severity
Several latency panels use sum() or sum by(stage)() on exported timer percentiles (median, upper_90, upper) while filtering with host=~"$host". With the default All hosts selection, multiple per-host percentile series are added, which inflates latency rather than reflecting a single pipeline or cluster percentile.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit 4ab5fbb. Configure here.
| "legendFormat": "get" | ||
| }, | ||
| { | ||
| "datasource": { | ||
| "type": "prometheus", | ||
| "uid": "${datasource}" | ||
| }, | ||
| "expr": "sum(rate(stats.counters.sentry.nodestore.get_multi.count{host=~\"$host\"}[$__rate_interval]))", | ||
| "refId": "A", | ||
| "legendFormat": "get_multi" | ||
| } |
There was a problem hiding this comment.
Bug: Multiple queries within the same Grafana dashboard panel share a non-unique refId: "A". Each query should have a unique refId to prevent potential rendering issues or incorrect behavior.
Severity: MEDIUM
Suggested Fix
Update the Grafana dashboard JSON files to ensure that within each panel, every query in the targets array has a unique refId. For example, instead of all targets having "refId": "A", they should be sequential, like "refId": "A", "refId": "B", "refId": "C", and so on.
Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.
Location: grafana-dashboards/sentry-foundational-storage.json#L374-L386
Potential issue: In multiple Grafana dashboard panels across all five new dashboard
files, several queries (`targets`) are configured with the same `refId: "A"`. According
to Grafana's design principles and documentation, each query within a panel must have a
unique `refId`. This identifier is used to reference query outputs in expressions,
transformations, and alerting. Using duplicate `refId`s is an incorrect configuration
that can lead to runtime visualization issues, where Grafana may fail to render all data
series correctly or behave unpredictably. While simple panels might render correctly
depending on the Grafana version, this pattern violates best practices and risks
breaking dashboards, especially if they are later modified to include expressions or
alerts.
Also affects:
grafana-dashboards/sentry-foundational-storage.json:480~500grafana-dashboards/sentry-foundational-storage.json:944~973grafana-dashboards/sentry-ingestion.json:374~394grafana-dashboards/sentry-tasks-manager.json:374~393grafana-dashboards/sentry-ui-api.json:471~491grafana-dashboards/sentry-event-processing.json:126~146
|
I plan to merge this before the next release, but I need to test it out again and gather feedbacks |
| "type": "piechart", | ||
| "gridPos": { | ||
| "h": 8, | ||
| "w": 8, |
There was a problem hiding this comment.
Bug: The "Event Items by Type" pie chart panel has a gridPos width of 8 instead of 12, creating a visual gap in the dashboard layout.
Severity: LOW
Suggested Fix
In grafana-dashboards/sentry-ingestion.json, locate the gridPos configuration for the panel with id=6. Change the value of the w property from 8 to 12. This will ensure the panel correctly occupies the second half of the dashboard row, creating a balanced layout consistent with other panels.
Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.
Location: grafana-dashboards/sentry-ingestion.json#L473
Potential issue: In the Grafana dashboard `sentry-ingestion.json`, the pie chart panel
"Event Items by Type" (panel id=6) has an incorrect `gridPos` configuration. Its width
is set to `w: 8` while its sibling panel in the same row has `w: 12`. On a 24-column
grid, this configuration leaves a 4-column empty gap on the right side of the row. Other
two-panel rows in the same dashboard correctly use a `w: 12` configuration for each
panel to fill the entire row width. This results in a visually unbalanced and broken
layout for this section of the dashboard.
| "type": "prometheus", | ||
| "uid": "${datasource}" | ||
| }, | ||
| "expr": "sum(stats.counters.sentry.relay.project_configs.post_v3.pending.count{host=~\"$host\"})", |
There was a problem hiding this comment.
Bug: The dashboard incorrectly queries a cumulative counter for "pending" items without rate(), showing a misleading, ever-increasing total instead of a meaningful operational metric.
Severity: MEDIUM
Suggested Fix
Wrap the queries for the pending counter metric with the rate() function to calculate its rate of change. For example, change sum(stats.counters...pending.count{...}) to sum(rate(stats.counters...pending.count{...}[$__rate_interval])). This will provide a meaningful metric consistent with sibling panels and allow for valid operational monitoring.
Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.
Location: grafana-dashboards/sentry-ingestion.json#L726
Potential issue: Several panels in the Grafana dashboard query a cumulative counter
metric, `stats.counters.sentry.relay.project_configs.post_v3.pending.count`, without
applying the `rate()` function. This causes the dashboard to display a misleading,
ever-increasing total number of items that have been in a pending state over time,
rather than the current queue depth or rate of change. This makes it impossible for
operators to determine if the pending queue is actually growing. In one panel, this raw
cumulative value is incorrectly compared directly against a per-second rate, creating a
nonsensical visualization.
Also affects:
grafana-dashboards/sentry-ingestion.json:1158~1158grafana-dashboards/sentry-ingestion.json:1358~1358
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 3 total unresolved issues (including 2 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 5eca131. Configure here.
| } | ||
| ] | ||
| }, | ||
| "unit": "mps" |
There was a problem hiding this comment.
Email panels use wrong unit
Low Severity
Several email throughput stat and timeseries panels set "unit": "mps". In Grafana, mps is a standard length unit (meters per second), not messages or emails per second, so labels and scaling disagree with titles like Emails Sent / sec and axes labeled emails per second.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit 5eca131. Configure here.
| "type": "prometheus", | ||
| "uid": "${datasource}" | ||
| }, | ||
| "expr": "sum(rate(stats.counters.sentry.nodestore.get.count{host=~\"$host\"}[$__rate_interval]))", |
There was a problem hiding this comment.
Bug: The rate() function is incorrectly applied to timer .count metrics, which are not cumulative counters, leading to incorrect data visualization on dashboards.
Severity: MEDIUM
Suggested Fix
Remove the rate() function from queries targeting stats.timers.*.count metrics. These metrics represent counts per flush interval and should be summed or aggregated over a time window, not have their rate calculated.
Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.
Location: grafana-dashboards/sentry-foundational-storage.json#L374
Potential issue: The `rate()` function is applied to StatsD timer `.count` metrics, such
as `stats.timers.sentry.filestore.read.count`. According to StatsD semantics, these
timer `.count` metrics represent a count of occurrences per flush interval, not a
monotonically increasing cumulative counter. Applying `rate()` to a non-cumulative,
gauge-like value that resets periodically results in a mathematically incorrect
calculation, essentially a "rate of a rate". This will cause the corresponding dashboard
panels to display meaningless or near-zero values, misrepresenting the actual activity.
Also affects:
grafana-dashboards/sentry-foundational-storage.json:411~411grafana-dashboards/sentry-foundational-storage.json:453~453grafana-dashboards/sentry-foundational-storage.json:464~464
| "placement": "right", | ||
| "showLegend": true, | ||
| "values": [ | ||
| "value", |
There was a problem hiding this comment.
Bug: A cumulative counter metric for "pending" items is displayed without the rate() function, showing a useless, ever-increasing total instead of a meaningful current value.
Severity: MEDIUM
Suggested Fix
Apply the rate() function to the stats.counters.sentry.relay.project_configs.post_v3.pending.count metric in the panel queries. This will convert the cumulative count into a per-second rate, providing a meaningful view of throughput. Alternatively, if the intent is to show a backlog, the underlying metric in the application should be changed to a gauge.
Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.
Location: grafana-dashboards/sentry-ingestion.json#L438
Potential issue: Dashboard panels display the raw value of a cumulative counter metric,
`stats.counters.sentry.relay.project_configs.post_v3.pending.count`, without applying
the `rate()` function. While the metric name suggests a gauge for "pending" items, its
`stats.counters` prefix indicates it is a cumulative counter that only increases over
time. Displaying the raw, ever-increasing total count is not useful for operational
monitoring, as it fails to represent the current number of pending items or the rate of
change. This provides a misleading view of the system's state.
Also affects:
grafana-dashboards/sentry-ingestion.json:654~654
3 participants
Closes #4391
Disclaimer: yes this is a vibe-coded Grafana JSON dashboard using Kimi 2.7 Code. And yes, I did test it against my instance.
My setup was: Self-hosted Sentry -> Statsd (using atlassian/gostatsd -- enable the
tagssupport, because Sentry uses dogstatsd client instead of regular statsd client) -> Graphite endpoint (provided by VictoriaMetrics) -> VictoriaMetrics. Then Grafana (the Grafana server I use in my company) queries to VictoriaMetrics via Prometheus datasource (NOT the Grafana VictoriaMetrics datasource).Then I use OpenCode as the AI agent harness, connect it to VictoriaMetrics' MCP then let it work on the GitHub issue with other directions.