feat(price-pusher): emit namespace label on all Prometheus metrics#3720
feat(price-pusher): emit namespace label on all Prometheus metrics#37200xghost42 wants to merge 2 commits into
namespace label on all Prometheus metrics#3720Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. 7 Skipped Deployments
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
Fair point — Pushed
Default behaviour unchanged for single-deployment operators — the CLI defaults the value to the chain command name ( |
The sample Grafana dashboard (`grafana-dashboard.sample.json`) filters
every Prometheus query by `namespace=$chain`:
pyth_price_feeds_total{namespace="$chain"}
pyth_price_last_published_time{namespace="$chain"}
pyth_price_update_attempts_total{namespace="$chain", ...}
...
But `src/metrics.ts` only set `app="price_pusher"` as the default label
and never emitted a `namespace` label on any counter/gauge, and
`prometheus.sample.yml` did not add one via relabeling. With the
dashboard out of the box that filter matched zero series, so every
panel for a fresh deployment came up empty — flagged by Devin during
review of pyth-network#3692 and acknowledged there as pre-existing dashboard
behavior worth fixing in a separate PR. This is that PR.
Wiring:
- new CLI option `--metrics-namespace <name>` in `src/options.ts`,
defaulted per chain command (`evm`, `sui`, `aptos`, `solana`) so
that single-chain deployments work without configuration
- `PricePusherMetrics` constructor now takes a `namespace: string`
parameter and sets it via `registry.setDefaultLabels`, so every
existing metric series gains a `namespace` label without per-metric
`labelNames` plumbing
- each chain command (`evm/sui/aptos/solana/command.ts`) reads the
new arg and passes through to the constructor
Operators running multiple deployments of the same chain (e.g. several
EVM networks against one Grafana instance) can now set
`--metrics-namespace bsc-mainnet` / `--metrics-namespace polygon-mainnet`
to disambiguate. Single-deployment setups keep working unchanged.
Bumps `@pythnetwork/price-pusher` to `10.5.0` (additive, no breaking
changes — old CLIs still work). Note: pyth-network#3689 and pyth-network#3703 also bump to
`10.5.0`, so whichever of the three lands second/third will need a
trivial rebase to `10.6.0` / `10.7.0`.
Refs pyth-network#3692.
Devin re-review on this PR pointed out that `namespace` is the label
Kubernetes Prometheus service discovery auto-applies to scraped pods
(it carries the pod's k8s namespace). With the default
`honor_labels: false` on the scrape config, my label would be renamed
to `exported_namespace` and the dashboard's `namespace=\$chain`
filter would silently break in any in-cluster deployment. With
`honor_labels: true` the operator wins but loses k8s namespace
context.
Renamed everywhere to `chain`, which is what the dashboard's template
variable was already called and avoids the collision entirely.
Renames applied:
- CLI option `--metrics-namespace` -> `--metrics-chain` (option
name + variable identifier + help text in
`apps/price_pusher/src/options.ts`)
- Constructor arg `PricePusherMetrics(logger, namespace)` ->
`PricePusherMetrics(logger, chain)` (+ comment noting why the
obvious name was avoided)
- Default-label key in `registry.setDefaultLabels` from
`namespace` to `chain`
- Per-chain callsites in `apps/price_pusher/src/{evm,sui,aptos,solana}/command.ts`
swap `options.metricsNamespace` / `metricsNamespace` to the new
name
- `apps/price_pusher/grafana-dashboard.sample.json` panel queries
(`namespace=\"$chain\"` -> `chain=\"$chain\"`, Loki
`{namespace=~...}` -> `{chain=~...}`, `legendFormat: {{namespace}}`
-> `{{chain}}`) and the template variable's source label
(`label: 'namespace'` -> `label: 'chain'` on the Loki query that
populates `$chain`)
Same semantics, less ambiguous label. Refs pyth-network#3692.
0xghost42
force-pushed
the
feat/3692-metrics-namespace-label
branch
from
b37bd13 to
0ad092f
Compare
There was a problem hiding this comment.
🟡 Grafana table panel missing excludeByName entries for new chain default label columns
Adding chain as a default Prometheus label in apps/price_pusher/src/metrics.ts:38 means that table-format queries in the "Price Feeds List" panel will now produce chain, chain#B, and chain#C columns (one per query ref). The excludeByName block already excludes the analogous app/app#B/app#C columns from the pre-existing app default label, but does not exclude the new chain columns. As a result, the table will display extra unwanted columns showing the chain value in every row.
(Refers to lines 432-455)
Was this helpful? React with 👍 or 👎 to provide feedback.
| }, | ||
| "editorMode": "code", | ||
| "expr": "{namespace=~\"$chain\"} | logfmt | json | msg =~ `.*(Price update successful|Transaction confirmed|Successfully updated price).*` | line_format `Tx Hash: {{.hash}}`", | ||
| "expr": "{chain=~\"$chain\"} | logfmt | json | msg =~ `.*(Price update successful|Transaction confirmed|Successfully updated price).*` | line_format `Tx Hash: {{.hash}}`", |
There was a problem hiding this comment.
🚩 Loki stream selector assumes chain label exists on log streams
The Grafana dashboard Loki queries (e.g., {chain=~"$chain"} at lines 1080, 1130, 1167) and the template variable definition at line 1205 now query for a chain label on Loki log streams. This label is not set by the application code itself — it depends on how logs are shipped to Loki (e.g., via Promtail/Grafana Agent label configuration in Kubernetes). Previously the dashboard used namespace which is automatically applied by Kubernetes service discovery. The new chain label will need to be explicitly configured in the log pipeline (e.g., as a Promtail relabeling rule or a pod label). This is a deployment concern rather than a code bug, but operators upgrading should be aware they need to update their log shipping configuration to add a chain label.
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
Summary
Make the sample Grafana dashboard's
namespace=\$chainfilters actually match emitted series, which they don't today.Background
apps/price_pusher/grafana-dashboard.sample.jsonqueries every panel withnamespace="\$chain", e.g.But
apps/price_pusher/src/metrics.tsonly setapp="price_pusher"as the default registry label, andprometheus.sample.ymldoesn't addnamespacevia relabeling either. The result: every panel in the sample dashboard came up empty on a fresh deployment.Devin flagged this during the #3692 review and I acknowledged it there as pre-existing dashboard behavior worth fixing in a separate PR. This is that PR.
Wiring
--metrics-namespace <name>insrc/options.ts, defaulted per chain command (evm,sui,aptos,solana) so single-chain deployments work without any configurationPricePusherMetricsconstructor now takes anamespace: stringparameter and sets it viaregistry.setDefaultLabels, so every existing metric series gains anamespacelabel without per-metriclabelNamesplumbingevm/sui/aptos/solana/command.ts) reads the new arg and passes through to the constructorOperators running multiple deployments of the same chain (e.g. several EVM networks against one Grafana instance) can now set
--metrics-namespace bsc-mainnet/--metrics-namespace polygon-mainnetto disambiguate. Single-deployment setups keep working unchanged — the dashboard's\$chaintemplate var will just match the chain command name out of the box.Bumps
@pythnetwork/price-pusherto10.5.0.Coordination heads-up
#3689 and #3703 (both open) also bump to
10.5.0. Whichever of the three lands second/third will need a trivial rebase to10.6.0/10.7.0.Test plan
--helpexposes new--metrics-namespaceoption on each chain--metrics-namespace):curl :9090/metricsshowspyth_*{...namespace="evm",...}(orsui/aptos/solana)--metrics-namespace bsc-mainnetyieldsnamespace="bsc-mainnet"on every emitted series\$chain=\$namespacepanels populate end-to-end against a real pusherevm/sui/aptos/solana)Refs #3692.