Skip to content

Include offending series labels in missing-metric-name push errors#7657

Open
anxkhn wants to merge 1 commit into
cortexproject:masterfrom
anxkhn:loop/cortex__001
Open

Include offending series labels in missing-metric-name push errors#7657
anxkhn wants to merge 1 commit into
cortexproject:masterfrom
anxkhn:loop/cortex__001

Conversation

@anxkhn

@anxkhn anxkhn commented Jun 30, 2026

Copy link
Copy Markdown

What this PR does:

When a pushed series has no __name__ label, the error Cortex returns to the
remote-write client gave no indication of which series was at fault, only
no metric name label (HTTP 500) or sample missing metric name (HTTP 400).
As the issue reporter described, the only way to find the culprit was to bisect
by stopping Prometheus instances one at a time.

This enriches both user-facing push paths so the offending series labels are
always included in the error:

  • noMetricNameError now carries the series and renders it via formatLabelSet,
    exactly like every sibling validation error in pkg/util/validation/errors.go
    (tooManyLabelsError, labelValueTooLongError, the native-histogram errors,
    etc.). This is the validation path (HTTP 400) taken when metric-name
    enforcement is enabled (the default).
  • The fatal error from tokenForLabels in the distributor (the
    shard_by_all_labels=false path that produced the original context-less HTTP
    500) is wrapped with the series labels via
    cortexpb.FromLabelAdaptersToMetric(ts.Labels).String().

Example, before vs after (validation path):

before:  sample missing metric name
after:   sample missing metric name metric: "{foo=\"bar\"}"

Which issue(s) this PR fixes:
Fixes #5802

Checklist

  • Tests updated
  • Documentation added (N/A - no user-facing config/docs change)
  • CHANGELOG.md updated
  • docs/configuration/v1-guarantees.md updated if this PR introduces experimental flags (N/A - no new flags)

AI assistance disclosure (per the project's GENAI_POLICY.md): this change was
prepared with the help of an AI coding assistant. I have reviewed every line,
understand the behaviour of both push paths, and validated it locally (build,
unit tests for both changed packages, golangci-lint, gofmt/goimports).

When a pushed series has no __name__ label, the error returned to the
client only said "no metric name label" / "sample missing metric name"
with no indication of which series caused it. Operators had to bisect by
stopping Prometheus instances one by one to find the culprit.

Enrich both user-facing push paths so the offending series labels are
always included:

- noMetricNameError now carries the series and renders it via
  formatLabelSet, matching every sibling validation error in the same
  file. This covers the validation (400) path used when metric name
  enforcement is enabled.
- The fatal error from tokenForLabels (the shard_by_all_labels=false
  path that produced the original 500 with no context) is wrapped with
  the series labels.

Fixes cortexproject#5802

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
@anxkhn anxkhn force-pushed the loop/cortex__001 branch from 090badd to 1565f0d Compare July 1, 2026 08:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The No metric name label error doesn't specify metric

1 participant