many_topics_test: allow Prometheus oversized-allocation WARN#30968
Closed
sjust-redpanda wants to merge 1 commit into
Closed
many_topics_test: allow Prometheus oversized-allocation WARN#30968sjust-redpanda wants to merge 1 commit into
sjust-redpanda wants to merge 1 commit into
Conversation
The admin-shard Prometheus exposition path (metric_aggregate_by_labels) allocates a >128 KiB hash-table bucket array per scrape at this test's ~40k-partition metric cardinality, tripping a non-fatal seastar_memory oversized-allocation WARN that fails raise_on_bad_logs. The harness scrapes metrics throughout every lifecycle variant, so allow the (deterministic, 208896-byte) line across the ManyTopicsTest lifecycle tests via a shared MANY_TOPICS_LOG_ALLOW_LIST. Matching the exact size avoids masking unrelated admin-shard oversized allocations. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
I am fixing the root of the issue here: redpanda-data/seastar#289 Not sure we need/want this. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ManyTopicsTestlifecycle tests failraise_on_bad_logswith a non-fatalseastar_memory - oversized allocation: 208896 bytesWARN on the admin shard.Symbolicating the backtrace (against the nightly binary actually under test)
shows it originates in Seastar's Prometheus text-exposition path: a
/metrics(or
/public_metrics) scrape makesmetric_aggregate_by_labelsbuild aper-request
unordered_map<label_key, labels_value>whose hash-table rehashallocates a >128 KiB bucket array at this test's ~40k-partition metric
cardinality. The harness scrapes metrics throughout every lifecycle variant, so
this can surface in any of them.
This adds a shared
MANY_TOPICS_LOG_ALLOW_LISTallowing the specific,deterministic
208896-byte admin-shard line across the lifecycle tests, to stopthe CI bleed while the underlying allocation is addressed separately. Matching
the exact size avoids masking unrelated admin-shard oversized allocations. The
underlying issue is tracked in CORE-16754 / CORE-16433.
Backports Required
Release Notes
🤖 Generated with Claude Code