Metrics aggregation warning v26.2.x#290
Open
WillemKauf wants to merge 6 commits into
Open
Conversation
Wire ankerl::unordered_dense into the build as a required dependency: find_package in SeastarDependencies, link unordered_dense::unordered_dense PUBLIC, a cooking recipe pinned to the commit Redpanda consumes (v4.4.0), and propagate its include flags to consumers via the pkg-config file. This is a prerequisite for the upcoming chunked_hash_map container.
Wire abseil into the build as a required dependency for chunked_hash_map's absl::Hash support: find_package(absl CONFIG), link absl::hash PUBLIC, a cooking recipe pinned to the same LTS release Redpanda consumes (20250814.1), the absl_hash pkg-config requirement for consumers, and the abseil dev packages for the distros that reliably carry them. This is a prerequisite for the upcoming chunked_hash_map container.
Port chunked_vector from Redpanda: a random-access vector whose storage is split across fixed-size fragments instead of one contiguous block, so it grows without large reallocations and keeps element addresses stable across growth. Adapted to Seastar conventions (seastar namespace, SEASTAR_ASSERT, .hh header) with a Boost.Test port of the test suite.
Port chunked_hash_map/chunked_hash_set from Redpanda: ankerl's segmented hash map/set backed by chunked_vector storage, suited to maps that grow large (scaling with partitions or topics) without large contiguous allocations. Hashing dispatches to absl::Hash when a type provides AbslHashValue, otherwise to unordered_dense's hash. Adapted to the seastar namespace, with a Boost.Test port of the test suite.
Replace the std::unordered_map backing metric_aggregate_by_labels with chunked_hash_map so per-label aggregation storage grows without large contiguous allocations when exporting many metric series.
When a single metric family aggregates to a very large number of series, that usually signals unbounded metric label cardinality, which is typically a bug. Add an optional config::aggregation_warn_threshold (default 10000) and emit a rate-limited warning, keyed on the metric family name, from both the text and protobuf scrape paths. Set the threshold to std::nullopt to opt out of the warning entirely.
There was a problem hiding this comment.
Pull request overview
This PR introduces a rate-limited warning for suspiciously high Prometheus metric aggregation cardinality, and adds new chunked container types (chunked_vector, chunked_hash_map) that are then used to back the aggregation map to reduce allocation overhead at large cardinalities.
Changes:
- Add configurable, per-metric-family, rate-limited logging when aggregation produces more than N series.
- Introduce
chunked_vectorandchunked_hash_map(plus unit tests) and switch Prometheus label-aggregation storage tochunked_hash_map. - Add build/dependency wiring for Abseil + unordered_dense (CMake, cooking, pkg-config, dependency install script).
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/CMakeLists.txt | Registers the new unit tests for the new containers. |
| tests/unit/chunked_vector_test.cc | Adds unit tests validating chunked_vector behavior and iterator properties. |
| tests/unit/chunked_hash_map_test.cc | Adds compile/usage tests for chunked_hash_map and chunked_hash_map_from_range. |
| src/core/prometheus.cc | Adds the aggregation-cardinality warning and invokes it from text/protobuf paths. |
| src/core/prometheus-impl.hh | Switches aggregation storage from std::unordered_map to chunked_hash_map and exposes .size(). |
| pkgconfig/seastar.pc.in | Exposes Abseil + unordered_dense to pkg-config consumers (cflags/requires). |
| install-dependencies.sh | Adds distro packages for Abseil. |
| include/seastar/core/prometheus.hh | Adds config::aggregation_warn_threshold. |
| include/seastar/core/chunked_vector.hh | Introduces chunked_vector container implementation. |
| include/seastar/core/chunked_hash_map.hh | Introduces chunked_hash_map/chunked_hash_set wrappers over unordered_dense segmented containers. |
| cooking_recipe.cmake | Adds cooking ingredients for unordered_dense and Abseil. |
| CMakeLists.txt | Fetches unordered_dense for Seastar’s own build (when not found) and links Abseil/unordered_dense. |
| cmake/SeastarDependencies.cmake | Adds dependency discovery for unordered_dense and Abseil. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+926
to
+928
| static thread_local std::unordered_map<sstring, logger::rate_limit> rate_limits; | ||
| auto it = rate_limits.try_emplace( | ||
| sstring(family_name), std::chrono::minutes(5)).first; |
Comment on lines
+929
to
+931
| seastar_logger.log( | ||
| log_level::error, it->second, | ||
| "prometheus: metric family '{}' aggregated to {} series, exceeding the " |
Comment on lines
21
to
29
| #include <boost/container_hash/hash_fwd.hpp> | ||
| #include <ranges> | ||
| #include <seastar/core/chunked_hash_map.hh> | ||
| #include <seastar/core/metrics.hh> | ||
| #include <seastar/core/metrics_api.hh> | ||
| #include <string_view> | ||
|
|
||
| #include <unordered_map> | ||
| #include <vector> | ||
| #include <string> |
Comment on lines
+138
to
+142
| ~chunked_vector() noexcept = default; | ||
|
|
||
| chunked_vector copy() const noexcept { return *this; } | ||
|
|
||
| auto get_allocator() const { return _frags.get_allocator(); } |
Comment on lines
+58
to
+63
| // calculate the maximum number of elements per fragment while | ||
| // keeping the element count a power of two | ||
| static consteval size_t calc_elems_per_frag() { | ||
| size_t max = max_allocation_size / sizeof(T); | ||
| return std::bit_floor(max); | ||
| } |
Comment on lines
+268
to
+288
| void reserve(size_t new_cap) { | ||
| static constexpr size_t elems_per_frag = calc_elems_per_frag(); | ||
| if (new_cap > _capacity) { | ||
| if (_frags.empty()) { | ||
| auto& frag = _frags.emplace_back(); | ||
| frag.reserve(std::min(elems_per_frag, new_cap)); | ||
| _capacity = frag.capacity(); | ||
| } else if (_frags.size() == 1) { | ||
| auto& frag = _frags.front(); | ||
| frag.reserve(std::min(elems_per_frag, new_cap)); | ||
| _capacity = frag.capacity(); | ||
| } | ||
| // We only reserve the first fragment as all fragments after the | ||
| // first are allocated at the maximum size, so we don't save | ||
| // anything in terms of reallocs after fully allocating the | ||
| // first fragment. In addition, due to cache locality, it's | ||
| // better to delay the allocations of those other fragments | ||
| // until they're going to be used. | ||
| } | ||
| update_generation(); | ||
| } |
Comment on lines
+368
to
+378
| iter& operator+=(ssize_t n) { | ||
| check_generation(); | ||
| _index += n; | ||
| return *this; | ||
| } | ||
|
|
||
| iter& operator-=(ssize_t n) { | ||
| check_generation(); | ||
| _index -= n; | ||
| return *this; | ||
| } |
Comment on lines
+440
to
+442
| friend ssize_t operator-(const iter& a, const iter& b) { | ||
| return a._index - b._index; | ||
| } |
Comment on lines
+91
to
96
| # Not REQUIRED: unordered_dense is header-only and not reliably packaged by | ||
| # distributions. When it is not installed, Seastar's own build fetches it | ||
| # (see CMakeLists.txt); consumers that lack it must provide it themselves. | ||
| seastar_find_dep (unordered_dense) | ||
| seastar_find_dep (absl CONFIG REQUIRED) | ||
| seastar_find_dep (GnuTLS 3.7.4) |
Comment on lines
24
to
40
| @@ -32,9 +33,9 @@ dpdk_libs=$<JOIN:@dpdk_LIBRARIES@, > | |||
| seastar_cflags=${seastar_include_flags} $<JOIN:$<TARGET_PROPERTY:seastar,INTERFACE_COMPILE_OPTIONS>, > -D$<JOIN:$<TARGET_PROPERTY:seastar,INTERFACE_COMPILE_DEFINITIONS>, -D> | |||
| seastar_libs=${libdir}/$<TARGET_FILE_NAME:seastar> @Seastar_SPLIT_DWARF_FLAG@ $<JOIN:@Seastar_Sanitizers_OPTIONS@, > | |||
|
|
|||
| Requires: liblz4 >= 1.7.3 | |||
| Requires: liblz4 >= 1.7.3, absl_hash | |||
| Requires.private: gnutls >= 3.2.26, protobuf >= 2.5.0, hwloc >= 1.11.2, $<$<BOOL:@Seastar_IO_URING@>:liburing $<ANGLE-R>= 2.0, >yaml-cpp >= 0.5.1 | |||
| Conflicts: | |||
| Cflags: @Seastar_CXX_COMPILE_OPTION@ ${boost_cflags} ${c_ares_cflags} ${fmt_cflags} ${liburing_cflags} ${lksctp_tools_cflags} ${seastar_cflags} | |||
| Cflags: @Seastar_CXX_COMPILE_OPTION@ ${boost_cflags} ${c_ares_cflags} ${fmt_cflags} ${unordered_dense_cflags} ${liburing_cflags} ${lksctp_tools_cflags} ${seastar_cflags} | |||
| Libs: ${seastar_libs} ${boost_program_options_libs} ${c_ares_libs} ${fmt_libs} | |||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.