[v26.2.x] Port chunked_vector, chunked_hash_map, and use it in prometheus-impl.hh#289
Merged
WillemKauf merged 5 commits intoJun 30, 2026
Merged
Conversation
Wire ankerl::unordered_dense into the build as a required dependency: find_package in SeastarDependencies, link unordered_dense::unordered_dense PUBLIC, a cooking recipe pinned to the commit Redpanda consumes (v4.4.0), and propagate its include flags to consumers via the pkg-config file. This is a prerequisite for the upcoming chunked_hash_map container.
Wire abseil into the build as a required dependency for chunked_hash_map's absl::Hash support: find_package(absl CONFIG), link absl::hash PUBLIC, a cooking recipe pinned to the same LTS release Redpanda consumes (20250814.1), the absl_hash pkg-config requirement for consumers, and the abseil dev packages for the distros that reliably carry them. This is a prerequisite for the upcoming chunked_hash_map container.
Port chunked_vector from Redpanda: a random-access vector whose storage is split across fixed-size fragments instead of one contiguous block, so it grows without large reallocations and keeps element addresses stable across growth. Adapted to Seastar conventions (seastar namespace, SEASTAR_ASSERT, .hh header) with a Boost.Test port of the test suite.
Port chunked_hash_map/chunked_hash_set from Redpanda: ankerl's segmented hash map/set backed by chunked_vector storage, suited to maps that grow large (scaling with partitions or topics) without large contiguous allocations. Hashing dispatches to absl::Hash when a type provides AbslHashValue, otherwise to unordered_dense's hash. Adapted to the seastar namespace, with a Boost.Test port of the test suite.
Replace the std::unordered_map backing metric_aggregate_by_labels with chunked_hash_map so per-label aggregation storage grows without large contiguous allocations when exporting many metric series.
There was a problem hiding this comment.
Pull request overview
This PR ports chunked_vector and chunked_hash_map into Seastar, wires in new third-party dependencies (ankerl::unordered_dense, Abseil), and switches Prometheus metric label aggregation to use chunked_hash_map to avoid large contiguous allocations in the aggregation path.
Changes:
- Added
include/seastar/core/chunked_vector.hhandinclude/seastar/core/chunked_hash_map.hhplus new unit tests for both. - Introduced build + packaging integration for
unordered_dense(including a FetchContent fallback) and added Abseil as a dependency. - Updated
src/core/prometheus-impl.hhto usechunked_hash_mapinstead ofstd::unordered_mapfor label aggregation storage.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
include/seastar/core/chunked_vector.hh |
New chunked vector container implementation (iterator + allocation strategy). |
include/seastar/core/chunked_hash_map.hh |
New segmented hash map/set aliases over unordered_dense + helpers/formatters. |
src/core/prometheus-impl.hh |
Swaps label aggregation map to chunked_hash_map to reduce oversized allocations. |
cmake/SeastarDependencies.cmake |
Adds discovery of new dependencies for both in-tree build and find_package(Seastar) consumers. |
CMakeLists.txt |
Adds FetchContent fallback for unordered_dense, installs/exports updated Seastar interface deps. |
pkgconfig/seastar.pc.in |
Extends pkg-config flags/requirements for new public headers/deps. |
cooking_recipe.cmake |
Adds pinned unordered_dense and absl ingredients. |
install-dependencies.sh |
Installs absl packages on Debian/Fedora/Arch. |
tests/unit/CMakeLists.txt |
Registers new unit tests. |
tests/unit/chunked_vector_test.cc |
Adds chunked_vector behavioral + iterator tests and internal validator. |
tests/unit/chunked_hash_map_test.cc |
Adds basic compile/move/from-range tests for chunked_hash_map. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| #include <initializer_list> | ||
| #include <iterator> | ||
| #include <ostream> | ||
| #include <ranges> |
| */ | ||
| template<typename T> | ||
| class chunked_vector { | ||
| static constexpr size_t max_allocation_size = 128UL * 1024; |
Comment on lines
+368
to
+372
| iter& operator+=(ssize_t n) { | ||
| check_generation(); | ||
| _index += n; | ||
| return *this; | ||
| } |
Comment on lines
+374
to
+378
| iter& operator-=(ssize_t n) { | ||
| check_generation(); | ||
| _index -= n; | ||
| return *this; | ||
| } |
Comment on lines
+440
to
+442
| friend ssize_t operator-(const iter& a, const iter& b) { | ||
| return a._index - b._index; | ||
| } |
Comment on lines
+86
to
+89
| if (f.capacity() > std::decay_t<decltype(v)>::max_frag_bytes()) { | ||
| return fmt::format( | ||
| "fragment {} capacity over max_frag_bytes ({})", i, calc_cap); | ||
| } |
| } while (0) | ||
|
|
||
| /** | ||
| * Proxy that applies a consistency check before deference |
| size_t | ||
| memory_usage_lower_bound(const chunked_hash_map<K, V, Hash, EqualTo>& m) { | ||
| return m.bucket_count() | ||
| * sizeof(typename chunked_hash_map<K, V>::bucket_type) |
Comment on lines
+425
to
+429
| if (NOT TARGET unordered_dense::unordered_dense) | ||
| include (FetchContent) | ||
| FetchContent_Declare (unordered_dense | ||
| URL https://github.com/martinus/unordered_dense/archive/f30ed41b58af8c79788e8581fe57a6faf856258e.tar.gz | ||
| URL_HASH MD5=0370e4a35c1e573aa6639fde97f0c93f) |
Comment on lines
+91
to
+95
| # Not REQUIRED: unordered_dense is header-only and not reliably packaged by | ||
| # distributions. When it is not installed, Seastar's own build fetches it | ||
| # (see CMakeLists.txt); consumers that lack it must provide it themselves. | ||
| seastar_find_dep (unordered_dense) | ||
| seastar_find_dep (absl CONFIG REQUIRED) |
StephanDollberg
approved these changes
Jun 30, 2026
This was referenced Jun 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ankerl::unordered_densedepabsldepchunked_vector.hchunked_hash_map.hchunked_hash_mapinprometheus-impl.hhto avoid oversized allocations in the metrics aggregation pathFor v26.1.x patch, see: #288