Skip to content

Two-stage ANALYZE with adaptive count-min sketch params #30206

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ztlpn merged 14 commits into from
Dec 9, 2025
Merged

Two-stage ANALYZE with adaptive count-min sketch params #30206

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
4a54224
statistics: support multiple stat types in SaveStatisticsQuery
ztlpn Oct 13, 2025
51be969
add proto msg for simple column statistics
ztlpn Dec 1, 2025
9baf8ae
calculate simple column statistics (count distinct)
ztlpn Dec 1, 2025
6355776
save column type information
ztlpn Dec 2, 2025
7970ffa
2-stage analyze with adaptive count-min sketch params
ztlpn Dec 2, 2025
e71469e
move TSelectBuilder to a separate file
ztlpn Dec 3, 2025
f6bfb00
add clarifying comment
ztlpn Dec 3, 2025
1f8c255
change Value column type to String
ztlpn Dec 4, 2025
026201c
statistics: refactor test helpers
ztlpn Dec 4, 2025
0b48dcc
tests: split CreateUniformTable into two functions
ztlpn Dec 4, 2025
00c0b07
more test helpers refactoring (passes existing tests)
ztlpn Dec 5, 2025
443978f
review fixes
ztlpn Dec 8, 2025
069fded
tests: single function to check count-min values/absence
ztlpn Dec 8, 2025
109e564
review fixes
ztlpn Dec 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions ydb/core/protos/statistics.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -198,3 +198,8 @@ message TEvAggregateStatisticsResponse {
}
repeated TFailedTablet FailedTablets = 3;
}

message TSimpleColumnStatistics {
optional uint64 Count = 1;
optional uint64 CountDistinct = 2;
};
28 changes: 14 additions & 14 deletions ydb/core/statistics/aggregator/aggregator_impl.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -608,25 +608,23 @@ void TStatisticsAggregator::SaveStatisticsToTable() {

PendingSaveStatistics = false;

std::vector<ui32> columnTags;
std::vector<TString> data;
auto count = CountMinSketches.size();
if (count == 0) {
Send(SelfId(), new TEvStatistics::TEvSaveStatisticsQueryResponse(
Ydb::StatusIds::SUCCESS, {}, TraversalPathId));
return;
}
columnTags.reserve(count);
data.reserve(count);
std::vector<TStatisticsItem> items = std::exchange(StatisticsToSave, {});

for (auto& [tag, sketch] : CountMinSketches) {
columnTags.push_back(tag);
if (!ColumnNames.contains(tag)) {
continue;
}
TString strSketch(sketch->AsStringBuf());
data.push_back(strSketch);
items.emplace_back(tag, EStatType::COUNT_MIN_SKETCH, std::move(strSketch));
}

if (items.empty()) {
Send(SelfId(), new TEvStatistics::TEvSaveStatisticsQueryResponse(
Ydb::StatusIds::SUCCESS, {}, TraversalPathId));
return;
}

Register(CreateSaveStatisticsQuery(SelfId(), Database,
TraversalPathId, EStatType::COUNT_MIN_SKETCH, std::move(columnTags), std::move(data)));
Register(CreateSaveStatisticsQuery(SelfId(), Database, TraversalPathId, std::move(items)));
}

void TStatisticsAggregator::DeleteStatisticsFromTable() {
Expand Down Expand Up @@ -677,6 +675,8 @@ void TStatisticsAggregator::ScheduleNextAnalyze(NIceDb::TNiceDb& db, const TActo
UpdateForceTraversalTableStatus(
TForceTraversalTable::EStatus::AnalyzeStarted, operation.OperationId, operationTable, db);

// operation.Types field is not used, TAnalyzeActor will determine suitable
// statistic types itself.
Comment on lines +678 to +679
Copy link

Copilot AI Dec 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment says "operation.Types field is not used" but this statement is incomplete. It should clarify what this field was previously used for and why it's no longer needed, to help future maintainers understand the change.

Suggested change
// operation.Types field is not used, TAnalyzeActor will determine suitable
// statistic types itself.
// Previously, the operation.Types field was used to specify which statistic types
// should be collected and analyzed for each force traversal operation. This approach
// was replaced to allow TAnalyzeActor to determine the suitable statistic types itself,
// based on the current table schema and configuration. As a result, operation.Types is
// no longer needed and is ignored here.

Copilot uses AI. Check for mistakes.
ctx.RegisterWithSameMailbox(new TAnalyzeActor(
SelfId(), operation.OperationId, operation.DatabaseName, operationTable.PathId,
operationTable.ColumnTags));
Expand Down
1 change: 1 addition & 0 deletions ydb/core/statistics/aggregator/aggregator_impl.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -284,6 +284,7 @@ class TStatisticsAggregator : public TActor<TStatisticsAggregator>, public NTabl

bool IsStatisticsTableCreated = false;
bool PendingSaveStatistics = false;
std::vector<TStatisticsItem> StatisticsToSave;
bool PendingDeleteStatistics = false;

std::vector<NScheme::TTypeInfo> KeyColumnTypes;
Expand Down
Loading
Loading