feat: add ReplacePartitions core class (PR1 of 2 for #637) by shangxinli · Pull Request #776 · apache/iceberg-cpp

shangxinli · 2026-06-24T00:05:57Z

Summary

Adds the iceberg::ReplacePartitions class — the dynamic partition overwrite operation. Each AddFile() registers the file's partition for replacement of all existing data files in that partition. Produces an overwrite snapshot with "replace-partitions"="true" in the summary. Unpartitioned tables replace all existing data files.

This is PR1 of 2 for #637. See tracking issue #775 for the split rationale.

What's in PR1

iceberg::ReplacePartitions class extending MergingSnapshotUpdate:
- Builder API: AddFile, ValidateAppendOnly, ValidateFromSnapshot, ValidateNoConflictingData, ValidateNoConflictingDeletes
- Apply / Summary / CleanUncommitted overrides
- Uses ManifestFilterManager for partition-based manifest filtering
SnapshotSummaryFields::kReplacePartitions = "replace-partitions" constant
ReplacePartitions forward declaration in type_fwd.h
CMake + Meson registration

What's deferred to PR2

Transaction::NewReplacePartitions() API wiring on Table / Transaction
End-to-end test file (replace_partitions_test.cc)

Design notes

Follows the Java ReplacePartitions semantics: operation type is overwrite, summary carries "replace-partitions"="true".
Mirrors the existing C++ FastAppend / MergeAppend patterns (factory Make(), MergingSnapshotUpdate base, builder error collector).
Unpartitioned table case treated as table-wide replace via DeleteByRowFilter(AlwaysTrue()), matching Java BaseReplacePartitions.

Test plan

Clean build (cmake --build build)
clang-format clean
End-to-end behavioral tests land in PR2 alongside the Table/Transaction API wiring

Closes part of #637 (PR2 will close it).
Tracking: #775

Adds iceberg::ReplacePartitions — the dynamic partition overwrite operation. Each AddFile() registers the file's partition for replacement of all existing data and delete files in that partition. Produces an overwrite snapshot with "replace-partitions=true" in the summary. Unpartitioned tables replace all existing data files. Extends MergingSnapshotUpdate (matching Java's BaseReplacePartitions) so the full data+delete manifest pipeline, custom-summary-property handling, and conflict-validation helpers are inherited. AddFile() unconditionally calls DropPartition(spec_id, file->partition) — for unpartitioned specs the partition value is empty and the filter manager matches every file in that spec, so no separate AlwaysTrue path is needed. Touched partitions are tracked in a PartitionSet; Validate() uses the partition-scoped overloads of ValidateAddedDataFiles / ValidateNoNewDeleteFiles, or skips entirely when no partitions were staged. Changes: * iceberg::ReplacePartitions extending MergingSnapshotUpdate with builder API (AddFile, ValidateAppendOnly, ValidateFromSnapshot, ValidateNoConflictingData, ValidateNoConflictingDeletes) and Validate() override. * SnapshotSummaryFields::kReplacePartitions = "replace-partitions". * MergingSnapshotUpdate::SetSummaryProperty promoted from private to protected so subclasses can stash custom summary entries that survive commit retry via the cached-rebuild path. * Forward declaration in type_fwd.h. * CMake + Meson source registration. Public API wiring (Table::NewReplacePartitions(), Transaction::NewReplacePartitions()) and end-to-end tests are deferred to PR2. Tracking: apache#775 Related: apache#637

manuzhang · 2026-06-24T09:16:51Z

+        current_metadata, starting_snapshot_id_, replaced_partitions_, snapshot, io));
+  }
+  if (validate_conflicting_deletes_) {
+    ICEBERG_RETURN_UNEXPECTED(ValidateNoNewDeleteFiles(


Java BaseReplacePartitions.validate also calls validateDeletedDataFiles, so concurrent overwrite/delete commits in the replaced partition are rejected

manuzhang · 2026-06-24T09:19:25Z

+  // the partition values are empty and naturally match every file under that
+  // spec — no separate AlwaysTrue path is needed, and validation stays scoped
+  // to the spec rather than the whole table.
+  ICEBERG_BUILDER_RETURN_IF_ERROR(DropPartition(spec_id, file->partition));


unpartitioned replace is spec-scoped, but Java treats it as table-wide.

…aFiles - AddFile() now uses DeleteByRowFilter(AlwaysTrue()) for unpartitioned specs instead of DropPartition with empty partition values, matching Java BaseReplacePartitions which treats unpartitioned tables as a table-wide replace. - Validate() now also calls ValidateDeletedDataFiles when ValidateNoConflictingDeletes is enabled, mirroring Java where validateNewDeletes gates both checks. This rejects concurrent overwrite/delete commits in the replaced partitions. - New replace_by_row_filter_ flag drives the AlwaysTrue path in Validate() for the unpartitioned case.

manuzhang · 2026-06-25T03:44:49Z

+  // No-op update: no partitions were staged and no table-wide replace was
+  // requested, so there is nothing to conflict with. Calling the validators
+  // with AlwaysTrue here would turn an empty builder into a full-table check.
+  if (!replace_by_row_filter_ && replaced_partitions_.empty()) {


Java rejects no-data-files replace. Suggest to call DataSpec() like Java.

Not sure where it's been implemented. Can you check again?

manuzhang

It looks replace_partitions.h is missing from src/iceberg/update/meson.build.

zhjwpku · 2026-06-25T07:30:56Z

In the PR description, should "the iceberg::ReplacePartitions class extending SnapshotUpdate" be changed to "the iceberg::ReplacePartitions class extending MergingSnapshotUpdate"?

zhjwpku · 2026-06-25T09:55:35Z

+
+namespace iceberg {
+
+/// \brief Replaces partitions in a table with new data files.


Do we need to mention This is provided to implement SQL compatible with Hive table operations but is not recommended. Instead, use the {@link OverwriteFiles overwrite API} to explicitly overwrite data. like java?

Changed to '/// \note This is provided to implement SQL compatible with Hive table operations but is not recommended. Instead, use OverwriteFiles to explicitly overwrite data.'

…date - Add replace_partitions.h to src/iceberg/update/meson.build (manuzhang) - Add Java-style note recommending OverwriteFiles for non-Hive use (zhjwpku) - Validate(): call DataSpec() to reject no-data-files replace, matching Java BaseReplacePartitions which throws when no file was added (manuzhang)

DataSpec() also errors on multi-spec stages, which would silently break a ReplacePartitions that touches files from more than one spec. Use the flags AddFile() sets (replace_by_row_filter_ / replaced_partitions_) instead — the same condition that already gated the no-op early-return below, now promoted to an error to match Java BaseReplacePartitions.

shangxinli · 2026-06-25T18:46:42Z

In the PR description, should "the iceberg::ReplacePartitions class extending SnapshotUpdate" be changed to "the iceberg::ReplacePartitions class extending MergingSnapshotUpdate"?

Updated the PR description — "extending SnapshotUpdate" → "extending MergingSnapshotUpdate". Thanks for catching that.

wgtmac · 2026-07-02T04:41:21Z

+  ICEBERG_BUILDER_ASSIGN_OR_RETURN(auto spec, base().PartitionSpecById(spec_id));
+
+  ICEBERG_BUILDER_RETURN_IF_ERROR(AddDataFile(file));
+  if (spec->fields().empty()) {


Java's PartitionSpec.isUnpartitioned() returns true not only for an empty field list but also for specs where all partition transforms are void/alwaysNull. With an all-void partition spec, BaseReplacePartitions.apply goes through the unpartitioned path and deletes by alwaysTrue(), replacing the whole table. This check only looks at fields().empty(), so C++ will treat an all-void spec as partitioned and call DropPartition, leaving files from other equivalent unpartitioned/all-void specs instead of doing the table-wide replace.

wgtmac · 2026-07-02T04:41:23Z

+  // Match Java BaseReplacePartitions: require at least one staged data file.
+  // Use the flags AddFile() sets — `DataSpec()` would also error on multi-spec
+  // stages and is not the guard we want here.
+  if (!replace_by_row_filter_ && replaced_partitions_.empty()) {


Java still calls dataSpec() from validate/apply, which enforces that exactly one partition spec is represented by the staged data files. This guard only checks that at least one partition was recorded, so a replace containing files from multiple spec IDs can commit in C++ even though Java rejects it. That can produce snapshot semantics that don't align with Java after spec evolution.

shangxinli force-pushed the feat/replace-partitions-core branch 4 times, most recently from ae973f6 to e33f433 Compare June 24, 2026 01:01

shangxinli mentioned this pull request Jun 24, 2026

feat: implement ReplacePartitions (two-PR plan) #775

Open

shangxinli force-pushed the feat/replace-partitions-core branch from e33f433 to 12dc230 Compare June 24, 2026 01:13

manuzhang reviewed Jun 24, 2026

View reviewed changes

manuzhang reviewed Jun 25, 2026

View reviewed changes

manuzhang suggested changes Jun 25, 2026

View reviewed changes

zhjwpku reviewed Jun 25, 2026

View reviewed changes

shangxinli added 2 commits June 25, 2026 09:36

wgtmac reviewed Jul 2, 2026

View reviewed changes


		namespace iceberg {

		/// \brief Replaces partitions in a table with new data files.

Uh oh!

Conversation

shangxinli commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's in PR1

What's deferred to PR2

Design notes

Test plan

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

manuzhang left a comment

Choose a reason for hiding this comment

Uh oh!

zhjwpku commented Jun 25, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shangxinli commented Jun 25, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

shangxinli commented Jun 24, 2026 •

edited

Loading