RFC ts-ct-migration: archival flag + structural routing [PR 4]#30892
Draft
sjust-redpanda wants to merge 14 commits into
Draft
RFC ts-ct-migration: archival flag + structural routing [PR 4]#30892sjust-redpanda wants to merge 14 commits into
sjust-redpanda wants to merge 14 commits into
Conversation
This was referenced Jun 23, 2026
sjust-redpanda
force-pushed
the
sjust/ts-ct-migration-pr4
branch
from
bd5f139 to
f9e4fcc
Compare
In order to dynamically switch a partition from TS to CT/TSv2, we need to be able to enable the ctp_stm on the fly. The simplest way to do that is to simply attach an empty ctp_stm to every partition. Unfortunately, with LRO unset, ctp_stm previously returned offset::min() for get_max_collectible_offset() preventing log trimming. This commit modifies that behavior to return offset::max() if _max_applied_epoch is empty making the presence of a ctp_stm harmless until advance_epoch() is called. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tions is_applicable_for: pre-install the (idle) ctp_stm on tiered-storage partitions when cloud topics are available, so a tiered->cloud migration needs no runtime STM install -- the STM is already present and inert until the partition becomes a cloud topic. STM membership is fixed at construction, so it has to be there beforehand. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…verload Split advance_reconciled_offset so the reconciled log offset can be supplied explicitly instead of always deriving it from the raft offset translator. The existing kafka-offset-only entry point now delegates, deriving the log offset as before. This lets a caller seed the reconciliation baseline of a TS-migrated partition directly to the migration boundary -- passing the precise raft offset of the last uploaded tiered-storage segment -- so the already reconciled region can be trimmed without re-deriving the offset. ctp_stm carries no migration state of its own; this is expressed purely as "reconciliation starts at offset N". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
During migration, we need the newly configured TSv2 or CT partition to continue to behave as an archival stm. Thus, we need the archival stm to continue to exist after the config switch. The simplest way to do this is to allow it to exist generally on CT/TSv2 topics. Drop the cloud_topic_enabled() == false guard from archival_metadata_stm_factory::is_applicable_for, so every cloud-topic partition carries an archival STM. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A cloud-mode partition whose storage mode has been flipped reports is_archival_enabled() == false, which would make max_removable_local_log_offset collect_all and stop constraining local-log truncation -- evicting tiered-storage data not yet uploaded. Keep constraining via cloud_recoverable_offset() while the partition still holds archived data (a non-empty live manifest or a spillover archive), introduced here as holds_archived_data(); once the manifest is emptied, trimming resumes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Drop the cloud_topic_enabled() guard from log_eviction_stm_factory so the eviction STM is installed on cloud-topic partitions too. A partition migrating from tiered storage is still served from its local log + archival manifest and must keep supporting prefix truncation (DeleteRecords) while migrating; the trigger flips storage_mode to cloud without reconstructing the partition, and a still-migrating partition is reconstructed cloud-mode on restart/recovery, so the STM must already be present. On a native cloud topic it is an inert passenger. Subsequent integration tests will provide coverage. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Persist a partition-local storage mode (model::redpanda_storage_mode) in the partition properties STM, alongside the existing writes_disabled property. This is the partition's own durable record of its storage mode, decoupled from the topic config; a later change exposes it on ntp_config as partition_mode (vs the topic-config-derived topic_mode) so the TS->CT migration can be modelled as reconciling partition_mode toward topic_mode. Adds update_partition_mode_cmd / operation_type::update_partition_mode, carries partition_mode on state_snapshot and raft_snapshot (serde version bump to v2, compat_version unchanged so older snapshots decode with partition_mode = unset), and the set_partition_mode / partition_mode / sync_partition_mode API. apply is refactored to dispatch on operation_type and copy-modify the current snapshot so each property update preserves the others. partition_mode updates are idempotent by value and ordered by log offset (no per-property revision). Also adds a change callback (set_partition_mode_change_callback), fired when partition_mode changes on apply or on raft-snapshot restore, so an owner can re-read partition_mode() and propagate it (the STM had no change notification; writes_disabled is polled). Tests cover: default unset, set/idempotent/change with cross-node consistency; partition_mode and writes_disabled are independent on the shared snapshot; raft snapshot carries partition_mode at the right offsets; recovery (local kvstore snapshot + replay, and raft-snapshot recovery after dropping a node's data); and the change callback fires on a real change but not on an idempotent no-op. Inert: no callers yet (the command is wired up and gated behind a feature in follow-up commits). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Declare feature::tiered_to_cloud_migration (bit 18, explicit_only, v26_2_1). It gates all behaviors of the TS->CT migration capability across the PR stack: the partition_mode write introduced in this base branch, and the migration trigger/cutover in the later PRs. explicit_only means it does not auto-activate and carries no mixed-version compatibility commitment until an operator opts in. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…mode
Introduce two storage-mode accessors on ntp_config:
- topic_mode() the mode the topic config requests (the existing
_overrides->storage_mode; the "desired" mode).
- partition_mode() the partition's own durable mode, fed from
partition_properties via set_partition_mode(); the
"actual" mode. Falls back to topic_mode() when unset.
The serving/behavior predicates (is_archival_enabled, is_remote_fetch_enabled,
is_tiered_storage, cloud_topic_enabled, is_tiered_cloud) now key on
partition_mode() instead of reading _overrides->storage_mode directly. The
legacy shadow_indexing fallback (used when the mode is unset) is preserved
unchanged, since archival-only/fetch-only SI modes have no redpanda_storage_mode
equivalent.
TS->CT migration wil lbe modelled as partition_mode lagging topic_mode
(still `tiered` while the topic config says `cloud`) until cutover, with
serving keyed on the actual mode.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… ntp_config On partition start, read partition_properties_stm::partition_mode() and push it into the log's ntp_config (storage::log gains a set_partition_mode method, mirroring set_overrides), and register the STM's change callback so the value is re-pushed whenever the STM applies a partition_mode update or restores a raft snapshot. ntp_config::partition_mode() therefore tracks the partition's durable mode, falling back to topic_mode() while unset. No feature guard here: reading replicated STM state and pushing it into the local ntp_config is always safe. Writing partition_mode (which requires the tiered_to_cloud_migration feature) is added in the bootstrap commit; until then the value is always unset. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Record the partition's durable storage mode (partition_mode) in
partition_properties. It is populated:
- at partition creation, from the topic-config mode, when the migration
feature is active (partition_manager::manage);
- on becoming leader, for a partition that predates this (maybe_bootstrap_
partition_mode); a node that became leader after an in-place ts->ct switch
keys on the archival manifest (holds_archived_data) since topic_mode()
already reads cloud;
- on migration-feature activation, sweeping current leaders that were elected
while the feature was inactive.
The creation-time classification (set_creation_partition_mode) is preferred by
the bootstrap over re-reading topic_mode(), so the mode does not depend on the
async config propagation a runtime re-derivation would race.
Also add partition_properties_stm::seed_partition_mode(), which pre-writes the
kvstore-backed local snapshot before the STM starts so a partition restored with
a non-empty log comes up with partition_mode already set (the seed survives
start(), whose initial cleanup only runs for an empty log). Used by the
migration-recovery path.
partition_mode stays inert here; serving/behavior accessors begin keying on it
in later PRs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…artition A partition served from tiered storage whose local log does not cover its cloud data (e.g. mid tiered->cloud migration after cluster recovery wiped the local log) reports high watermark 0, hiding the recovered prefix from ListOffsets and consumers. Report max(local, next_cloud_offset()) when cloud data is available; for a healthy partition the cloud watermark never exceeds the local one, so this is a no-op. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Assert that flipping a topic's storage mode to cloud (topic_mode) while the durable partition_mode lags at tiered leaves the partition served as tiered storage with no observable change: replicated_partition offsets / size estimates / timequery are byte-for-byte unchanged, make_partition_proxy still routes to the replicated_partition path, and prefix truncation still works and advances the start. Regresses the timequery corner (must consult the imported prefix, not the local-log start). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sjust-redpanda
force-pushed
the
sjust/ts-ct-migration-pr4
branch
from
f9e4fcc to
8525d9d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🚧 NOT YET READY FOR REVIEW
Summary
Add the partition-raft state and routing/trim rules that let a cloud-mode partition keep being served from tiered storage while it still holds tiered-storage data. Routing keys on a structural signal — the archival STM still holding data — not on the storage-mode config, so it is identical on every replica and robust to unordered config propagation. Behavior-preserving for native cloud topics and plain tiered partitions, and inert until the migration trigger is enabled (a later PR).
Stacked on PR2. Base is
dev, so the diff is cumulative (it includes PR2); commits new in this PR: the archival-STM migration flag, serve-cloud-reads, the kafka structural route, and two routing unit tests.What's in this PR
archival: a{none, migrating}flag on the archival STM (set_migration_state) plusreset_metadatato empty the manifest, with no offset/boundary carried.partition::is_remote_fetch_enabled_or_migrating()— keyed on the archival STM holding tiered data (live manifest non-empty or spillover archive present). (The matchingmax_removable_local_log_offsetlocal-trim clamp lives in PR2.)make_partition_proxy: a highest-priority branch routing a partition that holds tiered data to the tiered-storage path; native cloud topics (empty archival STM) fall through to the cloud-topic path.Testing
cloud_storage_e2e_test: a spilled partition with an empty live manifest still routes to tiered storage;replicated_partitionAPI parity for a migrating partition vs. a plain tiered one. Both green.archival_metadata_stm_test/_gtestgreen.🤖 Generated with Claude Code
Design Doc
PR 0 · PR 1 · PR 2 · PR 3 · PR 4 · PR 5 · PR 6 · PR 7 · PR 8