RFC ts-ct-migration: partition_mode substrate [PR 0]#30977
Draft
sjust-redpanda wants to merge 5 commits into
Draft
RFC ts-ct-migration: partition_mode substrate [PR 0]#30977sjust-redpanda wants to merge 5 commits into
sjust-redpanda wants to merge 5 commits into
Conversation
This was referenced Jun 30, 2026
Contributor
Author
|
Working on updating downstream PRs to use this mechanism rather than clumsily looking at the archival_metadata_stm state. |
Persist a partition-local storage mode (model::redpanda_storage_mode) in the partition properties STM, alongside the existing writes_disabled property. This is the partition's own durable record of its storage mode, decoupled from the topic config; a later change exposes it on ntp_config as partition_mode (vs the topic-config-derived topic_mode) so the TS->CT migration can be modelled as reconciling partition_mode toward topic_mode. Adds update_partition_mode_cmd / operation_type::update_partition_mode, carries partition_mode on state_snapshot and raft_snapshot (serde version bump to v2, compat_version unchanged so older snapshots decode with partition_mode = unset), and the set_partition_mode / partition_mode / sync_partition_mode API. apply is refactored to dispatch on operation_type and copy-modify the current snapshot so each property update preserves the others. partition_mode updates are idempotent by value and ordered by log offset (no per-property revision). Also adds a change callback (set_partition_mode_change_callback), fired when partition_mode changes on apply or on raft-snapshot restore, so an owner can re-read partition_mode() and propagate it (the STM had no change notification; writes_disabled is polled). Tests cover: default unset, set/idempotent/change with cross-node consistency; partition_mode and writes_disabled are independent on the shared snapshot; raft snapshot carries partition_mode at the right offsets; recovery (local kvstore snapshot + replay, and raft-snapshot recovery after dropping a node's data); and the change callback fires on a real change but not on an idempotent no-op. Inert: no callers yet (the command is wired up and gated behind a feature in follow-up commits). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Declare feature::tiered_to_cloud_migration (bit 18, explicit_only, v26_2_1). It gates all behaviors of the TS->CT migration capability across the PR stack: the partition_mode write introduced in this base branch, and the migration trigger/cutover in the later PRs. explicit_only means it does not auto-activate and carries no mixed-version compatibility commitment until an operator opts in. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…mode
Introduce two storage-mode accessors on ntp_config:
- topic_mode() the mode the topic config requests (the existing
_overrides->storage_mode; the "desired" mode).
- partition_mode() the partition's own durable mode, fed from
partition_properties via set_partition_mode(); the
"actual" mode. Falls back to topic_mode() when unset.
The serving/behavior predicates (is_archival_enabled, is_remote_fetch_enabled,
is_tiered_storage, cloud_topic_enabled, is_tiered_cloud) now key on
partition_mode() instead of reading _overrides->storage_mode directly. The
legacy shadow_indexing fallback (used when the mode is unset) is preserved
unchanged, since archival-only/fetch-only SI modes have no redpanda_storage_mode
equivalent.
TS->CT migration wil lbe modelled as partition_mode lagging topic_mode
(still `tiered` while the topic config says `cloud`) until cutover, with
serving keyed on the actual mode.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… ntp_config On partition start, read partition_properties_stm::partition_mode() and push it into the log's ntp_config (storage::log gains a set_partition_mode method, mirroring set_overrides), and register the STM's change callback so the value is re-pushed whenever the STM applies a partition_mode update or restores a raft snapshot. ntp_config::partition_mode() therefore tracks the partition's durable mode, falling back to topic_mode() while unset. No feature guard here: reading replicated STM state and pushing it into the local ntp_config is always safe. Writing partition_mode (which requires the tiered_to_cloud_migration feature) is added in the bootstrap commit; until then the value is always unset. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Record the partition's durable storage mode (partition_mode) in
partition_properties. It is populated:
- at partition creation, from the topic-config mode, when the migration
feature is active (partition_manager::manage);
- on becoming leader, for a partition that predates this (maybe_bootstrap_
partition_mode); a node that became leader after an in-place ts->ct switch
keys on the archival manifest (holds_archived_data) since topic_mode()
already reads cloud;
- on migration-feature activation, sweeping current leaders that were elected
while the feature was inactive.
The creation-time classification (set_creation_partition_mode) is preferred by
the bootstrap over re-reading topic_mode(), so the mode does not depend on the
async config propagation a runtime re-derivation would race.
Also add partition_properties_stm::seed_partition_mode(), which pre-writes the
kvstore-backed local snapshot before the STM starts so a partition restored with
a non-empty log comes up with partition_mode already set (the seed survives
start(), whose initial cleanup only runs for an empty log). Used by the
migration-recovery path.
partition_mode stays inert here; serving/behavior accessors begin keying on it
in later PRs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sjust-redpanda
force-pushed
the
sjust/ts-ct-migration-pr0
branch
from
e6a289d to
39313f9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🚧 NOT YET READY FOR REVIEW
Summary
Foundational base branch for the tiered-storage→cloud-topics migration.
Introduces
partition_mode: a partition's own durable storage mode, recordedin
partition_propertiesand decoupled from the topic config, so the migrationcan be modelled as
partition_modelagging the topic-config-derivedtopic_modeuntil cutover.
Commits:
cluster: carrypartition_modeinpartition_properties_stm— persist aper-partition
model::redpanda_storage_modealongsidewrites_disabled(serdebumped to v2,
compat_versionunchanged so older snapshots decodeunset);set_partition_mode/partition_mode()/sync_partition_modeAPI; and achange callback fired on the partition_mode apply branch and on raft-snapshot
restore.
features: add thetiered_to_cloud_migrationfeature — the singleexplicit_onlyflag that gates the whole migration capability across thestack; the
partition_modewrite is guarded on it, and the later PRs reuse it.storage: splitntp_configstorage mode intotopic_mode+partition_mode—
topic_mode()is the topic-config-requested mode;partition_mode()is thepartition's actual mode (falls back to
topic_mode()when unset). Servingpredicates key on
partition_mode(); the legacyshadow_indexingfallback ispreserved. No-op (the two are always equal here).
cluster/partition: feedpartition_modeintontp_config— push theSTM's
partition_modeinto the log'sntp_configat start and on the STMchange callback (
storage::loggainsset_partition_mode, mirroringset_overrides).cluster/partition: bootstrappartition_modeon leadership — on becomingleader with the feature active, record the partition's current mode (keying on
the archival manifest, via the new
archival_metadata_stm::holds_archived_data(),when the partition is really tiered mid-migration). One-time, idempotent.
Testing
partition_properties_stm_test:partition_modedefault-unset; set / idempotent/ change with cross-node consistency; independence from
writes_disabledon theshared snapshot; raft-snapshot round-trip at the right offsets; recovery (local
kvstore snapshot + replay, and raft-snapshot recovery after dropping a node's
data); and the change callback fires on a real change but not an idempotent
no-op.
🤖 Generated with Claude Code
Design Doc
PR 0 · PR 1 · PR 2 · PR 3 · PR 4 · PR 5 · PR 6 · PR 7 · PR 8