RFC ts-ct-migration: metastore object removal [PR 3]#30891
Draft
sjust-redpanda wants to merge 10 commits into
Draft
RFC ts-ct-migration: metastore object removal [PR 3]#30891sjust-redpanda wants to merge 10 commits into
sjust-redpanda wants to merge 10 commits into
Conversation
This was referenced Jun 23, 2026
sjust-redpanda
force-pushed
the
sjust/ts-ct-migration-pr3
branch
3 times, most recently
from
fc4f371 to
5bda2db
Compare
remote_segment_index.{h,cc} contained two independent things sharing a TU: the
read/write `offset_index` (used by readers) and `remote_segment_index_builder`, a
storage::batch_consumer that pulls in raft (raft::offset_translator_batch_types).
offset_index itself has none of that coupling -- only model, bytes, utils/delta_for
and serde.
Move `offset_index` into cloud_storage/offset_index.{h,cc} behind a new
//src/v/cloud_storage:offset_index cc_library.
remote_segment_index.{h,cc} keeps segment_record_stats + the builder, now
including offset_index.h; the cloud_storage monolith depends on :offset_index.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add an optional descriptor that lets an L1 object/extent point at a segment that physically lives in the tiered-storage bucket instead of an L1-native object. Metadata/encoding only; no read, write, or delete path consumes it yet. - imported_ts_info (object_id.h): the per-segment descriptor -- ts_path, delta_offset/delta_offset_end (log<->kafka offset translation), segment_term, and last_kafka_offset. Stored split by ownership: ts_path on the object row (imported_ts_object_location), delta/term on the extent row (imported_ts_segment_info); absent means a native L1 object. - Thread it through the metastore read/interface types on both backends: an optional imported_ts_info on object_response / extent_object_info (recomposed from the storage split on read), plus rpc_types and the replicated/domain passthrough. - Encode it: serde-versioned on both backends (object_entry/extent and the LSM rows), the LSM debug-serde path, and the ImportedTsObjectLocation / ImportedTsSegmentInfo protobufs. Old-version decode defaults to absent. Testing: serde round-trip unit tests for the new fields, including old-version decode. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a durable, per-partition boolean migrating to each backend's partition record, set true while a partition is mid tiered->cloud migration. - state: add the field to partition_state (serde v0->v1) and the LSM metadata_row_value (v1->v2), defaulting to false so snapshots predating the field decode as native cloud topics; carry it through the MetadataValue debug proto. - op: add set_migrating(partition, bool) across the metastore interface, the RPC service/client, the leader router (+probe), both domain managers, and both backends. It is idempotent (setting the current value is a no-op), applies in either direction, and creates the partition record when set on an absent partition; metadata-row rewrites (add_objects, set_start_offset) preserve the flag. - read: surface the flag on get_offsets. Tests (both backends): create-on-absent, set-then-clear, idempotent set, and preservation across add_objects; serde back-compat for both records. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Let the metastore register tiered-storage segments as imported L1 extents
by reference, reusing the existing add_objects extent-addition path rather
than a dedicated command.
- add_objects now handles imported objects:
- skips the pre-registration check for imported objects (they reference an
existing tiered-storage segment; nothing is written to L1)
- copies imported_ts_location / imported_ts_delta onto the object and
extent rows
- sets start_offset from the first extent for an empty migrating partition
(a non-zero start -- the TS->CT mount), rather than requiring offset 0
- the object builder's add_imported(ntp_metadata) registers one imported
segment as a finished, single-extent object: with no L1 write to reserve it
creates the object id and records it as finished without pre-registration,
to be committed by a subsequent add_objects. meta.imported must be set; the
read-replica metastore rejects it.
Tests (both backends): a migrating partition adopts a non-zero start at the
first imported extent; successive batches forward-append; a non-contiguous
batch is rejected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sjust-redpanda
force-pushed
the
sjust/ts-ct-migration-pr3
branch
2 times, most recently
from
e95afb1 to
85c80b1
Compare
A generic, header-only data source that fetches a byte range one fixed-size chunk at a time, lazily, and presents it as a single contiguous stream: only the chunks a reader actually consumes are fetched, so a read that stops early never pulls the tail. It is generic over the fetch callable and its error type, so it is not tied to tiered storage. A `use_chunk_aligned_reads` bool_class controls where the first chunk starts: when yes, chunk boundaries are chunk_size aligned so overlapping reads request identical (position, length) chunks that a downstream cache can dedup; when no, the first chunk starts exactly at the read's position (for a caller with no cache to dedup against). It has no default -- every caller states its choice. get() checks the abort source on every read, so a cancelled read fails loudly rather than ending as a truncated clean end-of-stream. Later commits build the reader and the open_object read path on top of this. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… abstraction Introduce open_object() and the object_handle/object_index abstraction for reading an L1 object: a handle exposes an index (seek by offset/timestamp to a byte position) and opens readers at a seek result. This commit wires the native L1 object path (footer index + record reader). Chunking lives here, above the cache, not inside read_object. read_object is a single-range primitive -- fetch exactly the requested bytes, via the cache or (skip_cache) a single cache-bypassing ranged GET. open_object wraps that per-range fetch in a chunk_data_source sized for the read: a cache-bypassing read is bounded by the streaming chunk size (request-relative), while a cached native read is left un-chunked because read_object caches the whole extent to a file, which already bounds memory. Keeping the chunking in open_object lets a later imported read chunk at the cache-chunk granularity for dedup, independently of how the cache-bypass path fragments a read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the building blocks for serving an imported tiered-storage segment, not yet
wired to any io backend (inert on their own):
- tiered_storage_object_reader (ts_reader.{h,cc}): an object_reader over a raw
Kafka-format segment stream. Maintains a running offset delta, emits only data
batches, drops aborted-transaction ranges (from the .tx manifest) so the
region is committed-only, and stamps the segment term as each batch's leader
epoch.
- ts_segment_index (ts_object.{h,cc}): object_index over a segment's
offset_index -- translates a target offset/timestamp into a byte position +
delta via find_kaf_offset/find_timestamp.
- ts_object_handle (ts_object.{h,cc}): object_handle whose byte transport is
injected as a fetch_range_fn, so file_io (download + cache) and fake_io
(in-memory) can share the seek/reader wiring. open_reader serves the segment
as a lazily-fetched sequence of fixed-size chunks through the
ts_chunk_data_source (only the chunks a read touches are downloaded);
chunk_size == 0 disables chunking and fetches the whole suffix in one range.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Read an imported (migrated) tiered-storage segment through the same open_object/object_handle path as a native L1 object. open_imported_object builds the segment's index from its .index sidecar (io::fetch_ts_index), its aborted-transaction set from the .tx sidecar (io::fetch_ts_tx, gated by the import-time tx_state), and returns a ts_object_handle whose byte transport reads ranges back through io::read_object routed to the segment's ts_path. An imported read is chunked at the cache-chunk granularity, chunk-aligned, so overlapping reads request identical (pos,size) chunks the cache can dedup -- distinct from the cache-bypassing streaming read, which is request-relative. The chunking is applied by open_object (chunked_fetch), so the handle and read_object stay format-agnostic. Tests (fake_io grows read_object-call recording so they can assert the chunking): imported reads with delta translation, index-backed seeks, aborted and control batch stripping, non-data/offset-translator batch handling, compacted segments with a hole at the front/middle/end, and the per-variant (cache/nocache x native/imported) chunk size and chunk alignment. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When an imported L1 object is removed, delete its backing tiered-storage objects (the segment, its .tx range manifest, and its .index -- the same set the archiver would have deleted) so nothing is orphaned. - delete_objects (abstract_io/file_io/fake_io) takes object_location instead of object_id, so it routes by the imported location: present -> delete from the tiered-storage bucket; absent -> delete from the L1 bucket as before. - Both garbage collectors (metastore and lsm) build the removal list from each object's imported_ts_location. Native L1 deletion is unchanged. Testing: an imported-segment GC test (import via add_objects on a migrating partition, then trim + GC) asserts the backing TS segment is removed, routed by ts_path; fake_io grows has_ts_segment to observe it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Factor the collector's delete-the-backing-objects-then-drop-the-metastore-rows step into a remove_objects() method on both the simple and LSM garbage collectors; remove_unreferenced delegates to it. Sync the STM before building the removal update. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sjust-redpanda
force-pushed
the
sjust/ts-ct-migration-pr3
branch
from
85c80b1 to
6304839
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🚧 NOT YET READY FOR REVIEW
Summary
A small, behavior-preserving refactor that factors object removal in the L1 metastore into a single
remove_objects()path. Part of the tiered→cloud migration stack.Stacked on PR1. Base is
dev, so the diff is cumulative (it includes PR1); the only commit new in this PR is cloud_topics/l1: factor object removal into remove_objects().What's in this PR
cloud_topics/l1: factor object removal intoremove_objects()— a single removal path that deletes the backing object (segment +.tx+.index, routed byts_pathfor an imported segment) and then the rows. The earlier{detach, gc}removal mode is deleted — removal always deletes the backing objects and their rows.Testing
//src/v/cloud_topics/level_one/metastore/...tests green (19/19).🤖 Generated with Claude Code
Design Doc
PR 0 · PR 1 · PR 2 · PR 3 · PR 4 · PR 5 · PR 6 · PR 7 · PR 8