RFC ts-ct-migration: metastore and L1 reader [PR 1] by sjust-redpanda · Pull Request #30800 · redpanda-data/redpanda

sjust-redpanda · 2026-06-12T23:23:35Z

Backports Required

none - not a bug fix

Release Notes

TBD

Features

Add the L1 metastore and reader foundation for an imported segment: one that physically lives in the tiered-storage bucket but is described and addressed by reference from L1. First PR of the tiered→cloud migration stack.

Changes by area

cloud_storage prep — split offset_index out of remote_segment_index into its own //src/v/cloud_storage:offset_index library, so the L1 ts_segment_index can reuse the read/write offset index without pulling in remote_segment_index_builder (which drags in raft). Behavior-preserving.
Schema / encoding — teach the metastore to describe a segment that lives in the tiered-storage bucket, referenced from L1 rather than written natively. The encoding changes, all serde-versioned (old snapshots decode with no imported info and migrating = false):
- imported_ts_info (object_id.h) -- optional metadata for object_extent required to read ts extents, absent on L1 object_extent
  - ts_path
  - log↔Kafka delta_offset / delta_offset_end
  - segment_term
  - last_kafka_offset. Carried as an optional; absent = a native L1 object.
- object row: object_entry gains imported_ts_object_location — the backing segment's ts_path.
- extent row: gains imported_ts_segment_info (imported_ts_delta) — the per-data delta/term; the extent's own offsets carry the Kafka bounds.
- partition record: partition_state (v0→v1) and LSM metadata_row_value (v1→v2) gain the boolean migrating flag. Used by recovery/read-replica to determine whether to treat the partition as TS or CT
- proto debug representation: ImportedTsObjectLocation (ObjectValue.imported_ts_location), ImportedTsSegmentInfo (ExtentValue.imported_ts_delta), and MetadataValue.migrating.
Write path
- add_objects also now supports imported segments via object_metadata_builder with appropriate additions to new_object
- fresh partition marked migrating (via the idempotent set_migrating op, surfaced on get_offsets) adopts its log at the first imported extent's base rather than at 0.
Read path —
- Factors L1 object format handling and indexing into an object_handle abstraction via io::open_object, with an implementation for the existing native L1 behavior.
- Adds TS support through a ts_object_handle/ts_segment_index for the TS index + transaction files. The handle serves a segment as lazily-fetched fixed-size chunks (ts_chunk_data_source) -- only the chunks a read touches are downloaded (chunk_size == 0 falls back to a whole-suffix fetch).
- The native L1 read and the imported-TS fetch share one cache/download helper, which runs under the per-shard single_flight (concurrent cache-misses for the same extent de-dup into one download) and reports byte-read counters via the file_io_probe (footer / ts-index / ts-tx bytes, under the cloud_topics_level_one_reader metric group).
Deletion — route object GC by the imported info: an imported extent's backing tiered-storage objects (segment + .tx + .index) are deleted from the tiered-storage bucket; native objects route to the L1 bucket as before.

Testing

Unit tests, green standalone (bazel test //src/v/cloud_topics/level_one/...): serde round-trip / old-version decode, state_update op semantics on both backends, the imported reader (offset translation, txn strip, epoch stamping, download errors), and imported-segment GC.

Design Doc

PR 0 · PR 1 · PR 2 · PR 3 · PR 4 · PR 5 · PR 6 · PR 7 · PR 8

sjust-redpanda · 2026-06-15T15:24:38Z

/ci-repeat 1

sjust-redpanda · 2026-06-15T15:37:17Z

/ci-repeat 1

sjust-redpanda · 2026-06-17T01:36:54Z

I think this is ready for review/feedback, but it probably shouldn't be merged until the remaining pieces are at least available in draft form and have been looked at.

Lazin · 2026-06-24T12:50:29Z

+    }
+
+    /// Opaque tiered-storage segment object path (sname_format path).
+    ts_segment_path ts_path;


is it possible to reuse an object_id field in the extent for that?

Lazin · 2026-06-24T15:38:54Z

+              // Downloads the whole segment suffix [pos, segment_end) up front,
+              // even when the fetch stops early (max_offset/max_bytes).
+              // Bounding it to the fetch window was deferred: it is a perf
+              // refinement (the native L1 path downloads seek..end the same


That could be problematic because the segment size may be large and it could cause a cache churn.
Also, if the cached range name is based on pos and len then the reader may not be able to find the right object. E.g. one reader starts from pos=100, another starts from pos=200. Would the second reader be able to use cached data?

In TSv1 we have chunked reads. The segment index is downloaded first. Then the smaller index is created. This index has 16MiB granularity. Then the read path is fetching these 16MiB chunks. In dedicated at least we configure the segment size to be pretty small IIRC. But it's not guaranteed that the segments are small.

Added a chunked read mechanism also internal to ts_object.

Lazin · 2026-06-26T14:39:29Z

    ],
 )

+redpanda_cc_library(


The code is removed, not moved

I'm not sure what you mean, offset_index.h/cc seem to have what was removed from remote_segment_index.h/cc.

For me the diff appears as if it only removes the code

maybe middleman renders the diff incorrectly

Maybe, I see 6e0f837 adding offset_index.h/cc, and I seem them still present in the final unified diff.

Oh, gitk does an interesting thing. It actually shows offset_index.h/cc as a copy of remote_segment_index.h/cc with lines removed. That might be what you're seeing.

Lazin · 2026-07-02T13:30:51Z

    }
 }

+ss::future<std::expected<std::unique_ptr<object_handle>, io::errc>>


maybe it makes sense to split the PR. everything prior to this commit should be OK to merge I think

Might make sense later, no urgency to merge the metastore structure changes yet though.

Lazin · 2026-07-02T16:52:43Z

+    // Open an object for reading: returns a handle that exposes the object's
+    // index (seek by offset/timestamp) and opens readers at a seek point.
+    virtual ss::future<std::expected<std::unique_ptr<object_handle>, errc>>
+    open_object(object_extent, ss::abort_source*, cloud_io::group_id g) = 0;


I'm a still a bit torn by this approach. This makes it impossible to decouple index downloads from payload downloads. Or maybe it is more correct to say that it's more difficult to decouple them in this case. The open_object method is always waiting for the footer/index download and the payload download can only happen afterwards. This is what we're doing in TSv1 but we're prefetching the footers in TSv2 (or planning to do this).

I can certainly defer the metadata reads until we know what the seek is going to be. I'll look into that tomorrow.

Lazin · 2026-07-02T16:52:44Z

+          cloud_storage::remote_segment_sampling_step_bytes,
+          model::timestamp::missing());
+        auto index_iobuf = co_await download_raw_iobuf(
+          _remote, _bucket, index_path.native(), as);


I think this step is only needed if the scan starts in the middle of the segment. You can expect that the Kafka client fetches long runs of batches. So for the very first segment it will need to download the index to start from the right place. But every subsequent segment will be scanned from the begining.
In TSv1 we download every object's index because the chunks are always aligned by the batch boundary. The read path has to fetch the index to be able to chunk-download the segment even if it scans the segment from the beginning to the end.
But here it seems that the chunks are not necessary aligned by the batch boundary so there is no need to pul the index for anything except the very first segment that the reader touches.

remote_segment_index.{h,cc} contained two independent things sharing a TU: the read/write `offset_index` (used by readers) and `remote_segment_index_builder`, a storage::batch_consumer that pulls in raft (raft::offset_translator_batch_types). offset_index itself has none of that coupling -- only model, bytes, utils/delta_for and serde. Move `offset_index` into cloud_storage/offset_index.{h,cc} behind a new //src/v/cloud_storage:offset_index cc_library. remote_segment_index.{h,cc} keeps segment_record_stats + the builder, now including offset_index.h; the cloud_storage monolith depends on :offset_index. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add an optional descriptor that lets an L1 object/extent point at a segment that physically lives in the tiered-storage bucket instead of an L1-native object. Metadata/encoding only; no read, write, or delete path consumes it yet. - imported_ts_info (object_id.h): the per-segment descriptor -- ts_path, delta_offset/delta_offset_end (log<->kafka offset translation), segment_term, and last_kafka_offset. Stored split by ownership: ts_path on the object row (imported_ts_object_location), delta/term on the extent row (imported_ts_segment_info); absent means a native L1 object. - Thread it through the metastore read/interface types on both backends: an optional imported_ts_info on object_response / extent_object_info (recomposed from the storage split on read), plus rpc_types and the replicated/domain passthrough. - Encode it: serde-versioned on both backends (object_entry/extent and the LSM rows), the LSM debug-serde path, and the ImportedTsObjectLocation / ImportedTsSegmentInfo protobufs. Old-version decode defaults to absent. Testing: serde round-trip unit tests for the new fields, including old-version decode. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add a durable, per-partition boolean migrating to each backend's partition record, set true while a partition is mid tiered->cloud migration. - state: add the field to partition_state (serde v0->v1) and the LSM metadata_row_value (v1->v2), defaulting to false so snapshots predating the field decode as native cloud topics; carry it through the MetadataValue debug proto. - op: add set_migrating(partition, bool) across the metastore interface, the RPC service/client, the leader router (+probe), both domain managers, and both backends. It is idempotent (setting the current value is a no-op), applies in either direction, and creates the partition record when set on an absent partition; metadata-row rewrites (add_objects, set_start_offset) preserve the flag. - read: surface the flag on get_offsets. Tests (both backends): create-on-absent, set-then-clear, idempotent set, and preservation across add_objects; serde back-compat for both records. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Let the metastore register tiered-storage segments as imported L1 extents by reference, reusing the existing add_objects extent-addition path rather than a dedicated command. - add_objects now handles imported objects: - skips the pre-registration check for imported objects (they reference an existing tiered-storage segment; nothing is written to L1) - copies imported_ts_location / imported_ts_delta onto the object and extent rows - sets start_offset from the first extent for an empty migrating partition (a non-zero start -- the TS->CT mount), rather than requiring offset 0 - the object builder's add_imported(ntp_metadata) registers one imported segment as a finished, single-extent object: with no L1 write to reserve it creates the object id and records it as finished without pre-registration, to be committed by a subsequent add_objects. meta.imported must be set; the read-replica metastore rejects it. Tests (both backends): a migrating partition adopts a non-zero start at the first imported extent; successive batches forward-append; a non-contiguous batch is rejected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

… abstraction Introduce an object_handle/object_index indirection for the L1 read path so a reader can seek and stream from an object without knowing its on-disk format. This is the seam a later commit uses to serve imported tiered-storage segments; here it is wired for native L1 only and is behavior-preserving. - object_handle.{h,cc}: object_index (seek_to_offset/seek_to_timestamp -> seek_result) and object_handle (index() + open_reader) interfaces, plus the native implementations l1_footer_index and l1_native_object_handle. The handle reads byte ranges via a fetch callback over io::read_object, so file_io and fake_io share one implementation. - open_object.{h,cc}: a free open_object() factory that reads the object footer via io::fetch_native_footer and returns an l1_native_object_handle (footer-indexed) whose fetch callback reads byte ranges back through io::read_object. - io::read_object_as_iobuf, whose sole use is reading the footer, is renamed to fetch_native_footer; file_io overrides it to record footer bytes on its probe (the footer_read_bytes metric). - level_one_reader obtains an object_handle via open_object and reads through its index/open_reader instead of reading the footer directly; group_id is threaded through. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add chunk_data_source: an ss::data_source_impl that fetches a byte range one fixed-size chunk at a time, lazily, behind a single contiguous input_stream. Only the chunks a read actually consumes are downloaded -- when the reader stops at its limit it stops pulling, and the tail is never fetched. Chunks are fixed byte ranges (not aligned to any structure in the data); the source concatenates one chunk's tail with the next chunk's head, so a record spanning a boundary is read seamlessly, and EOF is signalled only at the true end (never mistaken for a chunk boundary). It is a templated header-only library, generic over the fetch callable and its error type, so it depends on nothing L1-specific; make_chunk_data_source deduces the fetch type and wraps it in an ss::data_source. No caller yet uses it; the imported tiered-storage read path adopts it in the following commit. Tested: chunk_data_source_test covers the source directly (concatenation byte-identical to a whole download, mid-chunk head skip, early-termination fetches only touched chunks, single/exact-multiple sizes, EOF at total length, fetch-error propagation). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add the building blocks for serving an imported tiered-storage segment, not yet wired to any io backend (inert on their own): - tiered_storage_object_reader (ts_reader.{h,cc}): an object_reader over a raw Kafka-format segment stream. Maintains a running offset delta, emits only data batches, drops aborted-transaction ranges (from the .tx manifest) so the region is committed-only, and stamps the segment term as each batch's leader epoch. - ts_segment_index (ts_object.{h,cc}): object_index over a segment's offset_index -- translates a target offset/timestamp into a byte position + delta via find_kaf_offset/find_timestamp. - ts_object_handle (ts_object.{h,cc}): object_handle whose byte transport is injected as a fetch_range_fn, so file_io (download + cache) and fake_io (in-memory) can share the seek/reader wiring. open_reader serves the segment as a lazily-fetched sequence of fixed-size chunks through the ts_chunk_data_source (only the chunks a read touches are downloaded); chunk_size == 0 disables chunking and fetches the whole suffix in one range. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Extend the open_object factory to dispatch an imported extent (one carrying imported_ts_info) to the tiered-storage read path, so a single L1 reader serves native L1 objects and imported segments interchangeably. The segment's index (.index) and aborted-transaction ranges (.tx manifest) are fetched through two new io methods, fetch_ts_index and fetch_ts_tx. They return raw index bytes and parsed tx ranges rather than cloud_storage domain objects, so the widely-depended-on abstract_io interface stays free of a cloud_storage dependency. open_object assembles them into a ts_segment_index and an aborted set, gates the .tx fetch on the import-time tx_state, and returns a ts_object_handle whose byte transport reads chunks back through io::read_object (routed to the segment's ts_path, with cache/skip handling). - file_io: read_object routes an imported extent to its ts_path; fetch_ts_index / fetch_ts_tx download the .index and .tx sidecars (via a download_raw_iobuf helper) and record their byte counts on the probe. - fake_io: an injected segment (put_ts_segment, optionally with a serialized offset_index) is served through the same path with an in-memory fetch; without an index the ts_segment_index is empty (a full-segment scan), matching file_io's missing-.index fallback. The imported chunk size is taken from config (cloud_storage_cache_chunk_size / cloud_storage_disable_chunk_reads); tests set it small via scoped_config to exercise multi-chunk reads. Testing: imported open_object/reader unit tests (offset translation, the index-backed seek, read-committed strip, leader-epoch stamping, control-batch drop) against fake_io, plus the imported reader fixture. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

When an imported L1 object is removed, delete its backing tiered-storage objects (the segment, its .tx range manifest, and its .index -- the same set the archiver would have deleted) so nothing is orphaned. - delete_objects (abstract_io/file_io/fake_io) takes object_location instead of object_id, so it routes by the imported location: present -> delete from the tiered-storage bucket; absent -> delete from the L1 bucket as before. - Both garbage collectors (metastore and lsm) build the removal list from each object's imported_ts_location. Native L1 deletion is unchanged. Testing: an imported-segment GC test (import via add_objects on a migrating partition, then trim + GC) asserts the backing TS segment is removed, routed by ts_path; fake_io grows has_ts_segment to observe it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

sjust-redpanda · 2026-07-05T20:30:55Z

Change object_handle to layer on top of abstract_io:

Avoids awkward use of io::read_object
Avoids duplicated logic between file_io and fake_io
Significantly reduces changes to file_io helpers

Move ts_chunk_data_source.h/cc to chunk_data_source.h. Didn't need to be ts specific.

Next step is avoiding blocking read of L1 index when not necessary.

sjust-redpanda self-assigned this Jun 12, 2026

github-actions Bot added area/build area/redpanda labels Jun 12, 2026

sjust-redpanda force-pushed the sjust/ts-ct-migration-pr1 branch 4 times, most recently from 609249b to e79962a Compare June 13, 2026 04:15

sjust-redpanda force-pushed the sjust/ts-ct-migration-pr1 branch 2 times, most recently from a7f0aa2 to 41625b5 Compare June 15, 2026 21:24

sjust-redpanda commented Jun 15, 2026

View reviewed changes

Comment thread src/v/cloud_topics/level_one/metastore/lsm/state_update.cc Outdated

Comment thread src/v/cloud_topics/level_one/metastore/lsm/state_update.cc Outdated

Comment thread src/v/cloud_topics/level_one/metastore/lsm/state_update.cc Outdated

sjust-redpanda force-pushed the sjust/ts-ct-migration-pr1 branch 9 times, most recently from 8cf7d87 to afc1be9 Compare June 16, 2026 21:33

sjust-redpanda marked this pull request as ready for review June 16, 2026 23:54

sjust-redpanda requested review from WillemKauf and andrwng and removed request for andrwng June 16, 2026 23:54

sjust-redpanda force-pushed the sjust/ts-ct-migration-pr1 branch from afc1be9 to 6491b9d Compare June 17, 2026 00:27

sjust-redpanda changed the title ~~ts-ct-migration: metastore and L1 reader [PR 1]~~ RFC ts-ct-migration: metastore and L1 reader [PR 1] Jun 17, 2026

sjust-redpanda marked this pull request as draft June 17, 2026 02:47

Lazin reviewed Jun 24, 2026

View reviewed changes

sjust-redpanda force-pushed the sjust/ts-ct-migration-pr1 branch from 302623a to 0293bdc Compare June 25, 2026 18:33

Lazin reviewed Jun 26, 2026

View reviewed changes

sjust-redpanda commented Jun 26, 2026

View reviewed changes

Comment thread src/v/cloud_storage/offset_index.cc Outdated

sjust-redpanda force-pushed the sjust/ts-ct-migration-pr1 branch 2 times, most recently from 23875db to d9cb941 Compare June 29, 2026 17:49

sjust-redpanda mentioned this pull request Jun 30, 2026

RFC ts-ct-migration: partition_mode substrate [PR 0] #30977

Draft

Lazin reviewed Jul 2, 2026

View reviewed changes

sjust-redpanda and others added 4 commits July 2, 2026 09:58

sjust-redpanda force-pushed the sjust/ts-ct-migration-pr1 branch from d9cb941 to 5fd534d Compare July 2, 2026 20:57

sjust-redpanda commented Jul 2, 2026

View reviewed changes

Comment thread src/v/cloud_topics/level_one/common/object_handle.cc Outdated

sjust-redpanda commented Jul 2, 2026

View reviewed changes

Comment thread src/v/cloud_topics/level_one/common/fake_io.cc Outdated

sjust-redpanda and others added 5 commits July 2, 2026 18:33

sjust-redpanda force-pushed the sjust/ts-ct-migration-pr1 branch from 5fd534d to 5b8a9f6 Compare July 5, 2026 20:26

Uh oh!

Conversation

sjust-redpanda commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Backports Required

Release Notes

Features

Changes by area

Testing

Uh oh!

sjust-redpanda commented Jun 15, 2026

Uh oh!

sjust-redpanda commented Jun 15, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sjust-redpanda commented Jun 17, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sjust-redpanda commented Jul 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sjust-redpanda commented Jun 12, 2026 •

edited

Loading

sjust-redpanda commented Jul 5, 2026 •

edited

Loading