Support offline checkpoint catalog inspection, CSV dump, and restore script generation#25324
Support offline checkpoint catalog inspection, CSV dump, and restore script generation#25324jiangxinmeng1 wants to merge 79 commits into
Conversation
- EarliestTS/LatestTS: skip entries with empty start/end timestamps - EarliestTS also considers end timestamps, not just start - LatestTS: skip empty end values - Add parallel CSV pipeline with memory backpressure for storage mode - Fix block read memory leak: release now cleans batch vectors Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… DDL names - ListCatalogTables: merge catalog entries from raw and schema-projected mo_tables views - add logicalViewDataOffset for header detection without hardcoding meta width - normalizeCreateTableDDLName: replace catalog DDL table name with real name - add SQL token parser for CREATE TABLE name normalization (CREATE TABLE IF NOT EXISTS) - add checkpoint_test.go Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Remove the single-table dump by name (--table flag) functionality from ckp dump. The resolveTableByName function and all related validation logic are removed. Also change the default --jobs value from 1 to 5 for batch dump concurrency. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
XuPeng-SH
left a comment
There was a problem hiding this comment.
Requesting changes. The feature direction is useful, but I found several correctness gaps in the new dump/restore and multipart paths that should be closed before merge.
-
cmd/mo-object-tool/ckp/checkpoint.go:553drops the outputClose()error for single-table dumps. For local files this can lose a flush error; for--out-s3/--out-backendit is much more serious becausefileServiceWriteCloser.Close()is where the backgroundfs.Write/ multipart completion result is returned. TodayDumpTableCSVComposedcan succeed,CompleteMultipartUploadcan fail, and the command still printsTable ... dumped. Please close explicitly after the dump and combine dump/close errors the same waydumpOneTablealready does. -
pkg/tools/checkpointtool/table_dump.go:1185silently ignores everyGetObjectEntrieserror. That turns checkpoint corruption, permission errors, remote read failures, or decode errors into either a partial export or a misleadingno data entries for tableerror. This should only skip the known GC/missing-file case (isDataFileNotFound); all other errors need to be returned/aggregated. Please add an unhappy-path regression test for a non-missing checkpoint entry error. -
pkg/tools/checkpointtool/checkpoint_reader.go:424does not actually pick the latest usable GCKP.loadEntriessorts entries newest-first, butComposeAtiterates fromlen(entries)-1down and breaks on the firstend <= ts, which is the oldest qualifying GCKP. That can compose a snapshot from an unnecessarily old base plus too many incrementals, and can produce wrong catalog/table state when multiple global checkpoints exist. Iterate newest-to-oldest, skipping only GC'd bases, and cover this with a two-GCKP test. -
cmd/mo-object-tool/README.md:100documents--row-order=storageas object/block/row scan order, butpkg/tools/checkpointtool/table_dump.go:2065processes objects concurrently andwriteCSVChunkswrites chunks in completion order. The default output is therefore nondeterministic and no longer matches the documented storage-order contract. Either preserve order with a small reorder buffer keyed by(objectIdx, blockIdx)or change the contract/flag name and tests to make unordered streaming explicit. -
pkg/fileservice/aws_sdk_v2.go:713andpkg/fileservice/qcloud_sdk.go:566can resend the same pending buffer aftersendJobfails. On context cancellation, max-part overflow, or worker-triggered cancellation,sendJobalready records the error and may return the buffer to the pool, but the loop breaks withpendingN > 0, so lines 724/577 callsendJobagain for the same buffer. This can double-return buffers, advancepartNumincorrectly, or continue upload state after cancellation. Please clearpendingN/return immediately onsendJobfailure and add tests that cancel/fail after at least one full pending chunk, not only before upload starts.
Local checks run:
make err-checkgo test -count=1 ./pkg/tools/checkpointtool ./cmd/mo-object-tool/ckp ./pkg/sql/colexec/externalgo test -count=1 ./pkg/tools/toolfs ./pkg/fileservicewith macOS CGo rpath adjusted for local dylibs
What type of PR is this?
Which issue(s) this PR fixes:
issue #25323
What this PR does / why we need it:
Support offline checkpoint catalog inspection, CSV dump, and restore script generation