Skip to content

Fix schema restore failure with partitioned tables and add --data-only flag#775

Open
blakewatters wants to merge 1 commit intoxataio:mainfrom
blakewatters:baselayer
Open

Fix schema restore failure with partitioned tables and add --data-only flag#775
blakewatters wants to merge 1 commit intoxataio:mainfrom
blakewatters:baselayer

Conversation

@blakewatters
Copy link

Fix schema restore failure with partitioned tables and add --data-only flag

Summary

  • Fix --init / --snapshot-tables failing on databases with partitioned tables (PostgreSQL 17/18)
  • Add --data-only flag to allow data-only snapshots when schema is pre-populated on the target

Bug: Schema restore treats partition errors as fatal

When restoring a schema from a source database that uses declarative partitioning, pgstream's pg_dump-based schema restore fails with errors like:

ERROR:  "linking_queue_000" is already a partition
ERROR:  "sos_officers_p000" is already a partition

Root cause: PostgreSQL's pg_dump -Fp output for partitioned tables includes both:

  1. CREATE TABLE child PARTITION OF parent ... — which attaches the child to the parent
  2. ALTER TABLE parent ATTACH PARTITION child ... — which fails because the child is already attached from step 1

The error message "already a partition" is not matched by the existing ignorable error pattern, which only checks for "already exists". These errors fall through to the default case in parseErrorLine() and are classified as critical, causing the entire restore to fail.

This is the same class of harmless, idempotent error as "already exists" — the partition is correctly attached, the redundant ATTACH PARTITION just fails because it's already done.

Fix: Add strings.Contains(line, "already a partition") to the ignorable error patterns in parseErrorLine(), classified as ErrRelationAlreadyExists.

Feature: --data-only flag for --snapshot-tables

Currently, --snapshot-tables always triggers a full schema + data snapshot. There is no way to perform a data-only initial snapshot when the schema already exists on the target.

This is needed for workflows where the schema is restored separately (e.g., via pg_dump --schema-only | psql) to work around the partition bug above, or when the target schema is managed independently.

Change: Add --data-only boolean flag to pgstream run. When combined with --snapshot-tables, it sets source.postgres.snapshot.mode to "data" instead of "full", skipping the schema restore phase entirely.

# Schema already exists on target — only snapshot data, then start CDC
pgstream run \
  --source postgres --source-url "postgres://..." \
  --target postgres --target-url "postgres://..." \
  --snapshot-tables "public.*,private.*" \
  --data-only

Without --data-only, behavior is unchanged.

Files Changed

File Change
internal/postgres/pg_restore.go Add "already a partition" to ignorable error patterns
internal/postgres/pg_restore_test.go Add test cases for partition error classification
cmd/root_cmd.go Register --data-only flag on run command
cmd/run_cmd.go Wire --data-only to set snapshot mode to "data"

How We Found This

We're using pgstream to replicate an 838 GB Cloud SQL database (with ~2,100 tables including heavily partitioned tables with 200+ partitions each) to a local PostgreSQL 18 instance on a GCP VM. Both --init and --snapshot-tables failed on the schema restore phase due to the partition error. The data restore and CDC replication work correctly — only the schema restore is affected.

Testing

All existing tests pass. New test cases added:

  • TestParsePgRestoreOutputErrs/partition already attached error — verifies "already a partition" is classified as ignorable
  • TestParseErrorLine/already a partition — verifies parseErrorLine returns ErrRelationAlreadyExists for partition errors

@blakewatters blakewatters changed the title Better handle partitions Fix schema restore failure with partitioned tables and add --data-only flag Mar 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant