Former primary stuck at 1/2 after failover due to barman-cloud-check-wal-archive "Expected empty archive"

[MonicaMagoniGW]

Description

After a failover/switchover on a running cluster (bootstrapped with initdb, not a recovery/restore), the plugin-barman-cloud sidecar on replica pods keeps failing with "WAL archive check failed: Expected empty archive". This prevents the replica from becoming fully ready (1/2 containers).

The root cause is that barman-cloud-check-wal-archive is executed on every WAL Archive gRPC call, including on replica pods, and it fails because the S3 bucket already contains WAL files from previous timelines (which is expected after a switchover).

Environment

• CloudNativePG operator: v1.28.1
• plugin-barman-cloud: v0.11.0
• PostgreSQL: 18.3 (ghcr.io/cloudnative-pg/postgresql:18.3)
• Kubernetes: v1.31
• Object storage: Ceph RGW (S3-compatible, via Rook)

Cluster configuration

The cluster is bootstrapped with initdb (no recovery, no externalClusters):

spec:
  instances: 3
  bootstrap:
    initdb:
      database: system
      encoding: UTF8
      owner: app
  plugins:
    - name: barman-cloud.cloudnative-pg.io
      enabled: true
      isWALArchiver: true
      parameters:
        barmanObjectName: my-object-store

Steps to reproduce

1. Create a 3-instance CNPG cluster with initdb bootstrap and plugin-barman-cloud with isWALArchiver: true
2. Wait for all 3 pods to be 2/2 Running
3. A failover or switchover occurs (automatic or manual), changing the timeline (e.g., timeline 1 → 2 → 3)
4. After the failover, one or more replica pods get stuck at 1/2 — the plugin-barman-cloud sidecar blocks WAL archiving

What happens

After the switchover, the former primary is demoted to replica. The instance manager detects leftover WAL files and explicitly triggers archiving on the demoted pod:

{"msg":"Detected ready WAL files in a former primary, triggering WAL archiving"}

This causes the plugin-barman-cloud sidecar to attempt WAL archiving. However, the S3 bucket legitimately contains WAL files from previous timelines (archived by this same pod when it was primary). The plugin executes barman-cloud-check-wal-archive, finds the bucket is not empty, and fails:

barman-cloud-check-wal-archive: ERROR: WAL archive check failed for server <cluster-name>: Expected empty archive

plugin-barman-cloud sidecar log (replica pod):

{"level":"info","msg":"barman-cloud-check-wal-archive checking the first wal"}
{"level":"info","logger":"barman-cloud-check-wal-archive","msg":"ERROR: WAL archive check failed for server <name>: Expected empty archive","pipe":"stderr"}
{"level":"error","msg":"Error invoking barman-cloud-check-wal-archive",
  "options":["--endpoint-url","http://<ceph-rgw>","--cloud-provider","aws-s3","s3://<bucket>","<server-name>"],
  "exitCode":-1,"error":"exit status 1"}

Expected behavior

All the cluster pods are correctly running after a switchover/failover.

[dosubot]

Hi @MonicaMagoniGW! I'm Dosu and I’m helping the plugin-barman-cloud team.

This looks like a genuine gap in the plugin's logic. After looking at the code, the barman-cloud-check-wal-archive check is triggered during WALServiceImplementation.Archive() via CheckBackupDestination() [1] [2], and it runs on any pod that attempts WAL archiving — there's no logic to skip it on replicas or when the archive already contains WAL files from a previous timeline.

The only replica-specific skip in the codebase is for retention policy enforcement [3], but nothing similar exists for the empty archive check. The check is gated on two conditions — the IsEmptyWalArchiveCheckEnabled annotation being set and the .check-empty-wal-archive marker file existing in $PGDATA [4] — but neither condition accounts for a former primary that legitimately archived WAL files before being demoted.

This is related to a broader set of timeline-handling challenges after failover. A cross-timeline WAL segment issue after failover/switchover has been discussed as an open architectural concern [5], and several timeline-related replica fixes have been applied over time [6] [7], but the specific scenario of the empty archive check failing on a demoted primary doesn't appear to have a fix yet.

As a workaround, if your cluster has the cnpg.io/skipEmptyWalArchiveCheck annotation available, disabling the check could unblock the pods — though this trades off the safety mechanism. Alternatively, using a distinct serverName per bootstrap/lifecycle can isolate the archive path and avoid the collision [8].

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}
   

[MonicaMagoniGW]

Thanks @dosu for the analysis, that confirms our findings.

A few notes:

The cnpg.io/skipEmptyWalArchiveCheck annotation works as a workaround, but it disables a safety check globally — not ideal for production clusters.

Using a distinct serverName doesn't help here — this is not a restore/recovery scenario. The cluster is created once with initdb, runs normally, and the issue appears after a regular failover. The serverName stays the same throughout the cluster's lifecycle, as expected.

Is there a plan to address this?

[dosubot]

Good points — you're right that the serverName workaround doesn't apply here since this isn't a restore scenario.

To clarify the code path: when using a CNPG-I plugin like plugin-barman-cloud, the ArchiveAllReadyWALs function on the demoted primary detects leftover WAL files [1] and calls internalRun, which routes through gRPC to the plugin sidecar [2]. The plugin sidecar then checks for the .check-empty-wal-archive marker file in $PGDATA [3], and if it exists, runs barman-cloud-check-wal-archive via CheckBackupDestination() [4]. On the demoted primary, the archive is obviously non-empty, so the check fails.

The core issue is that the .check-empty-wal-archive marker file was created during the initial initdb bootstrap [5] and is supposed to be removed once continuous archiving starts successfully [6]. If it wasn't removed before the failover — or if the removal logic has a gap — the marker persists on the demoted primary, triggering the check against an archive that legitimately contains WAL files.

I don't see an open issue or active PR specifically tracking this scenario. The related fixes (PR #458, PR #9306, PR #685) addressed the check for restore/volume-snapshot scenarios but don't appear to cover the demoted-primary-after-initdb case you're describing. This looks like it could be a new bug worth tracking explicitly — particularly around why the marker file persists (or gets re-evaluated) on a demoted primary that had previously been archiving successfully.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}
   

[JamesatStarling]

@dosu I've also hit this same issue when patching container vulnerabilities or performing a minor version upgrade. The old primary is unable to be reintroduced to the cluster and we see the following error:
barman-cloud-check-wal-archive: ERROR: WAL archive check failed for server <cluster-name>: Expected empty archive

We can use cnpg.io/skipEmptyWalArchiveCheck for demo but not for production.