ObjectStore’s maxParallel doesn’t actually increase the number of parallel uploads

[johanfleury]

Say you have 10 WAL files ready for archiving and maxParallel is set to 3.

• PG runs archive_command for WAL file 1:
  • barman-plugin looks at ready files and run the archiver for WAL 1, 2 and 3.
  • Those 3 files are uploaded to the archive.
  • Command exits successfully and PG marks WAL 1 as done.
• Then PG runs archive_command for WAL file 2:
  • barman-plugin looks are ready file and runs the archiver for WAL 2, 3 and 4
  • WAL file 2 and 3 are already archived.
  • WAL file 4 is uploaded to the archive.
  • Command exits successfully and PG marks WAL 2 as done.
• And so on, PG archives WAL file 3 and barman-plugin uploads WAL file 5.

This results in always uploading one single WAL file at a time which is pretty slow and is contrary to what the doc says: “Number of WAL files to be […] archived in parallel”.

What I would expect to happen is:

• PG runs archive_command for WAL file 1:
  • WAL file 1, 2 and 3 are uploaded
  • WAL file 1 is marked as done
• PG runs archive_command for WAL file 2:
  • WAL file 4, 5, 6 are uploaded
  • WAL file 2 is marked as done
• And so on until there are no more files to upload and the archive_command simply exits successfully without doing anything.

I think this happens because internalRun does not pass WALs that have already been archived to GatherReadyWALFiles:

plugin-barman-cloud/internal/cnpgi/common/wal.go

Line 206 in 376e178

SkipWALs: []string{baseWalName},

.

[dosubot]

Hi @johanfleury! I'm Dosu and I’m helping the plugin-barman-cloud team.

Your diagnosis is correct! This is a known bug documented in Issue #266.

The root cause is a protocol limitation in CNPG-i. When the plugin archives files 1, 2, and 3 in parallel, it returns an intentionally empty WALArchiveResult — there's no mechanism to report which files were actually archived. The operator then only marks file 1 as .done since it has no visibility into the opportunistically archived files 2 and 3.

This creates exactly the sliding window pattern you observed: all files reach S3 successfully, but each archive_command redundantly re-uploads files that were already archived.

Additional issue: Issue #516 reports that maxParallel > 1 also breaks WAL restore with "end of wal reached" errors.

Workaround: Set maxParallel: 1 until this is fixed. A proper fix would require extending the CNPG-i protocol to let the plugin report which files were archived, or giving the plugin direct access to mark files as .done.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[amerello]

Could someone confirm if the Workaround suggested by @dosu, namely to "Set maxParallel: 1 until this is fixed." is accurate and if so, if a fix is planned? I'm struggling with performance issues during backup/restore operations and was hoping maxParallel could help.