new compression method, pbzip2

[nevesigor]

Please add when possible support for pbzip2, it's parallel bzip2 implementation, much faster.

[zipurman]

I had tried to use pbzip2 and other methods previously. The script uses dd with pv to monitor the data stream. When I did my testing using pbzip2 and others, I found them to be too slow when implemented this way. If you want to have a look at the code and let me know exactly how you'd suggest implementing pbzip2, I will do some testing and reassess.

[nevesigor]

I'm doing a lot of testing right now using attached storage for backup (not NFS as you suggest on your manual), I have changed some code (backup_imaging.php) regarding pv and dd, I don't know why yet, but my storage it's about to handle about 20 Gbit (nvme) but the backup/imaging using pv and dd it's taking ages to do the backup (about 200Mbit only)... this is very slow for production environments, once i'm done on speed, I will check for this.

[zipurman]

I have a large SAN and I have tried everything for speed, NFS was the fastest in my tests. I use iSCSI on the SAN for all VM storage. I did try a ramdisk on the BackupVM using 10GBe and that was crazy fast, but not practical.

I am also looking at this for a possible addon as well: https://ovirt.org/develop/release-management/features/storage/incremental-backup/

[nevesigor]

If I use fio tool to benchmark /mnt/backups I get 20Gbit, indeed pv+dd in the code can't go for more then 200Mbit, same machine, same disk!

[nevesigor]

You probably did not find pbzip2 useful because your backup storage does not go faster then one of your CPU, so you don't take any performance gain from doing parallel bzip2.

Meaning, one cpu doing ZIP was enough to fulfill your storage, if that's the case, pbzip2 was only doing overhead and getting the process worst/slow.

[zipurman]

In testing, pbzip2 seems to fail almost all of the time. I have done some recoding to get the other features into the project, but pbzip2 will fail on local storage in almost every test I run. The disk ends up partially corrupt and I have to fsck at best. Using LZO there is no issue at all and every test passes. I am pushing out the code with warnings on the pbzip2 and bzip2 settings, feel free to test and let me know what you find.