Maybe using libdeflate for decompression is faster?

[HatakeKakaxi]

When decompressing a file that has been compressed with gzip, kseqpp relies on zlib for decompression. However, there is a library called libdeflate which offers faster decompression speed compared to zlib. Perhaps using libdeflate could result in improved performance?

[rob-p]

Hi @HatakeKakaxi,

This is a nice suggestion. However, one downside of libdeflate is that it performs whole-buffer decompression. This is unlikely to be a good idea for FASTQ-sized data.

On the other hand, having done some testing of different zlib implementations, I can say that cloudlflare-zlib is a drop-in replacement for the standard zlib that is substantially faster, and also widely compatible and easy to build.

If you're willing to go through the extra pain of building it, the absolute fastest zlib library seems to be Intel's isa-l, which can be a few times faster than the standard system zlib.

I agree it would be nice to have a config option to use these if they are available on the user's system, or, for cloudflare at least, an option to pull down and compile the library if the user wants.

--Rob

[cartoonist]

Hey @HatakeKakaxi and @rob-p,

Thanks @rob-p for the insight into zlib alternatives! I didn't know the performance differences--really good to know.

As for libdeflate, I'd like to add that since it does not support streaming, it seems incompatible with the current design and use case. However, supporting different zlib implementations is a good idea. Adding a config option to support these implementations or libraries with zlib-compatible API seems straightforward. I'll take a look as soon as I find some free time.