Decompression speed is slow

[district10]

Summary

I'm benchmarking a data pipeline, measuring ros msg -> protobuf msg (Decode), protobuf -> zstd compressed (Compress), zstd compressed -> protobuf (Decompress). I found backports.zstd==1.3.0 decompression is much slower (than zstd==1.5.7.2). You may need to investigate.

backports.zstd

if sys.version_info >= (3, 14):
    from compression import zstd
else:
    from backports import zstd

zstd

if sys.version_info >= (3, 14):
    from compression import zstd
else:
    import zstd

Platform

Linux 5.15.0-60-generic x86_64 GNU/Linux

Python version

Python 3.10.19

Library version

1.3.0

Installation method

No response

[Rogdham]

Hello and thank you for your interest!

---

I have performed some benchmarks in the past; below is a recap for comparing backports.zstd==0.5.0 to zstd==1.5.72 on CPython 3.10.18 (Linux):

Case	Ref time	zstd
compress 1k level 3	<1ms	🟢 + 6.28%
compress 1k level 10	<1ms	🟢 + 8.04%
compress 1k level 17	<1ms	⚪ + 4.31%
compress 1M level 3	6ms	🟩 +20.21%
compress 1M level 10	29ms	⚪ + 1.51%
compress 1M level 17	171ms	⚪ + 4.01%
compress 1G level 3	5.89s	⚪ + 4.11%
decompress 1k level 3	<1ms	🔴 - 6.58%
decompress 1k level 10	<1ms	🔴 - 9.50%
decompress 1k level 17	<1ms	🔴 - 6.97%
decompress 1M level 3	1ms	🟩 +24.89%
decompress 1M level 10	1ms	🟩 +23.82%
decompress 1M level 17	1ms	🟩 +23.97%
decompress 1G level 3	1.45s	🟢 + 5.68%
decompress 1G level 10	1.58s	🔴 - 7.47%
decompress 1G level 17	1.66s	🔴 - 6.28%

Legend:

• 🟩: backports.zstd is significantly faster
• 🟢: backports.zstd is sightly faster
• ⚪: both are similar
• 🔴: backports.zstd is sightly slower

The results are very different to what you showed in your own benchmarks.

---

Could you share how your benchmarks were performed?

I am interested in the code that runs as well as the data that is (de)compressed.

This would allow to reproduce and troubleshoot.

Thanks!

[district10]

Code:

from pathlib import Path
import time
from backports import zstd as zstd1
import zstd as zstd2
import zstandard as zstd3
import msgpack
import numpy as np
for lib, zstd in [("backports.zstd", zstd1), ("zstd", zstd2), ("zstandard", zstd3)]:
    print("-" * 80, lib)
    with Path("exports.msgpack").open("rb") as f:
        exports = msgpack.unpackb(f.read())
    for label, msgs in exports.items():
        level = int(label.split(":")[-2].split("=")[-1])
        dur_enc = []
        dur_dec = []
        bytes_raw = 0
        bytes_enc = 0
        for pb in msgs:
            t0 = time.perf_counter_ns()
            comp = zstd.compress(pb, level)
            t1 = time.perf_counter_ns()
            decomp = zstd.decompress(comp)
            t2 = time.perf_counter_ns()
            dur_enc.append(t1 - t0)
            dur_dec.append(t2 - t1)
            bytes_raw += len(pb)
            bytes_enc += len(comp)
        dur_enc = np.array(dur_enc) / 1e3
        dur_dec = np.array(dur_dec) / 1e3
        label = label[label.index("/data")+1:]
        print(f"{label}: enc={np.mean(dur_enc):.2f}us (std={np.std(dur_enc):.2f}), dec={np.mean(dur_dec):.2f}us (std={np.std(dur_dec):.2f}), ratio={bytes_enc / bytes_raw:.2%} = {bytes_enc:,}/{bytes_raw:,}")

Log:

-------------------------------------------------------------------------------- backports.zstd
topic=/data1:level=1:pb: enc=4.70us (std=2.04), dec=1.95us (std=0.60), ratio=80.57% = 3,764,552/4,672,402
topic=/data1:level=3:pb: enc=4.72us (std=1.88), dec=1.95us (std=0.32), ratio=80.57% = 3,764,552/4,672,402
topic=/data1:level=6:pb: enc=4.64us (std=1.69), dec=1.93us (std=2.69), ratio=80.57% = 3,764,552/4,672,402
topic=/data1:level=9:pb: enc=4.66us (std=1.74), dec=1.93us (std=0.44), ratio=80.57% = 3,764,552/4,672,402
topic=/data1:level=15:pb: enc=4.71us (std=2.42), dec=1.93us (std=0.44), ratio=80.57% = 3,764,552/4,672,402
-------------------------------------------------------------------------------- zstd
topic=/data1:level=1:pb: enc=4.34us (std=2.88), dec=0.50us (std=0.35), ratio=80.57% = 3,764,552/4,672,402
topic=/data1:level=3:pb: enc=4.33us (std=1.99), dec=0.50us (std=0.26), ratio=80.57% = 3,764,552/4,672,402
topic=/data1:level=6:pb: enc=4.46us (std=2.37), dec=0.52us (std=0.16), ratio=80.57% = 3,764,552/4,672,402
topic=/data1:level=9:pb: enc=4.42us (std=1.86), dec=0.51us (std=0.11), ratio=80.57% = 3,764,552/4,672,402
topic=/data1:level=15:pb: enc=4.38us (std=1.84), dec=0.51us (std=0.12), ratio=80.57% = 3,764,552/4,672,402

[district10]

exports.msgpack.zip

[district10]

Weird, it'm more like 'why sergey-dryabzhinsky/python-zstd so fast', not 'why 'Rogdham/backports.zstd so slow'...

-------------------------------------------------------------------------------- backports.zstd
data1:level=1:pb: enc=4.57us (std=1.69), dec=1.91us (std=0.93), ratio=80.78% = 3,774,270/4,672,402
data1:level=3:pb: enc=4.63us (std=1.65), dec=1.90us (std=0.70), ratio=80.57% = 3,764,552/4,672,402
data1:level=6:pb: enc=5.37us (std=19.15), dec=1.89us (std=0.32), ratio=80.47% = 3,759,689/4,672,402
data1:level=9:pb: enc=5.57us (std=1.99), dec=1.91us (std=0.26), ratio=80.47% = 3,759,679/4,672,402
data1:level=15:pb: enc=11.92us (std=5.78), dec=1.97us (std=1.14), ratio=80.44% = 3,758,668/4,672,402
-------------------------------------------------------------------------------- zstd
data1:level=1:pb: enc=4.21us (std=2.00), dec=0.50us (std=0.10), ratio=80.78% = 3,774,270/4,672,402
data1:level=3:pb: enc=4.36us (std=1.83), dec=0.51us (std=0.33), ratio=80.57% = 3,764,552/4,672,402
data1:level=6:pb: enc=4.84us (std=2.16), dec=0.52us (std=0.13), ratio=80.47% = 3,759,689/4,672,402
data1:level=9:pb: enc=5.29us (std=2.08), dec=0.51us (std=0.10), ratio=80.47% = 3,759,679/4,672,402
data1:level=15:pb: enc=9.50us (std=1.27), dec=0.54us (std=0.15), ratio=80.14% = 3,744,516/4,672,402
-------------------------------------------------------------------------------- zstandard
data1:level=1:pb: enc=4.39us (std=4.16), dec=1.58us (std=0.29), ratio=80.78% = 3,774,270/4,672,402
data1:level=3:pb: enc=4.49us (std=1.74), dec=1.57us (std=0.26), ratio=80.57% = 3,764,552/4,672,402
data1:level=6:pb: enc=5.01us (std=1.84), dec=1.58us (std=0.23), ratio=80.47% = 3,759,689/4,672,402
data1:level=9:pb: enc=5.39us (std=1.94), dec=1.59us (std=0.26), ratio=80.47% = 3,759,679/4,672,402
data1:level=15:pb: enc=11.71us (std=5.58), dec=1.62us (std=0.31), ratio=80.44% = 3,758,668/4,672,402

[emmatyping]

Each protocol buffer seems tiny, ~100 bytes. The vast majority of the time spent decompressing that small of data is going to go into allocating an output buffer. The allocation patterns that backports.zstd currently uses are not optimized for tiny compressions like that since most users compress much larger datasets. I'd recommend looking if you can batch these together to get better compresison and performance.