feat: multi-threaded parallel compression

[polaz]

Summary

The C reference supports job-based parallel compression via ZSTDMT — splitting input into jobs processed by worker threads. This is essential for large file compression and saturating modern multi-core CPUs.

C reference implementation (zstdmt_compress.c)

Architecture

• Job-based: Input split into chunks (min 512KB per job)
• Worker thread pool: Generic pool from pool.c/h
• Overlap: Sequential jobs share context via overlap region
• Buffer pool: Pre-allocated reusable buffers with mutex

Key parameters

• ZSTD_c_nbWorkers: 1-256 threads (0 = single-threaded)
• ZSTD_c_jobSize: Per-thread workload (512KB-1GB)
• ZSTD_c_overlapLog: Overlap fraction between jobs

Challenges

• Job overlap ensures cross-boundary matches are found
• Results must be assembled in order
• Non-deterministic (scheduling-dependent) unless overlap=full
• Memory scales with thread count

What needs to be implemented

1. Job splitting — divide input into chunks with overlap regions
2. Thread pool — worker thread management (could use rayon or custom)
3. Per-job compression — each job independently compresses its chunk
4. Ordered assembly — collect job outputs in original order
5. Overlap management — each job gets tail of previous job as prefix context
6. Buffer pool — reuse allocated buffers across jobs
7. Parameter API — nbWorkers, jobSize, overlapLog

Acceptance criteria

• Multi-threaded compression produces valid zstd output
• C zstd can decompress multi-threaded output
• Speed scales roughly linearly with thread count on large inputs
• Configurable thread count and job size
• Single-threaded mode (nbWorkers=0) has no overhead

Dependencies

• feat: implement Default compression level (zstd level 3, dfast strategy) #5 (Default level) or at minimum working Fastest level

Time estimate

5d

Blocked by

• feat: implement Default compression level (zstd level 3, dfast strategy) #5 (Default level) — at minimum need a working compression level beyond Fastest