Ratio + speed round 2: xz/zstd/lz4 near parity; bwt/mtf/range faster

CHANGELOG.md

@@ -7,6 +7,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Performance
+
+- **Round 2 of encoder ratio + codec speed work** (encoder-only for ratio;
+  decoders unchanged and every format still decodes byte-for-byte with its
+  reference tool). Ratios on a 2.9 MB real-source corpus, our max level vs the
+  reference's max (`ours/ref`, lower is better):
+  - **xz / lzma2**: 1.51 → **1.10** — the LZMA2 chunk encoder now keeps the LZ
+    dictionary **continuous across chunks** (emits `0xC0` continue-dict control
+    bytes after the first `0xE0`, with a single match-finder spanning the whole
+    input) instead of resetting every 64 KiB. Closes nearly all of the gap to
+    the `.lzma` path (1.07). Also fixes the raw-LZMA2 decoder to feed
+    uncompressed (stored) chunks into the dictionary.
+  - **zstd**: 1.40 → **1.04 vs `zstd -19`** at max level (now beats `zstd -12`)
+    — cross-block matching over a retained sliding window (≤8 MiB, within the
+    advertised window) plus a two-pass, statistics-driven optimal parse
+    (btultra2-style repricing) and repeat-offset-aware DP pricing.
+  - **lz4**: 1.18 → **1.02** (frame, `-l 12` beats `lz4 -9`) — implemented LZ4
+    frame **block-linked** mode so matches reference up to 64 KiB of prior
+    blocks' output, not just the current block.
+- **Standalone-codec encode throughput** (output byte-identical):
+  - **bwt** encode ~3× faster — replaced the prefix-doubling rotation sort with
+    linear-time SA-IS suffix-array construction.
+  - **mtf** encode ~2.3× faster (single-pass scan-and-shift); **rangecoder**
+    encode/decode ~+15% (tightened hot loops).
+
+  Note: the xz/lzma2 and zstd encoders now buffer the whole input to drive a
+  single continuous match-finder (same memory profile the `.lzma` path already
+  had); higher levels trade encode time for ratio, decode speed is unchanged.
+
 ## [0.6.3](https://github.com/KarpelesLab/compcol/compare/v0.6.2...v0.6.3) - 2026-06-15
 
 ### Added

src/bwt/mod.rs

@@ -36,15 +36,19 @@
 //!
 //! ## Forward transform
 //!
-//! For each block we build the order of its cyclic rotations with a
-//! prefix-doubling (Manber–Myers) sort — `O(n log n)` ranking rounds, each an
-//! `O(n)` counting sort on the `(rank[i], rank[i + k])` pairs. From the sorted
-//! rotation order `sa` (where `sa[r]` is the starting offset of the rotation
-//! ranked `r`) the BWT last column is `L[r] = block[(sa[r] + n - 1) mod n]`,
-//! and the primary index is the rank of the rotation that starts at offset 0.
+//! For each block we build the order of its cyclic rotations by suffix-sorting
+//! the doubled block `T + T + $` (sentinel `$` smaller than any byte) with
+//! **SA-IS** (Suffix Array by Induced Sorting; Nong, Zhang & Chan, 2009), a
+//! linear-time `O(n)` construction. From the sorted rotation order `sa` (where
+//! `sa[r]` is the starting offset of the rotation ranked `r`) the BWT last
+//! column is `L[r] = block[(sa[r] + n - 1) mod n]`, and the primary index is
+//! the rank of the rotation that starts at offset 0.
 //!
-//! The prefix-doubling sort is `O(n log n)` and handles the pathological cases
-//! (all-equal bytes, long repeats) without degrading to `O(n² log n)`.
+//! SA-IS is `O(n)` and handles the pathological cases (all-equal bytes, long
+//! repeats) without degrading. A small KMP-based tie-break pass reorders any
+//! run of *equal* cyclic rotations (only possible for fully periodic blocks)
+//! by ascending offset, so the emitted `(L, primary)` pair is identical to
+//! that of a stable cyclic-rotation sort.
 //!
 //! ## Inverse transform
 //!
@@ -64,8 +68,9 @@
 //! ## Licensing
 //!
 //! Clean-room from the published BWT algorithm description (Burrows & Wheeler,
-//! 1994) and the textbook prefix-doubling rotation sort. No code was copied
-//! from `src/bzip2/` or any third-party source.
+//! 1994) and the published SA-IS suffix-array construction (Nong, Zhang &
+//! Chan, 2009). Implemented independently within this module; no code was
+//! copied from `src/bzip2/` or any third-party source.
 
 #![cfg_attr(docsrs, doc(cfg(feature = "bwt")))]