Roadmap: structured-zstd feature parity with C zstd

[polaz]

Execution order (dependency-aware)

Last actualized: 2026-04-12

Phase 1: Correctness & Core (P0-critical) — DONE ✅

These are blocking bugs and the most-used missing feature.

Order	Issue	What	Est
1.1	✅ #15	fix: large literals block panic	1d
1.2	✅ #17	feat: FSE table reuse + offset history (broken encoding)	2d
1.3	✅ #5	feat: Default compression level (dfast, level 3)	3d

Dependency chain: ✅ #15 and ✅ #17 are independent, both should be done before ✅ #5 (correct encoding foundation).

Phase 2: CoordiNode Critical Path (P1-high) — DONE ✅

Core features needed for the CoordiNode use case.

Order	Issue	What	Blocked by	Est
2.1	✅ #24	test: benchmark suite	—	2d
2.2	✅ #14	perf: encoder match interleaving	—	2d
2.3	✅ #12	perf: sequence execution wildcopy	—	3d
2.4	✅ #8	feat: dictionary compression	✅ #17	3d
2.5	✅ #9	feat: streaming encoder	—	2d
2.6	✅ #6	feat: Better level (lazy2, level 7)	✅ #5	4d

Parallelizable: ✅ #24, ✅ #14, ✅ #12, ✅ #9 are independent. ✅ #8 needs ✅ #17. ✅ #6 needs ✅ #5.

Phase 3: Performance Parity (P2-medium) — DONE ✅

Close the 1.4-3.5x decompression gap with C zstd.

Order	Issue	What	Blocked by	Est
3.1	✅ #10	perf: Huffman 4-stream parallel	—	3d
3.2	✅ #11	perf: FSE batched refill for state updates	—	2d
3.3	✅ #13	perf: bitstream reader	—	2d
3.4	✅ #20	perf: decoder pre-allocation	—	4h
3.5	✅ #16	feat: frame content size	—	1d
3.6	✅ #7	feat: Best level (btlazy2, level 11)	✅ #6	4d
3.7	✅ #21	feat: numeric levels 1-22	✅ #5, ✅ #6, ✅ #7	1d
3.8	✅ #25	feat: FastCOVER dict builder	—	3d
3.9	✅ #56	perf: packed FSE Entry layout (4-byte + bulk spread + cache alignment)	✅ #11	2d 4h
3.10	✅ #47	perf: reuse encoded scratch buffer across streaming blocks	—	1h
3.11	✅ #51	perf: rebase HC table positions (remove 4 GiB cutoff)	—	2d
3.12	✅ #67	perf(encoding): row-based match finder for fast/dfast levels	—	3d

Parallelizable: ✅ #25, ✅ #56, ✅ #47, ✅ #51, ✅ #67 were independent.

Phase 4: SIMD & Hardware Acceleration (P1-high / P2-medium) — DONE ✅

Architecture-specific optimizations to reach C zstd parity. Ordered by impact.

Order	Issue	What	Blocked by	Est
4.1	✅ #68	perf(decoding): SIMD wildcopy for literal+match memcpy	—	3d
4.2	✅ #69	perf(decoding): branchless offset history + stride prefetch + BMI2 pext	—	1d 4h
4.3	✅ #66	perf(decoding): SIMD HUF decode kernels (BMI2/AVX2/VBMI2/NEON)	✅ #56 (recommended)	4d
4.4	✅ #88	perf(encoding): eliminate default-level small-input dfast reset/table clear cliff	—	2d
4.5	✅ #70	perf(encoding): SIMD match-length comparison (SSE2/AVX2/NEON)	—	1d
4.6	✅ #97	perf(encoding): early incompressible fast-path for fastest/default encode + default decodecorpus dfast parity vs C	—	2d 4h
4.7	✅ #71	perf: ARM platform optimizations (CRC32 hash, NEON, SVE2 histcnt)	#68 (NEON wildcopy)	2d 4h

Critical path update: ✅ #68 (PR #85, 2026-04-09), ✅ #69 (PR #90, 2026-04-09), ✅ #66 (PR #92, 2026-04-09), ✅ #70 (PR #96, 2026-04-09), ✅ #97 (PR #99, 2026-04-11), ✅ #88 (verified+closed, 2026-04-11), ✅ #71 (PR #104, 2026-04-11), and ✅ #86 (PR #105, 2026-04-12) are completed. Next highest-impact open item is #22.

Parallelizable: Phase 4B item #86 is completed; Phase 5 items remain independently schedulable by dependency order.

Expected combined impact:

• Decode throughput: +60-100% (from 1.4-3.5x slower → ~1.0-1.5x slower vs C zstd)
• Encode throughput: +15-30% at levels 5+ (SIMD match comparison + ARM CRC32 hash)
• Encode throughput (incompressible/random): major latency reduction from early no-compress fast-path (perf(encoding): early incompressible fast-path for fastest/default encode #97)
• Encode latency (tiny payloads, default level): remove 100x-class cliff vs C for small-4k-log-lines path

Phase 4B: Dictionary Decode Hot Path (P2-medium) — DONE ✅

Latency follow-up immediately after SIMD/HW phase.

Order	Issue	What	Blocked by	Est
4B.1	✅ #86	perf(decoding): pre-parsed dictionary handle for repeated dict decode	Phase 4 complete	1d 4h

Phase 5: Advanced Features (P3-low)

Full feature parity with C zstd.

Order	Issue	What	Blocked by	Est
5.1	#22	feat: optimal parsing (levels 16-22)	✅ #7, ✅ #21	5d
5.2	#23	feat: block splitting	✅ #5	3d
5.3	#18	feat: long distance matching	—	3d
5.4	#26	feat: magicless format	—	4h
5.5	#27	feat: configurable parameters API	✅ #21	1d
5.6	#19	feat: multi-threaded compression	✅ #5	5d
5.7	#72	perf(decoding): parallel block decompression	—	3d

Dependency graph

Phase 1-2 (DONE):
✅#15 ──→ ✅#5 ──→ ✅#6 ──→ ✅#7 ──→ ✅#21
✅#17 ──→ ✅#8      │         │        │
✅#9  ✅#12 ✅#14   │         │        ├──→ #27 (params API)
✅#24 ✅#10 ✅#11   │         │        └──→ #22 (optimal parsing)
✅#13 ✅#20 ✅#16   ├──→ #19  │
                   └──→ #23  └──→ #22

Phase 3 (Performance Parity — DONE):
  ✅#56 (packed FSE) ──→ ✅#66 (SIMD HUF decode)
  ✅#25 (FastCOVER)
  ✅#47 (scratch reuse)
  ✅#51 (HC rebase)
  ✅#67 (row matcher)

Phase 4 (SIMD & HW Acceleration — DONE):
  ✅#68 (SIMD wildcopy) ──→ ✅#71 (ARM optimizations, NEON backend)
  ✅#69 (branchless offset + prefetch + pext)
  ✅#66 (SIMD HUF kernels, BMI2/AVX2/VBMI2/NEON)
  ✅#88 (default-level small-input dfast cliff)
  ✅#70 (SIMD match-length comparison)
  ✅#97 (early incompressible fast-path + default decodecorpus dfast parity vs C)
  ✅#86 (pre-parsed dictionary handle for repeated dict decode)

Phase 5 (Advanced Features):
  #22 (optimal parsing)
  #23 (block splitting)
  #18 (LDM)
  #26 (magicless)
  #27 (params API)
  #19 (multi-thread compress)
  #72 (parallel block decompress)

Recommended execution order (next actions)

Highest impact per effort — do these next:

1. feat: optimal parsing compression (btopt/btultra/btultra2 strategies) #22 (optimal parsing compression) — 5d — highest-impact remaining feature-parity item
2. feat: block splitting for improved compression ratio #23 (block splitting) — 3d — follow-up ratio/feature parity improvement
3. perf(decoding): parallel block decompression for multi-block frames #72 (parallel block decompression) — 3d — next decode throughput multiplier after parity features

After these 3 items, decode should stay in the ~1.0-1.3x range vs C zstd on literals-heavy paths with additional headroom from parallel block decompression, while remaining encode parity work concentrates in optimal parsing + block splitting. The early incompressible fast-path item remains completed in ✅ #97.

Total estimate

• Phase 3 remaining: ~0d
• Phase 4 (new): ~0d
• Phase 4B: ~0d
• Phase 5: ~20d 4h
• Total remaining: ~20d 4h (~20-21 working days) for full feature parity

Roadmap tail backlog

7. research(decoding): evaluate advanced SIMD wildcopy paths beyond current baseline #87 (research: advanced SIMD wildcopy paths) — 2d 4h — evaluate post-Phase-4 SIMD candidates and benchmark go/no-go decisions

[polaz]

Roadmap actualized after merge of #56 (PR #76).

Updated:

• Marked perf: packed FSE Entry layout (4-byte entries + bulk table spreading) #56 as done (perf: packed FSE Entry layout (4-byte entries + bulk table spreading) #56) in Phase 3 table and dependency graph
• Updated Phase 4 dependency note for perf(decoding): SIMD HUF decode kernels with runtime CPU dispatch #66 (blocked-by prerequisite now closed)
• Removed perf: packed FSE Entry layout (4-byte entries + bulk table spreading) #56 from "next actions" list and renumbered remaining priorities
• Recalculated rough remaining estimates (Phase 3 and total)\n\nVerification source: issue perf: packed FSE Entry layout (4-byte entries + bulk table spreading) #56 state = CLOSED, PR perf(fse): pack decoder entries and align decode tables #76 state = MERGED.

[polaz]

Roadmap actualized (2026-04-07):

• Verified referenced issue states against repository issues (no status-marker mismatches found).
• Updated roadmap issue header with wtmp begins Thu Dec 18 20:35:54 EET 2025.
• Recalculated remaining estimate based on currently open roadmap issues:
  • Phase 3: ~5d 1h
  • Phase 4: ~11d 8h
  • Phase 5: ~20d 4h
  • Total: ~36d 13h (~37 working days).

No roadmap items were added/removed in this pass; execution order remains unchanged.

[polaz]

Roadmap actualized after merge of #47.

Changes applied:

• Marked perf(encoding): reuse encoded scratch buffer across streaming blocks #47 as completed in the Phase 3 table (3.10), parallelizable list, and dependency graph (done marker).
• Updated estimates: Phase 3 remaining changed from 5d 1h to 5d; total remaining changed from 36d 13h to 36d 12h.
• Updated Last actualized date to 2026-04-08.

No additional issue status changes were detected in this pass.

[polaz]

Roadmap actualization (2026-04-08):

• verified roadmap issue references against current GitHub issue states
• migrated completed entries from strikethrough format () to issue-mode format ()
• updated Phase 3 to mark as closed
• normalized closed blockers in dependency columns to

Current plan state: completed through Phase 3.11; next highest-impact open queue remains .

[polaz]

Roadmap actualization (2026-04-08):

• verified roadmap issue references against current GitHub issue states
• migrated completed entries from strikethrough format to issue-mode format (✅ #N)
• updated Phase 3 to mark ✅ perf(encoding): rebase HC table positions to remove 4 GiB Better level cutoff #51 as closed
• normalized closed blockers in dependency columns to ✅ #N

Current plan state: completed through Phase 3.11; next highest-impact open queue remains #68 -> #69 -> #66 -> #70 -> #67 -> #71.

Note: previous comment in this thread had shell-escaping artifacts; this comment is the corrected summary.

[polaz]

Roadmap actualized after merge of #67 (PR #84).

Updated in issue body:

• Marked Phase 3 as DONE ✅ and switched 3.12 to ✅ perf(encoding): implement row-based match finder for fast/dfast levels #67
• Updated Phase 3 parallelizable/dependency graph entries to completed state
• Removed perf(encoding): implement row-based match finder for fast/dfast levels #67 from next-actions queue and renumbered priorities
• Recalculated estimates:
  • Phase 3 remaining: ~0d
  • Phase 4: ~11d 8h
  • Phase 5: ~20d 4h
  • Total remaining: ~31d 12h (~32 working days)

Verification source:

• perf(encoding): implement row-based match finder for fast/dfast levels #67 state = CLOSED
• perf(decoding): SIMD wildcopy for literal and match memcpy #68, perf(decoding): branchless offset history, stride prefetch, BMI2 pext #69, perf(decoding): SIMD HUF decode kernels with runtime CPU dispatch #66, perf(encoding): SIMD match-length comparison (SSE4.2/AVX2/NEON) #70, perf: ARM platform optimizations (CRC32 hash, NEON copy, SVE2 histcnt) #71, feat: optimal parsing compression (btopt/btultra/btultra2 strategies) #22, feat: block splitting for improved compression ratio #23, feat: long distance matching (LDM) for compression #18, feat: magicless frame format support #26, feat: configurable compression parameters API #27, feat: multi-threaded parallel compression #19, perf(decoding): parallel block decompression for multi-block frames #72 = OPEN

[polaz]

Roadmap actualized after merge of PR #85 (2026-04-09).

Changes applied:

• Marked as complete in Phase 4 and updated critical-path note.
• Reordered next actions to start from (since is done).
• Recomputed remaining estimates: Phase 4 from to , total from to .
• Kept in Phase 4 as the encoding default small-input latency-cliff fix.

Also closed issue #68 explicitly (PR #85 did not auto-close it because it lacked a closing keyword).

[polaz]

Roadmap actualized after merge of PR #85 (2026-04-09).

Changes applied:

• Marked perf(decoding): SIMD wildcopy for literal and match memcpy #68 as complete in Phase 4 and updated critical-path note.
• Reordered next actions to start from perf(decoding): branchless offset history, stride prefetch, BMI2 pext #69 (since perf(decoding): SIMD wildcopy for literal and match memcpy #68 is done).
• Recomputed remaining estimates: Phase 4 from ~13d 8h to ~10d 8h, total from ~34d 16h to ~31d 16h.
• Kept perf(encoding): eliminate default-level small-input dfast reset/table clear cliff #88 in Phase 4 as the encoding default small-input latency-cliff fix.

Also closed issue #68 explicitly (PR #85 did not auto-close it because it lacked a closing keyword).