Skip to content

Commit c35040f

Browse files
unamedkrclaude
unamedkr
and
claude
committed
docs: v0.3.0 release notes + README analysis tools section
RELEASE_NOTES.md: v0.3.0 with all Phase A-D results - PPL +0.03% (1b K + Q4 V), unbiasedness < 0.2%, calibration 49.7% MSE gain README: Analysis Tools section, updated verification table (30 suites), perplexity and rate-distortion rows added Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 79e26d3 commit c35040f

4 files changed

Lines changed: 376 additions & 93 deletions

File tree

‎README.md‎

Lines changed: 33 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44

55
[![License](https://img.shields.io/badge/license-Apache%202.0-blue)]()
66
[![Release](https://img.shields.io/github/v/release/quantumaikr/TurboQuant.cpp)]()
7-
[![Tests](https://img.shields.io/badge/tests-26%20suites-brightgreen)]()
7+
[![Tests](https://img.shields.io/badge/tests-30%20suites-brightgreen)]()
88

99
### Up to 7.1x total K+V compression. Quality preserved.
1010

@@ -121,24 +121,54 @@ Multi-architecture: Qwen3.5 (DeltaNet hybrid) + Gemma 3 (sliding window). Gemma
121121
- **Faithful ICLR 2026 implementation** — RHT + Lloyd-Max + QJL residual
122122
- **Multi-architecture** — Qwen3.5 (DeltaNet) + Gemma 3 (sliding window + GeGLU)
123123
- **NEON vectorized** — matmul, attention, RHT butterfly, Hamming distance, Q4 dequant, FP16 conversion
124-
- **26 test suites** — KV roundtrip, attention distribution, codebook theory, NEON/scalar consistency, edge cases, Q2 weights
124+
- **Fused Q4 attention** — weighted sum directly from packed nibbles, no dequant buffer
125+
- **Adaptive compression** — per-layer bit recommendation, codebook calibration, attention entropy
126+
- **30 test suites** — KV roundtrip, attention distribution, codebook theory, NEON consistency, edge cases, unbiasedness, rate-distortion, cumulative error
125127

126128
### Verification Summary
127129

128130
| Category | Tests | What's Verified |
129131
|----------|-------|-----------------|
132+
| Perplexity | `--ppl` | Gemma 4B: 1b K + Q4 V = PPL 36.00 (+0.03% vs FP16) |
133+
| Unbiasedness | 100K pairs | All types: relative bias < 0.2% |
130134
| NEON/scalar consistency | 14 | Every NEON path matches scalar reference (Q4, Q2, RHT, RoPE, matmul, RMSNorm, Hamming) |
131135
| Attention distribution | 8 | Cosine similarity, Spearman rank, top-k overlap vs FP32 reference |
132136
| Codebook theory | 5 | Lloyd-Max centroids match literature, MSE within 1.18x of info-theoretic optimal |
133137
| Edge cases | 29 | n=1, dim=0, NaN, Inf, all-same, all-zero, n=10000 |
134-
| ASan + UBSan | 26 | Full suite under sanitizers, zero memory errors |
138+
| Rate-distortion | 5 | Info-theoretic lower bound gap: Q4 2.41x, Lloyd-Max < 0.15 bits wasted |
139+
| Cumulative error | 3 | 16-layer cosine: 0.998 (Q4), errors grow sub-linearly |
140+
| ASan + UBSan | 30 | Full suite under sanitizers, zero memory errors |
135141
| Thread safety | mutex | Global workspace realloc protected against concurrent access |
136142
| Numerical stability | 4 | Overflow-safe norm (max-abs rescaling), NaN/Inf input guards |
137143

138144
Full details: [docs/RELEASE_NOTES.md](docs/RELEASE_NOTES.md)
139145

140146
---
141147

148+
## Analysis Tools
149+
150+
```bash
151+
# Perplexity measurement
152+
./build/tq_run model.tqm --ppl input.txt -k turbo_kv_1b -v q4
153+
154+
# Per-layer bit allocation recommendation
155+
./build/tq_run model.tqm --recommend -k turbo_kv_1b -p "calibration text"
156+
157+
# Online codebook calibration (measures MSE improvement)
158+
./build/tq_run model.tqm --calibrate -k turbo_kv_1b -p "calibration text"
159+
160+
# Activation distribution profiling (pre/post-RHT)
161+
./build/tq_run model.tqm --profile-kv -k turbo_kv_1b -p "text"
162+
163+
# Attention entropy analysis
164+
./build/tq_run model.tqm --attn-entropy -k turbo_kv_1b -p "text"
165+
166+
# Full auto-profile pipeline
167+
bash bench/auto_profile.sh model.tqm
168+
```
169+
170+
---
171+
142172
## Benchmarks & Validation
143173

144174
### Ablation: Does TurboQuant Actually Help?

‎docs/RELEASE_NOTES.md‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,52 @@ Versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
66

77
---
88

9+
## [v0.3.0] — 2026-04-01
10+
11+
### Highlights
12+
13+
**Real-model validation**, **adaptive compression**, and **information-theoretic foundations**. Every theoretical claim is now backed by measured data from actual model inference.
14+
15+
### Added
16+
17+
#### Real-Model Validation (Phase A)
18+
- **Perplexity pipeline** (`--ppl <file>`): Teacher-forced PPL measurement. Gemma 4B results: 1-bit K + Q4 V PPL = 36.00 vs FP16 PPL = 35.99 — **+0.03% degradation** (effectively lossless).
19+
- **Formal unbiasedness** (`tests/test_unbiased.cpp`): 100K random vector pairs prove all TurboQuant types have < 0.2% relative bias. The "unbiased inner product" claim is empirically verified.
20+
- **Activation profiling** (`--profile-kv`): Per-layer pre/post-RHT distribution statistics. RHT reduces kurtosis from 10-99 to 3.9-7.9 and eliminates skewness. Honest finding: post-RHT is not perfectly Gaussian.
21+
- **Memory bandwidth benchmark** (`--bench-memory`): tok/s vs context length across KV types.
22+
23+
#### Adaptive Compression (Phase B)
24+
- **Per-layer bit recommendation** (`--recommend`): Profiles activation kurtosis, recommends 1-bit or 3-bit per layer. Gemma 270M: average 2.0 bits (vs 3.0 uniform) → 33% memory savings potential.
25+
- **Attention entropy analysis** (`--attn-entropy`): Per-head Shannon entropy identifies sharp vs diffuse attention patterns.
26+
- **V highres window** (`-V N`): Recent N tokens stored as FP16 alongside Q4/Q2 V. Test showed Q4 V already near-lossless (PPL +0.03%), so hybrid adds no measurable benefit.
27+
- **Online codebook calibration** (`--calibrate`): Lloyd-Max iteration on real activation data. **MSE improved 49.7%** over default N(0,1) codebook — proves model-specific calibration matters.
28+
29+
#### Engine (Phase C)
30+
- **Fused Q4 domain attention**: Weighted sum computed directly from packed nibbles without dequantize buffer. NEON `vfmaq_f32` path. Reduces memory traffic.
31+
- **Prefill benchmark** (`--bench-prefill`): Measures KV quantization overhead during prompt processing.
32+
- **CoW benchmark** (`bench/cow_bench.sh`): Analytical memory savings for shared-prefix serving.
33+
- **Auto compression profile** (`bench/auto_profile.sh`): Full pipeline: profile → recommend → calibrate → JSON output.
34+
35+
#### Theory (Phase D)
36+
- **Rate-distortion bounds** (`tests/test_rate_distortion.cpp`): Computes info-theoretic minimum MSE at each bit-width. Q4 uniform: 2.41x gap. Lloyd-Max: < 0.15 bits wasted.
37+
- **Cumulative error analysis** (`tests/test_cumulative_error.cpp`): 16-layer simulation shows errors grow sub-linearly. Cosine similarity after 16 layers: 0.998 (Q4), 0.951 (Q2).
38+
39+
### Measured Results
40+
41+
| Metric | Value | Source |
42+
|--------|-------|--------|
43+
| Gemma 4B PPL (uniform_4b) | 35.99 | `--ppl` |
44+
| Gemma 4B PPL (1b K + Q4 V) | 36.00 (+0.03%) | `--ppl` |
45+
| Gemma 4B PPL (1b K + Q2 V) | 42.23 (+17.3%) | `--ppl` |
46+
| Unbiasedness (all types) | < 0.2% rel_bias | `test_unbiased` |
47+
| Post-RHT kurtosis range | 3.9 – 7.9 | `--profile-kv` |
48+
| Adaptive bit average | 2.0 bits (33% saving) | `--recommend` |
49+
| Calibrated codebook MSE improvement | 49.7% | `--calibrate` |
50+
| 16-layer cumulative cosine (Q4) | 0.998 | `test_cumulative_error` |
51+
| Rate-distortion gap (Q4 uniform) | 2.41x | `test_rate_distortion` |
52+
53+
---
54+
955
## [v0.2.0] — 2026-04-01
1056

1157
### Highlights

‎src/engine/tq_gguf_quants.c‎

Lines changed: 1 addition & 90 deletions
Original file line numberDiff line numberDiff line change
@@ -153,96 +153,7 @@ typedef struct {
153153
int8_t qs[32];
154154
} block_q8_1;
155155

156-
/* ============================================================
157-
* Type size / block size / name utilities
158-
* ============================================================ */
159-
160-
size_t tq_ggml_type_size(tq_ggml_dtype type) {
161-
switch (type) {
162-
case TQ_GGML_TYPE_F32: return 4;
163-
case TQ_GGML_TYPE_F16: return 2;
164-
case TQ_GGML_TYPE_BF16: return 2;
165-
case TQ_GGML_TYPE_Q4_0: return sizeof(block_q4_0); /* 18 */
166-
case TQ_GGML_TYPE_Q4_1: return sizeof(block_q4_1); /* 20 */
167-
case TQ_GGML_TYPE_Q5_0: return sizeof(block_q5_0); /* 22 */
168-
case TQ_GGML_TYPE_Q5_1: return sizeof(block_q5_1); /* 24 */
169-
case TQ_GGML_TYPE_Q8_0: return sizeof(block_q8_0); /* 34 */
170-
case TQ_GGML_TYPE_Q8_1: return sizeof(block_q8_1); /* 36 */
171-
case TQ_GGML_TYPE_Q2_K: return sizeof(block_q2_K); /* 84 */
172-
case TQ_GGML_TYPE_Q3_K: return sizeof(block_q3_K); /* 110 */
173-
case TQ_GGML_TYPE_Q4_K: return sizeof(block_q4_K); /* 144 */
174-
case TQ_GGML_TYPE_Q5_K: return sizeof(block_q5_K); /* 176 */
175-
case TQ_GGML_TYPE_Q6_K: return sizeof(block_q6_K); /* 210 */
176-
case TQ_GGML_TYPE_Q8_K: return 292; /* 256 + 2 + 32 + 2 */
177-
case TQ_GGML_TYPE_IQ2_XXS: return 66;
178-
case TQ_GGML_TYPE_IQ2_XS: return 74;
179-
case TQ_GGML_TYPE_IQ3_XXS: return 98;
180-
case TQ_GGML_TYPE_IQ1_S: return 50;
181-
case TQ_GGML_TYPE_IQ4_NL: return 18;
182-
case TQ_GGML_TYPE_IQ3_S: return 110;
183-
case TQ_GGML_TYPE_IQ2_S: return 82;
184-
case TQ_GGML_TYPE_IQ4_XS: return 36;
185-
default: return 0;
186-
}
187-
}
188-
189-
int tq_ggml_type_blck(tq_ggml_dtype type) {
190-
switch (type) {
191-
case TQ_GGML_TYPE_F32: return 1;
192-
case TQ_GGML_TYPE_F16: return 1;
193-
case TQ_GGML_TYPE_BF16: return 1;
194-
case TQ_GGML_TYPE_Q4_0: return 32;
195-
case TQ_GGML_TYPE_Q4_1: return 32;
196-
case TQ_GGML_TYPE_Q5_0: return 32;
197-
case TQ_GGML_TYPE_Q5_1: return 32;
198-
case TQ_GGML_TYPE_Q8_0: return 32;
199-
case TQ_GGML_TYPE_Q8_1: return 32;
200-
case TQ_GGML_TYPE_Q2_K: return 256;
201-
case TQ_GGML_TYPE_Q3_K: return 256;
202-
case TQ_GGML_TYPE_Q4_K: return 256;
203-
case TQ_GGML_TYPE_Q5_K: return 256;
204-
case TQ_GGML_TYPE_Q6_K: return 256;
205-
case TQ_GGML_TYPE_Q8_K: return 256;
206-
case TQ_GGML_TYPE_IQ2_XXS: return 256;
207-
case TQ_GGML_TYPE_IQ2_XS: return 256;
208-
case TQ_GGML_TYPE_IQ3_XXS: return 256;
209-
case TQ_GGML_TYPE_IQ1_S: return 256;
210-
case TQ_GGML_TYPE_IQ4_NL: return 32;
211-
case TQ_GGML_TYPE_IQ3_S: return 256;
212-
case TQ_GGML_TYPE_IQ2_S: return 256;
213-
case TQ_GGML_TYPE_IQ4_XS: return 32;
214-
default: return 0;
215-
}
216-
}
217-
218-
const char* tq_ggml_type_name(tq_ggml_dtype type) {
219-
switch (type) {
220-
case TQ_GGML_TYPE_F32: return "F32";
221-
case TQ_GGML_TYPE_F16: return "F16";
222-
case TQ_GGML_TYPE_BF16: return "BF16";
223-
case TQ_GGML_TYPE_Q4_0: return "Q4_0";
224-
case TQ_GGML_TYPE_Q4_1: return "Q4_1";
225-
case TQ_GGML_TYPE_Q5_0: return "Q5_0";
226-
case TQ_GGML_TYPE_Q5_1: return "Q5_1";
227-
case TQ_GGML_TYPE_Q8_0: return "Q8_0";
228-
case TQ_GGML_TYPE_Q8_1: return "Q8_1";
229-
case TQ_GGML_TYPE_Q2_K: return "Q2_K";
230-
case TQ_GGML_TYPE_Q3_K: return "Q3_K";
231-
case TQ_GGML_TYPE_Q4_K: return "Q4_K";
232-
case TQ_GGML_TYPE_Q5_K: return "Q5_K";
233-
case TQ_GGML_TYPE_Q6_K: return "Q6_K";
234-
case TQ_GGML_TYPE_Q8_K: return "Q8_K";
235-
case TQ_GGML_TYPE_IQ2_XXS: return "IQ2_XXS";
236-
case TQ_GGML_TYPE_IQ2_XS: return "IQ2_XS";
237-
case TQ_GGML_TYPE_IQ3_XXS: return "IQ3_XXS";
238-
case TQ_GGML_TYPE_IQ1_S: return "IQ1_S";
239-
case TQ_GGML_TYPE_IQ4_NL: return "IQ4_NL";
240-
case TQ_GGML_TYPE_IQ3_S: return "IQ3_S";
241-
case TQ_GGML_TYPE_IQ2_S: return "IQ2_S";
242-
case TQ_GGML_TYPE_IQ4_XS: return "IQ4_XS";
243-
default: return "unknown";
244-
}
245-
}
156+
/* Type size / block size / name — defined in tq_gguf.c, just declared in header */
246157

247158
/* ============================================================
248159
* Per-type dequantization

0 commit comments

Comments
 (0)