diff --git a/doc/EMULATOR_MODEL_CONFIG_MATRIX.md b/doc/EMULATOR_MODEL_CONFIG_MATRIX.md index fb6737b6..73715c0c 100644 --- a/doc/EMULATOR_MODEL_CONFIG_MATRIX.md +++ b/doc/EMULATOR_MODEL_CONFIG_MATRIX.md @@ -1,6 +1,6 @@ # PLENA Emulator Model/Config Results Matrix -Generated: `2026-05-22` | Last updated: `2026-05-25` +Generated: `2026-05-22` | Last updated: `2026-06-02` Simulator checkout: ```text @@ -14,6 +14,15 @@ compiler submodule HEAD: 287f6fb > **Note:** History was squashed on 2026-05-27. All prior sim commits in the > results table now map to `45c02b8`. +> **⚠️ Lineage / staleness note (2026-06-02):** Every row dated **≤ 2026-05-26** was produced on +> the now-superseded `exp/roll-attention-head-batch` lineage (sim `45c02b8`). `main` has since +> advanced well past it — sub-64 single-batch (PLENA_Compiler #51/#54), native non-packed +> **batch_size>1 incl. true sub-64** (#55 / Simulator #79 / Tools #6), and the accelerator +> refactors (#68/#71/#73/#75). Those rows' **`Repro @HEAD?` is therefore unverified against current +> `main`** and should be treated as stale unless re-run. New rows below are pinned to their actual +> producing commits. (Marked stale rather than re-run: the older matrix is 40+ rows, several +> multi-hour on the shared box.) + Key fixes in this update (2026-05-25): - **Golden RoPE bug fix**: golden was using X_embed (un-normed) instead of X_norm for RoPE cos term, causing 82% FAIL on SmolLM2/SmolVLM2 due to peaked-softmax amplification. Now 100% at 1L. - **Q/O projection for mlen=256**: `linear_projection` ISA ops + W_q/W_o weight loading. All 5 decoder models pass 100% at mlen=256 through 10 layers. @@ -43,8 +52,20 @@ Key fixes in this update (2026-05-25): All rows sorted by date descending. Allclose bar is 90% unless noted. -| Date | Sim commit | Comp commit | Mode | Model | Arch | L | MLEN | BLEN | seq | hidden | inter | batch | ISA | Wall | Allclose | Pass | MAE | Repro @HEAD? | Notes | +> **`Sim lat` column = simulated latency** (`executor.now()`, the modeled hardware time — the real +> accelerator metric). For 2026-06-01+ rows this is the emulator's `Simulation completed. Latency …` +> value; host wall time (box-load-dependent, **not** a hardware metric) is noted inline only as +> context. **Earlier rows show host *wall* time in this column** (it was labelled `Wall`) and are +> stale per the lineage note — sim latency was never captured for them because the harness ran the +> emulator at `--log-level warn`, which suppressed the INFO latency line. That harness gap is now +> fixed (`RUST_LOG=warn,transactional_emulator=info`), so `sim_latency_ns` is recorded automatically. + +| Date | Sim commit | Comp commit | Mode | Model | Arch | L | MLEN | BLEN | seq | hidden | inter | batch | ISA | Sim lat | Allclose | Pass | MAE | Repro @HEAD? | Notes | |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| 2026-06-01 | `56ed1e7`‡ | `0f55571`‡ | native | SmolVLM2-256M-text | LlamaForCausalLM | 1 | 16 | 4 | 4 | 576 | 1536 | 2 | 1.24M | 10.03ms | **99.07%** | PASS | 0.037 | YES | **true sub-64** head_dim 64>mlen 16 (4 col-blocks). Was `NotImplementedError` before #55. Multi-batch via kernel batch-loop + rpb comparison (Tools #6). Host ~17s | +| 2026-06-01 | `56ed1e7`‡ | `0f55571`‡ | native | SmolVLM2-256M-text | LlamaForCausalLM | 1 | 32 | 4 | 4 | 576 | 1536 | 2 | 1.16M | 16.42ms | **99.67%** | PASS | 0.034 | YES | true sub-64 head_dim 64>mlen 32 (2 col-blocks). Sim latency normal (cf 16/4 = 10.03ms) — **no pathology**. The original host run took 3.37h **under heavy box contention**; an identical quiet-box re-run was 106s (114× faster). Host wall is unreliable; sim latency is the metric | +| 2026-06-01 | `56ed1e7`‡ | `0f55571`‡ | native | SmolVLM2-256M-text | LlamaForCausalLM | 1 | 64 | 16 | 4 | 576 | 1536 | 2 | 148K | 2.39ms | **97.37%** | PASS | 0.052 | YES | head_dim==mlen batched path. Byte-identical ISA before/after the kernel edit (no-op proof). Was a 29% false-negative before Tools #6. Host ~33s | +| 2026-06-01 | `56ed1e7`‡ | `0f55571`‡ | native | SmolVLM2-256M-text | LlamaForCausalLM | 1 | 64 | 16 | 4 | 576 | 1536 | 1 | 73K | 1.28ms | **99.96%** | PASS | 0.031 | YES | batch=1 baseline (no regression); seq