Skip to content

docs: track sub-64 multi-batch runs in results matrix; mark old lineage stale #80

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
booth-algo merged 2 commits into from
Jun 2, 2026
Merged

docs: track sub-64 multi-batch runs in results matrix; mark old lineage stale #80

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3335bc6
docs: track sub-64 multi-batch decoder runs in results matrix; mark o…
booth-algo Jun 1, 2026
39a90a0
docs: report simulated latency (not host wall) for sub-64 multi-batch…
booth-algo Jun 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 31 additions & 2 deletions doc/EMULATOR_MODEL_CONFIG_MATRIX.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# PLENA Emulator Model/Config Results Matrix

Generated: `2026-05-22` | Last updated: `2026-05-25`
Generated: `2026-05-22` | Last updated: `2026-06-02`

Simulator checkout:
```text
Expand All @@ -14,6 +14,15 @@ compiler submodule HEAD: 287f6fb
> **Note:** History was squashed on 2026-05-27. All prior sim commits in the
> results table now map to `45c02b8`.

> **⚠️ Lineage / staleness note (2026-06-02):** Every row dated **≤ 2026-05-26** was produced on
> the now-superseded `exp/roll-attention-head-batch` lineage (sim `45c02b8`). `main` has since
> advanced well past it — sub-64 single-batch (PLENA_Compiler #51/#54), native non-packed
> **batch_size>1 incl. true sub-64** (#55 / Simulator #79 / Tools #6), and the accelerator
> refactors (#68/#71/#73/#75). Those rows' **`Repro @HEAD?` is therefore unverified against current
> `main`** and should be treated as stale unless re-run. New rows below are pinned to their actual
> producing commits. (Marked stale rather than re-run: the older matrix is 40+ rows, several
> multi-hour on the shared box.)

Key fixes in this update (2026-05-25):
- **Golden RoPE bug fix**: golden was using X_embed (un-normed) instead of X_norm for RoPE cos term, causing 82% FAIL on SmolLM2/SmolVLM2 due to peaked-softmax amplification. Now 100% at 1L.
- **Q/O projection for mlen=256**: `linear_projection` ISA ops + W_q/W_o weight loading. All 5 decoder models pass 100% at mlen=256 through 10 layers.
Expand Down Expand Up @@ -43,8 +52,20 @@ Key fixes in this update (2026-05-25):

All rows sorted by date descending. Allclose bar is 90% unless noted.

| Date | Sim commit | Comp commit | Mode | Model | Arch | L | MLEN | BLEN | seq | hidden | inter | batch | ISA | Wall | Allclose | Pass | MAE | Repro @HEAD? | Notes |
> **`Sim lat` column = simulated latency** (`executor.now()`, the modeled hardware time — the real
> accelerator metric). For 2026-06-01+ rows this is the emulator's `Simulation completed. Latency …`
> value; host wall time (box-load-dependent, **not** a hardware metric) is noted inline only as
> context. **Earlier rows show host *wall* time in this column** (it was labelled `Wall`) and are
> stale per the lineage note — sim latency was never captured for them because the harness ran the
> emulator at `--log-level warn`, which suppressed the INFO latency line. That harness gap is now
> fixed (`RUST_LOG=warn,transactional_emulator=info`), so `sim_latency_ns` is recorded automatically.

| Date | Sim commit | Comp commit | Mode | Model | Arch | L | MLEN | BLEN | seq | hidden | inter | batch | ISA | Sim lat | Allclose | Pass | MAE | Repro @HEAD? | Notes |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2026-06-01 | `56ed1e7`‡ | `0f55571`‡ | native | SmolVLM2-256M-text | LlamaForCausalLM | 1 | 16 | 4 | 4 | 576 | 1536 | 2 | 1.24M | 10.03ms | **99.07%** | PASS | 0.037 | YES | **true sub-64** head_dim 64>mlen 16 (4 col-blocks). Was `NotImplementedError` before #55. Multi-batch via kernel batch-loop + rpb comparison (Tools #6). Host ~17s |
| 2026-06-01 | `56ed1e7`‡ | `0f55571`‡ | native | SmolVLM2-256M-text | LlamaForCausalLM | 1 | 32 | 4 | 4 | 576 | 1536 | 2 | 1.16M | 16.42ms | **99.67%** | PASS | 0.034 | YES | true sub-64 head_dim 64>mlen 32 (2 col-blocks). Sim latency normal (cf 16/4 = 10.03ms) — **no pathology**. The original host run took 3.37h **under heavy box contention**; an identical quiet-box re-run was 106s (114× faster). Host wall is unreliable; sim latency is the metric |
| 2026-06-01 | `56ed1e7`‡ | `0f55571`‡ | native | SmolVLM2-256M-text | LlamaForCausalLM | 1 | 64 | 16 | 4 | 576 | 1536 | 2 | 148K | 2.39ms | **97.37%** | PASS | 0.052 | YES | head_dim==mlen batched path. Byte-identical ISA before/after the kernel edit (no-op proof). Was a 29% false-negative before Tools #6. Host ~33s |
| 2026-06-01 | `56ed1e7`‡ | `0f55571`‡ | native | SmolVLM2-256M-text | LlamaForCausalLM | 1 | 64 | 16 | 4 | 576 | 1536 | 1 | 73K | 1.28ms | **99.96%** | PASS | 0.031 | YES | batch=1 baseline (no regression); seq<mlen path. Host ~15s |
| 2026-05-26 | `45c02b8` | `ab4f52c` | sliced | CLM-60M | LlamaForCausalLM | 1 | 256 | 64 | 256 | 256 | 512 | 1 | — | — | **100.00%** | PASS | — | YES | GQA hq=3 hkv=1 (packed 3 Q heads). First fused GQA sliced test |
| 2026-05-26 | `45c02b8` | `ab4f52c` | sliced | SmolLM2-135M | LlamaForCausalLM | 1 | 256 | 64 | 256 | 256 | 512 | 1 | — | — | 95.69% | FAIL | — | N/A | GQA hq=3: packed-head precision sensitivity |
| 2026-05-26 | `45c02b8` | `ab4f52c` | sliced | SmolVLM2-256M-text | LlamaForCausalLM | 1 | 256 | 64 | 256 | 256 | 512 | 1 | — | — | 96.46% | FAIL | — | N/A | GQA hq=3: packed-head precision sensitivity |
Expand Down Expand Up @@ -95,6 +116,14 @@ All rows sorted by date descending. Allclose bar is 90% unless noted.

† Golden-only compile — no emulator run.

‡ Multi-batch rows were measured on the content-equivalent local commits (compiler `069930b`,
sim `2ac6b71` on `fix/vision-smolvlm2`); `program_attention.py` / `plena_frontend.py` /
`emulator_runner.py` are byte-identical to the cited PR commits (compiler #55 `0f55571`, sim #79
`56ed1e7`, tools #6 `a6ac9e3`), so the results reproduce there. `Sim lat` = simulated latency
(`executor.now()`) read back from the emulator's `Simulation completed. Latency …ns` line, recovered
by re-running the emulator on the saved build artifacts (the original runs were at `--log-level
warn`, which suppressed it). Inline "Host ~Ns" is uncontended host wall time, for context only.

## How to run tests

### Sliced
Expand Down
Loading