From 622d07a1664f53a01e12e4730cc009aca31b42ce Mon Sep 17 00:00:00 2001 From: booth-algo Date: Wed, 3 Jun 2026 18:18:18 +0100 Subject: [PATCH 01/10] feat(emulator): weight-streaming memory models (HostStream + LayerSwapping) wired into --memory-model Adds lib/memory/streaming.rs (HostStream = direct host->SRAM streaming bypassing DDR3 for weights; LayerSwapping = capacity-limited DDR3 that swaps layer weights on demand; WeightManifest parsed from JSON), and re-implements HBM-model instantiation in runner.rs (the emulator was refactored since this work began: dispatch/Accelerator moved out of the monolithic main.rs into src/accelerator/, so the original main.rs wiring no longer applied). runner.rs now preloads the backing store up front and builds Arc via match opts.memory_model {hbm,layer-swap,host-stream}. The default --memory-model hbm path is byte-for-byte identical (verified: HBM stats, Latency, vram_dump unchanged); cargo build --release clean. Streaming-model statistics are not yet surfaced through the erased trait (follow-up). --- doc/SMOLVLM2_ISA_PROFILE.md | 268 ++++++++++++++++++ doc/memory-footprint-and-streaming.md | 161 +++++++++++ transactional_emulator/Cargo.lock | 2 + transactional_emulator/Cargo.toml | 1 + transactional_emulator/lib/memory/Cargo.toml | 1 + transactional_emulator/lib/memory/src/lib.rs | 1 + .../lib/memory/src/streaming.rs | 219 ++++++++++++++ transactional_emulator/src/cli.rs | 30 ++ transactional_emulator/src/runner.rs | 165 ++++++++--- 9 files changed, 815 insertions(+), 33 deletions(-) create mode 100644 doc/SMOLVLM2_ISA_PROFILE.md create mode 100644 doc/memory-footprint-and-streaming.md create mode 100644 transactional_emulator/lib/memory/src/streaming.rs diff --git a/doc/SMOLVLM2_ISA_PROFILE.md b/doc/SMOLVLM2_ISA_PROFILE.md new file mode 100644 index 00000000..21abc3a1 --- /dev/null +++ b/doc/SMOLVLM2_ISA_PROFILE.md @@ -0,0 +1,268 @@ +# SmolVLM2-256M ISA Cycle Profile + +**Config:** 12L vision encoder + connector + 1L text decoder, mlen=256, blen=64, seq=256 +**Board:** Nexys Video — Artix-7 XC7A200T @ 100MHz, DDR3-1600 x16 (1.6 GB/s peak, ~1.0 GB/s effective) +**Note:** mlen=256 exceeds Artix-7 synthesis resources. This profile is a what-if analysis. Realisable configs use mlen=64 (proven) or mlen=128 (untested). +**ASM source:** `transactional_emulator/testbench/conv/build/smolvlm2_native_module/` +**Profiler:** `transactional_emulator/testbench/aten/compare/asm_profiler.py --board nexys_a7 --mlen 256` +**Date:** 2026-05-28 (corrected from 05-27: x16 bus width, validated against published references) + +## Summary + +| Stage | ISA lines | Dynamic instrs | Est cycles | Est time @100MHz | +|-------|-----------|----------------|-----------|------------------| +| Vision + Connector (12L) | 1,444,035 | 5,580,752 | 28.0M | **280 ms** | +| Text Decoder (1L) | 1,206,857 | 2,638,488 | 17.5M | **175 ms** | +| **Total VLM e2e** | 2,650,892 | 8,219,240 | 45.5M | **455 ms** | + +## Vision + Connector (12L vision + connector) + +**28.0M cycles @ Nexys A7 (dc_lib_dis, DDR3)** + +### Per-opcode breakdown (top 15) + +| Op | Dynamic count | Cyc/op | Total cycles | % | +|----|--------------|--------|-------------|---| +| M_MM | 44,496 | 256 | 11,390,976 | 40.7% | +| V_RED_SUM | 271,872 | 20 | 5,437,440 | 19.5% | +| S_ADDI_INT | 2,054,228 | 1 | 2,054,228 | 7.3% | +| V_MUL_VF | 297,216 | 5 | 1,486,080 | 5.3% | +| V_MUL_VV | 271,872 | 5 | 1,359,360 | 4.9% | +| H_PREFETCH_M | 3,033 | 400 | 1,213,200 | 4.3% | +| C_LOOP_END | 766,161 | 1 | 766,161 | 2.7% | +| V_ADD_VV | 343,296 | 2 | 686,592 | 2.5% | +| M_TMM | 2,304 | 256 | 589,824 | 2.1% | +| V_EXP_V | 73,728 | 6 | 442,368 | 1.6% | +| S_ST_FP | 381,125 | 1 | 381,125 | 1.4% | +| S_ADD_FP | 326,450 | 1 | 326,450 | 1.2% | +| H_PREFETCH_V | 15,492 | 20 | 309,840 | 1.1% | +| V_RECI_V | 36,864 | 7 | 258,048 | 0.9% | +| S_MAP_V_FP | 832 | 256 | 212,992 | 0.8% | + +### Category summary + +| Category | Cycles | % | +|----------|--------|---| +| **Matrix compute** | 11,998,272 | **42.9%** | +| **Vector compute** | 10,076,928 | **36.0%** | +| Scalar overhead | 3,537,049 | 12.7% | +| DDR3 memory | 1,546,080 | **5.5%** | +| Control | 796,938 | 2.9% | + +## Text Decoder (1L) + +**17.5M cycles @ Nexys A7 (dc_lib_dis, DDR3)** + +### Per-opcode breakdown (top 15) + +| Op | Dynamic count | Cyc/op | Total cycles | % | +|----|--------------|--------|-------------|---| +| M_MM | 32,040 | 256 | 8,202,240 | 46.9% | +| S_MAP_V_FP | 7,680 | 256 | 1,966,080 | 11.2% | +| H_PREFETCH_M | 3,240 | 400 | 1,296,000 | 7.4% | +| V_RED_SUM | 63,552 | 20 | 1,271,040 | 7.3% | +| S_ADDI_INT | 875,152 | 1 | 875,152 | 5.0% | +| V_MUL_VV | 161,472 | 5 | 807,360 | 4.6% | +| V_MUL_VF | 138,432 | 5 | 692,160 | 4.0% | +| V_EXP_V | 63,360 | 6 | 380,160 | 2.2% | +| V_RECI_V | 46,080 | 7 | 322,560 | 1.8% | +| C_LOOP_END | 267,883 | 1 | 267,883 | 1.5% | +| V_ADD_VV | 103,872 | 2 | 207,744 | 1.2% | +| S_LUI_INT | 201,149 | 1 | 201,149 | 1.1% | +| S_ST_FP | 197,760 | 1 | 197,760 | 1.1% | +| V_SUB_VF | 63,360 | 2 | 126,720 | 0.7% | +| V_ADD_VF | 46,080 | 2 | 92,160 | 0.5% | + +### Category summary + +| Category | Cycles | % | +|----------|--------|---| +| **Matrix compute** | 8,235,810 | **47.1%** | +| **Vector compute** | 4,055,424 | **23.2%** | +| Scalar overhead | 3,615,382 | 20.7% | +| DDR3 memory | 1,310,640 | **7.5%** | +| Control | 279,514 | 1.6% | + +## DDR3 Memory Impact + +The Nexys Video uses a single **MT41K256M16HA** DDR3-1600 chip with a **x16 bus** (2 bytes/transfer). + +| Parameter | Value | Notes | +|-----------|-------|-------| +| Chip | MT41K256M16HA | Single x16, 512 MiB | +| Peak bandwidth | **1.6 GB/s** | 2B × 800 MT/s (x16 bus) | +| Effective bandwidth | **~1.0 GB/s** | ~60-65% MIG efficiency for sequential streaming | +| MIG read latency | ~27.7 cycles | ZipCPU measured on Kintex-7 MIG | +| Burst latency | ~5 PLENA cycles (50ns) | CAS + transfer for 64B | +| Row miss penalty | +3 cycles (30ns) | tRP+tRCD = 27.5ns | +| H_PREFETCH_V cost | 25 cycles | 256B / 1.0 GB/s = 256ns | +| H_PREFETCH_M cost | 400 cycles | 4KB / 1.0 GB/s = 4000ns | + +| Stage | DDR3 cycles | % of total | Static prefetch count | +|-------|------------|-----------|---------------------| +| Vision+Connector | 1,629,300 | **5.8%** | 15,492 V + 3,033 M + 288 store | +| Text Decoder | 1,314,300 | **7.5%** | 5 V + 3,240 M + 180 store | + +The text decoder is **more DDR3-bound** (7.5%) despite fewer total prefetches because M_PREFETCH dominates (3,240 × 400 = 1.3M cycles) — each matrix prefetch loads 4KB of weights from DDR3. + +## Per-Kernel Breakdown + +### Vision + Connector (12L) — 280 ms + +| Kernel | Cycles | % | ms | Description | +|--------|--------|---|----|-------------| +| **Attention** | 13.5M | **48.4%** | 135 | Q/K/V projections + flash attention × 12 layers | +| **Layer Norm** | 8.4M | **30.0%** | 84 | RMS norm × 25 (2 per layer + final). V_RED_SUM (20 cyc) + V_MUL_VV (5 cyc) per element | +| **Conv2d im2col** | 6.0M | **21.4%** | 60 | SigLIP K=16 patch embedding via vector reductions (no matrix unit) | +| Connector | 0.07M | 0.2% | 0.7 | Linear projection 12288→576 | + +### Text Decoder (1L) — 182 ms + +| Kernel | Cycles | % | ms | Description | +|--------|--------|---|----|-------------| +| **FFN** | 8.7M | **48.0%** | 88 | Gate+Up linear (M_MM) + SiLU + Down linear (M_MM) | +| **Attention** | 5.3M | **29.1%** | 53 | Packed GQA (3 heads × online softmax + P@V via M_MM) | +| **Q/K/V Projection** | 2.5M | **13.8%** | 25 | Weight load from DDR3 + M_MM for Q/K/V/O | +| **RMS Norm** | 1.6M | **8.6%** | 16 | 2× per layer + final norm | +| Layer setup | 0.07M | 0.4% | 0.7 | VRAM zero-fill, residual copy | + +### Observations + +1. **Vision is attention-dominated (48%)** — 12 layers of self-attention with head projections dominate +2. **Vision layer norm is 30%** — 25 norm operations at 3.4ms each, driven by expensive V_RED_SUM (20 cycles on Artix-7) +3. **Conv2d im2col is 21% of vision** — patch embedding done entirely with vector ops, matrix unit idle +4. **Text decoder is FFN-dominated (48%)** — 3 large M_MM operations (gate, up, down projections: 576→1536→576) +5. **QKV projection is 14% of text** — weight loading from DDR3 is a significant fraction +6. **Connector is trivial (0.2%)** — single linear projection, not a bottleneck + +## Optimization Opportunities + +Ranked by estimated cycle savings on Nexys A7 (with DDR3): + +| # | Target | Cycles saved | % of total | Approach | +|---|--------|-------------|------------|----------| +| 1 | **V_RED_SUM in vision im2col** | ~5.4M | 11.9% | Conv2d patch embedding uses V_RED_SUM (271K × 20 cyc) for dot products. Restructure im2col to gather patches into matrix SRAM and dispatch M_MM. | +| 2 | **DDR3 prefetch stalls** | ~2.9M | 6.3% | 6,273 M_PREFETCH × 400 cyc. Prefetch scheduling, double-buffering, or weight caching in BRAM could hide latency. | +| 3 | **S_ADDI_INT loop overhead** | ~2.9M | 6.4% | Loop counter increments (2.9M calls). Hardware auto-increment or compiler loop fusion. | +| 4 | **S_MAP_V_FP in text decoder** | ~2.0M | 4.4% | 7,680 scalar→vector broadcasts × 256 cyc. Vector fill instruction or register caching. | +| 5 | **V_MUL_VV + V_MUL_VF** | ~3.5M | 7.7% | 509K calls combined. Fused multiply-accumulate (V_MAC) would halve cycles. | +| 6 | **Softmax chain** | ~1.5M | 3.3% | V_EXP + V_RECI + V_RED_MAX + V_SUB form softmax. Fused instruction reduces pipeline stalls. | + +### Conv2d analysis + +The vision encoder's conv2d (SigLIP patch embedding, K=16, C=3) compiles via im2col → vector reductions. The matrix unit handles linear projections inside transformer layers (M_MM at 40.7%) but **not the initial patch convolution**. + +Restructuring im2col to fill matrix SRAM and dispatch M_MM would reduce patch embedding from ~5.4M cycles to ~100K cycles — a **17% improvement** in vision encoder time. + +### DDR3 vs HBM analysis + +On V80 with HBM2e (819 GB/s), memory prefetch is effectively free (pipelined). On Nexys A7 with DDR3, memory accounts for 5-7% of cycles. For memory-bound workloads (e.g. weight loading for large models), DDR3 becomes the primary bottleneck — this profile is compute-bound because SmolVLM2-256M is a small model. + +## Memory Footprint vs Nexys Video DDR3 + +### Actual HBM binary sizes (MXFP8 quantised weights + activations) + +| Stage | HBM binary | Notes | +|-------|-----------|-------| +| Vision + Connector (12L) | 295 MB | 12 vision layers + connector weights | +| Text Decoder (1L) | 205 MB | 1/30 decoder layers | +| **Combined (1v+1t)** | **500 MB** | Fits in 512 MB DDR3 (barely) | +| Text Decoder (30L, estimated) | ~6 GB | Full 30-layer decoder | + +### The 512 MB DDR3 problem + +The Nexys Video board has **512 MB DDR3**. SmolVLM2-256M at MXFP8 quantisation: +- 12L vision + connector: 295 MB +- 1L text decoder: 205 MB → 30L would be ~6 GB + +A single VLM e2e pass (12L vision + 1L text) barely fits at 500 MB. **Full 30-layer +text decoding is impossible** without weight streaming — loading decoder layers one +at a time from host, running each layer, then replacing it with the next. + +### Implications for the emulator + +The Rust transactional emulator currently models HBM as a flat address space +(configurable size, default 16 GB). It does not model: + +1. **DDR3 capacity limits** — no OOM when allocations exceed 512 MB +2. **Weight streaming** — no mechanism to swap layer weights in/out during execution +3. **DDR3 bank conflicts** — prefetch latency is modeled as constant, not address-dependent +4. **Refresh cycles** — DDR3 refresh steals bandwidth periodically + +To model the Nexys Video memory constraint accurately, the emulator would need: +- A configurable HBM capacity cap (512 MB) that raises an error on overflow +- A weight-streaming ISA extension or runtime mechanism to load weights per-layer +- Bank-conflict-aware prefetch latency (address → bank → row hit/miss → cycles) + +For now, the cycle profile assumes all weights fit in DDR3. The 1L text decoder +config is realistic for the Nexys Video; the 30L config is V80-only. + +## Validation + +### Static profiler vs Rust emulator + +The ASM profiler was validated against the Rust transactional emulator (Ramulator HBM2 timing, dc_en=1) on the linear test (64×128 × 128×64 matmul, mlen=64): + +| Source | Cycles | +|--------|--------| +| Rust emulator (Ramulator HBM2) | 146,935 | +| Static profiler (dc_lib_en) | 144,938 | +| **Difference** | **1.4%** | + +### DDR3 bandwidth cross-references + +| Source | Bandwidth | Notes | +|--------|-----------|-------| +| **Our model** | 1.0 GB/s effective | x16 DDR3-1600, ~60-65% efficiency | +| Vitis AI DPU B1152 (PG338) | 684-1017 MB/s avg | Production DDR3/DDR4 accelerator | +| Flare on XC7A100T (PMC 2024) | 3.125 GB/s | x32 bus (2× our x16) | +| ultraembedded DDR3 on Arty A7 | 400 MB/s | Open-source controller @100MHz | +| ZipCPU MIG memtest (2025) | 27.7 cyc avg read | MIG controller latency measured | +| Gemmini (DAC 2021) | 8 bytes/cycle | LPDDR4 model for ASIC | + +Our 1.0 GB/s effective matches the Vitis AI DPU B1152 average range (684-1017 MB/s). The 400 cycles per 4KB M_PREFETCH derives from 4KB / 1.0 GB/s = 4μs = 400 cycles @100MHz. + +### Weight streaming architecture validation + +Published FPGA accelerators use the same DDR layer-by-layer streaming pattern as our LayerSwapping model: + +| System | Architecture | Reference | +|--------|-------------|-----------| +| Apache TVM VTA | Tiled DRAM-to-SRAM DMA, access-execute decoupled | arXiv 1807.04188 | +| Vitis AI DPU | Layer-sequential DDR streaming via micro-coded dispatch | AMD PG338 | +| FINN | On-chip default; `load_external_weights()` for DDR streaming | finn.readthedocs.io | +| DNNWeaver | DDR3 bandwidth as tiling constraint | MICRO 2016 | +| AutoWS | Automated on-chip/DDR weight partitioning for layer-pipelined | arXiv 2311.04764 | +| FlightLLM | HBM weight streaming at 65.9% utilization | FPGA 2024 | + +Host-to-SRAM direct streaming (our HostStream model) is **not standard practice** for real-time inference — USB 2.0 at 38 MB/s is too slow. It is valid for modeling what-if scenarios (e.g., PCIe at 2 GB/s) and batch/offline inference. + +### Weight streaming emulator validation + +All 3 memory models produce **identical VRAM output** (verified via binary diff). Only simulated latency differs. Tested on the ATen linear test (64×128 × 128×64 matmul, mlen=64): + +| Model | CLI flag | Latency (ns) | Slowdown | Notes | +|-------|----------|-------------|----------|-------| +| HBM (Ramulator) | `--memory-model hbm` | 146,935 | 1.0x | Pipelined HBM2, sub-cycle prefetch | +| LayerSwap (DDR3) | `--memory-model layer-swap` | 1,120,727 | **7.6x** | DDR3 SimpleTiming replaces HBM2 — no swap penalty (46KB fits 512MB), slowdown is pure DDR3 vs HBM timing | +| HostStream (USB) | `--memory-model host-stream` | 1,869,354 | **12.7x** | 38 MB/s serialized — each 64B weight chunk costs 1,684ns | + +The 7.6x LayerSwap slowdown represents the **DDR3 vs HBM access latency gap** — this is the real cost of running on Nexys Video DDR3 instead of V80 HBM. The 12.7x HostStream shows the additional penalty of streaming weights from host over USB. + +## Reproducing + +```bash +# Profile with Nexys A7 DDR3 timing +python3 transactional_emulator/testbench/aten/compare/asm_profiler.py \ + transactional_emulator/testbench/conv/build/smolvlm2_native_module/vision_connector/generated_asm_code.asm \ + --board nexys_a7 --mlen 256 + +python3 transactional_emulator/testbench/aten/compare/asm_profiler.py \ + transactional_emulator/testbench/conv/build/smolvlm2_native_module/text_decoder/generated_asm_code.asm \ + --board nexys_a7 --mlen 256 + +# Compare with V80 (HBM, 400MHz) +python3 transactional_emulator/testbench/aten/compare/asm_profiler.py \ + transactional_emulator/testbench/conv/build/smolvlm2_native_module/text_decoder/generated_asm_code.asm \ + --board v80 --mlen 256 +``` diff --git a/doc/memory-footprint-and-streaming.md b/doc/memory-footprint-and-streaming.md new file mode 100644 index 00000000..e10cbb63 --- /dev/null +++ b/doc/memory-footprint-and-streaming.md @@ -0,0 +1,161 @@ +# Memory Footprint & Host-Link Streaming Analysis + +Reasoning notes on whether SmolVLM2-256M needs weight streaming on the PLENA +target boards, and how local memory (DDR3 / HBM) compares to the host link +(USB2 / PCIe). Summary up front: **for SmolVLM2-256M at 8-bit, everything fits +in 512 MB of onboard DDR — keep weights resident and skip weight streaming.** + +## 1. Does SmolVLM2-256M fit in 512 MB? + +Yes, comfortably. + +| Component | Size (MXFP8, 8-bit) | Notes | +|-----------|---------------------|-------| +| Weights | **~288 MB** | 256M params × (8 bits/elem + 8-bit scale per block-of-8 ≈ +12.5%) = ~1.125 B/elem | +| KV cache | tens of MB | ~27 MB @ 1K tokens, ~106 MB @ 4K; sub-MB for the sub-64 bring-up | +| Activations | single-MB / layer | small | +| **Total** | **~300–400 MB** | fits in 512 MB with ~120–200 MB headroom | + +**The "~1 GB" you might observe on a GPU is not the model — it's CUDA +overhead:** the CUDA context (~300–500 MB), cuBLAS/cuDNN workspaces, allocator +fragmentation, and framework tensors. PLENA has none of that. (For reference, +fp32 weights *would* be ~1 GB — 256M × 4 B — so int8 is a 4× cut.) + +At long context the **KV cache** — not the weights — is the only thing that +eventually pressures the 512 MB ceiling. + +## 2. Implication: the streaming models are not for this model + +`transactional_emulator/lib/memory/src/streaming.rs` provides two models, both +designed for the case where **weights do not fit in DDR**: + +- **`HostStream`** — weight reads bypass DDR and pay a fixed host-transfer cost + (stream weights host→SRAM). +- **`LayerSwapping`** — capacity-limited DDR; a read to a non-resident layer + pays a swap-from-host penalty (read-only weights, so eviction is free). + +Since SmolVLM2-256M fits in 512 MB, you use **neither** — you use the plain +**resident-DDR** model: load all weights into DDR once, no streaming, no +swapping. The streaming models remain valuable for models that genuinely exceed +DDR (larger models, or future targets), but they are unnecessary here. + +## 3. Instructions: load once, don't stream + +The ISA program is small (~5–20 MB for sub-64 SmolVLM2) and **fixed/reused +across the decode loop**. With ~200 MB of DDR free after weights, load the +program **once** into DDR and fetch it locally. The host link's only job is the +one-time load (program + weights + input) and the output readback; during +compute it is idle. Continuously fetching instructions from the host adds stall +risk for zero benefit. + +## 4. DDR3 vs the host link (PCIe / USB) + +First, a framing point: **DDR3 is attached memory; PCIe is a transport** to host +RAM. A "PCIe fetch" pays the PCIe link latency **plus** the host's own DRAM +latency on the far end — they are not the same kind of thing. + +### Raw bandwidth (config-dependent) + +| Link | Bandwidth | +|------|-----------| +| DDR3 single ×16 chip (Nexys-class) | ~3.2 GB/s peak (DDR3-1600, 16-bit); ~1 GB/s effective | +| DDR3 desktop DIMM (64-bit channel) | ~12.8 GB/s/channel (25.6 dual-channel) | +| PCIe Gen3 ×16 | ~16 GB/s | +| PCIe Gen4 ×16 | ~32 GB/s | +| PCIe Gen5 ×16 | ~64 GB/s | + +On raw bandwidth a *single low-end DDR3 chip* is slower than a modern PCIe ×16 +link; a *full DDR3 DIMM* is comparable; and wide **HBM (V80: 819 GB/s) dwarfs** +any PCIe. + +### Latency (local memory always wins) + +- DDR3 access: **~15–50 ns** (directly attached). +- PCIe round-trip: **~0.5–1 µs+** (serialization + protocol + host memory + controller) — roughly **10–50× higher latency**. + +For random / latency-sensitive access (weight & activation fetch during +compute), local DDR3 wins regardless of the bandwidth headline. PCIe is good for +**bulk one-time loads**, not for being in the per-cycle fetch path. + +## 5. For the PLENA target boards specifically + +| Board | Local memory | Host link | Verdict | +|-------|--------------|-----------|---------| +| **Nexys Video** (Artix-7) | DDR3 ~1–3.2 GB/s | **USB2 ~38 MB/s** (FT2232H) — *no PCIe* | local DDR3 **~30–80× faster** + far lower latency | +| **V80** (Versal) | HBM **819 GB/s** | PCIe Gen5 ×16 ~64 GB/s | HBM **~13× faster** | +| **Custom Artix-7 + PCIe 2.0 ×4** | DDR3 ~1–3.2 GB/s | **PCIe 2.0 ×4 ~1.6–1.8 GB/s** | *comparable* — streaming becomes viable (§5.1) | + +For **Nexys Video** and **V80** the local memory beats the host link on bandwidth +*and* latency — which is exactly why "keep weights resident, don't stream from +host" is the right call there. (Note: the host link on Nexys Video is **USB2, not +PCIe**; PCIe is the V80 path, and the V80's 32 GB HBM makes the whole model a +rounding error.) The custom Artix-7 board below is the interesting exception. + +### 5.1 Custom Artix-7 with PCIe 2.0 ×4 — streaming flips to viable + +Put the *same* Artix-7 on a custom board with a **PCIe 2.0 ×4** host link and the +calculus changes: + +- **The link:** PCIe 2.0 = 5 GT/s/lane; ×4 = **20 GT/s raw** ("20 Gbps"). After + 8b/10b encoding (×0.8) → **16 Gbps = 2.0 GB/s** usable; after TLP/protocol + overhead (~10–20%) → **~1.6–1.8 GB/s effective payload.** +- **vs local DDR3** (single ×16 DDR3-1600: ~3.2 GB/s peak, ~1–2 GB/s effective): + PCIe 2.0 ×4 is now **at parity with — or faster than — the local DDR3.** That is + ~40× the USB2 figure; the host link is no longer the weak point. + +What that unlocks: + +1. **Weight streaming over PCIe becomes viable — even preferable.** The + `HostStream` model (stream weights host→SRAM, bypassing DDR3) runs at ~1.6 GB/s, + matching or beating the modeled DDR3 effective (~1.0 GB/s). On this board the + streaming models (`HostStream` / `LayerSwapping`) are the *right* tool — the + opposite of the USB2 verdict. +2. **Capacity is the real win:** streaming from host RAM (GBs) removes the 512 MB + ceiling → run models **larger than 512 MB** on this Artix-7. (SmolVLM2-256M + doesn't need it; a larger LM would.) +3. **Best-of-both split:** weights (read-only, sequential, prefetchable) streamed + over PCIe; activations/KV (read-write, random) in local DDR3. + +The one caveat is **latency, not bandwidth:** PCIe round-trip ~µs vs DDR3 ~tens of +ns. Hide it with **weight-prefetch FIFOs** — weight access is sequential, so this +works well (the board config already exposes `hbm_m_prefetch_amount` / prefetch +knobs). Latency would only bite random access, which weights are not. + +**Perf sketch:** streaming 288 MB at ~1.6 GB/s ≈ 180 ms/token → ~5–6 tok/s, vs +~3–4 tok/s for DDR3-resident (at the config's 1.0 GB/s) — a modest speedup *plus* +the bigger-model capability. Stream the **weights** (large, sequential, +bottlenecking), not the instructions (still tiny → load once into DDR). + +## 6. The real bottleneck is DDR bandwidth + +On Nexys, effective DDR ≈ **1.0 GB/s**, so reading ~288 MB of weights per token +≈ **~290 ms → ~3–4 tok/s**. That is the performance wall, *not* the host link. +For contrast, streaming those weights from the host over USB2 (38 MB/s) would be +**~7.6 s/token — ~25× worse** — which is the quantitative reason resident DDR is +the only sane choice. (On V80, HBM at 819 GB/s removes the wall entirely.) + +## 7. Caveat + how to validate in the emulator + +The emulator currently **models DDR as 128 GiB** (`modeled_size_bytes` in +`transactional_emulator/testbench/board_configs/nexys_a7.yaml`), not the physical +512 MB — so it does **not** enforce the real capacity today. + +To turn the "fits in 512 MB" argument into a measured result, run SmolVLM2-256M +through **`LayerSwapping` with `capacity = 512 MB`** and confirm **zero swaps** +(every layer loads once and never evicts). If it ever *does* swap at some +context length, that is the KV cache hitting the ceiling — the genuinely +interesting limit. + +## Bottom line + +For SmolVLM2-256M at 8-bit: keep everything resident in 512 MB DDR and skip +weight streaming. Caveats: (a) the Nexys host link is **USB2, not PCIe**; +(b) load the instruction program into DDR once rather than streaming it; and +(c) the performance wall is **DDR bandwidth (~3–4 tok/s on Nexys)**, not the +host link. + +The exception is a **custom Artix-7 board with PCIe 2.0 ×4** (§5.1): there the +host link (~1.6 GB/s) reaches parity with local DDR3, so weight streaming +becomes viable — and worthwhile mainly because it lifts the 512 MB capacity +ceiling (run from host RAM), with weights over PCIe and activations/KV in DDR3. diff --git a/transactional_emulator/Cargo.lock b/transactional_emulator/Cargo.lock index fb28d483..a75bc430 100644 --- a/transactional_emulator/Cargo.lock +++ b/transactional_emulator/Cargo.lock @@ -795,6 +795,7 @@ dependencies = [ "proptest", "rand 0.9.2", "runtime", + "serde", "tokio", "trait-variant", "zerocopy", @@ -1722,6 +1723,7 @@ dependencies = [ "ramulator", "runtime", "serde", + "serde_json", "sram", "tch", "tokio", diff --git a/transactional_emulator/Cargo.toml b/transactional_emulator/Cargo.toml index c8bbe7ad..76c6c5c0 100644 --- a/transactional_emulator/Cargo.toml +++ b/transactional_emulator/Cargo.toml @@ -15,6 +15,7 @@ tch = { version = "0.20.0", features = ["download-libtorch"] } half = "2" clap = { version = "4", features = ["derive"] } serde = { version = "1.0", features = ["derive"] } +serde_json = "1.0" toml = "0.8" tracing = "0.1" tracing-subscriber = { version = "0.3", features = ["env-filter"] } diff --git a/transactional_emulator/lib/memory/Cargo.toml b/transactional_emulator/lib/memory/Cargo.toml index a123f125..6ff7b1bd 100644 --- a/transactional_emulator/lib/memory/Cargo.toml +++ b/transactional_emulator/lib/memory/Cargo.toml @@ -6,6 +6,7 @@ edition = "2024" [dependencies] async-trait = "0.1.88" trait-variant = "0.1.2" +serde = { version = "1", features = ["derive"] } zerocopy = "0.8" runtime = { path = "../runtime" } tokio = { version = "1.45.1", features = ["sync"] } diff --git a/transactional_emulator/lib/memory/src/lib.rs b/transactional_emulator/lib/memory/src/lib.rs index 4b26e866..456143a6 100644 --- a/transactional_emulator/lib/memory/src/lib.rs +++ b/transactional_emulator/lib/memory/src/lib.rs @@ -2,6 +2,7 @@ pub mod chunked; mod frfcfs; mod naive; mod simple; +pub mod streaming; pub mod testutils; use std::mem::ManuallyDrop; diff --git a/transactional_emulator/lib/memory/src/streaming.rs b/transactional_emulator/lib/memory/src/streaming.rs new file mode 100644 index 00000000..f918770b --- /dev/null +++ b/transactional_emulator/lib/memory/src/streaming.rs @@ -0,0 +1,219 @@ +use std::sync::Mutex; + +use runtime::Duration; +use tokio::sync::Semaphore; + +use crate::MemoryModel; + +#[derive(Debug, Clone, serde::Deserialize)] +pub struct WeightRegion { + pub layer_id: u32, + pub offset: u64, + pub size: u64, +} + +#[derive(Debug, Clone, serde::Deserialize)] +pub struct WeightManifest { + pub activation_ceiling: u64, + pub regions: Vec, +} + +impl WeightManifest { + pub fn is_weight_addr(&self, addr: u64) -> bool { + if addr < self.activation_ceiling { + return false; + } + self.regions + .iter() + .any(|r| addr >= r.offset && addr < r.offset + r.size) + } + + pub fn layer_for_addr(&self, addr: u64) -> Option { + self.regions + .iter() + .find(|r| addr >= r.offset && addr < r.offset + r.size) + .map(|r| r.layer_id) + } +} + +#[derive(Debug, Default)] +pub struct StreamingStats { + pub weight_reads: u64, + pub weight_bytes: u64, + pub activation_reads: u64, + pub activation_bytes: u64, +} + +/// Direct host-to-SRAM streaming memory model. +/// +/// Weight reads bypass DDR3 and model a host transfer at a fixed bandwidth. +/// Activation reads go through the inner memory model (DDR3 timing). +/// Data is always served from the backing store — only timing differs. +pub struct HostStream { + inner: T, + manifest: WeightManifest, + host_transfer_time_per_chunk: Duration, + host_link: Semaphore, + stats: Mutex, +} + +impl HostStream { + pub fn new(inner: T, manifest: WeightManifest, host_bandwidth_bytes_per_sec: u64) -> Self { + let nanos_per_chunk = 64 * 1_000_000_000 / host_bandwidth_bytes_per_sec; + Self { + inner, + manifest, + host_transfer_time_per_chunk: Duration::from_nanos(nanos_per_chunk), + host_link: Semaphore::new(1), + stats: Mutex::new(StreamingStats::default()), + } + } + + pub fn statistics(&self) -> StreamingStats { + let guard = self.stats.lock().unwrap(); + StreamingStats { + weight_reads: guard.weight_reads, + weight_bytes: guard.weight_bytes, + activation_reads: guard.activation_reads, + activation_bytes: guard.activation_bytes, + } + } +} + +impl MemoryModel for HostStream { + async fn read(&self, addr: u64) -> [u8; 64] { + if self.manifest.is_weight_addr(addr) { + let _permit = self.host_link.acquire().await.unwrap(); + let rt = runtime::Executor::current(); + rt.resolve_at(self.host_transfer_time_per_chunk).await; + { + let mut stats = self.stats.lock().unwrap(); + stats.weight_reads += 1; + stats.weight_bytes += 64; + } + self.inner.read(addr).await + } else { + { + let mut stats = self.stats.lock().unwrap(); + stats.activation_reads += 1; + stats.activation_bytes += 64; + } + self.inner.read(addr).await + } + } + + async fn write(&self, addr: u64, bytes: [u8; 64]) { + self.inner.write(addr, bytes).await + } +} + +#[derive(Debug, Default)] +pub struct LayerSwapStats { + pub total_swaps: u64, + pub total_swap_bytes: u64, + pub total_swap_nanos: u64, +} + +/// Layer-swapping DDR3 memory model. +/// +/// Simulates a capacity-limited DDR3 where only a few layers' weights fit +/// at a time. When a weight read targets a non-resident layer, the model +/// charges a swap penalty (evict + load) based on host bandwidth. +/// Activation reads always go through DDR3 timing. +pub struct LayerSwapping { + inner: T, + manifest: WeightManifest, + capacity: u64, + swap_bandwidth_bytes_per_sec: u64, + state: Mutex, + stats: Mutex, +} + +#[derive(Debug)] +struct LayerSwapState { + resident_layers: Vec, + used_bytes: u64, +} + +impl LayerSwapping { + pub fn new( + inner: T, + manifest: WeightManifest, + capacity: u64, + swap_bandwidth_bytes_per_sec: u64, + ) -> Self { + Self { + inner, + manifest, + capacity, + swap_bandwidth_bytes_per_sec, + state: Mutex::new(LayerSwapState { + resident_layers: Vec::new(), + used_bytes: 0, + }), + stats: Mutex::new(LayerSwapStats::default()), + } + } + + pub fn statistics(&self) -> LayerSwapStats { + let guard = self.stats.lock().unwrap(); + LayerSwapStats { + total_swaps: guard.total_swaps, + total_swap_bytes: guard.total_swap_bytes, + total_swap_nanos: guard.total_swap_nanos, + } + } + + fn layer_size(&self, layer_id: u32) -> u64 { + self.manifest + .regions + .iter() + .filter(|r| r.layer_id == layer_id) + .map(|r| r.size) + .sum() + } + + fn ensure_resident(&self, layer_id: u32) -> Option { + let mut state = self.state.lock().unwrap(); + if state.resident_layers.contains(&layer_id) { + return None; + } + + let needed = self.layer_size(layer_id); + + while state.used_bytes + needed > self.capacity && !state.resident_layers.is_empty() { + let evicted = state.resident_layers.remove(0); + let freed = self.layer_size(evicted); + state.used_bytes = state.used_bytes.saturating_sub(freed); + } + + let load_nanos = needed * 1_000_000_000 / self.swap_bandwidth_bytes_per_sec; + state.resident_layers.push(layer_id); + state.used_bytes += needed; + + { + let mut stats = self.stats.lock().unwrap(); + stats.total_swaps += 1; + stats.total_swap_bytes += needed; + stats.total_swap_nanos += load_nanos; + } + + Some(Duration::from_nanos(load_nanos)) + } +} + +impl MemoryModel for LayerSwapping { + async fn read(&self, addr: u64) -> [u8; 64] { + if let Some(layer_id) = self.manifest.layer_for_addr(addr) { + if let Some(swap_duration) = self.ensure_resident(layer_id) { + let rt = runtime::Executor::current(); + rt.resolve_at(swap_duration).await; + } + } + self.inner.read(addr).await + } + + async fn write(&self, addr: u64, bytes: [u8; 64]) { + self.inner.write(addr, bytes).await + } +} diff --git a/transactional_emulator/src/cli.rs b/transactional_emulator/src/cli.rs index 74774565..95869b8c 100644 --- a/transactional_emulator/src/cli.rs +++ b/transactional_emulator/src/cli.rs @@ -106,6 +106,16 @@ fn parse_size(s: &str) -> Result { .ok_or_else(|| format!("size {:?} overflows usize", s)) } +#[derive(Clone, Copy, Debug, ValueEnum)] +pub(crate) enum MemoryModelKind { + /// Unlimited HBM with Ramulator HBM2 timing (current default) + Hbm, + /// DDR3 with layer swapping (host streams layers on demand) + LayerSwap, + /// Direct host-to-SRAM streaming (no DDR3 for weights) + HostStream, +} + #[derive(Parser)] pub(crate) struct Opts { #[arg(long)] @@ -139,6 +149,26 @@ pub(crate) struct Opts { #[arg(long, help_heading = "Logging")] pub(crate) log_file: Option, + #[arg(long, value_enum, default_value = "hbm")] + /// Memory model to simulate. + /// + /// hbm — Unlimited HBM with Ramulator timing (default) + /// layer-swap — DDR3 with layer swapping (requires --weight-manifest) + /// host-stream — Direct host-to-SRAM streaming (requires --weight-manifest) + pub(crate) memory_model: MemoryModelKind, + + #[arg(long)] + /// Path to weight manifest JSON (required for layer-swap and host-stream). + pub(crate) weight_manifest: Option, + + #[arg(long, value_parser = parse_size, default_value = "512M")] + /// DDR3 capacity for layer-swap model. + pub(crate) ddr3_capacity: Option, + + #[arg(long, default_value = "60000000")] + /// Host-to-board bandwidth in bytes/sec (default: 60 MB/s = USB 2.0 bulk). + pub(crate) host_bandwidth: Option, + #[arg(long, value_parser = parse_size)] /// Override HBM allocation size (default: from plena_settings.toml). /// diff --git a/transactional_emulator/src/runner.rs b/transactional_emulator/src/runner.rs index ef522341..d5cdb8bf 100644 --- a/transactional_emulator/src/runner.rs +++ b/transactional_emulator/src/runner.rs @@ -129,10 +129,94 @@ pub(crate) async fn run_from_cli() { effective_hbm_size, effective_hbm_size as f64 / (1024.0 * 1024.0 * 1024.0) ); - let hbm = Arc::new(memory::WithStats::new(memory::WithTiming::new( - ManuallyDrop::new(ramulator::Ramulator::hbm2_preset(8).unwrap()), - memory::MemoryBacked::with_capacity(effective_hbm_size), - ))); + // Build and preload the HBM backing store *before* wrapping it in a memory + // model. The streaming models (LayerSwap, HostStream) consume the backing + // store by value, so the preload has to happen up front; the default Hbm + // model preloads here too, which is behaviorally identical to the old + // post-construction `hbm.model().data().with_data(...)` step. + let hbm_backing = memory::MemoryBacked::with_capacity(effective_hbm_size); + let hbm_data = std::fs::read(&opts.hbm) + .unwrap_or_else(|err| panic!("failed to read HBM preload file {:?}: {err}", opts.hbm)); + hbm_backing.with_data(|f| { + f[..hbm_data.len()].copy_from_slice(&hbm_data); + }); + + // Concrete type of the default HBM model. Kept as a typed handle for the + // Hbm case so we can still surface `.statistics()` and dump the backing + // store via `.model().data()` (the `ErasedMemoryModel` trait exposes + // neither). For the streaming models this stays `None`. + type HbmModel = memory::WithStats< + memory::WithTiming, memory::MemoryBacked>, + >; + + // Wrap the preloaded backing store in the selected memory model. `hbm` is + // the erased handle handed to the accelerator; `hbm_concrete` is a typed + // alias of the same allocation, present only for the default Hbm path. + let (hbm, hbm_concrete): (Arc, Option>) = + match opts.memory_model { + cli::MemoryModelKind::Hbm => { + let model: Arc = + Arc::new(memory::WithStats::new(memory::WithTiming::new( + ManuallyDrop::new(ramulator::Ramulator::hbm2_preset(8).unwrap()), + hbm_backing, + ))); + (model.clone(), Some(model)) + } + cli::MemoryModelKind::LayerSwap => { + let manifest_path = opts + .weight_manifest + .as_ref() + .expect("--weight-manifest required for layer-swap memory model"); + let manifest_json = std::fs::read_to_string(manifest_path).unwrap_or_else(|err| { + panic!("failed to read weight manifest {:?}: {err}", manifest_path) + }); + let manifest: memory::streaming::WeightManifest = + serde_json::from_str(&manifest_json).unwrap_or_else(|err| { + panic!("failed to parse weight manifest {:?}: {err}", manifest_path) + }); + let capacity = opts.ddr3_capacity.unwrap_or(512 << 20) as u64; + let bandwidth = opts.host_bandwidth.unwrap_or(60_000_000); + tracing::info!( + "Memory model: layer-swap (DDR3 capacity={} MB, host bandwidth={} MB/s)", + capacity / (1 << 20), + bandwidth / 1_000_000 + ); + let ddr3_timing = memory::SimpleTiming::preset_ddr4_2400p(1); + let model = Arc::new(memory::WithStats::new( + memory::streaming::LayerSwapping::new( + memory::WithTiming::new(ddr3_timing, hbm_backing), + manifest, + capacity, + bandwidth, + ), + )); + (model, None) + } + cli::MemoryModelKind::HostStream => { + let manifest_path = opts + .weight_manifest + .as_ref() + .expect("--weight-manifest required for host-stream memory model"); + let manifest_json = std::fs::read_to_string(manifest_path).unwrap_or_else(|err| { + panic!("failed to read weight manifest {:?}: {err}", manifest_path) + }); + let manifest: memory::streaming::WeightManifest = + serde_json::from_str(&manifest_json).unwrap_or_else(|err| { + panic!("failed to parse weight manifest {:?}: {err}", manifest_path) + }); + let bandwidth = opts.host_bandwidth.unwrap_or(60_000_000); + tracing::info!( + "Memory model: host-stream (host bandwidth={} MB/s)", + bandwidth / 1_000_000 + ); + let model = Arc::new(memory::WithStats::new(memory::streaming::HostStream::new( + hbm_backing, + manifest, + bandwidth, + ))); + (model, None) + } + }; let mut accelerator = Accelerator::new(m_machine, v_machine, hbm.clone()); @@ -151,14 +235,6 @@ pub(crate) async fn run_from_cli() { }) .collect(); - // Memory Initialization - // - HBM Preload - let hbm_data = std::fs::read(&opts.hbm) - .unwrap_or_else(|err| panic!("failed to read HBM preload file {:?}: {err}", opts.hbm)); - hbm.model().data().with_data(|f| { - f[..hbm_data.len()].copy_from_slice(&hbm_data); - }); - // Load fpsram and intsram as raw bytes and map to the vector files. // - fpsram Preload let fpsram_data = std::fs::read(&opts.fpsram).unwrap_or_else(|err| { @@ -210,26 +286,49 @@ pub(crate) async fn run_from_cli() { let fpsram_bytes = accelerator.fpsram_dump_bytes(); dump_to_file("fpsram_dump.bin", &fpsram_bytes); - // Dump HBM — skipped unless DEBUG tracing is enabled because HBM_SIZE may - // be 128 GiB+. Tests run with --log-level warn and don't need hbm_dump.bin; - // only manual debug runs dump HBM. - if tracing::enabled!(tracing::Level::DEBUG) { - let hbm_size = effective_hbm_size; - let mut hbm_bytes = vec![0u8; hbm_size]; - hbm.model().data().with_data(|f| { - let len = std::cmp::min(hbm_size, f.len()); - hbm_bytes[..len].copy_from_slice(&f[..len]); - }); - dump_to_file("hbm_dump.bin", &hbm_bytes); - } + // Dump HBM + report statistics. + // + // Both the HBM dump (which reads back the backing store via + // `.model().data()`) and the byte-count statistics (`.statistics()`) are + // concrete `WithStats>` methods that the erased + // `ErasedMemoryModel` trait does not expose. They are therefore gated on + // the typed `hbm_concrete` handle, which is only populated for the default + // Hbm model. For that default path this is byte-for-byte the old behavior; + // for the streaming models we log the model's own streaming statistics + // instead and skip the (model-less) HBM dump. + match hbm_concrete { + Some(ref hbm) => { + // Dump HBM — skipped unless DEBUG tracing is enabled because + // HBM_SIZE may be 128 GiB+. Tests run with --log-level warn and + // don't need hbm_dump.bin; only manual debug runs dump HBM. + if tracing::enabled!(tracing::Level::DEBUG) { + let hbm_size = effective_hbm_size; + let mut hbm_bytes = vec![0u8; hbm_size]; + hbm.model().data().with_data(|f| { + let len = std::cmp::min(hbm_size, f.len()); + hbm_bytes[..len].copy_from_slice(&f[..len]); + }); + dump_to_file("hbm_dump.bin", &hbm_bytes); + } - let memory_stats = hbm.statistics(); - let utilization = (memory_stats.total_bytes_read + memory_stats.total_bytes_written) as f64 - / Executor::current().now().to_secs(); - tracing::info!( - "HBM Statistics - Bytes read: {:?} | Bytes written: {:?} | Utilization: {:.2e} bytes/sec", - memory_stats.total_bytes_read, - memory_stats.total_bytes_written, - utilization - ); + let memory_stats = hbm.statistics(); + let utilization = (memory_stats.total_bytes_read + memory_stats.total_bytes_written) + as f64 + / Executor::current().now().to_secs(); + tracing::info!( + "HBM Statistics - Bytes read: {:?} | Bytes written: {:?} | Utilization: {:.2e} bytes/sec", + memory_stats.total_bytes_read, + memory_stats.total_bytes_written, + utilization + ); + } + None => { + // Streaming models track their own timing/bandwidth statistics + // internally; the unified byte-count/HBM dump is unavailable. + tracing::info!( + "Memory model: {:?} | Simulation complete (HBM byte statistics unavailable for streaming models)", + opts.memory_model + ); + } + } } From 3d5f1848d5ee47ee4130863349f09da4b14a9c0a Mon Sep 17 00:00:00 2001 From: booth-algo Date: Wed, 3 Jun 2026 18:18:18 +0100 Subject: [PATCH 02/10] feat(profiler): charge DDR3 H_PREFETCH/H_STORE latency from board config memory cycles SimulatorCycleModel gains prefetch_v/prefetch_m/store_v_cycles (default 0); cycle_model_from_board reads them from the board memory: section, so --board nexys_a7 now charges DDR3 latency (V=25, M=400, store=25 cyc). Defaults are 0, so the plena_settings.toml path and every existing caller keep the prior async behaviour unchanged. --- .../testbench/aten/compare/isa_analysis.py | 31 ++++++++++++++++--- 1 file changed, 26 insertions(+), 5 deletions(-) diff --git a/transactional_emulator/testbench/aten/compare/isa_analysis.py b/transactional_emulator/testbench/aten/compare/isa_analysis.py index 15b0f95b..3fa38814 100644 --- a/transactional_emulator/testbench/aten/compare/isa_analysis.py +++ b/transactional_emulator/testbench/aten/compare/isa_analysis.py @@ -74,13 +74,26 @@ class SimulatorCycleModel: scalar_fp_sqrt_cycles: int scalar_fp_reci_cycles: int scalar_int_basic_cycles: int + # DDR3 / memory cycle costs. Board configs supply these (memory: section); + # the default 0 keeps H_PREFETCH/H_STORE async (charged lazily by a later + # SRAM access), matching the behaviour-simulator model used everywhere else. + prefetch_v_cycles: int = 0 + prefetch_m_cycles: int = 0 + store_v_cycles: int = 0 @property def description(self) -> str: + if self.prefetch_v_cycles or self.prefetch_m_cycles or self.store_v_cycles: + mem_desc = ( + "H_PREFETCH/H_STORE statically charged " + f"(V={self.prefetch_v_cycles}, M={self.prefetch_m_cycles}, store={self.store_v_cycles} cyc)" + ) + else: + mem_desc = "H_PREFETCH/H_STORE async memory timing not statically charged" return ( f"behavior simulator constants from {self.settings_path} " f"(DC_EN={self.dc_en}, latency_profile={self.latency_profile or 'dc_lib_dis'}; " - "H_PREFETCH/H_STORE async memory timing not statically charged)" + f"{mem_desc})" ) def instruction_cycles(self, opcode: str) -> int: @@ -116,10 +129,14 @@ def instruction_cycles(self, opcode: str) -> int: return self.scalar_int_basic_cycles if opcode in CONTROL_ONE_CYCLE_OPS: return 1 - if opcode.startswith("H_PREFETCH") or opcode.startswith("H_STORE"): - # The simulator starts an async memory transfer here; the wait is - # paid by a later SRAM read/write if the transfer has not resolved. - return 0 + if opcode == "H_PREFETCH_V": + return self.prefetch_v_cycles + if opcode == "H_PREFETCH_M": + return self.prefetch_m_cycles + if opcode.startswith("H_STORE"): + return self.store_v_cycles + if opcode.startswith("H_PREFETCH"): + return self.prefetch_v_cycles return 1 @@ -319,6 +336,7 @@ def load_board_config(board: str) -> dict[str, Any]: def cycle_model_from_board(board_cfg: dict[str, Any], *, mlen: int = 64, vlen: int = 64) -> SimulatorCycleModel: """Build a SimulatorCycleModel from a board config's `latency:` section.""" lat = board_cfg["latency"] + mem = board_cfg.get("memory", {}) dc_en = 1 if lat.get("dc_lib_en", False) else 0 return SimulatorCycleModel( settings_path=Path(board_cfg.get("name", "board_config")), @@ -338,6 +356,9 @@ def cycle_model_from_board(board_cfg: dict[str, Any], *, mlen: int = 64, vlen: i scalar_fp_sqrt_cycles=lat["scalar_fp_sqrt_cycles"], scalar_fp_reci_cycles=lat["scalar_fp_reci_cycles"], scalar_int_basic_cycles=lat["scalar_int_basic_cycles"], + prefetch_v_cycles=mem.get("prefetch_v_cycles", 0), + prefetch_m_cycles=mem.get("prefetch_m_cycles", 0), + store_v_cycles=mem.get("store_v_cycles", 0), ) From 9bc69ea274523044f92051cedee03861105b1ef9 Mon Sep 17 00:00:00 2001 From: booth-algo Date: Wed, 3 Jun 2026 22:38:12 +0100 Subject: [PATCH 03/10] feat(emulator): surface streaming-model statistics through ErasedMemoryModel Adds MemoryModel::statistics_summary (default None) forwarded via ErasedMemoryModel::box_statistics_summary, overridden by WithStats (the existing HBM byte-stats line), HostStream (weight/activation stream counts), and LayerSwapping (swap count/bytes/time). runner.rs now reports stats uniformly for any --memory-model via the trait, dropping the 'statistics unavailable for streaming models' fallback; the concrete hbm_concrete handle is kept only for the DEBUG-only HBM dump (data read-back isn't on the trait). Default --memory-model hbm output is byte-for-byte unchanged (verified: Bytes read 49152 / written 0 / 5.22e9 bytes/sec, Latency 9410ns); cargo build + cargo fmt clean. --- transactional_emulator/lib/memory/src/lib.rs | 30 ++++++++ .../lib/memory/src/streaming.rs | 16 +++++ transactional_emulator/src/runner.rs | 69 ++++++++----------- 3 files changed, 73 insertions(+), 42 deletions(-) diff --git a/transactional_emulator/lib/memory/src/lib.rs b/transactional_emulator/lib/memory/src/lib.rs index 456143a6..bab8db83 100644 --- a/transactional_emulator/lib/memory/src/lib.rs +++ b/transactional_emulator/lib/memory/src/lib.rs @@ -45,12 +45,24 @@ pub trait MemoryModel: Send + Sync { /// Write 64-bytes of memory. async fn write(&self, addr: u64, bytes: [u8; 64]); + + /// Render a human-readable, per-model statistics summary, or `None` if the + /// model does not track statistics. `elapsed_secs` is the simulated wall + /// time used for bandwidth/utilization figures. + /// + /// This is intentionally a non-async default so that `trait_variant` (which + /// only rewrites async methods) passes it through unchanged, letting it be + /// forwarded uniformly via `ErasedMemoryModel`. + fn statistics_summary(&self, _elapsed_secs: f64) -> Option { + None + } } #[async_trait::async_trait] pub trait ErasedMemoryModel: Send + Sync { async fn box_read(&self, addr: u64) -> [u8; 64]; async fn box_write(&self, addr: u64, bytes: [u8; 64]); + fn box_statistics_summary(&self, elapsed_secs: f64) -> Option; } #[async_trait::async_trait] @@ -62,6 +74,10 @@ impl ErasedMemoryModel for T { async fn box_write(&self, addr: u64, bytes: [u8; 64]) { self.write(addr, bytes).await } + + fn box_statistics_summary(&self, elapsed_secs: f64) -> Option { + self.statistics_summary(elapsed_secs) + } } impl MemoryModel for dyn ErasedMemoryModel { @@ -72,6 +88,10 @@ impl MemoryModel for dyn ErasedMemoryModel { async fn write(&self, addr: u64, bytes: [u8; 64]) { self.box_write(addr, bytes).await } + + fn statistics_summary(&self, elapsed_secs: f64) -> Option { + self.box_statistics_summary(elapsed_secs) + } } /// A memory that discards all written data. @@ -207,6 +227,16 @@ impl MemoryModel for WithStats { } self.model.write(addr, bytes).await } + + fn statistics_summary(&self, elapsed_secs: f64) -> Option { + let s = self.statistics(); + Some(format!( + "HBM Statistics - Bytes read: {:?} | Bytes written: {:?} | Utilization: {:.2e} bytes/sec", + s.total_bytes_read, + s.total_bytes_written, + (s.total_bytes_read + s.total_bytes_written) as f64 / elapsed_secs + )) + } } #[cfg(test)] diff --git a/transactional_emulator/lib/memory/src/streaming.rs b/transactional_emulator/lib/memory/src/streaming.rs index f918770b..58e89f69 100644 --- a/transactional_emulator/lib/memory/src/streaming.rs +++ b/transactional_emulator/lib/memory/src/streaming.rs @@ -105,6 +105,14 @@ impl MemoryModel for HostStream { async fn write(&self, addr: u64, bytes: [u8; 64]) { self.inner.write(addr, bytes).await } + + fn statistics_summary(&self, _elapsed_secs: f64) -> Option { + let s = self.statistics(); + Some(format!( + "Host-stream memory - weight reads: {} ({} bytes) | activation reads: {} ({} bytes)", + s.weight_reads, s.weight_bytes, s.activation_reads, s.activation_bytes + )) + } } #[derive(Debug, Default)] @@ -216,4 +224,12 @@ impl MemoryModel for LayerSwapping { async fn write(&self, addr: u64, bytes: [u8; 64]) { self.inner.write(addr, bytes).await } + + fn statistics_summary(&self, _elapsed_secs: f64) -> Option { + let s = self.statistics(); + Some(format!( + "Layer-swap memory - swaps: {} | swapped bytes: {} | swap time: {} ns", + s.total_swaps, s.total_swap_bytes, s.total_swap_nanos + )) + } } diff --git a/transactional_emulator/src/runner.rs b/transactional_emulator/src/runner.rs index d5cdb8bf..4748c0f2 100644 --- a/transactional_emulator/src/runner.rs +++ b/transactional_emulator/src/runner.rs @@ -286,49 +286,34 @@ pub(crate) async fn run_from_cli() { let fpsram_bytes = accelerator.fpsram_dump_bytes(); dump_to_file("fpsram_dump.bin", &fpsram_bytes); - // Dump HBM + report statistics. + // Dump HBM (DEBUG only) + report statistics. // - // Both the HBM dump (which reads back the backing store via - // `.model().data()`) and the byte-count statistics (`.statistics()`) are - // concrete `WithStats>` methods that the erased - // `ErasedMemoryModel` trait does not expose. They are therefore gated on - // the typed `hbm_concrete` handle, which is only populated for the default - // Hbm model. For that default path this is byte-for-byte the old behavior; - // for the streaming models we log the model's own streaming statistics - // instead and skip the (model-less) HBM dump. - match hbm_concrete { - Some(ref hbm) => { - // Dump HBM — skipped unless DEBUG tracing is enabled because - // HBM_SIZE may be 128 GiB+. Tests run with --log-level warn and - // don't need hbm_dump.bin; only manual debug runs dump HBM. - if tracing::enabled!(tracing::Level::DEBUG) { - let hbm_size = effective_hbm_size; - let mut hbm_bytes = vec![0u8; hbm_size]; - hbm.model().data().with_data(|f| { - let len = std::cmp::min(hbm_size, f.len()); - hbm_bytes[..len].copy_from_slice(&f[..len]); - }); - dump_to_file("hbm_dump.bin", &hbm_bytes); - } - - let memory_stats = hbm.statistics(); - let utilization = (memory_stats.total_bytes_read + memory_stats.total_bytes_written) - as f64 - / Executor::current().now().to_secs(); - tracing::info!( - "HBM Statistics - Bytes read: {:?} | Bytes written: {:?} | Utilization: {:.2e} bytes/sec", - memory_stats.total_bytes_read, - memory_stats.total_bytes_written, - utilization - ); - } - None => { - // Streaming models track their own timing/bandwidth statistics - // internally; the unified byte-count/HBM dump is unavailable. - tracing::info!( - "Memory model: {:?} | Simulation complete (HBM byte statistics unavailable for streaming models)", - opts.memory_model - ); + // The DEBUG-only HBM dump reads back the backing store via + // `.model().data()`, a concrete `WithStats>` method that the + // erased `ErasedMemoryModel` trait does not expose. It is therefore still + // gated on the typed `hbm_concrete` handle, which is only populated for the + // default Hbm model. + if let Some(hbm) = &hbm_concrete { + // Dump HBM — skipped unless DEBUG tracing is enabled because + // HBM_SIZE may be 128 GiB+. Tests run with --log-level warn and + // don't need hbm_dump.bin; only manual debug runs dump HBM. + if tracing::enabled!(tracing::Level::DEBUG) { + let hbm_size = effective_hbm_size; + let mut hbm_bytes = vec![0u8; hbm_size]; + hbm.model().data().with_data(|f| { + let len = std::cmp::min(hbm_size, f.len()); + hbm_bytes[..len].copy_from_slice(&f[..len]); + }); + dump_to_file("hbm_dump.bin", &hbm_bytes); } } + + // Statistics are surfaced uniformly through the erased memory model. Each + // concrete model renders its own summary (the default Hbm path reproduces + // the historical "HBM Statistics - ..." line byte-for-byte); models that + // don't track statistics return `None`. + let elapsed_secs = Executor::current().now().to_secs(); + if let Some(summary) = hbm.box_statistics_summary(elapsed_secs) { + tracing::info!("{summary}"); + } } From 2bdd7cdc994538223ec7cfd01f0254951390d933 Mon Sep 17 00:00:00 2001 From: booth-algo Date: Wed, 3 Jun 2026 23:44:28 +0100 Subject: [PATCH 04/10] fix(emulator): address code-review findings on the streaming models Resolves the confirmed correctness + cleanup findings from the PR #58 review: (1) WithStats::statistics_summary now appends the inner model's summary so HostStream/LayerSwapping stats actually surface (were shadowed by the outer WithStats); the HBM path is unchanged (inner returns None). (2) HostStream's inner is now wrapped in WithTiming(ddr3) so activation reads incur DDR3 latency (matches the doc; LayerSwap already did this). (3+6) --host-bandwidth / --ddr3-capacity are now non-Option with a value_parser that rejects 0 (was a divide-by-zero panic via Some(0) slipping past unwrap_or; also removes the dead Option/unwrap_or). (4+8) bytes->nanos extracted to transfer_nanos() with a u128 intermediate, fixing u64 overflow for layers larger than ~18 GB and de-duplicating the two call sites. (5) ensure_resident now warns when a single layer's weights exceed DDR3 capacity (restores the dropped guard). (7) clarifying comment that the DDR4-2400 preset approximates the board's DDR3-1600. Default --memory-model hbm output byte-for-byte unchanged; cargo build + fmt clean; --host-bandwidth 0 now rejected at the CLI. --- transactional_emulator/Cargo.lock | 1 + transactional_emulator/lib/memory/Cargo.toml | 1 + transactional_emulator/lib/memory/src/lib.rs | 12 +++++++++-- .../lib/memory/src/streaming.rs | 21 +++++++++++++++++-- transactional_emulator/src/cli.rs | 21 ++++++++++++++++--- transactional_emulator/src/runner.rs | 13 ++++++++---- 6 files changed, 58 insertions(+), 11 deletions(-) diff --git a/transactional_emulator/Cargo.lock b/transactional_emulator/Cargo.lock index a75bc430..e051acc2 100644 --- a/transactional_emulator/Cargo.lock +++ b/transactional_emulator/Cargo.lock @@ -797,6 +797,7 @@ dependencies = [ "runtime", "serde", "tokio", + "tracing", "trait-variant", "zerocopy", ] diff --git a/transactional_emulator/lib/memory/Cargo.toml b/transactional_emulator/lib/memory/Cargo.toml index 6ff7b1bd..59b69016 100644 --- a/transactional_emulator/lib/memory/Cargo.toml +++ b/transactional_emulator/lib/memory/Cargo.toml @@ -12,6 +12,7 @@ runtime = { path = "../runtime" } tokio = { version = "1.45.1", features = ["sync"] } futures-util = "0.3.31" rand = "0.9.2" +tracing = "0.1" [dev-dependencies] tokio = { version = "1.45.1", features = ["sync", "rt-multi-thread", "macros"] } diff --git a/transactional_emulator/lib/memory/src/lib.rs b/transactional_emulator/lib/memory/src/lib.rs index bab8db83..2c14f9f0 100644 --- a/transactional_emulator/lib/memory/src/lib.rs +++ b/transactional_emulator/lib/memory/src/lib.rs @@ -230,12 +230,20 @@ impl MemoryModel for WithStats { fn statistics_summary(&self, elapsed_secs: f64) -> Option { let s = self.statistics(); - Some(format!( + let hbm_line = format!( "HBM Statistics - Bytes read: {:?} | Bytes written: {:?} | Utilization: {:.2e} bytes/sec", s.total_bytes_read, s.total_bytes_written, (s.total_bytes_read + s.total_bytes_written) as f64 / elapsed_secs - )) + ); + // `WithStats` wraps the streaming models (LayerSwapping/HostStream) in the + // runner, so forward the inner model's summary too instead of shadowing it. + // The default HBM path's inner is `WithTiming`, whose summary is `None`, so + // this returns `hbm_line` unchanged and the HBM output stays byte-for-byte. + match self.model.statistics_summary(elapsed_secs) { + Some(inner) => Some(format!("{hbm_line}\n{inner}")), + None => Some(hbm_line), + } } } diff --git a/transactional_emulator/lib/memory/src/streaming.rs b/transactional_emulator/lib/memory/src/streaming.rs index 58e89f69..4d5e41e1 100644 --- a/transactional_emulator/lib/memory/src/streaming.rs +++ b/transactional_emulator/lib/memory/src/streaming.rs @@ -5,6 +5,13 @@ use tokio::sync::Semaphore; use crate::MemoryModel; +/// Convert a byte count into a transfer time in nanoseconds at a fixed +/// bandwidth. The multiplication is done in `u128` so large layers (where +/// `bytes * 1_000_000_000` would overflow `u64`) compute correctly. +fn transfer_nanos(bytes: u64, bandwidth_bytes_per_sec: u64) -> u64 { + (bytes as u128 * 1_000_000_000u128 / bandwidth_bytes_per_sec as u128) as u64 +} + #[derive(Debug, Clone, serde::Deserialize)] pub struct WeightRegion { pub layer_id: u32, @@ -59,7 +66,7 @@ pub struct HostStream { impl HostStream { pub fn new(inner: T, manifest: WeightManifest, host_bandwidth_bytes_per_sec: u64) -> Self { - let nanos_per_chunk = 64 * 1_000_000_000 / host_bandwidth_bytes_per_sec; + let nanos_per_chunk = transfer_nanos(64, host_bandwidth_bytes_per_sec); Self { inner, manifest, @@ -195,7 +202,17 @@ impl LayerSwapping { state.used_bytes = state.used_bytes.saturating_sub(freed); } - let load_nanos = needed * 1_000_000_000 / self.swap_bandwidth_bytes_per_sec; + // A single layer that exceeds total capacity can never stay resident: + // every access reloads it. We still admit/charge it (preserving the + // timing semantics) but surface the condition so it isn't silent. + if needed > self.capacity { + tracing::warn!( + "layer {layer_id} weights ({needed} bytes) exceed DDR3 capacity ({} bytes); modelling as reload on every access", + self.capacity + ); + } + + let load_nanos = transfer_nanos(needed, self.swap_bandwidth_bytes_per_sec); state.resident_layers.push(layer_id); state.used_bytes += needed; diff --git a/transactional_emulator/src/cli.rs b/transactional_emulator/src/cli.rs index 95869b8c..d0a3501e 100644 --- a/transactional_emulator/src/cli.rs +++ b/transactional_emulator/src/cli.rs @@ -106,6 +106,21 @@ fn parse_size(s: &str) -> Result { .ok_or_else(|| format!("size {:?} overflows usize", s)) } +/// Parse a `u64` bandwidth value, rejecting zero. +/// +/// A zero bandwidth would divide-by-zero when converting bytes to a transfer +/// time, so it is rejected at the CLI rather than panicking later. +fn parse_nonzero_u64(s: &str) -> Result { + let value: u64 = s + .trim() + .parse() + .map_err(|_| format!("invalid number {:?}", s))?; + if value == 0 { + return Err("must be > 0".to_string()); + } + Ok(value) +} + #[derive(Clone, Copy, Debug, ValueEnum)] pub(crate) enum MemoryModelKind { /// Unlimited HBM with Ramulator HBM2 timing (current default) @@ -163,11 +178,11 @@ pub(crate) struct Opts { #[arg(long, value_parser = parse_size, default_value = "512M")] /// DDR3 capacity for layer-swap model. - pub(crate) ddr3_capacity: Option, + pub(crate) ddr3_capacity: usize, - #[arg(long, default_value = "60000000")] + #[arg(long, value_parser = parse_nonzero_u64, default_value = "60000000")] /// Host-to-board bandwidth in bytes/sec (default: 60 MB/s = USB 2.0 bulk). - pub(crate) host_bandwidth: Option, + pub(crate) host_bandwidth: u64, #[arg(long, value_parser = parse_size)] /// Override HBM allocation size (default: from plena_settings.toml). diff --git a/transactional_emulator/src/runner.rs b/transactional_emulator/src/runner.rs index 4748c0f2..806ad68f 100644 --- a/transactional_emulator/src/runner.rs +++ b/transactional_emulator/src/runner.rs @@ -174,13 +174,14 @@ pub(crate) async fn run_from_cli() { serde_json::from_str(&manifest_json).unwrap_or_else(|err| { panic!("failed to parse weight manifest {:?}: {err}", manifest_path) }); - let capacity = opts.ddr3_capacity.unwrap_or(512 << 20) as u64; - let bandwidth = opts.host_bandwidth.unwrap_or(60_000_000); + let capacity = opts.ddr3_capacity as u64; + let bandwidth = opts.host_bandwidth; tracing::info!( "Memory model: layer-swap (DDR3 capacity={} MB, host bandwidth={} MB/s)", capacity / (1 << 20), bandwidth / 1_000_000 ); + // DDR4-2400 preset approximating the board's DDR3-1600 timing. let ddr3_timing = memory::SimpleTiming::preset_ddr4_2400p(1); let model = Arc::new(memory::WithStats::new( memory::streaming::LayerSwapping::new( @@ -204,13 +205,17 @@ pub(crate) async fn run_from_cli() { serde_json::from_str(&manifest_json).unwrap_or_else(|err| { panic!("failed to parse weight manifest {:?}: {err}", manifest_path) }); - let bandwidth = opts.host_bandwidth.unwrap_or(60_000_000); + let bandwidth = opts.host_bandwidth; tracing::info!( "Memory model: host-stream (host bandwidth={} MB/s)", bandwidth / 1_000_000 ); + // DDR4-2400 preset approximating the board's DDR3-1600 timing. + // Activation reads go through this timing (weight reads bypass it + // via the host link), matching the HostStream struct doc. + let ddr3_timing = memory::SimpleTiming::preset_ddr4_2400p(1); let model = Arc::new(memory::WithStats::new(memory::streaming::HostStream::new( - hbm_backing, + memory::WithTiming::new(ddr3_timing, hbm_backing), manifest, bandwidth, ))); From 1400aae826b860e107a7515dc9da41520f933322 Mon Sep 17 00:00:00 2001 From: booth-algo Date: Thu, 4 Jun 2026 00:23:08 +0100 Subject: [PATCH 05/10] feat(boards): rename nexys_a7 -> nexys_video (USB-only) + add custom_a7 (PCIe Gen2 x4) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The board mislabelled nexys_a7 is actually the Nexys Video (XC7A200T, USB-only via FT2232H, no PCIe) — renamed, and its usb2_bandwidth_mbps replaced with a structured host_link {type: usb2, bandwidth_bytes_per_sec: 4750000}. Added custom_a7.yaml: same XC7A200T/512MB DDR3 but host_link {type: pcie_gen2_x4, bandwidth_bytes_per_sec: 1.5e9} (2.0 GB/s theoretical, ~1.5 GB/s sustained). Both gain capacity_bytes=512MiB for the capacity-aware memory model. Updated --board default to nexys_video and the latency preset nexys_a7_150mhz -> nexys_video_150mhz. Profiler verified on both boards; ruff clean. --- .../testbench/aten/compare/isa_analysis.py | 4 +- .../testbench/aten/configurable.py | 4 +- .../testbench/board_configs/custom_a7.yaml | 79 +++++++++++++++++++ .../{nexys_a7.yaml => nexys_video.yaml} | 17 ++-- 4 files changed, 95 insertions(+), 9 deletions(-) create mode 100644 transactional_emulator/testbench/board_configs/custom_a7.yaml rename transactional_emulator/testbench/board_configs/{nexys_a7.yaml => nexys_video.yaml} (80%) diff --git a/transactional_emulator/testbench/aten/compare/isa_analysis.py b/transactional_emulator/testbench/aten/compare/isa_analysis.py index 3fa38814..3c191be3 100644 --- a/transactional_emulator/testbench/aten/compare/isa_analysis.py +++ b/transactional_emulator/testbench/aten/compare/isa_analysis.py @@ -322,7 +322,7 @@ def analyze_asm( def load_board_config(board: str) -> dict[str, Any]: - """Load a board config YAML by name (e.g. 'nexys_a7', 'v80').""" + """Load a board config YAML by name (e.g. 'nexys_video', 'custom_a7', 'v80').""" import yaml path = _BOARD_CONFIG_DIR / f"{board}.yaml" @@ -370,7 +370,7 @@ def main() -> None: parser = argparse.ArgumentParser(description="Profile PLENA ASM cycle cost against a board config") parser.add_argument("asm_file", type=Path, help="Path to generated_asm_code.asm") parser.add_argument( - "--board", default="nexys_a7", help=f"Board config name (from board_configs/). Available: {available}" + "--board", default="nexys_video", help=f"Board config name (from board_configs/). Available: {available}" ) parser.add_argument("--mlen", type=int, default=64, help="MLEN for matrix-op cycle cost (default: 64)") parser.add_argument("--clock-mhz", type=float, default=None, help="Override clock (default: board clock_mhz)") diff --git a/transactional_emulator/testbench/aten/configurable.py b/transactional_emulator/testbench/aten/configurable.py index fbd889d5..115beb17 100644 --- a/transactional_emulator/testbench/aten/configurable.py +++ b/transactional_emulator/testbench/aten/configurable.py @@ -37,7 +37,7 @@ LATENCY_PROFILE_PRESETS: dict[str, dict[str, int]] = { # Measured on current main + khl/addr-pipeline-fix RTL, behavioral mode. # See ~/docs/plena_latency_audit_20260520.md. - "nexys_a7_150mhz": { + "nexys_video_150mhz": { "SCALAR_FP_EXP_CYCLES": 18, "SCALAR_FP_RECI_CYCLES": 6, "VECTOR_ADD_CYCLES": 11, @@ -307,7 +307,7 @@ def add_common_args(parser: argparse.ArgumentParser, *, default_build_dir: Path) parser.add_argument( "--latency-profile", default=None, - help=("Optional per-FPGA latency profile for DC_EN=0. Known preset: nexys_a7_150mhz."), + help=("Optional per-FPGA latency profile for DC_EN=0. Known preset: nexys_video_150mhz."), ) parser.add_argument("--seed", type=int, default=42) parser.add_argument("--no-run", action="store_true") diff --git a/transactional_emulator/testbench/board_configs/custom_a7.yaml b/transactional_emulator/testbench/board_configs/custom_a7.yaml new file mode 100644 index 00000000..176cefad --- /dev/null +++ b/transactional_emulator/testbench/board_configs/custom_a7.yaml @@ -0,0 +1,79 @@ +# Custom Artix-7 board — Xilinx XC7A200T, 512 MB DDR3, PCIe Gen2 x4 host link. +# Same FPGA fabric / DDR3 as the Nexys Video, but the host link is PCIe Gen2 x4 +# (~1.5 GB/s effective, ~315x the Nexys Video USB2 link) instead of USB. + +name: custom_a7 +description: Custom Artix-7 XC7A200T, 512 MB DDR3, PCIe Gen2 x4 host link +vendor: xilinx +family: artix7 +part: xc7a200t + +clock_mhz: 100 + +memory: + type: ddr3 + spec: DDR3-1600 + chip: MT41K256M16HA + bus_width: x16 + physical_size_bytes: 536870912 # 512 MiB DDR3 + capacity_bytes: 536870912 # usable DDR3 capacity for the capacity-aware memory model + modeled_size_bytes: 137438953472 # 128 GiB (emulator flat-HBM model) + bus_width_bits: 512 + peak_bandwidth_gbps: 1.6 + effective_bandwidth_gbps: 1.0 + matrix_sram_tiles: 4096 + vector_sram_depth: 4194304 + hbm_m_prefetch_amount: 64 + hbm_v_prefetch_amount: 4 + hbm_v_writeback_amount: 4 + ddr3_burst_cycles: 5 + ddr3_row_miss_penalty: 3 + prefetch_v_cycles: 25 + prefetch_m_cycles: 400 + store_v_cycles: 25 + +# Host link for streaming weights/KV that don't fit in DDR3. +# PCIe Gen2 x4: 5 GT/s/lane x 8b/10b = 500 MB/s/lane x4 = 2.0 GB/s theoretical +# (16 Gbps); realistic sustained after TLP overhead ~1.5 GB/s (~12 Gbps). +host_link: + type: pcie_gen2_x4 + bandwidth_bytes_per_sec: 1500000000 # ~1.5 GB/s sustained (theoretical 2.0 GB/s) + theoretical_bandwidth_bytes_per_sec: 2000000000 + +precision: + matrix_sram: {type: fp, sign: true, exponent: 8, mantissa: 7} # BF16 + vector_sram: {type: fp, sign: true, exponent: 8, mantissa: 7} # BF16 + hbm_act: {exponent: 4, mantissa: 3, scale_exponent: 8, block: 8} # MXFP8 E4M3 + hbm_weight: {exponent: 4, mantissa: 3, scale_exponent: 8, block: 8} + +latency: + dc_lib_en: false + systolic_processing_overhead: 0 + vector_add_cycles: 2 + vector_mul_cycles: 5 + vector_exp_cycles: 6 + vector_reci_cycles: 7 + vector_max_cycles: 4 + vector_sum_cycles: 20 + vector_shift_cycles: 1 + vector_prefix_scan_cycles: 9 + scalar_fp_basic_cycles: 1 + scalar_fp_exp_cycles: 2 + scalar_fp_sqrt_cycles: 2 + scalar_fp_reci_cycles: 2 + scalar_int_basic_cycles: 1 + +power: + tdp_watts: 7.5 + io_voltage: 3.3 + +resources: + luts: 134600 + ffs: 269200 + bram_36k: 365 + dsp48: 740 + +tile_hints: + - {mlen: 64, vlen: 64, blen: 4, batch_size: 1, status: proven} + - {mlen: 64, vlen: 64, blen: 16, batch_size: 1, status: proven} + - {mlen: 256, vlen: 256, blen: 64, batch_size: 1, status: exceeds_resources} diff --git a/transactional_emulator/testbench/board_configs/nexys_a7.yaml b/transactional_emulator/testbench/board_configs/nexys_video.yaml similarity index 80% rename from transactional_emulator/testbench/board_configs/nexys_a7.yaml rename to transactional_emulator/testbench/board_configs/nexys_video.yaml index 56ff95a4..bd3d9dcb 100644 --- a/transactional_emulator/testbench/board_configs/nexys_a7.yaml +++ b/transactional_emulator/testbench/board_configs/nexys_video.yaml @@ -1,8 +1,9 @@ # Digilent Nexys Video — Xilinx Artix-7 XC7A200T -# FPGA fabric only, no dedicated compute library (dc_lib_dis) +# FPGA fabric only, no dedicated compute library (dc_lib_dis). +# Host link is USB only (FT2232H) — NO PCIe on this board. -name: nexys_a7 -description: Artix-7 XC7A200T on Nexys Video board +name: nexys_video +description: Artix-7 XC7A200T on Digilent Nexys Video (512 MB DDR3, USB host link, no PCIe) vendor: xilinx family: artix7 part: xc7a200t @@ -15,7 +16,8 @@ memory: chip: MT41K256M16HA # single x16 chip on Nexys Video bus_width: x16 # 2 bytes per transfer physical_size_bytes: 536870912 # 512 MiB (Nexys Video onboard) - modeled_size_bytes: 137438953472 # 128 GiB (emulator model) + capacity_bytes: 536870912 # usable DDR3 capacity for the capacity-aware memory model + modeled_size_bytes: 137438953472 # 128 GiB (emulator flat-HBM model) bus_width_bits: 512 # PLENA HBM abstraction layer width peak_bandwidth_gbps: 1.6 # x16 bus: 2B × 800 MT/s effective_bandwidth_gbps: 1.0 # ~60-65% efficiency (matches Vitis AI DPU B1152 avg) @@ -34,7 +36,12 @@ memory: prefetch_v_cycles: 25 # 256B at ~1.0 GB/s effective prefetch_m_cycles: 400 # 4KB at ~1.0 GB/s effective store_v_cycles: 25 # write-back same as prefetch V - usb2_bandwidth_mbps: 38 # FT2232H practical (not theoretical 60) + +# Host link used to stream weights/KV when they do not fit in DDR3. +# Nexys Video has no PCIe — only USB2 via the FT2232H. +host_link: + type: usb2 + bandwidth_bytes_per_sec: 4750000 # FT2232H ~38 Mbps practical (not theoretical 60) = ~4.75 MB/s precision: matrix_sram: {type: fp, sign: true, exponent: 8, mantissa: 7} # BF16 From 485c9ac614e06c4321f33cce096904febfe402a4 Mon Sep 17 00:00:00 2001 From: booth-algo Date: Thu, 4 Jun 2026 00:37:01 +0100 Subject: [PATCH 06/10] feat(emulator): capacity-aware memory model with resident / kv-swap / weight-stream regimes Adds RegionKind {Weight,Kv,Activation} (serde default Weight, back-compat with the weight-only manifest) + ddr_capacity_bytes to WeightManifest, and a CapacityModel that models the three DDR-fit regimes: Resident (footprint <= capacity: everything in DDR, no host transfer), KvSwap (weights pinned in DDR, KV/activations LRU-swapped to host on capacity pressure, reload-on-access no recompute), WeightStream (weights streamed from host, HostStream semantics). choose_regime() auto-selects from footprint_by_kind vs capacity; --memory-model gains {auto,resident,kv-swap,weight-stream}. Capacity = --ddr3-capacity (or manifest ddr_capacity_bytes); host bandwidth = --host-bandwidth (board host_link). Stats surface via WithStats/FIX-1. hbm/layer-swap/host-stream unchanged; default hbm byte-for-byte identical (9410ns). Verified regime ordering resident(13454ns) < kv-swap(47457ns) < weight-stream(849689ns). --- .../lib/memory/src/streaming.rs | 274 ++++++++++++++++++ transactional_emulator/src/cli.rs | 22 +- transactional_emulator/src/runner.rs | 60 ++++ 3 files changed, 353 insertions(+), 3 deletions(-) diff --git a/transactional_emulator/lib/memory/src/streaming.rs b/transactional_emulator/lib/memory/src/streaming.rs index 4d5e41e1..f077cb73 100644 --- a/transactional_emulator/lib/memory/src/streaming.rs +++ b/transactional_emulator/lib/memory/src/streaming.rs @@ -12,17 +12,38 @@ fn transfer_nanos(bytes: u64, bandwidth_bytes_per_sec: u64) -> u64 { (bytes as u128 * 1_000_000_000u128 / bandwidth_bytes_per_sec as u128) as u64 } +/// Classification of a manifest region by the kind of tensor data it holds. +/// +/// Used by the capacity-aware [`CapacityModel`] to decide which regions are +/// pinned in DDR, swapped under a capacity bound, or streamed from the host. +/// Defaults to `Weight` so legacy weight-only manifests (which omit the field) +/// still deserialize with every region treated as a weight region. +#[derive(Debug, Clone, Copy, PartialEq, Default, serde::Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum RegionKind { + #[default] + Weight, + Kv, + Activation, +} + #[derive(Debug, Clone, serde::Deserialize)] pub struct WeightRegion { pub layer_id: u32, pub offset: u64, pub size: u64, + #[serde(default)] + pub kind: RegionKind, } #[derive(Debug, Clone, serde::Deserialize)] pub struct WeightManifest { pub activation_ceiling: u64, pub regions: Vec, + /// Optional DDR capacity (bytes) the board exposes. When present it can be + /// used as the capacity for regime auto-selection in place of a CLI flag. + #[serde(default)] + pub ddr_capacity_bytes: Option, } impl WeightManifest { @@ -41,6 +62,30 @@ impl WeightManifest { .find(|r| addr >= r.offset && addr < r.offset + r.size) .map(|r| r.layer_id) } + + /// Classify an address by the kind of the region containing it, or `None` + /// if no region covers it. + pub fn region_kind(&self, addr: u64) -> Option { + self.regions + .iter() + .find(|r| addr >= r.offset && addr < r.offset + r.size) + .map(|r| r.kind) + } + + /// Sum region sizes per kind, returning `(weight, kv, activation)` bytes. + pub fn footprint_by_kind(&self) -> (u64, u64, u64) { + let mut weight = 0u64; + let mut kv = 0u64; + let mut activation = 0u64; + for r in &self.regions { + match r.kind { + RegionKind::Weight => weight += r.size, + RegionKind::Kv => kv += r.size, + RegionKind::Activation => activation += r.size, + } + } + (weight, kv, activation) + } } #[derive(Debug, Default)] @@ -250,3 +295,232 @@ impl MemoryModel for LayerSwapping { )) } } + +/// DDR-capacity regime for a fixed-size board. +/// +/// Selected explicitly on the CLI, or automatically by [`choose_regime`] from +/// the model's footprint relative to the board's DDR capacity. +#[derive(Debug, Clone, Copy, PartialEq)] +pub enum Regime { + /// Everything fits in DDR: all kinds pinned resident, pure DDR timing. + Resident, + /// Weights fit in DDR (pinned); KV + activations are swapped under the + /// remaining capacity, charging a host transfer on each miss. + KvSwap, + /// Weights do not fit in DDR: weights stream from the host link per access; + /// KV + activations stay resident in DDR. + WeightStream, +} + +impl Regime { + fn label(self) -> &'static str { + match self { + Regime::Resident => "resident", + Regime::KvSwap => "kv-swap", + Regime::WeightStream => "weight-stream", + } + } +} + +/// Choose a DDR-capacity regime from a model's footprint. +/// +/// * `total <= capacity` → [`Regime::Resident`] (everything is pinned) +/// * else `weight <= capacity` → [`Regime::KvSwap`] (weights pinned, KV swaps) +/// * else → [`Regime::WeightStream`] (weights stream) +pub fn choose_regime(weight_bytes: u64, total_bytes: u64, capacity: u64) -> Regime { + if total_bytes <= capacity { + Regime::Resident + } else if weight_bytes <= capacity { + Regime::KvSwap + } else { + Regime::WeightStream + } +} + +#[derive(Debug, Default)] +pub struct CapacityStats { + /// Bytes served straight from DDR (pinned/resident) without a host transfer. + pub resident_bytes: u64, + /// Bytes charged as KV/activation swap-ins under the capacity bound. + pub kv_swapped_bytes: u64, + /// Number of KV/activation swap-in events (misses). + pub swap_count: u64, + /// Bytes charged as host-streamed weight chunks. + pub weight_streamed_bytes: u64, +} + +/// Swappable-region LRU residency for the KV-swap regime. +/// +/// Keyed on `(offset, size)` regions (not layer ids) so KV and activation +/// regions are admitted/evicted independently of weights. Mirrors +/// [`LayerSwapping`]'s mechanics: a miss charges `transfer_nanos(size, host_bw)` +/// and the least-recently-admitted region is evicted when over capacity. +#[derive(Debug)] +struct SwapState { + /// Resident regions as `(offset, size)`, oldest-first (LRU at index 0). + resident: Vec<(u64, u64)>, + used_bytes: u64, +} + +/// Capacity-aware memory model for a fixed-size DDR board. +/// +/// Wraps a DDR-timed inner model and applies a per-kind pinning policy chosen +/// by [`Regime`]. Data is always served from `inner`; only the extra host +/// transfer timing differs between regimes. +pub struct CapacityModel { + inner: T, + manifest: WeightManifest, + regime: Regime, + /// DDR capacity available to swappable kinds (KV/activation) in `KvSwap`: + /// `ddr_capacity - weight_footprint`, saturating at 0. + swap_capacity: u64, + host_bandwidth_bytes_per_sec: u64, + /// Per-64B host-stream chunk time for the `WeightStream` regime. + host_transfer_time_per_chunk: Duration, + /// 1-permit link serialising host streaming (mirrors `HostStream`). + host_link: Semaphore, + swap_state: Mutex, + stats: Mutex, +} + +impl CapacityModel { + pub fn new( + inner: T, + manifest: WeightManifest, + ddr_capacity_bytes: u64, + host_bandwidth_bytes_per_sec: u64, + regime: Regime, + ) -> Self { + let (weight_footprint, _kv, _act) = manifest.footprint_by_kind(); + let swap_capacity = ddr_capacity_bytes.saturating_sub(weight_footprint); + let nanos_per_chunk = transfer_nanos(64, host_bandwidth_bytes_per_sec); + Self { + inner, + manifest, + regime, + swap_capacity, + host_bandwidth_bytes_per_sec, + host_transfer_time_per_chunk: Duration::from_nanos(nanos_per_chunk), + host_link: Semaphore::new(1), + swap_state: Mutex::new(SwapState { + resident: Vec::new(), + used_bytes: 0, + }), + stats: Mutex::new(CapacityStats::default()), + } + } + + pub fn statistics(&self) -> CapacityStats { + let g = self.stats.lock().unwrap(); + CapacityStats { + resident_bytes: g.resident_bytes, + kv_swapped_bytes: g.kv_swapped_bytes, + swap_count: g.swap_count, + weight_streamed_bytes: g.weight_streamed_bytes, + } + } + + /// The `(offset, size)` of the region containing `addr`, if any. + fn region_bounds(&self, addr: u64) -> Option<(u64, u64)> { + self.manifest + .regions + .iter() + .find(|r| addr >= r.offset && addr < r.offset + r.size) + .map(|r| (r.offset, r.size)) + } + + /// Ensure the swappable region containing `addr` is resident under the + /// capacity bound. Returns the swap-in delay to charge on a miss, or `None` + /// if already resident. + fn ensure_swappable_resident(&self, addr: u64) -> Option { + let (offset, size) = self.region_bounds(addr)?; + let mut state = self.swap_state.lock().unwrap(); + if state.resident.iter().any(|&(o, _)| o == offset) { + return None; + } + + while state.used_bytes + size > self.swap_capacity && !state.resident.is_empty() { + let (_, freed) = state.resident.remove(0); + state.used_bytes = state.used_bytes.saturating_sub(freed); + } + + // A region larger than the whole swap capacity can never stay resident: + // every access reloads it. Admit/charge it anyway (preserving timing). + if size > self.swap_capacity { + tracing::warn!( + "KV/activation region at offset {offset} ({size} bytes) exceeds swap capacity ({} bytes); modelling as reload on every access", + self.swap_capacity + ); + } + + let load_nanos = transfer_nanos(size, self.host_bandwidth_bytes_per_sec); + state.resident.push((offset, size)); + state.used_bytes += size; + + { + let mut stats = self.stats.lock().unwrap(); + stats.swap_count += 1; + stats.kv_swapped_bytes += size; + } + + Some(Duration::from_nanos(load_nanos)) + } +} + +impl MemoryModel for CapacityModel { + async fn read(&self, addr: u64) -> [u8; 64] { + let kind = self.manifest.region_kind(addr); + match self.regime { + // Everything pinned in DDR: pure inner timing. + Regime::Resident => { + self.stats.lock().unwrap().resident_bytes += 64; + self.inner.read(addr).await + } + // Weights pinned; KV + activations swap under the capacity bound. + Regime::KvSwap => match kind { + Some(RegionKind::Kv) | Some(RegionKind::Activation) => { + if let Some(swap_duration) = self.ensure_swappable_resident(addr) { + let rt = runtime::Executor::current(); + rt.resolve_at(swap_duration).await; + } + self.inner.read(addr).await + } + // Weight regions / non-manifest / below ceiling → resident DDR. + _ => { + self.stats.lock().unwrap().resident_bytes += 64; + self.inner.read(addr).await + } + }, + // Weights stream from the host link; KV + activations resident. + Regime::WeightStream => match kind { + Some(RegionKind::Weight) => { + let _permit = self.host_link.acquire().await.unwrap(); + let rt = runtime::Executor::current(); + rt.resolve_at(self.host_transfer_time_per_chunk).await; + self.stats.lock().unwrap().weight_streamed_bytes += 64; + self.inner.read(addr).await + } + _ => { + self.stats.lock().unwrap().resident_bytes += 64; + self.inner.read(addr).await + } + }, + } + } + + async fn write(&self, addr: u64, bytes: [u8; 64]) { + self.inner.write(addr, bytes).await + } + + fn statistics_summary(&self, _elapsed_secs: f64) -> Option { + let s = self.statistics(); + Some(format!( + "Capacity model [{}] - resident bytes: {} | kv swapped bytes: {} (swaps: {}) | weight streamed bytes: {}", + self.regime.label(), + s.resident_bytes, + s.kv_swapped_bytes, + s.swap_count, + s.weight_streamed_bytes + )) + } +} diff --git a/transactional_emulator/src/cli.rs b/transactional_emulator/src/cli.rs index d0a3501e..1052a073 100644 --- a/transactional_emulator/src/cli.rs +++ b/transactional_emulator/src/cli.rs @@ -129,6 +129,14 @@ pub(crate) enum MemoryModelKind { LayerSwap, /// Direct host-to-SRAM streaming (no DDR3 for weights) HostStream, + /// Auto-select a capacity regime from the manifest footprint vs DDR capacity + Auto, + /// Capacity regime: everything pinned resident in DDR (pure DDR timing) + Resident, + /// Capacity regime: weights pinned, KV/activations swapped under capacity + KvSwap, + /// Capacity regime: weights stream from the host link, KV/activations resident + WeightStream, } #[derive(Parser)] @@ -167,9 +175,17 @@ pub(crate) struct Opts { #[arg(long, value_enum, default_value = "hbm")] /// Memory model to simulate. /// - /// hbm — Unlimited HBM with Ramulator timing (default) - /// layer-swap — DDR3 with layer swapping (requires --weight-manifest) - /// host-stream — Direct host-to-SRAM streaming (requires --weight-manifest) + /// hbm — Unlimited HBM with Ramulator timing (default) + /// layer-swap — DDR3 with layer swapping (requires --weight-manifest) + /// host-stream — Direct host-to-SRAM streaming (requires --weight-manifest) + /// auto — Capacity-aware: pick regime from footprint vs DDR capacity + /// resident — Capacity-aware: all kinds pinned resident in DDR + /// kv-swap — Capacity-aware: weights pinned, KV/activations swap + /// weight-stream — Capacity-aware: weights stream from host, KV/act resident + /// + /// The capacity-aware regimes (auto/resident/kv-swap/weight-stream) require + /// --weight-manifest, and use --ddr3-capacity as the board DDR capacity and + /// --host-bandwidth as the host link bandwidth. pub(crate) memory_model: MemoryModelKind, #[arg(long)] diff --git a/transactional_emulator/src/runner.rs b/transactional_emulator/src/runner.rs index 806ad68f..baa10629 100644 --- a/transactional_emulator/src/runner.rs +++ b/transactional_emulator/src/runner.rs @@ -221,6 +221,66 @@ pub(crate) async fn run_from_cli() { ))); (model, None) } + cli::MemoryModelKind::Auto + | cli::MemoryModelKind::Resident + | cli::MemoryModelKind::KvSwap + | cli::MemoryModelKind::WeightStream => { + use memory::streaming::Regime; + + let manifest_path = opts + .weight_manifest + .as_ref() + .expect("--weight-manifest required for capacity-aware memory model"); + let manifest_json = std::fs::read_to_string(manifest_path).unwrap_or_else(|err| { + panic!("failed to read weight manifest {:?}: {err}", manifest_path) + }); + let manifest: memory::streaming::WeightManifest = + serde_json::from_str(&manifest_json).unwrap_or_else(|err| { + panic!("failed to parse weight manifest {:?}: {err}", manifest_path) + }); + + // DDR capacity: prefer the manifest's value if present, else the + // CLI --ddr3-capacity flag. + let capacity = manifest + .ddr_capacity_bytes + .unwrap_or(opts.ddr3_capacity as u64); + let bandwidth = opts.host_bandwidth; + + // For Auto, pick the regime from the model footprint; the other + // modes select their regime explicitly. + let regime = match opts.memory_model { + cli::MemoryModelKind::Auto => { + let (weight, kv, activation) = manifest.footprint_by_kind(); + let total = weight + kv + activation; + memory::streaming::choose_regime(weight, total, capacity) + } + cli::MemoryModelKind::Resident => Regime::Resident, + cli::MemoryModelKind::KvSwap => Regime::KvSwap, + cli::MemoryModelKind::WeightStream => Regime::WeightStream, + _ => unreachable!("outer match restricts to capacity-aware kinds"), + }; + + tracing::info!( + "Memory model: capacity-aware [{:?}] (DDR capacity={} MB, host bandwidth={} MB/s)", + regime, + capacity / (1 << 20), + bandwidth / 1_000_000 + ); + + // DDR4-2400 preset approximating the board's DDR3-1600 timing + // (same inner DDR-timed model used by layer-swap). + let ddr3_timing = memory::SimpleTiming::preset_ddr4_2400p(1); + let model = Arc::new(memory::WithStats::new( + memory::streaming::CapacityModel::new( + memory::WithTiming::new(ddr3_timing, hbm_backing), + manifest, + capacity, + bandwidth, + regime, + ), + )); + (model, None) + } }; let mut accelerator = Accelerator::new(m_machine, v_machine, hbm.clone()); From b5e2bf6949dca10bd8e9308e1e6e5efe06cf9e57 Mon Sep 17 00:00:00 2001 From: booth-algo Date: Thu, 4 Jun 2026 01:22:26 +0100 Subject: [PATCH 07/10] feat(testbench): kind-tagged weight-manifest generator from real compiles make_weight_manifest.py turns a compile's hbm_addrs / hbm_sizes / tensor_layouts into the capacity-model WeightManifest (weight / kv / activation regions, sized in MXFP8 bytes, plus the board's DDR capacity), so the resident / kv-swap / weight-stream regimes run on a real workload instead of a synthetic manifest. run_model.py dumps the HBM region maps next to the ISA (covers --compile-only and full runs). Classifies by the real plena_frontend.py name conventions (decoder W_*, vision V_W_/V_B_/V_LN1_/V_LN2_/V_POST_LN_/V_PATCH_*/V_CONNECTOR_* weights; K_stored_/V_stored_ KV; everything else activation), with size precedence hbm_sizes -> tensor_layouts(prod*1.125) -> address-delta, overlap clamping, and a uniform-layer full-model footprint estimate. Validated on clm60m (1 layer, mlen=256): 9 weight / 4 kv / 6 activation, all sized from authoritative hbm_sizes; emulator regimes fire correctly on real addresses (weight-stream streams 4.26 MB of weight reads; kv-swap swaps 884 KB of real KV; auto -> resident). --- .../testbench/make_weight_manifest.py | 365 ++++++++++++++++++ transactional_emulator/testbench/run_model.py | 18 + 2 files changed, 383 insertions(+) create mode 100644 transactional_emulator/testbench/make_weight_manifest.py diff --git a/transactional_emulator/testbench/make_weight_manifest.py b/transactional_emulator/testbench/make_weight_manifest.py new file mode 100644 index 00000000..06559f28 --- /dev/null +++ b/transactional_emulator/testbench/make_weight_manifest.py @@ -0,0 +1,365 @@ +#!/usr/bin/env python3 +"""Build a kind-tagged weight manifest from a real PLENA compile. + +The capacity-aware memory model in the emulator (``lib/memory/src/streaming.rs``) +selects a DDR-residency regime (resident / kv-swap / weight-stream) from a +``WeightManifest`` that tags every persistent HBM region as ``weight``, ``kv`` or +``activation`` and carries the board's DDR capacity. This script turns the +artifacts a real compile drops into ```` into exactly that manifest, so +the regimes fire on a real SmolVLM2 / SmolLM2 / clm workload instead of a +synthetic stand-in. + +Inputs (written by ``run_model.py`` right after the ISA, for any --case): + * ``hbm_addrs.json`` name -> HBM base byte offset (REQUIRED) + * ``hbm_sizes.json`` name -> region bytes (authoritative; incl. KV) (preferred) + * ``tensor_layouts.json`` name -> {storage_shape, ...} for INPUT tensors (fallback) + +Sizing precedence per region: + 1. ``hbm_sizes[name]`` - authoritative; covers KV stores. + 2. ``int(prod(storage_shape) * 1.125)`` - exact for inputs/weights; misses KV. + (MXFP8 E4M3, block=8, scale_exponent=8 -> real_data_ratio = 9/8 = 1.125, + matching the compiler: program_tensors.py ``hbm_size = int(size*1.125)``.) + 3. sorted-offset address delta - last resort, contiguous prefix only. + +Classification by region name (see plena_frontend.py emission conventions): + * KV : name contains a ``_stored_`` token preceded by K or V + (K_stored_*, V_stored_*, V_K_stored_*, V_V_stored_*). + * weight : decoder matmul weights W_*; vision attention/FFN weights V_W_*; + vision biases V_B_* (uppercase B: V_B_q/k/v/o/fc1/fc2); vision + layernorm params V_LN1_*/V_LN2_*/V_POST_LN_* (UPPERCASE); and the + patch-embed/connector weights V_PATCH_W, V_PATCH_BIAS_POS, + V_CONNECTOR_W, V_CONNECTOR_B. + * activation : everything else (X, POS, COS, SIN, R_rope, causal_mask, V_PIXELS, + the post-LN output vision_post_ln, and per-layer intermediates like + K_{l}_h{kv}, O_proj_{l}, V_FC1_{l}, V_PATCH_OUT, V_CONNECTOR_OUT). + NB classify KV by the ``_stored_`` token, NOT a leading K/V: bare K_{l}_h{kv} is a + pre-store projection activation, and the V_ prefix is the *vision* namespace. + +Usage: + python testbench/make_weight_manifest.py \ + --build-dir testbench/build/clm60m_native_64x64x16_b1_decoder \ + --board testbench/board_configs/custom_a7.yaml + # optional: --total-layers 30 (extrapolate full-model footprint in the report) + # --ddr-capacity 256M (override board capacity, e.g. to force a regime) + # -o path/to/weight_manifest.json +""" + +from __future__ import annotations + +import argparse +import json +import re +import sys +from math import prod +from pathlib import Path + +# MXFP8 (E4M3, block=8, scale_exponent=8): 1 data byte/elem + 1 scale byte / 8 +# elems = 9/8 bytes per element. Hardcoded in the compiler as real_data_ratio. +REAL_DATA_RATIO = 1.125 + +_KV_RE = re.compile(r"(^|_)[KV]_stored_") +# Weight (persistent parameter) name prefixes. Decoder matmul weights are W_*; +# vision attention/FFN weights V_W_*; vision biases V_B_* (uppercase B, incl. +# V_B_q/k/v/o/fc1/fc2); vision layernorm params are UPPERCASE V_LN1_*/V_LN2_*. +_WEIGHT_PREFIXES = ("W_", "V_W_", "V_B_", "V_LN1_", "V_LN2_", "V_POST_LN_") +# Vision patch-embed + connector weights/biases (exact names: their _OUT / +# _SHUFFLED siblings are VRAM alloc intermediates, never HBM regions). NB the +# bare "vision_post_ln" is the post-LN *output* activation — the params are +# V_POST_LN_weight/bias (covered by the V_POST_LN_ prefix above). +_WEIGHT_EXACT = frozenset({"V_PATCH_W", "V_PATCH_BIAS_POS", "V_CONNECTOR_W", "V_CONNECTOR_B"}) +# Per-head suffix and digit-bearing tokens (vision fc1/fc2/ln1/ln2) that are NOT +# the layer index; stripped/neutralised before extracting the layer id. +_HEAD_SUFFIX_RE = re.compile(r"_h\d+$") +_LAYER_TOKEN_RE = re.compile(r"(?i)(fc|ln)[12]") +_FIRST_INT_RE = re.compile(r"\d+") + +# RegionKind strings the Rust deserializer accepts (#[serde(rename_all="lowercase")]). +KIND_WEIGHT = "weight" +KIND_KV = "kv" +KIND_ACTIVATION = "activation" + + +def classify(name: str) -> str: + """Map a region name to a RegionKind string.""" + if _KV_RE.search(name): # classify KV by the _stored_ token, not a leading K/V + return KIND_KV + if name in _WEIGHT_EXACT or name.startswith(_WEIGHT_PREFIXES): + return KIND_WEIGHT + return KIND_ACTIVATION + + +def layer_id(name: str) -> int: + """Layer index from the name; 0 for global/shared tensors. + + Strips the per-head suffix (_h) and neutralises digit-bearing tokens that + are not the layer (vision fc1/fc2/ln1/ln2 -> fc/ln) so the first remaining + integer is the layer index (e.g. V_W_fc1_4 -> 4, vision_l4_ln1 -> 4). + """ + base = _HEAD_SUFFIX_RE.sub("", name) + base = _LAYER_TOKEN_RE.sub(lambda m: m.group(0)[:-1], base) + m = _FIRST_INT_RE.search(base) + return int(m.group()) if m else 0 + + +def parse_size(text: str) -> int: + """Parse '512M' / '512MiB' / '256Mi' / '1G' / plain bytes -> integer bytes. + + Binary units (K/M/G = 2^10/2^20/2^30), matching the Rust --ddr3-capacity + parser and the board YAML capacity_bytes values. Rejects malformed or + non-positive sizes with a clear error rather than a deep ValueError. + """ + t = str(text).strip().upper() + if t.endswith("B"): + t = t[:-1] + if t.endswith("I"): # MiB/Mi -> M + t = t[:-1] + mult = 1 + if t.endswith("K"): + mult, t = 1 << 10, t[:-1] + elif t.endswith("M"): + mult, t = 1 << 20, t[:-1] + elif t.endswith("G"): + mult, t = 1 << 30, t[:-1] + try: + val = int(float(t) * mult) + except ValueError: + raise SystemExit(f"--ddr-capacity: cannot parse size {text!r}") + if val <= 0: + raise SystemExit(f"--ddr-capacity must be a positive size, got {text!r}") + return val + + +def load_board_capacity(board_path: Path) -> tuple[int, str]: + """Return (capacity_bytes, board_name) from a board YAML's memory section.""" + import yaml + + cfg = yaml.safe_load(board_path.read_text()) + mem = cfg.get("memory", {}) + cap = mem.get("capacity_bytes") or mem.get("physical_size_bytes") + if cap is None: + raise SystemExit(f"board {board_path} has no memory.capacity_bytes / physical_size_bytes") + return int(cap), cfg.get("name", board_path.stem) + + +def tl_size(entry: dict) -> int | None: + """Region bytes from a tensor_layouts entry: int(prod(storage_shape)*1.125).""" + shape = entry.get("storage_shape") + if not shape: + return None + return int(prod(int(d) for d in shape) * REAL_DATA_RATIO) + + +def build_regions(build_dir: Path) -> tuple[list[dict], dict]: + """Build sorted, kind-tagged regions from a build dir's HBM artifacts. + + Returns (regions, diag) where diag records which size source was used. + """ + addrs_path = build_dir / "hbm_addrs.json" + if not addrs_path.exists(): + raise SystemExit( + f"missing {addrs_path}\n" + "Recompile with run_model.py (any case) to emit hbm_addrs.json; e.g.\n" + " python testbench/run_model.py clm60m --config native_64x64x16_b1 " + "--case decoder --layers 1 --compile-only" + ) + addrs: dict[str, int] = json.loads(addrs_path.read_text()) + + sizes_path = build_dir / "hbm_sizes.json" + sizes: dict[str, int | None] = json.loads(sizes_path.read_text()) if sizes_path.exists() else {} + layouts_path = build_dir / "tensor_layouts.json" + layouts: dict[str, dict] = json.loads(layouts_path.read_text()) if layouts_path.exists() else {} + + # Sorted by offset so the delta fallback (and human-readable output) is + # stable. Skip entries without a usable integer offset (e.g. an unmapped + # input with hbm_addr None) rather than crashing the sort. + ordered = sorted( + ((n, o) for n, o in addrs.items() if isinstance(o, int)), + key=lambda kv: kv[1], + ) + dropped_offset = [n for n, o in addrs.items() if not isinstance(o, int)] + src_count = {"hbm_sizes": 0, "tensor_layouts": 0, "delta": 0} + unsized: list[str] = [] + regions: list[dict] = [] + + for idx, (name, offset) in enumerate(ordered): + # Resolve size by precedence; accept a source only if it yields a + # positive int. A None/0 from hbm_sizes must NOT shadow a usable + # tensor_layouts/delta size, so fall through on each miss. + size = None + source = None + cand = sizes.get(name) + if isinstance(cand, int) and cand > 0: + size, source = cand, "hbm_sizes" + if size is None: + entry = layouts.get(name) + cand = tl_size(entry) if entry is not None else None + if isinstance(cand, int) and cand > 0: + size, source = cand, "tensor_layouts" + if size is None and idx + 1 < len(ordered): + # contiguous-prefix address delta: an UPPER bound (stride incl. any + # padding/gap), reliable only for the never-freed prefix. + cand = ordered[idx + 1][1] - offset + if isinstance(cand, int) and cand > 0: + size, source = cand, "delta" + if size is None: + unsized.append(name) + continue + src_count[source] += 1 + regions.append( + { + "layer_id": layer_id(name), + "offset": int(offset), + "size": int(size), + "kind": classify(name), + "_name": name, # diagnostic only; stripped before writing + } + ) + + # Guard overlaps: Rust region_kind()/region_bounds() are first-match-wins on + # offset-sorted regions, so an oversized region would shadow (misclassify) + # the next. Clamp to abut the next region and record the correction. + overlaps: list[tuple] = [] + for i in range(len(regions) - 1): + end = regions[i]["offset"] + regions[i]["size"] + nxt = regions[i + 1]["offset"] + if end > nxt: + overlaps.append((regions[i]["_name"], regions[i]["size"], nxt - regions[i]["offset"])) + regions[i]["size"] = nxt - regions[i]["offset"] + + return regions, { + "src_count": src_count, + "unsized": unsized, + "dropped_offset": dropped_offset, + "overlaps": overlaps, + "n_addrs": len(addrs), + } + + +def footprint(regions: list[dict]) -> dict[str, int]: + f = {KIND_WEIGHT: 0, KIND_KV: 0, KIND_ACTIVATION: 0} + for r in regions: + f[r["kind"]] += r["size"] + return f + + +def choose_regime(weight: int, total: int, capacity: int) -> str: + """Mirror of streaming.rs::choose_regime.""" + if total <= capacity: + return "resident" + if weight <= capacity: + return "kv-swap" + return "weight-stream" + + +def fmt_mb(n: int) -> str: + return f"{n / (1 << 20):.2f} MiB" if n else "0" + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--build-dir", required=True, type=Path, help="compile output dir with hbm_addrs.json") + ap.add_argument("--board", required=True, type=Path, help="board YAML (supplies memory.capacity_bytes)") + ap.add_argument("--ddr-capacity", type=str, default=None, help="override board capacity, e.g. 256M") + ap.add_argument( + "--total-layers", type=int, default=None, help="extrapolate per-layer footprint to N layers in the report" + ) + ap.add_argument( + "-o", "--output", type=Path, default=None, help="manifest path (default: /weight_manifest.json)" + ) + args = ap.parse_args() + + regions, diag = build_regions(args.build_dir) + if not regions: + raise SystemExit("no sizeable HBM regions found") + + capacity, board_name = load_board_capacity(args.board) + if args.ddr_capacity: + capacity = parse_size(args.ddr_capacity) + + fp = footprint(regions) + weight, kv, act = fp[KIND_WEIGHT], fp[KIND_KV], fp[KIND_ACTIVATION] + total = weight + kv + act + + # activation_ceiling: legacy-only field (consumed by is_weight_addr in the + # host-stream/layer-swap models, NOT by the capacity model). Set it to the + # lowest weight offset so the legacy view still separates weights from the + # activations/embeddings laid out before them. + weight_offsets = [r["offset"] for r in regions if r["kind"] == KIND_WEIGHT] + activation_ceiling = min(weight_offsets) if weight_offsets else 0 + + out_regions = [{k: r[k] for k in ("layer_id", "offset", "size", "kind")} for r in regions] + manifest = { + "activation_ceiling": activation_ceiling, + "regions": out_regions, + "ddr_capacity_bytes": capacity, + } + + out_path = args.output or (args.build_dir / "weight_manifest.json") + out_path.write_text(json.dumps(manifest, indent=2)) + + # ---- report ----------------------------------------------------------- + by_kind = {KIND_WEIGHT: 0, KIND_KV: 0, KIND_ACTIVATION: 0} + for r in regions: + by_kind[r["kind"]] += 1 + print(f"Wrote {out_path}") + print(f" board: {board_name} (DDR capacity {fmt_mb(capacity)})") + print( + f" regions: {len(regions)} ({by_kind[KIND_WEIGHT]} weight, {by_kind[KIND_KV]} kv, {by_kind[KIND_ACTIVATION]} activation)" + ) + print( + f" sizes: {diag['src_count']['hbm_sizes']} from hbm_sizes, " + f"{diag['src_count']['tensor_layouts']} from tensor_layouts, " + f"{diag['src_count']['delta']} from address-delta" + ) + if diag["unsized"]: + print( + f" WARNING: {len(diag['unsized'])} region(s) had no size and were DROPPED (coverage hole): {diag['unsized']}" + ) + if diag["src_count"]["delta"]: + print( + f" WARNING: {diag['src_count']['delta']} region(s) sized by address-delta (upper bound; footprint may be inflated). " + "Recompile with an hbm_sizes-emitting compiler for exact sizes." + ) + if diag.get("overlaps"): + print(f" WARNING: clamped {len(diag['overlaps'])} overlapping region(s) to abut the next: {diag['overlaps']}") + if diag.get("dropped_offset"): + print( + f" WARNING: {len(diag['dropped_offset'])} region(s) had no integer offset and were skipped: {diag['dropped_offset']}" + ) + print( + f" footprint (this build): weight {fmt_mb(weight)} | kv {fmt_mb(kv)} | activation {fmt_mb(act)} | total {fmt_mb(total)}" + ) + print(f" regime @ {fmt_mb(capacity)} (this build): {choose_regime(weight, total, capacity)}") + + if args.total_layers: + n = args.total_layers + # Uniform-layer extrapolation for the REPORT only (the manifest keeps the + # real regions). Per-layer weight/kv = this build's weight/kv divided by + # the number of distinct layers actually present, so a multi-layer + # (--layers N) build extrapolates correctly. Shared globals (embeddings, + # rope tables) and per-layer activation intermediates are NOT modelled, so + # the activation term is a floor, not the full-model activation footprint. + layers_in_build = len({r["layer_id"] for r in regions if r["kind"] in (KIND_WEIGHT, KIND_KV)}) or 1 + per_layer_w = weight / layers_in_build + per_layer_kv = kv / layers_in_build + est_w = int(per_layer_w * n) + est_kv = int(per_layer_kv * n) + est_total = est_w + est_kv + act + print( + f" --- estimated full model ({n} layers; build holds {layers_in_build}; embeddings/lm_head + per-layer activations excluded) ---" + ) + print( + f" footprint (est): weight {fmt_mb(est_w)} | kv {fmt_mb(est_kv)} | activation>={fmt_mb(act)} | total>={fmt_mb(est_total)}" + ) + regime = choose_regime(est_w, est_total, capacity) + print(f" regime @ {fmt_mb(capacity)} (est full model): {regime}") + print( + f" (manifest contains the REAL build regions; to reproduce the full-model regime\n" + f" on this build, run the emulator with --memory-model {regime})" + ) + + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/transactional_emulator/testbench/run_model.py b/transactional_emulator/testbench/run_model.py index b8e4f9dd..4263c87f 100644 --- a/transactional_emulator/testbench/run_model.py +++ b/transactional_emulator/testbench/run_model.py @@ -296,6 +296,24 @@ def main(): print(f"\nASM written to: {asm_path}") print(f"ISA lines: {len(result['isa'].splitlines())}") + # Dump the HBM region map so the weight-manifest generator + # (testbench/board_configs/make_weight_manifest.py) can build a kind-tagged + # capacity-model manifest from a real compile. hbm_addrs (name->offset) is + # always present; hbm_sizes (name->bytes, incl. KV stores) is emitted by + # newer compilers and is the authoritative size source; tensor_layouts + # (input tensors only) is the fallback for sizing. + import json as _json + + for _key, _fname in ( + ("hbm_addrs", "hbm_addrs.json"), + ("hbm_sizes", "hbm_sizes.json"), + ("tensor_layouts", "tensor_layouts.json"), + ): + _data = result.get(_key) + if _data is not None: + (build_dir / _fname).write_text(_json.dumps(_data, indent=2, sort_keys=True)) + print(f"{_key} written to: {build_dir / _fname}") + if args.compile_only: print("\n[compile-only] Skipping emulator run.") return From b8318c6a2df6e7dbf067469d8f1642d8d420cfa3 Mon Sep 17 00:00:00 2001 From: booth-algo Date: Thu, 4 Jun 2026 01:53:04 +0100 Subject: [PATCH 08/10] chore: keep streaming/profile working docs local-only (doc/_local) These are AI-authored working references, not shipped docs; move them out of tracking into gitignored doc/_local/ (consistent with #89) so this PR carries only code. --- doc/SMOLVLM2_ISA_PROFILE.md | 268 -------------------------- doc/memory-footprint-and-streaming.md | 161 ---------------- 2 files changed, 429 deletions(-) delete mode 100644 doc/SMOLVLM2_ISA_PROFILE.md delete mode 100644 doc/memory-footprint-and-streaming.md diff --git a/doc/SMOLVLM2_ISA_PROFILE.md b/doc/SMOLVLM2_ISA_PROFILE.md deleted file mode 100644 index 21abc3a1..00000000 --- a/doc/SMOLVLM2_ISA_PROFILE.md +++ /dev/null @@ -1,268 +0,0 @@ -# SmolVLM2-256M ISA Cycle Profile - -**Config:** 12L vision encoder + connector + 1L text decoder, mlen=256, blen=64, seq=256 -**Board:** Nexys Video — Artix-7 XC7A200T @ 100MHz, DDR3-1600 x16 (1.6 GB/s peak, ~1.0 GB/s effective) -**Note:** mlen=256 exceeds Artix-7 synthesis resources. This profile is a what-if analysis. Realisable configs use mlen=64 (proven) or mlen=128 (untested). -**ASM source:** `transactional_emulator/testbench/conv/build/smolvlm2_native_module/` -**Profiler:** `transactional_emulator/testbench/aten/compare/asm_profiler.py --board nexys_a7 --mlen 256` -**Date:** 2026-05-28 (corrected from 05-27: x16 bus width, validated against published references) - -## Summary - -| Stage | ISA lines | Dynamic instrs | Est cycles | Est time @100MHz | -|-------|-----------|----------------|-----------|------------------| -| Vision + Connector (12L) | 1,444,035 | 5,580,752 | 28.0M | **280 ms** | -| Text Decoder (1L) | 1,206,857 | 2,638,488 | 17.5M | **175 ms** | -| **Total VLM e2e** | 2,650,892 | 8,219,240 | 45.5M | **455 ms** | - -## Vision + Connector (12L vision + connector) - -**28.0M cycles @ Nexys A7 (dc_lib_dis, DDR3)** - -### Per-opcode breakdown (top 15) - -| Op | Dynamic count | Cyc/op | Total cycles | % | -|----|--------------|--------|-------------|---| -| M_MM | 44,496 | 256 | 11,390,976 | 40.7% | -| V_RED_SUM | 271,872 | 20 | 5,437,440 | 19.5% | -| S_ADDI_INT | 2,054,228 | 1 | 2,054,228 | 7.3% | -| V_MUL_VF | 297,216 | 5 | 1,486,080 | 5.3% | -| V_MUL_VV | 271,872 | 5 | 1,359,360 | 4.9% | -| H_PREFETCH_M | 3,033 | 400 | 1,213,200 | 4.3% | -| C_LOOP_END | 766,161 | 1 | 766,161 | 2.7% | -| V_ADD_VV | 343,296 | 2 | 686,592 | 2.5% | -| M_TMM | 2,304 | 256 | 589,824 | 2.1% | -| V_EXP_V | 73,728 | 6 | 442,368 | 1.6% | -| S_ST_FP | 381,125 | 1 | 381,125 | 1.4% | -| S_ADD_FP | 326,450 | 1 | 326,450 | 1.2% | -| H_PREFETCH_V | 15,492 | 20 | 309,840 | 1.1% | -| V_RECI_V | 36,864 | 7 | 258,048 | 0.9% | -| S_MAP_V_FP | 832 | 256 | 212,992 | 0.8% | - -### Category summary - -| Category | Cycles | % | -|----------|--------|---| -| **Matrix compute** | 11,998,272 | **42.9%** | -| **Vector compute** | 10,076,928 | **36.0%** | -| Scalar overhead | 3,537,049 | 12.7% | -| DDR3 memory | 1,546,080 | **5.5%** | -| Control | 796,938 | 2.9% | - -## Text Decoder (1L) - -**17.5M cycles @ Nexys A7 (dc_lib_dis, DDR3)** - -### Per-opcode breakdown (top 15) - -| Op | Dynamic count | Cyc/op | Total cycles | % | -|----|--------------|--------|-------------|---| -| M_MM | 32,040 | 256 | 8,202,240 | 46.9% | -| S_MAP_V_FP | 7,680 | 256 | 1,966,080 | 11.2% | -| H_PREFETCH_M | 3,240 | 400 | 1,296,000 | 7.4% | -| V_RED_SUM | 63,552 | 20 | 1,271,040 | 7.3% | -| S_ADDI_INT | 875,152 | 1 | 875,152 | 5.0% | -| V_MUL_VV | 161,472 | 5 | 807,360 | 4.6% | -| V_MUL_VF | 138,432 | 5 | 692,160 | 4.0% | -| V_EXP_V | 63,360 | 6 | 380,160 | 2.2% | -| V_RECI_V | 46,080 | 7 | 322,560 | 1.8% | -| C_LOOP_END | 267,883 | 1 | 267,883 | 1.5% | -| V_ADD_VV | 103,872 | 2 | 207,744 | 1.2% | -| S_LUI_INT | 201,149 | 1 | 201,149 | 1.1% | -| S_ST_FP | 197,760 | 1 | 197,760 | 1.1% | -| V_SUB_VF | 63,360 | 2 | 126,720 | 0.7% | -| V_ADD_VF | 46,080 | 2 | 92,160 | 0.5% | - -### Category summary - -| Category | Cycles | % | -|----------|--------|---| -| **Matrix compute** | 8,235,810 | **47.1%** | -| **Vector compute** | 4,055,424 | **23.2%** | -| Scalar overhead | 3,615,382 | 20.7% | -| DDR3 memory | 1,310,640 | **7.5%** | -| Control | 279,514 | 1.6% | - -## DDR3 Memory Impact - -The Nexys Video uses a single **MT41K256M16HA** DDR3-1600 chip with a **x16 bus** (2 bytes/transfer). - -| Parameter | Value | Notes | -|-----------|-------|-------| -| Chip | MT41K256M16HA | Single x16, 512 MiB | -| Peak bandwidth | **1.6 GB/s** | 2B × 800 MT/s (x16 bus) | -| Effective bandwidth | **~1.0 GB/s** | ~60-65% MIG efficiency for sequential streaming | -| MIG read latency | ~27.7 cycles | ZipCPU measured on Kintex-7 MIG | -| Burst latency | ~5 PLENA cycles (50ns) | CAS + transfer for 64B | -| Row miss penalty | +3 cycles (30ns) | tRP+tRCD = 27.5ns | -| H_PREFETCH_V cost | 25 cycles | 256B / 1.0 GB/s = 256ns | -| H_PREFETCH_M cost | 400 cycles | 4KB / 1.0 GB/s = 4000ns | - -| Stage | DDR3 cycles | % of total | Static prefetch count | -|-------|------------|-----------|---------------------| -| Vision+Connector | 1,629,300 | **5.8%** | 15,492 V + 3,033 M + 288 store | -| Text Decoder | 1,314,300 | **7.5%** | 5 V + 3,240 M + 180 store | - -The text decoder is **more DDR3-bound** (7.5%) despite fewer total prefetches because M_PREFETCH dominates (3,240 × 400 = 1.3M cycles) — each matrix prefetch loads 4KB of weights from DDR3. - -## Per-Kernel Breakdown - -### Vision + Connector (12L) — 280 ms - -| Kernel | Cycles | % | ms | Description | -|--------|--------|---|----|-------------| -| **Attention** | 13.5M | **48.4%** | 135 | Q/K/V projections + flash attention × 12 layers | -| **Layer Norm** | 8.4M | **30.0%** | 84 | RMS norm × 25 (2 per layer + final). V_RED_SUM (20 cyc) + V_MUL_VV (5 cyc) per element | -| **Conv2d im2col** | 6.0M | **21.4%** | 60 | SigLIP K=16 patch embedding via vector reductions (no matrix unit) | -| Connector | 0.07M | 0.2% | 0.7 | Linear projection 12288→576 | - -### Text Decoder (1L) — 182 ms - -| Kernel | Cycles | % | ms | Description | -|--------|--------|---|----|-------------| -| **FFN** | 8.7M | **48.0%** | 88 | Gate+Up linear (M_MM) + SiLU + Down linear (M_MM) | -| **Attention** | 5.3M | **29.1%** | 53 | Packed GQA (3 heads × online softmax + P@V via M_MM) | -| **Q/K/V Projection** | 2.5M | **13.8%** | 25 | Weight load from DDR3 + M_MM for Q/K/V/O | -| **RMS Norm** | 1.6M | **8.6%** | 16 | 2× per layer + final norm | -| Layer setup | 0.07M | 0.4% | 0.7 | VRAM zero-fill, residual copy | - -### Observations - -1. **Vision is attention-dominated (48%)** — 12 layers of self-attention with head projections dominate -2. **Vision layer norm is 30%** — 25 norm operations at 3.4ms each, driven by expensive V_RED_SUM (20 cycles on Artix-7) -3. **Conv2d im2col is 21% of vision** — patch embedding done entirely with vector ops, matrix unit idle -4. **Text decoder is FFN-dominated (48%)** — 3 large M_MM operations (gate, up, down projections: 576→1536→576) -5. **QKV projection is 14% of text** — weight loading from DDR3 is a significant fraction -6. **Connector is trivial (0.2%)** — single linear projection, not a bottleneck - -## Optimization Opportunities - -Ranked by estimated cycle savings on Nexys A7 (with DDR3): - -| # | Target | Cycles saved | % of total | Approach | -|---|--------|-------------|------------|----------| -| 1 | **V_RED_SUM in vision im2col** | ~5.4M | 11.9% | Conv2d patch embedding uses V_RED_SUM (271K × 20 cyc) for dot products. Restructure im2col to gather patches into matrix SRAM and dispatch M_MM. | -| 2 | **DDR3 prefetch stalls** | ~2.9M | 6.3% | 6,273 M_PREFETCH × 400 cyc. Prefetch scheduling, double-buffering, or weight caching in BRAM could hide latency. | -| 3 | **S_ADDI_INT loop overhead** | ~2.9M | 6.4% | Loop counter increments (2.9M calls). Hardware auto-increment or compiler loop fusion. | -| 4 | **S_MAP_V_FP in text decoder** | ~2.0M | 4.4% | 7,680 scalar→vector broadcasts × 256 cyc. Vector fill instruction or register caching. | -| 5 | **V_MUL_VV + V_MUL_VF** | ~3.5M | 7.7% | 509K calls combined. Fused multiply-accumulate (V_MAC) would halve cycles. | -| 6 | **Softmax chain** | ~1.5M | 3.3% | V_EXP + V_RECI + V_RED_MAX + V_SUB form softmax. Fused instruction reduces pipeline stalls. | - -### Conv2d analysis - -The vision encoder's conv2d (SigLIP patch embedding, K=16, C=3) compiles via im2col → vector reductions. The matrix unit handles linear projections inside transformer layers (M_MM at 40.7%) but **not the initial patch convolution**. - -Restructuring im2col to fill matrix SRAM and dispatch M_MM would reduce patch embedding from ~5.4M cycles to ~100K cycles — a **17% improvement** in vision encoder time. - -### DDR3 vs HBM analysis - -On V80 with HBM2e (819 GB/s), memory prefetch is effectively free (pipelined). On Nexys A7 with DDR3, memory accounts for 5-7% of cycles. For memory-bound workloads (e.g. weight loading for large models), DDR3 becomes the primary bottleneck — this profile is compute-bound because SmolVLM2-256M is a small model. - -## Memory Footprint vs Nexys Video DDR3 - -### Actual HBM binary sizes (MXFP8 quantised weights + activations) - -| Stage | HBM binary | Notes | -|-------|-----------|-------| -| Vision + Connector (12L) | 295 MB | 12 vision layers + connector weights | -| Text Decoder (1L) | 205 MB | 1/30 decoder layers | -| **Combined (1v+1t)** | **500 MB** | Fits in 512 MB DDR3 (barely) | -| Text Decoder (30L, estimated) | ~6 GB | Full 30-layer decoder | - -### The 512 MB DDR3 problem - -The Nexys Video board has **512 MB DDR3**. SmolVLM2-256M at MXFP8 quantisation: -- 12L vision + connector: 295 MB -- 1L text decoder: 205 MB → 30L would be ~6 GB - -A single VLM e2e pass (12L vision + 1L text) barely fits at 500 MB. **Full 30-layer -text decoding is impossible** without weight streaming — loading decoder layers one -at a time from host, running each layer, then replacing it with the next. - -### Implications for the emulator - -The Rust transactional emulator currently models HBM as a flat address space -(configurable size, default 16 GB). It does not model: - -1. **DDR3 capacity limits** — no OOM when allocations exceed 512 MB -2. **Weight streaming** — no mechanism to swap layer weights in/out during execution -3. **DDR3 bank conflicts** — prefetch latency is modeled as constant, not address-dependent -4. **Refresh cycles** — DDR3 refresh steals bandwidth periodically - -To model the Nexys Video memory constraint accurately, the emulator would need: -- A configurable HBM capacity cap (512 MB) that raises an error on overflow -- A weight-streaming ISA extension or runtime mechanism to load weights per-layer -- Bank-conflict-aware prefetch latency (address → bank → row hit/miss → cycles) - -For now, the cycle profile assumes all weights fit in DDR3. The 1L text decoder -config is realistic for the Nexys Video; the 30L config is V80-only. - -## Validation - -### Static profiler vs Rust emulator - -The ASM profiler was validated against the Rust transactional emulator (Ramulator HBM2 timing, dc_en=1) on the linear test (64×128 × 128×64 matmul, mlen=64): - -| Source | Cycles | -|--------|--------| -| Rust emulator (Ramulator HBM2) | 146,935 | -| Static profiler (dc_lib_en) | 144,938 | -| **Difference** | **1.4%** | - -### DDR3 bandwidth cross-references - -| Source | Bandwidth | Notes | -|--------|-----------|-------| -| **Our model** | 1.0 GB/s effective | x16 DDR3-1600, ~60-65% efficiency | -| Vitis AI DPU B1152 (PG338) | 684-1017 MB/s avg | Production DDR3/DDR4 accelerator | -| Flare on XC7A100T (PMC 2024) | 3.125 GB/s | x32 bus (2× our x16) | -| ultraembedded DDR3 on Arty A7 | 400 MB/s | Open-source controller @100MHz | -| ZipCPU MIG memtest (2025) | 27.7 cyc avg read | MIG controller latency measured | -| Gemmini (DAC 2021) | 8 bytes/cycle | LPDDR4 model for ASIC | - -Our 1.0 GB/s effective matches the Vitis AI DPU B1152 average range (684-1017 MB/s). The 400 cycles per 4KB M_PREFETCH derives from 4KB / 1.0 GB/s = 4μs = 400 cycles @100MHz. - -### Weight streaming architecture validation - -Published FPGA accelerators use the same DDR layer-by-layer streaming pattern as our LayerSwapping model: - -| System | Architecture | Reference | -|--------|-------------|-----------| -| Apache TVM VTA | Tiled DRAM-to-SRAM DMA, access-execute decoupled | arXiv 1807.04188 | -| Vitis AI DPU | Layer-sequential DDR streaming via micro-coded dispatch | AMD PG338 | -| FINN | On-chip default; `load_external_weights()` for DDR streaming | finn.readthedocs.io | -| DNNWeaver | DDR3 bandwidth as tiling constraint | MICRO 2016 | -| AutoWS | Automated on-chip/DDR weight partitioning for layer-pipelined | arXiv 2311.04764 | -| FlightLLM | HBM weight streaming at 65.9% utilization | FPGA 2024 | - -Host-to-SRAM direct streaming (our HostStream model) is **not standard practice** for real-time inference — USB 2.0 at 38 MB/s is too slow. It is valid for modeling what-if scenarios (e.g., PCIe at 2 GB/s) and batch/offline inference. - -### Weight streaming emulator validation - -All 3 memory models produce **identical VRAM output** (verified via binary diff). Only simulated latency differs. Tested on the ATen linear test (64×128 × 128×64 matmul, mlen=64): - -| Model | CLI flag | Latency (ns) | Slowdown | Notes | -|-------|----------|-------------|----------|-------| -| HBM (Ramulator) | `--memory-model hbm` | 146,935 | 1.0x | Pipelined HBM2, sub-cycle prefetch | -| LayerSwap (DDR3) | `--memory-model layer-swap` | 1,120,727 | **7.6x** | DDR3 SimpleTiming replaces HBM2 — no swap penalty (46KB fits 512MB), slowdown is pure DDR3 vs HBM timing | -| HostStream (USB) | `--memory-model host-stream` | 1,869,354 | **12.7x** | 38 MB/s serialized — each 64B weight chunk costs 1,684ns | - -The 7.6x LayerSwap slowdown represents the **DDR3 vs HBM access latency gap** — this is the real cost of running on Nexys Video DDR3 instead of V80 HBM. The 12.7x HostStream shows the additional penalty of streaming weights from host over USB. - -## Reproducing - -```bash -# Profile with Nexys A7 DDR3 timing -python3 transactional_emulator/testbench/aten/compare/asm_profiler.py \ - transactional_emulator/testbench/conv/build/smolvlm2_native_module/vision_connector/generated_asm_code.asm \ - --board nexys_a7 --mlen 256 - -python3 transactional_emulator/testbench/aten/compare/asm_profiler.py \ - transactional_emulator/testbench/conv/build/smolvlm2_native_module/text_decoder/generated_asm_code.asm \ - --board nexys_a7 --mlen 256 - -# Compare with V80 (HBM, 400MHz) -python3 transactional_emulator/testbench/aten/compare/asm_profiler.py \ - transactional_emulator/testbench/conv/build/smolvlm2_native_module/text_decoder/generated_asm_code.asm \ - --board v80 --mlen 256 -``` diff --git a/doc/memory-footprint-and-streaming.md b/doc/memory-footprint-and-streaming.md deleted file mode 100644 index e10cbb63..00000000 --- a/doc/memory-footprint-and-streaming.md +++ /dev/null @@ -1,161 +0,0 @@ -# Memory Footprint & Host-Link Streaming Analysis - -Reasoning notes on whether SmolVLM2-256M needs weight streaming on the PLENA -target boards, and how local memory (DDR3 / HBM) compares to the host link -(USB2 / PCIe). Summary up front: **for SmolVLM2-256M at 8-bit, everything fits -in 512 MB of onboard DDR — keep weights resident and skip weight streaming.** - -## 1. Does SmolVLM2-256M fit in 512 MB? - -Yes, comfortably. - -| Component | Size (MXFP8, 8-bit) | Notes | -|-----------|---------------------|-------| -| Weights | **~288 MB** | 256M params × (8 bits/elem + 8-bit scale per block-of-8 ≈ +12.5%) = ~1.125 B/elem | -| KV cache | tens of MB | ~27 MB @ 1K tokens, ~106 MB @ 4K; sub-MB for the sub-64 bring-up | -| Activations | single-MB / layer | small | -| **Total** | **~300–400 MB** | fits in 512 MB with ~120–200 MB headroom | - -**The "~1 GB" you might observe on a GPU is not the model — it's CUDA -overhead:** the CUDA context (~300–500 MB), cuBLAS/cuDNN workspaces, allocator -fragmentation, and framework tensors. PLENA has none of that. (For reference, -fp32 weights *would* be ~1 GB — 256M × 4 B — so int8 is a 4× cut.) - -At long context the **KV cache** — not the weights — is the only thing that -eventually pressures the 512 MB ceiling. - -## 2. Implication: the streaming models are not for this model - -`transactional_emulator/lib/memory/src/streaming.rs` provides two models, both -designed for the case where **weights do not fit in DDR**: - -- **`HostStream`** — weight reads bypass DDR and pay a fixed host-transfer cost - (stream weights host→SRAM). -- **`LayerSwapping`** — capacity-limited DDR; a read to a non-resident layer - pays a swap-from-host penalty (read-only weights, so eviction is free). - -Since SmolVLM2-256M fits in 512 MB, you use **neither** — you use the plain -**resident-DDR** model: load all weights into DDR once, no streaming, no -swapping. The streaming models remain valuable for models that genuinely exceed -DDR (larger models, or future targets), but they are unnecessary here. - -## 3. Instructions: load once, don't stream - -The ISA program is small (~5–20 MB for sub-64 SmolVLM2) and **fixed/reused -across the decode loop**. With ~200 MB of DDR free after weights, load the -program **once** into DDR and fetch it locally. The host link's only job is the -one-time load (program + weights + input) and the output readback; during -compute it is idle. Continuously fetching instructions from the host adds stall -risk for zero benefit. - -## 4. DDR3 vs the host link (PCIe / USB) - -First, a framing point: **DDR3 is attached memory; PCIe is a transport** to host -RAM. A "PCIe fetch" pays the PCIe link latency **plus** the host's own DRAM -latency on the far end — they are not the same kind of thing. - -### Raw bandwidth (config-dependent) - -| Link | Bandwidth | -|------|-----------| -| DDR3 single ×16 chip (Nexys-class) | ~3.2 GB/s peak (DDR3-1600, 16-bit); ~1 GB/s effective | -| DDR3 desktop DIMM (64-bit channel) | ~12.8 GB/s/channel (25.6 dual-channel) | -| PCIe Gen3 ×16 | ~16 GB/s | -| PCIe Gen4 ×16 | ~32 GB/s | -| PCIe Gen5 ×16 | ~64 GB/s | - -On raw bandwidth a *single low-end DDR3 chip* is slower than a modern PCIe ×16 -link; a *full DDR3 DIMM* is comparable; and wide **HBM (V80: 819 GB/s) dwarfs** -any PCIe. - -### Latency (local memory always wins) - -- DDR3 access: **~15–50 ns** (directly attached). -- PCIe round-trip: **~0.5–1 µs+** (serialization + protocol + host memory - controller) — roughly **10–50× higher latency**. - -For random / latency-sensitive access (weight & activation fetch during -compute), local DDR3 wins regardless of the bandwidth headline. PCIe is good for -**bulk one-time loads**, not for being in the per-cycle fetch path. - -## 5. For the PLENA target boards specifically - -| Board | Local memory | Host link | Verdict | -|-------|--------------|-----------|---------| -| **Nexys Video** (Artix-7) | DDR3 ~1–3.2 GB/s | **USB2 ~38 MB/s** (FT2232H) — *no PCIe* | local DDR3 **~30–80× faster** + far lower latency | -| **V80** (Versal) | HBM **819 GB/s** | PCIe Gen5 ×16 ~64 GB/s | HBM **~13× faster** | -| **Custom Artix-7 + PCIe 2.0 ×4** | DDR3 ~1–3.2 GB/s | **PCIe 2.0 ×4 ~1.6–1.8 GB/s** | *comparable* — streaming becomes viable (§5.1) | - -For **Nexys Video** and **V80** the local memory beats the host link on bandwidth -*and* latency — which is exactly why "keep weights resident, don't stream from -host" is the right call there. (Note: the host link on Nexys Video is **USB2, not -PCIe**; PCIe is the V80 path, and the V80's 32 GB HBM makes the whole model a -rounding error.) The custom Artix-7 board below is the interesting exception. - -### 5.1 Custom Artix-7 with PCIe 2.0 ×4 — streaming flips to viable - -Put the *same* Artix-7 on a custom board with a **PCIe 2.0 ×4** host link and the -calculus changes: - -- **The link:** PCIe 2.0 = 5 GT/s/lane; ×4 = **20 GT/s raw** ("20 Gbps"). After - 8b/10b encoding (×0.8) → **16 Gbps = 2.0 GB/s** usable; after TLP/protocol - overhead (~10–20%) → **~1.6–1.8 GB/s effective payload.** -- **vs local DDR3** (single ×16 DDR3-1600: ~3.2 GB/s peak, ~1–2 GB/s effective): - PCIe 2.0 ×4 is now **at parity with — or faster than — the local DDR3.** That is - ~40× the USB2 figure; the host link is no longer the weak point. - -What that unlocks: - -1. **Weight streaming over PCIe becomes viable — even preferable.** The - `HostStream` model (stream weights host→SRAM, bypassing DDR3) runs at ~1.6 GB/s, - matching or beating the modeled DDR3 effective (~1.0 GB/s). On this board the - streaming models (`HostStream` / `LayerSwapping`) are the *right* tool — the - opposite of the USB2 verdict. -2. **Capacity is the real win:** streaming from host RAM (GBs) removes the 512 MB - ceiling → run models **larger than 512 MB** on this Artix-7. (SmolVLM2-256M - doesn't need it; a larger LM would.) -3. **Best-of-both split:** weights (read-only, sequential, prefetchable) streamed - over PCIe; activations/KV (read-write, random) in local DDR3. - -The one caveat is **latency, not bandwidth:** PCIe round-trip ~µs vs DDR3 ~tens of -ns. Hide it with **weight-prefetch FIFOs** — weight access is sequential, so this -works well (the board config already exposes `hbm_m_prefetch_amount` / prefetch -knobs). Latency would only bite random access, which weights are not. - -**Perf sketch:** streaming 288 MB at ~1.6 GB/s ≈ 180 ms/token → ~5–6 tok/s, vs -~3–4 tok/s for DDR3-resident (at the config's 1.0 GB/s) — a modest speedup *plus* -the bigger-model capability. Stream the **weights** (large, sequential, -bottlenecking), not the instructions (still tiny → load once into DDR). - -## 6. The real bottleneck is DDR bandwidth - -On Nexys, effective DDR ≈ **1.0 GB/s**, so reading ~288 MB of weights per token -≈ **~290 ms → ~3–4 tok/s**. That is the performance wall, *not* the host link. -For contrast, streaming those weights from the host over USB2 (38 MB/s) would be -**~7.6 s/token — ~25× worse** — which is the quantitative reason resident DDR is -the only sane choice. (On V80, HBM at 819 GB/s removes the wall entirely.) - -## 7. Caveat + how to validate in the emulator - -The emulator currently **models DDR as 128 GiB** (`modeled_size_bytes` in -`transactional_emulator/testbench/board_configs/nexys_a7.yaml`), not the physical -512 MB — so it does **not** enforce the real capacity today. - -To turn the "fits in 512 MB" argument into a measured result, run SmolVLM2-256M -through **`LayerSwapping` with `capacity = 512 MB`** and confirm **zero swaps** -(every layer loads once and never evicts). If it ever *does* swap at some -context length, that is the KV cache hitting the ceiling — the genuinely -interesting limit. - -## Bottom line - -For SmolVLM2-256M at 8-bit: keep everything resident in 512 MB DDR and skip -weight streaming. Caveats: (a) the Nexys host link is **USB2, not PCIe**; -(b) load the instruction program into DDR once rather than streaming it; and -(c) the performance wall is **DDR bandwidth (~3–4 tok/s on Nexys)**, not the -host link. - -The exception is a **custom Artix-7 board with PCIe 2.0 ×4** (§5.1): there the -host link (~1.6 GB/s) reaches parity with local DDR3, so weight streaming -becomes viable — and worthwhile mainly because it lifts the 512 MB capacity -ceiling (run from host RAM), with weights over PCIe and activations/KV in DDR3. From e4231d0320c8bad85aa5599d3085e2895219008b Mon Sep 17 00:00:00 2001 From: booth-algo Date: Thu, 4 Jun 2026 15:53:35 +0100 Subject: [PATCH 09/10] feat(testbench): apply the selected board's latency to the emulator Add --board to run_model.py: load the board YAML and write its latency: block into the per-build plena_settings.toml -- DC_EN from the board's dc_lib_en flag, and each per-op cycle cost into the column DC_EN selects (dc_lib_en when enabled, else dc_lib_dis). Previously run_model.py passed dc_en=None, so the emulator ran on the DC-library defaults (DC_EN=1) for every build regardless of board; now it runs on the board's actual per-op latencies. The board's memory: cycle params still feed the analytic profiler, not the emulator's memory model. --- .../testbench/aten/configurable.py | 56 +++++++++++++++++++ transactional_emulator/testbench/run_model.py | 33 ++++++++++- 2 files changed, 86 insertions(+), 3 deletions(-) diff --git a/transactional_emulator/testbench/aten/configurable.py b/transactional_emulator/testbench/aten/configurable.py index 115beb17..5efa350f 100644 --- a/transactional_emulator/testbench/aten/configurable.py +++ b/transactional_emulator/testbench/aten/configurable.py @@ -72,6 +72,56 @@ def apply_latency_profile_config( latency[name][latency_profile] = int(value) +# Board YAML `latency:` keys -> TOML LATENCY entry (uppercased). The TOML carries +# extra entries (VECTOR_BASIC/_LONGEST_OPERATE, SCALAR_FP_LONGEST_OPERATE) that the +# board does not specify; those keep their base defaults. +_BOARD_LATENCY_KEYS: tuple[str, ...] = ( + "systolic_processing_overhead", + "vector_add_cycles", + "vector_mul_cycles", + "vector_exp_cycles", + "vector_reci_cycles", + "vector_max_cycles", + "vector_sum_cycles", + "vector_shift_cycles", + "vector_prefix_scan_cycles", + "scalar_fp_basic_cycles", + "scalar_fp_exp_cycles", + "scalar_fp_sqrt_cycles", + "scalar_fp_reci_cycles", + "scalar_int_basic_cycles", +) + + +def apply_board_latency_config(config: dict[str, Any], board_cfg: dict[str, Any]) -> None: + """Apply a board YAML's `latency:` block to the per-build TOML. + + Sets DC_EN from the board's `dc_lib_en` flag and writes each board per-op + cycle cost into the column DC_EN selects (`dc_lib_en` when enabled, else + `dc_lib_dis`), so the transactional emulator runs on that board's latencies. + The board's `memory:` cycle params (prefetch/ddr3) feed the analytic profiler + (isa_analysis.py), not the emulator's memory model, so they are not applied + here. + """ + lat = board_cfg.get("latency") or {} + if not lat: + return + dc_en = 1 if lat.get("dc_lib_en", False) else 0 + column = "dc_lib_en" if dc_en else "dc_lib_dis" + for mode in ("TRANSACTIONAL", "ANALYTIC"): + if mode not in config: + continue + mode_config = config[mode].setdefault("CONFIG", {}) + mode_config.setdefault("DC_EN", {})["value"] = dc_en + latency = config[mode].setdefault("LATENCY", {}) + for key in _BOARD_LATENCY_KEYS: + if key not in lat: + continue + entry = latency.get(key.upper()) + if entry is not None: + entry[column] = int(lat[key]) + + @dataclass(frozen=True) class HardwareConfig: mlen: int @@ -85,6 +135,7 @@ class HardwareConfig: hbm_v_prefetch_amount: int | None hbm_v_writeback_amount: int | None real_data_ratio: float = DEFAULT_REAL_DATA_RATIO + board_cfg: dict[str, Any] | None = None @classmethod def from_args( @@ -154,6 +205,11 @@ def write_toml(self, build_dir: Path) -> Path: dc_en=self.dc_en, latency_profile=self.latency_profile, ) + # A board config (if given) sets the per-op cycle costs + DC_EN directly + # from its `latency:` block, overriding the base defaults so the emulator + # runs on that board's latencies. + if self.board_cfg: + apply_board_latency_config(config, self.board_cfg) # PlenaCompiler reads from BEHAVIOR.CONFIG — mirror tile dimensions AND # the HBM prefetch/writeback amounts there. The compiler's preload diff --git a/transactional_emulator/testbench/run_model.py b/transactional_emulator/testbench/run_model.py index 4263c87f..9254aec8 100644 --- a/transactional_emulator/testbench/run_model.py +++ b/transactional_emulator/testbench/run_model.py @@ -65,7 +65,23 @@ def _build_dir(nickname: str, config_name: str, case: str) -> Path: return base -def _write_toml(preset: HardwarePreset, build_dir: Path) -> Path: +def _load_board_cfg(name: str | None) -> dict | None: + """Load a board YAML from testbench/board_configs/.yaml (or a path).""" + if not name: + return None + import yaml + + p = Path(name) + if not p.exists(): + p = Path(__file__).parent / "board_configs" / f"{name}.yaml" + if not p.exists(): + avail = sorted(q.stem for q in (Path(__file__).parent / "board_configs").glob("*.yaml")) + raise SystemExit(f"unknown board {name!r}; available: {avail}") + with p.open() as f: + return yaml.safe_load(f) + + +def _write_toml(preset: HardwarePreset, build_dir: Path, board_cfg: dict | None = None) -> Path: hw = HardwareConfig( mlen=preset.mlen, vlen=preset.vlen, @@ -77,9 +93,14 @@ def _write_toml(preset: HardwarePreset, build_dir: Path) -> Path: hbm_m_prefetch_amount=None, hbm_v_prefetch_amount=None, hbm_v_writeback_amount=None, + board_cfg=board_cfg, ) toml_path = hw.write_toml(build_dir) os.environ["PLENA_SETTINGS_TOML"] = str(toml_path) + if board_cfg: + print( + f"Applied board latency: {board_cfg.get('name', '?')} (DC_EN={1 if board_cfg.get('latency', {}).get('dc_lib_en') else 0})" + ) return toml_path @@ -97,7 +118,7 @@ def compile_sliced(mc: ModelConfig, preset: HardwarePreset, args) -> tuple[dict, batch_size = args.batch_size or preset.batch_size build_dir = _build_dir(mc.nickname, args.config or "default", args.case or "decoder-chain") - _write_toml(preset, build_dir) + _write_toml(preset, build_dir, _load_board_cfg(getattr(args, "board", None))) case = args.case or ("decoder-layer" if layers == 1 else "decoder-chain") common = dict( @@ -149,7 +170,7 @@ def compile_native(mc: ModelConfig, preset: HardwarePreset, args) -> tuple[dict, batch_size = args.batch_size or preset.batch_size build_dir = _build_dir(mc.nickname, args.config or "default", case) - _write_toml(preset, build_dir) + _write_toml(preset, build_dir, _load_board_cfg(getattr(args, "board", None))) print(f"Loading model {mc.model_id}...") model = AutoModel.from_pretrained( @@ -243,6 +264,12 @@ def main(): parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) parser.add_argument("nickname", help="Model nickname (e.g. smollm2, llada-8b, smolvlm2)") parser.add_argument("--config", default=None, help="Hardware preset name (e.g. sliced_64x64x16_b1)") + parser.add_argument( + "--board", + default=None, + help="Board config (board_configs/.yaml or a path) whose latency: block " + "is applied to the emulator (per-op cycles + DC_EN). E.g. nexys_video, custom_a7.", + ) parser.add_argument("--compile-only", action="store_true", help="Generate ASM only, skip emulator") parser.add_argument( "--case", From c583426d8a4967c3735a12b8f778f247fde2122b Mon Sep 17 00:00:00 2001 From: booth-algo Date: Thu, 4 Jun 2026 16:54:16 +0100 Subject: [PATCH 10/10] chore(boards): align Artix-7 vector/systolic latencies to synthesized RTL Update nexys_video + custom_a7 latency: blocks to the RTL DC_LIB_DIS pipeline depths (configuration.svh): systolic_processing_overhead 0->8, vector_add 2->9, vector_mul 5->7, vector_exp 6->15, vector_reci 7->8, vector_sum 20->30. Corrects the emulator/profiler under-counting of vector FP latency (softmax/LayerNorm exp + sum-reduce) and the previously zero systolic fill/drain. Per-line provenance comments cite the RTL block. --- .../testbench/board_configs/custom_a7.yaml | 14 ++++++++------ .../testbench/board_configs/nexys_video.yaml | 14 ++++++++------ 2 files changed, 16 insertions(+), 12 deletions(-) diff --git a/transactional_emulator/testbench/board_configs/custom_a7.yaml b/transactional_emulator/testbench/board_configs/custom_a7.yaml index 176cefad..a88c947c 100644 --- a/transactional_emulator/testbench/board_configs/custom_a7.yaml +++ b/transactional_emulator/testbench/board_configs/custom_a7.yaml @@ -48,13 +48,15 @@ precision: latency: dc_lib_en: false - systolic_processing_overhead: 0 - vector_add_cycles: 2 - vector_mul_cycles: 5 - vector_exp_cycles: 6 - vector_reci_cycles: 7 + # Aligned to RTL pipeline_pkg DC_LIB_DIS block (src/definitions/configuration.svh, FPGA build). + # Prior values under-counted the vector FP pipeline (incl. the 100 MHz timing-closure pipeline regs). + systolic_processing_overhead: 8 # was 0; RTL=8 (systolic feed/drain: 2*COMPUTE_DIM + 8) + vector_add_cycles: 9 # was 2; RTL=9 (+1 fp_cp_adder partition register) + vector_mul_cycles: 7 # was 5; RTL=7 + vector_exp_cycles: 15 # was 6; RTL=15 (+1 fp_cp_exp normalize->cast PIPELINE_STAGE) + vector_reci_cycles: 8 # was 7; RTL=8 (+1 fp_cp_reciprocal) vector_max_cycles: 4 - vector_sum_cycles: 20 + vector_sum_cycles: 30 # was 20; RTL=30 (+1/level x5 reduction partition register) vector_shift_cycles: 1 vector_prefix_scan_cycles: 9 scalar_fp_basic_cycles: 1 diff --git a/transactional_emulator/testbench/board_configs/nexys_video.yaml b/transactional_emulator/testbench/board_configs/nexys_video.yaml index bd3d9dcb..b4448494 100644 --- a/transactional_emulator/testbench/board_configs/nexys_video.yaml +++ b/transactional_emulator/testbench/board_configs/nexys_video.yaml @@ -51,13 +51,15 @@ precision: latency: dc_lib_en: false - systolic_processing_overhead: 0 - vector_add_cycles: 2 - vector_mul_cycles: 5 - vector_exp_cycles: 6 - vector_reci_cycles: 7 + # Aligned to RTL pipeline_pkg DC_LIB_DIS block (src/definitions/configuration.svh, FPGA build). + # Prior values under-counted the vector FP pipeline (incl. the 100 MHz timing-closure pipeline regs). + systolic_processing_overhead: 8 # was 0; RTL=8 (systolic feed/drain: 2*COMPUTE_DIM + 8) + vector_add_cycles: 9 # was 2; RTL=9 (+1 fp_cp_adder partition register) + vector_mul_cycles: 7 # was 5; RTL=7 + vector_exp_cycles: 15 # was 6; RTL=15 (+1 fp_cp_exp normalize->cast PIPELINE_STAGE) + vector_reci_cycles: 8 # was 7; RTL=8 (+1 fp_cp_reciprocal) vector_max_cycles: 4 - vector_sum_cycles: 20 + vector_sum_cycles: 30 # was 20; RTL=30 (+1/level x5 reduction partition register) vector_shift_cycles: 1 vector_prefix_scan_cycles: 9 scalar_fp_basic_cycles: 1