[draft] Chained-KN @ K=11, data-subset, PAQ, chunker — 13 new submissions

[gabrielnan]

Summary

Builds on #3's gradient-free survey with a Pareto sweep:

• chained-KN n-gram family @ K=11/12/14 — found K=11 is the floor saturation depth
• data-subset paradigm — first 70% of WikiText-103 ≈ full data at byte scale
• Modified Kneser-Ney with per-count discounts (Chen-Goodman) — beats scalar D=0.5
• PAQ-style multi-order context mixing — paradigm validated, dominated by chained-KN
• Schmidhuber 1991 chunker — first hierarchical-surprise arch to pass on byte-LM
• 5-run AdamW reopen — closes the optimizer cluster: Muon essential at iso-energy

Leaderboard (val char-acc ≥ 0.70, NVML training joules ascending)

Submission	Val acc	Energy (J)	Mechanism
subset_70_mkn	0.7031	858	Chained-KN @ K=11 on first-70%-of-train + Chen-Goodman per-count discounts
gpu_ngram_w31_k11	0.7050	1,245	Chained Kneser-Ney @ K=11 on full train (GPU torch.unique table build)
paq_mixer_v3	0.7047	1,744	PAQ-style: 11 independent count tables + 860-param logistic mixer
deep_backoff_kn	0.7184	2,236	Order-14 chained backoff + Kneser-Ney smoothing (CPU multi-core build)
gpu_ngram_o14_xorfix	0.7184	3,172	Order-14 GPU n-gram with XOR-bit sort fix (eliminates 150s CPU re-sort at k≥9)
chunker_phase1_v1	0.7057	5,918	Schmidhuber 1991 chunker: surprise-gated lower-tier (n-gram) + d=192/L=4 upper-tier
lwta_k4_alpha_065	0.7382	13,174	LWTA-k=4 sparse activation + W31 n-gram at α=0.65 hybrid mix
alpha_06	0.7437	14,047	NN + W31 n-gram hybrid at α=0.60 (highest acc clean of any submission)

The subset_70_mkn win composes two paradigms: data-subset (locally stationary corpus → -30% J) and Modified KN (+0.0016pp at iso-K). Both are independently settled across ≥2 runs.

DQ — informative paradigm probes (acc < 0.70 floor or time exceeded)

Submission	Val acc	Energy (J)	Why it fails
gpu_ngram_w31_k10	0.6975	878	K=11 is the floor saturation depth; K=10 misses by 0.25pp
chunker_phase1_v2	0.5621	13,936	Surprise-gated routing essential — fixing α=0.6 (no gate) loses 14pp
adamw_lr3e3_wd0_long	0.7061	41,071	AdamW at proper LR + 3× more steps reaches floor at 2.8× Muon energy — Muon dominates at iso-J
bpe_internal_nn_v2	0.3973	24,417	Per-byte argmax over BPE marginalization disagrees with token-level top-1 (paradigm needs algorithmic redesign)
mamba_byte	NaN	60,864	Pure-PyTorch Mamba SSM without selective_scan_cuda kernel — NaN at step 300

Headline findings

1. Lowest validated NVML-J: subset_70_mkn at 858 J / 0.7031 — 60× under the modded_nanogpt baseline (51,704 J), 53× under lwta_k2 (46,132 J / 0.7146).
2. K=11 is the floor saturation depth for chained-KN. Drop to K=10 → 0.6975 (DQ).
3. Modified Kneser-Ney re-opens "KN discount sweep doesn't help." Chen-Goodman per-count D1/D2/D3+ beats scalar D=0.5 at iso-K with no J penalty.
4. WikiText-103 is locally stationary at the byte level. First-70% subset ≈ random-70% subset ≈ full data within 0.001pp acc; 30% J reduction nearly free.
5. PAQ paradigm validates but is structurally dominated by chained-KN at iso-K. Independent per-order tables + mixer pays +29% J for +0pp acc vs chained backoff. New cluster, paradigm-closed.
6. Schmidhuber 1991 chunker passes byte-LM floor for the first time. Lower-tier n-gram surprise gates a small upper-tier transformer trained only at surprise positions. Pareto-dominated but paradigm-validated.
7. Muon optimizer essential at this scale. 5-run AdamW reopen (lr ∈ {1e-3, 2e-3, 3e-3}, wd ∈ {0.05, 0}, extended steps): closes 0.70 floor only at 2.8× Muon's energy. AdamW NN in hybrid composition contributes 0pp acc above the n-gram backstop.

Related

• Add total-system-energy reporting (CodeCarbon CPU backend) #4 — total-system-energy reporting (CodeCarbon CPU backend). The NVML-J numbers above are kept for comparability with Gradient-free methods exploration, NBB diagnostic, research portfolio #3's leaderboard; Add total-system-energy reporting (CodeCarbon CPU backend) #4's total_energy_J adds the CPU share for honest cross-paradigm comparison.

Test plan

• Every submission has a Modal-validated result.json + run.log + nvml.json artifact
• Paradigm-novel submissions (chunker, PAQ, MKN, subset) each have explicit README documenting the mechanism
• subset_70_mkn replicated across 4 J samples (mean ~880 J at acc 0.7031, deterministic across replicates to 4 decimal places)
• gpu_ngram_w31_k11 replicated across 2 J samples (1,245 + 1,388 J, both at acc 0.7050)
• Maintainer review

🤖 Generated with Claude Code