Record: Int6 MLP3x + SmearGate + BigramHash + MuonWD + SWA (mean val_bpb=1.1483)

[raahilshah]

Int6 MLP3x + SmearGate + BigramHash + OrthoInit + Muon WD + SWA

Mean val_bpb: 1.1483 (3 seeds: 1.1488, 1.1485, 1.1476)

Trained on 8×H100 SXM in 600 seconds. 15.92MB artifact (int6+zstd-22).

Key Techniques

1. Per-row int6 quantization ([-32,31]) on MLP + attention weights, fp16 passthrough for tied embeddings and last-layer key projection. zstd level 22 compression.
2. 3× MLP expansion (hidden=1536) — enabled by int6 byte savings. Single largest improvement source.
3. SmearGate — learned gate blending each token embedding with the previous token's (~512 params).
4. BigramHash embedding — 4096-bucket hash table (dim=128→512) for token-pair context (~524K params).
5. Orthogonal init + muP scaling — orthogonal weight init, output projections scaled by 1/√(2·num_layers).
6. Muon WD=0.02 with momentum warmup 0.92→0.99 over 1500 steps. AdamW WD=0.01 for embeddings/scalars.
7. SWA over last 50% of training (every 200 steps) — smoother weights, better quantization.

Hyperparameters

9 layers, 512 dim, MLP 3×, seq2048, batch=786K, warmdown=3000, matrix_lr=0.02, grad_clip=0.3, muon_momentum=0.99.

Metrics

Seed	val_loss	val_bpb
1337	1.93978	1.14885
42	1.93923	1.14852
7	1.93762	1.14757
Mean	1.93888	1.14831

• Pre-quant val_bpb: 1.1640
• Steps: 7,373 in 600s (81.4 ms/step)
• Artifact: 15.92MB (int6+zstd-22)
• Improvement over current SOTA (1.1748): -0.0265 bpb / -0.046 nats

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 14cdf6f7a4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Avoid double-counting tail tokens in sliding eval

The sliding-window scorer can count the same validation tokens more than once near the end of the corpus. With s = 0 if ws == 0 else max(wlen - stride, 0), any non-first window where wlen < stride scores the entire short window, including tokens that were already scored by the previous window (e.g., seq_len=8, stride=4 double-scores the last two tokens). This biases the reported val_loss/val_bpb, which can skew experiment comparisons.

Useful? React with 👍 / 👎.

[raahilshah]

Updated submission — improved configuration after systematic hyperparameter sweeps:

• Muon weight decay: 0.02 → 0.04 (swept 0.01–0.05, optimal at 0.04)
• SWA frequency: every 200 steps → every 50 steps (swept 25–200, optimal at 50; ~30 checkpoint average)

New results (3 seeds):

Seed	val_loss	val_bpb
1337	1.93492	1.14597
42	1.93591	1.14656
7	1.93314	1.14492
Mean	1.93466	1.14582

Previous submission mean was 1.1483 → now 1.1458 (improvement of 0.0025 bpb from tuning WD and SWA frequency alone).