Sliding Window Eval + Muon6 (val_bpb 1.1973)

[beee003]

Summary

• Mean val_bpb: 1.1973 (3 seeds: 1337→1.1968, 42→1.1974, 7→1.1978, p<0.001)
• Artifact: ~15.9 MB (under 16 MB)
• Training: ~13,688 steps in 600s on 8xH100 SXM
• Eval: ~126s sliding window (within 10 min budget)

Key Techniques

1. Sliding Window Evaluation (stride=256): Each token scored with 768+ context instead of 0-1023 average. Added forward_logits() method for efficient inference.
2. Muon 6-step Newton-Schulz: More accurate gradient orthogonalization (MUON_BACKEND_STEPS=6).
3. Extended Momentum Warmup: MUON_MOMENTUM_WARMUP_STEPS=1000 (up from 500) stabilizes early training.
4. Longer Warmdown: WARMDOWN_ITERS=1500 (up from 1200) for smoother LR decay.

Reproduction

MUON_BACKEND_STEPS=6 MUON_MOMENTUM_WARMUP_STEPS=1000 WARMDOWN_ITERS=1500 EVAL_STRIDE=256 \
SEED=1337 RUN_ID=submission VAL_LOSS_EVERY=0 \
torchrun --standalone --nproc_per_node=8 train_gpt.py

Results

Seed	val_loss	val_bpb	Steps
1337	2.0208	1.1968	13,688
42	2.0217	1.1974	13,688
7	2.0225	1.1978	13,688
Mean	2.0217	1.1973

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ccf183ec4e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Prevent double-counting boundary tokens in sliding eval

On multi-GPU runs, eval_val_sliding partitions the validation set with my_start/my_end, but this scoring window still extends each rank by a full stride because score_end is only clamped to actual_len. In the 8xH100 path used for leaderboard submissions, that means the last window on rank r can score past my_end while rank r+1 starts again at its own my_start, so the boundary tokens are counted twice in the all-reduced val_loss/val_bpb. Please clamp the scored range to the local shard (or distribute work by windows instead of raw tokens) so the final metric matches the true dataset average.

Useful? React with 👍 / 👎.