[Eval] Benchmark against NanoGPT hackers

[TomLucidor]

Motivation

Why is this evaluation important for HOPE/TITAN reproduction?
To see if Nested Learning can live up to the hype of emergent learning when compared to AR and Diffusion training method for basic pre-training

Task details

• Dataset / benchmark: https://github.com/KellerJordan/modded-nanogpt
• Metric(s): time-to-Perplexity, basic LLM benchmarks
• Expected runtime / hardware: same hardware, same dataset

Implementation sketch

Outline scripts/flags needed (e.g., extend scripts/eval/zeroshot.py).

Acceptance criteria

Describe what needs to be captured (JSON fields, plots, etc.).
Reports on how over-optimized autoregression cannot outperform Nested Learning when given the same clock time and/or FLOPs, to target the same dataset but getting lower Perplexity.

[kmccleary3301]

Good request. I want to run this, but I need a strict protocol so the comparison is meaningful.

My proposed protocol:

• match tokenizer, train tokens, and data mixture as closely as possible
• report both wall-clock and token-normalized progress
• compare HOPE, TITAN, and transformer controls with the same eval scripts
• publish raw JSON outputs and run configs, not only summary numbers

Initial deliverable for this issue:

• one reproducible benchmark harness script
• one short run card with time-to-loss and zero-shot deltas
• explicit caveats where apples-to-apples is not possible

Scope note: I have not run anything close to the original paper’s largest training scale yet, and I am not planning that immediately due to compute constraints. That may change if I can allocate more compute.

I will keep this issue open and post a benchmark protocol before claiming any performance conclusion.