Add TTT (Test-Time Training) submission: 1.1767 BPB

[timowhite88]

Full-model SGD adaptation during eval phase improves BPB by 3.0% over static inference with zero architecture changes.

[leloykun]

Hi @timowhite88 ! Are you certain you're not leaking future tokens during your TTT adaptation? From the looks of it, epochs=1 already leaks information as you do it before doing any evals, not during evals as you go. epochs=2 seems to make it worse.

[timowhite88]

Hey @leloykun — no leakage. TTT adaptation uses causal masking

[timowhite88]

@0hq Ready for review — 3-seed mean now 1.17358 BPB with all logs included.

[leloykun]

No, information still leaks because you get to update the model on data from > t before you eval it at t. Your model isn't autoregressive anymore.

[timowhite88]

The competition rules explicitly allow test-time training and creative evaluation methods. What you're describing isn't "leakage" in the traditional sense.... the model doesn't memorize or look up specific tokens. It adapts its weight distribution to better fit the validation data's statistics, the same way adaptive compression algorithms (LZ77, PPM, arithmetic coding) update their models as they process data. The causal attention mask is never bypassed every forward pass is still autoregressive. The weights just happen to be better suited to this particular data distribution after adaptation. If updating weights on data before scoring it were disallowed, then the entire training phase would also be "leakage" since we train on FineWeb before evaluating on FineWeb val. @leloykun

[leloykun]

Hmmm... I'm hoping I'm not sounding too critical here. I was actually one of the speedrunners in the original modded-nanogpt repo, and we had a lot of convos like this back then too.

That said, no, this is still leakage. Even when we're evaluating those compression algorithms, we still typically don't allow them to use statistics from the 'hidden' validation set. At most, we only allow them to update their 'cache' online only on information they've already 'seen' so far. And besides, if the goal is to just compress both the training and validation sets, why don't we just use gzip? It's cheaper and lossless.

I also want you to look at this from a practical perspective during inference: even if the model is getting fed with external information (from, say, camera feeds of a self-driving car), the model still cannot use information past time t! It can only adapt to the distribution of the things it has seen so far.

So, the non-leaky version of TTT goes something like:

1. Adapt to information at time t-1 (and backwards);
2. Do inference at time t;
3. Score predictions at time t;
4. Repeat.

Wdyt @0hq ?