Skip to content

Fix Princeton cross-entropy replay exploit via phase-specific inputs#142

Merged
S1ro1 merged 1 commit intogpu-mode:mainfrom
Ammaar-Alam:fix/princeton-cross-entropy-replay-exploit
Apr 8, 2026
Merged

Fix Princeton cross-entropy replay exploit via phase-specific inputs#142
S1ro1 merged 1 commit intogpu-mode:mainfrom
Ammaar-Alam:fix/princeton-cross-entropy-replay-exploit

Conversation

@Ammaar-Alam
Copy link
Copy Markdown
Contributor

Summary

Fixes a benchmark replay exploit in problems/princeton/cross_entropy_py/eval.py where a submission can exploit the evaluator's reuse of the exact same (logits, targets, grad_output) tensors across warmup and timed phases.

In the current evaluator, a benchmark-aware submission can:

  1. compute correct outputs during warmup or earlier timed phases
  2. arm a replay path keyed on input identity / benchmark phase structure
  3. skip the real computation during the ranked combined phase while still returning already-populated correct outputs

This is especially severe for the Princeton task because the leaderboard ranks by median combined time only.

Changes

  1. add a private per-run seed schedule for benchmark inputs
  2. generate distinct inputs for each benchmark phase:
    • warmup
    • forward-only timing
    • backward-only timing
    • combined timing
  3. reuse the same private seed schedule for both the baseline and the submission under test so the speedup comparison still sees matched workloads

Why this fix

The exploit depends on the ranked combined loop reusing the same exact tensors that were already seen during warmup or earlier timing loops.

This patch breaks that assumption while keeping input generation outside the measured CUDA event region:

  • the combined timed inputs are no longer identical to warmup inputs
  • the combined timed inputs are no longer identical to forward-only or backward-only timed inputs
  • each combined timed iteration gets its own distinct inputs

That removes the replay surface without turning the benchmark into a measurement of evaluator overhead.

Why this approach instead of per-call sync timing

PR #104 discusses a stronger per-call timing / sync approach for grouped GEMM exploits. That style of fix is useful when call deferral across a window is the main threat, but it also adds meaningful evaluator overhead.

For this Princeton task, the primary vulnerability is simpler: the evaluator reuses the exact same tensors across warmup and the ranked combined loop. Distinct phase-specific inputs are enough to close that loophole while preserving the current low-overhead timing structure.

Maintainer note

After merge, the Princeton leaderboard should be resynced / redeployed so the hosted evaluator picks up the patched eval.py.

@Ammaar-Alam
Copy link
Copy Markdown
Contributor Author

(this pr was automated entirely w/ codex so double check carefully pls)

Copy link
Copy Markdown
Member

@S1ro1 S1ro1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@S1ro1 S1ro1 merged commit 16022b0 into gpu-mode:main Apr 8, 2026
@Ammaar-Alam Ammaar-Alam deleted the fix/princeton-cross-entropy-replay-exploit branch April 8, 2026 21:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants