How were training trajectories collected? Discrepancy in performances

[ds5423]

Thanks for the great work! I have a quick question on Section 3.3/4.1.

The paper reports collecting 1,500 successful trajectories from 2,000 training environments (75% success rate). However, the best frontier model (GPT-5.5) achieves only 34.5% Pass@1 on the evaluation set. Could you clarify how the collection agent achieved this substantially higher success rate? For example, was the reference solution provided to guide execution (similar to Stage IV validation), or were multiple attempts made per environment?

Understanding this would help contextualize the +23.7% improvement and aid reproducibility. Thanks!

[icexiaoche]

Thanks for the careful read! The gap is expected and comes from a few factors, so the two numbers aren't directly comparable:

1. Trajectories were collected across multiple agent frameworks and prompt configurations. For each training environment we didn't run a single agent once — we attempted it with different agent frameworks (e.g., our agentic loop and OpenHarness) and different prompting modes (tool mode and skill mode).
2. Each environment was usually attempted multiple times. Within each of those configurations we also sampled/retried many tasks several times, keeping the trajectory if any attempt succeeded. So the 75% is best-of-N coverage over a pool of (frameworks × modes × repeated attempts), not the single-framework, single-attempt Pass@1 that GPT-5.5 reports on the eval set.
3. Training tasks were built for higher yield. Compared to the benchmark, the training tasks were generated with less injected noise and fewer environment initialization rounds, making them easier to complete successfully and increasing the yield of usable trajectories.

As a result, 75% and 34.5% measure different things. The former is environment coverage from a best-of-N pool of frameworks/modes/repeated attempts on a yield-oriented training split; the latter is single-framework, single-attempt Pass@1 on a harder eval split. Thanks for the careful read again!