A job's rate, the trials behind it, the controls, then a comparison.
- Open Runs, By launch or All runs.
- Open a job: its live board while it runs, its trial outcomes once finished.
Foreign results open the same way: Open existing results.
| Metric | Meaning |
|---|---|
| Raw solve rate | Resolved trials ÷ all trials; infrastructure failures count as misses |
| Adjusted solve rate | Resolved trials ÷ trials excluding infrastructure failures |
| Mean score | The average graded score, for scored benchmarks |
— |
No verdict, not a zero |
!!! note "The gap between raw and adjusted is infrastructure" It is what rate limits and setup failures cost; it does not predict the excluded tasks.
| Step | Open | Ask |
|---|---|---|
| 1 | Outcome and error | Did the attempt reach the task, or fail in setup or a provider call? |
| 2 | Verifier output | Which check rejected it? |
| 3 | Conversation and tool calls | Which actions led there? |
| 4 | Trace and artifacts | Which calls failed or ran slowly? What files were produced? |
- Controls: the
nopandoraclejobs beside the agent's (Core concepts). - Feasibility: a passing
nopor a failingoracleopens in the feasibility view. - Without controls: the run holds no evidence about task validity.
- Open Compare, pick the runs, choose Matched tasks.
- Start with the tasks whose outcomes differ and open the attempts side by side.
- Check dataset, model, agent and setup before attributing a change.
!!! warning "Different task sets move the aggregate" A score can change with the task set alone.
- Configure a judge in Settings.
- Request a judgement from a trial or a job.
- Read disagreements with the recorded outcome in Reconcile.
!!! warning "A judgement is a paid model call" Start with one trial; a batch button names how many calls it makes.
Questions, records and cost: Judgement.
Metric field guide, under Reference → Metrics on the documentation website.