Skip to content

Commit d414916

Browse files
authored
fix(stats): include execution errors in pass/total denominator (#999)
Previously the summary showed passedCount/gradedCount where gradedCount excluded execution errors, making results appear better than they were. Now shows passedCount/total to match the Convex Evals convention that inspired this design: the denominator is all tests actually attempted, not just those that reached the grading stage. Execution errors are still reported separately in the detail section. Closes #998
1 parent 70e8df1 commit d414916

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

apps/cli/src/commands/eval/statistics.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -226,7 +226,7 @@ export function formatEvaluationSummary(
226226
} else {
227227
overallVerdict = overallPassed ? 'PASS' : 'FAIL';
228228
verdictColor = overallPassed ? '\x1b[32m' : '\x1b[31m';
229-
verdictText = `RESULT: ${overallVerdict} (${summary.passedCount}/${gradedCount} scored >= ${threshold}, mean: ${formatScore(summary.mean)})`;
229+
verdictText = `RESULT: ${overallVerdict} (${summary.passedCount}/${summary.total} scored >= ${threshold}, mean: ${formatScore(summary.mean)})`;
230230
}
231231

232232
lines.push('\n==================================================');

0 commit comments

Comments
 (0)