Clarify Table 5: Individual Tasks (10,115) vs LLM Judger (50)

[Ethereal-sakura]

Hi, thanks for the excellent work and for open-sourcing MCPEval — it’s very helpful to the community. I have two quick questions:

1. In Table 5, what’s the exact difference between Individual Tasks (10,115) and LLM Judger (50)? What does each sample count represent, and how is each aggregated?

2. The Trajectory Evaluation Aspects and Task Completion Evaluation Aspects are defined in Appendix B.2 LLM Judger Criteria. Why do they also appear under Individual Tasks (10,115)? Are the same rubrics applied at the task level vs. the model×domain level, and if not, what differs?

Thanks again!

[JimSalesforce]

Thanks for carefully reading our paper! The content in the appendix is to show that our evaluating results consist of 10115 evaluation data points, which is a combination of trajectory + completion across all 5 domains. For example, an evaluation data point is a (trajectory_score, task_id, domain_name, model_name) .
50 is to view the results by average all the tasks and scores, which is the num_domains * num_models.
It may be misleading to compare this name with the actual generated tasks. We may later change the statements and tables.