A request for guidance on a comparative experiment

[SimonHeye]

Thank you for your outstanding contributions to the community!
I would like to ask whether, when compared with other works, the same prompt was directly used to re-experiment on open-source models from other works, or whether the results from other works' papers were directly used? (Because I have seen that many other works only conduct experiments targeting a specific tool, rather than this multi-tool selection and invocation.)😎

[Zhuofeng-Li]

Hi, thanks for your kind words!

Since different baselines use different settings, please refer to the corresponding papers for more details.

[SimonHeye]

Hi, thanks for your kind words!嗨，感谢您的美言！

Since different baselines use different settings, please refer to the corresponding papers for more details.由于不同的基线使用了不同的设置，请参考相应的论文以获取更多详细信息。

Thank you for your reply. That is to say, agentflow is directly compared with other specialized models that only call one tool, rather than having other models like search-r1 also perform multi-tool selection and calling. The prompts for other models are set according to their papers, while agentflow seems to have an additional step of tool selection capability, is that correct?

[Zhuofeng-Li]

Not exactly — some baselines invoke multiple tools (e.g., Autogen). Please refer to the specific implementations in the original papers.

[SimonHeye]

Not exactly — some baselines invoke multiple tools (e.g., Autogen). Please refer to the specific implementations in the original papers.

For ToRL, which only calls one tool, is it set up using the prompt in their paper?

[IsaacGHX]

Hi @SimonHeye, sorry for the delay,

For ToRL and other baselines, we follow the tool implementations and prompting strategies described in their original papers. As @Zhuofeng-Li mentioned above, since different methods adopt different experimental settings and tool usage paradigms, we recommend referring to the original papers for the exact details of how each tool is implemented and invoked.

That said, it's believed that a way better reinforcement learning method with strong generalization ability should, in principle, enable the model to spontaneously acquire performance gains when interacting with unseen tools, rather than being limited to a fixed or single-tool setup. Exploring this direction is very promising, and we encourage further investigation along these lines. A recent example that aligns well with this perspective is this work: From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by Composing Old Ones, which may be of interest.

We are happy to continue the discussion and exchange thoughts on this topic.

[Zhuofeng-Li]

Yes, you’re right — LLM-as-judge only returns a binary label (correct / incorrect).

However, it’s worth clarifying that not all test sets contain exactly 100 samples. Some of the benchmarks include fewer or more than 100 instances. The exact distribution and evaluation protocol follow the settings described in our original paper, which we recommend reporting for full details.

For other baselines, we try to adhere as closely as possible to the experimental settings reported in their respective papers. To ensure a fair comparison and reduce potential performance variance, we run multiple trials and report the average results whenever applicable (especially when randomness or sampling is involved).

This way, we aim to maintain consistency with prior work while ensuring fairness and robustness in the comparisons.