Does training on Claw-Anything data improve performance on Claw-Eval?

[yaoshengyu]

Hi, thanks for the great work!

The paper reports +23.7pp on the Claw-Anything eval set after fine-tuning.
We're wondering if training on this data also improves performance on Claw-Eval.

Given the large distribution gap (Claw-Anything has ~18x more context, 10x more services,
plus event logs that Claw-Eval doesn't have), we're concerned the trained model might
over-explore on simpler Claw-Eval tasks and hurt performance.

Did you run any cross-benchmark experiments? Any insight would be appreciated!

[icexiaoche]

Hi,

Thanks for the insightful question and for following our work!

You've raised a very solid point regarding the distribution gap. We are actually planning and preparing for these cross-benchmark experiments right now, and we will definitely share the results here as soon as they are available.

Theoretically, our pipeline is designed to be flexible enough to cover the distribution of a substantial number of tasks in Claw-Eval. To simulate these simpler Claw-Eval task distributions and mitigate the "over-exploration" risk you mentioned, we can adjust several hyper-parameters within our pipeline. For instance, we can:

1. Decrease the number of simulation rounds.
2. Control and restrict the number of available services.
3. Reduce the noise level to zero.

By tuning these parameters, we believe the trained model can better adapt to simpler scenarios and ultimately improve its cross-benchmark generalization capability.

Stay tuned, and we'll keep you updated on the progress! Thanks again for your interest in our work!

[icexiaoche]

Hi,

We have some exciting updates to share regarding the cross-benchmark performance!

We recently completed an RL training run on Qwen3.5-27B using our original training data. Excitingly, the post-RL model not only shows significant improvements on Claw-Anything, but it also delivers a substantial performance boost on Claw-Eval, significantly outperforming Qwen3.6-27B.

While we are still fine-tuning the final experiments, these preliminary results strongly indicate that RL training on Claw-Anything effectively enhances cross-benchmark generalization to Claw-Eval, effectively alleviating the "over-exploration" concern on simpler tasks.

Thanks for following our work! We will release more experimental details in the near future. Stay tuned!

[yaoshengyu]

Hi @icexiaoche

Thank you so much for the detailed update — this is really exciting to hear!

It's great to see that RL training on Claw-Anything not only improves in-domain performance but also generalizes well to Claw-Eval, and even surpasses Qwen3.6-27B. That definitely addresses the over-exploration concern we had.

Looking forward to the final experimental details and concrete comparison metrics whenever they're ready. Please keep us posted!

Thanks again for the great work and for being so responsive.