[Question] Mock service design: why is source code visible to agents?

[passionate11]

Hi team, thanks for the awesome benchmark! Really enjoying working with it.

I had a few questions about how mock services are set up — just want to make sure I'm understanding the design correctly:

1. Mock service source code visible to agents — I noticed that mock_services/server.py ends up in the agent's workspace after setup. We've seen some models read the source and try to start the services themselves (causing port conflicts). Was this intentional, or would it make sense to hide the implementation from agents?

2. Missing tmp/ in task_4 and task_6 — In category3, their warmup scripts reference /tmp_workspace/tmp/messages.json, but there's no tmp/ directory in the workspace. Just wondering if this is intentional or an oversight?

Appreciate any clarification. Thanks again for the great work!

[Mark12Ding]

Hi @passionate11,

Thanks for the detailed feedback and for your interest in our benchmark! Regarding your questions:

Mock service source code visibility:
You’re absolutely right that hiding the implementation would be a more "clean" design, especially to avoid issues like the port conflicts you mentioned.
Our initial intention was to treat mock_services/server.py as a distractor (noise) within the workspace. Since the server code doesn't contain any information useful for solving the tasks, we wanted to see if the agents could correctly identify it as irrelevant. However, we agree that it shouldn't interfere with the agent's execution logic. We'll take this into account for future design iterations.

Missing tmp/ directory:
This was indeed an oversight on our part. Thank you for catching that!
We have updated the tmp/ directory and the relevant workspace files. You can find the fix here: internlm/WildClawBench - Social_Interaction

Thanks again for helping us improve the benchmark! Let us know if you have any other questions.

[passionate11]

thx！