reproduction help!

[kw9-21]

hi authors!
I’m really hoping to reproduce the results of your paper in the ScienceWorld environment. I’ve already integrated ScienceWorld and got it running, but I haven’t come up with a good prompt yet, so I was wondering if you could share some of the prompts you used in your experiments? Specifically, prompts for generating insights from successful and failed trajectories, as well as the format specifications for those insights.

[bingreeky]

We have already uploaded SciWorld! Could you plz take a look and see whether it is helpful? Thx!

[kw9-21]

May I ask if I can confirm that Sciworld can now be run directly? I seem to have only found envs/sciworld_env.py, and when I run bash run_mas.sh, I get the following error. Also, I would really like to know if there should be a file named sciworld_prompt.py in the directory tasks/prompts.
I would greatly appreciate it if you could provide some necessary files, such as data/sciworld/test.jsonl and tasks/prompts/sciworld_prompt.py!
P.S.: When I tried to integrate sciworld myself earlier, I could only load the dataset from the SciWorld environment in code ; I couldn’t find any open-source versions available for direct download online.

Traceback (most recent call last):
File "/home/GMemory/tasks/run.py", line 21, in
from envs import BaseEnv, BaseRecorder, get_env, get_recorder, get_task
File "/home/GMemory/tasks/envs/init.py", line 31, in
with open(TASKS_PATH['sciworld'], 'r+', encoding='utf-8') as f:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'data/sciworld/test.jsonl'

[bingreeky]

data/sciworld/test.jsonl added! Sorry for the miss.

[kw9-21]

Thanks for your added data! But I still have two questions:

1. why data/sciworld/test.jsonl only has 90 samples? The statistics I obtained using the code below are Train: 3592 (49.8%)| Dev: 1796 (24.9%)| Test: 1819 (25.2%) . How do you get those 90?
   train_vars = env.get_variations_train()
   dev_vars = env.get_variations_dev()
   test_vars = env.get_variations_test()
2. Could you please provide tasks/prompts/sciworld_prompt.py？I'm afraid I cannot reproduce the result in paper without corresponding prompts.

[kw9-21]

To add to the second question: using your provided test.jsonl and my prompt, got result 35.6%(qwen14b), while the table 1 in paper shows 67.4%(gpt-4o-mini). Even though the models are different, surely there wouldn't be such a big gap?

[bingreeky]

Sure, we have updated the config and prompts. The test data is from the AgentSquare codebase; you may refer to it if interested. Many previous issues also encountered reproducibility difficulties, but they were eventually able to identify the causes and reproduce the results, e.g., #19. The 35.6% is surely not expected, and you may start from using the update codebase for reproduction. There is no need to rush — we can continue to discuss and work through this step by step : ) Thank you for the patience!

[kw9-21]

Sorry for getting a little worked up, I just really want to get a good result as soon as possible, haha.
I download your updated codebase and bash run_mas.sh directly, finally the result is ave reward: 0.5638227513227513, ave done: 0.2111111111111111, which is also strange. Could you plz help me?

python3 tasks/run.py
--task sciworld
--reasoning io
--mas_memory g-memory
--max_trials 30
--mas_type autogen
--model Qwen2.5-14B-Instruct \