bad performace in robotwin evaluation

[justimyhxu]

Hi, thanks for the excellent work and for releasing the codebase!

I followed the README instructions to set up the evaluation environment for RobotWin with Motus, and tested the rollout trajectories on the task “put the object in the cabinet.” However, I observed a noticeable discrepancy between the reported and reproduced performance.

Test results:

• Reproduced success rate: ~45%
• Reported in the paper (supplementary):
  • ~88% in the clean setting
  • ~71% in the randomized setting

I would like to ask:

1. Are there any additional evaluation details (e.g., environment version, random seed handling, number of rollouts, or success criteria) that might affect this result?
2. Were the reported numbers averaged over multiple runs or random seeds?
3. Have you tested the reproducibility of this task multiple times on a fresh environment to confirm the reported performance?

I would really appreciate any guidance on what might cause this gap, or pointers to specific evaluation settings to double-check. I’m happy to provide more details (logs, configs, seeds) if helpful.

Thanks again for sharing such a great project!

[HongzheBi]

The success rate index should not be so much worse, 1-2 points of fluctuation is normal. I can check it remotely for you if it's convenient. You can add me on wechat.

About your question: The environment is normal installation. Random seeds have no special settings. The evaluation is the standard 100 runs.

[HongzheBi]

In addition, are the success metrics of other tasks so different? You can calculate if there is a severe drop in the overall average success rate for the 50 tasks.

[hanyangyu1021]

Any progress? My robotwin performance is also much lower than this table.

[HongzheBi]

The test performance on my side is normal. Please give us all the results and average metrics of the 50 tasks. It is possible that the wrong model weights were uploaded.

[strive789]

Have you solved this issue? I can reproduce the reported scores using the open-source weights. However, when I train on RoboTwin with the same settings as in the paper—batch size 32×8, 40,000 steps—under three settings (no pretrain, loading Stage-1 weights, loading Stage-2 weights), the success rate in the Random setting is all below 60%. Did you encounter this problem as well?
@HongzheBi @hanyangyu1021 @justimyhxu

[HuailiangMa]

Have you solved this issue? I can reproduce the reported scores using the open-source weights. However, when I train on RoboTwin with the same settings as in the paper—batch size 32×8, 40,000 steps—under three settings (no pretrain, loading Stage-1 weights, loading Stage-2 weights), the success rate in the Random setting is all below 60%. Did you encounter this problem as well? @HongzheBi @hanyangyu1021 @justimyhxu

我也遇到了这个问题，用Robotwin生成的550条数据做单任务后训练，训练15000步之后模型效果跟论文报告有很大差别，理论上多任务训练40000步，我这个训练15000步单任务应该可以，但实际上单任务成功率只有20%，远不如报告里的88%