Evaluating Provided Checkpoint Does Not Reproduce Paper

[gunnxx]

I run the organize_items provided checkpoint. However, the "average" is only 35% way below the reported one in paper 52%. Attached is the detailed output. Did I miss something or is it expected?

evaluate_ret_02.json

[gunnxx]

I rerun it just to be sure:
evaluate_ret_01.json

The commands are

# Policy Server Side
uv run --no-sync scripts/serve_policy.py --host='0.0.0.0' --port=5555 policy:checkpoint --policy.config=organize_items --policy.dir ./checkpoints/organize_items/29999/

# Genie Sim Side
geniesim --config source/geniesim/config/organize_items.yaml --app.headless True

I set benchmark.num_episode=10.

Paper : https://arxiv.org/pdf/2601.02078

[Anonymous-694]

@gunnxx Hi, thanks for your comments. The paper results are from GenieSim3.0, could you provide more detailed info about your simulation env commit and assets, so that we can check the version consistency. Moreover, the paper will be updated in a few days, including some new experiment results. Please stay tuned and collaborations and discussions are welcome.

[gunnxx]

genie_sim commit [link]
As for the asset commit hash, I couldnt recover it reliably. However, I downloaded it from modelscope on April 2nd.

Out of curiosity, is the provided checkpoint is the 500 eps sim result or 1500 eps sim result? I just assumed it was 1500 but I realized it was not stated explicitly.

Cant wait for the updated paper and thanks for the great work!