Datasets used for training network in paper

[ChristianMilianti]

Hi there,

Nice project!

I was wondering if you provide the complete datasets that were utilised to train both the single object, and muti object networks in your paper?

[thomasweng15]

Hi Christian, apologies for my delayed response. The datasets for the networks are provided in the Google Drive link in the Readme. Are you having any issues with the datasets?

Thank you,
Thomas

[ChristianMilianti]

tldr: I'm having difficulty in reproducing the results you report on intra-category generalisation performance .63+/- .5 on the 30 bottle trials. My experiment only ran for 7 trials instead of the 30, however, I only achieved 1/7 successful grasps.
I tried retraining my own network using the data you provided, but my performance didn't improve either when using the new weights. Just wanted to confirm if there are any other files or weights you used to train or evaluate the results from the paper.

---

Thanks for your reply!

Yes, those are the ones I used.

I just wanted to confirm those were the ones used to train the network from the paper.

When running multi object eval bash scripts/eval/reach_and_grasp/multobj.sh with NGDF only and only bottle ( 7 bottles) I got the following error below.

The error:

Traceback (most recent call last):
  File "/home/christian/Documents/GitHub/NGDF_private/OMG-Planner/omg_bullet/panda_scene.py", line 123, in main
    info = planning_scene.step(category=category, pc_dict=pc_dict, viz_env=env)
  File "/home/christian/Documents/GitHub/NGDF_private/OMG-Planner/omg/core.py", line 566, in step
    self.traj, category=category, pc_dict=pc_dict, viz_env=viz_env
  File "/home/christian/Documents/GitHub/NGDF_private/OMG-Planner/omg/planner.py", line 823, in plan
    pc_dict["points_world"], category=category
  File "/home/christian/Documents/GitHub/NGDF_private/OMG-Planner/omg/../omg_bullet/methods/ngdf_pred.py", line 86, in get_shape_code
    latent = self.models[category].shape_model.extract_latent(shape_mi)
KeyError: 'Bowl'

In order to make it work I had to manually amend dataset to only include bottles but I'm not sure if I did it correctly.

When I was able to get bash scripts/eval/reach_and_grasp/multobj.sh to run:

My experiment only ran for 7 trials instead of the 30, however I only achieved 1/7 successful grasps.

Just wanted to confirm if there are any other files or weights you used to train or evaluate the results from the paper.

Thanks!

Christian

[thomasweng15]

Hi Christian, thanks for the detailed description. For the intra-category Bottle experiment, we trained on 7 bottles and tested on the same bottle used in the single-object experiment. Is that the training and evaluation you are running? There is definitely room to scale up further here. I am currently traveling until mid-July, but when I have time I can try running the intra-category evaluation script on my system to debug. Thank you, Thomas

…

On Fri, Jul 7, 2023 at 12:09 AM Christian Milianti ***@***.***> wrote: tldr: I'm having difficulty in reproducing the results you report on intra-category generalisation performance .63+/- .5 on the 30 bottle trials. *My experiment only ran for 7 trials instead of the 30, however, I only achieved 1/7 successful grasps.* I tried retraining my own network using the data you provided, but my performance didn't improve either when using the new weights. Just wanted to confirm if there are any other files or weights you used to train or evaluate the results from the paper. ------------------------------ Thanks for your reply! Yes, those are the ones I used. I just wanted to confirm those were the ones used to train the network from the paper. When running multi object eval bash scripts/eval/reach_and_grasp/multobj.sh with NGDF only and only bottle ( 7 bottles) I got the following error below. The error: Traceback (most recent call last): File "/home/christian/Documents/GitHub/NGDF_private/OMG-Planner/omg_bullet/panda_scene.py", line 123, in main info = planning_scene.step(category=category, pc_dict=pc_dict, viz_env=env) File "/home/christian/Documents/GitHub/NGDF_private/OMG-Planner/omg/core.py", line 566, in step self.traj, category=category, pc_dict=pc_dict, viz_env=viz_env File "/home/christian/Documents/GitHub/NGDF_private/OMG-Planner/omg/planner.py", line 823, in plan pc_dict["points_world"], category=category File "/home/christian/Documents/GitHub/NGDF_private/OMG-Planner/omg/../omg_bullet/methods/ngdf_pred.py", line 86, in get_shape_code latent = self.models[category].shape_model.extract_latent(shape_mi) KeyError: 'Bowl' In order to make it work I had to manually amend dataset to only include bottles but I'm not sure if I did it correctly. When I was able to get bash scripts/eval/reach_and_grasp/multobj.sh to run: *My experiment only ran for 7 trials instead of the 30, however I only achieved 1/7 successful grasps.* Just wanted to confirm if there are any other files or weights you used to train or evaluate the results from the paper. Thanks! Christian — Reply to this email directly, view it on GitHub <#17 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIQDKZHAAJXT2FWLJNI3KLXO6DWTANCNFSM6AAAAAAZX72DOI> . You are receiving this because you commented.Message ID: ***@***.***>

[ChristianMilianti]

Hi Thomas,

Thanks for the prompt reply.

I was testing on 1 instance of each of the 7 different bottles that were used for training.

I will try just testing on the single bottle from the single-object experiment.

In the paper you mention you evaluated on a held-out Bottle instance. Does that mean the bottles were always upright? Or did you use the same rotations as per the single-object experiments?

Thank you,

Christian

[ChristianMilianti]

Hi Thomas,
Just following this up.

I tried testing on 30 instances of the same bottle from the single object experiment (with the same rotations you provided for the single object experiment) and achieved 14/30 success (46.67%). Are these the rotations you used?

Cheers,

Christian

[ChristianMilianti]

Hi @thomasweng15, just wanted to confirm the execution success results.

In the paper, you report 0.63 +- 0.5 execution success for the intra category generalisation experiment on the bottle.

That's seems like a lot of variance? Or is it a typo and meant to be 0.63 +- 0.05 ? (5%)

I hope you can clear this up for me! 😀

Kind Regards,

Christian Milianti

[thomasweng15]

Apologies for the delay!

In the paper you mention you evaluated on a held-out Bottle instance. Does that mean the bottles were always upright? Or did you use the same rotations as per the single-object experiments?

We used the same rotations as per the single-object experiments.

I tried testing on 30 instances of the same bottle from the single object experiment (with the same rotations you provided for the single object experiment) and achieved 14/30 success (46.67%). Are these the rotations you used?

These are the same rotations that we used. I'm not sure why there is a drop in performance, I will check by running the intra-category evaluation on my machine as you mentioned there were some additional steps you had to take to get it to work.

In the paper, you report 0.63 +- 0.5 execution success for the intra category generalisation experiment on the bottle.
That's seems like a lot of variance? Or is it a typo and meant to be 0.63 +- 0.05 ? (5%)

The final version of the paper removed the variance as we determined it was the incorrect (or at least unhelpful) way to compute variance for a binary success metric. We considered using Clopper-Pearson confidence intervals but opted to follow other papers in reporting just the mean for binary success metrics. This wikipedia page may be a helpful resource: link

We also found that the training for NGDF could be unstable, so it may be helpful to evaluate intermediate checkpoints if you are doing your own training to see if you get better performance.

Thank you,
Thomas

[ChristianMilianti]

Apologies for the delay!

No worries at all!

These are the same rotations that we used. I'm not sure why there is a drop in performance, I will check by running the intra-category evaluation on my machine as you mentioned there were some additional steps you had to take to get it to work.

Thanks for that. I will wait for your response.

The final version of the paper removed the variance as we determined it was the incorrect (or at least unhelpful) way to compute variance for a binary success metric.

Do you have a link to the final version of the paper where the variance is removed? The version I have downloaded seems to have it in the Intra-Category generalisation section still. I also checked both the arxiv, and the ICRA versions. (See screenshot attached)

We also found that the training for NGDF could be unstable, so it may be helpful to evaluate intermediate checkpoints if you are doing your own training to see if you get better performance.

Thank you for the insight, do you have any hypothesis as to why that could be?

Thank you,
Christian

[nananQyQ]

Hi Christian, apologies for my delayed response. The datasets for the networks are provided in the Google Drive link in the Readme. Are you having any issues with the datasets?

Thank you, Thomas

Hi Tomas,

Nice work!
I just wondered where the point cloud is. I saw this 'pc_data_path: null # ${data_root}/acronym_perobj/shape-dataset' in config, but when I downloaded from google link, I just got perobj.hdf5. I have no idea about how to get the shape dataset...

Thanks!