Low proportion of goodcase during filtering gen_imgs based on gen_MPVPE.

[pjl013]

I encountered a serious issue during the final stage of image generation. The proportion of samples that meet the filtering criteria based on gen_MPVPE and other metrics and are selected as good cases is only around 10%. This means that if I want to generate a synthetic dataset of approximately 400,000 samples, similar in size to the DexYCB dataset, I would need to perform around 4 million generations, which would be extremely time-consuming.

If possible, could you share the final proportion of generated images in your case that met the required criteria?

[hxwork]

Hi,

The proportion of good samples should be at least >50%. May I know your settings? Which generative model do you use to generate images? What is the pre-trained model you use to do the filtering?

[pjl013]

I directly use the code you provided to generate rendered images as control conditions and employ the model weights and code of your diffusion model for image generation. Finally, I use the method provided in your code to filter the results by using Mobrecon.

The difference is that your code aligns result from Dexgrasp-net using the hand's position information while generating novel conditions, where I use the object's position information from the real dataset for alignment, which seems to have had a significant impact.

However,I encounter some issues while trying to reproduce this part of your code. Could you please tell me how these files was generated? Also, did you complete Cross-distribution Sampling before or after this step?

[hxwork]

Hi,

These files are generated by our intra- and inter distribution sampling. You can use the original data to check if you can generate higher-quality images. The content in real_info is the same as the original data. Here you can see how to align with an existing sample's global orientation.

[pjl013]

Could you tell me the specific values of M and N, which represent the sizes of the obtained real-world and synthetic datasets after performing Intra-distribution Sampling, respectively for the DexYCB and HO3D datasets?

[hxwork]

Empirically, for the DexYCB dataset (both "s0" and "s1" splits), we set M=500 and N=1,000 for the intra-distribution sampling. For the HO3D dataset, we set M=10 and N=500.

[pjl013]

Hi,

The proportion of good samples should be at least >50%. May I know your settings? Which generative model do you use to generate images? What is the pre-trained model you use to do the filtering?

Thanks for your reply! I try to generate images using the raw DextCB data, but the pass rate in the train/test sets is around 30%, not the 50% you mentioned. When using only right-hand data, the pass rate is ~43%, and dropped to ~35% while using the filtered grasping data.Could you give me some advice on how to improve the pass rate (>50%)?

[hxwork]

How many samples do you generate? Could you please detail your settings, such as the generative model you used, image resolution, and filtering thresholds? Does the raw DexYCB data mean "s0" or "s1" split?

[pjl013]

I use the model weights you provided (based on DexYCB s0 dataset) to generate images at the resolution of 128×128 , reconstruct them with your boosted MobRecon model, then filter the data follow your code’s thresholds: gen_MPJPE & gen_MPVPE < 25, gen_PA_MPJPE & gen_PA_MPVPE < 10. From the filtered DexYCB s0 grasping data, I generate 231,928 images (115,808 for the right hand), with 40,521 passing the filter (35.0%).

[hxwork]

Thanks for your detailed reply. I used the original MobRecon model (only trained on the DexYCB "s0" split), not the boosted one. I am not sure if this is why your ratio is lower than 50%.

[pjl013]

Thanks for your reply,I used your provided booster MobRecon model to filter the generated results, and with a filtering pass rate of around 45%, the final trained model was able to reach the performance reported in your paper.
However, when I used my own trained original MobRecon model (trained only on the DexYCB "s0" split) to filter the same generated results, the pass rate dropped to about 25%, even though the model performance itself did not show a significant difference.
Could you please advise what might be the reason behind this discrepancy in filtering effectiveness?

[hxwork]

Happy to see that you solved the problem. If I remember correctly, I didn't do anything special with the MobRecon model. Also, MobRecon is not the only choice; you can also try some recently released methods.

[pjl013]

Thank you for your previous response. After checking the details, I’ve managed to improve the final filtering pass rate to over 40%. However, I still have some questions regarding the implementation details of the cross sampling module. Specifically, I would like to understand how the final sampling probability mentioned in the paper is used to generate the similarity_sampled_gen_info.json file. Could you please clarify how this mapping is done?

[hxwork]

The cross-distribution sampling follows the following steps:

1. Construct the pose vectors, as shown in Figure 2 and Equation 4.
2. Compute the cosine similarity between the real-world and synthetic pose vectors, and form the sampling possibilities, as described in Equation 6 (higher similarity -> lower sampling possibility).
3. Sample the synthetic poses based on the possibilities.

Hope it clarifies.

[pjl013]

Thank you for your response. Additionally, during both the training and generation phases of the image generation model, I use 2D conditions with a resolution of 640×480, which are then cropped and transformed to 128×128. I would like to ask at which stage the 256×256 2D conditions mentioned in your paper are obtained and applied?

[hxwork]

Hi,

"2D conditions are 256256" means that the depth map, normal map, and the texture image are rendered in the resolution of 256256. When training the diffusion model to generate images, the resolution is set to 128*128.