Can't reproduce the performance

[ivylinden]

Hello,Thanks for the job,but when I run the code on the dataset wheat599, I can't reproduce the performance that reported in the paper

[RlChen0]

I can't reproduce the performance, too.I use the wheat599_pca, env1 only get 0.612 for 10-crossvilad(with SGD lr=0.01, batch_size=19, and weight_decay=0.01). The paper do not mention the conv layer type, the optimizer(SGD, AdamW and so on) and the weight decay.

[ivylinden]

I can't reproduce the performance, too.I use the wheat599_pca, env1 only get 0.612 for 10-crossvilad(with SGD lr=0.01, batch_size=19, and weight_decay=0.01). The paper do not mention the conv layer type, the optimizer(SGD, AdamW and so on) and the weight decay.
Are the results 0.612 you reproduced averaged out of ten fold?

[m-hrachov]

Hi! I also try to reproduce results. The batch size recommendation is different in the paper Supplementary and in the actual manual example (19 vs 28 respectively). Both haven't result in my case in something better than 0.59. By cross-validation, do you mean changing the seed value? Or something else?

I can't reproduce the performance, too.I use the wheat599_pca, env1 only get 0.612 for 10-crossvilad(with SGD lr=0.01, batch_size=19, and weight_decay=0.01). The paper do not mention the conv layer type, the optimizer(SGD, AdamW and so on) and the weight decay.
Are the results 0.612 you reproduced averaged out of ten fold?

[RlChen0]

Hi! @ivylinden, it has been a long time. But I'm quite sure it can reach the 0.612(mean of 10-fold).
hi @m-hrachov, you can check about scikit learn for details of cross-validtion.
By the way, I do not think the results can be reproduced.

[m-hrachov]

Hi! @ivylinden, it has been a long time. But I'm quite sure it can reach the 0.612(mean of 10-fold). hi @m-hrachov, you can check about scikit learn for details of cross-validtion. By the way, I do not think the results can be reproduced.

Thank you for your reply, but my question was rather what is (was) the seed for a split of your data? The script seems to perform one replication of a 10-fold CV (one split, one prediction). Changing the seed changes the split, and hence has an effect on prediction values.

For my project, I need a 5-fold CV performed 50 times, because my datasets are different in size. This is something I also don't see how to change.

I will definitely contact the authors, but maybe you have an idea how to proceed?

[RlChen0]

I do not think I will set a seed when reproducing other's work. But I might make a mistake, for the 0.59 is quite familiar to me.

…

---Original--- From: ***@***.***> Date: Wed, May 10, 2023 16:44 PM To: ***@***.***>; Cc: ***@***.******@***.***>; Subject: Re: [AIBreeding/DNNGP] Can't reproduce the performance (Issue #2) Hi! @ivylinden, it has been a long time. But I'm quite sure it can reach the 0.612(mean of 10-fold). hi @m-hrachov, you can check about scikit learn for details of cross-validtion. By the way, I do not think the results can be reproduced. Thank you for your reply, but my question was rather what is (was) the seed for a split of your data? The script seems to perform one replication of a 10-fold CV (one split, one prediction). Changing the seed changes the split, and hence has an effect on prediction values. For my project, I need a 5-fold CV performed 50 times, because my datasets are different in size. This is something I also don't see how to change. I will definitely contact the authors, but maybe you have an idea how to proceed? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

[z4668640]

The nature of deep learning models makes it impossible to reproduce the results 100%. At present, we can only guarantee that the same parameters on the same machine can achieve the same results. So it is necessary to adjust the training parameters when training models on different machines to achieve good results. When the h5 file is generated, this training result is fixed, that is, predictions with the same data on any machine can obtain the same result.