Remove some randomization in the example and in the code base.

[lionelkusch]

I try to improve the reproducibility of the example by setting seeds in the example and better management of the random generator in the code base.

However, there is still some randomisation in PermutationFeatureImportance and in Model_X_Knockoff.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.07%. Comparing base (6eb643c) to head (c8734d1).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #344      +/-   ##
==========================================
- Coverage   98.08%   98.07%   -0.01%     
==========================================
  Files          22       22              
  Lines        1147     1144       -3     
==========================================
- Hits         1125     1122       -3     
  Misses         22       22              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bthirion]

The PR LGTM overall.

[jpaillard]

I can't reproduce the reproducibility issue you describe for CFI and knockoffs. Can you point more precisely to the problematic example/test?

[lionelkusch]

I can't reproduce the reproducibility issue you describe for CFI and knockoffs. Can you point more precisely to the problematic example/test?

The problem is with PFI and CFI. For PFI: if you run example: plot_pitfalls_permutation_importance.pyin parallel (the file: tools/examples/debugger_script/try_reproducibility.py), the second figure is always a bit different. Forknockoff, if you run example: plot_knockoff_aggregation.py` in parallel, the figure for the empirical FDR is always a bit different.

The difference is quite minor but there is always some variations.

[jpaillard]

Thanks for the details. I could reproduce the issue.

It seems that the problem comes from using nested parallel loops: parallel calls of try_reproducibility.run_joblib, which has inner parallelization of plot_knockoff_aggregation.single_run

The inner processes might unpredictably inherit some state of the parent.

To fix it you can use Parallel(n_jobs=<nb_of_jobs>, require='sharedmem') or simply set n_jobs=1 in try_reproducibility.

[lionelkusch]

I fixed the issue for plot_knockoff_aggregation.
I was missing setting some random seed in LassoCV.

[lionelkusch]

For plot_pitfalls_permutation_importance, I find that seaborn requires a seed for the barplot. This was the cause of the difference.

[lionelkusch]

There is still a bit of uncontrolled randomness for plot_knockoff_aggregation but it will be easier to debug it after reformatting with the new API.

[lionelkusch]

I updated the management of the seed because I forgot that it's better to use a range of values for setting a seed than to use a random generator. The problem is because the random generator can generate 2 times the same numbers.
I merged the branch with main.

[jpaillard]

Thank you.
I have one last comment.

[bthirion]

I have some very minor comments.
There is no need to make KFold random in the examples (Better use ShuffleSplit if we want a random splitter, but this is the user's choice then).

[lionelkusch]

Last review before merging.

[jpaillard]

To be consistent with the changes suggested in #360 we should avoid showcasing the use of RandomState in examples and replace them with np.random.default_rng

[jpaillard]

Looks good, thx