[1766][1743][1332] lint and unit-test fix#1802
Conversation
bee363e to
6e83cf3
Compare
6e83cf3 to
9d075ae
Compare
tjhunter
left a comment
There was a problem hiding this comment.
@simone99n thanks for tackling these issues. There is an issue with your logic, let's discuss it.
| ruff check --target-version py312 \ | ||
| --fix \ | ||
| src/ scripts/ packages/ | ||
| echo "LINTING WORKFLOW STARTING..." |
There was a problem hiding this comment.
this is not correct. lint is here to run the formatting commands, the -check are here to ensure that you have run the commands first. The output for example should not mention a reformatting:
lint-check...
Installed 1 package in 93ms
1 file reformatted, 102 files left unchanged <-- should not reformat
| echo "LINTING WORKFLOW STARTING..." | ||
|
|
||
| echo "lint-check..." | ||
| ./scripts/actions.sh lint-check |
There was a problem hiding this comment.
add && chaining to ensure they all run and fail together
| echo "lint-check..." | ||
| ./scripts/actions.sh lint-check | ||
| echo "lint-fix..." | ||
| ./scripts/actions.sh lint-fix |
| echo "toml-checking..." | ||
| ./scripts/actions.sh toml-check | ||
| echo "type-checking..." | ||
| ./scripts/actions.sh type-check |
There was a problem hiding this comment.
we can leave this one as warning (it should not fail the linting)
9d075ae to
f945561
Compare
|
@simone99n let's whitelist these few lines for the time being and fix them later. |
|
@tjhunter done. |
tjhunter
left a comment
There was a problem hiding this comment.
It is better to put a message such asth #noqa on the lines that are concerned rather than the whole file, because then people can add more problematic code in the meantime in the file. Let's merge for now.
* Improve support for latent losses (#1963) * Revert 2D rope to false by default (#1967) Set to True by accident * Implementation of DataReaderMesh (#1840) * First implementation of DataReaderMesh * Move to datareaders extra * ruff * ruff2 * Undo ruff * undo auto-linting * correct typo in eval config (#1971) * Added all-physical-streams option and x/y axis limits (#1972) * Added all-physical-streams option and x/y axis limits * Fix * Changed flag for all streams * Removed old code * moved metric parsing to eval_from_config (#1977) Co-authored-by: buschow1 <buschow1@jwlogin04.juwels> * Fixed integration test (#1980) * [1974][model] Add fallback to config loading (#1985) * Add fallback to config loading * Adjust error message to be not misleading * Homegenize naming convention * Introduce bias/diff maps and animations (#1912) * Introduce bias/diff maps and animations * minor correction * Changes based on review * Introduce "plot_bias" in evaluation configuration (#1986) * Fixed index ordering to not have shuffled output (#1982) * Fixed idxs_inv to revert data point shuffeling * Fixed output handling * Handling of empty data case, addressing reviewer comment * [1893][eval] csvreader cleanup (#1906) * refactor csvreader * check if dataarray size is 0 * fix and use original logic for empty data * linting fixes * revert assertions back * [1890][eval] Move MergeReader to own module (#1892) * move mergereader * use assertions only * implement scoring for the sub-steps within the forecast window (#1896) * work in progress * working for forecast_step * working version * restore no valid times option * lint * Rename scale_z_channels to _scale_z_channels * fix 1 sample bug * Remove points_per_sample from ReaderOutput Remove points_per_sample from ReaderOutput return. * remove n_point_per_sample * fix lead time coord in compute_scores * lint * fix integration test * Fix integration test single stream (#1996) * fix test single * change yml extension and minor fixes --------- Co-authored-by: cosi1 <cosi1@jwlogin21.juwels> Co-authored-by: cosi1 <cosi1@jwb0001.juwels> * [1907][eval] clean up wegen_reader.py (#1911) * clean up wegen_reader.py * remove exception * consistent reader naming * add blank line * use assertions only * make names consistent * Merge branch 'develop' into 1907-wegenreader-cleanup * revert is_regular --------- Co-authored-by: iluise <72020169+iluise@users.noreply.github.com> Co-authored-by: Ilaria Luise <luise.ilaria@gmail.com> * [1888][eval] Refactor Reader class (#1889) * refactor Reader * use assertion only * fix npp atms --------- Co-authored-by: iluise <72020169+iluise@users.noreply.github.com> Co-authored-by: Ilaria Luise <luise.ilaria@gmail.com> * [1975][model] Load model path from private repo instead of json (#1998) * Load model path from private repo instead of json * Lint * Script to compute spatial autocorrelation of structured/unstructured datasets (#1955) * standalone script to compute spatial autocorrelation of variables in a structured or unstructured dataset * remove commits that should be in pr 1951 * lint * addressed comments * removed last failure returning 500km default, and moved to packages science * updated a note * rename autocorrelation script * update example usage * Correct EMA halflife_steps calculation with rampup_ratio (#2001) Corrected rampup calculation: https://github.com/NVlabs/edm2/blob/4bf8162f601bcc09472ce8a32dd0cbe8889dc8fc/training/phema.py#L145 Co-authored-by: Wael <wael.almikaeel.95@gmail.com> * Reduce verbosity of output during inference and evaluation (#2006) * Fix incorrect length in validation progress bar * Removing too verbose output * [1766][1743][1332] lint and unit-test fix (#1802) * [1766][1742] fix lint and unit-test * [1766] fix linter * [1766] lint local and global consistent * [1332] add script to detect bad functions (getattr) * code quality: lint and bad functions * [1766] disable some checks * [1877] Script to populate PR labels from linked issues (#1878) * script * branch * more dirs * typo * enable * Fixed bug in linear embedding (#2012) * Adding forecast_steps feature to plot_train (#2010) * Adding forecast_steps feature to plot_train * Renamed arguement to conform to hyphen convention * Added forecast step to filename --------- Co-authored-by: Seb Hickman <56727418+shmh40@users.noreply.github.com> --------- Co-authored-by: Christian Lessig <christian.lessig@ecmwf.int> Co-authored-by: Seb Hickman <56727418+shmh40@users.noreply.github.com> Co-authored-by: Kacper Nowak <kacper.nowak@awi.de> Co-authored-by: Till Hauer <till@web-hauer.de> Co-authored-by: s6sebusc <49226935+s6sebusc@users.noreply.github.com> Co-authored-by: buschow1 <buschow1@jwlogin04.juwels> Co-authored-by: Matthias Karlbauer <matthias.karlbauer@ecmwf.int> Co-authored-by: Savvas Melidonis <79579567+SavvasMel@users.noreply.github.com> Co-authored-by: Michael Tarnawa <18899420+mtar@users.noreply.github.com> Co-authored-by: iluise <72020169+iluise@users.noreply.github.com> Co-authored-by: pierluigicosi <91318382+pierluigicosi@users.noreply.github.com> Co-authored-by: cosi1 <cosi1@jwlogin21.juwels> Co-authored-by: cosi1 <cosi1@jwb0001.juwels> Co-authored-by: Ilaria Luise <luise.ilaria@gmail.com> Co-authored-by: Wael <wael.almikaeel.95@gmail.com> Co-authored-by: Simone Norberti <63310821+simone99n@users.noreply.github.com> Co-authored-by: Timothy Hunter <tim.hunter@ecmwf.int>
* Improve support for latent losses (#1963) * Revert 2D rope to false by default (#1967) Set to True by accident * Implementation of DataReaderMesh (#1840) * First implementation of DataReaderMesh * Move to datareaders extra * ruff * ruff2 * Undo ruff * undo auto-linting * correct typo in eval config (#1971) * Added all-physical-streams option and x/y axis limits (#1972) * Added all-physical-streams option and x/y axis limits * Fix * Changed flag for all streams * Removed old code * moved metric parsing to eval_from_config (#1977) Co-authored-by: buschow1 <buschow1@jwlogin04.juwels> * Fixed integration test (#1980) * [1974][model] Add fallback to config loading (#1985) * Add fallback to config loading * Adjust error message to be not misleading * Homegenize naming convention * Introduce bias/diff maps and animations (#1912) * Introduce bias/diff maps and animations * minor correction * Changes based on review * Introduce "plot_bias" in evaluation configuration (#1986) * Fixed index ordering to not have shuffled output (#1982) * Fixed idxs_inv to revert data point shuffeling * Fixed output handling * Handling of empty data case, addressing reviewer comment * [1893][eval] csvreader cleanup (#1906) * refactor csvreader * check if dataarray size is 0 * fix and use original logic for empty data * linting fixes * revert assertions back * [1890][eval] Move MergeReader to own module (#1892) * move mergereader * use assertions only * implement scoring for the sub-steps within the forecast window (#1896) * work in progress * working for forecast_step * working version * restore no valid times option * lint * Rename scale_z_channels to _scale_z_channels * fix 1 sample bug * Remove points_per_sample from ReaderOutput Remove points_per_sample from ReaderOutput return. * remove n_point_per_sample * fix lead time coord in compute_scores * lint * fix integration test * Fix integration test single stream (#1996) * fix test single * change yml extension and minor fixes --------- Co-authored-by: cosi1 <cosi1@jwlogin21.juwels> Co-authored-by: cosi1 <cosi1@jwb0001.juwels> * [1907][eval] clean up wegen_reader.py (#1911) * clean up wegen_reader.py * remove exception * consistent reader naming * add blank line * use assertions only * make names consistent * Merge branch 'develop' into 1907-wegenreader-cleanup * revert is_regular --------- Co-authored-by: iluise <72020169+iluise@users.noreply.github.com> Co-authored-by: Ilaria Luise <luise.ilaria@gmail.com> * [1888][eval] Refactor Reader class (#1889) * refactor Reader * use assertion only * fix npp atms --------- Co-authored-by: iluise <72020169+iluise@users.noreply.github.com> Co-authored-by: Ilaria Luise <luise.ilaria@gmail.com> * [1975][model] Load model path from private repo instead of json (#1998) * Load model path from private repo instead of json * Lint * Script to compute spatial autocorrelation of structured/unstructured datasets (#1955) * standalone script to compute spatial autocorrelation of variables in a structured or unstructured dataset * remove commits that should be in pr 1951 * lint * addressed comments * removed last failure returning 500km default, and moved to packages science * updated a note * rename autocorrelation script * update example usage * Correct EMA halflife_steps calculation with rampup_ratio (#2001) Corrected rampup calculation: https://github.com/NVlabs/edm2/blob/4bf8162f601bcc09472ce8a32dd0cbe8889dc8fc/training/phema.py#L145 Co-authored-by: Wael <wael.almikaeel.95@gmail.com> * Reduce verbosity of output during inference and evaluation (#2006) * Fix incorrect length in validation progress bar * Removing too verbose output * [1766][1743][1332] lint and unit-test fix (#1802) * [1766][1742] fix lint and unit-test * [1766] fix linter * [1766] lint local and global consistent * [1332] add script to detect bad functions (getattr) * code quality: lint and bad functions * [1766] disable some checks * [1877] Script to populate PR labels from linked issues (#1878) * script * branch * more dirs * typo * enable * Fixed bug in linear embedding (#2012) * Adding forecast_steps feature to plot_train (#2010) * Adding forecast_steps feature to plot_train * Renamed arguement to conform to hyphen convention * Added forecast step to filename --------- Co-authored-by: Seb Hickman <56727418+shmh40@users.noreply.github.com> * add noise distribution plotting * plot noise distribution and decoded noised tokens * fix noise level in validation to p_mean * rm noise and token distribution plotting --------- Co-authored-by: Christian Lessig <christian.lessig@ecmwf.int> Co-authored-by: Seb Hickman <56727418+shmh40@users.noreply.github.com> Co-authored-by: Kacper Nowak <kacper.nowak@awi.de> Co-authored-by: Till Hauer <till@web-hauer.de> Co-authored-by: s6sebusc <49226935+s6sebusc@users.noreply.github.com> Co-authored-by: buschow1 <buschow1@jwlogin04.juwels> Co-authored-by: Matthias Karlbauer <matthias.karlbauer@ecmwf.int> Co-authored-by: Savvas Melidonis <79579567+SavvasMel@users.noreply.github.com> Co-authored-by: Michael Tarnawa <18899420+mtar@users.noreply.github.com> Co-authored-by: iluise <72020169+iluise@users.noreply.github.com> Co-authored-by: pierluigicosi <91318382+pierluigicosi@users.noreply.github.com> Co-authored-by: cosi1 <cosi1@jwlogin21.juwels> Co-authored-by: cosi1 <cosi1@jwb0001.juwels> Co-authored-by: Ilaria Luise <luise.ilaria@gmail.com> Co-authored-by: Wael <wael.almikaeel.95@gmail.com> Co-authored-by: Simone Norberti <63310821+simone99n@users.noreply.github.com> Co-authored-by: Timothy Hunter <tim.hunter@ecmwf.int>
* grad accumulation first try * better memory * revert grad accumulation * Jk/mk/mh/diffusion single sample fix val (#2045) * Improve support for latent losses (#1963) * Revert 2D rope to false by default (#1967) Set to True by accident * Implementation of DataReaderMesh (#1840) * First implementation of DataReaderMesh * Move to datareaders extra * ruff * ruff2 * Undo ruff * undo auto-linting * correct typo in eval config (#1971) * Added all-physical-streams option and x/y axis limits (#1972) * Added all-physical-streams option and x/y axis limits * Fix * Changed flag for all streams * Removed old code * moved metric parsing to eval_from_config (#1977) Co-authored-by: buschow1 <buschow1@jwlogin04.juwels> * Fixed integration test (#1980) * [1974][model] Add fallback to config loading (#1985) * Add fallback to config loading * Adjust error message to be not misleading * Homegenize naming convention * Introduce bias/diff maps and animations (#1912) * Introduce bias/diff maps and animations * minor correction * Changes based on review * Introduce "plot_bias" in evaluation configuration (#1986) * Fixed index ordering to not have shuffled output (#1982) * Fixed idxs_inv to revert data point shuffeling * Fixed output handling * Handling of empty data case, addressing reviewer comment * [1893][eval] csvreader cleanup (#1906) * refactor csvreader * check if dataarray size is 0 * fix and use original logic for empty data * linting fixes * revert assertions back * [1890][eval] Move MergeReader to own module (#1892) * move mergereader * use assertions only * implement scoring for the sub-steps within the forecast window (#1896) * work in progress * working for forecast_step * working version * restore no valid times option * lint * Rename scale_z_channels to _scale_z_channels * fix 1 sample bug * Remove points_per_sample from ReaderOutput Remove points_per_sample from ReaderOutput return. * remove n_point_per_sample * fix lead time coord in compute_scores * lint * fix integration test * Fix integration test single stream (#1996) * fix test single * change yml extension and minor fixes --------- Co-authored-by: cosi1 <cosi1@jwlogin21.juwels> Co-authored-by: cosi1 <cosi1@jwb0001.juwels> * [1907][eval] clean up wegen_reader.py (#1911) * clean up wegen_reader.py * remove exception * consistent reader naming * add blank line * use assertions only * make names consistent * Merge branch 'develop' into 1907-wegenreader-cleanup * revert is_regular --------- Co-authored-by: iluise <72020169+iluise@users.noreply.github.com> Co-authored-by: Ilaria Luise <luise.ilaria@gmail.com> * [1888][eval] Refactor Reader class (#1889) * refactor Reader * use assertion only * fix npp atms --------- Co-authored-by: iluise <72020169+iluise@users.noreply.github.com> Co-authored-by: Ilaria Luise <luise.ilaria@gmail.com> * [1975][model] Load model path from private repo instead of json (#1998) * Load model path from private repo instead of json * Lint * Script to compute spatial autocorrelation of structured/unstructured datasets (#1955) * standalone script to compute spatial autocorrelation of variables in a structured or unstructured dataset * remove commits that should be in pr 1951 * lint * addressed comments * removed last failure returning 500km default, and moved to packages science * updated a note * rename autocorrelation script * update example usage * Correct EMA halflife_steps calculation with rampup_ratio (#2001) Corrected rampup calculation: https://github.com/NVlabs/edm2/blob/4bf8162f601bcc09472ce8a32dd0cbe8889dc8fc/training/phema.py#L145 Co-authored-by: Wael <wael.almikaeel.95@gmail.com> * Reduce verbosity of output during inference and evaluation (#2006) * Fix incorrect length in validation progress bar * Removing too verbose output * [1766][1743][1332] lint and unit-test fix (#1802) * [1766][1742] fix lint and unit-test * [1766] fix linter * [1766] lint local and global consistent * [1332] add script to detect bad functions (getattr) * code quality: lint and bad functions * [1766] disable some checks * [1877] Script to populate PR labels from linked issues (#1878) * script * branch * more dirs * typo * enable * Fixed bug in linear embedding (#2012) * Adding forecast_steps feature to plot_train (#2010) * Adding forecast_steps feature to plot_train * Renamed arguement to conform to hyphen convention * Added forecast step to filename --------- Co-authored-by: Seb Hickman <56727418+shmh40@users.noreply.github.com> * add noise distribution plotting * plot noise distribution and decoded noised tokens * fix noise level in validation to p_mean * rm noise and token distribution plotting --------- Co-authored-by: Christian Lessig <christian.lessig@ecmwf.int> Co-authored-by: Seb Hickman <56727418+shmh40@users.noreply.github.com> Co-authored-by: Kacper Nowak <kacper.nowak@awi.de> Co-authored-by: Till Hauer <till@web-hauer.de> Co-authored-by: s6sebusc <49226935+s6sebusc@users.noreply.github.com> Co-authored-by: buschow1 <buschow1@jwlogin04.juwels> Co-authored-by: Matthias Karlbauer <matthias.karlbauer@ecmwf.int> Co-authored-by: Savvas Melidonis <79579567+SavvasMel@users.noreply.github.com> Co-authored-by: Michael Tarnawa <18899420+mtar@users.noreply.github.com> Co-authored-by: iluise <72020169+iluise@users.noreply.github.com> Co-authored-by: pierluigicosi <91318382+pierluigicosi@users.noreply.github.com> Co-authored-by: cosi1 <cosi1@jwlogin21.juwels> Co-authored-by: cosi1 <cosi1@jwb0001.juwels> Co-authored-by: Ilaria Luise <luise.ilaria@gmail.com> Co-authored-by: Wael <wael.almikaeel.95@gmail.com> Co-authored-by: Simone Norberti <63310821+simone99n@users.noreply.github.com> Co-authored-by: Timothy Hunter <tim.hunter@ecmwf.int> * implement zero-weight physical loss * plot config file * deletion * merged with noised plotting * revert some changes * revert change for fixed eta * revert changes * revert more changes * fixed noised plotting * config changes --------- Co-authored-by: Julian Kuehnert <Jubeku@users.noreply.github.com> Co-authored-by: Christian Lessig <christian.lessig@ecmwf.int> Co-authored-by: Seb Hickman <56727418+shmh40@users.noreply.github.com> Co-authored-by: Kacper Nowak <kacper.nowak@awi.de> Co-authored-by: Till Hauer <till@web-hauer.de> Co-authored-by: s6sebusc <49226935+s6sebusc@users.noreply.github.com> Co-authored-by: buschow1 <buschow1@jwlogin04.juwels> Co-authored-by: Matthias Karlbauer <matthias.karlbauer@ecmwf.int> Co-authored-by: Savvas Melidonis <79579567+SavvasMel@users.noreply.github.com> Co-authored-by: Michael Tarnawa <18899420+mtar@users.noreply.github.com> Co-authored-by: iluise <72020169+iluise@users.noreply.github.com> Co-authored-by: pierluigicosi <91318382+pierluigicosi@users.noreply.github.com> Co-authored-by: cosi1 <cosi1@jwlogin21.juwels> Co-authored-by: cosi1 <cosi1@jwb0001.juwels> Co-authored-by: Ilaria Luise <luise.ilaria@gmail.com> Co-authored-by: Wael <wael.almikaeel.95@gmail.com> Co-authored-by: Simone Norberti <63310821+simone99n@users.noreply.github.com> Co-authored-by: Timothy Hunter <tim.hunter@ecmwf.int>
Description
./script/actions.sh unit-testshould restore the original venv after execution #1743)getattr(), and add support to add more bad-functions (Lint - disable using getattr in the project #1332)Issue Number
./script/actions.sh unit-testshould restore the original venv after execution #1743Checklist before asking for review
./scripts/actions.sh lint./scripts/actions.sh unit-test./scripts/actions.sh integration-testlaunch-slurm.py --time 60