Skip to content

Add lockfiles to RAiDER #762

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
jlmaurer merged 7 commits into from
Aug 24, 2025
Merged

Add lockfiles to RAiDER #762

jlmaurer merged 7 commits into from
Aug 24, 2025

Conversation

@nate-kean
Copy link
Contributor

@nate-kean nate-kean commented Aug 22, 2025 •
edited
Loading

Description

Adds lockfiles to RAiDER via conda/conda-lock. Makes creating a RAiDER environment at least 10-40x faster.1

  1. Adds lockfiles generated against the current dev branch for every version of Python the project currently supports.
    • conda-lock.yml at project root is for our highest supported Python version, currently 3.12.
    • I put the others in .circleci/, since I expect they will be chiefly used for the CircleCI runners.
  2. Updates README and CONTRIBUTING to reflect how the addition of the lockfile system will affect developers.
  3. Configures the CircleCI runners to take advantage of the lockfiles.

Motivation and Context

  • This is primarily meant as an alternative solution to Switch to micromamba to fix CircleCI timeout #757 to fix the present CircleCI timeouts. Giving the CI runners access to a lockfile allows them to create their RAiDER environment without solving it each time, which for this project is a step that sometimes would take so long that it would singlehandedly elapse the runners' 60 minute timeout.
  • This will also allow developers, particularly those who are setting up RAiDER for the first time, to create local environments faster.
  • One caveat is that this will add friction while submitting changes to environment.yml: any change to this file will now require also regenerating the lockfiles to keep them up to date with it. In other package managers/languages, this is done automatically, but unfortunately with conda we will have to remember to do this ourselves.
    I wrote and included a script that aids in doing this for us. See https://github.com/dbekaert/RAiDER/pull/762/files#diff-a590e97322c09b68d2481c457c63f1611410317c627f1019375a93aa3d39c2d3

How Has This Been Tested?

The runners now all pass.

Type of change

  • Bug fix (cures CircleCI timeouts)
  • New feature (makes CircleCI tests and new env installs faster)
  • Breaking change (developers have an extra thing to do when modifying environment.yml)

Checklist:

  • I have added an explanation of what your changes do and why you'd like us to include them.
  • I have written new tests for your core changes, as applicable.
  • I have successfully ran tests with your changes locally.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.

Footnotes

  1. The "Create RAiDER environment" step used to take the CircleCI runners at least 15 minutes, and sometimes would time out after solving for 60 minutes. Now it takes about 1m30s including the extra time to install conda-lock. ↩

nate-kean added a commit to nate-kean/RAiDER that referenced this pull request Aug 22, 2025
@nate-kean
Update CHANGELOG.md for dbekaert#762
c7ec911
jlmaurer
jlmaurer reviewed Aug 23, 2025
View reviewed changes
@nate-kean nate-kean mentioned this pull request Aug 23, 2025
Closed
8 tasks
@nate-kean nate-kean marked this pull request as ready for review August 23, 2025 22:06
@jlmaurer jlmaurer merged commit b32655c into dbekaert:dev Aug 24, 2025
8 checks passed
nate-kean added a commit to nate-kean/RAiDER that referenced this pull request Aug 24, 2025
@nate-kean
Merge branch 'dev' into remove-erai
2d9a63e 
Squashed commit of the following:

commit b32655c
Merge: 5727651 3fd5cf5
Author: Jeremy Maurer <maurer.jeremy@gmail.com>
Date:   Sat Aug 23 21:24:38 2025 -0500

    Merge pull request dbekaert#762 from nate-kean/add-lock-file

    Add lockfiles to RAiDER

commit 3fd5cf5
Author: Nate Kean <natekean65535@gmail.com>
Date:   Sat Aug 23 16:20:28 2025 -0500

    Create regenerate-locks.py

commit a999d56
Author: Nate Kean <natekean65535@gmail.com>
Date:   Fri Aug 22 16:15:06 2025 -0500

    Update CHANGELOG.md for dbekaert#762

commit 56df113
Author: Nate Kean <natekean65535@gmail.com>
Date:   Fri Aug 22 16:10:52 2025 -0500

    Update documentation for lockfiles

commit d809892
Author: Nate Kean <natekean65535@gmail.com>
Date:   Sat Aug 23 16:32:32 2025 -0500

    Update CircleCI config to use lockfiles

commit d58b08b
Author: Nate Kean <natekean65535@gmail.com>
Date:   Sat Aug 23 16:32:59 2025 -0500

    Generate lockfiles for all other supported Python versions @ 5727651

commit a78ec5a
Author: Nate Kean <natekean65535@gmail.com>
Date:   Fri Aug 22 16:09:05 2025 -0500

    Generate lockfile for Python 3.12 @ 5727651

commit 0bb2068
Author: Nate Kean <natekean65535@gmail.com>
Date:   Thu Aug 21 20:36:44 2025 -0500

    Fix tests: use xarray for MERRA2._fetch()

commit 5727651
Merge: 784237b 6d12fd4
Author: Charlie Marshak <cmarshak@users.noreply.github.com>
Date:   Thu Aug 21 13:11:09 2025 -0700

    Merge pull request dbekaert#760 from dbekaert/herman-smale-branch

    Browse Imagery Fix

commit 6d12fd4
Merge: 784237b a1d5d3c
Author: Charlie Marshak <cmarshak@users.noreply.github.com>
Date:   Thu Aug 21 11:38:18 2025 -0700

    Merge pull request dbekaert#759 from jacquelynsmale/add_browse_s3_tag

    Add `browse` s3 tag to  uploaded `.png` files

commit a1d5d3c
Author: jacquelynsmale <jacquelyn.smale@gmail.com>
Date:   Wed Aug 20 16:29:46 2025 -0800

    fix 135

commit 0ee91ab
Author: jacquelynsmale <jacquelyn.smale@gmail.com>
Date:   Wed Aug 20 16:09:46 2025 -0800

    fix patch side effect

commit c7db467
Author: jacquelynsmale <jacquelyn.smale@gmail.com>
Date:   Wed Aug 20 15:12:35 2025 -0800

    fix path in test

commit 4704db3
Author: jacquelynsmale <jacquelyn.smale@gmail.com>
Date:   Wed Aug 20 13:52:37 2025 -0800

    switch to accept only Path

commit 5f929ba
Author: jacquelynsmale <jacquelyn.smale@gmail.com>
Date:   Tue Aug 19 15:46:07 2025 -0800

    fix changelog to reflect tag

commit 1a16ac8
Author: jacquelynsmale <jacquelyn.smale@gmail.com>
Date:   Tue Aug 19 15:36:23 2025 -0800

    add in get_tag_set to aws.py
@nate-kean nate-kean deleted the add-lock-file branch August 24, 2025 02:34
nate-kean added a commit to nate-kean/RAiDER that referenced this pull request Aug 24, 2025
@nate-kean
Merge branch 'dev' into accept-yaml-int-timestamp-format
eda584c 
Squashed commit of the following:

commit b32655c
Merge: 5727651 3fd5cf5
Author: Jeremy Maurer <maurer.jeremy@gmail.com>
Date:   Sat Aug 23 21:24:38 2025 -0500

    Merge pull request dbekaert#762 from nate-kean/add-lock-file

    Add lockfiles to RAiDER

commit 3fd5cf5
Author: Nate Kean <natekean65535@gmail.com>
Date:   Sat Aug 23 16:20:28 2025 -0500

    Create regenerate-locks.py

commit a999d56
Author: Nate Kean <natekean65535@gmail.com>
Date:   Fri Aug 22 16:15:06 2025 -0500

    Update CHANGELOG.md for dbekaert#762

commit 56df113
Author: Nate Kean <natekean65535@gmail.com>
Date:   Fri Aug 22 16:10:52 2025 -0500

    Update documentation for lockfiles

commit d809892
Author: Nate Kean <natekean65535@gmail.com>
Date:   Sat Aug 23 16:32:32 2025 -0500

    Update CircleCI config to use lockfiles

commit d58b08b
Author: Nate Kean <natekean65535@gmail.com>
Date:   Sat Aug 23 16:32:59 2025 -0500

    Generate lockfiles for all other supported Python versions @ 5727651

commit a78ec5a
Author: Nate Kean <natekean65535@gmail.com>
Date:   Fri Aug 22 16:09:05 2025 -0500

    Generate lockfile for Python 3.12 @ 5727651

commit 0bb2068
Author: Nate Kean <natekean65535@gmail.com>
Date:   Thu Aug 21 20:36:44 2025 -0500

    Fix tests: use xarray for MERRA2._fetch()

commit 5727651
Merge: 784237b 6d12fd4
Author: Charlie Marshak <cmarshak@users.noreply.github.com>
Date:   Thu Aug 21 13:11:09 2025 -0700

    Merge pull request dbekaert#760 from dbekaert/herman-smale-branch

    Browse Imagery Fix

commit 6d12fd4
Merge: 784237b a1d5d3c
Author: Charlie Marshak <cmarshak@users.noreply.github.com>
Date:   Thu Aug 21 11:38:18 2025 -0700

    Merge pull request dbekaert#759 from jacquelynsmale/add_browse_s3_tag

    Add `browse` s3 tag to  uploaded `.png` files

commit a1d5d3c
Author: jacquelynsmale <jacquelyn.smale@gmail.com>
Date:   Wed Aug 20 16:29:46 2025 -0800

    fix 135

commit 0ee91ab
Author: jacquelynsmale <jacquelyn.smale@gmail.com>
Date:   Wed Aug 20 16:09:46 2025 -0800

    fix patch side effect

commit c7db467
Author: jacquelynsmale <jacquelyn.smale@gmail.com>
Date:   Wed Aug 20 15:12:35 2025 -0800

    fix path in test

commit 4704db3
Author: jacquelynsmale <jacquelyn.smale@gmail.com>
Date:   Wed Aug 20 13:52:37 2025 -0800

    switch to accept only Path

commit 5f929ba
Author: jacquelynsmale <jacquelyn.smale@gmail.com>
Date:   Tue Aug 19 15:46:07 2025 -0800

    fix changelog to reflect tag

commit 1a16ac8
Author: jacquelynsmale <jacquelyn.smale@gmail.com>
Date:   Tue Aug 19 15:36:23 2025 -0800

    add in get_tag_set to aws.py

commit 784237b
Merge: 9eecf23 a259c66
Author: Jeremy Maurer <maurer.jeremy@gmail.com>
Date:   Thu Jul 3 20:38:01 2025 -0500

    Merge pull request dbekaert#746 from garlic-os/numpy-2.0-upgrade

    Upgrade to numpy 2.0

commit a259c66
Author: Nate Kean <14845347+garlic-os@users.noreply.github.com>
Date:   Wed Jul 2 17:56:17 2025 -0500

    Support both numpy v1 and v2

    - environment.yml: Remove requirement for numpy to be at least v2
    - Add polyfill for np.trapezoid for numpy v1

commit 4a5d4ea
Author: Nate Kean <14845347+garlic-os@users.noreply.github.com>
Date:   Mon Jun 30 21:14:40 2025 -0500

    Update CHANGELOG.md for dbekaert#746

commit 46e8883
Author: Nate Kean <14845347+garlic-os@users.noreply.github.com>
Date:   Mon Jun 30 21:07:39 2025 -0500

    Migrate np.trapz to np.trapezoid

commit 7d3c98c
Author: Nate Kean <14845347+garlic-os@users.noreply.github.com>
Date:   Mon Jun 30 21:06:01 2025 -0500

    Add ruff ruleset for numpy 2.0 migration

commit b5e518d
Author: Nate Kean <14845347+garlic-os@users.noreply.github.com>
Date:   Mon Jun 30 21:05:45 2025 -0500

    Set minimum numpy version to 2.0
nate-kean added a commit to nate-kean/RAiDER that referenced this pull request Aug 24, 2025
@nate-kean
Merge branch 'dev' into fix-for-era5-api-change
ad8e253 
Squashed commit of the following:

commit b32655c
Merge: 5727651 3fd5cf5
Author: Jeremy Maurer <maurer.jeremy@gmail.com>
Date:   Sat Aug 23 21:24:38 2025 -0500

    Merge pull request dbekaert#762 from nate-kean/add-lock-file

    Add lockfiles to RAiDER

commit 3fd5cf5
Author: Nate Kean <natekean65535@gmail.com>
Date:   Sat Aug 23 16:20:28 2025 -0500

    Create regenerate-locks.py

commit a999d56
Author: Nate Kean <natekean65535@gmail.com>
Date:   Fri Aug 22 16:15:06 2025 -0500

    Update CHANGELOG.md for dbekaert#762

commit 56df113
Author: Nate Kean <natekean65535@gmail.com>
Date:   Fri Aug 22 16:10:52 2025 -0500

    Update documentation for lockfiles

commit d809892
Author: Nate Kean <natekean65535@gmail.com>
Date:   Sat Aug 23 16:32:32 2025 -0500

    Update CircleCI config to use lockfiles

commit d58b08b
Author: Nate Kean <natekean65535@gmail.com>
Date:   Sat Aug 23 16:32:59 2025 -0500

    Generate lockfiles for all other supported Python versions @ 5727651

commit a78ec5a
Author: Nate Kean <natekean65535@gmail.com>
Date:   Fri Aug 22 16:09:05 2025 -0500

    Generate lockfile for Python 3.12 @ 5727651

commit 0bb2068
Author: Nate Kean <natekean65535@gmail.com>
Date:   Thu Aug 21 20:36:44 2025 -0500

    Fix tests: use xarray for MERRA2._fetch()

commit 5727651
Merge: 784237b 6d12fd4
Author: Charlie Marshak <cmarshak@users.noreply.github.com>
Date:   Thu Aug 21 13:11:09 2025 -0700

    Merge pull request dbekaert#760 from dbekaert/herman-smale-branch

    Browse Imagery Fix

commit 6d12fd4
Merge: 784237b a1d5d3c
Author: Charlie Marshak <cmarshak@users.noreply.github.com>
Date:   Thu Aug 21 11:38:18 2025 -0700

    Merge pull request dbekaert#759 from jacquelynsmale/add_browse_s3_tag

    Add `browse` s3 tag to  uploaded `.png` files

commit a1d5d3c
Author: jacquelynsmale <jacquelyn.smale@gmail.com>
Date:   Wed Aug 20 16:29:46 2025 -0800

    fix 135

commit 0ee91ab
Author: jacquelynsmale <jacquelyn.smale@gmail.com>
Date:   Wed Aug 20 16:09:46 2025 -0800

    fix patch side effect

commit c7db467
Author: jacquelynsmale <jacquelyn.smale@gmail.com>
Date:   Wed Aug 20 15:12:35 2025 -0800

    fix path in test

commit 4704db3
Author: jacquelynsmale <jacquelyn.smale@gmail.com>
Date:   Wed Aug 20 13:52:37 2025 -0800

    switch to accept only Path

commit 5f929ba
Author: jacquelynsmale <jacquelyn.smale@gmail.com>
Date:   Tue Aug 19 15:46:07 2025 -0800

    fix changelog to reflect tag

commit 1a16ac8
Author: jacquelynsmale <jacquelyn.smale@gmail.com>
Date:   Tue Aug 19 15:36:23 2025 -0800

    add in get_tag_set to aws.py
nate-kean added a commit to nate-kean/RAiDER that referenced this pull request Aug 24, 2025
@nate-kean
Merge branch 'dev' into parallelize-tests
def756e 
Squashed commit of the following:

commit b32655c
Merge: 5727651 3fd5cf5
Author: Jeremy Maurer <maurer.jeremy@gmail.com>
Date:   Sat Aug 23 21:24:38 2025 -0500

    Merge pull request dbekaert#762 from nate-kean/add-lock-file

    Add lockfiles to RAiDER

commit 3fd5cf5
Author: Nate Kean <natekean65535@gmail.com>
Date:   Sat Aug 23 16:20:28 2025 -0500

    Create regenerate-locks.py

commit a999d56
Author: Nate Kean <natekean65535@gmail.com>
Date:   Fri Aug 22 16:15:06 2025 -0500

    Update CHANGELOG.md for dbekaert#762

commit 56df113
Author: Nate Kean <natekean65535@gmail.com>
Date:   Fri Aug 22 16:10:52 2025 -0500

    Update documentation for lockfiles

commit d809892
Author: Nate Kean <natekean65535@gmail.com>
Date:   Sat Aug 23 16:32:32 2025 -0500

    Update CircleCI config to use lockfiles

commit d58b08b
Author: Nate Kean <natekean65535@gmail.com>
Date:   Sat Aug 23 16:32:59 2025 -0500

    Generate lockfiles for all other supported Python versions @ 5727651

commit a78ec5a
Author: Nate Kean <natekean65535@gmail.com>
Date:   Fri Aug 22 16:09:05 2025 -0500

    Generate lockfile for Python 3.12 @ 5727651

commit 0bb2068
Author: Nate Kean <natekean65535@gmail.com>
Date:   Thu Aug 21 20:36:44 2025 -0500

    Fix tests: use xarray for MERRA2._fetch()

commit 5727651
Merge: 784237b 6d12fd4
Author: Charlie Marshak <cmarshak@users.noreply.github.com>
Date:   Thu Aug 21 13:11:09 2025 -0700

    Merge pull request dbekaert#760 from dbekaert/herman-smale-branch

    Browse Imagery Fix

commit 6d12fd4
Merge: 784237b a1d5d3c
Author: Charlie Marshak <cmarshak@users.noreply.github.com>
Date:   Thu Aug 21 11:38:18 2025 -0700

    Merge pull request dbekaert#759 from jacquelynsmale/add_browse_s3_tag

    Add `browse` s3 tag to  uploaded `.png` files

commit a1d5d3c
Author: jacquelynsmale <jacquelyn.smale@gmail.com>
Date:   Wed Aug 20 16:29:46 2025 -0800

    fix 135

commit 0ee91ab
Author: jacquelynsmale <jacquelyn.smale@gmail.com>
Date:   Wed Aug 20 16:09:46 2025 -0800

    fix patch side effect

commit c7db467
Author: jacquelynsmale <jacquelyn.smale@gmail.com>
Date:   Wed Aug 20 15:12:35 2025 -0800

    fix path in test

commit 4704db3
Author: jacquelynsmale <jacquelyn.smale@gmail.com>
Date:   Wed Aug 20 13:52:37 2025 -0800

    switch to accept only Path

commit 5f929ba
Author: jacquelynsmale <jacquelyn.smale@gmail.com>
Date:   Tue Aug 19 15:46:07 2025 -0800

    fix changelog to reflect tag

commit 1a16ac8
Author: jacquelynsmale <jacquelyn.smale@gmail.com>
Date:   Tue Aug 19 15:36:23 2025 -0800

    add in get_tag_set to aws.py

commit 784237b
Merge: 9eecf23 a259c66
Author: Jeremy Maurer <maurer.jeremy@gmail.com>
Date:   Thu Jul 3 20:38:01 2025 -0500

    Merge pull request dbekaert#746 from garlic-os/numpy-2.0-upgrade

    Upgrade to numpy 2.0

commit a259c66
Author: Nate Kean <14845347+garlic-os@users.noreply.github.com>
Date:   Wed Jul 2 17:56:17 2025 -0500

    Support both numpy v1 and v2

    - environment.yml: Remove requirement for numpy to be at least v2
    - Add polyfill for np.trapezoid for numpy v1

commit 4a5d4ea
Author: Nate Kean <14845347+garlic-os@users.noreply.github.com>
Date:   Mon Jun 30 21:14:40 2025 -0500

    Update CHANGELOG.md for dbekaert#746

commit 46e8883
Author: Nate Kean <14845347+garlic-os@users.noreply.github.com>
Date:   Mon Jun 30 21:07:39 2025 -0500

    Migrate np.trapz to np.trapezoid

commit 7d3c98c
Author: Nate Kean <14845347+garlic-os@users.noreply.github.com>
Date:   Mon Jun 30 21:06:01 2025 -0500

    Add ruff ruleset for numpy 2.0 migration

commit b5e518d
Author: Nate Kean <14845347+garlic-os@users.noreply.github.com>
Date:   Mon Jun 30 21:05:45 2025 -0500

    Set minimum numpy version to 2.0
nate-kean added a commit to nate-kean/RAiDER that referenced this pull request Aug 26, 2025
@nate-kean
Update CHANGELOG.md for dbekaert#762
28f6a32
@jlmaurer jlmaurer mentioned this pull request Sep 6, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jlmaurer jlmaurer jlmaurer approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@nate-kean @jlmaurer