Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
127 commits
Select commit Hold shift + click to select a range
d3a2245
resolved conflict
mariusaurus Dec 1, 2025
0189d52
update changelog
dallasfoster Sep 19, 2025
f11b18b
move seed initialization and fix dxwrapper tests
dallasfoster Sep 19, 2025
d063760
tempest extremes diagnostic model
mariusaurus Sep 15, 2025
a4d2544
error message
mariusaurus Sep 25, 2025
c1cdca0
testing if TE is available and works
mariusaurus Sep 25, 2025
016f16b
started working on support for batch sizes >1, currently works for bs 1
mariusaurus Sep 25, 2025
68e33b5
halfway to larger batch support
mariusaurus Sep 25, 2025
7bd60e1
enabling TE for batch sizes of >1. async version seems to work as wel…
mariusaurus Sep 26, 2025
3b0c00e
option to pass file names to TE connector
mariusaurus Sep 29, 2025
1e9bbe8
array equal test
mariusaurus Sep 29, 2025
d6be6dd
first stable try
mariusaurus Oct 7, 2025
1e9b275
support for per-member parallel execution and lets user controll max …
mariusaurus Oct 7, 2025
b5f5c18
precommit hooks
mariusaurus Oct 7, 2025
af8bc71
vibe-coded some tests, need to be hand-tested and selected
mariusaurus Oct 7, 2025
a9fd2bc
vibe-coded some tests, need to be hand-tested and selected
mariusaurus Oct 7, 2025
526e6bf
passing all pre-commit tests, still need to sub-select tests as there…
mariusaurus Oct 7, 2025
c3258d9
subselected tests
mariusaurus Oct 7, 2025
3fd145d
install doc
mariusaurus Oct 8, 2025
c26f453
throwing an error in case cleanup is not called before object goes ou…
mariusaurus Oct 8, 2025
d2a8e4a
custom depenmdency failure message for TE
mariusaurus Oct 8, 2025
0ab6d67
moved tensor tiling and concatenation to utils
mariusaurus Oct 8, 2025
8ca3fae
enable setting fcn3 random seed
dallasfoster Sep 19, 2025
e93932e
add proper noise handling for fcn3
dallasfoster Sep 19, 2025
bc9e3ac
fix linting and test issues
dallasfoster Sep 19, 2025
2685f90
update lockfile
dallasfoster Sep 19, 2025
e3a4e3d
move seed initialization and fix dxwrapper tests
dallasfoster Sep 19, 2025
1dec990
tc tracking pipeline
mariusaurus Oct 13, 2025
02945f1
update
mariusaurus Oct 13, 2025
f89efe3
updated uv.lock
mariusaurus Oct 13, 2025
92896eb
seems to work now
mariusaurus Oct 13, 2025
9e0e106
wind gust from HRRR analysis
mariusaurus Oct 14, 2025
5550ad9
minor updates
mariusaurus Oct 14, 2025
869b8fe
stability test
mariusaurus Oct 24, 2025
343d035
version check for torch-harmonics import
mariusaurus Oct 24, 2025
037a5a7
addressed greptile comments
mariusaurus Oct 30, 2025
608315b
time import
mariusaurus Oct 30, 2025
3f55702
comma
mariusaurus Oct 30, 2025
7fefe5b
merged main
mariusaurus Jan 9, 2026
dc0cd79
updated env
mariusaurus Jan 9, 2026
70361a7
moved tempest_extremes
mariusaurus Jan 9, 2026
e172e50
wip
mariusaurus Jan 9, 2026
edb1978
exploring aifs ensemble capability
mariusaurus Jan 13, 2026
e3b2ed0
Merge branch 'main' into mkoch/tc_tracking
mariusaurus Jan 13, 2026
8c3c848
thread issue with writing to netcdf in threads
mariusaurus Jan 13, 2026
d41351e
automated testing of writing TE files and their reproducibility. bug …
mariusaurus Jan 14, 2026
780e458
README for tc_hunt test
mariusaurus Jan 14, 2026
4360ea4
second test for extracting historic data
mariusaurus Jan 14, 2026
8f766f5
added aux data for tests
mariusaurus Jan 14, 2026
5bd90c6
merged main
mariusaurus Jan 15, 2026
f3286fb
test for reference track extraction
mariusaurus Jan 15, 2026
3638bcc
wip
mariusaurus Jan 16, 2026
5b7ae87
track plotting notebook
mariusaurus Jan 16, 2026
3002fd0
field and track notebook
mariusaurus Jan 16, 2026
0b968b3
plotting tracks and fields notebook
mariusaurus Jan 19, 2026
67f7fc6
case study notebook
mariusaurus Jan 20, 2026
2df89f7
Merge branch 'main' into mkoch/tc_tracking
mariusaurus Jan 20, 2026
0f046a1
REAMEs and markdowns in notebooks
mariusaurus Jan 20, 2026
fbdbc9f
drafted readme
mariusaurus Jan 21, 2026
b495b31
first iteration over readme
mariusaurus Jan 22, 2026
ad4717c
final touches README
mariusaurus Jan 22, 2026
34fda29
...gif
mariusaurus Jan 22, 2026
78dc346
wip
mariusaurus Jan 23, 2026
b76a7d3
Merge branch 'main' into mkoch/tc_tracking
mariusaurus Jan 23, 2026
fb7f207
verified plotting for west-pacific
mariusaurus Jan 23, 2026
d4926cf
final touches
mariusaurus Jan 23, 2026
f0fcff0
removed some configs
mariusaurus Jan 23, 2026
2c8eaf3
pyproject project name
mariusaurus Jan 23, 2026
b4af03e
README comment about conainer build time
mariusaurus Jan 23, 2026
1bf9850
removed TE from models/dx/__init__
mariusaurus Jan 23, 2026
99ae8ba
moving a bracket around
mariusaurus Jan 23, 2026
6048522
Merge branch 'main' into mkoch/tc_tracking
mariusaurus Feb 10, 2026
5da7854
updated base container
mariusaurus Feb 18, 2026
ff5dcba
Merge branch 'main' into mkoch/tc_tracking
mariusaurus Feb 19, 2026
acd0354
Merge branch 'mkoch/tc_tracking' of github.com:mariusaurus/earth2stud…
mariusaurus Feb 19, 2026
f067969
merged main, might be broken
mariusaurus Mar 10, 2026
6c85a95
Merge branch 'mkoch/tc_tracking' of github.com:mariusaurus/earth2stud…
mariusaurus Mar 10, 2026
2f4b495
fixed some bugs to be in line with new main
mariusaurus Mar 11, 2026
84609e3
split plan
mariusaurus Mar 11, 2026
b53a3eb
first round of LLM reduction
mariusaurus Mar 11, 2026
8032f20
doc strings and type hints for generate_ensembles.py
mariusaurus Mar 11, 2026
5edac8b
doc strings and type hints for tempest_extremes.py
mariusaurus Mar 11, 2026
41c3248
doc strings and type hints for src/utils.py
mariusaurus Mar 11, 2026
9c2104d
doc strings and type hints for data
mariusaurus Mar 11, 2026
6f8c3d5
orography from HF
mariusaurus Mar 18, 2026
3167014
git lfs in dockerfile
mariusaurus Mar 18, 2026
16d18fe
final touches
mariusaurus Mar 18, 2026
aa360b2
final touches
mariusaurus Mar 18, 2026
6d58f4f
Merge branch 'main' into mkoch/tc_hunt_1
mariusaurus Mar 18, 2026
6bcbb5a
reverted fcn3 changes
mariusaurus Mar 18, 2026
3a9a5cf
updated dockerfile to latest physics nemo container and reduced numbe…
mariusaurus Mar 18, 2026
342caf5
first round of pre-commit hooks
mariusaurus Mar 18, 2026
731231b
licenses
mariusaurus Mar 18, 2026
e0b01b2
renamed some files and removed tempest_extremes testing
mariusaurus Mar 18, 2026
cecdcb7
TE list files unique by time stamp
mariusaurus Mar 19, 2026
a2c7510
Te timeouts
mariusaurus Mar 19, 2026
82fa33c
improved catching of TE failures
mariusaurus Mar 19, 2026
98b80c8
satisfying the greptile
mariusaurus Mar 19, 2026
a897748
TE workers
mariusaurus Mar 19, 2026
1e59b6e
collecting all tasks in wait_for_completion and in cleanup
mariusaurus Apr 8, 2026
63bac87
re-using wait for compeltion and cosmetics
mariusaurus Apr 8, 2026
01b1a00
removed some stale code, replaced use_ram with shm location, some cle…
mariusaurus Apr 8, 2026
f099214
replace TempestExtremes module-level globals with singleton and flatt…
mariusaurus Apr 8, 2026
c86bfe0
removed SFNO support, replaced print statements with loggers.
mariusaurus Apr 9, 2026
2252140
refactored pyproject.toml
mariusaurus Apr 9, 2026
29c75a7
Merge branch 'main' into mkoch/tc_hunt_1
mariusaurus Apr 9, 2026
a6c0b3b
updated install notes
mariusaurus Apr 14, 2026
3d3adcd
added reproducibility
mariusaurus Apr 14, 2026
8198ac1
store overwrite
mariusaurus Apr 14, 2026
fca4d39
pt 3 init
mariusaurus Apr 15, 2026
0434fe7
updated install docs
mariusaurus Apr 15, 2026
eb341e7
Merge branch 'main' into mkoch/tc_hunt_1
mariusaurus Apr 15, 2026
f3cd9dc
Merge branch 'mkoch/tc_hunt_1' into mkoch/tc_hunt_2
mariusaurus Apr 15, 2026
05feb7c
Merge branch 'mkoch/tc_hunt_2' into mkoch/tc_hunt_3
mariusaurus Apr 15, 2026
d9cdda9
Merge branch 'main' into mkoch/tc_hunt_1
NickGeneva Apr 16, 2026
58a361f
setuptools version
mariusaurus Apr 16, 2026
02a900c
optional AIFS/FCN3 deps
mariusaurus Apr 16, 2026
36de792
Merge branch 'mkoch/tc_hunt_1' into mkoch/tc_hunt_2
mariusaurus Apr 16, 2026
5cdb8b9
Merge branch 'mkoch/tc_hunt_2' into mkoch/tc_hunt_3
mariusaurus Apr 16, 2026
c7d01c7
data sources in config
mariusaurus Apr 16, 2026
cdca059
greptile feedback
mariusaurus Apr 16, 2026
4105167
plotting
mariusaurus Apr 16, 2026
6d634e0
updated jupyter cells
mariusaurus May 20, 2026
45cf91d
merged main
mariusaurus May 20, 2026
4f3ae2b
claude review
mariusaurus May 22, 2026
8e3853c
removed some dead code
mariusaurus May 22, 2026
1e7397d
Merge branch 'main' into mkoch/tc_hunt_4
mariusaurus May 22, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 21 additions & 25 deletions recipes/tc_tracking/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,13 +43,12 @@ process.
- [2.1 Generate Ensemble](#21-generate-ensemble)
- [2.2 Reproduce Individual Ensemble Members](#22-reproduce-individual-ensemble-members)
- [2.3 Extract Reference Tracks from ERA5](#23-extract-reference-tracks-from-era5)
3. [Visualisation](#3-visualisation) *(coming soon)*
3. [Visualisation](#3-visualisation)
4. [TempestExtremes Integration](#4-tempestextremes-integration)
5. [Example Workflow](#5-example-workflow)
- [5.1 Extract Baseline](#51-extract-baseline-optional)
- [5.2 Produce Ensemble Forecasts](#52-produce-ensemble-forecasts)
- [5.3 Analyse Tracks](#53-analyse-tracks)
*(coming soon)*
- [5.4 Reproduce Interesting Members](#54-reproduce-interesting-members-to-extract-fields)

## 1. Setting up the Environment
Expand Down Expand Up @@ -570,29 +569,36 @@ and a warning is logged.

## 3. Visualisation

> [!Note]
> Visualisation tools will be available in a future update.

<!-- markdownlint-disable MD033 -->
<details>
<summary>Preview</summary>

Two Jupyter notebooks are provided in `./plotting` for
analysing and visualising tropical cyclone tracking results:
Two [JupyText](https://jupytext.readthedocs.io/) notebook scripts are
provided in `./plotting` for analysing and visualising tropical cyclone
tracking results:

- **`tracks_slayground.ipynb`**: Ensemble track analysis
- **`tracks_slayground_notebook.py`**: Ensemble track analysis
including spaghetti plots (trajectory visualisation),
absolute and relative intensity metrics (wind speed,
MSLP), comparisons against ERA5 reference tracks and
IBTrACS observations, extreme value statistics, and
error moment analysis over lead time.

- **`plot_tracks_n_fields.ipynb`**: Create animated
- **`plot_tracks_n_fields_notebook.py`**: Create animated
visualisations of storm tracks overlaid on atmospheric
field data.

</details>
<!-- markdownlint-enable MD033 -->
Both scripts can be run as plain Python files or converted to Jupyter
notebooks via JupyText. From the recipe root:

```bash
cd plotting
jupytext --to notebook tracks_slayground_notebook.py
jupytext --to notebook plot_tracks_n_fields_notebook.py
jupyter notebook tracks_slayground.ipynb
```

`./plotting/` must be the working directory because the notebooks import
their helpers (`analyse_n_plot`, `plotting_helpers`, `data_handling`) by
bare module name. See `./plotting/README.md` for the full layout, including
the `analyse_n_plot.py` batch entry point for running the analysis across
many storms at once.

## 4. TempestExtremes Integration

Expand Down Expand Up @@ -708,13 +714,6 @@ trajectories.

### 5.3 Analyse Tracks

> [!Note]
> Visualisation tools will be available in a future update.

<!-- markdownlint-disable MD033 -->
<details>
<summary>Preview</summary>

Visualise the results using the notebook
`plotting/tracks_slayground.ipynb`.

Expand All @@ -739,9 +738,6 @@ tru_track_dir = '/path/to/outputs_reference_tracks'
# tru_track_dir = '/path/to/test/aux_data'
```

</details>
<!-- markdownlint-enable MD033 -->

### 5.4 Reproduce Interesting Members to Extract Fields

Suppose that after conducting the above analysis you want
Expand Down
2 changes: 2 additions & 0 deletions recipes/tc_tracking/plotting/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
plots/
*.ipynb
50 changes: 50 additions & 0 deletions recipes/tc_tracking/plotting/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# Analysing and Plotting TC Tracks

## Notebooks

- **`plot_tracks_n_fields_notebook.py`** – Plotting tracks and fields for
individual ensemble members
- **`tracks_slayground_notebook.py`** – Analysing and plotting complete
tracks from a full ensemble run for a case study on a given storm

Both scripts are [JupyText](https://jupytext.readthedocs.io/) Python files
that can be run directly or converted to Jupyter notebooks:

```bash
# Convert to a Jupyter notebook
jupytext --to notebook plot_tracks_n_fields_notebook.py
jupytext --to notebook tracks_slayground_notebook.py
```

> [!Note]
> The notebooks and the modules in this directory use bare module names
> (`from analyse_n_plot import ...`, `from plotting_helpers import ...`).
> Run them with `plotting/` as the working directory so Python can resolve
> those imports:
>
> ```bash
> cd recipes/tc_tracking/plotting
> jupyter notebook tracks_slayground.ipynb
> ```

## Scripts and Library Modules

- **`analyse_n_plot.py`** – Batch entry point. Drives
`analyse_individual_storms` (one plot set per storm) and
`analyse_ensemble_of_storms` (error metrics aggregated across many
storms). Run with `python analyse_n_plot.py` after editing the storm
selection and paths near the bottom of the file.
- **`data_handling.py`** – Library: track ingestion, matching against the
reference, ensemble averaging on the sphere, and lead-time error
metrics. Imported by the notebooks; not intended to be run directly.
- **`plotting_helpers.py`** – Library: the individual plotting routines
(spaghetti, intensities over time, histograms, error metrics). Also
imported by the notebooks; not intended to be run directly.

## Additional Information

- Each notebook specifies at the beginning what data is required and how to
produce it using the TC tracking pipeline.
- All plotting and analysis routines take a `time_step` keyword argument
defaulting to 6 h, matching the stock FCN3 and AIFS-ENS configurations.
Override it if you run the upstream pipeline at a different cadence.
Loading