Release v0.13 #488

gtrevisan · 2025-11-07T17:13:54Z

C-MOD

DIII-D

Use local scratch rather than tmp for DIII-D #494

HBT-EP

MAST

Introduce MAST workflows #509

Framework

Automation

Longer period before stale #515

Documentation

Dependencies

Bumps [actions/checkout](https://github.com/actions/checkout) from 5 to 6. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v5...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* use localscratch rather than tmp if available * fixup! use localscratch rather than tmp if available

* add generic methods to docs, tweak menu, rename and tweak others * Specify "C-Mod" in the Disruption Parameter Descriptions table header --------- Co-authored-by: yumouwei <46117079+yumouwei@users.noreply.github.com>

* update id for Trevisan APS-DPP 2025 * Update wietfeldt APS 2025 poster number * Add Wei APS-DPP 2025 poster * Update reference link APS Zapata Cornejo Link corrected. I accept also your suggestion of removing the hyphen from my second name, if that still OK. * Updated zkeith APS 2025 poster number * fixed poster ID for AR Saperstein * fixed text --------- Co-authored-by: hwietfeldt <henry.wietfeldt@gmail.com> Co-authored-by: yumouwei <46117079+yumouwei@users.noreply.github.com> Co-authored-by: Enrique Zapata <zapata_e@psfc.mit.edu> Co-authored-by: ZanderKeith <zkeith@mit.edu> Co-authored-by: Alex R. Saperstein <saperstein@psfc.mit.edu>

* add a dummy mds class as 3rd layer * prefer MDSplus, warn about mdsthin, disregard if no connection * add a couple of generic MDSplus exceptions

* remove layouts from readme * revamp installation section in readme * tweak install * fixup! revamp installation section in readme * add link to contributors page * separate installation from execution in section B * minor tweaks

* unpin pandas and netcdf4 * lock deps * black! disruption_py/core/physics_method/params.py * black! disruption_py/machine/cmod/physics.py * black! disruption_py/machine/generic/physics.py * black! disruption_py/settings/shotlist_setting.py * black! docs/examples/custom_time_setting.py

* Add hbtep fastcam shotlist and descriptions * Update disruption_py/data/README.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update readme * Add publications that used the hbt fast camera shotlist * remove BOM bits * add shortcut for HBT-EP shotlist * tweak HBT-EP publications --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: gtrevisan <gtrevisan@users.noreply.github.com>

* add get_metadata utility function * use get_metadata in logging header * add metadata as dataset attributes * add data vars attributes from config * explicit attrs logic in xr.concat * catch AttributeError, too * fixup! catch AttributeError, too * prefer dict.get over try-except

@gtrevisan

* Add CMOD attributes * build imas url as derived attribute * drop imas description * prefer units to unit * add further imas specifier to gaps * use shortcut for imas * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @gtrevisan Co-authored-by: Gregorio L. Trevisan <gtrevisan@users.noreply.github.com> * reviewed and updated cmod's config.toml file * add script to check imas urls * fixup! add script to check imas urls * remove unnecessary import * fix some CMOD imas paths * Correct imas addresses and typos. Add "/value" when necessary * Fix another typo found by copilot --------- Co-authored-by: gtrevisan <gtrevisan@users.noreply.github.com> Co-authored-by: AlexSaperstein <saperstein@psfc.mit.edu> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* lock deps * try MDSplus first, then fall back onto mdsthin * harmonize MDSplus imports and exceptions for C-MOD * harmonize MDSplus imports and exceptions for DIII-D * fixup! harmonize MDSplus imports and exceptions for DIII-D * try to fix str conversions for arrays * add skeleton of MAST tokamak * Add compatibility with mdsthin * Update with first data from FAIR MAST * Remove print statement * Re-add mast as a method * Update pyproj.toml, poetry/uv locks * Add support for different timebases to MAST * Add test info to config * Refactor s3 -> xarray backend, make more generic * add MAST to readme * add MAST methods to docs * add MAST docs to menu * Rough first pass at adding a few parameter groups * Update with datatree backend * Change xarray to use path. More generic * fixup! add MAST methods to docs * add doc reference to MAST physics methods * Minimise formatting changes git diff * Add verbose level across all loggers * Fix missed rebase/merge * Update fsspec deps * minimize diff further * Add ip threshold params for MAST Also fix one or two small inconsistencies * Update with first data from FAIR MAST * Refactor s3 -> xarray backend, make more generic * Update fsspec deps * Remove unused import * Remove print statement * Update with more gracefully missing data handling * Update config * Update doc strings * Small optimisation for opening datatree * Clean up efit calls * Fix linting errors * disable pylint warning * Fix mds error, fix cleanup method * Add fix linting warnings * Add missing method, missing exception flag * Fix missing mdsplus flag, fix interp for efit * Add getting started doc page * drop safe_cast from utils/misc * Add a few more physics props * Add parquet support * Handle missing xarray data * Update docs * fix merge issues * minimize diff for tokamak resolution * rename builtin to physics methods * run linters * fix too-many-return-statements in tokamak resolution * bugfix wrong class for mds_conn * drop unnecessary dummy driver * add python shebangs * adapt mdsplus example for MAST * adapt sql example for MAST * adapt efit example for MAST * run linter on MAST config * cannot test any MAST column atm * cache MAST get_ip_parameters for generic get_time_domain * add MAST ip for test_output_setting.py * fix mast test shotlist * skip some tests for MAST * revert logger changes for better debugging * add MAST to remote tests * tweak new dependencies * lock deps * readd EAST return statement * Fix copilot code review issues. * refactor xarray backend as xr * unpin pandas and netcdf4 * lock deps * Small updates from code review * Add a few more test shots * fix imports * toml-sort! disruption_py/machine/mast/config.toml * black! disruption_py/settings/shotlist_setting.py * Revert "black! disruption_py/settings/shotlist_setting.py" This reverts commit 7a758cc. * Add attributes, rename variables * Rename dalpha to d_alpha to make it consistent with D3D h_alpha * Change disruption_time_start to include the ramp up phase in the returned dataset * Swap shot number, sort toml * minor changes to mast toml * downgrade opening data message to trace * build imas url as derived attribute * use shortcut for imas * add script to check imas urls * fixup! add script to check imas urls * remove unnecessary import * fix some MAST imas paths * Update disruption_py/machine/mast/config.toml Co-authored-by: Gregorio L. Trevisan <gtrevisan@users.noreply.github.com> * Update disruption_py/machine/mast/config.toml Co-authored-by: Gregorio L. Trevisan <gtrevisan@users.noreply.github.com> * update pulse schedule ip reference following C-MOD 3eb275a * third time's the charm * use shot_msg when reporting path failure, drop PID * sort efit properties, readd wmhd * log resolved tokamak as info * fixup! log resolved tokamak as info --------- Co-authored-by: gtrevisan <gtrevisan@users.noreply.github.com> Co-authored-by: Samuel Jackson <samuel.jackson@ukaea.ac.uk> Co-authored-by: yumouwei <46117079+yumouwei@users.noreply.github.com>

Copilot

Pull request overview

Release prep for v0.13, expanding DisruptionPy to support the open-access MAST workflow (S3/Xarray-based I/O) alongside existing MDSplus-based devices, plus associated documentation/CI/dependency updates.

Changes:

Add MAST device support (Tokamak enum, config, physics/EFIT methods) and an Xarray/S3-backed connection implementation.
Add richer provenance/metadata + per-variable attribute propagation to xarray outputs (incl. IMAS URL generation).
Update docs navigation/references, CI workflows (new tokamak matrix entries), examples/tests skips, and dependency pins/extras.

Reviewed changes

Copilot reviewed 57 out of 59 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
tests/test_time_setting.py	Skip unsupported time-setting tests for MAST/HBT-EP.
tests/test_retrieval_settings.py	Skip framework retrieval-settings test for MAST/HBT-EP.
tests/test_output_setting.py	Adjust output column choice for MAST/HBT-EP.
scripts/check_imas.py	New script to validate IMAS doc links in xarray outputs.
pyproject.toml	Add S3/zarr deps; update pandas/netcdf4 constraints; bump dev version.
mkdocs.yml	Reorganize physics methods nav; add MAST pages.
examples/sql.py	Skip SQL example for MAST/HBT-EP.
examples/mdsplus.py	Skip MDSplus example for MAST.
examples/efit.py	Add MAST EFIT example path/expected shape.
docs/usage/sql_database.md	Remove outdated link line.
docs/usage/physics_methods/physics_method_reference.md	Update physics method reference links (generic + per-device + MAST).
docs/usage/physics_methods/mast_method_reference.md	New mkdocstrings page for MAST methods.
docs/usage/physics_methods/hbtep_method_reference.md	Rename heading to “HBT-EP Physics Methods”.
docs/usage/physics_methods/generic_method_reference.md	New mkdocstrings page for generic methods.
docs/usage/physics_methods/east_method_reference.md	Rename heading to “EAST Physics Methods”.
docs/usage/physics_methods/disruption_parameters_reference.md	Clarify parameter table is C‑Mod-specific.
docs/usage/physics_methods/d3d_method_reference.md	Rename heading to “DIII-D Physics Methods”.
docs/usage/physics_methods/cmod_method_reference.md	Rename heading to “C-Mod Physics Methods”.
docs/examples/custom_time_setting.py	Minor formatting cleanup.
disruption_py/workflow.py	Support xarray connection selection; improve import/tokamak logging.
disruption_py/settings/time_setting.py	Add MAST timebase support via MastUtilMethods (EFIT/disruption/Ip).
disruption_py/settings/shotlist_setting.py	Support parquet shotlists; add hbtep_fastcam included shotlist.
disruption_py/settings/output_setting.py	Explicit `combine_attrs` on xarray concat to avoid warning.
disruption_py/settings/log_settings.py	Use centralized get_metadata() for startup/source logging.
disruption_py/machine/tokamak.py	Add Tokamak.MAST; change env resolution fallback behavior.
disruption_py/machine/method_holders.py	Include MAST method holders (with generic).
disruption_py/machine/mast/util.py	New MAST helper methods + interpolation wrapper.
disruption_py/machine/mast/physics.py	New MAST physics methods using XarrayConnection.
disruption_py/machine/mast/efit.py	New MAST EFIT parameter retrieval using XarrayConnection.
disruption_py/machine/mast/config.toml	New MAST configuration (xarray endpoint + attributes + tests/time params).
disruption_py/machine/mast/init.py	Register MAST method holders.
disruption_py/machine/hbtep/physics.py	Docstring naming fix (“HBT-EP”).
disruption_py/machine/generic/physics.py	Add MAST handling for generic time-domain logic.
disruption_py/machine/east/util.py	Add shebang / header consistency.
disruption_py/machine/east/physics.py	Fix incorrect docstring device name.
disruption_py/machine/d3d/util.py	Add shebang / header consistency.
disruption_py/machine/cmod/physics.py	Minor cleanup of tuple unpacking.
disruption_py/machine/cmod/config.toml	Update MDSplus host; add extensive physics attribute metadata.
disruption_py/inout/xr.py	New XarrayConnection for zarr/netcdf datatree access (incl. S3 endpoint).
disruption_py/inout/sql.py	Reduce dummy-DB noise (debug instead of warning).
disruption_py/inout/mds.py	Make MDSplus import fallback more robust (MDSplus → mdsthin → dummy).
disruption_py/data/hbtep_fastcam.csv	New included HBT-EP fastcam shotlist.
disruption_py/data/README.md	Document new HBT-EP fastcam shotlist.
disruption_py/core/utils/misc.py	Add get_metadata(); improve git hash + localscratch temp folder behavior.
disruption_py/core/retrieval_manager.py	Avoid MDS-specific nickname wiring for non-MDS connections.
disruption_py/core/physics_method/runner.py	Add dataset + variable attrs; set merge compat explicitly.
disruption_py/core/physics_method/params.py	Minor formatting.
disruption_py/config.py	Auto-generate IMAS documentation URLs for attribute metadata.
REFERENCES.md	Add database references; update APS-DPP IDs.
README.md	Revamp device/install docs; add MAST; update CLI guidance.
INSTALL.md	Update install guidance (uv lock/export mention).
.github/workflows/tests.yml	Add HBT-EP/MAST matrix; update checkout; adjust tunnel setup; conditionalize steps.
.github/workflows/stale.yml	Increase stale window; reduce schedule frequency.
.github/workflows/lint.yml	Bump actions/checkout to v6.
.github/workflows/install.yml	Bump actions/checkout to v6.
.github/workflows/docs.yml	Bump actions/checkout to v6.
.github/workflows/build.yml	Bump actions/checkout to v6.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-10T15:41:42Z

disruption_py/inout/xr.py

+    def __init__(
+        self,
+        file_path: str,
+        file_ext: str = "zarr",
+        endpoint_url: str | None = None,
+    ):
+        if file_path is None:
+            return
+
+        self.endpoint_url = endpoint_url
+        self.file_path = file_path
+        self.file_ext = file_ext
+        self.data_tree: xr.DataTree | None = None


XarrayConnection.__init__ returns early when file_path is None, leaving the instance without required attributes like file_path/file_ext/data_tree. This will surface later as attribute errors in folder_path/get_shot_connection/get_data. Prefer raising a ValueError for missing config, or fully initialize a “disabled” connection object with explicit error messages on use.

@samueljackson92 you can push a fix if you want, but IMHO we can skip over this since we will revamp the back-ends soon.
@samc24 for awareness.

Copilot · 2026-02-10T15:41:42Z

disruption_py/inout/xr.py

+    def get_data(self, shot_id: int, path: str, return_xarray: bool = False):
+        """Get data from the connection."""
+        if self.data_tree is None:
+            self.get_shot_connection(shot_id)
+


get_data ignores the provided shot_id once data_tree is set: it only opens a new tree when self.data_tree is None. If the same XarrayConnection instance is reused across multiple shots without an explicit get_shot_connection call, this can silently return data from the wrong shot. Track the currently-open shot_id and reopen when it changes (or make get_data always ensure the correct shot is loaded).

@samueljackson92 you can push a fix if you want, but IMHO we can skip over this since we will revamp the back-ends soon.
@samc24 for awareness.

Copilot · 2026-02-10T15:41:42Z

disruption_py/inout/xr.py

+    def cleanup(self):
+        """Cleanup the connection."""
+
+    def reconnect(self):
+        """Reconnect the connection."""


cleanup()/reconnect() are empty, but PhysicsMethodParams.cleanup() always calls mds_conn.cleanup(). For XarrayConnection, this likely leaves open file/store handles. Implement cleanup() to close the opened datatree/store (and clear data_tree), and implement reconnect() consistently (e.g., by forcing a reopen on next access).

@samueljackson92 you can push a fix if you want, but IMHO we can skip over this since we will revamp the back-ends soon.
@samc24 for awareness.

Copilot · 2026-02-10T15:41:43Z

disruption_py/machine/mast/config.toml

+[mast.physics.attributes.beta_p]
+description = "Beta poloidal (EFIT)."
+imas = "/equilibrium/time_slice(itime)/global_quantities/beta_pol"
+unitss = "dimensionless"


Typo in attribute metadata key: unitss should be units. As-is, this attribute won’t be picked up by the code that propagates per-variable attrs into xarray outputs.

Suggested change

unitss = "dimensionless"

units = "dimensionless"

ouch, how did we miss this? @yumouwei

Copilot · 2026-02-10T15:41:43Z

disruption_py/machine/mast/config.toml

+
+[mast.physics.attributes.dipprog_dt]
+description = "Time derivative of the programmed plasma current."
+units = "A"


dipprog_dt is a time derivative (computed via np.gradient in code), but the config declares units = "A". This should be A/s to avoid publishing incorrect metadata.

Suggested change

units = "A"

units = "A/s"

ouch, how did we miss this? @yumouwei

Copilot · 2026-02-10T15:41:43Z

disruption_py/machine/mast/config.toml

+[mast.physics.attributes.triagbot]
+description = "Bottom triangularity (EFIT)."
+imas = "/equilibrium/time_slice(itime)/boundary/triangularity_lower"
+units = "dimensionless"
+
+[mast.physics.attributes.triagtop]


The triangularity attribute keys in the config (triagbot/triagtop) don’t match the variables produced by the MAST EFIT method (tribot/tritop). This prevents attribute propagation for these outputs and leaves dead config entries. Align the config keys with the actual column names returned by the physics methods.

Suggested change

[mast.physics.attributes.triagbot]

description = "Bottom triangularity (EFIT)."

imas = "/equilibrium/time_slice(itime)/boundary/triangularity_lower"

units = "dimensionless"

[mast.physics.attributes.triagtop]

[mast.physics.attributes.tribot]

description = "Bottom triangularity (EFIT)."

imas = "/equilibrium/time_slice(itime)/boundary/triangularity_lower"

units = "dimensionless"

[mast.physics.attributes.tritop]

ouch, how did we miss this? @yumouwei

Copilot · 2026-02-10T15:41:44Z

disruption_py/machine/mast/util.py

+        if len(x) != len(y) and np.isnan(y).all():
+            # if all y are NaN (is a missing signal)
+            # just return array of NaNs with same shape as x_new
+            return np.full_like(x_new, np.nan)
+        return interp1(x, y, x_new)


interpolate_1d’s missing-signal guard is incorrect: it only returns NaNs when both len(x) != len(y) and y is all-NaN. If y is all-NaN but lengths match (a likely case when get_data returns [nan]), this will fall through to interp1 and raise. Consider handling np.isnan(y).all() (and also len(x) < 2) first, and separately raising a clear error when len(x) != len(y) for non-missing signals.

@samueljackson92 do take this one, please!

Copilot · 2026-02-10T15:41:44Z

disruption_py/machine/mast/efit.py

+    }
+
+    @staticmethod
+    @physics_method(columns=list(efit_properties.keys()))


MastEfitMethods.get_efit_parameters is missing tokamak=Tokamak.MAST in the @physics_method decorator. Without this, the method is marked as runnable for any tokamak (metadata-wise), which can lead to unexpected selection if it’s ever included outside the MAST holder list or used as a custom method.

Suggested change

@physics_method(columns=list(efit_properties.keys()))

@physics_method(columns=list(efit_properties.keys()), tokamak="MAST")

Copilot · 2026-02-10T15:41:44Z

disruption_py/workflow.py

 def get_mdsplus_class(
    tokamak: Tokamak = None,
-) -> ProcessMDSConnection:
+) -> ProcessMDSConnection | XarrayConnection:
    """
    Get the MDSplus connection for the tokamak.
    """
    tokamak = resolve_tokamak_from_environment(tokamak)
-    return ProcessMDSConnection.from_config(tokamak=tokamak)
+
+    inout_cfg = config(tokamak).inout
+    if "mds" in inout_cfg:
+        return ProcessMDSConnection.from_config(tokamak=tokamak)
+
+    if "xarray" in inout_cfg:
+        return XarrayConnection.from_config(tokamak=tokamak)
+
+    raise ValueError("No valid MDSplus or xarray connection found.")


get_mdsplus_class now returns either ProcessMDSConnection or XarrayConnection, which is a breaking/ambiguous API (callers may assume the returned object has .conn and is MDSplus-backed). Consider renaming to something connection-agnostic (e.g., get_data_connection) or keeping get_mdsplus_class MDS-only and adding a new function for xarray/S3 connections.

@samueljackson92 you can push a fix if you want, but IMHO we can skip over this since we will revamp the back-ends soon.
@samc24 for awareness.

disruption_py/machine/cmod/physics.py

bump dev version to v0.13.0-dev

1fe3744

gtrevisan added the release New project releases label Nov 7, 2025

gtrevisan and others added 20 commits December 1, 2025 15:22

update deps (#491)

2e13125

suppress stderr when looking up commit (#492)

937d7bf

Use local scratch rather than tmp for DIII-D (#494)

6f57312

* use localscratch rather than tmp if available * fixup! use localscratch rather than tmp if available

Add generic methods to docs (#495)

a6479be

* add generic methods to docs, tweak menu, rename and tweak others * Specify "C-Mod" in the Disruption Parameter Descriptions table header --------- Co-authored-by: yumouwei <46117079+yumouwei@users.noreply.github.com>

More robust MDSplus fallback logic (#499)

0a4fb40

* add a dummy mds class as 3rd layer * prefer MDSplus, warn about mdsthin, disregard if no connection * add a couple of generic MDSplus exceptions

downgrade warning to debug for empty db config (#503)

036e607

lock deps (#502)

8e2ffc1

Revamp installation readmes (#501)

6bbb9d6

* remove layouts from readme * revamp installation section in readme * tweak install * fixup! revamp installation section in readme * add link to contributors page * separate installation from execution in section B * minor tweaks

set explicit compat for xr.merge (#507)

17cc6a7

add HBT-EP to remote tests (#510)

d4ec78f

add databases to references (#504)

24dc809

update C-MOD mdsplus server (#513)

4feeb4e

consider stale after 6 mo (#515)

b56ef10

gtrevisan requested a review from Copilot February 10, 2026 15:32

Copilot started reviewing on behalf of gtrevisan February 10, 2026 15:32 View session

Copilot AI reviewed Feb 10, 2026

View reviewed changes

	@physics_method(columns=list(efit_properties.keys()))
	@physics_method(columns=list(efit_properties.keys()), tokamak="MAST")

Release v0.13 #488

Are you sure you want to change the base?

Release v0.13 #488

Uh oh!

Conversation

gtrevisan commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

C-MOD

DIII-D

HBT-EP

MAST

Framework

Automation

Documentation

Dependencies

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gtrevisan commented Nov 7, 2025 •

edited

Loading