Skip to content

Refactor workflow to stateless run manifest architecture and SA input design coordination#3708

Merged
infotroph merged 90 commits intoPecanProject:developfrom
divine7022:run-manifest
Feb 26, 2026
Merged

Refactor workflow to stateless run manifest architecture and SA input design coordination#3708
infotroph merged 90 commits intoPecanProject:developfrom
divine7022:run-manifest

Conversation

@divine7022
Copy link
Copy Markdown
Member

@divine7022 divine7022 commented Dec 7, 2025

Description

fixes #3692

  • updated run.write.configs to generate a runs_manifest.csv file containing run metadata (run_id, site_id, pft_name, trait, quantile, type) instead of appending to samples.Rdata.
  • samples.Rdata is now treated as a read-only "Master parameter definition" file.
  • updated write.sa.configs and write.ensemble.configs to return manifest dataframes.

fixes #3693

  • refactored read.sa.output and read.ensemble.output to pass the full vector of requested variables to read.output in a single call.
  • eliminated the inefficient inner loop that opened/closed netcdf files for every individual variable.

Additionally this PR separates input design generation for SA and ensemble runs, fixing dimension mismatch errors in multisite workflows.
previously both SA and ensemble used a single input_design matrix, causing failures when:

  • Ensemble needed N runs (e.g. 50)
  • SA needed 1 (median) + (traits * quantiles) runs (e.g. 145)

causes in dimension mismatch errors in input design coordination.

runModule.run.write.configs generates separate ens_input_design and sa_input_design matrices.
run.write.configs accepts both input_design_ens and input_design_sa parameters.
Old input_design parameter still works with deprecation warning.

Motivation and Context

Review Time Estimate

  • Immediately
  • Within one week
  • When possible

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My change requires a change to the documentation.
  • My name is in the list of CITATION.cff
  • I agree that PEcAn Project may distribute my contribution under any or all of
    • the same license as the existing code,
    • and/or the BSD 3-clause license.
  • I have updated the CHANGELOG.md.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

Comment thread base/workflow/R/run.write.configs.R Outdated
Comment thread base/workflow/R/run.write.configs.R Outdated
Comment thread base/workflow/R/run.write.configs.R Outdated
Comment thread base/workflow/R/run.write.configs.R Outdated
Comment thread base/workflow/R/run.write.configs.R Outdated
Comment thread modules/uncertainty/R/ensemble.R Outdated
Comment thread modules/uncertainty/R/sensitivity.R
Comment thread modules/uncertainty/R/sensitivity.R Outdated
Comment thread modules/uncertainty/R/sensitivity.R
Comment thread base/workflow/R/runModule.run.write.configs.R Outdated
@dlebauer dlebauer added the ccmmf issues and pre related to the ccmmf project label Jan 20, 2026
@dlebauer dlebauer dismissed infotroph’s stale review February 25, 2026 19:57

All requests changes and suggestions have been addressed

@dlebauer dlebauer enabled auto-merge February 25, 2026 19:57
@dlebauer
Copy link
Copy Markdown
Member

dlebauer commented Feb 25, 2026

failing build - looks like base/workflows/NAMESPACE needs to be re-auto-generated

devtools::document('base/workflows')

@dlebauer dlebauer added this pull request to the merge queue Feb 25, 2026
github-merge-queue Bot pushed a commit that referenced this pull request Feb 25, 2026
Refactor workflow to stateless run manifest architecture and SA input design coordination
Comment thread base/workflow/NEWS.md Outdated
@@ -1,8 +1,12 @@
# PEcAn.workflow 1.10.0.9000
# Unreleased
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Unreleased
PEcAn.workflow 1.10.0.9000

@infotroph infotroph removed this pull request from the merge queue due to a manual request Feb 25, 2026
Comment thread CHANGELOG.md Outdated
Comment thread CHANGELOG.md Outdated
Comment thread base/workflow/NEWS.md Outdated
Comment on lines +7 to +9
# PEcAn.workflow 1.10.0.9000

* Removed `tests/Rcheck_reference.log`, which was used to ignore historic check messages that have now been fixed.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.9000 means unreleased, so if a .9000 section exists you should always add entries there rather than start new sections above it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed 04816ad

Comment thread base/workflow/R/run.write.configs.R Outdated
Comment thread base/workflow/R/runModule.run.write.configs.R Outdated
Comment thread base/workflow/R/runModule.run.write.configs.R
#' result object with the design matrix in \code{$X} plus additional
#' components for Sobol index calculations.
#'
#' @details
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move above params

Comment thread base/workflow/R/runModule.run.write.configs.R
@infotroph infotroph added this pull request to the merge queue Feb 26, 2026
Merged via the queue into PecanProject:develop with commit dab62a6 Feb 26, 2026
19 of 25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

base ccmmf issues and pre related to the ccmmf project dockerfile modules tests

Projects

None yet

4 participants