Add mpas analysis model vs model

[xylar]

Summary

Objectives:

• Add model vs. model (also called test vs. reference or main vs. control) comparison support for MPAS-Analysis

Issue resolution:

• Fixes [Feature]: model-vs-model for mpas-analysis #771

Select one: This pull request is...

• a new feature: increment the minor version

Please fill out either the "Small Change" or "Big Change" section (the latter includes the numbered subsections), and delete the other.

Small Change

• To merge, I will use "Squash and merge". That is, this change should be a single commit.
• Logic: I have visually inspected the entire pull request myself.
• Pre-commit checks: All the pre-commits checks have passed.

[xylar]

@forsyth2, would you like me to include bumping the minor version here? I assume you might want to hold off and bundle various minor updates together but let me know.

[xylar]

Testing

I created a control run in:

/lcrc/group/e3sm/ac.xylar/zppy/mpas_analysis_model_vs_model/v3.LR.historical_0051

closely based on the example config file for that simulation. I disabled all analysis other than MPAS-Analysis

Then, I set up a main run in:

/lcrc/group/e3sm/ac.xylar/zppy/mpas_analysis_model_vs_model/v3.LR.historical_0091

pointing to the control run as the reference_data_path. All analysis ran successfully except indexNino34, which revealed a bug that I have fixed in MPAS-Dev/MPAS-Analysis#1134 (only related to model vs. model runs).

I then reran the main run in:

/lcrc/group/e3sm/ac.xylar/zppy/mpas_analysis_model_vs_model/v3.LR.historical_0091_el_nino_fix

pointing to my MPAS-Analysis branch with the bug fix. This time, all analysis ran successfully and the comparison output can be seen at:
https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.xylar/zppy/mpas_analysis_model_vs_model/v3.LR.historical_0091_el_nino_fix/v3.LR.historical_0091/mpas_analysis/ts_1850-2014_climo_1985-2014/

[xylar]

@forsyth2 and @erinethomas, this is ready for you to have a look at.

[xylar]

The bug in the ENSO model vs. model analysis is unfortunate and zppy users will either have to point to their own environment with the latest MPAS-Analysis from the develop branch or disable ENSO in the model vs. model run for the time being, e.g. by adding no_index to the generate list:

[mpas_analysis]
generate = 'all', 'no_landIceCavities', 'no_BGC', 'no_icebergs', 'no_min', 'no_max', 'no_sose', 'no_waves', 'no_eke', 'no_climatologyMapAntarcticMelt', 'no_regionalTSDiagrams', 'no_timeSeriesAntarcticMelt', 'no_timeSeriesOceanRegions', 'no_climatologyMapSose', 'no_woceTransects', 'no_soseTransects', 'no_geojsonTransects', 'no_oceanRegionalProfiles', 'no_hovmollerOceanRegions', 'no_oceanConservation', 'no_index'

The issue will, of course, be fixed in the next E3SM-Unified.

[forsyth2]

would you like me to include bumping the minor version here?

No, I usually make specific PRs for that (e.g., the last version bump PR).

I assume you might want to hold off and bundle various minor updates together but let me know.

Are we trying to have this feature available for use before the next E3SM Unified environment is ready? We can make a dev environment based on main after it merges. We could also do a midterm release of zppy (e.g., v3.2.0) and release v3.3.0 for the Unified release. @chengzhuzhang Do you have any preferences here?

this is ready for you to have a look at.

Great, thank you for working on this! I should be able to do a review today.

[forsyth2]

@xylar From visual inspection, this all looks right. Thanks for adding documentation as well.

I see you tested yourself, but I would still like to add integration tests (like e3sm_diags atm_monthly_180x360_aave_mvm task in the v3 test cfg). I can add those as a new commit.

[forsyth2]

I would still like to add integration tests
I can add those as a new commit.

I made some progress on that today; I'm hoping to make more progress tomorrow.

[chengzhuzhang]

Are we trying to have this feature available for use before the next E3SM Unified environment is ready? We can make a dev environment based on main after it merges. We could also do a midterm release of zppy (e.g., v3.2.0) and release v3.3.0 for the Unified release. @chengzhuzhang Do you have any preferences here?

I have been maintaining a zppy development environment for v3.HR to leverage the latest zppy changes, and advanced users can create their own zppy development environment to use these new features as well. I don’t have a strong preference on whether zppy should have a midterm release; since we are not planning a midterm release for Unified, a zppy release is not strictly necessary.

[forsyth2]

I have been maintaining a zppy development environment for v3.HR to leverage the latest zppy changes

@chengzhuzhang Ok great; in that case, it's probably not worth the trouble of doing an intermediate release. Thanks!

[forsyth2]

@xylar I've been trying to add a test, but I keep running into issues.

Updating the test cfg

tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg on main has:

[mpas_analysis]
active = True
anomalyRefYear = 1985
climo_years = "1985-1989", "1990-1995",
enso_years = "1985-1989", "1990-1995",
environment_commands = "source <INSERT PATH TO CONDA>/conda.sh; conda activate <INSERT ENV NAME>"
mesh = "IcoswISC30E3r5"
parallelTaskCount = 6
partition = "compute"
qos = "regular"
shortTermArchive = True
ts_years = "1985-1989", "1985-1995",
walltime = "00:30:00"

(It's an auto-generated file, which is why environment_commands is set to a placeholder value).

My latest test of this branch ran with:

[mpas_analysis]
active = True
mesh = "IcoswISC30E3r5"
partition = "compute"
qos = "regular"
shortTermArchive = True

  [[ standard ]]
  anomalyRefYear = 1985
  environment_commands = "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh"
  parallelTaskCount = 6
  ts_years = "1985-1989", "1985-1995",
  climo_years = "1985-1989", "1990-1995",
  enso_years = "1985-1989", "1990-1995",
  walltime = "00:30:00"

  [[ mvm ]]
  anomalyRefYear = 1850
  environment_commands = "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh"
  parallelTaskCount = 32
  ts_years = "1985-1989", "1985-1995",
  climo_years = "1985-1989", "1990-1995",
  enso_years = "1985-1989", "1990-1995",
  reference_data_path = "/lcrc/group/e3sm/ac.xylar/zppy/mpas_analysis_model_vs_model/v3.LR.historical_0051"
  ref_ts_years = "1850-2014",
  ref_climo_years = "1985-2014",
  ref_enso_years = "1850-2014",
  walltime = "4:00:00"

Your test zppy cfg has:

[mpas_analysis]
active = True
environment_commands = "source /home/ac.xylar/chrysalis/miniforge3/etc/profile.d/conda.sh && conda activate mpas_analysis_dev"

anomalyRefYear = 1850
climo_years = "1985-2014",
enso_years = "1850-2014",
mesh = "IcoswISC30E3r5"
parallelTaskCount = 32
shortTermArchive = True
ts_years = "1850-2014",
walltime = "4:00:00"

reference_data_path = "/lcrc/group/e3sm/ac.xylar/zppy/mpas_analysis_model_vs_model/v3.LR.historical_0051"
ref_ts_years = "1850-2014",
ref_climo_years = "1985-2014",
ref_enso_years = "1850-2014",

So, a few notes:

1. I know we can now separate mpas_analysis into subtasks because of the addition of [[__many__]] in zppy/defaults/default.ini in this PR.
2. zppy/mpas_analysis.py has the following:

    if not path.is_dir():
        raise FileNotFoundError(
            f"MPAS-Analysis run path does not exist: {run_output_dir}"
        )

That file also notes:

            # These zppy options are expected to point to either:
            #  - a prior zppy run's output directory (containing a post/ directory),
            #  - the post/ directory itself,
            #  - an MPAS-Analysis directory (analysis/mpas_analysis),
            #  - the cfg/ directory, or
            #  - directly to the specific mpas_analysis_*.cfg file.

Taken together, it's clear the model-vs-model mpas_analysis is expecting both the controlRunConfigFile and mainRunConfigFile to already exist (or be the empty string by setting reference_data_path="" or test_data_path="" respectively).

Question 1: It appears both controlRunConfigFile and mainRunConfigFile must exist on the file system at the time zppy is run (that's when we run mpas_analysis.py), leaving no chance to reference a config file that's created later on. That is, even if we want to set up a dependency on a different mpas-analysis subtask (standard in this case), we can't make it work because the ConfigFile won't exist at launch time. Is that correct?

Question 2: You're not setting test_data_path in your cfg, so it must be the case that mainRunConfigFile="". Is that a special value in mpas_analysis that means something like "look for RunConfigFile later" or "use current directory"?

Setting up a model-vs-model test in a repeatable way

Point 2 above makes sense in real-world use cases, since zppy is typically run on one simulation's output, but unfortunately it does complicate testing.

For the e3sm_diags model-vs-model testing, I've gotten around the problem of needing to store reference data in some other post/ directory (as we did above by setting reference_data_path = "/lcrc/group/e3sm/ac.xylar/zppy/mpas_analysis_model_vs_model/v3.LR.historical_0051"), by simply comparing against different years from the same simulation data (e.g., these expected results). In tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg, I set all the reference paths to be the same data paths as model-vs-obs used:

  [[ atm_monthly_180x360_aave_mvm ]]
  # Test model-vs-model using the same files as the reference
  diff_title = "Difference"
  partition = "compute"
  qos = "regular"
  ref_name = "v3.LR.historical_0051"
  run_type = "model_vs_model"
  short_ref_name = "same simulation"
  swap_test_ref = False
  tag = "model_vs_model"
  ts_num_years_ref = 2
  # Reference paths
  reference_data_path = "/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_pr772_mpas_mvm_try5/v3.LR.historical_0051/post/atm/180x360_aave/clim"
  # mvm streamflow only
  gauges_path = "/lcrc/group/e3sm/diagnostics/observations/Atm/time-series/GSIM/GSIM_catchment_characteristics_all_1km2.csv"
  reference_data_path_ts_rof = "/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_pr772_mpas_mvm_try5/v3.LR.historical_0051/post/rof/native/ts/monthly"
  # mvm diurnal_cycle only
  reference_data_path_climo_diurnal = "/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_pr772_mpas_mvm_try5/v3.LR.historical_0051/post/atm/180x360_aave/clim_diurnal_8xdaily"
  # mvm "enso_diags", "qbo", "area_mean_time_series"
  reference_data_path_ts = "/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_pr772_mpas_mvm_try5/v3.LR.historical_0051/post/atm/180x360_aave/ts/monthly"
  # mvm tropical_subseasonal only
  reference_data_path_ts_daily = "/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_pr772_mpas_mvm_try5/v3.LR.historical_0051/post/atm/180x360_aave/ts/daily"

I suppose the equivalent thing to do here would be to set controlRunConfigFile to be the same value as mainRunConfigFile but that returns me to question 1 (re: referencing config files that are yet to be created).

More closely matching your use case

Even just using reference_data_path = "/lcrc/group/e3sm/ac.xylar/zppy/mpas_analysis_model_vs_model/v3.LR.historical_0051" from your test zppy cfg, I end up getting errors:

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_pr772_mpas_mvm_try5/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# mpas_analysis_mvm_ts_1985-1989_climo_1985-1989_vs_ref_ts_1850-2014_climo_1985-2014.status:ERROR (1)
# mpas_analysis_mvm_ts_1985-1995_climo_1990-1995_vs_ref_ts_1850-2014_climo_1985-2014.status:WAITING 1125661
emacs mpas_analysis_mvm_ts_1985-1989_climo_1985-1989_vs_ref_ts_1850-2014_climo_1985-2014.o1125660
# The following commands may be helpful:
#   cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_pr772_mpas_mvm_try5/v3.LR.historical_0051/post/scripts/../analysis/mpas_analysis/ts_1985-1989_climo_1985-1989/logs
#   grep Error *.log
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_pr772_mpas_mvm_try5/v3.LR.historical_0051/post/scripts/../analysis/mpas_analysis/ts_1985-1989_climo_1985-1989/logs
grep Error *.log
# indexNino34.log:ValueError: could not broadcast input array from shape (5,) into shape (1,)
# timeSeriesOHCAnomaly_computeAnomaly.log:    raise ValueError('The data set contains no Time entries between '
# timeSeriesOHCAnomaly_computeAnomaly.log:ValueError: The data set contains no Time entries between dates 1850-01-01 00:00:00 and 1850-12-31 23:59:59.
# timeSeriesSalinityAnomaly_computeAnomaly.log:    raise ValueError('The data set contains no Time entries between '
# timeSeriesSalinityAnomaly_computeAnomaly.log:ValueError: The data set contains no Time entries between dates 1850-01-01 00:00:00 and 1850-12-31 23:59:59.
# timeSeriesSSHAnomaly_computeAnomaly.log:    raise ValueError('The data set contains no Time entries between '
# timeSeriesSSHAnomaly_computeAnomaly.log:ValueError: The data set contains no Time entries between dates 1850-01-01 00:00:00 and 1850-12-31 23:59:59.
# timeSeriesTemperatureAnomaly_computeAnomaly.log:    raise ValueError('The data set contains no Time entries between '
# timeSeriesTemperatureAnomaly_computeAnomaly.log:ValueError: The data set contains no Time entries between dates 1850-01-01 00:00:00 and 1850-12-31 23:59:59.

All analysis ran successfully except indexNino34

That explains the indexNino34.log error, but there are more errors.

It looks like 1850 data is missing. Looking at reference_data_path = "/lcrc/group/e3sm/ac.xylar/zppy/mpas_analysis_model_vs_model/v3.LR.historical_0051":

ls /lcrc/group/e3sm/ac.xylar/zppy/mpas_analysis_model_vs_model/v3.LR.historical_0051/post/analysis/mpas_analysis/
# ts_1850-2014_climo_1985-2014
ls /lcrc/group/e3sm/ac.xylar/zppy/mpas_analysis_model_vs_model/v3.LR.historical_0051/post/analysis/mpas_analysis/cfg/
# mpas_analysis_ts_1850-2014_climo_1985-2014.cfg

Question 3: I'm not sure why I'm still running into errors, even using an explicit reference_data_path (rather than trying to be clever with referencing files made during the zppy run). Is there an obvious issue?

I should note that the standard run did appear to complete successfully.

Summary notes

Let me know if it would be easier to discuss this in a short meeting. I mainly just want to make sure we're not introducing a major new feature without also merging an accompanying test. I'll try to read up more on how to use MPAS-Analysis as well. Thanks!

[xylar]

@forsyth2, your message came in on a Friday and I guess I forgot about it. Sorry about that.

Question 1: It appears both controlRunConfigFile and mainRunConfigFile must exist on the file system at the time zppy is run (that's when we run mpas_analysis.py), leaving no chance to reference a config file that's created later on. That is, even if we want to set up a dependency on a different mpas-analysis subtask (standard in this case), we can't make it work because the ConfigFile won't exist at launch time. Is that correct?

Thanks you for pointing out this issue. It clearly needs to be fixed. I agree with you that it is important that the "control" or "reference" run can be generated first in the same zppy run so it should not have to exist already at zppy launch.

I will think about the best solution, likely just moving the check to runtime (presumably already present in MPAS-Analysis in any case).

Question 2: You're not setting test_data_path in your cfg, so it must be the case that mainRunConfigFile="". Is that a special value in mpas_analysis that means something like "look for RunConfigFile later" or "use current directory"?

The idea is that there are 2 common modes of running model-vs-model analysis. In one case, you have already run the analysis on both the "main" or "test" analysis run and the "control" or "reference" run because you wanted the comparison with observations to be available. This is pretty common, and in this case you can save a lot of time in MPAS-Analysis by pointing to both runs -- all the time is spent plotting, and little other computation is required.

In another mode, you don't care about the model vs. obs run for the "main" or "test" run, you just want to jump straight to model vs. model. In this situation, you just omit the mainRunConfigFile. The computational side of the analysis for the "main" run must be performed in full because there is no previous data to refer to but no comparison plots are made with observations, just with the "reference" run.

Does that help?

In your test script, you refer to my run:

  reference_data_path = "/lcrc/group/e3sm/ac.xylar/zppy/mpas_analysis_model_vs_model/v3.LR.historical_0051"

I hope this was a temporary test, as I have no intention of preserving these files (indeed, I have already deleted them because I was close to my quota on scratch space). Instead, this needs to refer to the location where the [[standard]] run will be output. If it isn't easy to know that location in advance, we need to have some kind of a placeholder that signals to zppy that it needs to fill in that location based on another run in its workflow. The same would presumably be needed for the test_data_path.

The anomaly errors you are seeing are because you have set:

anomalyRefYear = 1850

This year is not available in the data, which is an invalid configuration for MPAS-Analysis. The anomaly year must be present in the source data or it is simply not possible to compute anomalies.

Question 3: I'm not sure why I'm still running into errors, even using an explicit reference_data_path (rather than trying to be clever with referencing files made during the zppy run). Is there an obvious issue?

I believe it is because of the incorrect anomaly year above.

[xylar]

I tried to make headway on this today but wasn't able to access my home directory after logging in to Chrysalis. hopefully, it will be resolved later today.

[xylar]

@forsyth2, I believe I have fixed the issues you brought up. Thank you for thinking about them and for testing.

I used the following two config files in my testing. I waited for the jobs from the first zppy run to finish before launching the second:

# This is an example zppy v3.1.0 configuration file for Chrysalis.
# zppy v3.1.0 is included in E3SM Unified 1.12.
#
# NOTE: This configuration generates approximately 220 tasks/jobs when fully enabled.
# Users should only enable the tasks and time periods they need based on their specific
# analysis requirements to reduce computational load and queue time.

[default]
# Change these two paths to include your username instead
# You can also change these to match your preferred directory structure
output = /lcrc/group/e3sm/ac.xylar/zppy/mpas_analysis_model_vs_model/v3.LR.historical_0051
www = /lcrc/group/e3sm/public_html/diagnostic_output/ac.xylar/zppy/mpas_analysis_model_vs_model/v3.LR.historical_0051
# Other paths
input = /lcrc/group/e3sm2/ac.wlin/E3SMv3/v3.LR.historical_0051

# zppy auto-determines and sources latest e3sm-unified on each supported machine with environment_commands commented out.
#environment_commands = "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh"
# For testing Unified rc:
#environment_commands = "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.12.0rc2_chrysalis.sh"
# Other parameters:
campaign = "water_cycle"
case = v3.LR.historical_0051
partition = compute

[mpas_analysis]
active = True
environment_commands = "source /home/ac.xylar/chrysalis/miniforge3/etc/profile.d/conda.sh && conda activate mpas_analysis_dev"
anomalyRefYear = 1985
mesh = "IcoswISC30E3r5"
parallelTaskCount = 32
shortTermArchive = True
walltime = "2:00:00"

  [[ reference ]]
  ts_years = "1985-1989",
  climo_years = "1985-1989",
  enso_years = "1985-1989",

  [[ test ]]
  ts_years = "1985-1995",
  climo_years = "1990-1995",
  enso_years = "1990-1995",

  [[ mvm ]]
  reference_data_path = [[ reference ]]
  test_data_path = [[ test ]]

# This is an example zppy v3.1.0 configuration file for Chrysalis.
# zppy v3.1.0 is included in E3SM Unified 1.12.
#
# NOTE: This configuration generates approximately 220 tasks/jobs when fully enabled.
# Users should only enable the tasks and time periods they need based on their specific
# analysis requirements to reduce computational load and queue time.

[default]
# Change these two paths to include your username instead
# You can also change these to match your preferred directory structure
output = /lcrc/group/e3sm/ac.xylar/zppy/mpas_analysis_model_vs_model/v3.LR.historical_0091_el_nino_fix
www = /lcrc/group/e3sm/public_html/diagnostic_output/ac.xylar/zppy/mpas_analysis_model_vs_model/v3.LR.historical_0091_el_nino_fix
# Other paths
input = /lcrc/group/e3sm2/ac.wlin/E3SMv3/v3.LR.historical_0091

# zppy auto-determines and sources latest e3sm-unified on each supported machine with environment_commands commented out.
#environment_commands = "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh"
# For testing Unified rc:
#environment_commands = "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.12.0rc2_chrysalis.sh"
# Other parameters:
campaign = "water_cycle"
case = v3.LR.historical_0091
partition = compute

[mpas_analysis]
active = True
environment_commands = "source /home/ac.xylar/chrysalis/miniforge3/etc/profile.d/conda.sh && conda activate mpas_analysis_dev"

anomalyRefYear = 1985
climo_years = "1985-2014",
enso_years = "1985-2014",
mesh = "IcoswISC30E3r5"
parallelTaskCount = 32
shortTermArchive = True
ts_years = "1985-2014",
walltime = "4:00:00"

reference_case = v3.LR.historical_0051
reference_data_path = "/lcrc/group/e3sm/ac.xylar/zppy/mpas_analysis_model_vs_model/v3.LR.historical_0051"
ref_ts_years = "1985-1995",
ref_climo_years = "1990-1995",
ref_enso_years = "1990-1995",

The first performs model-vs-model analysis within a given simulation (relatively uncommon but practical for testing). Results are here:
https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.xylar/zppy/mpas_analysis_model_vs_model/v3.LR.historical_0051/v3.LR.historical_0051/mpas_analysis_mvm/ts_1985-1995_climo_1990-1995_vs_ts_1985-1989_climo_1985-1989/

The second performs a more typical model vs model run (no model vs. obs, though, which is also uncommon). Results are here:
https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.xylar/zppy/mpas_analysis_model_vs_model/v3.LR.historical_0091_el_nino_fix/v3.LR.historical_0091/mpas_analysis_mvm/v3.LR.historical_0091_ts_1985-2014_climo_1985-2014_vs_v3.LR.historical_0051_ts_1985-1995_climo_1990-1995/

[xylar]

As you will see, I added support for references like reference_data_path = [[ reference ]].

I also needed to put the output in an mpas_analysis_mvm subdirectory instead of just mpas_analysis. And I took care of giving the runs unique (and therefore long) names like:

ts_1985-1995_climo_1990-1995_vs_ts_1985-1989_climo_1985-1989
v3.LR.historical_0091_ts_1985-2014_climo_1985-2014_vs_v3.LR.historical_0051_ts_1985-1995_climo_1990-1995

I'm open to suggestions on how to revise any of this.

[xylar]

I ran into 3 more MPAS-Analysis bugs that I'm working to fix:
MPAS-Dev/MPAS-Analysis#1135
MPAS-Dev/MPAS-Analysis#1136
MPAS-Dev/MPAS-Analysis#1137

[forsyth2]

your message came in on a Friday and I guess I forgot about it. Sorry about that.

@xylar No worries, sorry as well for getting back to this late. Do we just want this done by the release candidate deadline or do we need it earlier? I might have some time today/tomorrow to revise the integration test file, but after that I don't expect much opportunity to work on this for a couple weeks.

2 common modes of running model-vs-model analysis.

you can save a lot of time in MPAS-Analysis by pointing to both runs

you just want to jump straight to model vs. model. In this situation, you just omit the mainRunConfigFile. The computational side of the analysis for the "main" run must be performed in full

Does that help?

Yes, thanks, that explains why mainRunConfigFile="".

I hope this was a temporary test, as I have no intention of preserving these files

Correct, I was just trying to get the test to work in any capacity. I try to avoid having external depdencies like that in the tests.

The anomaly year must be present in the source data

Ah ok, in retrospect that does make sense since the refernce_data_path had v3.LR.historical_0051.

I believe I have fixed the issues you brought up. Thank you for thinking about them and for testing.

Thank you for working on these!! I will work on revising my testing file accordingly.

I ran into 3 more MPAS-Analysis bugs that I'm working to fix:

Thanks!

[xylar]

@forsyth2

Do we just want this done by the release candidate deadline or do we need it earlier?

I think it would be nice not to lose momentum on this but there's no need to have it finished before the next RC.

[forsyth2]

The new commit above adds the test case to the cfg. This is the first change to the test cfgs since v3.1.0 was released. To address this, I merged #769, enabling the v3.1.0 legacy test cfgs to retain testing of the cfgs as they existed at time of release, for backwards compatibility. I was able to generate model-vs-model output at this page.

[forsyth2]

@xylar I did a high-level visual inspection of the code changes and I was able to generate model-vs-model output, so I think this should be good to merge. Thanks!

[erinethomas]

yes! thank you for this one @xylar and @forsyth2!

[xylar]

Okay, great, @forsyth2, feel free to merge when you're ready. I don't have the privileges. Thanks for working on this and for the review!

[forsyth2]

I'll use a merge commit (rather than squash and merge) here since there were actually a decent number of changes + commits from multiple people.