Create Python-based Scorecard

[bikegeek]

Replace italics below with details for this issue.

Describe the New Feature

Generate a scorecard, utilizing the following:

• METdataio METreformat code to format the MET .stat output based on linetype
• METcalcpy agg_stat.py module to calculate the bootstrap confidence limits using the reformatted METdataio data as input
• METcalcpy scorecard.py to calculate the p-values and DIFF or DIFF_STAT derived statistics used to determine degrees of statistical significance between models
  • [https://github.com/Make necessary modifications to scorecard module to support a basic scorecard METcalcpy#467](METcalcpy Feature 461 make compare images configurable #467)
• METplotpy scorecard_plot.py to control all aspects of generating the scorecard plot (from MET .stat input to a plot)

Acceptance Testing

Time Estimate

~ 1 1/2 months

Sub-Issues

Consider breaking the new feature down into sub-issues.

• METcalcpy scorecard.py working with YAML config file

• METdataio reformatting of sample data

  • Allow flexibility in the directory format for input data

• METcalcpy agg_stat.py to calculate bootstrap confidence limits on sample data

  • Use ECNT linetype data provided by Gwen Chen/NOAA with agg_stat.py

• METcalcpy scorecard.py on output from agg_stat.py to calculate p-values, etc.

• METplotpy generate scorecard plot via scorecard_plot.py

  • Common YAML config file encompassing all steps
    • Reformatting
    • Filtering
    • METcalcpy agg_stat for calculating CI's
    • METcalcpy scorecard.py for p-values
    • METplotpy for customizing scorecard plot
  • invoke METcalcpy event_equalize.py BEFORE invoking METcalcpy scorecard.py
  • Identify how to refine configurations used in METdataio and METcalcpy
  • Consider subsetting and filtering of data early in process to avoid performance degradation for large datasets
  • Create tests for the filtering code

• METplotpy scorecard plot

  • Incorporate plottable code base as sub-module in repository
  • Clean up jpeg images for triangles and boxes
  • Clean up appearance of tables
  • Support multiple tables (to support multiple levels, stats, etc.)
  • Create the DIFF_SIG (list of lists inside code)
    • Remove DIFF_SIG entries from YAML config file
  • Add support for START TIME, END TIME, INCR in YAML config file
    • Incorporate the start, end, incr values in filtering logic
    • For AI model v2 evaluation, generate at least 2 scorecards (AIGEFSv2 vs AIGEFSv1 and HGEFSv2 vs HGEFSv1) for a specified period (from the start date to the end date; this period can be longer than 90 days

• Compare METplotpy with METviewer-generated scorecard to verify that results are consistent

• Provide documentation for NOAA scientists to run using their data

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

NOAA SW METplus

Define the Metadata

Assignee

• Select engineer(s)
• Select scientist(s) NOAA EMC: Gwen Chen

Labels

• Review default alert labels
• Select component(s)
• Select priority
• Select requestor(s)

Milestone and Projects###

• Select Milestone as a METplotpy-X.Y.Z version, Consider for Next Release, or Backlog of Development Ideas
• For a METplotpy-X.Y.Z version, select the METplotpy-X.Y.Z Development project

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy

New Feature Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Add any new Python packages to the METplus Components Python Requirements table.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Development issue
  Select: Milestone as the next official version
  Select: METplotpy-X.Y.Z Development project for development toward the next official release
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.