Action to run all published rules' tests

[SFJohnson24]

This will involve creating a script to:

• get all published rules (this is now changed since rules are associated in github)
• read through the sharepoint directory of test data
  • This part may be difficult because the folder and file structure may be inconsistent. Might involve some reorganization of the folder structures to be consistent or the manual creation of a mapping file.
• execute each rule with the related test data
• parse the result data compare the result data with the result.json NOTE: we may need to run a script to ensure result.json in the cdisc-open-rules repo exists and is accurate
  • for positive test data, there should be at least 1 positive result and no negatiive results. skips can be ignored
  • for negative test data, there should be at least 1 negative result
• summarize the test results

Depending on runtime, we could run this action on push or create a cron job to run periodically using git actions:
https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows#schedule

[SFJohnson24]

This would likely involve:

1. standardizing the folder structure and file names for sharepoint (checking to make sure files are in this structure) this has been done with Initial Rules and Data cdisc-open-rules#2

2. a private git repo with an action that scrapes sharepoint based on this structure -- taking into account the name/directory structure with potential for multiple pos/negative data. It will also grab Define.xml when applicable. This is sorted into standard/coreID folders with (multiple) positive and negative sub directories. This would likely use the sharepoint API:
   https://learn.microsoft.com/en-us/sharepoint/dev/sp-add-ins/get-to-know-the-sharepoint-rest-service?tabs=csom a new action for engine repo that grabs cdisc-open-rules, it will need to iterate through Published/ grabbing all test data and running based on the .env in the test data and using the the rule.yml file in each Published/CORE-XXXX folder.

3. a second git action on this repo that gets engine, updates cache, uses rules_dictionary.pkl to ID all the published rules. We then iterate through the rule IDs, grab the corresponding folder based on ID since they are unique (will likely need a system for rules that have a/b/c parts as I believe these exist), make the command based on folder contents (if condition for define), then check the resultant json-- expecting no issues for positive data, negative results for negative data.

4. Compare results.json results.csv for each piece of test data. It would be good to also create a ledger that is given to the user as an artifact on the action run and the number of issues found for positive/negative. It could create a second sheet where problem rules/data are added if there is a misalignment between number of issues and their dataset type, differences in rule outputs should be stashed into an artifact folder that is given with this ledger.

a second copy of this second action could be aimed at the SQL branch for testing

[alexfurmenkov]

@SFJohnson24
Right now, in your rules_2 branch, each rule contains positive and negative folders, and each of those has a 01 subfolder with a data folder inside.

What should we expect in the action? Is it possible that there will be multiple folders like 01, 02, each with their own data?

Also, in the results folder, the outputs are currently in CSV format, but the task description mentions that the reference results will be in JSON. Should we generate those in advance, or should we expect the results to remain in CSV?

[SFJohnson24]

What should we expect in the action? Is it possible that there will be multiple folders like 01, 02, each with their own data?

there is possibly multiple datasets, different numbers for each. I would expect it to iterate through all test cases and compare the result.csv in the repo with the results.csv from the validation run.

Also, in the results folder, the outputs are currently in CSV format, but the task description mentions that the reference results will be in JSON. Should we generate those in advance, or should we expect the results to remain in CSV?

we changed it to CSV for diffing, this ticket has not been updated. rules_test added the logic in .github/scripts that converts the results.json to the results.csv as well as diffing two results.csv. This action can use those to convert+compare