Allow Rule Types that include define-xml to start the dataset builder from define-xml (and not only from dataset).

[eljanssens]

Standard

CDISC SDTM/SDTMIG

Reference Rule ID(s)

CG0015

Conformance Rule ID(s) (if published in CORE)

CDISC.SDTMIG.CG0015

JIRA Ticket

https://jira.cdisc.org/browse/CORERULES-45?jql=project%20%3D%20CORERULES%20AND%20resolution%20%3D%20Unresolved%20AND%20text%20~%20%22CG0015%22%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC

Proposed Rule YAML

A# Variable: GEN
# Condition: Variable Core Status = Permissible and data is collected
# Rule: Variable present in dataset and ^= null
Authorities:
  - Organization: CDISC
    Standards:
      - Name: SDTMIG
        References:
          - Citations:
              - Cited Guidance: The sponsor does not have the discretion to exclude permissible
                  variables when they contain data.
                Document: IG v3.2
                Section: 4.1.1.5
            Origin: SDTM and SDTMIG Conformance Rules
            Rule Identifier:
              Id: CG0015
              Version: '1'
            Version: '2.0'
        Version: '3.2'
      - Name: SDTMIG
        References:
          - Citations:
              - Cited Guidance: The sponsor does not have the discretion to exclude permissible
                  variables when they contain data.
                Document: IG v3.3
                Section: 4.1.5
            Origin: SDTM and SDTMIG Conformance Rules
            Rule Identifier:
              Id: CG0015
              Version: '1'
            Version: '2.0'
        Version: '3.3'
      - Name: SDTMIG
        References:
          - Citations:
              - Cited Guidance: If a study includes a data item that would be represented in a
                  Permissible variable, then that variable must be included in
                  the SDTM dataset, even if null.
                Document: IG v3.4
                Section: 4.1.5
            Origin: SDTM and SDTMIG Conformance Rules
            Rule Identifier:
              Id: CG0015
              Version: '1'
            Version: '2.0'
        Version: '3.4'
Check:
  all:
    - name: define_variable_name
      operator: non_empty
    - name: library_variable_core
      operator: equal_to
      value: Perm
    - name: define_variable_has_no_data
      operator: empty
    - any:
        - name: variable_name
          operator: not_exists
        - name: variable_is_empty
          operator: equal_to
          value: 'Yes'
Core:
  Id: CDISC.SDTMIG.CG0015
  Status: Draft
  Version: '1'
Description: Raise an error when a variable is permissible, data is collected as
  per define-xml document but variable is not present in the dataset or has no
  data.
Executability: Fully Executable
Outcome:
  Message: 'Variable is permissible and data is collected as per define-xml
    document but variable is not present in the dataset or variable has no
    data.'
  Output Variables:
    - library_variable_name
    - library_variable_core
    - define_variable_name
    - define_variable_has_no_data
    - variable_name
    - variable_is_empty
Rule Type: Variable Metadata Check against Define XML and Library Metadata
Scope:
  Classes:
    Include:
      - ALL
  Domains:
    Include:
      - ALL
Sensitivity: Record

Attach any sample test data file(s)

Available on SharePoint: https://cdisc.sharepoint.com/sites/CORERules/Shared%20Documents/Forms/AllItems.aspx?id=%2Fsites%2FCORERules%2FShared%20Documents%2FunitTesting%2FSDTMIG%2FCG0015&viewid=ecd009a9%2Db73b%2D415b%2D9e41%2Df9d04a69edf4&FolderCTID=0x012000DB2B68AD3779324A8BE66373AE3E0E60

Expected output

sv.xpt - VISIT (in define but missing in dataset) - SVENDY & SVUPDES (in define and in dataset but completely empty)
tv.xpt - VISIT and TVENRL (in define and in dataset but completely empty)

Any Additional Information

Discussed with Gerry and Sam previously and this would mean enabling an outer join.

[SFJohnson24]

I spoke with @DmitryMK and @rich-verisian today regarding this:
idea being to implement a flag under Rule Type in the yaml that signals the direction to build from. This would then be accessible to all builders and prevent the need to create copies of builders in opposite directions (starting from dataset vs starting from define.xml) as future rules need different directionality. So for this ticket, it would use the current 'Variable Metadata Check against Define XML' rule type but a flag would tell the builder to start from the define and use all of its variable metadata and merge the variable metadata from provided datasets onto it. we can just change the builder to an outer join to facilitate checks in both directions