CQL Unit Testing

[JPercival]

EDIT: Updated based on discussions below

The language server should support CQL unit tests. The first step is to define what a unit test looks like in CQL. We propose adding support for some @tags to support specifying unit tests and their input requirements in CQL

Tag	Value	Description
@test	N/A	If specified on a Library, marks it as a test suite. If specified on a definition, marks it as a test
@parameter	Name CQLValue	set input parameters for Libraries
@asof	CQLDatetime	evaluate as of a specific date
@context	Context=Value	the value of the context
@data	Path	source directory for test data
@terminology	Path	source directory for test terminology
@mock	Definition=Value	(future work) specifies a mock value for a CQL definition
@parameterized	<tag>	(future work) specifies that a test should be repeated with a different set of inputs
@dataprovider	ExpressionReference	(future work) supplies a set of data used to run tests
@ignore	N/A	(future work) Report the results of this test, but don't fail the overall suite if this fails

An example of a test Library using these tags:

// @test
// @parameter: "Measurement Interval" [@2019,@2020]
// @asof: @2020-10-01
// @context: Patient=654
// @terminology: tests/vocab
library DQMTest

include DQM as targetLibrary
include org.opencds.cqf.cql.Asserts version '1.0.0'
include TestHelpers

// @test
// @data: tests/data
// @context: Patient=123
define "In Initial Population":
  assert(targetLibrary."Initial Population").isTrue()
  
// @test
// @data: tests/data
// @context: Patient=123
define "Has Required Attributes":
   TestHelpers.HasRequiredAttributes(Patient)

Evaluating a definition marked with @test and getting a Message with the severity level of Error is a test failure.

http://cql.hl7.org/09-b-cqlreference.html#message

@test is required to use the other tags. If any are present without @test, it's an error condition. IOW, these tags are only allowed on unit test libraries.

Any @tag defined at the Library level sets the default @tag value for the set of tests within the Library. @tag values on a definition override the default Library @tag.

Each @test definition is evaluated independently and with the appropriate input parameters as defined by the merged definition and library @tag values. It's up to the test runtime environment to optimize data or context caching to speed up test execution.

If during the evaluation of a test, the CQL definition references any other CQL definitions that are also marked as @test (or with any other of the proposed tags), those @tags are ignored. Only the @tags of the entry point apply. That is to say, during the evaluation of a test the context, input parameters, terminology, or data may not change mid-evaluation.

CQL Test libraries SHOULD NOT be shipped as part of an executable package, such as a FHIR npm package.

The example org.opencds.cqf.cql.Asserts library does not exist at the time of this writing. The contents would be a set of helper functions to assist in specifying expected results for tests.

[JPercival]

Relevant API for VS Code:
https://code.visualstudio.com/api/extension-guides/testing

[JPercival]

Had some discussions with Bryn. We think that these test tags should be disallowed outside the context of a test Library to avoid confusion with runtime behavior of CQL code.

[JPercival]

Test functions should throw an Exception if the test is a failure. The way to do that currently is to use the "Message" operator with an "Error" status in CQL: http://cql.hl7.org/09-b-cqlreference.html#message

[JPercival]

Proposed tags:
@test = marks a Library as a group of tests, marks a specific definition as a test within the Library
@parameter Name CQLValue = set input parameters for Libraries
@asof CQLDatetime = evaluate as of a specific date
@context Context Value = the value of the context
@source Path = source directory for test data

@test is required to use the other tags. If any are present without @test, it's an error condition.

[JPercival]

Open questions:
Multiple contexts supported?
Use of tags outside tests? (we think no, since authors could come to depend on it for runtime behavior of, say, a Measure)
Use of remote server for data?
Terminology tag needed for terminology source?
Shared data across tests? (workaround, load everything and set context)
Should this convention map to the XML test definitions for FHIRPath / CQL Engine compliance? (we think no, since those include compiler compliance as well which is a different requirement than unit testing)
Should the @source tag be inferred for an IG?

[c-schuler]

Very cool!

I would advocate for being able to use a remote server for data and/or terminology. I think that would promote shareability and a terminology source allows for more robust testing.

[JPercival]

Note that this specifically excludes testing of Knowledge Artifacts that may leverage CQL, such as FHIR Measures and PlanDefinitions. Both Measures and PD may not use CQL at all, or be defined in terms of a different language altogether, such as FHIRPath. In other words, full authoring support of those types of artifacts are beyond the scope of this current discussion, and debugging / testing those needs to occur at a higher, FHIR-based layer within the context of a CQL/FHIR authoring environment.

[JPercival]

I would advocate for being able to use a remote server for data and/or terminology. I think that would promote shareability and a terminology source allows for more robust testing.

The main issue/consideration there is that "unit tests" for other languages are generally expected to run quickly and not require any external data or input. IOW, you wouldn't expect a unit test to fail if a remote server were down.

Bryn and I also discussed some alternate test conventions. We did a quick survey of various tools and languages out there such as C#, Java, Python, Haskell, F#, and SQL and found that all of them had a widely available XUnit inspired framework. So that seems to be the most common and familiar paradigm for a wide variety of developers. In principle, as a functional language, CQL could adopt some other convention such as "all test definitions return true." We think that the "Assert expected result" paradigm is the least surprising, and allows authors to inspect test output for the sake of a quick "edit, run, debug" development loop.

[vitorpamplona]

It would be nice if the tool took a screenshot of useful data within a database/FHIR Server and saved that as input to the test. In my experience, CQL changes are always a reaction to new data in a server as opposed to being specification driven. Most CQL constructs that I have seen never treat edge cases. They are only created to match the specific way an institution is utilizing FHIR.

[cmoesel]

Writing unit tests for CQL in CQL... neat! We developed a test framework for CQL here, but it's quite different in its approach. I like the idea of adding an XUnit flavored approach in CQL. Very cool.

If it is built into the language server, what does that mean for being able to run automated tests from the command line and/or in a headless CI environment? Ideally, I'd like a command I can put on my path so I could just run cql-test from a CQL project folder and have it just work (perhaps with a little required configuration). Does integrating it into the language server still allow for that use case?

Have you considered building it on top of an existing framework so you can take advantage of runners, reporters, etc? That might get you some extra bang for your buck -- but perhaps it also limits some of the features you can provide. Just throwing out some ideas...

[vitorpamplona]

I would make a distinction between making a tool for development, debugging, and troubleshooting new CQLs (developer experience) and making a tool for automated, interoperability testing, and production auditing (DevOps experience). Both can use the same underlying principles and languages but are vastly different in their use.

[cmoesel]

I'm on a team that has developed CQL-based CDS in the past, and used a git repository for collaboration. In that case, having tests automatically run for PRs is quite helpful in ensuring that code meets a certain standard of quality before being merged to main. That's what I was advocating for (not full-on interoperability testing).

[JPercival]

@cmoesel

We developed a test framework for CQL here,

The approach you've created there is aligned with what I had in mind for higher-level PD/Measure testing. My thinking was that for CQL it'd be nice to try to keep the developer in the CQL as much as possible.

If it is built into the language server, what does that mean for being able to run automated tests from the command line

This is a good point. The cql-evaluator can already run CQL from the command line and accepts the equivalent of these tags as arguments. The new piece is test case discovery from the tags. My thinking had been to build the discovery bits into the language-server as a "trial use" thing and go from there, but putting the discovery bits in the evaluator would allow command line execution too.

Have you considered building it on top of an existing framework

The defacto build tool for most CQL projects is the IG Publisher. It's not particularly extensible so it's a challenge to integrate with it cleanly. It does already have a cql-translator integration so maybe it's not too difficult extend that a bit. Reporting in that scenario would look like the typical QA errors.

I take a look at some other options too and see what I come up with. Thanks for the suggestion.

[JPercival]

@vitorpamplona

It would be nice if the tool took a screenshot of useful data within a database/FHIR Server

This would be useful feature. I'm not sure how'd we'd implement it though. The trick is that if you have a test server with 1000s of Patients, how do you select only the data relevant for a particular test case? That may span ValueSets, non-Patient-compartment data like Medications, etc. Maybe we could start with the $everthing operation. Or generate data-requirements for some CQL and run the queries. Another option would be some type of "picker" UI.