st2-run-pack-tests can already measure test coverage (via the -c flag), but it can only ever produce pytest-cov's default terminal report. There is no way to:
- emit a machine-readable coverage report (e.g. Cobertura XML, LCOV) or write it to a known path, or
produce a single aggregated coverage report across all packs (or a given subset of packs).
- This makes it effectively impossible to wire pack coverage into standard CI / code-quality tooling without bypassing the script. I'd like to propose adding small, backwards-compatible options to close this gap.
Current behavior
When -c is passed, the script enables coverage by appending one --cov= per pack subdirectory (excluding tests/) and then runs:
python3 -m pytest -s -v [--cov=<subdir> ...] <pack>/tests/
It never appends a --cov-report=..., so pytest-cov falls back to the terminal-only report. The option parser is a fixed set (getopts ":p:f:xjvct") with no pass-through for additional pytest/coverage arguments, so a user cannot supply --cov-report themselves. The script also operates on exactly one pack per invocation and does not enable --cov-append, so successive runs overwrite each other's coverage data.
Why this change is needed
Coverage data is produced but thrown away in an unusable form. The underlying .coverage data file exists after a run, but the script only surfaces a human-readable terminal table. Every modern coverage consumer — SonarQube, Codecov, Coveralls, GitLab/GitHub coverage visualization — requires a machine-readable report (Cobertura XML or LCOV). Today there is no supported way for the script to emit one.
There is no story for multi-pack coverage. Pack repositories — Exchange-style or internal mono-repos containing many packs — have no way to produce a single combined coverage report. Because each invocation overwrites the data file (no --cov-append) and there is no shared data-file workflow, every pack is an island. Reviewers and CI cannot see aggregate coverage for a change that spans packs.
The gap is small and the capability already exists. pytest-cov is already a hardcoded dependency of this script. --cov-report and --cov-append are stock pytest-cov options; coverage.py already honors the COVERAGE_FILE environment variable. The script simply doesn't expose any of this. Exposing it unlocks coverage-driven quality gates for pack development — increasingly expected in enterprise and regulated environments — with a very small, additive change.
Proposed solution
Add two backwards-compatible flags that forward to options pytest-cov already supports:
- -r <report_spec> — render coverage report(s) in the given format/path, mapping to --cov-report (e.g. -r xml:coverage.xml, or a comma list like -r xml:coverage.xml,term). Enables machine-readable output.
- -a — enable --cov-append, so coverage data accumulates across multiple pack invocations into one data file.
Combined with the already-honored COVERAGE_FILE env var (documented to be an absolute path, since the script pushdes into the pack directory), this enables a single aggregated report across all or a chosen subset of packs:
for p in packs/pack_a packs/pack_b packs/pack_c; do
st2-run-pack-tests -x -c -a -p "$p"
done
coverage xml --data-file="$COVERAGE_FILE" -o coverage.xml
Willingness to contribute
I'm happy to submit a PR implementing the -r/-a flags (with updated usage() help and a CHANGELOG.rst entry) and to adjust the design per any feedback here.
st2-run-pack-testscan already measure test coverage (via the -c flag), but it can only ever produce pytest-cov's default terminal report. There is no way to:produce a single aggregated coverage report across all packs (or a given subset of packs).
Current behavior
When -c is passed, the script enables coverage by appending one --cov= per pack subdirectory (excluding tests/) and then runs:
python3 -m pytest -s -v [--cov=<subdir> ...] <pack>/tests/It never appends a --cov-report=..., so pytest-cov falls back to the terminal-only report. The option parser is a fixed set (getopts ":p:f:xjvct") with no pass-through for additional pytest/coverage arguments, so a user cannot supply --cov-report themselves. The script also operates on exactly one pack per invocation and does not enable --cov-append, so successive runs overwrite each other's coverage data.
Why this change is needed
Coverage data is produced but thrown away in an unusable form. The underlying .coverage data file exists after a run, but the script only surfaces a human-readable terminal table. Every modern coverage consumer — SonarQube, Codecov, Coveralls, GitLab/GitHub coverage visualization — requires a machine-readable report (Cobertura XML or LCOV). Today there is no supported way for the script to emit one.
There is no story for multi-pack coverage. Pack repositories — Exchange-style or internal mono-repos containing many packs — have no way to produce a single combined coverage report. Because each invocation overwrites the data file (no --cov-append) and there is no shared data-file workflow, every pack is an island. Reviewers and CI cannot see aggregate coverage for a change that spans packs.
The gap is small and the capability already exists. pytest-cov is already a hardcoded dependency of this script. --cov-report and --cov-append are stock pytest-cov options; coverage.py already honors the COVERAGE_FILE environment variable. The script simply doesn't expose any of this. Exposing it unlocks coverage-driven quality gates for pack development — increasingly expected in enterprise and regulated environments — with a very small, additive change.
Proposed solution
Add two backwards-compatible flags that forward to options pytest-cov already supports:
Combined with the already-honored COVERAGE_FILE env var (documented to be an absolute path, since the script pushdes into the pack directory), this enables a single aggregated report across all or a chosen subset of packs:
Willingness to contribute
I'm happy to submit a PR implementing the -r/-a flags (with updated usage() help and a CHANGELOG.rst entry) and to adjust the design per any feedback here.