Skip to content

Commit 4813d01

Browse files
bkhouribkhouri
committed
SwiftPM: HTML Coverage Report
1 parent 769e862 commit 4813d01

File tree

1 file changed

+204
-0
lines changed

1 file changed

+204
-0
lines changed

‎proposals/NNNN-swiftpm-html-coverage-report.md‎

Lines changed: 204 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,204 @@
1+
# HTML Coverage Report
2+
3+
* Proposal: [SE-NNNN](NNNN-swiftpm-html-coverage-report.md)
4+
* Authors: [Sam Khouri](https://github.com/bkhouri)
5+
* Review Manager: TBD
6+
* Status: **Awaiting implementation**
7+
* Implementation: [swiftlang/swift-package-manager#9076][PR]
8+
* Decision Notes: [Pitch](https://forums.swift.org/t/pitch-adding-html-coverage-support/82358)
9+
10+
## Introduction
11+
12+
Currently, `swift test` supports generating a JSON coverage report, which is
13+
great for ingesting into various systems. The JSON, however, is not very
14+
"human-readable" while iterating at-desk.
15+
16+
This proposes adding an additional command line argument to `swift test` that
17+
would allow the caller to select the generation of an HTML coverage report.
18+
19+
20+
## Motivation
21+
22+
JSON coverage report is great for ingesting into external tools that post-process
23+
the coverage data. If SwiftPM could generate an HTML coverage report:
24+
- said report can be uploaded to CI systems for visual inspection
25+
- developer can generate the report at-desk, giving faster feedback to determine
26+
if the current changes are sufficiently covered to their liking.
27+
28+
## Proposed solution
29+
30+
If users currently want an HTML report, the user must manually construct the
31+
`llvm-cov` binary directly with the correct command line arguments.
32+
33+
e.g.:
34+
```sh
35+
❯ swift test --enable-code-coverage
36+
37+
❯ swift test --show-codecov-path
38+
39+
❯ llvm-cov show \
40+
--project-title="HelloWorld" \
41+
--format="html" \
42+
--output-dir=".coverage" \
43+
--instr-profile=".build/arm64-apple-macosx/debug/codecov/default.profdata" \
44+
".build/.../HelloWorldPackageTests.xctest/Contents/MacOS/HelloWorldPackageTests" \
45+
"Sources"
46+
```
47+
48+
Since SwiftPM currently orchestrates the JSON coverage data, the solution adds a
49+
new command line argument, e.g.: `--coverage-format` to `swift test` which can
50+
be specified multiple times, to generate multiple coverage report type from a
51+
single test execution.
52+
53+
While processing the coverage data, SwiftPM will loop on all the unique coverage
54+
format options to generate the specified reports.
55+
56+
Unless otherwise specified, this proposal applies only to the HTML Coverage
57+
report. The generation of the JSON Coverage report is unchanged and is out of
58+
scope.
59+
60+
## Detailed design
61+
62+
Existing tools in LLVM have been around for several years (and maybe even decades),
63+
and provide robust tools for code coverage analysis. The LLVM tools are
64+
well-documented, and have been used in production for many years. SwiftPM will
65+
make use of LLVM's tools and construct the proper command line arguments to the
66+
`llvm-cov show` utility, which will generate the HTML report.
67+
68+
The proposted command line changes are as follows:
69+
70+
71+
### Format Selection
72+
73+
The `swift test` command line will have an option named `--coverage-format`,
74+
which accepts either `json` or `html`. This option can be specified multiple
75+
times on the command line, and a report will be generated for each format
76+
specified.
77+
78+
The command line option will be similar to:
79+
80+
```sh
81+
--codecov-format, --code-coverage-format, --coverage-format <format>
82+
Format of the code coverage output. Can be specified multiple times. (default: json)
83+
json - Produces a JSON coverage report.
84+
html - Produces an HTML report produced by llvm-cov.
85+
```
86+
87+
88+
### Coverage Report configuration
89+
90+
`llvm-cov show` has several report configurability options. In order to
91+
prevent a "command line arguments" explosion to `swift test`, the configuration
92+
options will be read from a response file. The optional response file will be
93+
located in `<repo>/.swiftpm/configuration/coverage.html.report.args.txt`. The
94+
response file will be supported.
95+
96+
The user can include `--format=text`, or a variation thereof, in the response
97+
file. In order to ensure SwiftPM will always generate an HTML report, SwiftPM
98+
will add `--format=html` after the response file argument to ensure `llvm-cov`
99+
will generate an HTML report.
100+
101+
102+
SwiftPM will not perform any validation on the response file contents, except
103+
to determine the output location.
104+
105+
### Coverage report location
106+
107+
By default, the HTML report will be created in location under the scratch path
108+
(ie: the build directory). However, this can be overridden using the response file.
109+
110+
Some CI system, such as [Jenkins](https://www.jenkins.io), only allow archiving
111+
contents files/directories that belong in a "sandbox" location. It can be a safe
112+
assumption that the CI system will have a copy of the repository in the "sandbox"
113+
location, allowing this system to upload the HTML report.
114+
115+
```
116+
--show-codecov-path [mode], --show-code-coverage-path [mode], --show-coverage-path [mode]
117+
Print the path of the exported code coverage files. The mode specifies how to
118+
display the paths of the selected code coverage file formats. (default: text)
119+
json - Display the output in JSON format.
120+
text - Display the output as plain text.
121+
--show-codecov-path-mode, --show-code-coverage-path-mode, --show-coverage-path-mode <show-codecov-path-mode>
122+
--enable-codecov, --enable-code-coverage, --enable-coverage/--disable-codecov, --disable-code-coverage, --disable-coverage
123+
Enable code coverage. (default: --disable-codecov)
124+
--codecov-format, --code-coverage-format, --coverage-format <format>
125+
Format of the code coverage output. Can be specified multiple times. (values: json, html; default: Produces a JSON coverage report.)
126+
```
127+
128+
### Show coverage path
129+
Prior to this proposal `swift test --show-coverage-path` would display a single
130+
absolute path location to the JSON coverage report.
131+
132+
Since `--coverage-format` can be specified multiple times, it's output must be
133+
changed to reflect the new functionality.
134+
135+
If the `--coverage-format` option is specified on the `swift test` command line
136+
a single time (or is not specified at all), there is no change to the output.
137+
138+
139+
if `--coverage-format` is specified multiple times, the output must reflect this.
140+
The `--show-coverage-path` command line argument will be modified to be an optional
141+
with a default value. The supported values are `json` or `text`, with the default
142+
being `text` to preverse existing behaviour.
143+
144+
A value of `json` will output a JSON object with the key representing the format,
145+
and the value representing the output location of said format.
146+
147+
A value of `text` will omit the format in the output if a single `coverage-format`
148+
is requested. Otherwise, the output will be similar to
149+
150+
```sh
151+
❯ swift test -c release --build-system swiftbuild --show-coverage-path --coverage-format html --coverage-format json
152+
Building for debugging...
153+
[5/5] Write swift-version-4B9677C1F510A69F.txt
154+
Build complete! (0.37s)
155+
Html: /swift-package-manager/.build/arm64-apple-macosx/Products/Release/codecov/Simple-html
156+
Json: /swift-package-manager/.build/arm64-apple-macosx/Products/Release/codecov/Simple.json
157+
158+
159+
❯ swift test -c release --build-system swiftbuild --show-coverage-path text --coverage-format html --coverage-format json
160+
Building for debugging...
161+
[5/5] Write swift-version-4B9677C1F510A69F.txt
162+
Build complete! (0.37s)
163+
Html: /swift-package-manager/.build/arm64-apple-macosx/Products/Release/codecov/Simple-html
164+
Json: /swift-package-manager/.build/arm64-apple-macosx/Products/Release/codecov/Simple.json
165+
166+
167+
❯ swift test -c release --build-system swiftbuild --show-coverage-path json --coverage-format html --coverage-format json
168+
Building for debugging...
169+
[1/1] Write swift-version-4B9677C1F510A69F.txt
170+
Build of product 'swift-test' complete! (0.40s)
171+
{
172+
"html" : "/Users/bkhouri/Documents/git/public/swiftlang/swift-package-manager/.build/arm64-apple-macosx/Products/Release/codecov/SwiftPM-html",
173+
"json" : "/Users/bkhouri/Documents/git/public/swiftlang/swift-package-manager/.build/arm64-apple-macosx/Products/Release/codecov/SwiftPM.json"
174+
}
175+
```
176+
177+
## Security
178+
179+
Since SwiftPM will use `llvm-cov show`, there may be security implications from
180+
the `llvm-cov` utility, but this is outside of the Swift organizations control.
181+
182+
The LLVM project has a [LLVM Security Response Group](https://llvm.org/docs/Security.html),
183+
which has a process for handling security vulnerabilities.
184+
185+
## Impact on existing packages
186+
187+
No impact is expected.
188+
189+
## Alternatives considered
190+
191+
- In addition to the response file, the coverage report generation can support
192+
a command line argument similar to `-Xlinker`, `-Xcc` and others, which will
193+
pass the arguments to `llvm-cov show` and override the values in the response
194+
file. This has _not_ been implemented in the [PR].
195+
196+
- Instead of having a `--show-codecov-path` as a tri-state, we could preserve
197+
`--show-codecov-path` original behaviour and add an additional command line
198+
argument to indicate the output mode. The comand line argument would be
199+
`--show-codecov-path-mode <mode>`, where `<mode>` is either `text` or `json`.
200+
This was not favours as `--show-codecov-path-mode` would have a dependency on
201+
`--show-codecov-path` argument, and may lead to some confusion.
202+
203+
204+
[PR]: https://github.com/swiftlang/swift-package-manager/pull/9076

0 commit comments

Comments
 (0)