Cobertura reports ignoring closures when calculating executable lines

[may246]

Hello! I've noticed that the coverage values listed in the Cobertura reports generated by Xcresultparser do no match the coverage values reported by Xcode.

After doing some digging, it seems that the reason for this is that Xcode includesclosures when calculating executable lines, while Cobertura does not.

I can't show our actual code files, but here's an example of what I mean -- the XML data shows a single executable line, while Xcode shows three, including two closures.

In this minimal example, the final coverage percentage happens to not be impacted (since the coverage state of all three lines is the same anyway), but more often than not the final coverage ends up being significantly different due to the ignored closures. E.g., in another file with 688 executable lines according to Xcode, Cobertura only reports 451 lines 😬

I strongly suspect this issue isn't unique to Xcresultparser, since we saw similar problems with Slather and Xcov as well. But I'm wondering if it might be possible to fix this issue on the Xcresultparser side, because the correct percentages are actually already reported in Xcresultparser's HTML and markdown output types? Any input on this or other potential workarounds would be appreciated! Thanks.

---

Other Notes:

• It looks like Sonar has the exact same issue as Cobertura: https://sonarsource.atlassian.net/browse/SONARSWIFT-560

• Implicit closures and variable initialization expressions are also included in Xcode's calculation of executable lines:

[a7ex]

Hi may246,
thanks for taking the time to examine the problem and document it. It will certainly help me to try and find the reason for the discrepancy.
I was about to answer, that xcresultparser can only report, what it finds in the xcresult bundle without knowing anything about the code behind it, but if the html and markdown output show the correct values, then it may be possible to find the culprit of the discrepancy.
But I can not promise, that it can be fixed, as xcresultparser only reports, what it "finds" in the in the bundle, or in case of coverage, what Xcode's xccov CLI tool reports to have found in the xcresult bundle.
There are however two different methods xccov offers, the newer method creates a JSON result with all information at once and is WAY faster, than the older method, where the coverage is gathered for each single file.
The first step will be to check, whether the results of both are the same.
As soon as I have some spare time, I will investigate and come back to you on this.

[a7ex]

Hm, I am a bit at a loss here. To get the coverage data from the xcresult bundle I use the Xcode tool xccov.
In one case (Cobertura/Sonar xml) I need the information for each line and compile a list of code lines.
It uses the command xccov view --archive --json <path-to-xcresult-bundle>
In the other case (simple output as text/html/md) another subcommand of xccov is used: --report
It uses the command xccov view --report --json <path-to-xcresult-bundle>

The output of the --report (which according to you seems to be correct), can't be converted to a list of coverage of single lines of a file, which is required for sonar or cobertura.
The --report subcommand list the percentage per function. For each function, I can see, at which line it starts, and how many executable lines it has. I see, at which line the next function starts, so I can determine how many lines are in between and I have the info how many lines are covered and how many lines are executable from this delta. But what I can not do is to determine which of the lines, is excutable, which is covered and which is not-executable. Which means I can not create the required format for cobertura and sonar. (A simple list of all lines with. the information, whether the single line is executable and whether it is covered.)

E.g. what I get from xccov view --report --json:

[0] = {
    coveredLines = 1
    lineCoverage = 1
    lineNumber = 11
    name = "variable initialization expression of CLIResultFormatter.cliStyle"
    executableLines = 1
    executionCount = 1
  }
  [1] = {
    coveredLines = 1
    lineCoverage = 1
    lineNumber = 14
    name = "CLIResultFormatter.init()"
    executableLines = 1
    executionCount = 1
  }
  [2] = {
    coveredLines = 3
    lineCoverage = 1
    lineNumber = 16
    name = "CLIResultFormatter.documentPrefix(title:)"
    executableLines = 3
    executionCount = 1
  }
  [3] = {
    coveredLines = 3
    lineCoverage = 1
    lineNumber = 19
    name = "CLIResultFormatter.documentSuffix.getter"
    executableLines = 3
    executionCount = 1
  }
  [4] = {
    coveredLines = 3
    lineCoverage = 1
    lineNumber = 22
    name = "CLIResultFormatter.accordionOpenTag.getter"
    executableLines = 3
    executionCount = 3
  }
  [5] = {
    coveredLines = 3
    lineCoverage = 1
    lineNumber = 25
    name = "CLIResultFormatter.accordionCloseTag.getter"
    executableLines = 3
    executionCount = 3
  }
  [6] = {
    coveredLines = 0
    lineCoverage = 0
    lineNumber = 28
    name = "CLIResultFormatter.tableOpenTag.getter"
    executableLines = 3
    executionCount = 0
  }
  [7] = {
    coveredLines = 0
    lineCoverage = 0
    lineNumber = 31
    name = "CLIResultFormatter.tableCloseTag.getter"
    executableLines = 3
    executionCount = 0
  }
  [8] = {
    coveredLines = 3
    lineCoverage = 1
    lineNumber = 34
    name = "CLIResultFormatter.divider.getter"
    executableLines = 3
    executionCount = 2
  }
  [9] = {
    coveredLines = 0
    lineCoverage = 0
    lineNumber = 37
    name = "CLIResultFormatter.resultSummaryLine(_:failed:)"
    executableLines = 3
    executionCount = 0
  }
...

and what I get from xccov view --archive --json:

                    <lines>
                        <line number="11" branch="false" hits="1"/>
                        <line number="14" branch="false" hits="1"/>
                        <line number="16" branch="false" hits="1"/>
                        <line number="17" branch="false" hits="1"/>
                        <line number="18" branch="false" hits="1"/>
                        <line number="19" branch="false" hits="1"/>
                        <line number="20" branch="false" hits="1"/>
                        <line number="21" branch="false" hits="1"/>
                        <line number="22" branch="false" hits="3"/>
                        <line number="23" branch="false" hits="3"/>
                        <line number="24" branch="false" hits="3"/>
                        <line number="25" branch="false" hits="3"/>
                        <line number="26" branch="false" hits="3"/>
                        <line number="27" branch="false" hits="3"/>
                        <line number="28" branch="false" hits="0"/>
                        <line number="29" branch="false" hits="0"/>
                        <line number="30" branch="false" hits="0"/>
                        <line number="31" branch="false" hits="0"/>
                        <line number="32" branch="false" hits="0"/>
                        <line number="33" branch="false" hits="0"/>
                        <line number="34" branch="false" hits="2"/>
                        <line number="35" branch="false" hits="2"/>
                        <line number="36" branch="false" hits="2"/>
                        <line number="37" branch="false" hits="0"/>
                        <line number="38" branch="false" hits="0"/>
                        <line number="39" branch="false" hits="0"/>
                        <line number="40" branch="false" hits="0"/>
                        <line number="41" branch="false" hits="0"/>
                        <line number="42" branch="false" hits="0"/>
                        <line number="43" branch="false" hits="2"/>
...
</lines>

So I think at this time, we may need to wait, whether Apple "fixes" the xccov tool to report consistent data for both subcommands. :-(

I don't think I can fix it.

[may246]

Thank you for looking into this, Alex! I see that the problem lies with xccov, so I will see if I can file a feedback report with Apple to hopefully fix this discrepancy.

EDIT: I might also think about opening an issue with Cobertura itself. I wonder if it maybe lacks a way to represent coverage for closures, since they don't really have line numbers in a given the same way regular executable lines of code do? Hmm...

For now, my team has been able to leverage the txt and cli output options from xcresultparser as workaround to print the correct coverage data :) so that alone has been a huge help, thank you!