slow coverage data generation

[staabm]

Q	A
ParaTest version	7.8.4
PHPUnit version	11.5.43
PHP version	8.3.27

Summary

running paratest with pcov and coverage generation on phpstan-src reports, that after running the tests for 4min40s (which is fast), we need another minute to generate the coverage xml report

see https://github.com/phpstan/phpstan-src/actions/runs/19078182639/job/54499398288?pr=4525

I would hope that there is some low hanging fruit to speedup coverage xml export.
I did not yet succeed in extracting the gathered information to make the process easily reproducible (and measurable with a profiler).
maybe you have ideas on how this can be made faster or how to at best investigate the problem without a 8minute feedback loop?

Current behavior

coverage data generation seems slow

How to reproduce: command, code and error stack trace

• git clone phpstan-src
• composer install
• run make tests (which installs paratest)
• run php -d pcov.enabled=1 tests/vendor/bin/paratest --passthru-php="'-d' 'pcov.enabled=1'" --coverage-xml=tmp/coverage/coverage-xml --log-junit=tmp/coverage/junit.xml

Expected behavior

faster xml code coverage export

[Slamdunk]

The complete absence of #[CoversClass(***)] annotations for sure kills performance.
Without them, PHPUnit needs to track every source code for every test, multiple that by the number of processes, and merge everything at the end.

Just to check if that's the culprit, code this silly try:

1. for every FooTest.php file in the tests/ folder, add a single #[CoversClass(Foo::class)] annotation if Foo exists
2. otherwhise add a #[CoversNothing] one

If the drop in the test duration is much higher than the drop in the overall code coverage, you've found the missing piece of the puzzle

[Jean85]

Would it be possible to save some time by merging the coverage information "on the fly"? In Paraunit I do that incrementally as soon as a single process terminates its tests.

[Slamdunk]

@Jean85 I don't think that would make any meaninful difference, but I'd be happy to be proven wrong

[staabm]

I just added CoversNothing to 13 integration test-classes which had interessintg results:

no changes to the remaining thousands of test-classes.

before this PR:

Time: 07:57.258, Memory: 618.50 MB

OK, but some tests were skipped!
Tests: 13334, Assertions: 80601, Skipped: 171.

Generating code coverage report in PHPUnit XML format ... done [00:34.829]

after this PR:

Time: 03:44.550, Memory: 660.50 MB

OK, but some tests were skipped!
Tests: 13334, Assertions: 80601, Skipped: 171.

Generating code coverage report in PHPUnit XML format ... done [00:32.989]

it got twice as fast, but also requires more memory..?

[staabm]

Would it be possible to save some time by merging the coverage information "on the fly"? In Paraunit I do that incrementally as soon as a single process terminates its tests.

I agree that - as long as this report generation takes tens of seconds - it would likely speedup the overall process when it would be incrementally build

I will give it a try

[staabm]

I don't think that would make any meaninful difference, but I'd be happy to be proven wrong

@Slamdunk hopefully #1045 proved you wrong enough :-).

would be great to get your opinion on that one

[Slamdunk]

I'm waiting for the build to get green before reviewing it 😉

[staabm]

summary: the coverage-merging itself is not slow, but the report generation with coverageManager->generateReports - so its a phpunit-code-coverage root problem

[staabm]

PHPUnit Coverage XML Report generation will be ~2x faster with sebastianbergmann/php-code-coverage#1125