Tests for test suite reordering fail

[sebastianbergmann]

I pulled #6338 into 10.5 and merged it from there to 11.5, 12.4, 12.5, and main.

On PHPUnit >= 12, two tests, order-by-defects-test-classes-with-defects.phpt and order-by-defects-test-methods-with-defects.phpt, now fail.

These test failures need to be investigated. Hopefully, these tests fail for the obvious reason: that the result cache fixture file needs to be updated.

[sebastianbergmann]

@staabm I can't see the forest for the trees at the moment, maybe a fresh pair of eyes is needed? Could you take a look? Thanks!

[staabm]

Looking at tests/end-to-end/execution-order/order-by-defects-test-classes-with-defects.phpt, I wonder why the expectation is that after the 2 failled tests namely testThree and testTwo - the 2 succeeding tests are expected in the order testFour, testOne.

shouldn't they be executed in the order of declaration in the test-class, which means testOne before testFour (and thats what the test runner in PHPUnit 12.x is doing)

I guess what I am saying is, that the tests looks for me to work like expected starting with 12.x.
maybe I miss something?

[sebastianbergmann]

Thank you for your feedback, I did not think of this possibility.

As explained in #6338 (comment), to me, these new tests are characterization tests meaning they characterize the current behaviour. If you are correct, then the characterization tests I originally wrote for PHPUnit 10 now show us that PHPUnit 12 behaves differently.

[staabm]

One PR I remember related to sorting is #6263

which landed in PHPUnit 11.5+, which means at least for cases which use data-provider there might be a difference between PHPUnit 10 and PHPUnit 11+ (but this does not explain the difference we see regarding the above comments)

[sebastianbergmann]

The new tests work as-is for both PHPUnit 10 and PHPUnit 11, so the PR you mention is unrelated.