Passing expectedFailure (aka TODO test) should be treated as a failure

[mcepl]

My problem is with this text (in the “TODO tests” part of the current specification):

Should a todo test point begin succeeding, the harness may report it in some way that indicates that whatever was supposed to be done has been, and it should be promoted to a normal Test Point.

That’s actually a difference from what for example Python’s unittest.expectedFailure says:

Mark the test as an expected failure or error. If the test fails or errors in the test function itself (rather than in one of the test fixture methods) then it will be considered a success. If the test passes, it will be considered a failure.

Pytest at least treats passed xfails separately:

An xfail means that you expect a test to fail for some reason. A common example is a test for a feature not yet implemented, or a bug not yet fixed. When a test passes despite being expected to fail (marked with pytest.mark.xfail), it’s an xpass and will be reported in the test summary.

For me, treatment of passed expectedFailures was the reason why to abandon use of the TAP protocol for M2Crypto, and it seems to me that the messy state of the specification bothers others as well (that reporter is probably lobbying more for the pytest-style treatment of counting them separately, I would be happy if the passed TODO tests were just considered a failure).

Cc: @mblayman @lorcap

[Ovid]

I would be happy if the passed TODO tests were just considered a failure

So would I and I've argued for that in the past (not here, though). However, they've been like this for years, so suddenly changing them to fail if passing would break tons of test suites and have people very unhappy.

We can't break the contract. We can only extend it. I've seen enough test suites (ab)using TODO that I'm sure that even bumping this to version 15 and labeling it as a breaking change would hinder adoption.