Out of order IDs mixed with missing IDs

[rhansen]

TAPv14 says:

Test Points may be output in any order, but any Test Point ID provided must be within the range described by the Plan.

TAPv14 also says:

If there is no number, the harness must maintain its own counter until the script supplies test numbers again.

How do these two interact? For example, how should a harness process the following document:

1..4
ok 2
ok
ok 1
ok 3

I can see several reasonable choices for the implied ID of that second test point:

• 1 because that was the lowest ID not yet affiliated with any test points when the second test point was observed
• 2 because the ID-less test point was the second test point observed
• 3 because that is one greater than the highest affiliated test point ID when the second test point was observed
• 4 because that was the only ID not affiliated with another test point at the conclusion of the TAP document

My opinion: If the N^th test point in the TAP document does not have an ID, then the harness gives it ID N even if ID N is already affiliated with another test point. Thus, the above stream would be equivalent to:

1..4
ok 2
ok 2
ok 1
ok 3

Rationale:

• It is the easiest for a harness to implement.
• A test point's ID is immediately known when the test point is observed.
• The ID is guaranteed to be within the plan's range unless there are too many test points.
• It would effectively require TAP document producers to always include the ID with their test points if the test points might be written out of order.

Thoughts?

[kinow]

I'd say 4th option, but depending on the tool implementing it.

[Leont]

Wait, since when do we allow that? I think I missed that in #25, but I'm not sure that is compatible with "No changes will be accepted UNLESS they are already widely adopted by multiple implementations representing a majority of the TAP userbase."

I'm not sure I understand how such a thing could happen or what its use would be either to be honest. Why would you add numbers at all if they're going to be out of order anyway?

[rhansen]

I'd say 4th option, but depending on the tool implementing it.

With the 4th option, harnesses would not be able to display IDs for ID-less test points until after testing is complete. Also, repeated runs of the same test suite might result in different IDs assigned to the ID-less test points (the test points are not in order presumably because they run in parallel).

Thinking about this more, here's a 5th option that might be better than the 4th option:

5. The harness does not assign an ID to an ID-less test point; it remains unnumbered. With regards to ID enforcement, all the harness must do is:

   • Indicate an error if the total number of test points does not match the plan.
   • Indicate an error if a test point ID is outside the plan's range.
   • Indicate an error if the test point IDs observed so far have been monotonically increasing and a test point ID is repeated. (The monotonicity condition means that this check only requires O(1) state.)

   In addition, the harness should:

   • Indicate an error if any test point ID is repeated. (This requires maintaining O(N) state when test points are in arbitrary order, which is why it is a should instead of must.)

I'm not sure I understand how such a thing could happen or what its use would be either to be honest.

Out-of-order test points can happen if the test cases are running concurrently.

Why would you add numbers at all if they're going to be out of order anyway?

To make it easier for a human to locate the corresponding code when a test fails.

[Leont]

I'm not sure I understand how such a thing could happen or what its use would be either to be honest.

Out-of-order test points can happen if the test cases are running concurrently.

Why would you add numbers at all if they're going to be out of order anyway?

To make it easier for a human to locate the corresponding code when a test fails.

That doesn't directly answer my question, but taking a step back I'm guessing that what you're trying to tell me is that you use a TAP producer framework where you have to pass the identifiers manually? I'm not sure I understand why you'd do that.

[rhansen]

That doesn't directly answer my question

Sorry, I must have misunderstood your question. Would you please rephrase it?

I'm guessing that what you're trying to tell me is that you use a TAP producer framework where you have to pass the identifiers manually? I'm not sure I understand why you'd do that.

I'm currently working on updating the Automake TAP harness to follow the TAP14 spec. How to handle unnumbered test points when they could be out of order was a question that came up during development work.

---

In addition to the specific question posed in my original post above, I'm trying to understand the overall purpose of test point IDs in the spec to help me make the Automake TAP harness satisfy the intent. A similar effect can be achieved by putting an identifier in the description, so why does the spec go to the effort of documenting IDs at all? These are the reasons that immediately came to mind:

• To enable the harness to perform some basic sanity checks on the TAP document. This is nice, but not worth complicating the spec or implementations in my opinion.

• To provide a way for a harness to reliably correlate test points across multiple runs. This can be used to analyze flakiness, or to highlight the effects of a code change. This is indeed a very useful property, but only if:

  • the IDs are required (or the harness can reproducibly assign IDs to unnumbered test points), and
  • duplicate IDs are strictly prohibited, and
  • IDs are stable.

  Test point IDs in the TAP14 spec provide none of those guarantees:

  • "TAP expects the ok or not ok to be followed by an integer Test Point ID. If there is no number, ..."
  • "Test Point IDs should be unique within the TAP Document." And, "Harnesses may warn about repeated Test Point IDs or treat them as a test failure, ..."
  • "... any Test Point ID provided must be within the range described by the Plan." (If a test case is removed, at least one of the remaining test cases must have its ID changed to fit in the newly reduced plan range.)

  Because of the lack of guarantees, harnesses must rely on other mechanisms such as prohibiting description changes, similarity heuristics, or embedding UUIDs in the description. So test point IDs aren't very useful for this purpose.

Maybe there's something I'm missing?

[Ovid]

My thoughts, as the primary author of the TAP::Harness which currently ships with Perl (maintained by @Leont above). Short answer: I largely agree with @rhansen and much of what I write below is an extended ++

When our team was building out what was is now the test harness which ships with Perl, test numbers were one of many sources of pain. Hated 'em because the more state you have to maintain in an app, the more bugs you potentially have and bugs in a test harness are not a good thing. The v14 spec says this:

TAP expects the ok or not ok to be followed by an integer Test Point
ID. If there is no number, the harness must maintain its own counter
until the script supplies test numbers again.

Great more state. And that's where it gets ambiguous. I honestly don't know how to interpret the above. Tests are often run concurrently, with not guaranteed order, so subsequent test runs could result in different outputs, particularly if test descriptions are omitted. I like to maintain test history over time and now I can't correlate them.

It also means that I could easily wind up assigning a non-unique test point ID:

Test Point IDs should be unique within the TAP Document. Harnesses may
warn about repeated Test Point IDs or treat them as a test failure, but
must not treat a Test Point with a re-used ID as a non-TAP line.

Great. Now my ID is not longer an ID. How do I build tooling without guarantees? The "ID" is not guaranteed to be unique or to even exist. The test description is not guaranteed to exist.

Also, ideas such as this are problematic:

"4 because that was the only ID not affiliated with another test point at the conclusion of the TAP document"

Part of the rationale of TAP is that it is a streaming document. Unlike XML, you can parse TAP line-by-line, which means we can stream the output to a terminal as the tests are running. I love that. I am unsure how to safely do that if I have to wait for a document to complete and then have the harness go back and "backfill" test IDs.

I haven't given it a huge amount of thought, but I like @rhansen's suggestion:

The harness does not assign an ID to an ID-less test point; it remains unnumbered.

That fits with the intent of what TAP was originally designed to do. If I can't write better tooling for TAP because the test authors are writing bad output, so be it. That's where we are, but I'd hate for us to go and try to jury-rig a solution for problem which, I suspect, doesn't come up often.

I suspect it also means that, if agreed upon, that current TAP implementations probably already follow this.

---

Side note: I would love for a future version of TAP to figure out a way of identifying "strict" TAP that ensures that every line of TAP is unambiguous and can be clearly matched against similar lines, if existing, in other TAP documents. I've never quite figured out how to make that work without mandatory test descriptions and mandatory unique IDs.