[Feature request] Including instruction elements in the report

[olw-medcom]

Hey there,

Currently it seems that the pdf report only includes input parameters from an element and not the instructions.
I.e. if I have something like this (pseudocode):

<interact>
  <instruct desc="instruction">"Please submit a screenshot of a donkey"</instruct>
  <request inputType="UPLOAD"/>
</interact>

Then the pdf report omits the instructions that the screenshot should be of a donkey. This makes it tough to read a pdf report on it's own as it lacks the required context.
So would it be possible to include the elements?

Best regards

[costas80]

Sounds good to me. The way I see this, this will extend further than the PDF report but will rather consider instructions for all steps reports. This means:

• Viewing the step's report on the UI when clicking the step on the diagram.
• The step's XML report (as other XML reports that include step details).
• The step's PDF report (as other XML reports that include step details).

I'll let you know when this is available.

[olw-medcom]

You read my mind! It would be great to also have it on the step itself in the UI. Thank you!

[costas80]

Ready to pull!

With this update you can include instruct elements in reports by specifying report to true (this is false by default). We are not including instructions by default in reports to remain backwards-compatible in terms of behavior - instructions are often simple help notices that users may very well not want to include by default. With this attribute you choose which instructions you want preserved.

There is however something to note about the implementation which I illustrate below. Say you have an interaction step in your test case like this (added a couple instruction variants - see also the new report attribute used) ...

<interact id="userData" desc="User instructions">
   <request name="file1" inputType="UPLOAD" required="true" desc="Test image file"/>
   <instruct desc="Sample file" mimeType="application/json" report="true">$remoteJsonSample</instruct>
   <instruct desc="Next step" level="INFO" forceDisplay="true" showControls="false" report="true">"Please notify the administrator and include a direct link to the test or the session id found in this window."</instruct>
   <instruct desc="This is an additional message displayed without a title." forceDisplay="true" showControls="false" report="true"></instruct>
</interact>

... which renders while executing the test session as follows ...

... the step's report viewed through the UI is as follows ...

... and in the PDF report as follows:

As you likely noticed, the instruction elements are included but you don't get the same display as when executing the interaction. This is because attributes such as level and forceDisplay are ignored, given that the reporting output format used in steps' reports differs from the test case definition. I am reluctant to align these "1 to 1" given that the attributes in question are purely about presentation which is not what the reporting output format is about. In my view this makes sense in the context of the step report but I wanted to highlight and cross-check it with you.

[olw-medcom]

This is a massive improvement to us as reviewers. If I am to review, as an admin, what the vendors have made but I don't quite recall what they were asked to do, this helps a ton. It basically makes it so I can just click into their step report and see what they were asked to do.
I will say though that it would be even further improved if attributes such as level and forceDisplay were rendered so that it instruct elements are as close to the original step report as possible. Am I reading between the lines that the output format is a bit more troublesome to get these things rendered?

Regardless, this is a big improvement, so thank you, even if it stays as is.

[costas80]

Am I reading between the lines that the output format is a bit more troublesome to get these things rendered?

It's not a matter of this being troublesome, it's more related to the purpose of the reporting format. This currently records step outcomes and their data but not hints for visual tweaks (e.g., required/optional, always show text, severity level styling, show/hide controls). We could just replicate all this to the reporting format but it would then make it aware of how things are presented to users and rendered as input controls (of which the reporting format is currently agnostic).

Regardless, this is a big improvement, so thank you, even if it stays as is.

I hear you. I also understand that my concerns about separation of concerns probably seem academic from the outside. I'll give this some extra thought and see if we revisit before freezing the current release. Lets keep this issue open until then.

[olw-medcom]

Alright!