NXP backend: Improve test result gathering

[roman-janik-nxp]

Summary

Only last 2 commits are part of this PR. Requires upgrade to NC 3.1.2 PR.
Feature for creating improved test results for reporting when test log-level is set to DEBUG. Adds IR models to .outputs/test_dir, result tensor differences, text variant of input and result tensors, summary with test info.

Test plan

Test provided. All tests that use NSYS.

cc @robert-kalmar

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20024

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 New Failures

As of commit f3bd95d with merge base 9898bc5 ():

NEW FAILURES - The following jobs have failed:

• pull / test-multimodal-linux (gemma3-4b) / linux-job (gh)
  RuntimeError: Command docker exec -t fb75638c2e5b5126e20d14d22156dd812e467b3710b43160edc5574b0887c673 /exec failed with exit code 139
• pull / unittest-nxp-neutron / linux-job (gh)
  ../../dev/ir/converter/node_converter/test_sub_tensor_converter.py::TestSubTensor::test__w_conv_broadcast[2
• Test QNN Backend / test-qnn / test-backend-linux (qnn, operators) / linux-job (gh)
  RuntimeError: Command docker exec -t 0fd9de24f7ac0b181f5c2f793649b53ad7a6143748e14adb7de09c0b9ca29ff2 /exec failed with exit code 92
• trunk / test-qnn-optimum-model (fp32, pvt) / linux-job (gh)
  RuntimeError: Command docker exec -t bd917ecaa29f326c77aedffc133eeb0dfa6d20e648741b2ac405ae0d6995ed04 /exec failed with exit code 92

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[linux-foundation-easycla]

• ❌ - login: @roman-janik-nxp / name: roman-janik-nxp. The commit (379ea17, f3bd95d) is not authorized under a signed CLA. Please click here to be authorized. For further assistance with EasyCLA, please visit our EasyCLA portal and chat with our support bot.

[roman-janik-nxp]

@novak-vaclav

[MartinPavella]

How does this work when there are more delegated partitions? The file names suggest it only works with 1 such partition.

[roman-janik-nxp]

I counted with single partition, as multiple partitions means there is a problem. But we also need to take these cases into account. As Robert pointed out the copy-solution unfortunate, so I changed it to pass test_dir to Neutron backend via CompileSpec and store all partitions directly there.

[robert-kalmar]

This is a quite impractical workaround for the code in executorch/backends/nxp/nxp_backend.py::NeutronBackend::preprocess() which stores the debug files into current working directory.

Now you have:

• logic in lower_run_compare, which knows where to properly store the test files - derived from test ID from request
• logic in NeutronBackend::preprocess storing them into current working directory, what e.g. Arm handles with passing the intermediates folder in compile_spec, what is acceptable, see https://github.com/pytorch/executorch/blob/main/backends/arm/scripts/aot_arm_compiler.py#L511
• logic in this function (_run_delegated_executorch_program) which stores the pytorch/executorch visualized graphs (.json files) into the test_dir what is kind of OK as it is called from lower_run_compare but unflexible:

Unflexible because: the folder name for intermediate artefacts is created in lower_run_compare based on request fixture. The lower_run_compare is quite low level function to know about request [ https://en.wikipedia.org/wiki/Principle_of_least_privilege ] . Why not to move the test_dir name creation logic into test itself and pass it into utility functions. Such a change also opens the possibility to store intermediate artefacts also from the tests not calling the lower_run_compare , like assert_not_delegated tests.

And as Martin pointed out, you must rely on specific tag name to avoid mistakenly copying garbage.

[roman-janik-nxp]

I agree the copy-solution is unfortunate, so I changed it to pass test_dir to Neutron backend via CompileSpec and store all partitions directly there. Now the storing logic is in one place. I made the test_dir param inside our pipeline optional, so the default store dir is still cwd.
I also wouldn't move the test_name extraction outside of lower_run_compare as I consider it as the one compact API for NSYS test. Other type of tests typically don't generate such test results and the test_name can still be be extracted from request in those tests.

[novak-vaclav]

Provided additional information to some of my comments. Otherwise LGTM