Add NPU tests (New)

[vandah]

Description

This PR adds tests for testing devices with an NPU.
They are currently focused on Intel NPUs which use the accel kernel interface.
The driver for these NPUs is distributed through the intel-npu-driver snap which also includes a gtest-based testing utility npu-umd-test. The tests check the appropriate firmware version is loaded, the user has the appropriate permissions and the rest runs individual tests from the npu-umd-test utility.

The latest version of the intel-npu-driver snap now includes also the configuration file and model files necessary for running the npu-umd-test as well as a known-failures command which I've used to simplify the test filtering. The configuration file and list of known failures will be maintained as part of the intel-npu-driver snap as they can change from one version to another.

Known issues

Some of the test names coming from the npu-umd-test test suite are longer than 80 characters which triggers a warning in checkbox.

Tests

Tests have been run on Meteor Lake and Arrow Lake devices.

Meteor Lake run - submission_2026-02-20T00.04.24.655939.zip

[vandah]

@fernando79513 and @tomli380576, can you please review this PR?

[pseudocc]

Just a passersby reviewer, see inline comments.

[tomli380576]

Test scripts lgtm! Could you provide a bit of documentation on what the expectations are for NPU_UMD_TEST_CONFIG like where it's supposed to be placed at, permissions, what exactly are the expected contents (for example the tree output of a correct setup), etc.

One small question: is the driver snap expected to come preinstalled? If not, I think we should print an error somewhere or mention it in the manifest.

[vandah]

One small question: is the driver snap expected to come preinstalled? If not, I think we should print an error somewhere or mention it in the manifest.

The snap is not pre-installed but ideally the devices that do have the has_npu manifest entry should install the snap before running checkbox... Is there any way in checkbox to define this dependency and maybe even have the snap auto-installed by checkbox?

[pseudocc]

Looks good to me now, thanks!

[tomli380576]

To make this case depend on whether the driver snap exists, add snap.name == "intel-npu-driver" in the requires: section of the test job; but note that this would make checkbox "silently" skip the job and put it in the "job with failed dependencies" pile if the snap is not installed even if the manifest is set to true.

[codecov]

Codecov Report

❌ Patch coverage is 93.61702% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.21%. Comparing base (ab06cc3) to head (19d85dd).
⚠️ Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
providers/base/bin/npu_check_firmware_version.py	90.90%	2 Missing and 2 partials ⚠️
providers/base/bin/npu_check_accel_permissions.py	90.47%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2198      +/-   ##
==========================================
+ Coverage   55.56%   56.21%   +0.64%     
==========================================
  Files         413      425      +12     
  Lines       44692    45873    +1181     
  Branches     8235     8335     +100     
==========================================
+ Hits        24834    25786     +952     
- Misses      19003    19308     +305     
+ Partials      855      779      -76     

Flag	Coverage Δ
provider-base	32.52% <93.61%> (+0.34%)	⬆️
provider-gfx	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[vandah]

To make this case depend on whether the driver snap exists, add snap.name == "intel-npu-driver" in the requires: section of the test job; but note that this would make checkbox "silently" skip the job and put it in the "job with failed dependencies" pile if the snap is not installed even if the manifest is set to true.

I see, I have actually avoided using the dependency exactly so that it would be an explicit failure if the snap is not present but has_npu. In our testing pipeline, we install the snap with testflinger before running checkbox, but I was wondering how to make this more obvious if it gets run by someone else (and how does it work with dependencies when a device gets certified).

As a side note, I think it would be useful to have something like Ansible's failed_when which would allow the tests to fail based on the value of checkbox variables since as I understand, skipped tests have a bit different meaning than failed tests (not applicable vs. something is wrong).

[fernando79513]

I left a few more comments on the PR.
My main concern is with the "non-blocker" jobs. I am not sure under which conditions you expect them to fail, but we should not add them to the test plan if they are not expected to pass.

Apart from that, could you add some submissions of devices in the lab running the NPU tests with the new provider?

[fernando79513]

Also, I have a couple of questions.

• How are you going to install the intel-npu-driver snap on the devices that require it for testing? For that, we could use the setup-include feature. But since it's not fully implemented (It's lacking documentation and a few key features), I think we should include it in the corresponding testflinger job.
• Can you share the submission tar file of the checkbox run in the PR description? Or if you want, submit the checkbox job to C3, that's also OK.

[vandah]

Also, I have a couple of questions.

• How are you going to install the intel-npu-driver snap on the devices that require it for testing? For that, we could use the setup-include feature. But since it's not fully implemented (It's lacking documentation and a few key features), I think we should include it in the corresponding testflinger job.

@fernando79513 Thank you for your review! The intel-npu-driver snap is a functional dependency that's needed for the NPU to be used so it is expected to be present on the tested device so installing in a testflinger job sounds perfect 👍. We do that for the intel-npu-driver snap CI already but are there any extra steps needed for certification ? I.e. do we need to provide you with the testflinger jobs and then it gets run by the cert team directly or do we just run the tests and upload to C3 ourselves?

• Can you share the submission tar file of the checkbox run in the PR description? Or if you want, submit the checkbox job to C3, that's also OK.

Sure, I'll upload those.

[fernando79513]

are there any extra steps needed for certification ?

@vandah I think that's an ongoing question and I am not sure I can answer it right now, but if you can attend the office hours this week, it would be a good place to discuss about it

[vandah]

are there any extra steps needed for certification ?

@vandah I think that's an ongoing question and I am not sure I can answer it right now, but if you can attend the office hours this week, it would be a good place to discuss about it

Ok, I'll be there!

[fernando79513]

Thanks a lot for your contribution and your patience during the review process.

I think we are good to go now with this PR.

LGTM +1!

Already reviewed.