initial support for automated testing with NTFC

[szafonimateusz-mi]

Summary

First step to solve #17717

This PR adds support for NTFC in CI and migrates sim/citest to use NTFC instead of tools/ci/testrun tool.
The rest of citest configs uses the old testing method.

635cb70 must be also ported to nuttx-apps.
EDIT: PR for nuttx-apps is here apache/nuttx-apps#3291 [DONE]

Impact

NTFC performs more tests so sim-01 will take longer but will provide better code coverage.
EDIT: With this change Linux sim-01 took 1h 13m 56s, before this change it took 58m 25s.

Testing

Tested in a private repository, let's see if it works with upstream.

Here is an example run test for sim/citest on my fork with all tests passed:
https://github.com/szafonimateusz-mi/nuttx/actions/runs/20580444513/job/59106747416#step:11:154

I will update this section when upstream CI is finished.

EDIT: successful run is here:
https://github.com/apache/nuttx/actions/runs/20582851657/job/59113926720?pr=17728#step:11:247

Test start:

Test end:

Linux sim-01 took 1h 13m 56s, before this change it took 58m 25s, so tests for this target take 15 minutes more (not bad):

Summary from pytest before this PR:

================ 1058 passed, 10 skipped in 2735.05s (0:45:35) =================

Summary from pytest for this PR:

================ 1224 passed, 548 skipped in 3625.97s (1:00:25) ================

[szafonimateusz-mi]

I have updated the description according to the CI results. Linux sim-01 took 1h 13m 56s, before this change it took 58m 25s, so tests take 15 minutes more. For comparison, the entire CI workflow took 3 hours.

Summary from pytest before this PR:

================ 1058 passed, 10 skipped in 2735.05s (0:45:35) =================

Summary from pytest for this PR:

================ 1224 passed, 548 skipped in 3625.97s (1:00:25) ================

[acassis]

What do you want documented? The NTFC documentation is in the NTFC repository https://github.com/szafonimateusz-mi/nuttx-ntfc/

Sorry, I missed that Running CI Tests with tools/ci/testrun section. So I think it is fine

[szafonimateusz-mi]

@acassis No need to sorry, I updated this file after your comment :P I've also created a new top section in the docs: "Testing." Ultimately, CI's operation can be better described there.

[hartmannathan]

Hi all, as I said in my review comment, I hope the plan is to eventually move the framework and official test cases under the apache/nuttx repositories.

[simbit18]

Question:

Could this repository Apache NuttX Testing

be reused to contain both repositories NuttX Test Framework and NTFC test cases ?

[raiden00pl]

@simbit18 we need 2 separate repositories. NTFC is not dependent on NTFC test cases in any way, the user can use their own test cases. In addition, NTFC is organized as a Python package and has different requirements regarding maintainence and CI.

@hartmannathan yes, it will be moved to Apache, we just need one more repository for nuttx-ntfc. Do we need INFRA to create a new repository or can we do it ourselves?

[hartmannathan]

@simbit18 we need 2 separate repositories. NTFC is not dependent on NTFC test cases in any way, the user can use their own test cases. In addition, NTFC is organized as a Python package and has different requirements regarding maintainence and CI.

@hartmannathan yes, it will be moved to Apache, we just need one more repository for nuttx-ntfc. Do we need INFRA to create a new repository or can we do it ourselves?

@raiden00pl We can create repositories using INFRA's Self-Service tools, but I think that will create empty repositories. I think what we need in this case is to clone the repos; correct me if this isn't the case. I don't see a way to create a repo as a clone. We can ask INFRA to do it by opening a Jira ticket and telling them the URLs of the repos we want cloned... Thoughts?

[raiden00pl]

@hartmannathan We can clone the repo with INFRA, but we can also do it differently: create an empty repository and add all changes in one pull request. Both options are OK for me, the question is which one is faster :)

[raiden00pl]

Here's a PR for nuttx-testing adding test cases to repo: apache/nuttx-testing#102

[hartmannathan]

@hartmannathan We can clone the repo with INFRA, but we can also do it differently: create an empty repository and add all changes in one pull request. Both options are OK for me, the question is which one is faster :)

@raiden00pl I don't know, but let's try the way you suggested. Would you like to do the honors and setup the new repos? (If you'd like me to do it, I'll need to know what repository names you'd like...)

[lupyuen]

Sorry @szafonimateusz-mi would you be able to Retry the Failed Downloads for NTFC? According to the log below: When NTFC fails to download some dependencies, all the CI Jobs will fail. Which will waste GitHub Runners (1.5 hours elapsed time, 11 hours chargeable time) and frustrate our devs. Thanks :-)

FYI: We are now testing the Retry of CI Builds, with Random Exponential Backoff. Hopefully it will fix any Failed Downloads for ESP32 Runtime, LVGL, OpenAMP, Device Tree, etc:

• [Do Not Merge] Testing of CI Build Retry #18576

Download Failure for NTFC: https://github.com/apache/nuttx/actions/runs/23468549943/job/68286177498#step:10:64

Err:10 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages
  File has unexpected size (1302502 != 1302485). Mirror sync in progress? [IP: 91.189.91.83 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/dists/jammy-security/universe/binary-amd64/Packages.gz  File has unexpected size (1302502 != 1302485). Mirror sync in progress? [IP: 91.189.91.83 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/dists/jammy-security/universe/binary-i386/Packages.gz  
E: Failed to fetch http://security.ubuntu.com/ubuntu/dists/jammy-security/main/binary-i386/Packages.gz  

Update: Found another NTFC Download Failure from #17573: https://github.com/apache/nuttx/actions/runs/20914875922/job/60086076606#step:10:206

Err:2 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 python3.10-venv amd64 3.10.12-1~22.04.13
  404  Not Found [IP: 91.189.91.81 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/universe/p/python3.10/python3.10-venv_3.10.12-1%7e22.04.13_amd64.deb  404  Not Found [IP: 91.189.91.81 80]

Another one from the same PR #17573: https://github.com/apache/nuttx/actions/runs/20777559190/job/59667205045#step:10:95

E: Failed to fetch http://security.ubuntu.com/ubuntu/dists/jammy-security/main/binary-amd64/Packages.gz  File has unexpected size (3633035 != 3633630). Mirror sync in progress? [IP: 91.189.91.81 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/dists/jammy-security/universe/binary-amd64/Packages.gz  
E: Failed to fetch http://security.ubuntu.com/ubuntu/dists/jammy-security/main/binary-i386/Packages.gz  

FYI @simbit18 :-)

[lupyuen]

I made a video to explain the NTFC Download Failure. We have some evidence that NTFC Download Failure contributed to the Overuse of GitHub Runners in Jan-Feb 2026. So I really hope our NTFC Maintainers can implement the Retry of NTFC Downloads soon. Thanks :-)

• Apache NuttX RTOS: NTFC Download Failure

[raiden00pl]

@lupyuen most of these downloads should disappear when I move the package to pip. For now all these downloads are required to install venv support (even system update). I'll eventually build NTFC into Docker image, it won't take up much space, then the only package to download will be NTFC test repo. I'm also wondering whether NTFC test repo should be integrated into Docker image, then NTFC won't need any download

[lupyuen]

@raiden00pl Cool thanks! Please inform @simbit18 and me when the migration is done, we will have to monitor for errors.