kola --allow-rerun-success allows rerun success for any test

[dustymabe]

The original intent was that you'd allow rerun success for only tests that met certain conditions. For example kola run --rerun --allow-rerun-success tags=needs-internet but apparently this doesn't work as intended because even if a test doesn't have the needs-internet tag the framework will accept rerun success as success:

To show this I forced a failure with this in f-c-c:

$ git diff
diff --git a/tests/kola/files/aleph-version b/tests/kola/files/aleph-version
index acea8873..41a29f5e 100755
--- a/tests/kola/files/aleph-version
+++ b/tests/kola/files/aleph-version
@@ -10,5 +10,6 @@ set -xeuo pipefail
 # shellcheck disable=SC1091
 . "$KOLA_EXT_DATA/commonlib.sh"
 
+fatal 'forced failure'
 jq . < /sysroot/.coreos-aleph-version.json >/dev/null
 ok aleph

Then you can see it fail and attempt a rerun even with some fabricated tag:

Details

[coreos-assembler]$ cosa kola run --rerun --allow-rerun-success tags=foo '*aleph*'                                                                                                           
kola -p qemu run --rerun --allow-rerun-success tags=foo *aleph* --output-dir tmp/kola
=== RUN   ext.config.files.aleph-version
systemctl status kola-runext-19.service:
× kola-runext-19.service
     Loaded: loaded (/etc/systemd/system/kola-runext-19.service; static)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Tue 2026-05-05 13:29:03 UTC; 107ms ago
   Duration: 15ms
 Invocation: 766bced3c00242f9aa16ae6f10178afe
    Process: 1930 ExecStart=/usr/local/bin/kola-runext-aleph-version (code=exited, status=1/FAILURE)
   Main PID: 1930 (code=exited, status=1/FAILURE)
   Mem peak: 1.2M
        CPU: 5ms

May 05 13:29:03 qemu0 systemd[1]: Started kola-runext-19.service.
May 05 13:29:03 qemu0 kola-runext-aleph-version[1930]: + . /var/opt/kola/extdata-19/commonlib.sh
May 05 13:29:03 qemu0 kola-runext-aleph-version[1930]: ++ IFS=' '
May 05 13:29:03 qemu0 kola-runext-aleph-version[1930]: ++ read -r -a cmdline
May 05 13:29:03 qemu0 kola-runext-aleph-version[1930]: + fatal 'forced failure'
May 05 13:29:03 qemu0 kola-runext-aleph-version[1930]: + echo 'forced failure'
May 05 13:29:03 qemu0 kola-runext-aleph-version[1930]: forced failure
May 05 13:29:03 qemu0 kola-runext-aleph-version[1930]: + exit 1
May 05 13:29:03 qemu0 systemd[1]: kola-runext-19.service: Main process exited, code=exited, status=1/FAILURE
May 05 13:29:03 qemu0 systemd[1]: kola-runext-19.service: Failed with result 'exit-code'.
--- FAIL: ext.config.files.aleph-version (25.29s)
        harness.go:1308: kolet failed: : kolet run-test-unit failed:  Error: Unit kola-runext-19.service exited with code 1
2026-05-05T13:29:03Z cli: Unit kola-runext-19.service exited with code 1: Process exited with status 1
FAIL, output in tmp/kola


======== Re-running failed tests (flake detection) ========

=== RUN   ext.config.files.aleph-version
systemctl status kola-runext-19.service:
× kola-runext-19.service
     Loaded: loaded (/etc/systemd/system/kola-runext-19.service; static)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Tue 2026-05-05 13:29:29 UTC; 99ms ago
   Duration: 18ms
 Invocation: f805affa0d0a40dcba14ee87fb59f2ab
    Process: 1933 ExecStart=/usr/local/bin/kola-runext-aleph-version (code=exited, status=1/FAILURE)
   Main PID: 1933 (code=exited, status=1/FAILURE)
   Mem peak: 1.2M
        CPU: 5ms

May 05 13:29:29 qemu0 systemd[1]: Started kola-runext-19.service.
May 05 13:29:29 qemu0 kola-runext-aleph-version[1933]: + . /var/opt/kola/extdata-19/commonlib.sh
May 05 13:29:29 qemu0 kola-runext-aleph-version[1933]: ++ IFS=' '
May 05 13:29:29 qemu0 kola-runext-aleph-version[1933]: ++ read -r -a cmdline
May 05 13:29:29 qemu0 kola-runext-aleph-version[1933]: + fatal 'forced failure'
May 05 13:29:29 qemu0 kola-runext-aleph-version[1933]: + echo 'forced failure'
May 05 13:29:29 qemu0 kola-runext-aleph-version[1933]: forced failure
May 05 13:29:29 qemu0 kola-runext-aleph-version[1933]: + exit 1
May 05 13:29:29 qemu0 systemd[1]: kola-runext-19.service: Main process exited, code=exited, status=1/FAILURE
May 05 13:29:29 qemu0 systemd[1]: kola-runext-19.service: Failed with result 'exit-code'.
--- FAIL: ext.config.files.aleph-version (26.56s)
        harness.go:1308: kolet failed: : kolet run-test-unit failed:  Error: Unit kola-runext-19.service exited with code 1
2026-05-05T13:29:29Z cli: Unit kola-runext-19.service exited with code 1: Process exited with status 1
FAIL, output in tmp/kola/rerun
Error: harness: test suite failed
2026-05-05T13:29:34Z cli: harness: test suite failed
failed to execute cmd-kola: exit status 1

[dustymabe]

I think I'm going to argue here that we just make the current behavior the intended behavior (i.e. we update the design to just allow rerun success for any test and not try to limit it to tests of a certain tag.

It would change the semantics of the argument. i.e. --allow-rerun-success could now become a boolean and not accept any value, which would be ideal, but wouldn't be backwards compatible. So we'd need to either

1. backport this "fix" to older COSA branches still in operation
2. OR change the pipeline code to have different behavior based on the stream
3. not change the argument in a backwards incompatible way

[nikita-dubrovskii]

For those who are not aware, let me explain the idea behind --rerun and --allow-rerun-success tags= (@dustymabe, please correct me if I'm wrong):

• The --rerun option just tells kola to rerun failed tests to detect possible flakes, but the return value should still be EXIT_FAILURE (currently broken):

  [coreos-assembler]$ ./kola run --rerun '*aleph*' --output-dir tmp/kola
  === RUN   ext.config.files.aleph-version
  --- FAIL: ext.config.files.aleph-version (24.20s)
  ======== Re-running failed tests (flake detection) ========
  === RUN   ext.config.files.aleph-version
  --- PASS: ext.config.files.aleph-version (25.30s)
  PASS, output in tmp/kola/rerun
  Error: harness: test suite failed
  2026-05-13T09:17:07Z cli: harness: test suite failed
  [coreos-assembler]$ echo $?
  1

• --allow-rerun-success tags= was introduced to return EXIT_SUCCESS if reruns of tests with the given tags succeeded, or EXIT_FAILURE otherwise (currently broken):

  [coreos-assembler]$ ./kola run --rerun --allow-rerun-success tags=foo '*aleph*' --output-dir tmp/kola
  === RUN   ext.config.files.aleph-version
  --- FAIL: ext.config.files.aleph-version (25.28s)
  ======== Re-running failed tests (flake detection) ========
  === RUN   ext.config.files.aleph-version
  --- PASS: ext.config.files.aleph-version (24.00s)
  Error: harness: test suite failed
  2026-05-13T09:08:33Z cli: harness: test suite failed
  [coreos-assembler]$ echo $?
  1

  • We also added support for tags=all or tags=* to return EXIT_SUCCESS when all failed tests rerun successfully, regardless of tags:

    [coreos-assembler]$ ./kola run --rerun --allow-rerun-success tags=all '*aleph*' --output-dir tmp/kola
    === RUN   ext.config.files.aleph-version
    --- FAIL: ext.config.files.aleph-version (24.21s)
    ======== Re-running failed tests (flake detection) ========
    === RUN   ext.config.files.aleph-version
    --- PASS: ext.config.files.aleph-version (25.17s)
    [coreos-assembler]$ echo $?
    0

  • Later, kola became smarter and, in several failure scenarios, automatically added the special tag AllowRerunSuccessTag = "allow-rerun-success" to the list of tags set by the user.

Between proposed solutions

So we'd need to either

1. backport this "fix" to older COSA branches still in operation
2. OR change the pipeline code to have different behavior based on the stream
3. not change the argument in a backwards incompatible way

I would select 1) and 3). Alternatively, we could remove all the code for --allow-rerun-success tags=, but keep the option as a no-op (ignored). In that case, --rerun would effectively behave as --rerun --allow-rerun-success tags=all.

This patch broke it - dd4a78f#diff-9db99bb7fefa41cf89f75a965efa109bc8349ab0a55fdb093fa46775724791cbR866-R868 .

[dustymabe]

Thanks for your analysis @nikita-dubrovskii.

@marmijo @jbtrystram @prestist WDYT? I don't have a strong opinion.

[jbtrystram]

WDYT? I don't have a strong opinion.

I never dug into those details, today I learned, thanks @nikita-dubrovskii .
My assumption until now was that kola was behaving with --rerun --allow-rerun-success tags=all in the pipeline, and that makes total sense for me : rerun failed tests to make sure they're not flakes.

Now that I read this, yes, it makes sense to not rerun everything, especially when there is not a lot of room for flakyness (e.g. no network, no I/O heavy operations). But then, is the re-run really that expensive ? I think not.

I'd vote for making current behavior the intended behavior as well.

Alternatively, we could remove all the code for --allow-rerun-success tags=, but keep the option as a no-op (ignored). In that case, --rerun would effectively behave as --rerun --allow-rerun-success tags=all.

This makes a lot of sense to me, especially for the maintenance perspective as it would make the code simpler.

[nikita-dubrovskii]

So, should we proceed somehow here? Fix is ready anyhow