kola port clashing for qemu user mode networking

[dustymabe]

When we run kola in the pipeline we run many many tests in a single run, but only 4 at a time (--parallel=4).

It appears, though, that some qemu processes are hanging around longer. For example:

core@coreos-ppc64le-builder:~$ sudo ss -lutnp | grep qemu
udp   UNCONN 0      0                                     0.0.0.0:32771      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=143))
udp   UNCONN 0      0                                     0.0.0.0:33353      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=163))
udp   UNCONN 0      0                                     0.0.0.0:33513      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=185))
udp   UNCONN 0      0                                     0.0.0.0:35914      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=122))
udp   UNCONN 0      0                                     0.0.0.0:36387      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=152))
udp   UNCONN 0      0                                     0.0.0.0:36766      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=133))
udp   UNCONN 0      0                                     0.0.0.0:36842      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=169))
udp   UNCONN 0      0                                     0.0.0.0:36847      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=108))
udp   UNCONN 0      0                                     0.0.0.0:37014      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=118))
udp   UNCONN 0      0                                     0.0.0.0:37312      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=114))
udp   UNCONN 0      0                                     0.0.0.0:37328      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=156))
udp   UNCONN 0      0                                     0.0.0.0:37600      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=165))
udp   UNCONN 0      0                                     0.0.0.0:37746      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=145))
udp   UNCONN 0      0                                     0.0.0.0:37902      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=107))
udp   UNCONN 0      0                                     0.0.0.0:38864      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=168))
udp   UNCONN 0      0                                     0.0.0.0:38962      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=110))
udp   UNCONN 0      0                                     0.0.0.0:40629      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=112))
udp   UNCONN 0      0                                     0.0.0.0:41679      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=159))
udp   UNCONN 0      0                                     0.0.0.0:41731      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=142))
udp   UNCONN 0      0                                     0.0.0.0:42216      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=170))
udp   UNCONN 0      0                                     0.0.0.0:43546      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=149))
udp   UNCONN 0      0                                     0.0.0.0:44382      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=121))
udp   UNCONN 0      0                                     0.0.0.0:44477      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=146))
udp   UNCONN 0      0                                     0.0.0.0:44969      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=106))
udp   UNCONN 0      0                                     0.0.0.0:45113      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=135))
udp   UNCONN 0      0                                     0.0.0.0:45711      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=30)) 
udp   UNCONN 0      0                                     0.0.0.0:45852      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=130))
udp   UNCONN 0      0                                     0.0.0.0:46064      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=126))
udp   UNCONN 0      0                                     0.0.0.0:47320      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=154))
udp   UNCONN 0      0                                     0.0.0.0:47657      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=131))
udp   UNCONN 0      0                                     0.0.0.0:48291      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=187))
udp   UNCONN 0      0                                     0.0.0.0:48428      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=148))
udp   UNCONN 0      0                                     0.0.0.0:50286      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=155))
udp   UNCONN 0      0                                     0.0.0.0:50475      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=140))
udp   UNCONN 0      0                                     0.0.0.0:50478      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=120))
udp   UNCONN 0      0                                     0.0.0.0:51084      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=164))
udp   UNCONN 0      0                                     0.0.0.0:51150      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=158))
udp   UNCONN 0      0                                     0.0.0.0:51467      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=160))
udp   UNCONN 0      0                                     0.0.0.0:51639      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=115))
udp   UNCONN 0      0                                     0.0.0.0:51643      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=162))
udp   UNCONN 0      0                                     0.0.0.0:51867      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=138))
udp   UNCONN 0      0                                     0.0.0.0:51911      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=134))
udp   UNCONN 0      0                                     0.0.0.0:52978      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=144))
udp   UNCONN 0      0                                     0.0.0.0:53949      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=171))
udp   UNCONN 0      0                                     0.0.0.0:54155      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=129))
udp   UNCONN 0      0                                     0.0.0.0:54285      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=166))
udp   UNCONN 0      0                                     0.0.0.0:54296      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=167))
udp   UNCONN 0      0                                     0.0.0.0:54434      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=137))
udp   UNCONN 0      0                                     0.0.0.0:55749      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=189))
udp   UNCONN 0      0                                     0.0.0.0:55831      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=113))
udp   UNCONN 0      0                                     0.0.0.0:55904      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=139))
udp   UNCONN 0      0                                     0.0.0.0:56132      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=111))
udp   UNCONN 0      0                                     0.0.0.0:56266      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=182))
udp   UNCONN 0      0                                     0.0.0.0:56434      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=151))
udp   UNCONN 0      0                                     0.0.0.0:56636      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=141))
udp   UNCONN 0      0                                     0.0.0.0:56930      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=153))
udp   UNCONN 0      0                                     0.0.0.0:57442      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=117))
udp   UNCONN 0      0                                     0.0.0.0:57848      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=136))
udp   UNCONN 0      0                                     0.0.0.0:57890      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=150))
udp   UNCONN 0      0                                     0.0.0.0:58370      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=116))
udp   UNCONN 0      0                                     0.0.0.0:58738      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=157))
udp   UNCONN 0      0                                     0.0.0.0:59310      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=161))
udp   UNCONN 0      0                                     0.0.0.0:59466      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=128))
udp   UNCONN 0      0                                     0.0.0.0:60703      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=119))
udp   UNCONN 0      0                                           *:37513            *:*    users:(("qemu-system-ppc",pid=1975486,fd=127))
udp   UNCONN 0      0                                           *:39929            *:*    users:(("qemu-system-ppc",pid=1975486,fd=125))
udp   UNCONN 0      0                                           *:46748            *:*    users:(("qemu-system-ppc",pid=1975486,fd=124))
udp   UNCONN 0      0                                           *:49099            *:*    users:(("qemu-system-ppc",pid=1975486,fd=132))
udp   UNCONN 0      0                                           *:50749            *:*    users:(("qemu-system-ppc",pid=1975486,fd=123))
udp   UNCONN 0      0                                           *:54827            *:*    users:(("qemu-system-ppc",pid=1975486,fd=147))
tcp   LISTEN 0      1                                   127.0.0.1:33359      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975455,fd=19)) 
tcp   LISTEN 0      1                                   127.0.0.1:33229      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975987,fd=19)) 
tcp   LISTEN 0      1                                   127.0.0.1:38673      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975486,fd=19)) 
tcp   LISTEN 0      1                                   127.0.0.1:41731      0.0.0.0:*    users:(("qemu-system-ppc",pid=1975753,fd=19)) 

The majority of those are owned by the same process so there might be some opportunity for us to figure out how to clean those up, or maybe they are legitimate somehow?

The bigger problem however is that we had a port clash with the kdump.crash.nfs test. The test requires the use of the "Qemu Host" IP:Port 10.0.2.2:2049 so it can end up clashing if something else happens to be using that port already.

Something like this happened recently in the RHCOS pipeline:

Details

[2025-05-30T14:53:17.900Z] + set -xeuo pipefail

[2025-05-30T14:53:17.900Z] ++ umask

[2025-05-30T14:53:17.900Z] + '[' 0022 = 0000 ']'

[2025-05-30T14:53:17.900Z] + cd /home/jenkins/agent/workspace/build-arch

[2025-05-30T14:53:17.900Z] + cosa kola run --rerun --allow-rerun-success=tags=needs-internet --build=latest --output-dir=/home/jenkins/agent/workspace/build-arch/tmp/kola-Wndtl/kola --on-warn-failure-exit-77 --arch=ppc64le '--tag=!reprovision' '--tag=!openshift' --parallel=4

[2025-05-30T14:53:19.782Z] warning: /sys/fs/selinux appears to be mounted but should not be; enabling workaround

[2025-05-30T14:53:19.782Z] kola -p qemu --build latest run --rerun --allow-rerun-success=tags=needs-internet --on-warn-failure-exit-77 --arch=ppc64le --tag=!reprovision --tag=!openshift --parallel=4 --output-dir /home/jenkins/agent/workspace/build-arch/tmp/kola-Wndtl/kola

[2025-05-30T14:53:20.708Z] ⏭️  Skipping kola test pattern "ext.config.version.rhaos-pkgs-match-openshift":

[2025-05-30T14:53:20.708Z]   👉 https://issues.redhat.com/browse/OCPBUGS-42688

[2025-05-30T14:53:20.708Z] ⏭️  Skipping kola test pattern "ext.config.shared.multipath.resilient":

[2025-05-30T14:53:20.708Z]   👉 https://github.com/coreos/fedora-coreos-tracker/issues/1937

[2025-05-30T14:53:21.275Z] === RUN   ostree.hotfix

[2025-05-30T14:53:21.275Z] === RUN   coreos.unique.boot.failure

[2025-05-30T14:53:21.275Z] === RUN   ext.config.shared.disks.lvmdevices

[2025-05-30T14:53:21.275Z] === RUN   ext.config.shared.clhm.ignition-warnings

[2025-05-30T14:53:21.275Z] === RUN   ext.config.shared.containers.quadlet

[2025-05-30T14:53:21.275Z] === RUN   coreos.misc.disk.varlibcontainers

[2025-05-30T14:53:21.275Z] === RUN   coreos.ignition.mount.disks

[2025-05-30T14:53:21.275Z] === RUN   ext.config.shared.networking.nameserver

[2025-05-30T14:53:21.275Z] === RUN   ext.config.shared.boot.grub2-install

[2025-05-30T14:53:21.275Z] === RUN   fips.enable

[2025-05-30T14:53:21.275Z] === RUN   ext.config.shared.ntp.chrony.dhcp-propagation

[2025-05-30T14:53:21.275Z] === RUN   multipath.day1

[2025-05-30T14:53:21.275Z] === RUN   ext.config.shared.ignition.systemd-enable-units

[2025-05-30T14:53:21.275Z] === RUN   ext.config.shared.ignition.kargs

[2025-05-30T14:53:21.275Z] === RUN   ext.config.shared.var-mount.simple

[2025-05-30T14:53:21.275Z] === RUN   ext.config.shared.reboot

[2025-05-30T14:53:21.275Z] === RUN   ext.config.shared.ignition.delete-config

[2025-05-30T14:53:21.275Z] === RUN   ext.config.shared.networking.mtu-on-bond-ignition

[2025-05-30T14:53:21.275Z] === RUN   coreos.ignition.sethostname

[2025-05-30T14:53:21.275Z] === RUN   ext.config.shared.networking.ifname-karg.everyboot-systemd-link-file

[2025-05-30T14:53:21.275Z] === RUN   fips.enable.partitions

[2025-05-30T14:53:21.275Z] === RUN   rhcos.sssd

[2025-05-30T14:53:21.275Z] === RUN   ext.config.shared.networking.no-persist-ip

[2025-05-30T14:53:21.275Z] === RUN   ext.config.shared.gshadow

[2025-05-30T14:53:21.275Z] === RUN   rhcos.selinux.manage

[2025-05-30T14:53:21.275Z] === RUN   coreos.unique.boot.ignition.failure

[2025-05-30T14:53:21.275Z] === RUN   ext.config.shared.podman.rootless-systemd

[2025-05-30T14:53:21.275Z] === RUN   non-exclusive-test-bucket-0

[2025-05-30T14:53:21.275Z] === RUN   ext.config.shared.var-mount.luks

[2025-05-30T14:53:21.275Z] === RUN   ext.config.shared.networking.force-persist-ip

[2025-05-30T14:53:21.275Z] === RUN   ext.config.shared.ssh.custom-host-key-permissions

[2025-05-30T14:53:21.275Z] === RUN   ext.config.shared.networking.prefer-ignition-networking

[2025-05-30T14:53:21.276Z] === RUN   coreos.ignition.groups

[2025-05-30T14:53:21.276Z] === RUN   ext.config.shared.systemd.network-online

[2025-05-30T14:53:21.276Z] === RUN   coreos.selinux.boolean

[2025-05-30T14:53:21.276Z] === RUN   ext.config.shared.networking.rd-net-timeout-carrier

[2025-05-30T14:53:21.276Z] === RUN   ext.config.disk.nvme-compat-symlink

[2025-05-30T14:53:21.276Z] === RUN   ext.config.shared.boot.bootupd-validate

[2025-05-30T14:53:21.276Z] === RUN   coreos.ignition.systemd.enable-service

[2025-05-30T14:53:21.276Z] === RUN   rpmostree.upgrade-rollback

[2025-05-30T14:53:21.276Z] === RUN   ext.config.shared.ignition.stable-boot

[2025-05-30T14:53:21.276Z] === RUN   kdump.crash.ssh

[2025-05-30T14:53:21.276Z] === RUN   ext.config.shared.networking.hostname.fallback-hostname

[2025-05-30T14:53:21.276Z] === RUN   coreos.ignition.failure

[2025-05-30T14:53:21.276Z] === RUN   coreos.ignition.v2.users

[2025-05-30T14:53:21.276Z] === RUN   coreos.auth.verify

[2025-05-30T14:53:21.276Z] === RUN   multipath.day2

[2025-05-30T14:53:21.276Z] === RUN   ext.config.shared.networking.nmstate.policy

[2025-05-30T14:53:21.276Z] === RUN   ext.config.shared.ignition.remote

[2025-05-30T14:53:21.276Z] === RUN   basic

[2025-05-30T14:53:21.276Z] === RUN   rpmostree.status

[2025-05-30T14:53:21.276Z] === RUN   ext.config.shared.networking.team-dhcp-via-ignition

[2025-05-30T14:53:21.276Z] === RUN   ext.config.shared.networking.mtu-on-bond-kargs

[2025-05-30T14:53:21.276Z] === RUN   ext.config.rpm-ostree.replace-rt-kernel

[2025-05-30T14:53:21.276Z] === RUN   ostree.sync

[2025-05-30T14:53:21.276Z] === RUN   ext.config.shared.selinux.enforcing

[2025-05-30T14:53:21.276Z] === RUN   ext.config.shared.networking.ifname-karg.udev-rule-firstboot-propagation

[2025-05-30T14:53:21.276Z] === RUN   ostree.remote

[2025-05-30T14:53:21.276Z] === RUN   ext.config.systemd.journal-compat

[2025-05-30T14:53:21.276Z] === RUN   coreos.ignition.once

[2025-05-30T14:53:21.276Z] === RUN   coreos.ignition.symlink

[2025-05-30T14:53:21.276Z] === RUN   ostree.unlock

[2025-05-30T14:53:21.276Z] === RUN   multipath.partition

[2025-05-30T14:53:21.276Z] === RUN   ext.config.shared.networking.kargs-rd-net

[2025-05-30T14:53:21.276Z] === RUN   rhcos.services-disabled

[2025-05-30T14:53:21.276Z] === RUN   kdump.crash.nfs

[2025-05-30T14:53:21.276Z] === RUN   ext.config.rpm-ostree.startlimit

[2025-05-30T14:53:21.276Z] === RUN   ext.config.shared.ignition.resource.remote

[2025-05-30T14:53:21.276Z] === RUN   rpmostree.install-uninstall

[2025-05-30T14:53:21.276Z] === RUN   ext.config.shared.kdump.crash

[2025-05-30T14:53:21.276Z] === RUN   ext.config.shared.networking.no-default-initramfs-net-propagation.bootif

[2025-05-30T14:53:21.276Z] === RUN   ext.config.shared.systemd.condition-needs-update

[2025-05-30T14:53:21.276Z] === RUN   ext.config.shared.networking.nmstate.state

[2025-05-30T14:53:21.276Z] === RUN   ext.config.shared.var-mount.scsi-id

[2025-05-30T14:53:21.276Z] === RUN   rhcos.network.multiple-nics

[2025-05-30T14:53:21.276Z] === RUN   podman.base

[2025-05-30T14:53:21.276Z] === RUN   rhcos.selinux.boolean.persist

[2025-05-30T14:53:21.276Z] === RUN   ext.config.shared.networking.bridge-static-via-kargs

[2025-05-30T14:53:21.276Z] === RUN   coreos.ignition.mount.partitions

[2025-05-30T14:53:59.909Z] === RUN   ostree.hotfix/unlock

[2025-05-30T14:54:01.789Z] --- PASS: rpmostree.status (40.41s)

[2025-05-30T14:54:03.679Z] --- PASS: ext.config.rpm-ostree.startlimit (42.28s)

[2025-05-30T14:54:03.934Z] --- PASS: ext.config.shared.systemd.network-online (42.65s)

[2025-05-30T14:54:18.756Z] === RUN   ostree.hotfix/install

[2025-05-30T14:54:18.756Z] === RUN   ostree.hotfix/uninstall

[2025-05-30T14:54:19.010Z] === RUN   ostree.hotfix/persist

[2025-05-30T14:54:40.871Z] === RUN   basic/MachineID

[2025-05-30T14:54:41.125Z] === RUN   basic/RHCOSGrowpart

[2025-05-30T14:54:41.380Z] === RUN   basic/PortSSH

[2025-05-30T14:54:41.380Z] === RUN   basic/DbusPerms

[2025-05-30T14:54:41.947Z] === RUN   basic/ServicesActive

[2025-05-30T14:54:41.947Z] === RUN   basic/ReadOnly

[2025-05-30T14:54:42.201Z] === RUN   basic/Useradd

[2025-05-30T14:54:44.084Z] --- PASS: ext.config.shared.networking.hostname.fallback-hostname (42.11s)

[2025-05-30T14:54:45.450Z] --- PASS: basic (41.98s)

[2025-05-30T14:54:45.450Z]     --- PASS: basic/MachineID (0.14s)

[2025-05-30T14:54:45.450Z]     --- PASS: basic/RHCOSGrowpart (0.17s)

[2025-05-30T14:54:45.450Z]     --- PASS: basic/PortSSH (0.14s)

[2025-05-30T14:54:45.450Z]     --- PASS: basic/DbusPerms (0.47s)

[2025-05-30T14:54:45.450Z]     --- PASS: basic/ServicesActive (0.15s)

[2025-05-30T14:54:45.450Z]     --- PASS: basic/ReadOnly (0.14s)

[2025-05-30T14:54:45.450Z]     --- PASS: basic/Useradd (0.26s)

[2025-05-30T14:54:53.518Z] === RUN   ostree.hotfix/rollback

[2025-05-30T14:55:15.379Z] --- PASS: ext.config.shared.ignition.remote (68.46s)

[2025-05-30T14:55:33.410Z] --- PASS: ostree.hotfix (131.57s)

[2025-05-30T14:55:33.410Z]     --- PASS: ostree.hotfix/unlock (19.34s)

[2025-05-30T14:55:33.410Z]     --- PASS: ostree.hotfix/install (1.01s)

[2025-05-30T14:55:33.410Z]     --- PASS: ostree.hotfix/uninstall (0.58s)

[2025-05-30T14:55:33.410Z]     --- PASS: ostree.hotfix/persist (33.57s)

[2025-05-30T14:55:33.411Z]     --- PASS: ostree.hotfix/rollback (36.74s)

[2025-05-30T14:55:51.430Z] --- PASS: coreos.auth.verify (38.99s)

[2025-05-30T14:56:13.300Z] --- PASS: coreos.ignition.v2.users (40.33s)

[2025-05-30T14:56:16.560Z] --- PASS: coreos.ignition.failure (25.20s)

[2025-05-30T14:56:19.066Z] --- PASS: multipath.day2 (93.63s)

[2025-05-30T14:56:57.700Z] --- PASS: coreos.ignition.systemd.enable-service (39.70s)

[2025-05-30T14:57:07.623Z] --- PASS: ext.config.shared.networking.nmstate.policy (142.97s)

[2025-05-30T14:57:14.139Z] --- PASS: ext.config.shared.networking.mtu-on-bond-ignition (54.41s)

[2025-05-30T14:57:35.993Z] --- PASS: ext.config.shared.ignition.stable-boot (42.16s)

[2025-05-30T14:57:48.145Z] --- PASS: coreos.ignition.groups (39.44s)

[2025-05-30T14:57:50.649Z] === RUN   rpmostree.upgrade-rollback/upgrade

[2025-05-30T14:58:29.279Z] --- PASS: ext.config.shared.networking.prefer-ignition-networking (48.79s)

[2025-05-30T14:58:29.279Z] --- PASS: ext.config.shared.ssh.custom-host-key-permissions (42.20s)

[2025-05-30T14:58:51.152Z] === RUN   rpmostree.upgrade-rollback/rollback

[2025-05-30T14:58:52.512Z] --- PASS: kdump.crash.ssh (155.81s)

[2025-05-30T14:58:52.512Z]         cluster.go:151: Running as unit: run-r1a8a98404bfa4f99ba5db0a15aceba72.service

[2025-05-30T14:59:14.374Z] --- PASS: ext.config.shared.networking.force-persist-ip (49.33s)

[2025-05-30T14:59:40.925Z] --- PASS: rpmostree.upgrade-rollback (147.12s)

[2025-05-30T14:59:40.925Z]     --- PASS: rpmostree.upgrade-rollback/upgrade (60.35s)

[2025-05-30T14:59:40.925Z]     --- PASS: rpmostree.upgrade-rollback/rollback (46.89s)

[2025-05-30T15:00:12.931Z] --- PASS: ext.config.shared.var-mount.luks (101.53s)

[2025-05-30T15:00:39.405Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.rpm-ostree.container-deps

[2025-05-30T15:00:39.405Z] --- PASS: fips.enable (105.63s)

[2025-05-30T15:00:39.405Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.files.license

[2025-05-30T15:00:39.964Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.disks.growfs

[2025-05-30T15:00:40.887Z] === RUN   non-exclusive-test-bucket-0/ext.config.files.localtime

[2025-05-30T15:00:41.811Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.files.check-symlink

[2025-05-30T15:00:42.369Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.systemd.no-systemd-firstboot

[2025-05-30T15:00:44.250Z] === RUN   non-exclusive-test-bucket-0/ext.config.files.check-qemu-ga

[2025-05-30T15:00:45.172Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.files.validate-symlinks

[2025-05-30T15:00:59.991Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.files.root-immutable-bit

[2025-05-30T15:00:59.991Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.files.initrd.executables

[2025-05-30T15:01:03.255Z] === RUN   non-exclusive-test-bucket-0/ext.config.files.conntrack-tools

[2025-05-30T15:01:03.815Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.files.sudoers

[2025-05-30T15:01:04.737Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.disks.partition-scheme

[2025-05-30T15:01:06.618Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.networking.nm-start

[2025-05-30T15:01:08.507Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.files.logrotate-service

[2025-05-30T15:01:09.867Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.files.rpmdb-sqlite

[2025-05-30T15:01:10.790Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.content-origins

[2025-05-30T15:01:12.671Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.files.aleph-version

[2025-05-30T15:01:13.231Z] === RUN   non-exclusive-test-bucket-0/ext.config.files.env-godebug

[2025-05-30T15:01:16.485Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.disks.no-google-device-links

[2025-05-30T15:01:17.045Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.composefs.enabled

[2025-05-30T15:01:18.928Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.podman.rootless-pasta-networking

[2025-05-30T15:01:25.454Z] --- PASS: ext.config.shared.reboot (74.49s)

[2025-05-30T15:01:27.974Z] === RUN   non-exclusive-test-bucket-0/ext.config.files.composefs-maybe

[2025-05-30T15:01:28.228Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.podman.dns

[2025-05-30T15:01:31.495Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.disks.root-prjquota

[2025-05-30T15:01:32.056Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.networking.network-online-service

[2025-05-30T15:01:32.614Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.ignition.journald-log

[2025-05-30T15:01:34.497Z] --- PASS: ext.config.shared.ignition.delete-config (141.08s)

[2025-05-30T15:01:34.497Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.networking.no-default-initramfs-net-propagation.default

[2025-05-30T15:01:35.419Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.files.setuid

[2025-05-30T15:01:37.301Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.networking.nm-dhcp-client

[2025-05-30T15:01:37.861Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.files.yum-repo-dir

[2025-05-30T15:01:38.785Z] === RUN   non-exclusive-test-bucket-0/ext.config.afterburn.sshkey-service

[2025-05-30T15:01:39.345Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.disks.root-boot-ro

[2025-05-30T15:01:40.270Z] === RUN   non-exclusive-test-bucket-0/ext.config.console.console-login-helper-service

[2025-05-30T15:01:41.201Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.disks.systemd-repart-service

[2025-05-30T15:01:41.761Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.logging.printk

[2025-05-30T15:01:42.684Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.networking.default-network-behavior-change

[2025-05-30T15:01:44.567Z] === RUN   non-exclusive-test-bucket-0/ext.config.ssh.config

[2025-05-30T15:01:46.447Z] === RUN   non-exclusive-test-bucket-0/ext.config.version.rhel-major-version

[2025-05-30T15:01:56.372Z] --- PASS: coreos.unique.boot.ignition.failure (21.50s)

[2025-05-30T15:02:00.528Z] --- PASS: ext.config.shared.podman.rootless-systemd (82.57s)

[2025-05-30T15:02:02.410Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.ntp.chrony.coreos-platform-chrony-config

[2025-05-30T15:02:02.969Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.clhm.network-device-info

[2025-05-30T15:02:03.893Z] === RUN   non-exclusive-test-bucket-0/ext.config.version.drop-azure-ptp

[2025-05-30T15:02:04.451Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.selinux.usrlocal-context

[2025-05-30T15:02:05.374Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.systemd.default-unit-timeouts

[2025-05-30T15:02:07.257Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.files.console-config

[2025-05-30T15:02:08.624Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.networking.tls

[2025-05-30T15:02:15.158Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.disks.tmpfs

[2025-05-30T15:02:15.412Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.kubernetes.systemd-env-read

[2025-05-30T15:02:17.292Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.files.etc-permissions

[2025-05-30T15:02:19.173Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.files.file-directory-permissions

[2025-05-30T15:02:20.531Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.selinux.unmodified-policy

[2025-05-30T15:02:25.763Z] === RUN   non-exclusive-test-bucket-0/ext.config.files.kata-osbuilder

[2025-05-30T15:02:26.322Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.files.setgid

[2025-05-30T15:02:28.203Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.selinux.podman-tmpfs-context

[2025-05-30T15:02:36.276Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.selinux.file-context-policy-match

[2025-05-30T15:02:38.157Z] --- PASS: ext.config.shared.var-mount.simple (73.45s)

[2025-05-30T15:02:50.312Z] --- PASS: ext.config.shared.ignition.kargs (54.48s)

[2025-05-30T15:03:05.133Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.networking.nm-ifcfg-rh-plugin

[2025-05-30T15:03:06.493Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.files.root-bash

[2025-05-30T15:03:07.415Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.firewall.iptables

[2025-05-30T15:03:08.337Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.networking.nic-naming

[2025-05-30T15:03:10.217Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.security.coreos-update-ca-trust

[2025-05-30T15:03:12.096Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.boot.bootupd

[2025-05-30T15:03:13.979Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.files.kernel-headers

[2025-05-30T15:03:14.537Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.files.initrd.compression

[2025-05-30T15:03:15.463Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.kubernetes.kube-watch

[2025-05-30T15:03:17.972Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.files.system-generators

[2025-05-30T15:03:19.330Z] --- PASS: ext.config.shared.ignition.systemd-enable-units (41.43s)

[2025-05-30T15:03:19.887Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.kdump.service

[2025-05-30T15:03:20.811Z] --- PASS: rhcos.selinux.manage (80.02s)

[2025-05-30T15:03:20.811Z] === RUN   non-exclusive-test-bucket-0/ext.config.shared.swap.zram-default

[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0 (224.07s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.rpm-ostree.container-deps (1.80s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.license (0.74s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.disks.growfs (0.82s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.files.localtime (0.75s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.check-symlink (0.80s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.systemd.no-systemd-firstboot (1.85s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.files.check-qemu-ga (0.77s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.validate-symlinks (12.47s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.root-immutable-bit (0.76s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.initrd.executables (4.80s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.files.conntrack-tools (0.79s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.sudoers (0.75s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.disks.partition-scheme (1.78s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.nm-start (1.77s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.logrotate-service (1.77s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.rpmdb-sqlite (0.79s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.content-origins (1.81s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.aleph-version (0.76s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.files.env-godebug (2.81s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.disks.no-google-device-links (0.78s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.composefs.enabled (1.81s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.podman.rootless-pasta-networking (8.80s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.files.composefs-maybe (0.79s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.podman.dns (2.82s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.disks.root-prjquota (0.79s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.network-online-service (0.81s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.ignition.journald-log (1.79s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.no-default-initramfs-net-propagation.default (0.80s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.setuid (1.81s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.nm-dhcp-client (0.78s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.yum-repo-dir (0.77s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.afterburn.sshkey-service (0.79s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.disks.root-boot-ro (0.76s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.console.console-login-helper-service (0.80s)

[2025-05-30T15:03:24.963Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.disks.systemd-repart-service (0.79s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.logging.printk (0.76s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.default-network-behavior-change (1.85s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.ssh.config (1.82s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.version.rhel-major-version (15.93s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.ntp.chrony.coreos-platform-chrony-config (0.78s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.clhm.network-device-info (0.76s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.version.drop-azure-ptp (0.75s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.selinux.usrlocal-context (0.74s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.systemd.default-unit-timeouts (1.78s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.console-config (1.76s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.tls (5.80s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.disks.tmpfs (0.76s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.kubernetes.systemd-env-read (1.78s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.etc-permissions (1.76s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.file-directory-permissions (1.79s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.selinux.unmodified-policy (4.79s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.files.kata-osbuilder (0.77s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.setgid (1.78s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.selinux.podman-tmpfs-context (7.83s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.selinux.file-context-policy-match (28.82s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.nm-ifcfg-rh-plugin (1.86s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.root-bash (0.79s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.firewall.iptables (0.84s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.nic-naming (1.85s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.security.coreos-update-ca-trust (1.82s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.boot.bootupd (1.81s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.kernel-headers (0.83s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.initrd.compression (0.81s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.kubernetes.kube-watch (2.86s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.system-generators (1.81s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.kdump.service (0.83s)

[2025-05-30T15:03:24.964Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.shared.swap.zram-default (0.90s)

[2025-05-30T15:03:33.032Z] --- PASS: ext.config.shared.gshadow (42.37s)

[2025-05-30T15:04:11.662Z] --- PASS: ext.config.shared.networking.no-persist-ip (48.25s)

[2025-05-30T15:04:13.543Z] --- PASS: rhcos.sssd (40.81s)

[2025-05-30T15:04:52.177Z] --- PASS: ext.config.shared.ntp.chrony.dhcp-propagation (86.90s)

[2025-05-30T15:04:54.683Z] --- PASS: multipath.day1 (95.28s)

[2025-05-30T15:05:41.270Z] --- PASS: coreos.ignition.sethostname (39.65s)

[2025-05-30T15:05:42.628Z] --- PASS: ext.config.shared.var-mount.scsi-id (51.02s)

[2025-05-30T15:05:49.154Z] --- PASS: ext.config.shared.networking.ifname-karg.everyboot-systemd-link-file (95.74s)

[2025-05-30T15:05:59.077Z] --- PASS: fips.enable.partitions (108.57s)

[2025-05-30T15:06:31.074Z] --- PASS: ext.config.shared.networking.bridge-static-via-kargs (48.44s)

[2025-05-30T15:06:33.583Z] === RUN   podman.base/info

[2025-05-30T15:06:34.505Z] === RUN   podman.base/resources

[2025-05-30T15:06:49.331Z] --- PASS: podman.base (50.21s)

[2025-05-30T15:06:49.331Z]     --- PASS: podman.base/info (1.07s)

[2025-05-30T15:06:49.331Z]     --- PASS: podman.base/resources (9.93s)

[2025-05-30T15:06:49.331Z] --- PASS: coreos.ignition.mount.partitions (73.95s)

[2025-05-30T15:06:50.254Z] --- FAIL: kdump.crash.nfs (2.02s)

[2025-05-30T15:06:50.254Z]         kdump.go:291: listen tcp :2049: bind: address already in use

[2025-05-30T15:07:05.075Z] --- PASS: rhcos.selinux.boolean.persist (75.01s)

[2025-05-30T15:07:26.941Z] --- PASS: ext.config.systemd.journal-compat (52.94s)

[2025-05-30T15:07:41.760Z] === RUN   ostree.unlock/unlock

[2025-05-30T15:07:42.016Z] === RUN   ostree.unlock/install

[2025-05-30T15:07:43.376Z] === RUN   ostree.unlock/uninstall

[2025-05-30T15:07:43.934Z] === RUN   ostree.unlock/discard

[2025-05-30T15:08:01.956Z] === RUN   rhcos.services-disabled/ServicesDisabled

[2025-05-30T15:08:03.840Z] --- PASS: multipath.partition (73.31s)

[2025-05-30T15:08:04.398Z] --- PASS: rhcos.services-disabled (40.27s)

[2025-05-30T15:08:04.398Z]     --- PASS: rhcos.services-disabled/ServicesDisabled (0.40s)

[2025-05-30T15:08:22.421Z] --- PASS: ostree.unlock (76.70s)

[2025-05-30T15:08:22.421Z]     --- PASS: ostree.unlock/unlock (1.03s)

[2025-05-30T15:08:22.422Z]     --- PASS: ostree.unlock/install (1.34s)

[2025-05-30T15:08:22.422Z]     --- PASS: ostree.unlock/uninstall (0.67s)

[2025-05-30T15:08:22.422Z]     --- PASS: ostree.unlock/discard (33.76s)

[2025-05-30T15:08:24.302Z] --- PASS: rhcos.network.multiple-nics (96.31s)

[2025-05-30T15:08:46.163Z] --- PASS: coreos.ignition.symlink (39.76s)

[2025-05-30T15:08:52.687Z] --- PASS: ext.config.shared.networking.kargs-rd-net (48.23s)

[2025-05-30T15:09:14.561Z] --- PASS: ext.config.shared.networking.no-default-initramfs-net-propagation.bootif (50.21s)

[2025-05-30T15:09:32.584Z] --- PASS: coreos.ignition.once (71.58s)

[2025-05-30T15:09:59.059Z] --- PASS: ext.config.shared.boot.grub2-install (43.53s)

[2025-05-30T15:09:59.059Z] --- PASS: coreos.misc.disk.varlibcontainers (75.83s)

[2025-05-30T15:09:59.059Z]         cluster.go:151: Created symlink /etc/systemd/system/var-lib-containers.mount.wants/systemd-mkfs@dev-disk-by-id-virtio-disk1.service → /etc/systemd/system/systemd-mkfs@dev-disk-by-id-virtio-disk1.service.

[2025-05-30T15:09:59.059Z]         cluster.go:151: Created symlink /etc/systemd/system/local-fs.target.wants/var-lib-containers.mount → /etc/systemd/system/var-lib-containers.mount.

[2025-05-30T15:10:55.197Z] --- PASS: ext.config.shared.networking.nameserver (48.43s)

[2025-05-30T15:10:55.197Z] --- PASS: ext.config.shared.networking.nmstate.state (75.73s)

[2025-05-30T15:11:03.279Z] --- PASS: ext.config.shared.systemd.condition-needs-update (130.35s)

[2025-05-30T15:11:13.206Z] --- PASS: coreos.ignition.mount.disks (72.43s)

[2025-05-30T15:11:19.736Z] === RUN   rpmostree.install-uninstall/install

[2025-05-30T15:11:29.662Z] --- PASS: ext.config.shared.clhm.ignition-warnings (40.83s)

[2025-05-30T15:11:56.145Z] --- PASS: ext.config.shared.containers.quadlet (44.55s)

[2025-05-30T15:12:08.297Z] --- PASS: ext.config.shared.ignition.resource.remote (38.49s)

[2025-05-30T15:12:09.657Z] --- FAIL: ostree.sync (2.02s)

[2025-05-30T15:12:09.657Z]         sync.go:117: listen tcp :2049: bind: address already in use

[2025-05-30T15:12:36.136Z] === RUN   rpmostree.install-uninstall/uninstall

[2025-05-30T15:13:14.783Z] --- PASS: ext.config.shared.disks.lvmdevices (75.45s)

[2025-05-30T15:13:32.806Z] --- PASS: ext.config.shared.networking.ifname-karg.udev-rule-firstboot-propagation (80.97s)

[2025-05-30T15:13:32.806Z] --- PASS: ext.config.shared.kdump.crash (149.09s)

[2025-05-30T15:13:42.730Z] --- PASS: rpmostree.install-uninstall (176.30s)

[2025-05-30T15:13:42.730Z]     --- PASS: rpmostree.install-uninstall/install (74.08s)

[2025-05-30T15:13:42.730Z]     --- PASS: rpmostree.install-uninstall/uninstall (65.40s)

[2025-05-30T15:14:09.212Z] === RUN   ostree.remote/add

[2025-05-30T15:14:09.212Z] === RUN   ostree.remote/list

[2025-05-30T15:14:09.212Z] === RUN   ostree.remote/show-url

[2025-05-30T15:14:09.212Z] === RUN   ostree.remote/refs

[2025-05-30T15:14:09.212Z] === RUN   ostree.remote/summary

[2025-05-30T15:14:09.212Z] === RUN   ostree.remote/delete

[2025-05-30T15:14:12.471Z] --- PASS: ostree.remote (41.97s)

[2025-05-30T15:14:12.471Z]     --- PASS: ostree.remote/add (0.33s)

[2025-05-30T15:14:12.471Z]     --- PASS: ostree.remote/list (0.13s)

[2025-05-30T15:14:12.471Z]     --- PASS: ostree.remote/show-url (0.25s)

[2025-05-30T15:14:12.471Z]     --- PASS: ostree.remote/refs (0.23s)

[2025-05-30T15:14:12.471Z]     --- PASS: ostree.remote/summary (0.52s)

[2025-05-30T15:14:12.471Z]     --- PASS: ostree.remote/delete (0.51s)

[2025-05-30T15:14:22.406Z] --- PASS: ext.config.shared.networking.mtu-on-bond-kargs (49.03s)

[2025-05-30T15:14:25.663Z] --- PASS: ext.config.shared.selinux.enforcing (73.70s)

[2025-05-30T15:14:25.663Z] mke2fs 1.47.2 (1-Jan-2025)

[2025-05-30T15:14:30.896Z] --- PASS: ext.config.shared.networking.team-dhcp-via-ignition (48.24s)

[2025-05-30T15:14:45.724Z] --- PASS: coreos.unique.boot.failure (18.75s)

[2025-05-30T15:15:03.744Z] --- PASS: ext.config.disk.nvme-compat-symlink (41.07s)

[2025-05-30T15:15:30.215Z] --- PASS: ext.config.shared.boot.bootupd-validate (59.30s)

[2025-05-30T15:16:16.824Z] --- PASS: ext.config.shared.networking.rd-net-timeout-carrier (86.07s)

[2025-05-30T15:16:16.824Z] --- PASS: coreos.selinux.boolean (71.74s)

[2025-05-30T15:18:53.236Z] KVM: Failed to create TCE64 table for liobn 0x80000001

[2025-05-30T15:18:59.774Z] --- PASS: ext.config.rpm-ostree.replace-rt-kernel (287.08s)

[2025-05-30T15:18:59.774Z] FAIL, output in /home/jenkins/agent/workspace/build-arch/tmp/kola-Wndtl/kola

[2025-05-30T15:18:59.774Z] 

[2025-05-30T15:18:59.774Z] 

[2025-05-30T15:18:59.774Z] ======== Re-running failed tests (flake detection) ========

[2025-05-30T15:18:59.774Z] 

[2025-05-30T15:18:59.774Z] ⏭️  Skipping kola test pattern "ext.config.version.rhaos-pkgs-match-openshift":

[2025-05-30T15:18:59.774Z]   👉 https://issues.redhat.com/browse/OCPBUGS-42688

[2025-05-30T15:18:59.774Z] ⏭️  Skipping kola test pattern "ext.config.shared.multipath.resilient":

[2025-05-30T15:18:59.774Z]   👉 https://github.com/coreos/fedora-coreos-tracker/issues/1937

[2025-05-30T15:19:00.028Z] === RUN   ostree.sync

[2025-05-30T15:19:00.028Z] === RUN   kdump.crash.nfs

[2025-05-30T15:19:00.585Z] qemu-system-ppc64: -netdev user,id=eth0,hostfwd=tcp:127.0.0.1:36287-:22,hostfwd=tcp:127.0.0.1:2049-:2049,hostname=qemu0: Could not set up host forwarding rule 'tcp:127.0.0.1:2049-:2049'

[2025-05-30T15:19:32.587Z] --- FAIL: ostree.sync (31.54s)

[2025-05-30T15:19:32.587Z]         sync.go:117: failed to establish qmp connection: dial unix /var/tmp/mantle-qemu2099425295/qmp-1748618340436644144.sock: connect: connection refused

[2025-05-30T15:21:38.968Z] --- PASS: kdump.crash.nfs (157.10s)

[2025-05-30T15:21:38.968Z]         cluster.go:151: Running as unit: run-rbc26bb21f6424c438e84e7781f60c9e3.service

[2025-05-30T15:21:38.968Z] FAIL, output in /home/jenkins/agent/workspace/build-arch/tmp/kola-Wndtl/kola/rerun

[2025-05-30T15:21:38.968Z] Error: harness: test suite failed

[2025-05-30T15:21:38.968Z] 2025-05-30T15:21:37Z cli: harness: test suite failed

[2025-05-30T15:21:38.968Z] failed to execute cmd-kola: exit status 1

[2025-05-30T15:19:00.585Z] qemu-system-ppc64: -netdev user,id=eth0,hostfwd=tcp:127.0.0.1:36287-:22,hostfwd=tcp:127.0.0.1:2049-:2049,hostname=qemu0: Could not set up host forwarding rule 'tcp:127.0.0.1:2049-:2049'

[2025-05-30T15:19:32.587Z] --- FAIL: ostree.sync (31.54s)

[2025-05-30T15:19:32.587Z]         sync.go:117: failed to establish qmp connection: dial unix /var/tmp/mantle-qemu2099425295/qmp-1748618340436644144.sock: connect: connection refused

Basically for whatever reason the host clashed: Could not set up host forwarding rule 'tcp:127.0.0.1:2049-:2049'. I don't know if another kdump.crash.nfs test was running for another pipeline (or if they would even clash).

You can also reproduce something similar with cosa kola run kdump.crash.nfs --multiply 2 --parallel 2.

[dustymabe]

From coreos/rhel-coreos-config#89 (comment)

Both ostree.sync and kdump.crash.nfs need to bind port 2049 to the host, if they are running at the same time, one will be failed with listen tcp :2049: bind: address already in use.

I wonder if the more correct approach here would be to just mark these various tests as conflicts.

I don't think retrying as is done in #4358 will be reliable because the retry at 12*5seconds = 1minute probably isn't long enough.

Note that I don't know if we've conflicts before so it might not work perfectly on first try or it might!

[HuijingHei]

I wonder if the more correct approach here would be to just mark these various tests as conflicts.

This sounds good to me and makes sense, thank you for the pointer.

I don't think retrying as is done in #4358 will be reliable because the retry at 12*5seconds = 1minute probably isn't long enough.

Agree it is not the good workaround.

Note that I don't know if we've conflicts before so it might not work perfectly on first try or it might!

I will give it a try.

[HuijingHei]

If I add Conflicts: []string{"kdump.crash.nfs"} in ostree.sync, then it will skip test ostree.sync, maybe this is by design. Actually we just do not want them to be running at the same time, or easier way is to skip ostree.sync running on qemu.

[coreos-assembler]$ kola run --qemu-image scos-9.0.20251029-0-qemu.x86_64.qcow2 ostree.sync kdump.crash.nfs --parallel 2
=== RUN   kdump.crash.nfs
--- PASS: kdump.crash.nfs (178.63s)
        cluster.go:151: Running as unit: run-r32f8b9d404f44ad18c3c9ac4425a3c77.service
PASS, output in _kola_temp/qemu-2025-11-06-0149-7071

[dustymabe]

hmm. not sure. When I do the same thing I get:

[dustymabe@media fcos]$ cosa kola run --parallel=2 ostree.sync kdump.crash.nfs
COREOS_ASSEMBLER_CONTAINER_RUNTIME_ARGS=-v /var/b/shared/code/github.com/coreos/coreos-assembler//bin/:/usr/local/bin/:ro -v /var/b/shared/code/github.com/coreos/coreos-assembler/:/src/:rw
COSA_BUILD_WITH_BUILDAH=1
COREOS_ASSEMBLER_GIT=/var/b/shared/code/github.com/coreos/coreos-assembler/
COREOS_ASSEMBLER_CONFIG_GIT=/var/b/shared/code/github.com/coreos/fedora-coreos-config/
COREOS_ASSEMBLER_CONTAINER_NAME=cosafcos
+ podman run --rm -ti --security-opt label=disable --privileged --userns keep-id:uid=1000,gid=1000 -v /var/b/shared/assembler/fcos:/srv/ --device /dev/kvm --device /dev/fuse --tmpfs /tmp -v /var/tmp:/var/tmp --name cosafcos -e COSA_BUIs
kola -p qemu run --parallel=2 ostree.sync kdump.crash.nfs --output-dir tmp/kola
panic: exclusive test cannot have non-empty conflicts entry

which I guess makes sense. I guess we only introduce conflicts to deal with non-exclusive tests that conflict with each other, so they couldn't all be run on the same VM.

I think it could make sense to extend this concept even to exclusive tests like ostree.sync and kdump.crash.nfs.

[HuijingHei]

[dustymabe@media fcos]$ cosa kola run --parallel=2 ostree.sync kdump.crash.nfs
COREOS_ASSEMBLER_CONTAINER_RUNTIME_ARGS=-v /var/b/shared/code/github.com/coreos/coreos-assembler//bin/:/usr/local/bin/:ro -v /var/b/shared/code/github.com/coreos/coreos-assembler/:/src/:rw
COSA_BUILD_WITH_BUILDAH=1
COREOS_ASSEMBLER_GIT=/var/b/shared/code/github.com/coreos/coreos-assembler/
COREOS_ASSEMBLER_CONFIG_GIT=/var/b/shared/code/github.com/coreos/fedora-coreos-config/
COREOS_ASSEMBLER_CONTAINER_NAME=cosafcos
+ podman run --rm -ti --security-opt label=disable --privileged --userns keep-id:uid=1000,gid=1000 -v /var/b/shared/assembler/fcos:/srv/ --device /dev/kvm --device /dev/fuse --tmpfs /tmp -v /var/tmp:/var/tmp --name cosafcos -e COSA_BUIs
kola -p qemu run --parallel=2 ostree.sync kdump.crash.nfs --output-dir tmp/kola
panic: exclusive test cannot have non-empty conflicts entry

You are correct, seems I do not run the updated code.

$ ./bin/kola run --qemu-image scos-9.0.20251029-0-qemu.x86_64.qcow2 ostree.sync kdump.crash.nfs --parallel 2
panic: exclusive test cannot have non-empty conflicts entry

which I guess makes sense. I guess we only introduce conflicts to deal with non-exclusive tests that conflict with each other, so they couldn't all be run on the same VM.

I think it could make sense to extend this concept even to exclusive tests like ostree.sync and kdump.crash.nfs.

In this case, should be make sure these 2 in the same bucket that would avoid to run parallelly?

[dustymabe]

In this case, should be make sure these 2 in the same bucket that would avoid to run parallelly?

It would take some internal work in kola, which might be useful.

It's kind of a hack, but another option is to tag them with reprovision, which forces them to run serially in our current setup. See coreos/fedora-coreos-config#2826

[HuijingHei]

Copy from coreos/fedora-coreos-config#2826 (comment):

It's always been a bit awkward how we mentioned the reprovision tag in the kola() step. We should probably add a new serial tag instead to all those tests and use that in kola().

I'd be happy with that. Though I think it would be nicer if there was a way to have it happen internally in kola instead of having to do --parallel=4 --tag=!serial and then --tag=serial.
tests with the serial tag would run serially even when --parallel=4 was specified

This looks sane, maybe can add this in cosa in longer term.

And for short term, add tag reprovision in ostree.sync. WDYT?

[dustymabe]

This looks sane, maybe can add this in cosa in longer term.

Yeah. The other option here is to have something like;

Locks: ["HostPort=2049"]

and then have both tests define that and then the test framework would take/wait for a lock before continuing.

Note: Locks doesn't exist today. We'd have to implement it.