BSS Not Returning Boot Params

[PowellTravis]

I frequently have been seeing BSS pulling the boot parameters and logging them in the console. But the parameters are never returned to the node attempting to boot. I do not know if this is related to other issues with frequent TLS timeouts on the system as well. This issue is intermittent, and usually resolves with several service reboots and acme-register restarts. But it keeps happening every few days.

Nov 03 16:50:52 tempest-ochami-mgmt.rci.montana.edu bss[149451]: 2025/11/03 21:50:52 DEBUG: BootscriptGet(): Received request /boot/v1/bootscript?mac=cc:48:3a:21:2a:6a
Nov 03 16:50:52 tempest-ochami-mgmt.rci.montana.edu bss[149451]: 2025/11/03 21:50:52 DEBUG: bd: {nomodeset ro root=live:http://10.226.104.4:9000/boot-images/compute/cpu/uefi-66df2971/rocky8.10-common-fstab-uefi ip=dhcp overlayroot=tmpfs overlayroot_cfgdisk=disabled apparmor=0 selinux=0 crashkernel=auto systemd.unified_cgroup_hierarchy=1 systemd.legacy_systemd_cgroup_controller=0 cgroup_no_v1=all console=ttyS0,115200 ip6=off cloud-init=enabled ds=nocloud-net;s=http://10.226.104.4:8081/cloud-init {http://10.226.104.4:9000/boot-images/efi-images/compute/cpu/uefi-66df2971/vmlinuz-4.18.0-553.72.1.el8_10.x86_64 } {http://10.226.104.4:9000/boot-images/efi-images/compute/cpu/uefi-66df2971/initramfs-4.18.0-553.72.1.el8_10.x86_64.img } {map[] map[] {      }} }
Nov 03 16:50:52 tempest-ochami-mgmt.rci.montana.edu bss[149451]: 2025/11/03 21:50:52 DEBUG: comp: {{x1000c1s7b19n0 Node On  0xc000818fd2  Compute  119     false false}  [CC:48:3A:21:2A:6A cc:48:3a:21:2a:6a] true}
Nov 03 16:50:52 tempest-ochami-mgmt.rci.montana.edu bss[149451]: 2025/11/03 21:50:52 DEBUG: buildBootScript({nomodeset ro root=live:http://10.226.104.4:9000/boot-images/compute/cpu/uefi-66df2971/rocky8.10-common-fstab-uefi ip=dhcp overlayroot=tmpfs overlayroot_cfgdisk=disabled apparmor=0 selinux=0 crashkernel=auto systemd.unified_cgroup_hierarchy=1 systemd.legacy_systemd_cgroup_controller=0 cgroup_no_v1=all console=ttyS0,115200 ip6=off cloud-init=enabled ds=nocloud-net;s=http://10.226.104.4:8081/cloud-init {http://10.226.104.4:9000/boot-images/efi-images/compute/cpu/uefi-66df2971/vmlinuz-4.18.0-553.72.1.el8_10.x86_64 } {http://10.226.104.4:9000/boot-images/efi-images/compute/cpu/uefi-66df2971/initramfs-4.18.0-553.72.1.el8_10.x86_64.img } {map[] map[] {      }} }, {x1000c1s7b19n0 119 }, chain https://${SYSTEM_URL}/apis/bss/boot/v1/bootscript?mac=cc:48:3a:21:2a:6a&retry=1, Compute, , MAC cc:48:3a:21:2a:6a (x1000c1s7b19n0))
Nov 03 16:50:52 tempest-ochami-mgmt.rci.montana.edu bss[149451]: 2025/11/03 21:50:52 DEBUG: checkParam("nomodeset ro root=live:http://10.226.104.4:9000/boot-images/compute/cpu/uefi-66df2971/rocky8.10-common-fstab-uefi ip=dhcp overlayroot=tmpfs overlayroot_cfgdisk=disabled apparmor=0 selinux=0 crashkernel=auto systemd.unified_cgroup_hierarchy=1 systemd.legacy_systemd_cgroup_controller=0 cgroup_no_v1=all console=ttyS0,115200 ip6=off cloud-init=enabled ds=nocloud-net;s=http://10.226.104.4:8081/cloud-init", "xname=", "x1000c1s7b19n0")
Nov 03 16:50:52 tempest-ochami-mgmt.rci.montana.edu bss[149451]: 2025/11/03 21:50:52 DEBUG: checkParam returning "nomodeset ro root=live:http://10.226.104.4:9000/boot-images/compute/cpu/uefi-66df2971/rocky8.10-common-fstab-uefi ip=dhcp overlayroot=tmpfs overlayroot_cfgdisk=disabled apparmor=0 selinux=0 crashkernel=auto systemd.unified_cgroup_hierarchy=1 systemd.legacy_systemd_cgroup_controller=0 cgroup_no_v1=all console=ttyS0,115200 ip6=off cloud-init=enabled ds=nocloud-net;s=http://10.226.104.4:8081/cloud-init xname=x1000c1s7b19n0"
Nov 03 16:50:52 tempest-ochami-mgmt.rci.montana.edu bss[149451]: 2025/11/03 21:50:52 DEBUG: checkParam("nomodeset ro root=live:http://10.226.104.4:9000/boot-images/compute/cpu/uefi-66df2971/rocky8.10-common-fstab-uefi ip=dhcp overlayroot=tmpfs overlayroot_cfgdisk=disabled apparmor=0 selinux=0 crashkernel=auto systemd.unified_cgroup_hierarchy=1 systemd.legacy_systemd_cgroup_controller=0 cgroup_no_v1=all console=ttyS0,115200 ip6=off cloud-init=enabled ds=nocloud-net;s=http://10.226.104.4:8081/cloud-init xname=x1000c1s7b19n0", "nid=", "119")
Nov 03 16:50:52 tempest-ochami-mgmt.rci.montana.edu bss[149451]: 2025/11/03 21:50:52 DEBUG: checkParam returning "nomodeset ro root=live:http://10.226.104.4:9000/boot-images/compute/cpu/uefi-66df2971/rocky8.10-common-fstab-uefi ip=dhcp overlayroot=tmpfs overlayroot_cfgdisk=disabled apparmor=0 selinux=0 crashkernel=auto systemd.unified_cgroup_hierarchy=1 systemd.legacy_systemd_cgroup_controller=0 cgroup_no_v1=all console=ttyS0,115200 ip6=off cloud-init=enabled ds=nocloud-net;s=http://10.226.104.4:8081/cloud-init xname=x1000c1s7b19n0 nid=119"
Nov 03 16:50:52 tempest-ochami-mgmt.rci.montana.edu bss[149451]: 2025/11/03 21:50:52 DEBUG: checkParam("nomodeset ro root=live:http://10.226.104.4:9000/boot-images/compute/cpu/uefi-66df2971/rocky8.10-common-fstab-uefi ip=dhcp overlayroot=tmpfs overlayroot_cfgdisk=disabled apparmor=0 selinux=0 crashkernel=auto systemd.unified_cgroup_hierarchy=1 systemd.legacy_systemd_cgroup_controller=0 cgroup_no_v1=all console=ttyS0,115200 ip6=off cloud-init=enabled ds=nocloud-net;s=http://10.226.104.4:8081/cloud-init xname=x1000c1s7b19n0 nid=119", "ds=", "nocloud-net;s=localhost/")
Nov 03 16:50:52 tempest-ochami-mgmt.rci.montana.edu bss[149451]: 2025/11/03 21:50:52 DEBUG: checkParam returning "nomodeset ro root=live:http://10.226.104.4:9000/boot-images/compute/cpu/uefi-66df2971/rocky8.10-common-fstab-uefi ip=dhcp overlayroot=tmpfs overlayroot_cfgdisk=disabled apparmor=0 selinux=0 crashkernel=auto systemd.unified_cgroup_hierarchy=1 systemd.legacy_systemd_cgroup_controller=0 cgroup_no_v1=all console=ttyS0,115200 ip6=off cloud-init=enabled ds=nocloud-net;s=http://10.226.104.4:8081/cloud-init xname=x1000c1s7b19n0 nid=119"
Nov 03 16:50:52 tempest-ochami-mgmt.rci.montana.edu bss[149451]: 2025/11/03 21:50:52 BSS request succeeded for MAC cc:48:3a:21:2a:6a (x1000c1s7b19n0)
Nov 03 16:50:52 tempest-ochami-mgmt.rci.montana.edu postgres[147827]: 2025-11-03 21:50:52.255 UTC [28] ERROR:  relation "endpoint_access" does not exist at character 13
Nov 03 16:50:52 tempest-ochami-mgmt.rci.montana.edu postgres[147827]: 2025-11-03 21:50:52.255 UTC [28] STATEMENT:  INSERT INTO endpoint_access (name, endpoint, last_epoch) VALUES ($1, $2, $3);
Nov 03 16:50:52 tempest-ochami-mgmt.rci.montana.edu bss[149451]: 2025/11/03 21:50:52 Failed to store last access timestamp for endpoint="bootscript" name="x1000c1s7b19n0" to postgres DB: postgres.LogEndpointAccess: Error executing query to add endpoint access {x1000c1s7b19n0 bootscript 1762206652}: pq: relation "endpoint_access" does not exist

Node config.

- name: tempest-epyc019
  nid: 119
  xname: x1000c1s7b19n0
  bmc_mac: cc:48:3a:27:03:2c
  bmc_ip: 10.226.100.119
  fqdn: tempest-epyc019.rci.montana.edu
  group: rome-cpu
  interfaces:
  - mac_addr: CC:48:3A:21:2A:6A
    ip_addrs:
    - name: management
      ip_addr: 10.226.104.119

[synackd]

Are you able to monitor traffic on the server-side? It looks like the request makes it to BSS but not back to the booting node. If a response originates from the interface, it's a network issue but if not, it might be an issue with BSS.

[PowellTravis]

@synackd It seems that the request never makes it out of the containers. We just started having the problem again and it fully prevents nodes from receiving params. We are looking to make the full switch in a few weeks during our maintenance window, but this seems to be a blocking factor for moving all nodes.

[synackd]

Without knowing your setup details, this smells like it could be a firewall/nftables issue, but it could also easily be something else. It would help if you are able to share more about your setup environment so that the issue can be reproduced. Are you using the release RPM (if so what version)? Do you have any extra firewall rules, SELinux policies, etc? Any setup instructions to help in reproduction would help.

Also, since this issue has been open, v0.6.0 of the ochami CLI has been released which has an updated discovery format. While the old format will still work, it is deprecated and will be removed in a future release. See the release notes for info on migrating to the new format (there's a script that will convert for you).

[PowellTravis]

We have the firewall completely disabled on this system. We have SELinux fully disabled on this system. We are using the release RPM v0.1.1. We are running on the latest Rocky 9.10 with the latest kernel release 5.14.0-570.17.1. I can send you a tarball of our configurations if that would help.

[synackd]

Hmm, I am unable to reproduce this issue on my Rocky 9 system running release v0.1.1 with SELinux and firewall disabled. I don't think a bad nodes.yaml or boot parameter configuration could cause it, but I suppose it wouldn't hurt to share configs.

[synackd]

This issue is intermittent, and usually resolves with several service reboots and acme-register restarts. But it keeps happening every few days.

And just to make sure, you don't have any custom iptables/nftables rules, correct? I wonder if something is messing with the Podman networks such that restarting OpenCHAMI services reapplies Podman's rules.

[PowellTravis]

We do not have any custom iptables/nftables rules.