flake: Non-empty /var/tmp/omicron_tmp (propolis zone filesystem hung around) #10063
Description
This CI run failed on a CI run on #10061
https://github.com/oxidecomputer/omicron/runs/67014051871
[12441](https://buildomat.eng.oxide.computer/wg/0/details/01KKMDHFKZPW5MF94CXZAN6E7V/WTNrd25u6CYkkz2rziQTmv8YI6x1Q0KzojDXDBBq9pJyqnyr/01KKMDJ6GWC3S57RZ1AEDZP2TJ#S12441) 2026-03-13T21:02:06.699Z + echo 'files in /var/tmp/omicron_tmp (none expected on success):'
[12442](https://buildomat.eng.oxide.computer/wg/0/details/01KKMDHFKZPW5MF94CXZAN6E7V/WTNrd25u6CYkkz2rziQTmv8YI6x1Q0KzojDXDBBq9pJyqnyr/01KKMDJ6GWC3S57RZ1AEDZP2TJ#S12442) 2026-03-13T21:02:06.699Z files in /var/tmp/omicron_tmp (none expected on success):
[12443](https://buildomat.eng.oxide.computer/wg/0/details/01KKMDHFKZPW5MF94CXZAN6E7V/WTNrd25u6CYkkz2rziQTmv8YI6x1Q0KzojDXDBBq9pJyqnyr/01KKMDJ6GWC3S57RZ1AEDZP2TJ#S12443) 2026-03-13T21:02:06.699Z + find /var/tmp/omicron_tmp -ls
[12444](https://buildomat.eng.oxide.computer/wg/0/details/01KKMDHFKZPW5MF94CXZAN6E7V/WTNrd25u6CYkkz2rziQTmv8YI6x1Q0KzojDXDBBq9pJyqnyr/01KKMDJ6GWC3S57RZ1AEDZP2TJ#S12444) 2026-03-13T21:02:06.699Z 164375 39 drwxr-xr-x 3 build build 3 Mar 13 21:02 /var/tmp/omicron_tmp
[12445](https://buildomat.eng.oxide.computer/wg/0/details/01KKMDHFKZPW5MF94CXZAN6E7V/WTNrd25u6CYkkz2rziQTmv8YI6x1Q0KzojDXDBBq9pJyqnyr/01KKMDJ6GWC3S57RZ1AEDZP2TJ#S12445) 2026-03-13T21:02:06.699Z 226509 1 drwxr-xr-x 3 build build 3 Mar 13 20:55 /var/tmp/omicron_tmp/.tmpridzi8
[12446](https://buildomat.eng.oxide.computer/wg/0/details/01KKMDHFKZPW5MF94CXZAN6E7V/WTNrd25u6CYkkz2rziQTmv8YI6x1Q0KzojDXDBBq9pJyqnyr/01KKMDJ6GWC3S57RZ1AEDZP2TJ#S12446) 2026-03-13T21:02:06.700Z 226516 1 drwxr-xr-x 3 build build 3 Mar 13 20:55 /var/tmp/omicron_tmp/.tmpridzi8/var
[12447](https://buildomat.eng.oxide.computer/wg/0/details/01KKMDHFKZPW5MF94CXZAN6E7V/WTNrd25u6CYkkz2rziQTmv8YI6x1Q0KzojDXDBBq9pJyqnyr/01KKMDJ6GWC3S57RZ1AEDZP2TJ#S12447) 2026-03-13T21:02:06.700Z 226517 1 drwxr-xr-x 3 build build 3 Mar 13 20:55 /var/tmp/omicron_tmp/.tmpridzi8/var/tmp
[12448](https://buildomat.eng.oxide.computer/wg/0/details/01KKMDHFKZPW5MF94CXZAN6E7V/WTNrd25u6CYkkz2rziQTmv8YI6x1Q0KzojDXDBBq9pJyqnyr/01KKMDJ6GWC3S57RZ1AEDZP2TJ#S12448) 2026-03-13T21:02:06.700Z 226518 1 drwxr-xr-x 3 build build 3 Mar 13 20:55 /var/tmp/omicron_tmp/.tmpridzi8/var/tmp/omicron_tmp
[12449](https://buildomat.eng.oxide.computer/wg/0/details/01KKMDHFKZPW5MF94CXZAN6E7V/WTNrd25u6CYkkz2rziQTmv8YI6x1Q0KzojDXDBBq9pJyqnyr/01KKMDJ6GWC3S57RZ1AEDZP2TJ#S12449) 2026-03-13T21:02:06.700Z 226519 1 drwxr-xr-x 3 build build 3 Mar 13 20:55 /var/tmp/omicron_tmp/.tmpridzi8/var/tmp/omicron_tmp/.tmp7eqlkt
[12450](https://buildomat.eng.oxide.computer/wg/0/details/01KKMDHFKZPW5MF94CXZAN6E7V/WTNrd25u6CYkkz2rziQTmv8YI6x1Q0KzojDXDBBq9pJyqnyr/01KKMDJ6GWC3S57RZ1AEDZP2TJ#S12450) 2026-03-13T21:02:06.700Z 226520 1 drwxr-xr-x 3 build build 3 Mar 13 20:55 /var/tmp/omicron_tmp/.tmpridzi8/var/tmp/omicron_tmp/.tmp7eqlkt/crypt
[12451](https://buildomat.eng.oxide.computer/wg/0/details/01KKMDHFKZPW5MF94CXZAN6E7V/WTNrd25u6CYkkz2rziQTmv8YI6x1Q0KzojDXDBBq9pJyqnyr/01KKMDJ6GWC3S57RZ1AEDZP2TJ#S12451) 2026-03-13T21:02:06.700Z 226521 11 drwxr-xr-x 3 build build 3 Mar 13 20:55 /var/tmp/omicron_tmp/.tmpridzi8/var/tmp/omicron_tmp/.tmp7eqlkt/crypt/zone
[12452](https://buildomat.eng.oxide.computer/wg/0/details/01KKMDHFKZPW5MF94CXZAN6E7V/WTNrd25u6CYkkz2rziQTmv8YI6x1Q0KzojDXDBBq9pJyqnyr/01KKMDJ6GWC3S57RZ1AEDZP2TJ#S12452) 2026-03-13T21:02:06.700Z 226522 1 drwxr-xr-x 3 build build 3 Mar 13 20:55 /var/tmp/omicron_tmp/.tmpridzi8/var/tmp/omicron_tmp/.tmp7eqlkt/crypt/zone/oxz_propolis-server_e8e95a60-2aaf-4453-90e4-e0e58f126762
[12453](https://buildomat.eng.oxide.computer/wg/0/details/01KKMDHFKZPW5MF94CXZAN6E7V/WTNrd25u6CYkkz2rziQTmv8YI6x1Q0KzojDXDBBq9pJyqnyr/01KKMDJ6GWC3S57RZ1AEDZP2TJ#S12453) 2026-03-13T21:02:06.700Z 226523 1 drwxr-xr-x 3 build build 3 Mar 13 20:55 /var/tmp/omicron_tmp/.tmpridzi8/var/tmp/omicron_tmp/.tmp7eqlkt/crypt/zone/oxz_propolis-server_e8e95a60-2aaf-4453-90e4-e0e58f126762/root
[12454](https://buildomat.eng.oxide.computer/wg/0/details/01KKMDHFKZPW5MF94CXZAN6E7V/WTNrd25u6CYkkz2rziQTmv8YI6x1Q0KzojDXDBBq9pJyqnyr/01KKMDJ6GWC3S57RZ1AEDZP2TJ#S12454) 2026-03-13T21:02:06.700Z 226525 1 drwxr-xr-x 3 build build 3 Mar 13 20:55 /var/tmp/omicron_tmp/.tmpridzi8/var/tmp/omicron_tmp/.tmp7eqlkt/crypt/zone/oxz_propolis-server_e8e95a60-2aaf-4453-90e4-e0e58f126762/root/var
[12455](https://buildomat.eng.oxide.computer/wg/0/details/01KKMDHFKZPW5MF94CXZAN6E7V/WTNrd25u6CYkkz2rziQTmv8YI6x1Q0KzojDXDBBq9pJyqnyr/01KKMDJ6GWC3S57RZ1AEDZP2TJ#S12455) 2026-03-13T21:02:06.700Z 226526 1 drwxr-xr-x 3 build build 3 Mar 13 20:55 /var/tmp/omicron_tmp/.tmpridzi8/var/tmp/omicron_tmp/.tmp7eqlkt/crypt/zone/oxz_propolis-server_e8e95a60-2aaf-4453-90e4-e0e58f126762/root/var/svc
[12456](https://buildomat.eng.oxide.computer/wg/0/details/01KKMDHFKZPW5MF94CXZAN6E7V/WTNrd25u6CYkkz2rziQTmv8YI6x1Q0KzojDXDBBq9pJyqnyr/01KKMDJ6GWC3S57RZ1AEDZP2TJ#S12456) 2026-03-13T21:02:06.700Z 226527 1 drwxr-xr-x 2 build build 3 Mar 13 20:55 /var/tmp/omicron_tmp/.tmpridzi8/var/tmp/omicron_tmp/.tmp7eqlkt/crypt/zone/oxz_propolis-server_e8e95a60-2aaf-4453-90e4-e0e58f126762/root/var/svc/profile
[12457](https://buildomat.eng.oxide.computer/wg/0/details/01KKMDHFKZPW5MF94CXZAN6E7V/WTNrd25u6CYkkz2rziQTmv8YI6x1Q0KzojDXDBBq9pJyqnyr/01KKMDJ6GWC3S57RZ1AEDZP2TJ#S12457) 2026-03-13T21:02:06.700Z 228871 1 -rw-r--r-- 1 build build 713 Mar 13 20:55 /var/tmp/omicron_tmp/.tmpridzi8/var/tmp/omicron_tmp/.tmp7eqlkt/crypt/zone/oxz_propolis-server_e8e95a60-2aaf-4453-90e4-e0e58f126762/root/var/svc/profile/site.xml
[12458](https://buildomat.eng.oxide.computer/wg/0/details/01KKMDHFKZPW5MF94CXZAN6E7V/WTNrd25u6CYkkz2rziQTmv8YI6x1Q0KzojDXDBBq9pJyqnyr/01KKMDJ6GWC3S57RZ1AEDZP2TJ#S12458) 2026-03-13T21:02:06.700Z + rmdir /var/tmp/omicron_tmp
[12459](https://buildomat.eng.oxide.computer/wg/0/details/01KKMDHFKZPW5MF94CXZAN6E7V/WTNrd25u6CYkkz2rziQTmv8YI6x1Q0KzojDXDBBq9pJyqnyr/01KKMDJ6GWC3S57RZ1AEDZP2TJ#S12459) 2026-03-13T21:02:06.700Z rmdir: directory "/var/tmp/omicron_tmp": Directory not empty
It looks like a filesystem was set up for a propolis-server, within /var/tmp/omicron_tmp, that was not removed.
This really should be unrelated to the associated PR, which is why I believe it's a test flake.
EDIT: I originally thought this was a live test issue, but I don't believe it is. Trying to crawl through the logs and see if I can get any clue about which test left this behind...