MUPdate failure: Downloading installinator appears stuck

[smklein]

On 6/23, I attempted to MUPdate the dogfood rack.

Sled 21 showed "Downloading installinator", with the following expanded message:

It's unclear to me why it is stuck here - it looks like it should be complete?

[jgallagher]

When I hopped on the console, instead of a login prompted I was greeted with

Enter user name for system maintenance (control-d to bypass):

I left it in this state for someone else to poke at.

[rcgoodfellow]

Found services in this state

root@:/var/adm# svcs -xv
svc:/network/physical:default (physical network interfaces)
 State: offline since Sun Dec 28 00:00:04 1986
Reason: Start method is running.
   See: http://illumos.org/msg/SMF-8000-C4
   See: man -M /usr/share/man -s 8 ifconfig
   See: /var/svc/log/network-physical:default.log
Impact: 26 dependent services are not running:
        svc:/milestone/network:default
        svc:/network/initial:default
        svc:/network/service:default
        svc:/network/dns/client:default
        svc:/milestone/name-services:default
        svc:/milestone/multi-user:default
        svc:/system/boot-config:default
        svc:/milestone/multi-user-server:default
        svc:/system/system-log:default
        svc:/system/cron:default
        svc:/network/netmask:default
        svc:/milestone/single-user:default
        svc:/milestone/sysconfig:default
        svc:/system/utmp:default
        svc:/system/console-login:default
        svc:/system/sac:default
        svc:/system/filesystem/local:default
        svc:/system/dumpadm:default
        svc:/system/hotplug:default
        svc:/system/t6init:default
        svc:/oxide/installinator:default
        svc:/system/boot-archive-update:default
        svc:/network/routing-setup:default
        svc:/system/identity:node
        svc:/system/picl:default
        svc:/system/identity:domain

svc:/network/physical:nwam (physical network interface autoconfiguration)
 State: disabled since Sun Dec 28 00:00:03 1986
Reason: Disabled by an administrator.
   See: http://illumos.org/msg/SMF-8000-05
   See: man -M /usr/share/man -s 8 nwamd
   See: http://hub.opensolaris.org/bin/view/Project+nwam/
Impact: 23 dependent services are not running:
        svc:/milestone/network:default
        svc:/network/initial:default
        svc:/network/service:default
        svc:/network/dns/client:default
        svc:/milestone/name-services:default
        svc:/milestone/multi-user:default
        svc:/system/boot-config:default
        svc:/milestone/multi-user-server:default
        svc:/system/system-log:default
        svc:/system/cron:default
        svc:/network/netmask:default
        svc:/milestone/single-user:default
        svc:/milestone/sysconfig:default
        svc:/system/utmp:default
        svc:/system/console-login:default
        svc:/system/sac:default
        svc:/system/filesystem/local:default
        svc:/system/dumpadm:default
        svc:/system/hotplug:default
        svc:/system/t6init:default
        svc:/oxide/installinator:default
        svc:/system/boot-archive-update:default
        svc:/network/routing-setup:default

svc:/system/rbac:default (Assemble the RBAC *attr files.)
 State: offline since Sun Dec 28 00:00:05 1986
Reason: Start method is running.
   See: http://illumos.org/msg/SMF-8000-C4
   See: /var/svc/log/system-rbac:default.log
Impact: 20 dependent services are not running:
        svc:/system/manifest-import:default
        svc:/system/boot-config:default
        svc:/milestone/single-user:default
        svc:/milestone/multi-user:default
        svc:/milestone/multi-user-server:default
        svc:/milestone/sysconfig:default
        svc:/system/system-log:default
        svc:/system/utmp:default
        svc:/system/console-login:default
        svc:/system/sac:default
        svc:/system/filesystem/local:default
        svc:/system/cron:default
        svc:/system/dumpadm:default
        svc:/system/hotplug:default
        svc:/system/t6init:default
        svc:/oxide/installinator:default
        svc:/system/boot-archive-update:default
        svc:/network/routing-setup:default
        svc:/system/coreadm:default
        svc:/system/name-service-cache:default

svc:/site/recovery/hostname:default (recovery hostname)
 State: offline since Sun Dec 28 00:00:05 1986
Reason: Start method is running.
   See: http://illumos.org/msg/SMF-8000-C4
   See: /var/svc/log/site-recovery-hostname:default.log
Impact: 19 dependent services are not running:
        svc:/system/identity:node
        svc:/milestone/single-user:default
        svc:/milestone/multi-user:default
        svc:/system/boot-config:default
        svc:/milestone/multi-user-server:default
        svc:/milestone/sysconfig:default
        svc:/system/system-log:default
        svc:/system/utmp:default
        svc:/system/console-login:default
        svc:/system/sac:default
        svc:/system/filesystem/local:default
        svc:/system/cron:default
        svc:/system/dumpadm:default
        svc:/system/hotplug:default
        svc:/system/t6init:default
        svc:/oxide/installinator:default
        svc:/system/boot-archive-update:default
        svc:/system/picl:default
        svc:/system/identity:domain

svc:/system/sysevent:default (system event notification)
 State: offline since Sun Dec 28 00:00:05 1986
Reason: Start method is running.
   See: http://illumos.org/msg/SMF-8000-C4
   See: man -M /usr/share/man -s 8 syseventd
   See: /var/svc/log/system-sysevent:default.log
Impact: 17 dependent services are not running:
        svc:/milestone/single-user:default
        svc:/milestone/multi-user:default
        svc:/system/boot-config:default
        svc:/milestone/multi-user-server:default
        svc:/milestone/sysconfig:default
        svc:/system/system-log:default
        svc:/system/utmp:default
        svc:/system/console-login:default
        svc:/system/sac:default
        svc:/system/filesystem/local:default
        svc:/system/cron:default
        svc:/system/dumpadm:default
        svc:/system/hotplug:default
        svc:/system/t6init:default
        svc:/oxide/installinator:default
        svc:/system/boot-archive-update:default
        svc:/system/picl:default

svc:/system/logadm-upgrade:default (logadm upgrade)
 State: offline since Sun Dec 28 00:00:05 1986
Reason: Start method is running.
   See: http://illumos.org/msg/SMF-8000-C4
   See: /var/svc/log/system-logadm-upgrade:default.log
Impact: This service is not running.

physical networking angry about no egrep.

root@:/var/adm# cat /var/svc/log/network-physical:default.log
[ Dec 28 00:00:03 Enabled. ]
[ Dec 28 00:00:04 Executing start method ("/lib/svc/method/net-physical"). ]
[ Dec 28 00:00:04 Timeout override by svc.startd.  Using infinite timeout. ]
/lib/svc/method/net-physical[469]: egrep: not found [No such file or directory]
dladm: warning: kstat open operation failed

[jclulow]

How did we decide to remove egrep?

[jgallagher]

It looks like this is another casualty of oxidecomputer/helios#88, but I'm a little confused how worked on the remaining sleds. Is the use of egrep conditional (and not particularly common)?

[rcgoodfellow]

It does not appear to be conditional.

• https://github.com/oxidecomputer/illumos-gate/blob/stlouis/usr/src/cmd/svc/milestone/net-physical#L469

However, this was in maintenance/single-user mode. So it's not clear if this is an artifact of being in maintenance mode or the reason we are in maintenance mode.

[jgallagher]

egrep was restored in oxidecomputer/helios#96, so if its lack was the root cause this is fixed. Leaving open but changing the milestone for now, until/if we see it again.