daemon communcation fault

[astronomerdave]

Got into a state where sequencer could not connect to focusd:

2025-12-11T07:55:51.042  (Sequencer::Sequence::connect_to_daemon) connected to focusd
2025-12-11T07:55:57.048  (Common::DaemonClient::send) TIMEOUT polling socket localhost/9004 on fd 31 for focusd
2025-12-11T07:55:57.048  (Common::DaemonClient::connect) closing existing connection to focusd socket localhost/9004 on fd 31
2025-12-11T07:55:57.048  (Common::DaemonClient::connect) connected to focusd at localhost/9004 on fd 31
2025-12-11T07:56:04.054  (Common::DaemonClient::send) TIMEOUT polling socket localhost/9004 on fd 31 for focusd
2025-12-11T07:56:04.054  (Common::DaemonClient::connect) closing existing connection to focusd socket localhost/9004 on fd 31
2025-12-11T07:56:04.055  (Common::DaemonClient::connect) connected to focusd at localhost/9004 on fd 31
2025-12-11T07:56:11.061  (Common::DaemonClient::send) TIMEOUT polling socket localhost/9004 on fd 31 for focusd
2025-12-11T07:56:11.061  (Common::DaemonClient::connect) closing existing connection to focusd socket localhost/9004 on fd 31
2025-12-11T07:56:11.062  (Common::DaemonClient::connect) connected to focusd at localhost/9004 on fd 31
2025-12-11T07:56:12.062  (Common::DaemonClient::send) [TEST] attempting to flush after timeout
2025-12-11T07:56:14.064  (Sequencer::Sequence::parse_state) ERROR parsing "" from Sequencer::Sequence::open_hardware: expected <state> <error>
2025-12-11T07:56:14.064  (Sequencer::Sequence::open_hardware) ERROR opening focusd hardware
2025-12-11T07:56:14.064  (Sequencer::Sequence::focus_init) ERROR initializing focus control
2025-12-11T07:56:14.064  (Sequencer::Sequence::broadcast_waitstate) WAITSTATE:
2025-12-11T07:56:14.064  (Sequencer::Sequence::startup) ERROR worker focus_init exception: could not open focus hardware

I discovered that sequencerd got stuck in CLOSE_WAIT:

$ sudo lsof -i :9004
COMMAND      PID      USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
focusd    233358 dataowner   14u  IPv4 1478252      0t0  TCP *:9004 (LISTEN)
focusd    233358 dataowner   16u  IPv4 7239300      0t0  TCP localhost:9004->localhost:59354 (CLOSE_WAIT)
sequencer 821844 dataowner   31u  IPv4 7701888      0t0  TCP localhost:56018->localhost:9004 (ESTABLISHED)

I had to restart both daemons. This is what it should look like:

$ sudo lsof -i :9004
COMMAND       PID      USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
focusd    1052863 dataowner   14u  IPv4 7755878      0t0  TCP *:9004 (LISTEN)
focusd    1052863 dataowner   16u  IPv4 7750558      0t0  TCP localhost:9004->localhost:40810 (ESTABLISHED)
sequencer 1053145 dataowner   32u  IPv4 7754080      0t0  TCP localhost:40810->localhost:9004 (ESTABLISHED)