HostManager failed to remove container #373
Description
Odd problem today (finally resolved at 2024-01-18 14:30:06 UTC) when ACU agent was misbehaving, and hostmanager "down" could not kill it (twice, though first attempt was brief). HM logs show:
2024-01-18T14:23:51+0000 start called for update
2024-01-18T14:23:51+0000 update:140 Status is now "starting".
2024-01-18T14:23:51+0000 update:140 Status is now "running".
2024-01-18T14:23:51+0000 update:140 Update requested.
2024-01-18T14:23:51+0000 update:140 Status is now "done".
2024-01-18T14:23:52+0000 manager:0 Requesting termination of ACUAgent:acu
2024-01-18T14:23:57+0000 manager:0 Agent instance ACUAgent:acu refused to die.
2024-01-18T14:23:58+0000 manager:0 Detected unexpected session for ACUAgent:acu (probably docker); it will ...
2024-01-18T14:24:03+0000 manager:0 Requesting termination of ACUAgent:acu
2024-01-18T14:24:08+0000 manager:0 Agent instance ACUAgent:acu refused to die.
2024-01-18T14:24:09+0000 manager:0 Detected unexpected session for ACUAgent:acu (probably docker); it will ...
2024-01-18T14:24:13+0000 manager:0 Requesting termination of ACUAgent:acu
2024-01-18T14:24:19+0000 manager:0 Agent instance ACUAgent:acu refused to die.
...
But running docker-compose rm --stop --force ocs-acu, as ocs user, did bring down the container, without any issues. (That command should be exactly what the HM runs.)
I have no useful analysis yet...