New connections to draining Runners

[mpass99]

Our current drain_on_shutdown strategy for stopping Nomad agents is:

• On Shutdown the Nomad agent gets ineligible and no new runners are being scheduled.
• In the drain-on-shutdown deadline all running executions have time to finish.
  • ⚡ We still start new Executions in runners on the draining agent that may not have enough time to finish
• After that the Nomad Agent shuts down

The executions that don't have enough time to finish result in a user-visible error.

We might need to "exclude" some runners for new executions as soon as the respective Nomad agent is about to shut down.

See #651

---

Unfortunately, we currently don't have any metric to count how often this issue occurs.

[MrSerth]

ToDo: Let's identify which error / log information / ... we get when above issue occurs.

[mpass99]

We've conducted a local reproduction of this scenario: nomadEventLog-ExecuteDraining.txt.
It shows that POSEIDON-3W (#590) with the sub-error the allocation was rescheduled indicates this error. This has not happened for at least 90 days.

If we consider a fix for this necessary in the future, we might consider listening to Nomad's Node events to receive drain updates, fetch all allocations of this node, and block new executions for these allocations/runners. Further, we should ensure that the drain deadline matches the maximum of all allowed execution timeouts (of CodeOcean).

[MrSerth]

This issue is still valid and could be a nice improvement. However, we don't expect that this problem occurs many times, so that it doesn't have a high priority.