Skip to content

feat: add warm queues#1680

Draft
ltalirz wants to merge 1 commit intoAzure:mainfrom
ltalirz:feat/add-warmup
Draft

feat: add warm queues#1680
ltalirz wants to merge 1 commit intoAzure:mainfrom
ltalirz:feat/add-warmup

Conversation

@ltalirz
Copy link
Copy Markdown
Contributor

@ltalirz ltalirz commented Sep 12, 2023

fixes #1172

Warm queues will always hold at least 1 idling node so that new jobs can start instantly. The cronjob that checks for this is run every 5 minutes, on workdays only.

Warm queues will always hold at least 1 idling node so that new jobs can start instantly.
The cronjob that checks for this is run every 5 minutes, on workdays only.
@ltalirz
Copy link
Copy Markdown
Contributor Author

ltalirz commented Sep 12, 2023

This is the setup we've been using for SLURM - it works (and is a big productivity win in practice) but would need to be generalized for other schedulers.

Of course it is not the ideal solution, since it creates some noise in the job scheduler (the job counter will go up). Ideally, CycleCloud or the scheduler could be configured to take care of this (and perhaps there is a way to get approximately what we do here).

@xpillons
Copy link
Copy Markdown
Collaborator

this is interesting enough to be mainstream. can you please also update the config.tpl.yml and the documentation ?

@xpillons xpillons added kind/feature New feature request do-not-merge labels Sep 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge kind/feature New feature request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Test queues

2 participants