Flux machine issues

[jwhite242]

I am currently seeing some issues with the flux scheduler that don't crop up in the others. The machine appears to struggle with lots of fast running jobs, possibly losing track of resource availability, leading to throughput grinding to a halt at some point -> looks to be related to there being a mix of using ats and the flux handler for resource tracking?

Additionally, the time argument to flux mini run is causing some issues, requiring large over estimates of job allocation times other wise there gets to be a race condition where ats thinks jobs are still remaining, but flux won't schedule any of them due to the time requested exceeding the remaining allocation time.

This is tested with a project specific ats wrapper, and on a flux scheduled cluster (not bootstrapped within slurm/etc).

Jeremy

[dawson6]

​[4:12 PM] Dawson, Shawn A.
This one needs to be better defined so we can find the real issue that flux is having or our use of flux is having. But the simple solution of not passing the '-t' option to flux will cause other types of errors. So that is up for discussion still.
​[4:12 PM] Dawson, Shawn A.
Is there a way we can create a simple reproducer for this? I have several small test codes, if it is just matter of ensuring I submit 100's of them of a certain size or what not, we can make that happen,
<https://teams.microsoft.com/l/message/19:53a10cb3fbc745cfa70aa0c38010493b@thread.skype/1679353822518?tenantId=a722dec9-ae4e-4ae3-9d75-fd66e2680a63&amp;groupId=07b41973-fc19-4954-88da-63aae39c8ca1&amp;parentMessageId=1678980904524&amp;teamName=ATS Testing System&channelName=General&createdTime=1679353822518&allowXTenantAccess=false>

[dawson6]

@jwhite242

The 7.0.114 release has significant changes with how flux is used. Can you test this out on rzvernal and see if there are isssues still?

[dawson6]

Closing this for now. The user should try ats version 7.0.120 or later and verify the issue still exists. If so, we can reopen this issue.

[jwhite242]

Well, not sure how i missed your initial response in march, so apologies; it's not really how ats uses it so much as a thing flux is doing: if you specify a time on the flux run calls and there isn't enough time left in an allocation it just won't schedule it at all. Implication being you have to be very precise about your estimated wall times when stacking jobs in time to avoid having later ones just never even being allowed to start, and certainly can't have the flux run times match the time set on the allocation due to scheduling overhead.

Certainly no need to reopen the issue at this point though (and i'm sure we're already on the later versions you tagged anyway).

[dawson6]

Thanks Jeremy,

Just taking a look at the outstanding issues as the support for ATS will be transitioning to Stephanie Dempsey, Ben Liu, and Ryan Quinlan in a few weeks.

-Shawn