⚠️ CI failed ⚠️ - test_deadlock fails intermittently

[github-actions]

Workflow Run URL

[ncclementi]

@jrbourbeau and @fjetter it looks like the deadlock might still be around? See Workflow run in the comment above.

[fjetter]

How do I debug this? Is there any information about this run? A coiled cluster ID, cluster dump, perf report or anything like that?

[fjetter]

Ok, I got it running locally after a while. Let's see if I can reproduce

[fjetter]

Thank you @ntabris for pointing out that there is a log line that references the cluster 🎉

coiled:cluster.py:552 Creating Cluster (name: test_deadlock-6a165f70a88347c1bafca759d0fb7a3e, https://cloud.coiled.io/dask-engineering/clusters/32950/details). This might take a few minutes...

so this would be https://cloud.coiled.io/dask-engineering/clusters/32950/details

[fjetter]

If this is again the swapping of pages we may need to consider adding limits on coiled, if that is possible

[jrbourbeau]

If this is again the swapping of pages we may need to consider adding limits on coiled, if that is possible

Sorry, I don't quite follow what you mean here @fjetter. Could you elaborate a bit more, or perhaps there's a ticket somewhere with additional context?

[ncclementi]

@fjetter While I was fixing some logic to get the cluster dump working for this particular test. I was able able to get a failure and a cluster dump of it.

CI failure: https://github.com/coiled/coiled-runtime/runs/7049211634?check_suite_focus=true#step:6:94

Cluster dump S3 uri: s3://coiled-runtime-ci/test-scratch/cluster_dumps/test_deadlock-994c7153388c45e0b34a77a1b6fe0a19/

I hope this helps. For future reference, every stability test that fails will have a line like this in the CI report where it points to the S3 URI (in the aws oss account) that has the file.
https://github.com/coiled/coiled-runtime/runs/7049211634?check_suite_focus=true#step:6:356