Skip to content

fix: race condition during merging#42

Open
alexander-held wants to merge 5 commits intomainfrom
fix/merge-race-condition
Open

fix: race condition during merging#42
alexander-held wants to merge 5 commits intomainfrom
fix/merge-race-condition

Conversation

@alexander-held
Copy link
Copy Markdown
Member

@alexander-held alexander-held commented Jan 20, 2026

The custom processing caused ReduceSchedulingError instances which seemed to be more frequent with more complex workloads. Switch to dask.distributed.as_completed instead, which does not run into these issues based on my testing.

Comment thread util/_dask.py Outdated
@alexander-held
Copy link
Copy Markdown
Member Author

In its current state this may have helped partially but hit a new case I have previously not seen:

ReduceSchedulingError: dataset [...] has 3 items in merge-buffer (should only be 1); chunks left to merge: 0

The more common version prior to the changes here was:

ReduceSchedulingError: dataset [...] has 0 items in merge-buffer (should only be 1); chunks left to merge: 0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant