Summary
Update scheduler-to-worker dataset routing logic to support multiple data node targets per fetch.
- Allow a
Fetch request to include multiple peer IDs (data nodes) with a random selection strategy.
- Adjust the scheduler to return a list of data nodes for each slice.
- Ensure fallback behavior is graceful: if one data node fails, retries will hit another peer.
- Reuse and extend the existing
Reference::Peers variant and SelectionStrategy::Random.
Background
Currently, a worker fetches a slice from only one data node, which is a single point of failure. This change enables fault-tolerant fetching and prepares the system for distributed data delivery at scale.
Summary
Update scheduler-to-worker dataset routing logic to support multiple data node targets per fetch.
Fetchrequest to include multiple peer IDs (data nodes) with a random selection strategy.Reference::Peersvariant andSelectionStrategy::Random.Background
Currently, a worker fetches a slice from only one data node, which is a single point of failure. This change enables fault-tolerant fetching and prepares the system for distributed data delivery at scale.