feat(spider-scheduler): Add scheduler crate skeleton with trait and type abstractions.

[LinZhihao-723]

Description

This PR introduces a new crate spider-scheduler and lands the trait and type abstractions that the scheduler will be built on top of. No concrete implementations are included — those follow in subsequent PRs.

Architecture

The scheduler is the serial decision maker that turns ready tasks discovered by the storage layer into assignments for execution managers. It owns placement and ordering policy, not dependency resolution: storage decides what is ready, and the scheduler decides in what order and with what throttling ready tasks are offered to the fleet.

The pipeline:

  storage  ── authoritative ready queue (owned by the storage layer, not this crate)
        │
        │  poll_ready / poll_commit_ready / poll_cleanup_ready  (SchedulerStorageClient)
        ▼
  ┌───────────────────┐
  │   SchedulerCore   │  serial loop: poll → decide → enqueue
  └───────────────────┘
        │
        │  enqueue             (DispatchQueueSink — writer side)
        ▼
  ┌───────────────────┐
  │  dispatch queue   │  bounded SPMC; a full queue back-pressures the core
  └───────────────────┘
        │
        │  dequeue             (DispatchQueueSource — reader side)
        ▼
  ┌───────────────────┐
  │ scheduler service │ ──▶ execution managers (concurrent fan-out)
  └───────────────────┘

Trait seams

• SchedulerStorageClient — the scheduler's view of storage. Three lane-specific polls (poll_ready, poll_commit_ready, poll_cleanup_ready) mirror storage's ReadyQueueReceiverHandle lanes; each returns (SessionId, Vec<InboundEntry>) so a stale-session batch can be detected downstream. job_state(JobId) -> JobState exposes a read-only lookup for placement policies that gate on job lifecycle.

• SchedulerCore — the algorithm seam. Owns its decision loop: poll the inbound queue through its associated StorageClient, apply the scheduling algorithm, and write assignments to its associated Sink. Generic over both, so a real algorithm and a mock can share the same runtime. The loop terminates when its tokio_util::sync::CancellationToken is cancelled.

• DispatchQueueSink — the writer side of the dispatching queue. enqueue(TaskAssignment) awaits when the bounded queue is full, providing the back-pressure that throttles the core to fleet drain rate. bump_session_id(SessionId) advances the queue's current session and invalidates everything currently queued; the core calls it when it observes a strictly-higher session from a poll.

• DispatchQueueSource — the reader side, drained by the EM-facing service. dequeue() -> (SessionId, TaskAssignment) returns the next assignment paired with the session it was enqueued under, so the EM can compare against storage's current session at registration time and discard stale assignments.

Checklist

• The PR satisfies the contribution guidelines.
• This is a breaking change and that has been indicated in the PR title, OR this isn't a
  breaking change.
• Necessary docs have been updated, OR no docs need to be updated.

Validation performed

• Ensure all workflows pass.

Summary by CodeRabbit

• New Features
  • Added the spider-scheduler component, a new scheduling infrastructure module enabling efficient task placement and queue management across the platform. This component provides standardized abstractions for storage integration, task dispatch operations, comprehensive error handling, and assignment workflows necessary for scheduler operations and execution coordination.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 84c88a2f-9f01-4b62-9f06-49a2afeb78a8

📥 Commits

Reviewing files that changed from the base of the PR and between 27091f0 and 2013cbf.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (8)

• Cargo.toml
• components/spider-scheduler/Cargo.toml
• components/spider-scheduler/src/core.rs
• components/spider-scheduler/src/dispatch_queue.rs
• components/spider-scheduler/src/error.rs
• components/spider-scheduler/src/lib.rs
• components/spider-scheduler/src/storage_client.rs
• components/spider-scheduler/src/types.rs

---

Walkthrough

This pull request introduces the spider-scheduler crate, a new Rust component that defines the trait contracts and data structures for task scheduling. The crate establishes abstractions for storage polling, task dispatch operations, and scheduler orchestration, along with foundational error handling.

Changes

Spider Scheduler Core

Layer / File(s)	Summary
Data types and error contracts components/spider-scheduler/src/types.rs, components/spider-scheduler/src/error.rs	InboundEntry and TaskAssignment structures represent task data at the storage/execution boundary. StorageClientError and SchedulerError enums provide error handling for storage operations and scheduler runtime failures.
Storage and dispatch queue abstractions components/spider-scheduler/src/storage_client.rs, components/spider-scheduler/src/dispatch_queue.rs	SchedulerStorageClient trait defines three polling methods (poll_ready, poll_commit_ready, poll_cleanup_ready) and a job state query API. DispatchQueueSink and DispatchQueueSource traits decouple scheduler placement decisions from execution-manager consumption via async enqueue, session bumping, and dequeue operations.
Scheduler core orchestration components/spider-scheduler/src/core.rs	SchedulerCore async trait defines the orchestration contract with associated types for storage client and dispatch sink, plus a run method that drives the scheduling loop with cancellation support.
Crate configuration and public surface Cargo.toml, components/spider-scheduler/Cargo.toml, components/spider-scheduler/src/lib.rs	Workspace membership is registered, crate manifest declares dependencies (async-trait, spider-core, thiserror, tokio-util), and the library root re-exports all public traits, error types, and data structures for downstream consumption.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Suggested reviewers

• sitaowang1998

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: introducing a new scheduler crate with trait and type abstractions.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[sitaowang1998]

We should probably have a batched enqueue method for better performance.

[LinZhihao-723]

The current planned implementation won't benefit from a batch operation:

• The dispatch queue is implemented using async channel, meaning that all enqueue operations will be serialized.
• The scheduler decision maker pops assignments from the queue one by one; a batch operation means we need to construct/destruct vector on top of the popped results, which introduces unnecessary overhead.