PRISM Compute Plane: miner-funded GPU worker plane (base)#37
Merged
Conversation
Add src/base/compute/ package: - provider.py: ProviderClient protocol, InstanceSpec (mandatory max_lifetime_hours/max_price_per_hour bounds), Offer/Instance, typed ProviderError/CostGuardrailError. - lium.py: LiumClient over httpx (X-API-Key, base https://lium.io/api). list_offers price filtering; provision refuses unbounded/over-priced specs before any network call, always sends bounded termination_hours, and terminates+verifies on any post-rent failure (try/finally); idempotent ensure_ssh_key/ensure_template; idempotent terminate; verify_terminated via GET /pods; stream_logs; watchtower digest; GET /users/me balance. API key never logged/repr'd/in errors. 50 offline respx unit tests (VAL-PROV-001/003/004/005/011/017/018 + secret hygiene). Full base gate green: 1316 passed, coverage 89.1%.
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (69)
📝 WalkthroughWalkthroughThis PR introduces a complete miner-funded GPU "worker plane" feature: new database tables/migrations for worker registration, assignments, faults, and nonces; master-side coordination, assignment engine, reconciliation, and unit-status services; a worker agent runtime with deploy/proof/execution support; Lium and Targon compute provider clients; CLI commands ( ChangesWorker Plane Feature
Estimated code review effort: 5 (Critical) | ~150 minutes Sequence Diagram(s)sequenceDiagram
participant Agent as WorkerAgent
participant Client as WorkerCoordinationClient
participant Master as Master Proxy
participant Engine as WorkerAssignmentEngine
Agent->>Client: register(binding)
Client->>Master: POST /v1/workers/register
Master->>Master: verify signature, upsert pending
Engine->>Master: assign_pending replicas
loop poll
Agent->>Client: pull()
Client->>Master: POST /v1/workers/assignments/pull
Agent->>Agent: execute + build ExecutionProof
Agent->>Client: post_result(proof)
Client->>Master: POST /v1/workers/assignments/{id}/result
end
sequenceDiagram
participant Recon as WorkerReconciliationService
participant DB as WorkerAssignment/WorkAssignment
participant Validator as Validator Audit
Recon->>DB: collect replica manifest hashes
alt hashes diverge
Recon->>DB: mark unit DISPUTED
Recon->>Validator: create+assign audit unit
Validator-->>Recon: authoritative manifest
Recon->>DB: record WorkerFault
else hashes match
Recon->>DB: mark unit COMPLETED
Recon->>Recon: forward_result once
end
Related Issues: None referenced in provided context. Related PRs: None referenced in provided context. Suggested labels: feature, worker-plane, compute, needs-review-large Suggested reviewers: Reviewers familiar with master orchestration/assignment services and worker security/authentication flows should prioritize the coordination, assignment, and reconciliation layers given their security- and consistency-sensitive nature. Poem: ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…-client secret hygiene
Add build_lium_worker_template (Lium CustomTemplateRequest payload) and build_targon_worker_app (Targon app definition) in src/base/compute/ worker_deployment.py. Both pin the docker image BY DIGEST via a shared pinned_image_reference helper; the image+digest default to the published prism-evaluator digest (M1 placeholder) and are inputs so M2 swaps in docker/Dockerfile.worker with a one-line change. Well-formedness is enforced (required fields, internal ports incl. 22, env plumbing, is_private, no embedded secrets) and covered by offline respx tests that also assert the whole compute suite makes zero real network calls under respx strict mode and needs no provider credentials (VAL-PROV-009/010/016).
…TESTS Add scripts/live_lium_e2e.py driving the M1 provider clients against production: read-only reachability (Lium users/me + executors + watchtower/digest, Targon inventory + apps) plus one batched Lium rental cycle (ensure ssh key + template -> rent cheapest suitable executor -> poll RUNNING -> ssh nvidia-smi -> logs -> DELETE -> verify gone -> record balance delta). Pod is deleted in a finally on every path; opt-in via BASE_LIVE_PROVIDER_TESTS=1 so the default suite stays offline.
…gpu shape - LiumClient.provision rejects sub-1-hour max_lifetime_hours so termination_hours never truncates to 0 (auto-termination stays on) - widen post-rent cleanup guard to key off rent success, not pod-id resolution: a transient GET /pods failure during resolution still best-effort terminates + verifies the just-rented pod - normalize Targon Offer.price_per_hour to per-GPU (cost_per_hour / gpu_count) so the per-GPU max_price cap filters multi-GPU shapes correctly - default WORKER_GPU_SHAPE 'h100' -> live-valid 'h100-small'
Add the miner-funded GPU worker plane registry (architecture.md sec 3.3),
gated behind compute.worker_plane_enabled:
- alembic 0009: worker_registrations + worker_faults + worker_request_nonces
on the existing chain (0008 -> 0009), no legacy table altered.
- WorkerCoordinationService mirroring validator_coordination patterns:
POST /v1/workers/register (sr25519 miner binding verified against the mock
metagraph, binding-nonce replay protection, no silent cross-owner rebind),
POST /v1/workers/{id}/heartbeat, GET /v1/workers (fleet: status/owner/
provider/last-seen/faults, authenticated-but-not-admin), and
GET /v1/workers/active?hotkey= (admission surface).
- Lifecycle pending -> active -> stale -> retired with
compute.worker_heartbeat_ttl_seconds (default 120); retired is terminal
(no heartbeat resurrection); staleness derived on read + a background pass.
- worker_auth: binding message/verify, signed-request verifier + registered-
worker/validator eligibility, worker nonce store.
- Wired into create_proxy_app + the master proxy CLI (flag-off => unmounted).
Unit + Postgres integration tests (15433) cover VAL-MASTER-001/002/015/016/
018/021 and VAL-AGENT-015.
Add the miner-funded WorkerAgent (src/base/worker/): register under a
miner-signed binding, heartbeat to stay active, pull gpu-only replicas,
execute via the AssignmentExecutor seam on its local broker, and post
results that always carry an ExecutionProof envelope (sr25519 over
sha256('{manifest_sha256}:{unit_id}'), pinned identically to prism).
Extract the shared agent-loop primitives into base/coordination/agent_loop
(BackoffPolicy, is_transient_error, AgentCycleSummary, sleep_until,
backoff_sleep); the validator agent now imports them with behavior unchanged.
Add the master worker assignment plane (worker_assignments table + 0010
migration, worker-authenticated pull/result routes gated on registration
and liveness, never a validator permit) and wire it behind
compute.worker_plane_enabled.
Fulfills VAL-AGENT-002/003/004/005/006/007/008/016/017/018.
Add a top-level `base worker` Typer app distinct from the legacy `base master worker` Swarm group: - deploy --provider local starts a miner-funded agent against a local master and reports it active within 60s. - deploy --provider lium|targon requires the provider key env (actionable refusal before any network), bounds offer selection by --max-price preferring an exact gpu_count executor (next-cheapest fallback), and never transmits the provider key to the master (pod env excludes it). - status renders the fleet from GET /v1/workers (signed as the worker key). Thread startup_commands through InstanceSpec, LiumClient.ensure_template and build_lium_worker_template (validated metachar-free) per the live-learned Lium rent constraint; keep docker/Dockerfile.worker's exec-form entrypoint metachar-free. Add WorkerSettings, worker/miner keypair resolvers, signed list_workers, config/worker.example.yaml, and unit tests. All behind compute.worker_plane_enabled; full base gate green.
Add WorkerAssignmentEngine that materializes gpu work-unit replicas onto ACTIVE distinct-owner workers behind compute.worker_plane_enabled: R=2 with self-evaluation exclusion (unit waits under sole-capacity scarcity), graceful degradation to R=1 with a recorded warning, per-worker gpu concurrency 1, and per-replica deadline/reassignment bounded by max_attempts. The validator AssignmentService skips worker-plane capabilities when the flag is on, so flag-OFF gpu routing to validators stays byte-identical to legacy.
… behind flag Reconcile replicated gpu worker results (architecture.md 3.3): matching ExecutionProof.manifest_sha256 forwards exactly one result to the challenge; divergent hashes dispute the unit (never forwarded, before or after audit) and create a validator-executor audit unit whose outcome writes worker_faults for the divergent worker(s), visible in fleet status. Single-replica reporting terminates deterministically (accept-after-degrade with warning); late/foreign posts stay rejected with replica state intact. All gated by compute.worker_plane_enabled (reconciler is None when off).
Legacy AssignmentService.reclaim_stale_assignments treated a null assigned_validator_hotkey as 'offline validator => reassignable', so under the worker plane a worker-owned prism PRIMARY (ASSIGNED, null hotkey by design) churned back to PENDING every MasterOrchestrationDriver.run_once pass. _assign_pending_in_session already skipped such units via worker_plane_capabilities; _reclaim_in_session did not. Factor the guard into AssignmentService._worker_plane_owns and apply it in BOTH paths so a worker-owned primary is neither assigned nor reclaimed here, while a genuinely stale validator unit (cpu, or gpu AUDIT with executor_kind= validator) is still reclaimed. Flag OFF keeps reclaim byte-identical to legacy.
Allow GET /v1/workers/active to authenticate via the prism<->master bridge shared bearer (the same token base forwards results to prism with) in addition to the signed-request path, so prism's live admission check works end-to-end (VAL-CROSS-004). The full-fleet GET /v1/workers stays signed-request only; the internal bearer is rejected there. Flag OFF => router unmounted (404) unchanged.
Add a mock-metagraph master service (base/scripts/mission/mission_master.py) and harness docs (docs/operations/mission-harness.md) for the local end-to-end drills (VAL-CROSS-001/002/003/004/009/011). Surface each worker's attributed fault count in 'base worker status' so the CLI fleet view agrees with GET /v1/workers.
Add GET /v1/workers/units (CoordinationReadEligibility signed-request auth, flag OFF => router unmounted/404) exposing per primary gpu unit: status incl. disputed, replicas (worker/owner/manifest/proof), and for disputed units the linked validator audit unit id + executor_kind + terminal outcome (pending/passed/mismatch-resolved). Makes the dispute->audit->invalidation->fault chain operator-discoverable via API alone (VAL-CROSS-011). Surfaced in the mission master launcher + harness docs. Unit + postgres(15433) integration tests.
Add an offline network-egress guard (pytest plugin -p no_external_egress) plus its unit test so both default suites can be proven to run with ZERO real egress to lium.io / api.targon.com (loopback + AF_UNIX stay allowed). Wire a worker_plane_enabled toggle into the mission master: with the flag OFF no capability is owned by the worker plane, so gpu units route to online gpu validators byte-for-byte as pre-mission (VAL-MASTER-013). Document the exact reproducible flags-OFF verification procedure in the harness docs.
…erminate (VAL-CROSS-005) Add an opt-in (BASE_LIVE_PROVIDER_TESTS=1) live validation that provisions a real Lium pod via the real `base worker deploy --provider lium` CLI, runs a base worker agent inside the pod (pod_worker_agent.py) that enrolls with a LOCAL mission master over a reverse SSH tunnel, pulls+executes a gpu unit on the CPU stub, and posts an ExecutionProof stamped with the lium provider + real pod id; then DELETE + verify gone + GET /pods empty + balance delta <= $2. Pod is terminated on every path (try/finally) so a failure never leaks a billable pod.
… image) base worker deploy --provider lium hit two real gaps: Lium's edge WAF 403s any POST body carrying a loopback URL (baked into the template env), and the private-namespace placeholder WORKER_IMAGE fails pod creation (CREATION_FAILED). - is_loopback_url() + build_worker_pod_env/LiumClient.ensure_template now strip loopback master/broker/gateway URLs from the WAF-sensitive template body; the agent resolves master_url at runtime from config. - require_worker_image() makes worker.deploy.image + image_digest a required, digest-pinned config for provider deploys (fail-fast, clear error); no more silent pin of an un-pullable private image. - docs/miner/worker-plane.md: publish + digest-pin procedure (flagged as a user/release action) + WAF/CREATION_FAILED troubleshooting; config example. Offline respx/unit tests only; full base gate green.
Confirmed live (single green-lit deploy) that Targon has no single
POST /workloads/deploy route (returns 405); the real flow is two-step
register-then-deploy: POST /workloads (-> uid, state registered) then
POST /workloads/{uid}/deploy (-> provisioning). Refactor TargonClient.deploy
to create-then-deploy, surfacing an insufficient-credit failure at either
step as the typed InsufficientCreditsError (never retried). Send the correct
uppercase type RENTAL. Mirror the LiumClient sub-1h lifetime guard into
provision: 0<max_lifetime_hours<1 raises CostGuardrailError instead of
int-truncating termination_hours to 0. respx tests updated; base gate green
(1540 passed, cov 88.4%, mypy/ruff clean).
…gging churn Attach a recording handler directly to the base.supervisor.image_updater logger (mirroring test_supervisor_weights.py / test_supervisor_weight_submit.py) instead of relying on caplog. Importing bittensor raises every already-created logger to CRITICAL, which filtered the WARNING/ERROR records before caplog could capture them, making these tests order-dependent in the full suite. No product code change.
… + flush harness master logs worker_unit_status keyed faulted_units by work_unit_id alone; harden to the (challenge_slug, work_unit_id) tuple so same-id units under different challenge slugs never collide (regression test added). Line-buffer + basicConfig the mission_master harness so its drill logs are non-empty/inspectable after SIGTERM teardown.
…ount fallback - lium.provision: move pod-id extraction inside the try so a 2xx rent with a non-JSON body still terminates+verifies the just-rented pod (no leaked pod); _extract_pod_id raises a typed LiumError on an unparseable body. - targon._extract_gpu_count: fall back to the top-level numeric gpu_count when spec.gpu_count is 0/unknown, so per-GPU price (and max_price filtering) is correct for multi-GPU shapes. - worker.deploy.image_tag documented as informational-only (never consumed by the digest-pinned deploy path) in settings, config, and the miner docs.
# Conflicts: # src/base/cli_app/main.py # tests/unit/test_image_updater.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mission branch for the PRISM Compute Plane (base repo). Delivered incrementally per milestone; this PR aggregates the mission work on `mission/compute-plane`.
M1 providers (in progress)
compute-provider-contract-and-lium-client
All new behavior is additive (new package); no existing behavior changed. Full base gate green: ruff, format, mypy, and pytest (1316 passed, coverage 89.1%).
Summary by CodeRabbit
New Features
Bug Fixes
Documentation