PRISM Compute Plane — miner-funded GPU worker plane (mission/compute-plane)#1
Merged
Conversation
…m-proof)
Add src/prism_challenge/proof.py: ExecutionProof v1 envelope (schema-compatible
with the base worker plane), canonical manifest hashing (sha256 over
json.dumps(manifest, sort_keys=True, indent=2) == the exact on-disk bytes of
prism_run_manifest.v2.json), tier computation from injected provider env
(tier 0 default; tier 1 needs BOTH image digest AND pod metadata; tier 2 only
with a non-null attestation payload), and an sr25519 worker signature over the
pinned message sha256(f'{manifest_sha256}:{unit_id}') (identical to the base
agent, VAL-AGENT-008/VAL-PRISM-006).
Wire emission into the validator finalization path: the dispatch result payload
carries exactly one signed proof at successful finalization, gated on a new
prism worker_plane config block (enabled by default OFF -> legacy behavior).
Proof construction reads only the manifest + a fixed non-secret provider env
allowlist and never the held-out split config or LLM keys (VAL-PRISM-008).
Fulfills VAL-PRISM-001/002/003/004/005/006/008.
|
Warning Review limit reached
Next review available in: 59 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (37)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…ion + idempotent ingestion (M3 prism-proof)
CREATE POST /internal/v1/work_units/result (body {work_unit_id, submission_ref,
result}), the counterpart to base's HttpChallengeResultForwarder. Verify the
forwarded ExecutionProof BEFORE scoring: reject missing/malformed proof
(VAL-PRISM-018) and tampered manifest / forged digest (VAL-PRISM-007) 422 with a
distinguishable reason, leaving the unit eligible for retry. Downgrade
unverifiable tier claims to their effective tier for audit sampling
(VAL-PRISM-019). Finalize idempotently via a guarded work_unit_results key so a
duplicate delivery is a no-op and a conflicting one is refused 409, never mutating
a finalized score/leaderboard (VAL-PRISM-017). Gated behind worker_plane.enabled
(404 when off).
…proof) Add a verify-only plausibility module (architecture.md 3.5) that rejects an implausible prism_run_manifest.v2 before scoring and passes a plausible manifest through unchanged: - schema/version + metrics-block validation - step-0 anomaly (initial loss below the from-scratch baseline) - loss-trajectory sanity (final/late online loss at/above ~ln(vocab)) - log/score consistency - wall-clock vs declared budget Wired into the base->prism verify-only ingestion path with a distinct plausibility_* rejection reason (never scored, unit stays retry-eligible); plausible results finalize identically to the legacy path (VAL-PRISM-009/010).
…3 prism-proof) Sample finalized worker results at per-tier rates (0.10/0.05/0.02, seeded); sampled results create DISTINCT-id validator audit units on the existing validator_dispatch path (pending-only listing). Audit mismatch invalidates the submission score in its epoch leaderboard and propagates to crown/weights; audit failure/timeout re-audits within bounds and never silently accepts. R=1-degraded results stay audit-eligible at their effective-tier rate; worker hotkeys still cannot publish checkpoints while resume refs reach reassigned units. Fulfills VAL-PRISM-011,012,013,023,024,025,026.
…essions (M3 prism-proof) Adds the fail-closed submission admission rule behind worker_plane.admission_requires_worker: POST /v1/submissions and the base bridge path /internal/v1/bridge/submissions both query the master's GET /v1/workers/active?hotkey= and reject with 403 NO_ACTIVE_WORKER unless >=1 active worker is confirmed. Master unreachable/timeout/5xx folds into the same bounded 403 shape (VAL-PRISM-014/020/021). Flag OFF is byte-identical to legacy with zero master calls (VAL-PRISM-015). Scoring and get_weights/leaderboard/epoch surfaces are unchanged ON vs OFF (VAL-PRISM-016/022).
…thout re-execution (M3 prism-finalization) Under worker_plane.enabled, ingest_work_unit_result now finalizes the submission score from the forwarded, verified+reconciled prism_run_manifest.v2 via a new PrismWorker.finalize_worker_result that takes NO GPU lease and NEVER constructs the evaluator (no evaluator.evaluate / _evaluate_within_wall_time / _augment_with_heldout). It reuses the _finalize_container_score tail with skip_heldout=True so the score is prequential-bpb-only and the master-only secret val split (base_eval_val_data_dir) is never read (architecture.md 4). Dedup/conflict and proof+plausibility gates run before finalization exactly as-is; flag OFF keeps the legacy process_submission re-execution path byte-identical. Fulfills VAL-FINAL-001..004.
… audit sampler (M3 prism-finalization) Wire the audit-only validator cycle end-to-end: enumerate pending audit: units (is_audit_unit_id), replay the audited submission's evaluation deterministically for a fresh manifest_sha256, and resolve via resolve_audit_unit -- matching replay leaves the finalized score untouched, divergent replay invalidates it and records a worker_fault. dispatch_assignment routes on unit type so validators run audits while workers still execute primaries + emit proofs under the flag; the autonomous run_validator_cycle is audit-only when the flag is on and legacy primary execution when off. Mix a server-side secret WorkerPlaneConfig.audit_salt (repr=False) into the AuditSampler seed so selection is unpredictable from submission_id yet reproducible for a fixed salt, preserving per-tier rates (VAL-FINAL-005/006).
…VAL-FINAL-007) On audit-mismatch invalidation, _recompute_family_after_invalidation now advances architecture_families.owner_hotkey (and owner_submission_id) to the surviving best submission's owner alongside canonical_submission_id/q_arch_best, so a proven-faulty family creator loses the 0.60 architecture emission share that get_weights rewards and the surviving co-owner is paid. The single-owner no-survivor burn (q_arch_best -> 0.0) and the normal (non-invalidated) persistent-crown-pays-creator semantics are unchanged. Adds a multi-owner-same-arch regression test proving the faulty owner loses the share and get_weights pays the survivor.
…ents Add worker_plane.cpu_reexec_test_mode config that installs the repo's own CPU re-exec seam (mock_reexec.cpu_reexec_run) with a tiny deterministic PrismContext, plus an evaluator/cpu_test_mode helper (stage data, install seam, normalize volatile manifest fields so honest replicas agree on one manifest_sha256) wired into create_app. Add the local mission worker/validator/prism agents and drill launcher (scripts/mission) for the cross-repo e2e harness (VAL-CROSS-001..011).
…units
Extend drill 3 to rebuild the full dispute->audit->invalidation->fault story from
operator APIs alone (VAL-CROSS-011): (a)/(b) from the new master GET
/v1/workers/units, (c) from prism GET /v1/submissions/{id}, (d) fault on GET
/v1/workers + base worker status. Adds a master_units() signed-read helper.
…n rule, worker_plane flag)
Add a legacy validator that runs an assigned prism PRIMARY unit through the real validator_dispatch path, plus a flags-OFF smoke orchestrator that stands up the local mock-metagraph deployment with the worker plane OFF and drives one legacy submission end-to-end. It asserts (via HTTP + master/prism DB reads) the pre-mission behavior: submission accepted with no admission 403, exactly one gpu work unit, the unit assigned to the VALIDATOR (never a worker) with no worker-plane rows/side effects, and a score finalized via validator_dispatch. Wire a worker_plane_enabled toggle into the mission prism service so the flag OFF disables the worker plane + admission gate.
…aim lease finalize_worker_result no longer returns a truthy submission_id when the source-static derivation FAILS: it reverts the CAS claim to pending (retryable) and raises WorkerFinalizationError, which ingestion surfaces as a distinct finalization_failed outcome (finalized=False, nothing recorded, HTTP 503) so a transient internal error is retried instead of being silently marked terminally finalized (status=failed). Happy path and all VAL-FINAL assertions preserved. run_validator_audit_cycle now claims each pending audit under a lightweight per-audit lease (repository.claim_audit_unit + audit_units.claimed_at/by, audit_claim_lease_seconds) before replaying it, so a multi-validator deployment no longer has every validator redundantly replay the same audits; each pending audit is replayed by at most one validator. The claim is cleared on resolution so a re-audited unit is immediately reclaimable, and a crashed claimant's lease expires. Single-validator behavior and VAL-FINAL-005 unchanged.
…; harness log/re-run hygiene
Strengthen the launch.py drills so PASS/FAIL derives strictly from operator-
observable API/CLI surfaces: CROSS-001 asserts the finalized score via prism
GET /v1/submissions/{id}; CROSS-002 asserts via GET /v1/workers/units that the
submitter's own worker never became a replica of its unit; CROSS-009 diffs
GET /v1/workers against parsed 'base worker status' (ids/owners/providers/
statuses/fault counts). Line-buffer + basicConfig mission_prism/mission_worker
so drill logs survive SIGTERM teardown; use a random submission nonce per run so
legacy_smoke (and the drills) are re-runnable against a reused --workdir.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft PR for the PRISM Compute Plane mission (worker plane behind feature flags).
This branch delivers the prism-side milestone
prism-proof. First landed:ExecutionProof emission (feature execution-proof-emission) — architecture.md §3.4/§3.5
src/prism_challenge/proof.py: ExecutionProof v1 envelope (schema-compatible with the base worker plane), canonical manifest hashing (sha256 overjson.dumps(manifest, sort_keys=True, indent=2)== the exact on-disk bytes ofprism_run_manifest.v2.json), tier computation from injected provider env (tier 0 default; tier 1 needs BOTH image digest AND pod metadata; tier 2 only with a non-null attestation payload), and an sr25519 worker signature over the pinned messagesha256(f'{manifest_sha256}:{unit_id}')(identical to the base agent).worker_planeconfig block (default OFF → legacy behavior byte-for-byte).Fulfills VAL-PRISM-001/002/003/004/005/006/008.
Gate: ruff + mypy clean;
pytest -m "not distributed_gloo"816 passed, coverage 83.21%.