Add MetaInstance declarative layer

[AlexCheema]

Motivation

Users currently manage instances directly, which means if a node disconnects or connections break, the instance dies and nothing recreates it. MetaInstance is a declarative primitive: "ensure an instance matching these parameters always exists." The reconciler watches for unhealthy or missing backing instances and re-places them automatically.

Changes

• MetaInstance type (meta_instance.py): declarative constraint with model_id, min_nodes, optional node_ids, and sharding
• Reconciler (reconcile.py): find_unsatisfied_meta_instances checks which MetaInstances lack a healthy backing instance, try_place_for_meta_instance creates one
• Master loop (main.py): periodically reconciles unsatisfied MetaInstances; immediate placement on CreateMetaInstance command
• API (api.py): create_meta_instance / delete_meta_instance / GET /meta_instances endpoints; delete cascades to backing instances with task cancellation
• Binding via meta_instance_id on Instance (instances.py): no separate binding event or backing map — the instance carries its parent MetaInstance ID directly, eliminating race conditions in the reconciler
• Dashboard: sidebar shows MetaInstances with their backing instance status; orphan instances (created directly) still shown separately
• Tests: constraint matching, connection health, unsatisfied detection, exclusive binding, cascade delete with task cancellation

Recent improvements

• fix: cancel active tasks on cascade delete — DeleteMetaInstance now emits TaskStatusUpdated(Cancelled) for any Pending/Running tasks on backing instances before emitting InstanceDeleted. Previously, cascade-deleting backing instances left orphaned task references in state.
• Lifecycle logging — added logger.info/logger.warning for: CreateMetaInstance (model, min_nodes, sharding), DeleteMetaInstance (with cascade count), reconciler placement success/failure, and retry decisions with attempt counts in InstanceHealthReconciler.
• GET /meta_instances endpoint — lists all meta-instances without needing to fetch full state.
• 2 regression tests — test_cascade_delete_cancels_active_tasks and test_cascade_delete_skips_completed_tasks verify the cascade-delete event sequence.

Why It Works

Putting meta_instance_id on BaseInstance makes binding inherent to instance creation. When the reconciler creates an instance for a MetaInstance, it tags it via model_copy. When the instance is deleted, the binding disappears with it. This avoids the two bugs that a separate binding mechanism would introduce:

1. Stale exclusion sets — the reconciler loop can't accidentally bind two MetaInstances to the same instance
2. Delete ordering race — no window between deleting an instance and its binding where the reconciler could re-place

Test Plan

Manual Testing

• Created MetaInstance via dashboard, verified instance placed
• Verified delete cascades (deleting MetaInstance removes backing instance)
• Verified orphan instances still work independently

Automated Testing

• 30 tests in test_meta_instance_edge_cases.py: lifecycle, retry logic, error handling, concurrent operations, cascade delete with task cancellation
• 24 tests in test_reconcile.py: constraint matching, connection health (single/multi-node, edge removal, IP changes), unsatisfied detection, exclusive binding, idempotency
• All 261 tests pass
• basedpyright 0 errors, ruff clean, dashboard builds

[AlexCheema]

Testing scenarios needed before merging

• Disconnect Ethernet with a Ring instance running — verify MetaInstance reconciler detects unhealthy connections and re-places
• Disconnect Thunderbolt 5 with an RDMA instance running — verify same recovery behavior
• Kill a node that's part of an Instance — verify node timeout triggers instance deletion and MetaInstance re-places on remaining nodes
• Delete a MetaInstance from the dashboard — verify backing instance is cascade-deleted
• Create multiple MetaInstances for the same model — verify each gets its own backing instance (exclusive binding)
• Create an orphan instance directly via API — verify it works independently and isn't affected by MetaInstance lifecycle

[AlexCheema]

Future work: placement preferences

MetaInstance currently places with no optimization preference. A natural next step is letting users specify a placement preference, e.g.:

• Highest interactivity — maximize tokens/sec per request (fewer nodes, lower latency)
• Highest throughput — maximize total tokens/sec across concurrent requests (more sharding, more parallelism)

These are different points on a throughput vs. interactivity Pareto curve. The placer would use the preference to score candidate placements differently rather than just picking the first valid one.

[AlexCheema]

JACCL RDMA Error Warning Banner

Added a dashboard warning that detects [jaccl] errors in MetaInstance failure messages. These errors indicate a problem with the experimental RDMA driver in macOS — the only fix is restarting the affected machine.

What it does:

• Scans metaInstances for lastFailureError containing [jaccl]
• Shows a red dismissible alert banner at the top-left of the topology view
• Hover tooltip explains the issue and tells the user to restart
• Re-appears if a new jaccl error arrives after dismissal

Banner:

Tooltip on hover:

[AlexCheema]

Deep Review of PR 1447: MetaInstance Layer

Reviewed: all 32 commits, ~2459 additions / ~228 deletions across 25+ files
Edge-case tests: 25 new tests pushed in src/exo/master/tests/test_meta_instance_edge_cases.py (all passing)
Full test suite: 256 passed, 1 skipped, 97 deselected (pre-existing slow tests)
Pre-commit checks: basedpyright 0 errors, ruff clean, nix fmt clean

---

Summary

The PR adds a MetaInstance declarative constraint layer that ensures a model instance matching given parameters always exists. When the backing instance fails, the system automatically retries placement up to MAX_INSTANCE_RETRIES (3). The implementation includes a clean ProcessManager protocol with three reconcilers (InstanceHealth, MetaInstance, NodeTimeout), new event types, worker-side retry coordination, and JACCL SideChannel FIFO relay.

Recommendation: Merge (with minor suggestions below)

Architecture is clean — pure reconciliation functions in reconcile.py, side-effectful orchestration in process managers, event application in apply.py. Follows existing patterns well. Test coverage is solid (43 existing + 25 new edge-case tests).

---

Issues Found

1. Race condition in delete_meta_instance (Medium-High)

In src/exo/master/api.py, the delete handler sends a DeleteMetaInstance command, then reads self.state() to find backing instances for cascade deletion. Since commands are processed asynchronously, state may be stale — could leave orphaned instances.

Suggestion: Read state before sending the delete command, or make cascade deletion part of the event handler.

2. apply_instance_retrying silently drops events for missing instances (Low-Medium)

In src/exo/shared/apply.py, when InstanceRetrying references a non-existent instance, the handler returns early without incrementing the MetaInstance failure counter. This is likely intentional (InstanceDeleted handles counting instead), but is undocumented and was confusing during review.

Suggestion: Add a brief comment explaining this design choice.

3. Reconcile loop runs ModelCard.load() async I/O (Medium)

try_place_for_meta_instance() calls ModelCard.load() inside the 1-second reconcile loop. If slow or failing, this blocks all reconciliation (health checks, node timeouts, other meta-instances).

Suggestion: Consider a timeout or running placement attempts outside the main reconcile cycle.

Minor Notes

• MAX_INSTANCE_RETRIES is hardcoded to 3 — works for now, could be configurable later
• Removed use_default validator from PlaceInstanceParams — intentional per commit message, minor breaking API change
• RDMA placement now properly raises ValueError instead of silently falling through — good fix
• No exponential backoff on retries (3 rapid attempts at 1s intervals for persistent failures)

---

Edge-Case Tests Added (25 tests)

Category	Tests
Lifecycle	create/delete roundtrip, frozen model, deletion removes from state
Retry logic	counter increments through cycle, max retries → deletion, resets on success
Error handling	retrying for missing instance, placement failure records error, double-delete idempotent
Backward compat	instances without meta_instance_id, legacy placement, state serialization
Concurrent	multiple meta-instances for same model, deleting one doesn't affect others
Constraints	node_ids subset matching, min_nodes enforcement, binding vs constraint semantics

🤖 Generated with Claude Code

[AlexCheema]

Full Summary of PR #1447 — Add MetaInstance Declarative Layer

Diff: 31 files changed, +3,241 / -238 lines

---

What is MetaInstance?

A declarative primitive: "ensure an instance matching these parameters always exists." If a node disconnects or connections break, the reconciler automatically re-creates the backing instance. Previously, users managed instances directly and dead instances stayed dead.

---

Core Changes (Python — 20 files)

New types and models:

• src/exo/shared/types/meta_instance.py — MetaInstance frozen Pydantic model with model_id, sharding, instance_meta, min_nodes, optional node_ids, failure tracking (consecutive_failures, last_failure_error, placement_error)
• src/exo/shared/types/common.py — Added MetaInstanceId type
• src/exo/shared/types/worker/instances.py — Added optional meta_instance_id field on BaseInstance, binding instances to their parent MetaInstance

Events and commands:

• src/exo/shared/types/events.py — New events: MetaInstanceCreated, MetaInstanceDeleted, MetaInstancePlacementFailed, InstanceRetrying
• src/exo/shared/types/commands.py — New commands: CreateMetaInstance, DeleteMetaInstance
• src/exo/shared/types/api.py — New API request/response types for meta-instance endpoints
• src/exo/shared/types/state.py — Added meta_instances dict to State

Event sourcing:

• src/exo/shared/apply.py — New apply functions for all meta-instance events; added explanatory comment on apply_instance_retrying documenting why it returns early for missing instances (avoids double-counting failures)

Reconciliation:

• src/exo/master/reconcile.py — find_unsatisfied_meta_instances() checks which MetaInstances lack a healthy backing instance; try_place_for_meta_instance() creates one using existing placement logic
• src/exo/master/process_managers/meta_instance.py — MetaInstanceReconciler runs in the master loop with 10s timeout and error handling for ModelCard.load(), emitting MetaInstancePlacementFailed events with dedup
• src/exo/master/process_managers/instance_health.py — InstanceHealthReconciler (extracted from inline code); MAX_INSTANCE_RETRIES = 3 retry logic for failed instances
• src/exo/master/process_managers/node_timeout.py — NodeTimeoutReconciler (extracted)
• src/exo/master/process_managers/__init__.py — Package init

Master and worker:

• src/exo/master/main.py — Reconcile loop runs 3 process managers every 1s; handles CreateMetaInstance/DeleteMetaInstance commands; cascade-delete removes backing instances
• src/exo/master/api.py — /create_meta_instance and /delete_meta_instance API endpoints
• src/exo/worker/plan.py — _create_runner now takes all_runners param to check for terminal peer runners before creating (prevents races during retry)

Other:

• src/exo/master/placement.py, src/exo/master/placement_utils.py — Minor refactors for reuse by meta-instance placement
• src/exo/main.py — Wires up new components
• src/exo/download/coordinator.py — Minor adjustment
• pyproject.toml — Test config update

---

Dashboard Changes (Svelte — 4 files)

• dashboard/src/routes/+page.svelte —
  • Shows MetaInstances in sidebar with backing instance status (placing, healthy, error states)
  • Bug fix: $derived(fn) → $derived.by(fn) for unifiedDisplayItems (was returning the function itself, not its result)
  • Bug fix: Removed tautological lastError ? ... : null check (always truthy when failures > 0)
  • getMetaInstancePlacingStatus() helper for UI state derivation
• dashboard/src/lib/stores/app.svelte.ts — MetaInstanceData interface, metaInstances reactive store
• dashboard/src/lib/components/ChatSidebar.svelte — Removed MlxIbvInstance references (consolidated to MlxJacclInstance)
• dashboard/src/lib/components/ModelCard.svelte — Same MlxIbv cleanup

---

Tests (3 files, ~28 new tests)

• src/exo/master/tests/test_reconcile.py — 24 new tests: constraint matching, connection health (single/multi-node, edge removal, IP changes), unsatisfied detection, exclusive binding, idempotency
• src/exo/master/tests/test_meta_instance_edge_cases.py — 28 tests: retry/failure flows, cascade delete, placement error dedup, ModelCard.load timeout/error handling
• src/exo/master/tests/test_placement_utils.py — Placement utility tests

---

Design Decisions

1. Binding via meta_instance_id on Instance — No separate binding event or backing map. The instance carries its parent MetaInstance ID directly, eliminating two classes of race conditions (stale exclusion sets, delete ordering races).
2. Process manager extraction — Instance health, node timeout, and meta-instance reconciliation are separate @final classes with a reconcile(state) -> Sequence[Event] interface.
3. ModelCard.load timeout — 10s anyio.fail_after prevents a slow/failing model card load from blocking the entire reconcile loop. Errors emit MetaInstancePlacementFailed with dedup against current state.
4. Retry strategy — Failed instances retry up to 3 times (MAX_INSTANCE_RETRIES), then get deleted. consecutive_failures and last_failure_error are always set together in apply.

---

Merge Conflict Resolution

The merge with origin/main resolved conflicts between this feature and main's new task cancellation feature:

• commands.py — Both CreateMetaInstance/DeleteMetaInstance and TaskCancelled
• plan.py — Added _cancel_tasks from main, kept all_runners param from this branch
• runner_supervisor.py — Merged _cancel_sender/cancelled fields with JACCL pipe relay fields
• reconcile.py — Pass empty tasks dict to get_transition_events (new param from main)

[AlexCheema]

Latest improvements (commit 3df896d)

Bug fix: cancel active tasks on meta-instance cascade delete

DeleteMetaInstance now emits TaskStatusUpdated(Cancelled) for any Pending/Running tasks on backing instances before emitting InstanceDeleted. Previously, cascade-deleting backing instances left orphaned task references in state — matching the pattern already used by get_transition_events() in placement.py.

Files: src/exo/master/main.py

Lifecycle logging

Added structured logging for meta-instance operations to aid debugging:

• main.py — logs CreateMetaInstance (model, min_nodes, sharding) and DeleteMetaInstance (with cascade instance count)
• process_managers/meta_instance.py — logs successful placement and placement failures
• process_managers/instance_health.py — logs retry attempts with (attempt N/3) and when retry limit is exceeded

GET /meta_instances API endpoint

Added GET /meta_instances to list all meta-instances directly, without needing to fetch full cluster state.

File: src/exo/master/api.py

Regression tests

2 new tests in test_meta_instance_edge_cases.py (30 total):

• test_cascade_delete_cancels_active_tasks — verifies the full event sequence (MetaInstanceDeleted → TaskStatusUpdated(Cancelled) → InstanceDeleted) correctly updates state
• test_cascade_delete_skips_completed_tasks — verifies only Pending/Running tasks are targeted, not completed ones

Pre-commit checks

All passing: basedpyright 0 errors, ruff clean, nix fmt clean, 261 tests passed (1 skipped, 97 deselected slow).

🤖 Generated with Claude Code

[AlexCheema]

Code Review: PR #1447 — MetaInstance Declarative Layer

Overall Assessment

Well-architected addition that introduces a declarative MetaInstance abstraction over the existing imperative instance management. The reconciliation pattern is clean, the retry logic is sound, and the test coverage is thorough (778 lines of edge-case tests). This is a significant architectural improvement.

Strengths

1. ProcessManager Protocol (process_managers/__init__.py): Clean, composable interface. @runtime_checkable is a nice touch for validation. The three reconcilers (InstanceHealth, NodeTimeout, MetaInstance) have clear single responsibilities.

2. _apply_and_broadcast eliminates the loopback processor hack. Centralizing event indexing, state mutation, persistence, and broadcast in one method is a major improvement. The docstring correctly notes Python's cooperative scheduling guarantees.

3. instance_runners_failed (reconcile.py): The logic is exactly right — requires ALL runners to be terminal AND at least one RunnerFailed. Correctly returns (False, None) when runners haven't reported yet (still starting) or when all are gracefully shut down. The node-identity-aware error messages are a nice UX touch.

4. find_unsatisfied_meta_instances correctly checks both meta_instance_id binding AND instance_connections_healthy, not just one or the other. A MetaInstance with a backing instance that has a broken connection will be detected.

5. Cascade delete (_command_processor, DeleteMetaInstance case): Correctly cancels active tasks before deleting the backing instance. The TaskStatus.Cancelled events are emitted before InstanceDeleted, ensuring proper cleanup ordering.

6. Comprehensive edge-case tests (test_meta_instance_edge_cases.py): Tests cover frozen model validation, create/delete roundtrips, nonexistent-ID safety, duplicate MetaInstances, retry counter resets, full retry cycles, and more.

Issues

7. try_place_for_meta_instance doesn't intersect node_ids with live topology (reconcile.py ~line 227): If a meta-instance pins node_ids=["node-a", "node-b"] and node-b goes down, placement will fail because node-b isn't in the topology. The placement could be made more resilient by intersecting meta_instance.node_ids with topology.list_nodes() and only requiring alive nodes. (Note: the other branch — #1484 — appears to have this fix with alive = set(meta_instance.node_ids) & live_nodes.)

8. Reconcile loop at 1-second interval (master/main.py): The previous _plan loop ran every 10 seconds. Now _reconcile runs every 1 second, and MetaInstanceReconciler calls ModelCard.load() for every unsatisfied meta-instance on each cycle. If model card loading hits the network (HuggingFace API), this could generate excessive requests when placement persistently fails. Consider:

   • Caching ModelCard.load() results
   • Adding a timeout to ModelCard.load() (prevent one slow model lookup from blocking the entire reconcile loop)
   • Exponential backoff for meta-instances with placement_error set

9. Placement strategy change (placement.py, placement_utils.py): get_smallest_cycles → get_largest_cycles changes the default from "minimize nodes used" to "maximize nodes used". This is a significant behavior change — users who previously got 2-node placements will now get 4-node placements. Should be documented and justified (performance? memory pressure?).

10. _reconcile manager ordering matters (master/main.py): Managers run sequentially with state updated between each. InstanceHealthReconciler runs before MetaInstanceReconciler, which is correct (delete broken instances before trying to re-place). But this ordering dependency is implicit. Consider adding a comment.

11. Potential double-place race in CreateMetaInstance handler (master/main.py): The command handler does an immediate placement attempt after _apply_and_broadcast(MetaInstanceCreated). It re-checks find_unsatisfied_meta_instances to avoid racing with the reconciler, which is good. However, between the await ModelCard.load() and the re-check, the reconciler could also be placing — the re-check mitigates but doesn't fully eliminate the race since the reconciler could be mid-placement (after find_unsatisfied but before emitting InstanceCreated).

Minor

12. PlacementResult as NamedTuple (reconcile.py): Rest of the codebase uses frozen Pydantic models or dataclasses. Minor inconsistency.

13. Magic number: MAX_INSTANCE_RETRIES = 3 in instance_health.py — consider making this configurable or at least documenting why 3.

Verdict

Strong architectural improvement. The declarative layer, reconciliation pattern, and retry logic are well-designed. The main concerns are the placement strategy change (largest vs smallest), the 1-second reconcile interval with potentially slow ModelCard.load(), and the node_ids/topology intersection issue. Would approve after addressing #7 (topology intersection) and #9 (placement strategy documentation).

🤖 Generated with Claude Code