From d14e8486788c21e4830af0769deb79c86da6c628 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Sat, 16 May 2026 11:31:19 +0100 Subject: [PATCH 01/37] docs: add pull-wake health check design spec Co-Authored-By: Claude Opus 4.6 --- ...026-05-16-pull-wake-health-check-design.md | 180 ++++++++++++++++++ 1 file changed, 180 insertions(+) create mode 100644 packages/agents-server/docs/superpowers/specs/2026-05-16-pull-wake-health-check-design.md diff --git a/packages/agents-server/docs/superpowers/specs/2026-05-16-pull-wake-health-check-design.md b/packages/agents-server/docs/superpowers/specs/2026-05-16-pull-wake-health-check-design.md new file mode 100644 index 0000000000..a5e142b000 --- /dev/null +++ b/packages/agents-server/docs/superpowers/specs/2026-05-16-pull-wake-health-check-design.md @@ -0,0 +1,180 @@ +# Pull-Wake Runner Health Check + +## Problem + +The pull-wake dispatch system is unreliable and there's no way to diagnose what's going wrong. We need a single endpoint that returns comprehensive diagnostic info about a runner's state — covering both the server-side DB state and the client-side connection state. + +## Design + +### Layer 1: Client-Side Diagnostics (PullWakeRunner) + +Add internal state tracking to `createPullWakeRunner` in `packages/agents-runtime/src/pull-wake-runner.ts`. + +**Tracked state:** + +| Field | Type | Description | +| ------------------------ | ------------------------------------- | -------------------------------------------------- | +| `started_at` | ISO string | When `start()` was called | +| `stream_connected` | boolean | Whether the stream iterator is actively yielding | +| `stream_connected_since` | ISO string | When the current stream connection was established | +| `reconnect_count` | number | Total reconnection attempts since start | +| `last_error` | string | Most recent error message | +| `last_error_at` | ISO string | When the last error occurred | +| `last_heartbeat_at` | ISO string | When the last heartbeat was sent | +| `last_heartbeat_ok` | boolean | Whether the last heartbeat succeeded | +| `last_claim_at` | ISO string | When the last claim attempt was made | +| `last_claim_result` | `"claimed"` / `"no_work"` / `"error"` | Result of the last claim | +| `last_dispatch_at` | ISO string | When the last wake was dispatched to the runtime | +| `events_received` | number | Total wake events received from the stream | +| `claims_succeeded` | number | Total successful claims | +| `claims_skipped` | number | Claims that returned no work / already claimed | +| `claims_failed` | number | Claims that errored | + +**New interface method:** + +```ts +export interface PullWakeRunner { + // ... existing + getHealth: () => PullWakeRunnerHealth +} + +export interface PullWakeRunnerHealth { + running: boolean + offset: string | undefined + started_at: string | null + stream_connected: boolean + stream_connected_since: string | null + reconnect_count: number + last_error: string | null + last_error_at: string | null + last_heartbeat_at: string | null + last_heartbeat_ok: boolean + last_claim_at: string | null + last_claim_result: 'claimed' | 'no_work' | 'error' | null + last_dispatch_at: string | null + events_received: number + claims_succeeded: number + claims_skipped: number + claims_failed: number +} +``` + +**Reporting to server:** The heartbeat POST body already sends `lease_ms` and `wake_stream_offset`. Extend it with a `diagnostics` field containing the tracked state above. The server persists this in the runners table. + +### Layer 2: Server-Side Storage + +Add a `diagnostics` JSONB column to the `runners` table via migration `0007_runner_diagnostics.sql`. + +The `heartbeatRunner` method in `PostgresRegistry` stores the diagnostics payload from the heartbeat request. + +The `ElectricAgentsRunner` type gains an optional `diagnostics` field. + +### Layer 3: Health Endpoint + +**Route:** `GET /_electric/runners/:id/health` + +Added to `runners-router.ts` alongside the existing runner CRUD routes. Same auth as `getRunner` — owner must match authenticated principal. + +**Response shape:** + +```json +{ + "runner": { + "id": "desktop-abc123", + "admin_status": "enabled", + "liveness_status": "online", + "lease_expires_at": "2026-05-16T12:00:30Z", + "lease_remaining_ms": 12345, + "wake_stream": "/runners/desktop-abc123/wake", + "wake_stream_offset": "0_3", + "last_seen_at": "2026-05-16T12:00:00Z", + "created_at": "2026-05-16T11:00:00Z" + }, + "client": { + "started_at": "2026-05-16T11:00:01Z", + "stream_connected": true, + "stream_connected_since": "2026-05-16T11:00:02Z", + "reconnect_count": 0, + "last_error": null, + "last_error_at": null, + "last_heartbeat_at": "2026-05-16T12:00:00Z", + "last_heartbeat_ok": true, + "last_claim_at": "2026-05-16T11:55:00Z", + "last_claim_result": "claimed", + "last_dispatch_at": "2026-05-16T11:55:01Z", + "events_received": 14, + "claims_succeeded": 10, + "claims_skipped": 3, + "claims_failed": 1 + }, + "claims": { + "active_count": 1, + "active": [ + { + "consumer_id": "wake-001", + "epoch": 3, + "entity_url": "/entities/coder/session-42", + "stream_path": "/coder/session-42/main", + "claimed_at": "2026-05-16T11:55:00Z", + "last_heartbeat_at": "2026-05-16T12:00:00Z", + "lease_expires_at": "2026-05-16T12:00:30Z" + } + ] + }, + "dispatch": { + "entities_with_active_claim": 1, + "entities_with_outstanding_wake": 0, + "entities_with_pending_work": 2 + }, + "health": { + "status": "healthy", + "issues": [] + } +} +``` + +**Health status derivation rules:** + +| Condition | Status | +| ----------------------------------------------- | ----------- | +| Lease expired (liveness_lease_expires_at < now) | `unhealthy` | +| admin_status is `disabled` | `unhealthy` | +| Client reports stream_connected = false | `degraded` | +| Client reports last_heartbeat_ok = false | `degraded` | +| reconnect_count > 5 (since last check) | `degraded` | +| No client diagnostics available | `degraded` | +| Otherwise | `healthy` | + +Each failing condition adds a human-readable string to the `issues` array. + +### Data Sources for the Endpoint + +| Section | Source | +| ---------- | --------------------------------------------------------------------- | +| `runner` | `runners` table row | +| `client` | `runners.diagnostics` JSONB (from last heartbeat) | +| `claims` | `consumer_claims` table where `runner_id = :id AND status = 'active'` | +| `dispatch` | `entity_dispatch_state` table where `active_runner_id = :id` | +| `health` | Derived from above | + +### Files Changed + +**New:** + +- `packages/agents-server/drizzle/0007_runner_diagnostics.sql` — adds `diagnostics` JSONB column to runners table + +**Modified:** + +- `packages/agents-runtime/src/pull-wake-runner.ts` — add diagnostics tracking, `getHealth()` method, report diagnostics in heartbeat +- `packages/agents-runtime/src/types.ts` — export `PullWakeRunnerHealth` type if needed +- `packages/agents-server/src/db/schema.ts` — add `diagnostics` column to `runners` table definition +- `packages/agents-server/src/entity-registry.ts` — extend `heartbeatRunner` to store diagnostics; add `getActiveClaimsForRunner` and `getDispatchStatsForRunner` queries +- `packages/agents-server/src/electric-agents-types.ts` — add `diagnostics` to `ElectricAgentsRunner`, add health response types +- `packages/agents-server/src/routing/runners-router.ts` — add `GET /:id/health` route and handler +- `packages/agents-server/src/routing/runners-router.ts` — extend heartbeat body schema with optional `diagnostics` + +### Testing + +- Unit test for health status derivation logic (pure function) +- Unit test for `getHealth()` on the PullWakeRunner +- Integration test extending the existing `horton-pull-wake-e2e.test.ts` to call the health endpoint after dispatch and verify diagnostics are populated From afe87f2d629d853587cb848992ab1e6176390944 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Sat, 16 May 2026 11:42:46 +0100 Subject: [PATCH 02/37] docs: update health check spec with principal rename and shape sync Co-Authored-By: Claude Opus 4.6 --- ...026-05-16-pull-wake-health-check-design.md | 50 ++++++++++++++++--- 1 file changed, 42 insertions(+), 8 deletions(-) diff --git a/packages/agents-server/docs/superpowers/specs/2026-05-16-pull-wake-health-check-design.md b/packages/agents-server/docs/superpowers/specs/2026-05-16-pull-wake-health-check-design.md index a5e142b000..e9d0373597 100644 --- a/packages/agents-server/docs/superpowers/specs/2026-05-16-pull-wake-health-check-design.md +++ b/packages/agents-server/docs/superpowers/specs/2026-05-16-pull-wake-health-check-design.md @@ -2,7 +2,12 @@ ## Problem -The pull-wake dispatch system is unreliable and there's no way to diagnose what's going wrong. We need a single endpoint that returns comprehensive diagnostic info about a runner's state — covering both the server-side DB state and the client-side connection state. +The pull-wake dispatch system is unreliable and there's no way to diagnose what's going wrong. We need: + +1. A **health check endpoint** (`GET /_electric/runners/:id/health`) for deep debugging — curl it to see comprehensive diagnostics about a runner's dispatch pipeline. +2. **Rich runner state in Postgres** so that apps can sync the `runners` table via an Electric Shape and show runner status on any device (e.g. see your laptop runner's status from your phone). + +The `diagnostics` JSONB column on the `runners` table serves both purposes: the health endpoint reads it for the detailed response, and Shape sync delivers it reactively to any connected client. ## Design @@ -69,6 +74,35 @@ The `heartbeatRunner` method in `PostgresRegistry` stores the diagnostics payloa The `ElectricAgentsRunner` type gains an optional `diagnostics` field. +### Principal-Based Ownership Rename + +The existing `owner_user_id` column and field name is a misnomer — principals aren't limited to users. They include `agent:`, `service:`, and `system:` actors. The canonical identifier is the principal URL (e.g. `/principal/user%3Aalice`), not the key (`user:alice`). + +As part of this work, rename across the runners system: + +- **DB column**: `owner_user_id` → `owner_principal` (migration) +- **Drizzle schema**: `ownerUserId` → `ownerPrincipal` +- **Types**: `ElectricAgentsRunner.owner_user_id` → `ElectricAgentsRunner.owner_principal` +- **API body/query**: `owner_user_id` → `owner_principal` +- **Auth checks**: compare against `ctx.principal.url` instead of `ctx.principal.key` +- **Registry methods**: `ownerUserId` param → `ownerPrincipal` + +The stored value is the principal URL (`/principal/user%3Aalice`), which is the primary identifier for everything in the system. + +### Multi-Device Runner Status via Shape Sync + +Apps sync the `runners` table via an Electric Shape scoped to `owner_principal`. This gives reactive runner status on any device without polling. The table row provides everything a UI needs: + +- **Online/offline**: derive from `liveness_lease_expires_at` vs current time +- **Admin status**: `admin_status` column (`enabled`/`disabled`) +- **Owner**: `owner_principal` — the principal URL, for Shape scoping (`WHERE owner_principal = :my_principal_url`) +- **Stream connected**: `diagnostics.stream_connected` +- **Last error**: `diagnostics.last_error` + `diagnostics.last_error_at` +- **Activity**: `diagnostics.last_claim_at`, `diagnostics.last_dispatch_at` +- **Counters**: `diagnostics.events_received`, `diagnostics.claims_succeeded`, etc. + +The health endpoint aggregates additional server-side state (active claims, dispatch stats) that isn't in the runners table — it's the "explain what's happening right now" view for debugging. The Shape is the "show me my runners" view for apps. + ### Layer 3: Health Endpoint **Route:** `GET /_electric/runners/:id/health` @@ -161,17 +195,17 @@ Each failing condition adds a human-readable string to the `issues` array. **New:** -- `packages/agents-server/drizzle/0007_runner_diagnostics.sql` — adds `diagnostics` JSONB column to runners table +- `packages/agents-server/drizzle/0007_runner_diagnostics_and_principal.sql` — adds `diagnostics` JSONB column, renames `owner_user_id` → `owner_principal` on runners table **Modified:** - `packages/agents-runtime/src/pull-wake-runner.ts` — add diagnostics tracking, `getHealth()` method, report diagnostics in heartbeat -- `packages/agents-runtime/src/types.ts` — export `PullWakeRunnerHealth` type if needed -- `packages/agents-server/src/db/schema.ts` — add `diagnostics` column to `runners` table definition -- `packages/agents-server/src/entity-registry.ts` — extend `heartbeatRunner` to store diagnostics; add `getActiveClaimsForRunner` and `getDispatchStatsForRunner` queries -- `packages/agents-server/src/electric-agents-types.ts` — add `diagnostics` to `ElectricAgentsRunner`, add health response types -- `packages/agents-server/src/routing/runners-router.ts` — add `GET /:id/health` route and handler -- `packages/agents-server/src/routing/runners-router.ts` — extend heartbeat body schema with optional `diagnostics` +- `packages/agents-server/src/db/schema.ts` — rename `ownerUserId` → `ownerPrincipal`, add `diagnostics` column +- `packages/agents-server/src/entity-registry.ts` — rename `ownerUserId` params → `ownerPrincipal`; extend `heartbeatRunner` to store diagnostics; add `getActiveClaimsForRunner` and `getDispatchStatsForRunner` queries +- `packages/agents-server/src/electric-agents-types.ts` — rename `owner_user_id` → `owner_principal` on `ElectricAgentsRunner`, add `diagnostics` field, add health response types +- `packages/agents-server/src/routing/runners-router.ts` — rename all `owner_user_id` references → `owner_principal`; switch auth checks to use `ctx.principal.url`; add `GET /:id/health` route; extend heartbeat body schema with optional `diagnostics` +- `packages/agents/src/server.ts` — update `BuiltinAgentsServer` registration to use `ownerPrincipal` +- `packages/agents-desktop/src/main.ts` — update runner registration to use `owner_principal` ### Testing From 6c9979e5ad98bba4bc2bc45b180dadb59bb8816e Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Sat, 16 May 2026 11:49:34 +0100 Subject: [PATCH 03/37] docs: add pull-wake health check implementation plan Co-Authored-By: Claude Opus 4.6 --- .../2026-05-16-pull-wake-health-check.md | 1574 +++++++++++++++++ 1 file changed, 1574 insertions(+) create mode 100644 packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md diff --git a/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md b/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md new file mode 100644 index 0000000000..ead050b551 --- /dev/null +++ b/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md @@ -0,0 +1,1574 @@ +# Pull-Wake Runner Health Check Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add comprehensive diagnostics to the pull-wake runner system: client-side state tracking reported via heartbeats, server-side storage + aggregation, and a `GET /_electric/runners/:id/health` endpoint. Also rename `owner_user_id` → `owner_principal` throughout the runners system, storing principal URLs instead of keys. + +**Architecture:** Three layers — (1) `PullWakeRunner` tracks 16 diagnostic fields internally and reports them to the server in each heartbeat, (2) the server stores client diagnostics in a new `diagnostics` JSONB column on the `runners` table, (3) a new health endpoint aggregates runner state, client diagnostics, active claims, and dispatch stats into a single response with derived health status. + +**Tech Stack:** TypeScript, Drizzle ORM, itty-router, Vitest, PostgreSQL + +--- + +### Task 1: Migration — rename `owner_user_id` and add `diagnostics` column + +**Files:** + +- Create: `packages/agents-server/drizzle/0007_runner_diagnostics_and_principal.sql` + +- [ ] **Step 1: Write the migration SQL** + +```sql +ALTER TABLE runners RENAME COLUMN owner_user_id TO owner_principal; +--> statement-breakpoint +DROP INDEX IF EXISTS idx_runners_owner_user_id; +--> statement-breakpoint +CREATE INDEX idx_runners_owner_principal ON runners (tenant_id, owner_principal); +--> statement-breakpoint +ALTER TABLE runners ADD COLUMN diagnostics jsonb; +``` + +- [ ] **Step 2: Commit** + +```bash +git add packages/agents-server/drizzle/0007_runner_diagnostics_and_principal.sql +git commit -m "feat(agents-server): add migration for runner diagnostics and principal rename" +``` + +--- + +### Task 2: Update Drizzle schema and types for principal rename + diagnostics + +**Files:** + +- Modify: `packages/agents-server/src/db/schema.ts:104-144` +- Modify: `packages/agents-server/src/electric-agents-types.ts:99-136` + +- [ ] **Step 1: Update the `runners` table in Drizzle schema** + +In `packages/agents-server/src/db/schema.ts`, change the `runners` table definition: + +```ts +// In the runners column definitions (line 109): +// REPLACE: + ownerUserId: text(`owner_user_id`).notNull(), +// WITH: + ownerPrincipal: text(`owner_principal`).notNull(), + +// After livenessLeaseExpiresAt (line 118), ADD: + diagnostics: jsonb(`diagnostics`), + +// In the table constraints (line 129): +// REPLACE: + index(`idx_runners_owner_user_id`).on(table.tenantId, table.ownerUserId), +// WITH: + index(`idx_runners_owner_principal`).on(table.tenantId, table.ownerPrincipal), +``` + +- [ ] **Step 2: Update the `ElectricAgentsRunner` type** + +In `packages/agents-server/src/electric-agents-types.ts`, update the runner types: + +```ts +// In ElectricAgentsRunner (line 106-120): +// REPLACE: +export interface ElectricAgentsRunner { + id: string + owner_user_id: string + label: string + kind: RunnerKind + admin_status: RunnerAdminStatus + liveness?: RunnerLiveness + last_seen_at?: string + liveness_lease_expires_at?: string + active_claims?: Array + wake_stream: string + wake_stream_offset?: string + created_at: string + updated_at: string +} +// WITH: +export interface ElectricAgentsRunner { + id: string + owner_principal: string + label: string + kind: RunnerKind + admin_status: RunnerAdminStatus + liveness?: RunnerLiveness + last_seen_at?: string + liveness_lease_expires_at?: string + active_claims?: Array + wake_stream: string + wake_stream_offset?: string + diagnostics?: Record + created_at: string + updated_at: string +} + +// In RegisterRunnerRequest (line 122-129): +// REPLACE: +export interface RegisterRunnerRequest { + id: string + owner_user_id: string + label: string + kind?: RunnerKind + admin_status?: RunnerAdminStatus + wake_stream?: string +} +// WITH: +export interface RegisterRunnerRequest { + id: string + owner_principal: string + label: string + kind?: RunnerKind + admin_status?: RunnerAdminStatus + wake_stream?: string +} +``` + +- [ ] **Step 3: Add `RunnerHealthResponse` and `RunnerHealthStatus` types** + +Append to `packages/agents-server/src/electric-agents-types.ts`: + +```ts +export type RunnerHealthStatus = `healthy` | `degraded` | `unhealthy` + +export interface RunnerHealthResponse { + runner: { + id: string + admin_status: RunnerAdminStatus + liveness_status: RunnerLiveness | `expired` + lease_expires_at: string | null + lease_remaining_ms: number | null + wake_stream: string + wake_stream_offset: string | null + last_seen_at: string | null + created_at: string + } + client: Record | null + claims: { + active_count: number + active: Array<{ + consumer_id: string + epoch: number + entity_url: string + stream_path: string + claimed_at: string + last_heartbeat_at: string | null + lease_expires_at: string | null + }> + } + dispatch: { + entities_with_active_claim: number + entities_with_outstanding_wake: number + entities_with_pending_work: number + } + health: { + status: RunnerHealthStatus + issues: Array + } +} +``` + +- [ ] **Step 4: Commit** + +```bash +git add packages/agents-server/src/db/schema.ts packages/agents-server/src/electric-agents-types.ts +git commit -m "feat(agents-server): rename owner_user_id to owner_principal in schema and types, add diagnostics" +``` + +--- + +### Task 3: Update entity registry — principal rename, diagnostics storage, health queries + +**Files:** + +- Modify: `packages/agents-server/src/entity-registry.ts:74-81, 132-190, 193-217, 1148-1168` + +- [ ] **Step 1: Rename `RegisterRunnerInput.ownerUserId` → `ownerPrincipal`** + +In `packages/agents-server/src/entity-registry.ts` (line 74-81): + +```ts +// REPLACE: +export interface RegisterRunnerInput { + id: string + ownerUserId: string + label: string + kind?: RunnerKind + adminStatus?: RunnerAdminStatus + wakeStream?: string +} +// WITH: +export interface RegisterRunnerInput { + id: string + ownerPrincipal: string + label: string + kind?: RunnerKind + adminStatus?: RunnerAdminStatus + wakeStream?: string +} +``` + +- [ ] **Step 2: Add `diagnostics` to `HeartbeatRunnerInput`** + +In `packages/agents-server/src/entity-registry.ts` (line 83-89): + +```ts +// REPLACE: +export interface HeartbeatRunnerInput { + runnerId: string + heartbeatAt?: Date + livenessLeaseExpiresAt?: Date + leaseMs?: number + wakeStreamOffset?: string +} +// WITH: +export interface HeartbeatRunnerInput { + runnerId: string + heartbeatAt?: Date + livenessLeaseExpiresAt?: Date + leaseMs?: number + wakeStreamOffset?: string + diagnostics?: Record +} +``` + +- [ ] **Step 3: Update `createRunner` to use `ownerPrincipal`** + +In the `createRunner` method (line 132-167), replace all `ownerUserId` → `ownerPrincipal` references: + +```ts + async createRunner( + input: RegisterRunnerInput + ): Promise { + const now = new Date() + const wakeStream = input.wakeStream ?? runnerWakeStream(input.id) + + await this.db + .insert(runners) + .values({ + tenantId: this.tenantId, + id: input.id, + ownerPrincipal: input.ownerPrincipal, + label: input.label, + kind: input.kind ?? `local`, + adminStatus: input.adminStatus ?? `enabled`, + wakeStream, + updatedAt: now, + }) + .onConflictDoUpdate({ + target: [runners.tenantId, runners.id], + set: { + ownerPrincipal: input.ownerPrincipal, + label: input.label, + kind: input.kind ?? `local`, + adminStatus: input.adminStatus ?? `enabled`, + wakeStream, + updatedAt: now, + }, + }) + + const runner = await this.getRunner(input.id) + if (!runner) { + throw new Error(`Failed to read back runner "${input.id}"`) + } + return runner + } +``` + +- [ ] **Step 4: Update `listRunners` filter** + +In `listRunners` (line 178-191): + +```ts +// REPLACE: + async listRunners(filter?: { + ownerUserId?: string + }): Promise> { + const conditions = [eq(runners.tenantId, this.tenantId)] + if (filter?.ownerUserId) { + conditions.push(eq(runners.ownerUserId, filter.ownerUserId)) + } +// WITH: + async listRunners(filter?: { + ownerPrincipal?: string + }): Promise> { + const conditions = [eq(runners.tenantId, this.tenantId)] + if (filter?.ownerPrincipal) { + conditions.push(eq(runners.ownerPrincipal, filter.ownerPrincipal)) + } +``` + +- [ ] **Step 5: Update `heartbeatRunner` to store diagnostics** + +In `heartbeatRunner` (line 193-217): + +```ts + async heartbeatRunner( + input: HeartbeatRunnerInput + ): Promise { + const now = input.heartbeatAt ?? new Date() + const leaseExpiresAt = + input.livenessLeaseExpiresAt ?? + new Date(now.getTime() + (input.leaseMs ?? DEFAULT_RUNNER_LEASE_MS)) + + const rows = await this.db + .update(runners) + .set({ + lastSeenAt: now, + livenessLeaseExpiresAt: leaseExpiresAt, + ...(input.wakeStreamOffset !== undefined + ? { wakeStreamOffset: input.wakeStreamOffset } + : {}), + ...(input.diagnostics !== undefined + ? { diagnostics: input.diagnostics } + : {}), + updatedAt: now, + }) + .where( + and(eq(runners.tenantId, this.tenantId), eq(runners.id, input.runnerId)) + ) + .returning() + + return rows[0] ? this.rowToRunner(rows[0]) : null + } +``` + +- [ ] **Step 6: Add `getActiveClaimsForRunner` query** + +Add after `materializeReleasedClaim` (around line 367): + +```ts + async getActiveClaimsForRunner( + runnerId: string + ): Promise> { + const rows = await this.db + .select() + .from(consumerClaims) + .where( + and( + eq(consumerClaims.tenantId, this.tenantId), + eq(consumerClaims.runnerId, runnerId), + eq(consumerClaims.status, `active`) + ) + ) + return rows.map((row) => this.rowToConsumerClaim(row)) + } +``` + +- [ ] **Step 7: Add `getDispatchStatsForRunner` query** + +Add right after `getActiveClaimsForRunner`: + +```ts + async getDispatchStatsForRunner( + runnerId: string + ): Promise<{ + entities_with_active_claim: number + entities_with_outstanding_wake: number + entities_with_pending_work: number + }> { + const rows = await this.db + .select() + .from(entityDispatchState) + .where( + and( + eq(entityDispatchState.tenantId, this.tenantId), + eq(entityDispatchState.activeRunnerId, runnerId) + ) + ) + + let activeClaim = 0 + let outstandingWake = 0 + let pendingWork = 0 + for (const row of rows) { + if (row.activeConsumerId) activeClaim++ + if (row.outstandingWakeId && !row.activeConsumerId) outstandingWake++ + const pending = row.pendingSourceStreams as Array | null + if (pending && pending.length > 0) pendingWork++ + } + + return { + entities_with_active_claim: activeClaim, + entities_with_outstanding_wake: outstandingWake, + entities_with_pending_work: pendingWork, + } + } +``` + +- [ ] **Step 8: Update `rowToRunner` to include `owner_principal` and `diagnostics`** + +In `rowToRunner` (line 1148-1168): + +```ts +// REPLACE: + private rowToRunner(row: typeof runners.$inferSelect): ElectricAgentsRunner { + const now = Date.now() + const livenessExpiry = row.livenessLeaseExpiresAt?.getTime() + return { + id: row.id, + owner_user_id: row.ownerUserId, + label: row.label, + kind: assertRunnerKind(row.kind), + admin_status: assertRunnerAdminStatus(row.adminStatus), + liveness: + livenessExpiry !== undefined && livenessExpiry > now + ? `online` + : `offline`, + last_seen_at: row.lastSeenAt?.toISOString(), + liveness_lease_expires_at: row.livenessLeaseExpiresAt?.toISOString(), + wake_stream: row.wakeStream, + wake_stream_offset: row.wakeStreamOffset ?? undefined, + created_at: row.createdAt.toISOString(), + updated_at: row.updatedAt.toISOString(), + } + } +// WITH: + private rowToRunner(row: typeof runners.$inferSelect): ElectricAgentsRunner { + const now = Date.now() + const livenessExpiry = row.livenessLeaseExpiresAt?.getTime() + return { + id: row.id, + owner_principal: row.ownerPrincipal, + label: row.label, + kind: assertRunnerKind(row.kind), + admin_status: assertRunnerAdminStatus(row.adminStatus), + liveness: + livenessExpiry !== undefined && livenessExpiry > now + ? `online` + : `offline`, + last_seen_at: row.lastSeenAt?.toISOString(), + liveness_lease_expires_at: row.livenessLeaseExpiresAt?.toISOString(), + wake_stream: row.wakeStream, + wake_stream_offset: row.wakeStreamOffset ?? undefined, + diagnostics: (row.diagnostics as Record) ?? undefined, + created_at: row.createdAt.toISOString(), + updated_at: row.updatedAt.toISOString(), + } + } +``` + +- [ ] **Step 9: Commit** + +```bash +git add packages/agents-server/src/entity-registry.ts +git commit -m "feat(agents-server): update entity registry for principal rename, diagnostics, and health queries" +``` + +--- + +### Task 4: Update runners router — principal rename, diagnostics in heartbeat, health endpoint + +**Files:** + +- Modify: `packages/agents-server/src/routing/runners-router.ts` + +- [ ] **Step 1: Update the registration body schema** + +In `packages/agents-server/src/routing/runners-router.ts` (line 36-53): + +```ts +// REPLACE: +const registerRunnerBodySchema = Type.Object({ + id: Type.String(), + owner_user_id: Type.Optional(Type.String()), + label: Type.String(), + kind: Type.Optional( + Type.Union([ + Type.Literal(`local`), + Type.Literal(`cloud-worker`), + Type.Literal(`sandbox`), + Type.Literal(`ci`), + Type.Literal(`server`), + ]) + ), + admin_status: Type.Optional( + Type.Union([Type.Literal(`enabled`), Type.Literal(`disabled`)]) + ), + wake_stream: Type.Optional(Type.String()), +}) +// WITH: +const registerRunnerBodySchema = Type.Object({ + id: Type.String(), + owner_principal: Type.Optional(Type.String()), + label: Type.String(), + kind: Type.Optional( + Type.Union([ + Type.Literal(`local`), + Type.Literal(`cloud-worker`), + Type.Literal(`sandbox`), + Type.Literal(`ci`), + Type.Literal(`server`), + ]) + ), + admin_status: Type.Optional( + Type.Union([Type.Literal(`enabled`), Type.Literal(`disabled`)]) + ), + wake_stream: Type.Optional(Type.String()), +}) +``` + +- [ ] **Step 2: Add `diagnostics` to heartbeat body schema** + +In the `heartbeatBodySchema` (line 55-60): + +```ts +// REPLACE: +const heartbeatBodySchema = Type.Object({ + lease_ms: Type.Optional(Type.Number()), + wake_stream_offset: Type.Optional(Type.String()), + wakeStreamOffset: Type.Optional(Type.String()), + liveness_lease_expires_at: Type.Optional(Type.String()), +}) +// WITH: +const heartbeatBodySchema = Type.Object({ + lease_ms: Type.Optional(Type.Number()), + wake_stream_offset: Type.Optional(Type.String()), + wakeStreamOffset: Type.Optional(Type.String()), + liveness_lease_expires_at: Type.Optional(Type.String()), + diagnostics: Type.Optional(Type.Record(Type.String(), Type.Unknown())), +}) +``` + +- [ ] **Step 3: Add the health route** + +After the existing routes (line 90), add: + +```ts +runnersRouter.get(`/:id/health`, runnerHealth) +``` + +- [ ] **Step 4: Update `registerRunner` handler to use `owner_principal` and `ctx.principal.url`** + +In `registerRunner` (line 103-136): + +```ts +// REPLACE: +async function registerRunner( + request: RunnersRouteRequest, + ctx: TenantContext +): Promise { + const parsed = routeBody(request) + const ownerUserId = parsed.owner_user_id ?? ctx.principal?.key + if (!ownerUserId) { + throw new ElectricAgentsError( + ErrCodeInvalidRequest, + `owner_user_id is required when no authenticated user is present`, + 400 + ) + } + if (ctx.principal && ownerUserId !== ctx.principal.key) { + throw new ElectricAgentsError( + ErrCodeUnauthorized, + `owner_user_id must match the authenticated user`, + 403 + ) + } + + const runner = await ctx.entityManager.registry.createRunner({ + id: parsed.id, + ownerUserId, + label: parsed.label, + kind: parsed.kind, + adminStatus: parsed.admin_status, + wakeStream: parsed.wake_stream, + }) + await ctx.streamClient.ensure(runner.wake_stream, { + contentType: `application/json`, + }) + return json(runner, { status: 201 }) +} +// WITH: +async function registerRunner( + request: RunnersRouteRequest, + ctx: TenantContext +): Promise { + const parsed = routeBody(request) + const ownerPrincipal = parsed.owner_principal ?? ctx.principal?.url + if (!ownerPrincipal) { + throw new ElectricAgentsError( + ErrCodeInvalidRequest, + `owner_principal is required when no authenticated principal is present`, + 400 + ) + } + if (ctx.principal && ownerPrincipal !== ctx.principal.url) { + throw new ElectricAgentsError( + ErrCodeUnauthorized, + `owner_principal must match the authenticated principal`, + 403 + ) + } + + const runner = await ctx.entityManager.registry.createRunner({ + id: parsed.id, + ownerPrincipal, + label: parsed.label, + kind: parsed.kind, + adminStatus: parsed.admin_status, + wakeStream: parsed.wake_stream, + }) + await ctx.streamClient.ensure(runner.wake_stream, { + contentType: `application/json`, + }) + return json(runner, { status: 201 }) +} +``` + +- [ ] **Step 5: Update `listRunners` handler** + +In `listRunners` (line 138-154): + +```ts +// REPLACE: +async function listRunners( + request: RunnersRouteRequest, + ctx: TenantContext +): Promise { + const requestedOwner = firstQueryValue(request.query.owner_user_id) + if (ctx.principal && requestedOwner && requestedOwner !== ctx.principal.key) { + throw new ElectricAgentsError( + ErrCodeUnauthorized, + `owner_user_id must match the authenticated user`, + 403 + ) + } + const runners = await ctx.entityManager.registry.listRunners({ + ownerUserId: ctx.principal?.key ?? requestedOwner, + }) + return json(runners) +} +// WITH: +async function listRunners( + request: RunnersRouteRequest, + ctx: TenantContext +): Promise { + const requestedOwner = firstQueryValue(request.query.owner_principal) + if (ctx.principal && requestedOwner && requestedOwner !== ctx.principal.url) { + throw new ElectricAgentsError( + ErrCodeUnauthorized, + `owner_principal must match the authenticated principal`, + 403 + ) + } + const runners = await ctx.entityManager.registry.listRunners({ + ownerPrincipal: ctx.principal?.url ?? requestedOwner, + }) + return json(runners) +} +``` + +- [ ] **Step 6: Update heartbeat handler to pass diagnostics** + +In `heartbeat` (line 165-185), add `diagnostics` to the `heartbeatRunner` call: + +```ts +const runner = await ctx.entityManager.registry.heartbeatRunner({ + runnerId, + leaseMs: parsed.lease_ms, + wakeStreamOffset: parsed.wake_stream_offset ?? parsed.wakeStreamOffset, + livenessLeaseExpiresAt: parsed.liveness_lease_expires_at + ? new Date(parsed.liveness_lease_expires_at) + : undefined, + diagnostics: parsed.diagnostics, +}) +``` + +- [ ] **Step 7: Update `assertRunnerOwnerIfAuthenticated` to use `principal.url`** + +In `assertRunnerOwnerIfAuthenticated` (line 297-308): + +```ts +// REPLACE: +function assertRunnerOwnerIfAuthenticated( + ctx: TenantContext, + ownerUserId: string +): void { + if (!ctx.principal) return + if (ownerUserId === ctx.principal.key) return + throw new ElectricAgentsError( + ErrCodeUnauthorized, + `Runner access requires the authenticated owner`, + 403 + ) +} +// WITH: +function assertRunnerOwnerIfAuthenticated( + ctx: TenantContext, + ownerPrincipal: string +): void { + if (!ctx.principal) return + if (ownerPrincipal === ctx.principal.url) return + throw new ElectricAgentsError( + ErrCodeUnauthorized, + `Runner access requires the authenticated owner`, + 403 + ) +} +``` + +- [ ] **Step 8: Update all callers of `assertRunnerOwnerIfAuthenticated`** + +Change all calls from `runner.owner_user_id` → `runner.owner_principal`: + +In `getRunner` (line 161): `assertRunnerOwnerIfAuthenticated(ctx, runner.owner_principal)` + +In `heartbeat` (line 171): `assertRunnerOwnerIfAuthenticated(ctx, existing.owner_principal)` + +In `setRunnerStatus` (line 208): `assertRunnerOwnerIfAuthenticated(ctx, existing.owner_principal)` + +- [ ] **Step 9: Update claim auth check** + +In `claimWake` (line 225): + +```ts +// REPLACE: + if (ctx.principal && runner.owner_user_id !== ctx.principal.key) { +// WITH: + if (ctx.principal && runner.owner_principal !== ctx.principal.url) { +``` + +- [ ] **Step 10: Implement `runnerHealth` handler** + +Add at the bottom of the file, before `notificationFromClaim`: + +```ts +async function runnerHealth( + request: RunnersRouteRequest, + ctx: TenantContext +): Promise { + const runnerId = routeParam(request, `id`) + const runner = await requireRunner(ctx, runnerId) + assertRunnerOwnerIfAuthenticated(ctx, runner.owner_principal) + + const now = Date.now() + const leaseExpiresAt = runner.liveness_lease_expires_at + ? new Date(runner.liveness_lease_expires_at).getTime() + : null + + const livenessStatus = + runner.admin_status === `disabled` + ? `offline` + : leaseExpiresAt !== null && leaseExpiresAt > now + ? `online` + : leaseExpiresAt !== null + ? `expired` + : `offline` + + const [activeClaims, dispatchStats] = await Promise.all([ + ctx.entityManager.registry.getActiveClaimsForRunner(runnerId), + ctx.entityManager.registry.getDispatchStatsForRunner(runnerId), + ]) + + const clientDiagnostics = runner.diagnostics ?? null + + const issues: Array = [] + let healthStatus: `healthy` | `degraded` | `unhealthy` = `healthy` + + if (runner.admin_status === `disabled`) { + healthStatus = `unhealthy` + issues.push(`Runner is disabled`) + } + if (livenessStatus === `expired`) { + healthStatus = `unhealthy` + const ago = leaseExpiresAt ? Math.round((now - leaseExpiresAt) / 1000) : 0 + issues.push(`Heartbeat lease expired ${ago}s ago`) + } + if (livenessStatus === `offline` && runner.admin_status === `enabled`) { + healthStatus = healthStatus === `unhealthy` ? `unhealthy` : `degraded` + issues.push(`Runner has never sent a heartbeat`) + } + if (clientDiagnostics) { + if (clientDiagnostics.stream_connected === false) { + if (healthStatus === `healthy`) healthStatus = `degraded` + issues.push(`Client reports stream disconnected`) + } + if (clientDiagnostics.last_heartbeat_ok === false) { + if (healthStatus === `healthy`) healthStatus = `degraded` + issues.push(`Client reports last heartbeat failed`) + } + if ( + typeof clientDiagnostics.reconnect_count === `number` && + clientDiagnostics.reconnect_count > 5 + ) { + if (healthStatus === `healthy`) healthStatus = `degraded` + issues.push( + `Client has reconnected ${clientDiagnostics.reconnect_count} times` + ) + } + } else if (runner.last_seen_at) { + if (healthStatus === `healthy`) healthStatus = `degraded` + issues.push(`No client diagnostics available`) + } + + return json({ + runner: { + id: runner.id, + admin_status: runner.admin_status, + liveness_status: livenessStatus, + lease_expires_at: runner.liveness_lease_expires_at ?? null, + lease_remaining_ms: + leaseExpiresAt !== null ? Math.max(0, leaseExpiresAt - now) : null, + wake_stream: runner.wake_stream, + wake_stream_offset: runner.wake_stream_offset ?? null, + last_seen_at: runner.last_seen_at ?? null, + created_at: runner.created_at, + }, + client: clientDiagnostics, + claims: { + active_count: activeClaims.length, + active: activeClaims.map((c) => ({ + consumer_id: c.consumer_id, + epoch: c.epoch, + entity_url: c.entity_url, + stream_path: c.stream_path, + claimed_at: c.claimed_at, + last_heartbeat_at: c.last_heartbeat_at ?? null, + lease_expires_at: c.lease_expires_at ?? null, + })), + }, + dispatch: dispatchStats, + health: { status: healthStatus, issues }, + }) +} +``` + +- [ ] **Step 11: Commit** + +```bash +git add packages/agents-server/src/routing/runners-router.ts +git commit -m "feat(agents-server): update runners router for principal rename, diagnostics, and health endpoint" +``` + +--- + +### Task 5: Client-side diagnostics in PullWakeRunner + +**Files:** + +- Modify: `packages/agents-runtime/src/pull-wake-runner.ts` + +- [ ] **Step 1: Add `PullWakeRunnerHealth` interface and diagnostics tracking** + +In `packages/agents-runtime/src/pull-wake-runner.ts`, after the existing `PullWakeRunner` interface (line 48-54), add: + +```ts +export interface PullWakeRunnerHealth { + running: boolean + offset: string | undefined + started_at: string | null + stream_connected: boolean + stream_connected_since: string | null + reconnect_count: number + last_error: string | null + last_error_at: string | null + last_heartbeat_at: string | null + last_heartbeat_ok: boolean + last_claim_at: string | null + last_claim_result: `claimed` | `no_work` | `error` | null + last_dispatch_at: string | null + events_received: number + claims_succeeded: number + claims_skipped: number + claims_failed: number +} +``` + +Add `getHealth` to the `PullWakeRunner` interface: + +```ts +export interface PullWakeRunner { + start: () => void + stop: () => Promise + waitForStopped: () => Promise + readonly running: boolean + readonly offset: string | undefined + getHealth: () => PullWakeRunnerHealth +} +``` + +- [ ] **Step 2: Add diagnostic state variables inside `createPullWakeRunner`** + +After the existing `let currentOffset = config.offset` (line 63), add: + +```ts +let startedAt: string | null = null +let streamConnected = false +let streamConnectedSince: string | null = null +let reconnectCount = 0 +let lastError: string | null = null +let lastErrorAt: string | null = null +let lastHeartbeatAt: string | null = null +let lastHeartbeatOk = false +let lastClaimAt: string | null = null +let lastClaimResult: PullWakeRunnerHealth[`last_claim_result`] = null +let lastDispatchAt: string | null = null +let eventsReceived = 0 +let claimsSucceeded = 0 +let claimsSkipped = 0 +let claimsFailed = 0 +``` + +- [ ] **Step 3: Build the diagnostics snapshot function** + +Add after the diagnostic variables: + +```ts +const buildDiagnostics = (): Omit< + PullWakeRunnerHealth, + `running` | `offset` +> => ({ + started_at: startedAt, + stream_connected: streamConnected, + stream_connected_since: streamConnectedSince, + reconnect_count: reconnectCount, + last_error: lastError, + last_error_at: lastErrorAt, + last_heartbeat_at: lastHeartbeatAt, + last_heartbeat_ok: lastHeartbeatOk, + last_claim_at: lastClaimAt, + last_claim_result: lastClaimResult, + last_dispatch_at: lastDispatchAt, + events_received: eventsReceived, + claims_succeeded: claimsSucceeded, + claims_skipped: claimsSkipped, + claims_failed: claimsFailed, +}) +``` + +- [ ] **Step 4: Update `heartbeat` to report diagnostics and track heartbeat state** + +Replace the existing `heartbeat` function (line 106-131): + +```ts +const heartbeat = async (signal: AbortSignal): Promise => { + try { + const headers = new Headers(await resolveHeaders()) + headers.set(`content-type`, `application/json`) + const res = await fetch(heartbeatUrl, { + method: `POST`, + headers, + body: JSON.stringify({ + lease_ms: leaseMs, + ...(currentOffset !== undefined + ? { wake_stream_offset: currentOffset } + : {}), + diagnostics: buildDiagnostics(), + }), + signal, + }) + lastHeartbeatAt = new Date().toISOString() + if (!res.ok) { + lastHeartbeatOk = false + throw new Error( + `Pull-wake runner heartbeat failed for ${config.runnerId}: ${res.status} ${await res.text()}` + ) + } + lastHeartbeatOk = true + } catch (err) { + if (!signal.aborted) { + lastHeartbeatOk = false + config.onError?.(err instanceof Error ? err : new Error(String(err))) + } + } +} +``` + +- [ ] **Step 5: Update `reportError` to track errors** + +Replace the existing `reportError` (line 101-104): + +```ts +const reportError = (err: unknown): void => { + const error = err instanceof Error ? err : new Error(String(err)) + lastError = error.message + lastErrorAt = new Date().toISOString() + if (config.onError?.(error) !== true) throw error +} +``` + +- [ ] **Step 6: Update `claimWake` to track claim results** + +Replace the existing `claimWake` (line 170-200): + +```ts +const claimWake = async ( + event: PullWakeEvent, + signal: AbortSignal +): Promise => { + lastClaimAt = new Date().toISOString() + const headers = new Headers(await resolveHeaders()) + headers.set(`content-type`, `application/json`) + try { + const response = await fetch(claimUrl, { + method: `POST`, + headers, + signal, + body: JSON.stringify(event), + }) + if (response.status === 204) { + lastClaimResult = `no_work` + claimsSkipped++ + return null + } + if (!response.ok) { + const text = await response.text() + if ( + response.status === 409 && + (text.includes(`ALREADY_CLAIMED`) || text.includes(`NO_PENDING_WORK`)) + ) { + lastClaimResult = `no_work` + claimsSkipped++ + return null + } + lastClaimResult = `error` + claimsFailed++ + throw new Error( + `Pull-wake claim failed for ${config.runnerId}: ${response.status} ${text}` + ) + } + const notification = (await response.json()) as WakeNotification & { + done?: boolean + } + if (notification.done) { + lastClaimResult = `no_work` + claimsSkipped++ + return null + } + lastClaimResult = `claimed` + claimsSucceeded++ + return notification + } catch (err) { + if (lastClaimResult !== `no_work` && lastClaimResult !== `error`) { + lastClaimResult = `error` + claimsFailed++ + } + throw err + } +} +``` + +- [ ] **Step 7: Update the `run` function to track stream and event state** + +Replace the existing `run` function (line 202-236): + +```ts +const run = async (): Promise => { + const signal = controller!.signal + try { + response = await streamFactory({ + url: wakeUrl, + headers: await resolveHeaders(), + offset: currentOffset, + signal, + }) + streamConnected = true + streamConnectedSince = new Date().toISOString() + for await (const event of response.jsonStream()) { + if (signal.aborted) break + if (event?.type !== `wake`) continue + eventsReceived++ + const notification = await claimWake(event, signal) + if (notification) { + config.runtime.dispatchWake(notification, { + claimHeaders: resolveClaimHeaders, + claimTokenHeader: config.claimTokenHeader, + }) + lastDispatchAt = new Date().toISOString() + await config.runtime.drainWakes() + } + if (response.offset !== undefined) currentOffset = response.offset + } + await response.closed?.catch((err) => { + if (!signal.aborted) throw err + }) + } catch (err) { + if (!signal.aborted) { + reconnectCount++ + reportError(err) + } + } finally { + streamConnected = false + stopHeartbeat() + response = null + controller = null + } +} +``` + +- [ ] **Step 8: Update `start()` to record `startedAt`** + +In the returned object's `start()` method (line 239-244): + +```ts + start() { + if (loop) return + controller = new AbortController() + startedAt = new Date().toISOString() + startHeartbeat(controller.signal) + loop = run().finally(() => { + loop = null + }) + }, +``` + +- [ ] **Step 9: Add `getHealth()` to the returned object** + +Add after the `offset` getter: + +```ts + getHealth(): PullWakeRunnerHealth { + return { + running: loop !== null, + offset: currentOffset, + ...buildDiagnostics(), + } + }, +``` + +- [ ] **Step 10: Update the runtime index exports** + +In `packages/agents-runtime/src/index.ts`, add `PullWakeRunnerHealth` to the exports (line 238-243): + +```ts +// REPLACE: +export type { + PullWakeEvent, + PullWakeRunner, + PullWakeRunnerConfig, + PullWakeStreamResponse, +} from './pull-wake-runner' +// WITH: +export type { + PullWakeEvent, + PullWakeRunner, + PullWakeRunnerConfig, + PullWakeRunnerHealth, + PullWakeStreamResponse, +} from './pull-wake-runner' +``` + +- [ ] **Step 11: Commit** + +```bash +git add packages/agents-runtime/src/pull-wake-runner.ts packages/agents-runtime/src/index.ts +git commit -m "feat(agents-runtime): add diagnostics tracking and getHealth() to PullWakeRunner" +``` + +--- + +### Task 6: Update BuiltinAgentsServer and desktop app for principal rename + +**Files:** + +- Modify: `packages/agents/src/server.ts:40-51, 393-422` +- Modify: `packages/agents-desktop/src/main.ts:219-274, 1544-1582` + +- [ ] **Step 1: Update `BuiltinAgentsServerOptions` in agents/server.ts** + +In `packages/agents/src/server.ts` (line 40-51): + +```ts +// REPLACE: + pullWake: { + runnerId: string + ownerUserId?: string + label?: string + registerRunner?: boolean +// WITH: + pullWake: { + runnerId: string + ownerPrincipal?: string + label?: string + registerRunner?: boolean +``` + +- [ ] **Step 2: Update `registerPullWakeRunner` to use `owner_principal`** + +In `packages/agents/src/server.ts` (line 393-422): + +```ts +// REPLACE: + body: JSON.stringify({ + id: pullWake.runnerId, + owner_user_id: pullWake.ownerUserId, + label: pullWake.label ?? `Built-in agents`, + kind: `local`, + admin_status: `enabled`, + }), +// WITH: + body: JSON.stringify({ + id: pullWake.runnerId, + owner_principal: pullWake.ownerPrincipal, + label: pullWake.label ?? `Built-in agents`, + kind: `local`, + admin_status: `enabled`, + }), +``` + +- [ ] **Step 3: Update desktop env var and function names** + +In `packages/agents-desktop/src/main.ts`: + +Rename the constant (line 227-228): + +```ts +// REPLACE: +const PULL_WAKE_OWNER_USER_ID = + process.env.ELECTRIC_DESKTOP_PULL_WAKE_OWNER_USER_ID?.trim() || +// WITH: +const PULL_WAKE_OWNER_PRINCIPAL = + process.env.ELECTRIC_DESKTOP_PULL_WAKE_OWNER_PRINCIPAL?.trim() || + process.env.ELECTRIC_DESKTOP_PULL_WAKE_OWNER_USER_ID?.trim() || +``` + +Rename the helper function (line 265-274). The server-side `registerRunner` handler already converts the principal key to a URL when storing, so the desktop just needs to pass the principal key it already has — the server handles the rest: + +```ts +// REPLACE: +function runnerOwnerUserIdFromHeaders( + headers: Record | undefined +): string { + const normalized = new Headers(headers) + return ( + normalized.get(`authorization`)?.trim() || + normalized.get(ELECTRIC_PRINCIPAL_HEADER)?.trim() || + PULL_WAKE_OWNER_USER_ID + ) +} +// WITH: +function runnerOwnerPrincipalFromHeaders( + headers: Record | undefined +): string { + const normalized = new Headers(headers) + return ( + normalized.get(ELECTRIC_PRINCIPAL_HEADER)?.trim() || + normalized.get(`authorization`)?.trim() || + PULL_WAKE_OWNER_PRINCIPAL + ) +} +``` + +Update usage (line 1544, 1551, 1575): + +```ts +// REPLACE: +const runnerOwnerUserId = runnerOwnerUserIdFromHeaders(runtimeHeaders) +// WITH: +const runnerOwnerPrincipal = runnerOwnerPrincipalFromHeaders(runtimeHeaders) +``` + +```ts +// REPLACE: + ownerUserId: PULL_WAKE_REGISTER_RUNNER ? runnerOwnerUserId : undefined, +// WITH: + ownerPrincipal: PULL_WAKE_REGISTER_RUNNER ? runnerOwnerPrincipal : undefined, +``` + +Update log messages referencing `owner user id` → `owner principal`. + +- [ ] **Step 4: Commit** + +```bash +git add packages/agents/src/server.ts packages/agents-desktop/src/main.ts +git commit -m "feat(agents, agents-desktop): rename ownerUserId to ownerPrincipal for runner registration" +``` + +--- + +### Task 7: Update tests for principal rename and health endpoint + +**Files:** + +- Modify: `packages/agents-server/test/runners-router.test.ts` +- Modify: `packages/agents-runtime/test/pull-wake-runner.test.ts` +- Modify: `packages/agents-server/test/horton-pull-wake-e2e.test.ts` +- Modify: `packages/agents-server/test/horton-title-generation.test.ts` +- Modify: `packages/agents-server/test/horton-spawn-worker.test.ts` +- Modify: `packages/agents-server/test/dispatch-policy-routing.test.ts` + +- [ ] **Step 1: Update runners-router.test.ts — principal rename and context** + +In `packages/agents-server/test/runners-router.test.ts`: + +Update the `runner()` helper (line 15-28): + +```ts +// REPLACE: + owner_user_id: `user:owner@example.com`, +// WITH: + owner_principal: `/principal/user%3Aowner%40example.com`, +``` + +Update `buildContext` registry mock (line 33-35): + +```ts +// REPLACE: + createRunner: vi.fn(async (input) => + runner({ + id: input.id, + owner_user_id: input.ownerUserId, +// WITH: + createRunner: vi.fn(async (input) => + runner({ + id: input.id, + owner_principal: input.ownerPrincipal, +``` + +Update all test assertions that reference `owner_user_id`: + +- Line 89: `owner_user_id: `other@example.com``→`owner_principal: `/principal/other` +- Line 118: `owner_user_id: `user:owner@example.com`` → `owner_principal: `/principal/user%3Aowner%40example.com`` +- Line 128-129: `ownerUserId: `user:owner@example.com`` → `ownerPrincipal: `/principal/user%3Aowner%40example.com`` +- Line 158-159: same replacement + +- [ ] **Step 2: Add health endpoint test to runners-router.test.ts** + +Add to the `runner routes` describe block: + +```ts +it(`returns runner health with diagnostics and claim state`, async () => { + const ctx = buildContext({ + principal: { + kind: `user`, + id: `owner@example.com`, + key: `user:owner@example.com`, + url: `/principal/user%3Aowner%40example.com`, + }, + }) + vi.mocked(ctx.entityManager.registry.getRunner).mockResolvedValue( + runner({ + owner_principal: `/principal/user%3Aowner%40example.com`, + liveness_lease_expires_at: new Date(Date.now() + 30_000).toISOString(), + last_seen_at: new Date().toISOString(), + diagnostics: { + stream_connected: true, + reconnect_count: 0, + last_heartbeat_ok: true, + }, + }) + ) + ctx.entityManager.registry.getActiveClaimsForRunner = vi.fn(async () => []) + ctx.entityManager.registry.getDispatchStatsForRunner = vi.fn(async () => ({ + entities_with_active_claim: 0, + entities_with_outstanding_wake: 0, + entities_with_pending_work: 0, + })) + + const response = await globalRouter.fetch( + request(`GET`, `/_electric/runners/runner-1/health`), + ctx + ) + + expect(response.status).toBe(200) + const body = (await response.json()) as Record + expect(body.runner).toMatchObject({ + id: `runner-1`, + liveness_status: `online`, + }) + expect(body.client).toMatchObject({ stream_connected: true }) + expect(body.claims).toMatchObject({ active_count: 0 }) + expect(body.health).toMatchObject({ status: `healthy`, issues: [] }) +}) + +it(`returns unhealthy when runner lease is expired`, async () => { + const ctx = buildContext({ + principal: { + kind: `user`, + id: `owner@example.com`, + key: `user:owner@example.com`, + url: `/principal/user%3Aowner%40example.com`, + }, + }) + vi.mocked(ctx.entityManager.registry.getRunner).mockResolvedValue( + runner({ + owner_principal: `/principal/user%3Aowner%40example.com`, + liveness_lease_expires_at: new Date(Date.now() - 10_000).toISOString(), + last_seen_at: new Date(Date.now() - 15_000).toISOString(), + }) + ) + ctx.entityManager.registry.getActiveClaimsForRunner = vi.fn(async () => []) + ctx.entityManager.registry.getDispatchStatsForRunner = vi.fn(async () => ({ + entities_with_active_claim: 0, + entities_with_outstanding_wake: 0, + entities_with_pending_work: 0, + })) + + const response = await globalRouter.fetch( + request(`GET`, `/_electric/runners/runner-1/health`), + ctx + ) + + expect(response.status).toBe(200) + const body = (await response.json()) as Record + expect((body.health as any).status).toBe(`unhealthy`) + expect((body.health as any).issues.length).toBeGreaterThan(0) +}) +``` + +- [ ] **Step 3: Add `getHealth()` test to pull-wake-runner.test.ts** + +Add to the `createPullWakeRunner` describe block in `packages/agents-runtime/test/pull-wake-runner.test.ts`: + +```ts +it(`exposes diagnostics via getHealth()`, async () => { + const event: PullWakeEvent = { + type: `wake`, + subscription_id: `runner:runner-1`, + stream: `chat/one/main`, + generation: 7, + ts: 123, + } + const notification: WakeNotification = { + consumerId: `wake-1`, + epoch: 7, + wakeId: `wake-1`, + streamPath: `/chat/one/main`, + streams: [{ path: `/chat/one/main`, offset: `12` }], + callback: `http://server/_electric/callback-forward/wake-1`, + claimToken: `claim-token`, + entity: { + type: `chat`, + status: `idle`, + url: `/chat/one`, + streams: { main: `/chat/one/main`, error: `/chat/one/error` }, + }, + } + const fetchMock = vi.fn(async (_input: RequestInfo | URL) => + Response.json(notification) + ) + vi.stubGlobal(`fetch`, fetchMock) + const streamFactory = vi.fn(async () => ({ + offset: `42`, + async *jsonStream() { + yield event + }, + closed: Promise.resolve(), + })) + + const runner = createPullWakeRunner({ + baseUrl: `http://server`, + runnerId: `runner-1`, + runtime: { + dispatchWake: vi.fn(), + drainWakes: vi.fn(async () => undefined), + abortWakes: vi.fn(), + }, + heartbeatIntervalMs: 0, + streamFactory, + }) + + const healthBefore = runner.getHealth() + expect(healthBefore.running).toBe(false) + expect(healthBefore.started_at).toBeNull() + expect(healthBefore.events_received).toBe(0) + + runner.start() + await runner.waitForStopped() + + const healthAfter = runner.getHealth() + expect(healthAfter.running).toBe(false) + expect(healthAfter.started_at).not.toBeNull() + expect(healthAfter.events_received).toBe(1) + expect(healthAfter.claims_succeeded).toBe(1) + expect(healthAfter.last_claim_result).toBe(`claimed`) + expect(healthAfter.last_dispatch_at).not.toBeNull() + expect(healthAfter.offset).toBe(`42`) +}) +``` + +- [ ] **Step 4: Update horton-pull-wake-e2e.test.ts for principal rename** + +In `packages/agents-server/test/horton-pull-wake-e2e.test.ts` (line 133): + +```ts +// REPLACE: + ownerUserId: testPrincipal.key, +// WITH: + ownerPrincipal: testPrincipal.url, +``` + +- [ ] **Step 5: Update horton-title-generation.test.ts and horton-spawn-worker.test.ts** + +In `packages/agents-server/test/horton-title-generation.test.ts` (line 39): + +```ts +// REPLACE: + ownerUserId: `test-user`, +// WITH: + ownerPrincipal: `/principal/system%3Atest-user`, +``` + +In `packages/agents-server/test/horton-spawn-worker.test.ts` (line 39): + +```ts +// REPLACE: + ownerUserId: `test-user`, +// WITH: + ownerPrincipal: `/principal/system%3Atest-user`, +``` + +- [ ] **Step 6: Update dispatch-policy-routing.test.ts** + +In `packages/agents-server/test/dispatch-policy-routing.test.ts` (line 71): + +```ts +// REPLACE: + owner_user_id: `user:owner@example.com`, +// WITH: + owner_principal: `/principal/user%3Aowner%40example.com`, +``` + +- [ ] **Step 7: Run all tests** + +Run: `cd packages/agents-runtime && pnpm vitest run test/pull-wake-runner.test.ts --reporter=dot` + +Run: `cd packages/agents-server && pnpm vitest run test/runners-router.test.ts --reporter=dot` + +Expected: All tests PASS + +- [ ] **Step 8: Commit** + +```bash +git add packages/agents-server/test/ packages/agents-runtime/test/ +git commit -m "test: update all tests for principal rename and add health endpoint tests" +``` + +--- + +### Task 8: Typecheck and final verification + +- [ ] **Step 1: Typecheck agents-runtime** + +Run: `pnpm -C packages/agents-runtime build` +Expected: No errors + +- [ ] **Step 2: Typecheck agents-server** + +Run: `pnpm --filter @electric-ax/agents-server typecheck` +Expected: No errors + +- [ ] **Step 3: Typecheck agents** + +Run: `pnpm --filter @electric-ax/agents typecheck` +Expected: No errors + +- [ ] **Step 4: Typecheck agents-desktop** + +Run: `pnpm --filter @electric-ax/agents-desktop typecheck` +Expected: No errors + +- [ ] **Step 5: Run unit tests** + +Run: `cd packages/agents-runtime && pnpm vitest run test/pull-wake-runner.test.ts --reporter=dot` +Run: `cd packages/agents-server && pnpm vitest run test/runners-router.test.ts --reporter=dot` +Expected: All PASS + +- [ ] **Step 6: Fix any issues and commit** + +If any typecheck or test failures, fix and commit: + +```bash +git commit -m "fix: address typecheck and test issues from health check implementation" +``` From e01b4d777c1d1c8946acab9cdda93636ba5ceca8 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Sat, 16 May 2026 11:59:43 +0100 Subject: [PATCH 04/37] =?UTF-8?q?fix(plan):=20address=20code=20review=20fi?= =?UTF-8?q?ndings=20=E2=80=94=20add=20canonicalizePrincipal,=20dispatch-po?= =?UTF-8?q?licy,=20server-utils,=20and=20electric-ax?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.6 --- .../2026-05-16-pull-wake-health-check.md | 132 +++++++++++++++--- 1 file changed, 112 insertions(+), 20 deletions(-) diff --git a/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md b/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md index ead050b551..c0adeac04a 100644 --- a/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md +++ b/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md @@ -458,11 +458,13 @@ git commit -m "feat(agents-server): update entity registry for principal rename, --- -### Task 4: Update runners router — principal rename, diagnostics in heartbeat, health endpoint +### Task 4: Update runners router, dispatch policy, and shape columns — principal rename, diagnostics in heartbeat, health endpoint **Files:** - Modify: `packages/agents-server/src/routing/runners-router.ts` +- Modify: `packages/agents-server/src/routing/dispatch-policy.ts:127` +- Modify: `packages/agents-server/src/utils/server-utils.ts:130-134` - [ ] **Step 1: Update the registration body schema** @@ -539,7 +541,26 @@ After the existing routes (line 90), add: runnersRouter.get(`/:id/health`, runnerHealth) ``` -- [ ] **Step 4: Update `registerRunner` handler to use `owner_principal` and `ctx.principal.url`** +- [ ] **Step 4: Add `canonicalizePrincipal` helper** + +Callers (desktop, electric-ax) may pass a principal key (`user:alice`) instead of a URL (`/principal/user%3Aalice`). Add a helper at the top of `runners-router.ts` that normalizes both forms, and add the import for `parsePrincipalKey`: + +```ts +// Add to imports at the top of the file: +import { parsePrincipalKey } from '../principal.js' + +// Add after the schema constants (after heartbeatBodySchema): +function canonicalizePrincipal(value: string): string { + if (value.startsWith(`/principal/`)) return value + try { + return parsePrincipalKey(value).url + } catch { + return value + } +} +``` + +- [ ] **Step 5: Update `registerRunner` handler to use `owner_principal` and canonicalize** In `registerRunner` (line 103-136): @@ -585,7 +606,9 @@ async function registerRunner( ctx: TenantContext ): Promise { const parsed = routeBody(request) - const ownerPrincipal = parsed.owner_principal ?? ctx.principal?.url + const ownerPrincipal = canonicalizePrincipal( + parsed.owner_principal ?? ctx.principal?.url ?? `` + ) if (!ownerPrincipal) { throw new ElectricAgentsError( ErrCodeInvalidRequest, @@ -616,7 +639,7 @@ async function registerRunner( } ``` -- [ ] **Step 5: Update `listRunners` handler** +- [ ] **Step 6: Update `listRunners` handler** In `listRunners` (line 138-154): @@ -659,7 +682,7 @@ async function listRunners( } ``` -- [ ] **Step 6: Update heartbeat handler to pass diagnostics** +- [ ] **Step 7: Update heartbeat handler to pass diagnostics** In `heartbeat` (line 165-185), add `diagnostics` to the `heartbeatRunner` call: @@ -675,7 +698,7 @@ const runner = await ctx.entityManager.registry.heartbeatRunner({ }) ``` -- [ ] **Step 7: Update `assertRunnerOwnerIfAuthenticated` to use `principal.url`** +- [ ] **Step 8: Update `assertRunnerOwnerIfAuthenticated` to use `principal.url`** In `assertRunnerOwnerIfAuthenticated` (line 297-308): @@ -708,7 +731,7 @@ function assertRunnerOwnerIfAuthenticated( } ``` -- [ ] **Step 8: Update all callers of `assertRunnerOwnerIfAuthenticated`** +- [ ] **Step 9: Update all callers of `assertRunnerOwnerIfAuthenticated`** Change all calls from `runner.owner_user_id` → `runner.owner_principal`: @@ -718,7 +741,7 @@ In `heartbeat` (line 171): `assertRunnerOwnerIfAuthenticated(ctx, existing.owner In `setRunnerStatus` (line 208): `assertRunnerOwnerIfAuthenticated(ctx, existing.owner_principal)` -- [ ] **Step 9: Update claim auth check** +- [ ] **Step 10: Update claim auth check** In `claimWake` (line 225): @@ -729,7 +752,29 @@ In `claimWake` (line 225): if (ctx.principal && runner.owner_principal !== ctx.principal.url) { ``` -- [ ] **Step 10: Implement `runnerHealth` handler** +- [ ] **Step 11: Update `assertDispatchPolicyAllowed` in dispatch-policy.ts** + +In `packages/agents-server/src/routing/dispatch-policy.ts` (line 127): + +```ts +// REPLACE: + if (ctx.principal && runner.owner_user_id !== ctx.principal.key) { +// WITH: + if (ctx.principal && runner.owner_principal !== ctx.principal.url) { +``` + +- [ ] **Step 12: Update runners Shape column allowlist in server-utils.ts** + +In `packages/agents-server/src/utils/server-utils.ts` (line 131-133): + +```ts +// REPLACE: +;`"tenant_id","id","owner_user_id","label","kind","admin_status","wake_stream","wake_stream_offset","last_seen_at","liveness_lease_expires_at","created_at","updated_at"` +// WITH: +`"tenant_id","id","owner_principal","label","kind","admin_status","wake_stream","wake_stream_offset","last_seen_at","liveness_lease_expires_at","diagnostics","created_at","updated_at"` +``` + +- [ ] **Step 13: Implement `runnerHealth` handler** Add at the bottom of the file, before `notificationFromClaim`: @@ -834,11 +879,11 @@ async function runnerHealth( } ``` -- [ ] **Step 11: Commit** +- [ ] **Step 14: Commit** ```bash -git add packages/agents-server/src/routing/runners-router.ts -git commit -m "feat(agents-server): update runners router for principal rename, diagnostics, and health endpoint" +git add packages/agents-server/src/routing/runners-router.ts packages/agents-server/src/routing/dispatch-policy.ts packages/agents-server/src/utils/server-utils.ts +git commit -m "feat(agents-server): update runners router, dispatch policy, and shape columns for principal rename, diagnostics, and health endpoint" ``` --- @@ -1158,11 +1203,12 @@ git commit -m "feat(agents-runtime): add diagnostics tracking and getHealth() to --- -### Task 6: Update BuiltinAgentsServer and desktop app for principal rename +### Task 6: Update BuiltinAgentsServer, electric-ax, and desktop app for principal rename **Files:** - Modify: `packages/agents/src/server.ts:40-51, 393-422` +- Modify: `packages/electric-ax/src/start.ts:131-139, 379, 395` - Modify: `packages/agents-desktop/src/main.ts:219-274, 1544-1582` - [ ] **Step 1: Update `BuiltinAgentsServerOptions` in agents/server.ts** @@ -1207,11 +1253,58 @@ In `packages/agents/src/server.ts` (line 393-422): }), ``` -- [ ] **Step 3: Update desktop env var and function names** +- [ ] **Step 3: Update electric-ax/src/start.ts** + +In `packages/electric-ax/src/start.ts`: + +Rename the function (line 131-139): + +```ts +// REPLACE: +export function resolvePullWakeOwnerId( + env: NodeJS.ProcessEnv = process.env, + fileEnv: Record = readDotEnvFile() +): string { + return ( + readConfigValue(env, fileEnv, [`ELECTRIC_AGENTS_IDENTITY`]) ?? + DEFAULT_PULL_WAKE_OWNER_ID + ) +} +// WITH: +export function resolvePullWakeOwnerPrincipal( + env: NodeJS.ProcessEnv = process.env, + fileEnv: Record = readDotEnvFile() +): string { + return ( + readConfigValue(env, fileEnv, [`ELECTRIC_AGENTS_IDENTITY`]) ?? + DEFAULT_PULL_WAKE_OWNER_ID + ) +} +``` + +Update the usage (line 379): + +```ts +// REPLACE: +const ownerUserId = resolvePullWakeOwnerId(env, fileEnv) +// WITH: +const ownerPrincipal = resolvePullWakeOwnerPrincipal(env, fileEnv) +``` + +Update the `BuiltinAgentsServer` call (line 395): + +```ts +// REPLACE: + ownerUserId, +// WITH: + ownerPrincipal, +``` + +- [ ] **Step 4: Update desktop env var and function names** In `packages/agents-desktop/src/main.ts`: -Rename the constant (line 227-228): +Rename the constant (line 227-228). No backwards-compat fallback — clean break: ```ts // REPLACE: @@ -1220,10 +1313,9 @@ const PULL_WAKE_OWNER_USER_ID = // WITH: const PULL_WAKE_OWNER_PRINCIPAL = process.env.ELECTRIC_DESKTOP_PULL_WAKE_OWNER_PRINCIPAL?.trim() || - process.env.ELECTRIC_DESKTOP_PULL_WAKE_OWNER_USER_ID?.trim() || ``` -Rename the helper function (line 265-274). The server-side `registerRunner` handler already converts the principal key to a URL when storing, so the desktop just needs to pass the principal key it already has — the server handles the rest: +Rename the helper function (line 265-274). The server-side `canonicalizePrincipal` converts principal keys to URLs when storing, so the desktop can pass whatever form it has: ```ts // REPLACE: @@ -1268,11 +1360,11 @@ const runnerOwnerPrincipal = runnerOwnerPrincipalFromHeaders(runtimeHeaders) Update log messages referencing `owner user id` → `owner principal`. -- [ ] **Step 4: Commit** +- [ ] **Step 5: Commit** ```bash -git add packages/agents/src/server.ts packages/agents-desktop/src/main.ts -git commit -m "feat(agents, agents-desktop): rename ownerUserId to ownerPrincipal for runner registration" +git add packages/agents/src/server.ts packages/electric-ax/src/start.ts packages/agents-desktop/src/main.ts +git commit -m "feat(agents, electric-ax, agents-desktop): rename ownerUserId to ownerPrincipal for runner registration" ``` --- From 83e2c41136a32a920b393fed2562a230e4bcda64 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Sat, 16 May 2026 12:04:23 +0100 Subject: [PATCH 05/37] =?UTF-8?q?fix(plan):=20strict=20no-compat=20?= =?UTF-8?q?=E2=80=94=20remove=20canonicalizePrincipal,=20validate=20URL=20?= =?UTF-8?q?form,=20callers=20convert=20keys=20to=20URLs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.6 --- .../2026-05-16-pull-wake-health-check.md | 67 +++++++++---------- 1 file changed, 30 insertions(+), 37 deletions(-) diff --git a/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md b/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md index c0adeac04a..e2ce6fd252 100644 --- a/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md +++ b/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md @@ -18,7 +18,11 @@ - [ ] **Step 1: Write the migration SQL** +Existing `owner_user_id` values are key-form strings (e.g., `local-desktop`). The new column expects principal URLs (e.g., `/principal/local-desktop`). Since we have no backwards compatibility, the migration truncates existing runner rows — runners are ephemeral and will re-register on next startup. + ```sql +DELETE FROM runners; +--> statement-breakpoint ALTER TABLE runners RENAME COLUMN owner_user_id TO owner_principal; --> statement-breakpoint DROP INDEX IF EXISTS idx_runners_owner_user_id; @@ -541,26 +545,9 @@ After the existing routes (line 90), add: runnersRouter.get(`/:id/health`, runnerHealth) ``` -- [ ] **Step 4: Add `canonicalizePrincipal` helper** - -Callers (desktop, electric-ax) may pass a principal key (`user:alice`) instead of a URL (`/principal/user%3Aalice`). Add a helper at the top of `runners-router.ts` that normalizes both forms, and add the import for `parsePrincipalKey`: - -```ts -// Add to imports at the top of the file: -import { parsePrincipalKey } from '../principal.js' - -// Add after the schema constants (after heartbeatBodySchema): -function canonicalizePrincipal(value: string): string { - if (value.startsWith(`/principal/`)) return value - try { - return parsePrincipalKey(value).url - } catch { - return value - } -} -``` +- [ ] **Step 4: Update `registerRunner` handler to use `owner_principal` with strict URL validation** -- [ ] **Step 5: Update `registerRunner` handler to use `owner_principal` and canonicalize** +No backwards compatibility for key-form principals. If `owner_principal` is provided it must be a URL (starts with `/principal/`); otherwise the server derives it from `ctx.principal.url`. Callers must send URLs. In `registerRunner` (line 103-136): @@ -606,9 +593,7 @@ async function registerRunner( ctx: TenantContext ): Promise { const parsed = routeBody(request) - const ownerPrincipal = canonicalizePrincipal( - parsed.owner_principal ?? ctx.principal?.url ?? `` - ) + const ownerPrincipal = parsed.owner_principal ?? ctx.principal?.url if (!ownerPrincipal) { throw new ElectricAgentsError( ErrCodeInvalidRequest, @@ -616,6 +601,13 @@ async function registerRunner( 400 ) } + if (!ownerPrincipal.startsWith(`/principal/`)) { + throw new ElectricAgentsError( + ErrCodeInvalidRequest, + `owner_principal must be a principal URL (starting with /principal/), got: ${ownerPrincipal}`, + 400 + ) + } if (ctx.principal && ownerPrincipal !== ctx.principal.url) { throw new ElectricAgentsError( ErrCodeUnauthorized, @@ -639,7 +631,7 @@ async function registerRunner( } ``` -- [ ] **Step 6: Update `listRunners` handler** +- [ ] **Step 5: Update `listRunners` handler** In `listRunners` (line 138-154): @@ -682,7 +674,7 @@ async function listRunners( } ``` -- [ ] **Step 7: Update heartbeat handler to pass diagnostics** +- [ ] **Step 6: Update heartbeat handler to pass diagnostics** In `heartbeat` (line 165-185), add `diagnostics` to the `heartbeatRunner` call: @@ -698,7 +690,7 @@ const runner = await ctx.entityManager.registry.heartbeatRunner({ }) ``` -- [ ] **Step 8: Update `assertRunnerOwnerIfAuthenticated` to use `principal.url`** +- [ ] **Step 7: Update `assertRunnerOwnerIfAuthenticated` to use `principal.url`** In `assertRunnerOwnerIfAuthenticated` (line 297-308): @@ -731,7 +723,7 @@ function assertRunnerOwnerIfAuthenticated( } ``` -- [ ] **Step 9: Update all callers of `assertRunnerOwnerIfAuthenticated`** +- [ ] **Step 8: Update all callers of `assertRunnerOwnerIfAuthenticated`** Change all calls from `runner.owner_user_id` → `runner.owner_principal`: @@ -741,7 +733,7 @@ In `heartbeat` (line 171): `assertRunnerOwnerIfAuthenticated(ctx, existing.owner In `setRunnerStatus` (line 208): `assertRunnerOwnerIfAuthenticated(ctx, existing.owner_principal)` -- [ ] **Step 10: Update claim auth check** +- [ ] **Step 9: Update claim auth check** In `claimWake` (line 225): @@ -752,7 +744,7 @@ In `claimWake` (line 225): if (ctx.principal && runner.owner_principal !== ctx.principal.url) { ``` -- [ ] **Step 11: Update `assertDispatchPolicyAllowed` in dispatch-policy.ts** +- [ ] **Step 10: Update `assertDispatchPolicyAllowed` in dispatch-policy.ts** In `packages/agents-server/src/routing/dispatch-policy.ts` (line 127): @@ -763,7 +755,7 @@ In `packages/agents-server/src/routing/dispatch-policy.ts` (line 127): if (ctx.principal && runner.owner_principal !== ctx.principal.url) { ``` -- [ ] **Step 12: Update runners Shape column allowlist in server-utils.ts** +- [ ] **Step 11: Update runners Shape column allowlist in server-utils.ts** In `packages/agents-server/src/utils/server-utils.ts` (line 131-133): @@ -774,7 +766,7 @@ In `packages/agents-server/src/utils/server-utils.ts` (line 131-133): `"tenant_id","id","owner_principal","label","kind","admin_status","wake_stream","wake_stream_offset","last_seen_at","liveness_lease_expires_at","diagnostics","created_at","updated_at"` ``` -- [ ] **Step 13: Implement `runnerHealth` handler** +- [ ] **Step 12: Implement `runnerHealth` handler** Add at the bottom of the file, before `notificationFromClaim`: @@ -879,7 +871,7 @@ async function runnerHealth( } ``` -- [ ] **Step 14: Commit** +- [ ] **Step 13: Commit** ```bash git add packages/agents-server/src/routing/runners-router.ts packages/agents-server/src/routing/dispatch-policy.ts packages/agents-server/src/utils/server-utils.ts @@ -1257,7 +1249,7 @@ In `packages/agents/src/server.ts` (line 393-422): In `packages/electric-ax/src/start.ts`: -Rename the function (line 131-139): +Rename the function and convert the resolved identity key to a principal URL (line 131-139): ```ts // REPLACE: @@ -1275,10 +1267,10 @@ export function resolvePullWakeOwnerPrincipal( env: NodeJS.ProcessEnv = process.env, fileEnv: Record = readDotEnvFile() ): string { - return ( + const key = readConfigValue(env, fileEnv, [`ELECTRIC_AGENTS_IDENTITY`]) ?? DEFAULT_PULL_WAKE_OWNER_ID - ) + return `/principal/${encodeURIComponent(key)}` } ``` @@ -1315,7 +1307,7 @@ const PULL_WAKE_OWNER_PRINCIPAL = process.env.ELECTRIC_DESKTOP_PULL_WAKE_OWNER_PRINCIPAL?.trim() || ``` -Rename the helper function (line 265-274). The server-side `canonicalizePrincipal` converts principal keys to URLs when storing, so the desktop can pass whatever form it has: +Rename the helper function (line 265-274). The server requires `owner_principal` to be a principal URL, so convert any key-form value to a URL before returning: ```ts // REPLACE: @@ -1334,11 +1326,12 @@ function runnerOwnerPrincipalFromHeaders( headers: Record | undefined ): string { const normalized = new Headers(headers) - return ( + const key = normalized.get(ELECTRIC_PRINCIPAL_HEADER)?.trim() || normalized.get(`authorization`)?.trim() || PULL_WAKE_OWNER_PRINCIPAL - ) + if (key.startsWith(`/principal/`)) return key + return `/principal/${encodeURIComponent(key)}` } ``` From c7c334250f87abea7f29dc41e1704982a376f033 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Sat, 16 May 2026 12:09:43 +0100 Subject: [PATCH 06/37] fix(plan): strict principal validation, clean up dependent tables in migration, drop authorization fallback MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Use principalKeyFromUrl for proper principal URL validation (rejects /principal/local-desktop) - Migration expires active claims and clears dispatch state before deleting runners - Desktop: don't use authorization header as principal source — return undefined and let server derive from ctx.principal.url - listRunners validates owner_principal query param Co-Authored-By: Claude Opus 4.6 --- .../2026-05-16-pull-wake-health-check.md | 65 ++++++++++++------- 1 file changed, 43 insertions(+), 22 deletions(-) diff --git a/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md b/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md index e2ce6fd252..a1310845ef 100644 --- a/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md +++ b/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md @@ -18,9 +18,13 @@ - [ ] **Step 1: Write the migration SQL** -Existing `owner_user_id` values are key-form strings (e.g., `local-desktop`). The new column expects principal URLs (e.g., `/principal/local-desktop`). Since we have no backwards compatibility, the migration truncates existing runner rows — runners are ephemeral and will re-register on next startup. +Existing `owner_user_id` values are key-form strings (e.g., `local-desktop`). The new column expects principal URLs (e.g., `/principal/system%3Alocal-desktop`). Since we have no backwards compatibility, the migration deletes existing runner rows — runners are ephemeral and will re-register on next startup. Must also clean up dependent tables (`consumer_claims` and `entity_dispatch_state`) since there are no FK constraints to cascade the deletes. ```sql +UPDATE consumer_claims SET status = 'expired', updated_at = NOW() WHERE status = 'active'; +--> statement-breakpoint +UPDATE entity_dispatch_state SET active_runner_id = NULL, active_consumer_id = NULL, active_epoch = NULL, active_claimed_at = NULL, active_lease_expires_at = NULL, updated_at = NOW() WHERE active_runner_id IS NOT NULL; +--> statement-breakpoint DELETE FROM runners; --> statement-breakpoint ALTER TABLE runners RENAME COLUMN owner_user_id TO owner_principal; @@ -545,9 +549,17 @@ After the existing routes (line 90), add: runnersRouter.get(`/:id/health`, runnerHealth) ``` -- [ ] **Step 4: Update `registerRunner` handler to use `owner_principal` with strict URL validation** +- [ ] **Step 4: Add `principalKeyFromUrl` import** + +Add to the imports at the top of `runners-router.ts`: -No backwards compatibility for key-form principals. If `owner_principal` is provided it must be a URL (starts with `/principal/`); otherwise the server derives it from `ctx.principal.url`. Callers must send URLs. +```ts +import { principalKeyFromUrl } from '../principal.js' +``` + +- [ ] **Step 5: Update `registerRunner` handler to use `owner_principal` with strict URL validation** + +No backwards compatibility for key-form principals. If `owner_principal` is provided it must be a valid principal URL that `principalKeyFromUrl` can parse (i.e., `/principal/${encodeURIComponent('kind:id')}`); otherwise the server derives it from `ctx.principal.url`. Callers must send URLs. In `registerRunner` (line 103-136): @@ -601,10 +613,10 @@ async function registerRunner( 400 ) } - if (!ownerPrincipal.startsWith(`/principal/`)) { + if (!principalKeyFromUrl(ownerPrincipal)) { throw new ElectricAgentsError( ErrCodeInvalidRequest, - `owner_principal must be a principal URL (starting with /principal/), got: ${ownerPrincipal}`, + `owner_principal must be a valid principal URL (e.g. /principal/user%3Aalice), got: ${ownerPrincipal}`, 400 ) } @@ -631,7 +643,7 @@ async function registerRunner( } ``` -- [ ] **Step 5: Update `listRunners` handler** +- [ ] **Step 6: Update `listRunners` handler** In `listRunners` (line 138-154): @@ -660,6 +672,13 @@ async function listRunners( ctx: TenantContext ): Promise { const requestedOwner = firstQueryValue(request.query.owner_principal) + if (requestedOwner && !principalKeyFromUrl(requestedOwner)) { + throw new ElectricAgentsError( + ErrCodeInvalidRequest, + `owner_principal must be a valid principal URL (e.g. /principal/user%3Aalice), got: ${requestedOwner}`, + 400 + ) + } if (ctx.principal && requestedOwner && requestedOwner !== ctx.principal.url) { throw new ElectricAgentsError( ErrCodeUnauthorized, @@ -674,7 +693,7 @@ async function listRunners( } ``` -- [ ] **Step 6: Update heartbeat handler to pass diagnostics** +- [ ] **Step 7: Update heartbeat handler to pass diagnostics** In `heartbeat` (line 165-185), add `diagnostics` to the `heartbeatRunner` call: @@ -690,7 +709,7 @@ const runner = await ctx.entityManager.registry.heartbeatRunner({ }) ``` -- [ ] **Step 7: Update `assertRunnerOwnerIfAuthenticated` to use `principal.url`** +- [ ] **Step 8: Update `assertRunnerOwnerIfAuthenticated` to use `principal.url`** In `assertRunnerOwnerIfAuthenticated` (line 297-308): @@ -723,7 +742,7 @@ function assertRunnerOwnerIfAuthenticated( } ``` -- [ ] **Step 8: Update all callers of `assertRunnerOwnerIfAuthenticated`** +- [ ] **Step 9: Update all callers of `assertRunnerOwnerIfAuthenticated`** Change all calls from `runner.owner_user_id` → `runner.owner_principal`: @@ -733,7 +752,7 @@ In `heartbeat` (line 171): `assertRunnerOwnerIfAuthenticated(ctx, existing.owner In `setRunnerStatus` (line 208): `assertRunnerOwnerIfAuthenticated(ctx, existing.owner_principal)` -- [ ] **Step 9: Update claim auth check** +- [ ] **Step 10: Update claim auth check** In `claimWake` (line 225): @@ -744,7 +763,7 @@ In `claimWake` (line 225): if (ctx.principal && runner.owner_principal !== ctx.principal.url) { ``` -- [ ] **Step 10: Update `assertDispatchPolicyAllowed` in dispatch-policy.ts** +- [ ] **Step 11: Update `assertDispatchPolicyAllowed` in dispatch-policy.ts** In `packages/agents-server/src/routing/dispatch-policy.ts` (line 127): @@ -755,7 +774,7 @@ In `packages/agents-server/src/routing/dispatch-policy.ts` (line 127): if (ctx.principal && runner.owner_principal !== ctx.principal.url) { ``` -- [ ] **Step 11: Update runners Shape column allowlist in server-utils.ts** +- [ ] **Step 12: Update runners Shape column allowlist in server-utils.ts** In `packages/agents-server/src/utils/server-utils.ts` (line 131-133): @@ -766,7 +785,7 @@ In `packages/agents-server/src/utils/server-utils.ts` (line 131-133): `"tenant_id","id","owner_principal","label","kind","admin_status","wake_stream","wake_stream_offset","last_seen_at","liveness_lease_expires_at","diagnostics","created_at","updated_at"` ``` -- [ ] **Step 12: Implement `runnerHealth` handler** +- [ ] **Step 13: Implement `runnerHealth` handler** Add at the bottom of the file, before `notificationFromClaim`: @@ -871,7 +890,7 @@ async function runnerHealth( } ``` -- [ ] **Step 13: Commit** +- [ ] **Step 14: Commit** ```bash git add packages/agents-server/src/routing/runners-router.ts packages/agents-server/src/routing/dispatch-policy.ts packages/agents-server/src/utils/server-utils.ts @@ -1307,7 +1326,7 @@ const PULL_WAKE_OWNER_PRINCIPAL = process.env.ELECTRIC_DESKTOP_PULL_WAKE_OWNER_PRINCIPAL?.trim() || ``` -Rename the helper function (line 265-274). The server requires `owner_principal` to be a principal URL, so convert any key-form value to a URL before returning: +Rename the helper function (line 265-274). Do NOT use the `authorization` header as a principal source — that's a bearer token, not a principal key. When the request has auth headers, the server middleware extracts `ctx.principal` from them and uses `ctx.principal.url` as the owner. So when only auth is present (no explicit `electric-principal` header), return `undefined` to let the server derive the owner. Only produce a principal URL from an explicit `electric-principal` key or the env var fallback: ```ts // REPLACE: @@ -1324,14 +1343,16 @@ function runnerOwnerUserIdFromHeaders( // WITH: function runnerOwnerPrincipalFromHeaders( headers: Record | undefined -): string { +): string | undefined { const normalized = new Headers(headers) - const key = - normalized.get(ELECTRIC_PRINCIPAL_HEADER)?.trim() || - normalized.get(`authorization`)?.trim() || - PULL_WAKE_OWNER_PRINCIPAL - if (key.startsWith(`/principal/`)) return key - return `/principal/${encodeURIComponent(key)}` + const principalKey = normalized.get(ELECTRIC_PRINCIPAL_HEADER)?.trim() + if (principalKey) { + return principalKey.startsWith(`/principal/`) + ? principalKey + : `/principal/${encodeURIComponent(principalKey)}` + } + if (normalized.has(`authorization`)) return undefined + return `/principal/${encodeURIComponent(PULL_WAKE_OWNER_PRINCIPAL)}` } ``` From 454ea9b18fe5cb18088ceea3e353953ca1358721 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Sat, 16 May 2026 12:18:03 +0100 Subject: [PATCH 07/37] fix(plan): scope migration to runner-owned claims, fix default principal keys, complete desktop constant replacement Co-Authored-By: Claude Opus 4.6 --- .../2026-05-16-pull-wake-health-check.md | 23 ++++++++++++++----- 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md b/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md index a1310845ef..88366fb2e3 100644 --- a/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md +++ b/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md @@ -21,7 +21,7 @@ Existing `owner_user_id` values are key-form strings (e.g., `local-desktop`). The new column expects principal URLs (e.g., `/principal/system%3Alocal-desktop`). Since we have no backwards compatibility, the migration deletes existing runner rows — runners are ephemeral and will re-register on next startup. Must also clean up dependent tables (`consumer_claims` and `entity_dispatch_state`) since there are no FK constraints to cascade the deletes. ```sql -UPDATE consumer_claims SET status = 'expired', updated_at = NOW() WHERE status = 'active'; +UPDATE consumer_claims SET status = 'expired', updated_at = NOW() WHERE status = 'active' AND runner_id IS NOT NULL; --> statement-breakpoint UPDATE entity_dispatch_state SET active_runner_id = NULL, active_consumer_id = NULL, active_epoch = NULL, active_claimed_at = NULL, active_lease_expires_at = NULL, updated_at = NOW() WHERE active_runner_id IS NOT NULL; --> statement-breakpoint @@ -559,7 +559,7 @@ import { principalKeyFromUrl } from '../principal.js' - [ ] **Step 5: Update `registerRunner` handler to use `owner_principal` with strict URL validation** -No backwards compatibility for key-form principals. If `owner_principal` is provided it must be a valid principal URL that `principalKeyFromUrl` can parse (i.e., `/principal/${encodeURIComponent('kind:id')}`); otherwise the server derives it from `ctx.principal.url`. Callers must send URLs. +No backwards compatibility for key-form principals. If `owner_principal` is provided it must be a valid principal URL accepted by `principalKeyFromUrl()` (e.g., `/principal/user%3Aalice`); otherwise the server derives it from `ctx.principal.url`. Callers must send URLs. In `registerRunner` (line 103-136): @@ -616,7 +616,7 @@ async function registerRunner( if (!principalKeyFromUrl(ownerPrincipal)) { throw new ElectricAgentsError( ErrCodeInvalidRequest, - `owner_principal must be a valid principal URL (e.g. /principal/user%3Aalice), got: ${ownerPrincipal}`, + `owner_principal must be a valid principal URL accepted by principalKeyFromUrl() (e.g. /principal/user%3Aalice), got: ${ownerPrincipal}`, 400 ) } @@ -1268,7 +1268,16 @@ In `packages/agents/src/server.ts` (line 393-422): In `packages/electric-ax/src/start.ts`: -Rename the function and convert the resolved identity key to a principal URL (line 131-139): +First, rename and fix the default constant (line 21). `builtin-agents` is not a valid principal key — must be `kind:id` format: + +```ts +// REPLACE: +const DEFAULT_PULL_WAKE_OWNER_ID = `builtin-agents` +// WITH: +const DEFAULT_PULL_WAKE_OWNER_PRINCIPAL = `system:builtin-agents` +``` + +Then rename the function and convert the resolved identity key to a principal URL (line 131-139): ```ts // REPLACE: @@ -1288,7 +1297,7 @@ export function resolvePullWakeOwnerPrincipal( ): string { const key = readConfigValue(env, fileEnv, [`ELECTRIC_AGENTS_IDENTITY`]) ?? - DEFAULT_PULL_WAKE_OWNER_ID + DEFAULT_PULL_WAKE_OWNER_PRINCIPAL return `/principal/${encodeURIComponent(key)}` } ``` @@ -1315,15 +1324,17 @@ Update the `BuiltinAgentsServer` call (line 395): In `packages/agents-desktop/src/main.ts`: -Rename the constant (line 227-228). No backwards-compat fallback — clean break: +Rename the constant (line 227-229). No backwards-compat fallback — clean break. The default must be a valid principal key (`kind:id` format): ```ts // REPLACE: const PULL_WAKE_OWNER_USER_ID = process.env.ELECTRIC_DESKTOP_PULL_WAKE_OWNER_USER_ID?.trim() || + `local-desktop` // WITH: const PULL_WAKE_OWNER_PRINCIPAL = process.env.ELECTRIC_DESKTOP_PULL_WAKE_OWNER_PRINCIPAL?.trim() || + `system:local-desktop` ``` Rename the helper function (line 265-274). Do NOT use the `authorization` header as a principal source — that's a bearer token, not a principal key. When the request has auth headers, the server middleware extracts `ctx.principal` from them and uses `ctx.principal.url` as the owner. So when only auth is present (no explicit `electric-principal` header), return `undefined` to let the server derive the owner. Only produce a principal URL from an explicit `electric-principal` key or the env var fallback: From 6530be34c6dc7afa0b3a4b1cfec57836063a0c00 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Sat, 16 May 2026 12:27:08 +0100 Subject: [PATCH 08/37] fix(plan): store principal URLs directly in constants, not keys Co-Authored-By: Claude Opus 4.6 --- .../2026-05-16-pull-wake-health-check.md | 21 +++++++++---------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md b/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md index 88366fb2e3..df95c25929 100644 --- a/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md +++ b/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md @@ -1268,16 +1268,16 @@ In `packages/agents/src/server.ts` (line 393-422): In `packages/electric-ax/src/start.ts`: -First, rename and fix the default constant (line 21). `builtin-agents` is not a valid principal key — must be `kind:id` format: +First, rename the default constant (line 21). Store a principal URL directly: ```ts // REPLACE: const DEFAULT_PULL_WAKE_OWNER_ID = `builtin-agents` // WITH: -const DEFAULT_PULL_WAKE_OWNER_PRINCIPAL = `system:builtin-agents` +const DEFAULT_PULL_WAKE_OWNER_PRINCIPAL = `/principal/system%3Abuiltin-agents` ``` -Then rename the function and convert the resolved identity key to a principal URL (line 131-139): +Then rename the function (line 131-139). `ELECTRIC_AGENTS_IDENTITY` is a principal key (`kind:id`), so convert it to a URL. The default is already a URL: ```ts // REPLACE: @@ -1295,10 +1295,9 @@ export function resolvePullWakeOwnerPrincipal( env: NodeJS.ProcessEnv = process.env, fileEnv: Record = readDotEnvFile() ): string { - const key = - readConfigValue(env, fileEnv, [`ELECTRIC_AGENTS_IDENTITY`]) ?? - DEFAULT_PULL_WAKE_OWNER_PRINCIPAL - return `/principal/${encodeURIComponent(key)}` + const identity = readConfigValue(env, fileEnv, [`ELECTRIC_AGENTS_IDENTITY`]) + if (identity) return `/principal/${encodeURIComponent(identity)}` + return DEFAULT_PULL_WAKE_OWNER_PRINCIPAL } ``` @@ -1324,7 +1323,7 @@ Update the `BuiltinAgentsServer` call (line 395): In `packages/agents-desktop/src/main.ts`: -Rename the constant (line 227-229). No backwards-compat fallback — clean break. The default must be a valid principal key (`kind:id` format): +Rename the constant (line 227-229). No backwards-compat fallback — clean break. Store a principal URL directly: ```ts // REPLACE: @@ -1334,10 +1333,10 @@ const PULL_WAKE_OWNER_USER_ID = // WITH: const PULL_WAKE_OWNER_PRINCIPAL = process.env.ELECTRIC_DESKTOP_PULL_WAKE_OWNER_PRINCIPAL?.trim() || - `system:local-desktop` + `/principal/system%3Alocal-desktop` ``` -Rename the helper function (line 265-274). Do NOT use the `authorization` header as a principal source — that's a bearer token, not a principal key. When the request has auth headers, the server middleware extracts `ctx.principal` from them and uses `ctx.principal.url` as the owner. So when only auth is present (no explicit `electric-principal` header), return `undefined` to let the server derive the owner. Only produce a principal URL from an explicit `electric-principal` key or the env var fallback: +Rename the helper function (line 265-274). Do NOT use the `authorization` header as a principal source — that's a bearer token, not a principal key. When the request has auth headers, the server middleware extracts `ctx.principal` from them and uses `ctx.principal.url` as the owner. So when only auth is present (no explicit `electric-principal` header), return `undefined` to let the server derive the owner: ```ts // REPLACE: @@ -1363,7 +1362,7 @@ function runnerOwnerPrincipalFromHeaders( : `/principal/${encodeURIComponent(principalKey)}` } if (normalized.has(`authorization`)) return undefined - return `/principal/${encodeURIComponent(PULL_WAKE_OWNER_PRINCIPAL)}` + return PULL_WAKE_OWNER_PRINCIPAL } ``` From 6c499825461fc0f085266d696dc3e4c451135616 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Sat, 16 May 2026 12:57:32 +0100 Subject: [PATCH 09/37] feat(agents): add pull-wake runner health diagnostics --- packages/agents-desktop/README.md | 12 +- packages/agents-desktop/src/main.ts | 31 ++-- packages/agents-runtime/src/index.ts | 1 + .../agents-runtime/src/pull-wake-runner.ts | 144 ++++++++++++++--- .../test/pull-wake-runner.test.ts | 65 ++++++++ .../2026-05-16-pull-wake-health-check.md | 120 +++++++------- .../0007_runner_diagnostics_and_principal.sql | 22 +++ .../agents-server/drizzle/meta/_journal.json | 7 + packages/agents-server/src/db/schema.ts | 8 +- .../src/electric-agents-types.ts | 44 ++++- packages/agents-server/src/entity-registry.ts | 67 +++++++- .../src/routing/dispatch-policy.ts | 2 +- .../src/routing/runners-router.ts | 151 +++++++++++++++--- .../agents-server/src/utils/server-utils.ts | 2 +- .../test/dispatch-policy-routing.test.ts | 6 +- .../test/horton-pull-wake-e2e.test.ts | 2 +- .../test/horton-spawn-worker.test.ts | 2 +- .../test/horton-title-generation.test.ts | 2 +- .../agents-server/test/runners-router.test.ts | 88 ++++++++-- packages/agents/src/server.ts | 4 +- packages/electric-ax/src/start.ts | 15 +- packages/electric-ax/test/start.test.ts | 15 +- 22 files changed, 642 insertions(+), 168 deletions(-) create mode 100644 packages/agents-server/drizzle/0007_runner_diagnostics_and_principal.sql diff --git a/packages/agents-desktop/README.md b/packages/agents-desktop/README.md index a89244ef62..6eac3be5be 100644 --- a/packages/agents-desktop/README.md +++ b/packages/agents-desktop/README.md @@ -18,12 +18,12 @@ This starts both the UI dev server (with HMR) and the Electron main process. ### Environment variables -| Variable | Default | Description | -| -------------------------------------------- | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `ELECTRIC_DESKTOP_PRINCIPAL` | _(none)_ | Sets the `electric-principal` header on all requests to the agents-server. Use `system:dev-local` for local development without auth. | -| `ELECTRIC_DESKTOP_PULL_WAKE_OWNER_USER_ID` | `local-desktop` | Override the `owner_user_id` used when registering the pull-wake runner. When `ELECTRIC_DESKTOP_PRINCIPAL` is set, this is derived from it automatically. | -| `ELECTRIC_DESKTOP_PULL_WAKE_RUNNER_ID` | _(auto-generated)_ | Fixed runner ID for the pull-wake runner. | -| `ELECTRIC_DESKTOP_PULL_WAKE_REGISTER_RUNNER` | `true` | Set to `false` to skip runner registration (runner must already exist on the server). | +| Variable | Default | Description | +| -------------------------------------------- | ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `ELECTRIC_DESKTOP_PRINCIPAL` | _(none)_ | Sets the `electric-principal` header on all requests to the agents-server. Use `system:dev-local` for local development without auth. | +| `ELECTRIC_DESKTOP_PULL_WAKE_OWNER_PRINCIPAL` | `/principal/system%3Alocal-desktop` | Override the `owner_principal` used when registering the pull-wake runner. When `ELECTRIC_DESKTOP_PRINCIPAL` is set, this is derived from it automatically. | +| `ELECTRIC_DESKTOP_PULL_WAKE_RUNNER_ID` | _(auto-generated)_ | Fixed runner ID for the pull-wake runner. | +| `ELECTRIC_DESKTOP_PULL_WAKE_REGISTER_RUNNER` | `true` | Set to `false` to skip runner registration (runner must already exist on the server). | ### Settings diff --git a/packages/agents-desktop/src/main.ts b/packages/agents-desktop/src/main.ts index 15053c8af9..dab20050cd 100644 --- a/packages/agents-desktop/src/main.ts +++ b/packages/agents-desktop/src/main.ts @@ -224,9 +224,9 @@ const PULL_WAKE_REGISTER_RUNNER = : [`1`, `true`].includes( process.env.ELECTRIC_DESKTOP_PULL_WAKE_REGISTER_RUNNER.trim().toLowerCase() ) -const PULL_WAKE_OWNER_USER_ID = - process.env.ELECTRIC_DESKTOP_PULL_WAKE_OWNER_USER_ID?.trim() || - `local-desktop` +const PULL_WAKE_OWNER_PRINCIPAL = + process.env.ELECTRIC_DESKTOP_PULL_WAKE_OWNER_PRINCIPAL?.trim() || + `/principal/system%3Alocal-desktop` const DEV_PRINCIPAL = ((): string | null => { const raw = process.env.ELECTRIC_DESKTOP_PRINCIPAL?.trim() || null if (!raw) return null @@ -262,15 +262,18 @@ function hasHeader( return headers ? new Headers(headers).has(name) : false } -function runnerOwnerUserIdFromHeaders( +function runnerOwnerPrincipalFromHeaders( headers: Record | undefined -): string { +): string | undefined { const normalized = new Headers(headers) - return ( - normalized.get(`authorization`)?.trim() || - normalized.get(ELECTRIC_PRINCIPAL_HEADER)?.trim() || - PULL_WAKE_OWNER_USER_ID - ) + const principalKey = normalized.get(ELECTRIC_PRINCIPAL_HEADER)?.trim() + if (principalKey) { + return principalKey.startsWith(`/principal/`) + ? principalKey + : `/principal/${encodeURIComponent(principalKey)}` + } + if (normalized.has(`authorization`)) return undefined + return PULL_WAKE_OWNER_PRINCIPAL } /** @@ -1541,14 +1544,14 @@ async function startRuntime(serverId: string): Promise { const serverWithPrincipal = injectDevPrincipalHeaders(activeServer) const runtimeHeaders = mergeHeaders(serverWithPrincipal.headers) - const runnerOwnerUserId = runnerOwnerUserIdFromHeaders(runtimeHeaders) + const runnerOwnerPrincipal = runnerOwnerPrincipalFromHeaders(runtimeHeaders) console.info( `[agents-desktop] Starting built-in agents runtime for server ${activeServer.url}` ) console.info(`[agents-desktop] Pull-wake runner id: ${runnerId}`) if (PULL_WAKE_REGISTER_RUNNER) { console.info( - `[agents-desktop] Pull-wake runner registration enabled; owner user id: ${runnerOwnerUserId}` + `[agents-desktop] Pull-wake runner registration enabled; owner principal: ${runnerOwnerPrincipal ?? `(derived from auth)`}` ) } else { console.info( @@ -1572,7 +1575,9 @@ async function startRuntime(serverId: string): Promise { pullWake: { runnerId, registerRunner: PULL_WAKE_REGISTER_RUNNER, - ownerUserId: PULL_WAKE_REGISTER_RUNNER ? runnerOwnerUserId : undefined, + ownerPrincipal: PULL_WAKE_REGISTER_RUNNER + ? runnerOwnerPrincipal + : undefined, label: `Electric Agents Desktop`, headers: runtimeHeaders, claimHeaders: runtimeHeaders, diff --git a/packages/agents-runtime/src/index.ts b/packages/agents-runtime/src/index.ts index 14ed10070d..e5f9a1d1d6 100644 --- a/packages/agents-runtime/src/index.ts +++ b/packages/agents-runtime/src/index.ts @@ -239,6 +239,7 @@ export type { PullWakeEvent, PullWakeRunner, PullWakeRunnerConfig, + PullWakeRunnerHealth, PullWakeStreamResponse, } from './pull-wake-runner' diff --git a/packages/agents-runtime/src/pull-wake-runner.ts b/packages/agents-runtime/src/pull-wake-runner.ts index a729787a2c..ccedaaf5d2 100644 --- a/packages/agents-runtime/src/pull-wake-runner.ts +++ b/packages/agents-runtime/src/pull-wake-runner.ts @@ -51,6 +51,27 @@ export interface PullWakeRunner { waitForStopped: () => Promise readonly running: boolean readonly offset: string | undefined + getHealth: () => PullWakeRunnerHealth +} + +export interface PullWakeRunnerHealth { + running: boolean + offset: string | undefined + started_at: string | null + stream_connected: boolean + stream_connected_since: string | null + reconnect_count: number + last_error: string | null + last_error_at: string | null + last_heartbeat_at: string | null + last_heartbeat_ok: boolean + last_claim_at: string | null + last_claim_result: `claimed` | `no_work` | `error` | null + last_dispatch_at: string | null + events_received: number + claims_succeeded: number + claims_skipped: number + claims_failed: number } export function createPullWakeRunner( @@ -61,6 +82,21 @@ export function createPullWakeRunner( let response: PullWakeStreamResponse | null = null let heartbeatTimer: ReturnType | null = null let currentOffset = config.offset + let startedAt: string | null = null + let streamConnected = false + let streamConnectedSince: string | null = null + let reconnectCount = 0 + let lastError: string | null = null + let lastErrorAt: string | null = null + let lastHeartbeatAt: string | null = null + let lastHeartbeatOk = false + let lastClaimAt: string | null = null + let lastClaimResult: PullWakeRunnerHealth[`last_claim_result`] = null + let lastDispatchAt: string | null = null + let eventsReceived = 0 + let claimsSucceeded = 0 + let claimsSkipped = 0 + let claimsFailed = 0 const wakePath = config.wakeStreamPath ?? @@ -78,6 +114,27 @@ export function createPullWakeRunner( `/_electric/runners/${encodeURIComponent(config.runnerId)}/claim` const claimUrl = appendPathToUrl(config.baseUrl, claimPath) + const buildDiagnostics = (): Omit< + PullWakeRunnerHealth, + `running` | `offset` + > => ({ + started_at: startedAt, + stream_connected: streamConnected, + stream_connected_since: streamConnectedSince, + reconnect_count: reconnectCount, + last_error: lastError, + last_error_at: lastErrorAt, + last_heartbeat_at: lastHeartbeatAt, + last_heartbeat_ok: lastHeartbeatOk, + last_claim_at: lastClaimAt, + last_claim_result: lastClaimResult, + last_dispatch_at: lastDispatchAt, + events_received: eventsReceived, + claims_succeeded: claimsSucceeded, + claims_skipped: claimsSkipped, + claims_failed: claimsFailed, + }) + const resolveHeaders = async (): Promise> => { const init = typeof config.headers === `function` @@ -100,6 +157,8 @@ export function createPullWakeRunner( const reportError = (err: unknown): void => { const error = err instanceof Error ? err : new Error(String(err)) + lastError = error.message + lastErrorAt = new Date().toISOString() if (config.onError?.(error) !== true) throw error } @@ -115,16 +174,21 @@ export function createPullWakeRunner( ...(currentOffset !== undefined ? { wake_stream_offset: currentOffset } : {}), + diagnostics: buildDiagnostics(), }), signal, }) + lastHeartbeatAt = new Date().toISOString() if (!res.ok) { + lastHeartbeatOk = false throw new Error( `Pull-wake runner heartbeat failed for ${config.runnerId}: ${res.status} ${await res.text()}` ) } + lastHeartbeatOk = true } catch (err) { if (!signal.aborted) { + lastHeartbeatOk = false config.onError?.(err instanceof Error ? err : new Error(String(err))) } } @@ -171,32 +235,56 @@ export function createPullWakeRunner( event: PullWakeEvent, signal: AbortSignal ): Promise => { + lastClaimAt = new Date().toISOString() + lastClaimResult = null const headers = new Headers(await resolveHeaders()) headers.set(`content-type`, `application/json`) - const response = await fetch(claimUrl, { - method: `POST`, - headers, - signal, - body: JSON.stringify(event), - }) - if (response.status === 204) return null - if (!response.ok) { - const text = await response.text() - if ( - response.status === 409 && - (text.includes(`ALREADY_CLAIMED`) || text.includes(`NO_PENDING_WORK`)) - ) { + try { + const response = await fetch(claimUrl, { + method: `POST`, + headers, + signal, + body: JSON.stringify(event), + }) + if (response.status === 204) { + lastClaimResult = `no_work` + claimsSkipped++ return null } - throw new Error( - `Pull-wake claim failed for ${config.runnerId}: ${response.status} ${text}` - ) - } - const notification = (await response.json()) as WakeNotification & { - done?: boolean + if (!response.ok) { + const text = await response.text() + if ( + response.status === 409 && + (text.includes(`ALREADY_CLAIMED`) || text.includes(`NO_PENDING_WORK`)) + ) { + lastClaimResult = `no_work` + claimsSkipped++ + return null + } + lastClaimResult = `error` + claimsFailed++ + throw new Error( + `Pull-wake claim failed for ${config.runnerId}: ${response.status} ${text}` + ) + } + const notification = (await response.json()) as WakeNotification & { + done?: boolean + } + if (notification.done) { + lastClaimResult = `no_work` + claimsSkipped++ + return null + } + lastClaimResult = `claimed` + claimsSucceeded++ + return notification + } catch (err) { + if (lastClaimResult !== `no_work` && lastClaimResult !== `error`) { + lastClaimResult = `error` + claimsFailed++ + } + throw err } - if (notification.done) return null - return notification } const run = async (): Promise => { @@ -208,15 +296,19 @@ export function createPullWakeRunner( offset: currentOffset, signal, }) + streamConnected = true + streamConnectedSince = new Date().toISOString() for await (const event of response.jsonStream()) { if (signal.aborted) break if (event?.type !== `wake`) continue + eventsReceived++ const notification = await claimWake(event, signal) if (notification) { config.runtime.dispatchWake(notification, { claimHeaders: resolveClaimHeaders, claimTokenHeader: config.claimTokenHeader, }) + lastDispatchAt = new Date().toISOString() await config.runtime.drainWakes() } if (response.offset !== undefined) currentOffset = response.offset @@ -226,9 +318,11 @@ export function createPullWakeRunner( }) } catch (err) { if (!signal.aborted) { + reconnectCount++ reportError(err) } } finally { + streamConnected = false stopHeartbeat() response = null controller = null @@ -239,6 +333,7 @@ export function createPullWakeRunner( start() { if (loop) return controller = new AbortController() + startedAt = new Date().toISOString() startHeartbeat(controller.signal) loop = run().finally(() => { loop = null @@ -263,5 +358,12 @@ export function createPullWakeRunner( get offset() { return currentOffset }, + getHealth(): PullWakeRunnerHealth { + return { + running: loop !== null, + offset: currentOffset, + ...buildDiagnostics(), + } + }, } } diff --git a/packages/agents-runtime/test/pull-wake-runner.test.ts b/packages/agents-runtime/test/pull-wake-runner.test.ts index e844fc6bed..ee73839096 100644 --- a/packages/agents-runtime/test/pull-wake-runner.test.ts +++ b/packages/agents-runtime/test/pull-wake-runner.test.ts @@ -152,6 +152,71 @@ describe(`createPullWakeRunner`, () => { expect(runner.offset).toBe(`42`) }) + it(`exposes diagnostics via getHealth()`, async () => { + const event: PullWakeEvent = { + type: `wake`, + subscription_id: `runner:runner-1`, + stream: `chat/one/main`, + generation: 7, + ts: 123, + } + const notification: WakeNotification = { + consumerId: `wake-1`, + epoch: 7, + wakeId: `wake-1`, + streamPath: `/chat/one/main`, + streams: [{ path: `/chat/one/main`, offset: `12` }], + callback: `http://server/_electric/callback-forward/wake-1`, + claimToken: `claim-token`, + entity: { + type: `chat`, + status: `idle`, + url: `/chat/one`, + streams: { main: `/chat/one/main`, error: `/chat/one/error` }, + }, + } + const fetchMock = vi.fn(async (_input: RequestInfo | URL) => + Response.json(notification) + ) + vi.stubGlobal(`fetch`, fetchMock) + const streamFactory = vi.fn(async () => ({ + offset: `42`, + async *jsonStream() { + yield event + }, + closed: Promise.resolve(), + })) + + const runner = createPullWakeRunner({ + baseUrl: `http://server`, + runnerId: `runner-1`, + runtime: { + dispatchWake: vi.fn(), + drainWakes: vi.fn(async () => undefined), + abortWakes: vi.fn(), + }, + heartbeatIntervalMs: 0, + streamFactory, + }) + + const healthBefore = runner.getHealth() + expect(healthBefore.running).toBe(false) + expect(healthBefore.started_at).toBeNull() + expect(healthBefore.events_received).toBe(0) + + runner.start() + await runner.waitForStopped() + + const healthAfter = runner.getHealth() + expect(healthAfter.running).toBe(false) + expect(healthAfter.started_at).not.toBeNull() + expect(healthAfter.events_received).toBe(1) + expect(healthAfter.claims_succeeded).toBe(1) + expect(healthAfter.last_claim_result).toBe(`claimed`) + expect(healthAfter.last_dispatch_at).not.toBeNull() + expect(healthAfter.offset).toBe(`42`) + }) + it(`preserves base URL query parameters on stream, claim, and heartbeat requests`, async () => { const fetchMock = vi.fn(async (_input: RequestInfo | URL) => { return new Response(null, { status: 204 }) diff --git a/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md b/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md index df95c25929..6d2f10b910 100644 --- a/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md +++ b/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md @@ -1,6 +1,6 @@ # Pull-Wake Runner Health Check Implementation Plan -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]` / `- [x]`) syntax for tracking. **Goal:** Add comprehensive diagnostics to the pull-wake runner system: client-side state tracking reported via heartbeats, server-side storage + aggregation, and a `GET /_electric/runners/:id/health` endpoint. Also rename `owner_user_id` → `owner_principal` throughout the runners system, storing principal URLs instead of keys. @@ -16,7 +16,7 @@ - Create: `packages/agents-server/drizzle/0007_runner_diagnostics_and_principal.sql` -- [ ] **Step 1: Write the migration SQL** +- [x] **Step 1: Write the migration SQL** Existing `owner_user_id` values are key-form strings (e.g., `local-desktop`). The new column expects principal URLs (e.g., `/principal/system%3Alocal-desktop`). Since we have no backwards compatibility, the migration deletes existing runner rows — runners are ephemeral and will re-register on next startup. Must also clean up dependent tables (`consumer_claims` and `entity_dispatch_state`) since there are no FK constraints to cascade the deletes. @@ -36,7 +36,7 @@ CREATE INDEX idx_runners_owner_principal ON runners (tenant_id, owner_principal) ALTER TABLE runners ADD COLUMN diagnostics jsonb; ``` -- [ ] **Step 2: Commit** +- [x] **Step 2: Commit** ```bash git add packages/agents-server/drizzle/0007_runner_diagnostics_and_principal.sql @@ -52,7 +52,7 @@ git commit -m "feat(agents-server): add migration for runner diagnostics and pri - Modify: `packages/agents-server/src/db/schema.ts:104-144` - Modify: `packages/agents-server/src/electric-agents-types.ts:99-136` -- [ ] **Step 1: Update the `runners` table in Drizzle schema** +- [x] **Step 1: Update the `runners` table in Drizzle schema** In `packages/agents-server/src/db/schema.ts`, change the `runners` table definition: @@ -73,7 +73,7 @@ In `packages/agents-server/src/db/schema.ts`, change the `runners` table definit index(`idx_runners_owner_principal`).on(table.tenantId, table.ownerPrincipal), ``` -- [ ] **Step 2: Update the `ElectricAgentsRunner` type** +- [x] **Step 2: Update the `ElectricAgentsRunner` type** In `packages/agents-server/src/electric-agents-types.ts`, update the runner types: @@ -134,7 +134,7 @@ export interface RegisterRunnerRequest { } ``` -- [ ] **Step 3: Add `RunnerHealthResponse` and `RunnerHealthStatus` types** +- [x] **Step 3: Add `RunnerHealthResponse` and `RunnerHealthStatus` types** Append to `packages/agents-server/src/electric-agents-types.ts`: @@ -178,7 +178,7 @@ export interface RunnerHealthResponse { } ``` -- [ ] **Step 4: Commit** +- [x] **Step 4: Commit** ```bash git add packages/agents-server/src/db/schema.ts packages/agents-server/src/electric-agents-types.ts @@ -193,7 +193,7 @@ git commit -m "feat(agents-server): rename owner_user_id to owner_principal in s - Modify: `packages/agents-server/src/entity-registry.ts:74-81, 132-190, 193-217, 1148-1168` -- [ ] **Step 1: Rename `RegisterRunnerInput.ownerUserId` → `ownerPrincipal`** +- [x] **Step 1: Rename `RegisterRunnerInput.ownerUserId` → `ownerPrincipal`** In `packages/agents-server/src/entity-registry.ts` (line 74-81): @@ -218,7 +218,7 @@ export interface RegisterRunnerInput { } ``` -- [ ] **Step 2: Add `diagnostics` to `HeartbeatRunnerInput`** +- [x] **Step 2: Add `diagnostics` to `HeartbeatRunnerInput`** In `packages/agents-server/src/entity-registry.ts` (line 83-89): @@ -242,7 +242,7 @@ export interface HeartbeatRunnerInput { } ``` -- [ ] **Step 3: Update `createRunner` to use `ownerPrincipal`** +- [x] **Step 3: Update `createRunner` to use `ownerPrincipal`** In the `createRunner` method (line 132-167), replace all `ownerUserId` → `ownerPrincipal` references: @@ -285,7 +285,7 @@ In the `createRunner` method (line 132-167), replace all `ownerUserId` → `owne } ``` -- [ ] **Step 4: Update `listRunners` filter** +- [x] **Step 4: Update `listRunners` filter** In `listRunners` (line 178-191): @@ -308,7 +308,7 @@ In `listRunners` (line 178-191): } ``` -- [ ] **Step 5: Update `heartbeatRunner` to store diagnostics** +- [x] **Step 5: Update `heartbeatRunner` to store diagnostics** In `heartbeatRunner` (line 193-217): @@ -343,7 +343,7 @@ In `heartbeatRunner` (line 193-217): } ``` -- [ ] **Step 6: Add `getActiveClaimsForRunner` query** +- [x] **Step 6: Add `getActiveClaimsForRunner` query** Add after `materializeReleasedClaim` (around line 367): @@ -365,7 +365,7 @@ Add after `materializeReleasedClaim` (around line 367): } ``` -- [ ] **Step 7: Add `getDispatchStatsForRunner` query** +- [x] **Step 7: Add `getDispatchStatsForRunner` query** Add right after `getActiveClaimsForRunner`: @@ -405,7 +405,7 @@ Add right after `getActiveClaimsForRunner`: } ``` -- [ ] **Step 8: Update `rowToRunner` to include `owner_principal` and `diagnostics`** +- [x] **Step 8: Update `rowToRunner` to include `owner_principal` and `diagnostics`** In `rowToRunner` (line 1148-1168): @@ -457,7 +457,7 @@ In `rowToRunner` (line 1148-1168): } ``` -- [ ] **Step 9: Commit** +- [x] **Step 9: Commit** ```bash git add packages/agents-server/src/entity-registry.ts @@ -474,7 +474,7 @@ git commit -m "feat(agents-server): update entity registry for principal rename, - Modify: `packages/agents-server/src/routing/dispatch-policy.ts:127` - Modify: `packages/agents-server/src/utils/server-utils.ts:130-134` -- [ ] **Step 1: Update the registration body schema** +- [x] **Step 1: Update the registration body schema** In `packages/agents-server/src/routing/runners-router.ts` (line 36-53): @@ -519,7 +519,7 @@ const registerRunnerBodySchema = Type.Object({ }) ``` -- [ ] **Step 2: Add `diagnostics` to heartbeat body schema** +- [x] **Step 2: Add `diagnostics` to heartbeat body schema** In the `heartbeatBodySchema` (line 55-60): @@ -541,7 +541,7 @@ const heartbeatBodySchema = Type.Object({ }) ``` -- [ ] **Step 3: Add the health route** +- [x] **Step 3: Add the health route** After the existing routes (line 90), add: @@ -549,7 +549,7 @@ After the existing routes (line 90), add: runnersRouter.get(`/:id/health`, runnerHealth) ``` -- [ ] **Step 4: Add `principalKeyFromUrl` import** +- [x] **Step 4: Add `principalKeyFromUrl` import** Add to the imports at the top of `runners-router.ts`: @@ -557,7 +557,7 @@ Add to the imports at the top of `runners-router.ts`: import { principalKeyFromUrl } from '../principal.js' ``` -- [ ] **Step 5: Update `registerRunner` handler to use `owner_principal` with strict URL validation** +- [x] **Step 5: Update `registerRunner` handler to use `owner_principal` with strict URL validation** No backwards compatibility for key-form principals. If `owner_principal` is provided it must be a valid principal URL accepted by `principalKeyFromUrl()` (e.g., `/principal/user%3Aalice`); otherwise the server derives it from `ctx.principal.url`. Callers must send URLs. @@ -643,7 +643,7 @@ async function registerRunner( } ``` -- [ ] **Step 6: Update `listRunners` handler** +- [x] **Step 6: Update `listRunners` handler** In `listRunners` (line 138-154): @@ -693,7 +693,7 @@ async function listRunners( } ``` -- [ ] **Step 7: Update heartbeat handler to pass diagnostics** +- [x] **Step 7: Update heartbeat handler to pass diagnostics** In `heartbeat` (line 165-185), add `diagnostics` to the `heartbeatRunner` call: @@ -709,7 +709,7 @@ const runner = await ctx.entityManager.registry.heartbeatRunner({ }) ``` -- [ ] **Step 8: Update `assertRunnerOwnerIfAuthenticated` to use `principal.url`** +- [x] **Step 8: Update `assertRunnerOwnerIfAuthenticated` to use `principal.url`** In `assertRunnerOwnerIfAuthenticated` (line 297-308): @@ -742,7 +742,7 @@ function assertRunnerOwnerIfAuthenticated( } ``` -- [ ] **Step 9: Update all callers of `assertRunnerOwnerIfAuthenticated`** +- [x] **Step 9: Update all callers of `assertRunnerOwnerIfAuthenticated`** Change all calls from `runner.owner_user_id` → `runner.owner_principal`: @@ -752,7 +752,7 @@ In `heartbeat` (line 171): `assertRunnerOwnerIfAuthenticated(ctx, existing.owner In `setRunnerStatus` (line 208): `assertRunnerOwnerIfAuthenticated(ctx, existing.owner_principal)` -- [ ] **Step 10: Update claim auth check** +- [x] **Step 10: Update claim auth check** In `claimWake` (line 225): @@ -763,7 +763,7 @@ In `claimWake` (line 225): if (ctx.principal && runner.owner_principal !== ctx.principal.url) { ``` -- [ ] **Step 11: Update `assertDispatchPolicyAllowed` in dispatch-policy.ts** +- [x] **Step 11: Update `assertDispatchPolicyAllowed` in dispatch-policy.ts** In `packages/agents-server/src/routing/dispatch-policy.ts` (line 127): @@ -774,7 +774,7 @@ In `packages/agents-server/src/routing/dispatch-policy.ts` (line 127): if (ctx.principal && runner.owner_principal !== ctx.principal.url) { ``` -- [ ] **Step 12: Update runners Shape column allowlist in server-utils.ts** +- [x] **Step 12: Update runners Shape column allowlist in server-utils.ts** In `packages/agents-server/src/utils/server-utils.ts` (line 131-133): @@ -785,7 +785,7 @@ In `packages/agents-server/src/utils/server-utils.ts` (line 131-133): `"tenant_id","id","owner_principal","label","kind","admin_status","wake_stream","wake_stream_offset","last_seen_at","liveness_lease_expires_at","diagnostics","created_at","updated_at"` ``` -- [ ] **Step 13: Implement `runnerHealth` handler** +- [x] **Step 13: Implement `runnerHealth` handler** Add at the bottom of the file, before `notificationFromClaim`: @@ -890,7 +890,7 @@ async function runnerHealth( } ``` -- [ ] **Step 14: Commit** +- [x] **Step 14: Commit** ```bash git add packages/agents-server/src/routing/runners-router.ts packages/agents-server/src/routing/dispatch-policy.ts packages/agents-server/src/utils/server-utils.ts @@ -905,7 +905,7 @@ git commit -m "feat(agents-server): update runners router, dispatch policy, and - Modify: `packages/agents-runtime/src/pull-wake-runner.ts` -- [ ] **Step 1: Add `PullWakeRunnerHealth` interface and diagnostics tracking** +- [x] **Step 1: Add `PullWakeRunnerHealth` interface and diagnostics tracking** In `packages/agents-runtime/src/pull-wake-runner.ts`, after the existing `PullWakeRunner` interface (line 48-54), add: @@ -944,7 +944,7 @@ export interface PullWakeRunner { } ``` -- [ ] **Step 2: Add diagnostic state variables inside `createPullWakeRunner`** +- [x] **Step 2: Add diagnostic state variables inside `createPullWakeRunner`** After the existing `let currentOffset = config.offset` (line 63), add: @@ -966,7 +966,7 @@ let claimsSkipped = 0 let claimsFailed = 0 ``` -- [ ] **Step 3: Build the diagnostics snapshot function** +- [x] **Step 3: Build the diagnostics snapshot function** Add after the diagnostic variables: @@ -993,7 +993,7 @@ const buildDiagnostics = (): Omit< }) ``` -- [ ] **Step 4: Update `heartbeat` to report diagnostics and track heartbeat state** +- [x] **Step 4: Update `heartbeat` to report diagnostics and track heartbeat state** Replace the existing `heartbeat` function (line 106-131): @@ -1031,7 +1031,7 @@ const heartbeat = async (signal: AbortSignal): Promise => { } ``` -- [ ] **Step 5: Update `reportError` to track errors** +- [x] **Step 5: Update `reportError` to track errors** Replace the existing `reportError` (line 101-104): @@ -1044,7 +1044,7 @@ const reportError = (err: unknown): void => { } ``` -- [ ] **Step 6: Update `claimWake` to track claim results** +- [x] **Step 6: Update `claimWake` to track claim results** Replace the existing `claimWake` (line 170-200): @@ -1105,7 +1105,7 @@ const claimWake = async ( } ``` -- [ ] **Step 7: Update the `run` function to track stream and event state** +- [x] **Step 7: Update the `run` function to track stream and event state** Replace the existing `run` function (line 202-236): @@ -1153,7 +1153,7 @@ const run = async (): Promise => { } ``` -- [ ] **Step 8: Update `start()` to record `startedAt`** +- [x] **Step 8: Update `start()` to record `startedAt`** In the returned object's `start()` method (line 239-244): @@ -1169,7 +1169,7 @@ In the returned object's `start()` method (line 239-244): }, ``` -- [ ] **Step 9: Add `getHealth()` to the returned object** +- [x] **Step 9: Add `getHealth()` to the returned object** Add after the `offset` getter: @@ -1183,7 +1183,7 @@ Add after the `offset` getter: }, ``` -- [ ] **Step 10: Update the runtime index exports** +- [x] **Step 10: Update the runtime index exports** In `packages/agents-runtime/src/index.ts`, add `PullWakeRunnerHealth` to the exports (line 238-243): @@ -1205,7 +1205,7 @@ export type { } from './pull-wake-runner' ``` -- [ ] **Step 11: Commit** +- [x] **Step 11: Commit** ```bash git add packages/agents-runtime/src/pull-wake-runner.ts packages/agents-runtime/src/index.ts @@ -1222,7 +1222,7 @@ git commit -m "feat(agents-runtime): add diagnostics tracking and getHealth() to - Modify: `packages/electric-ax/src/start.ts:131-139, 379, 395` - Modify: `packages/agents-desktop/src/main.ts:219-274, 1544-1582` -- [ ] **Step 1: Update `BuiltinAgentsServerOptions` in agents/server.ts** +- [x] **Step 1: Update `BuiltinAgentsServerOptions` in agents/server.ts** In `packages/agents/src/server.ts` (line 40-51): @@ -1241,7 +1241,7 @@ In `packages/agents/src/server.ts` (line 40-51): registerRunner?: boolean ``` -- [ ] **Step 2: Update `registerPullWakeRunner` to use `owner_principal`** +- [x] **Step 2: Update `registerPullWakeRunner` to use `owner_principal`** In `packages/agents/src/server.ts` (line 393-422): @@ -1264,7 +1264,7 @@ In `packages/agents/src/server.ts` (line 393-422): }), ``` -- [ ] **Step 3: Update electric-ax/src/start.ts** +- [x] **Step 3: Update electric-ax/src/start.ts** In `packages/electric-ax/src/start.ts`: @@ -1319,7 +1319,7 @@ Update the `BuiltinAgentsServer` call (line 395): ownerPrincipal, ``` -- [ ] **Step 4: Update desktop env var and function names** +- [x] **Step 4: Update desktop env var and function names** In `packages/agents-desktop/src/main.ts`: @@ -1384,7 +1384,7 @@ const runnerOwnerPrincipal = runnerOwnerPrincipalFromHeaders(runtimeHeaders) Update log messages referencing `owner user id` → `owner principal`. -- [ ] **Step 5: Commit** +- [x] **Step 5: Commit** ```bash git add packages/agents/src/server.ts packages/electric-ax/src/start.ts packages/agents-desktop/src/main.ts @@ -1404,7 +1404,7 @@ git commit -m "feat(agents, electric-ax, agents-desktop): rename ownerUserId to - Modify: `packages/agents-server/test/horton-spawn-worker.test.ts` - Modify: `packages/agents-server/test/dispatch-policy-routing.test.ts` -- [ ] **Step 1: Update runners-router.test.ts — principal rename and context** +- [x] **Step 1: Update runners-router.test.ts — principal rename and context** In `packages/agents-server/test/runners-router.test.ts`: @@ -1439,7 +1439,7 @@ Update all test assertions that reference `owner_user_id`: - Line 128-129: `ownerUserId: `user:owner@example.com`` → `ownerPrincipal: `/principal/user%3Aowner%40example.com`` - Line 158-159: same replacement -- [ ] **Step 2: Add health endpoint test to runners-router.test.ts** +- [x] **Step 2: Add health endpoint test to runners-router.test.ts** Add to the `runner routes` describe block: @@ -1523,7 +1523,7 @@ it(`returns unhealthy when runner lease is expired`, async () => { }) ``` -- [ ] **Step 3: Add `getHealth()` test to pull-wake-runner.test.ts** +- [x] **Step 3: Add `getHealth()` test to pull-wake-runner.test.ts** Add to the `createPullWakeRunner` describe block in `packages/agents-runtime/test/pull-wake-runner.test.ts`: @@ -1594,7 +1594,7 @@ it(`exposes diagnostics via getHealth()`, async () => { }) ``` -- [ ] **Step 4: Update horton-pull-wake-e2e.test.ts for principal rename** +- [x] **Step 4: Update horton-pull-wake-e2e.test.ts for principal rename** In `packages/agents-server/test/horton-pull-wake-e2e.test.ts` (line 133): @@ -1605,7 +1605,7 @@ In `packages/agents-server/test/horton-pull-wake-e2e.test.ts` (line 133): ownerPrincipal: testPrincipal.url, ``` -- [ ] **Step 5: Update horton-title-generation.test.ts and horton-spawn-worker.test.ts** +- [x] **Step 5: Update horton-title-generation.test.ts and horton-spawn-worker.test.ts** In `packages/agents-server/test/horton-title-generation.test.ts` (line 39): @@ -1625,7 +1625,7 @@ In `packages/agents-server/test/horton-spawn-worker.test.ts` (line 39): ownerPrincipal: `/principal/system%3Atest-user`, ``` -- [ ] **Step 6: Update dispatch-policy-routing.test.ts** +- [x] **Step 6: Update dispatch-policy-routing.test.ts** In `packages/agents-server/test/dispatch-policy-routing.test.ts` (line 71): @@ -1636,7 +1636,7 @@ In `packages/agents-server/test/dispatch-policy-routing.test.ts` (line 71): owner_principal: `/principal/user%3Aowner%40example.com`, ``` -- [ ] **Step 7: Run all tests** +- [x] **Step 7: Run all tests** Run: `cd packages/agents-runtime && pnpm vitest run test/pull-wake-runner.test.ts --reporter=dot` @@ -1644,7 +1644,7 @@ Run: `cd packages/agents-server && pnpm vitest run test/runners-router.test.ts - Expected: All tests PASS -- [ ] **Step 8: Commit** +- [x] **Step 8: Commit** ```bash git add packages/agents-server/test/ packages/agents-runtime/test/ @@ -1655,33 +1655,33 @@ git commit -m "test: update all tests for principal rename and add health endpoi ### Task 8: Typecheck and final verification -- [ ] **Step 1: Typecheck agents-runtime** +- [x] **Step 1: Typecheck agents-runtime** Run: `pnpm -C packages/agents-runtime build` Expected: No errors -- [ ] **Step 2: Typecheck agents-server** +- [x] **Step 2: Typecheck agents-server** Run: `pnpm --filter @electric-ax/agents-server typecheck` Expected: No errors -- [ ] **Step 3: Typecheck agents** +- [x] **Step 3: Typecheck agents** Run: `pnpm --filter @electric-ax/agents typecheck` Expected: No errors -- [ ] **Step 4: Typecheck agents-desktop** +- [x] **Step 4: Typecheck agents-desktop** Run: `pnpm --filter @electric-ax/agents-desktop typecheck` Expected: No errors -- [ ] **Step 5: Run unit tests** +- [x] **Step 5: Run unit tests** Run: `cd packages/agents-runtime && pnpm vitest run test/pull-wake-runner.test.ts --reporter=dot` Run: `cd packages/agents-server && pnpm vitest run test/runners-router.test.ts --reporter=dot` Expected: All PASS -- [ ] **Step 6: Fix any issues and commit** +- [x] **Step 6: Fix any issues and commit** If any typecheck or test failures, fix and commit: diff --git a/packages/agents-server/drizzle/0007_runner_diagnostics_and_principal.sql b/packages/agents-server/drizzle/0007_runner_diagnostics_and_principal.sql new file mode 100644 index 0000000000..b2573b9709 --- /dev/null +++ b/packages/agents-server/drizzle/0007_runner_diagnostics_and_principal.sql @@ -0,0 +1,22 @@ +UPDATE consumer_claims +SET status = 'expired', updated_at = NOW() +WHERE status = 'active' AND runner_id IS NOT NULL; +--> statement-breakpoint +UPDATE entity_dispatch_state +SET active_runner_id = NULL, + active_consumer_id = NULL, + active_epoch = NULL, + active_claimed_at = NULL, + active_lease_expires_at = NULL, + updated_at = NOW() +WHERE active_runner_id IS NOT NULL; +--> statement-breakpoint +DELETE FROM runners; +--> statement-breakpoint +ALTER TABLE runners RENAME COLUMN owner_user_id TO owner_principal; +--> statement-breakpoint +DROP INDEX IF EXISTS idx_runners_owner_user_id; +--> statement-breakpoint +CREATE INDEX idx_runners_owner_principal ON runners (tenant_id, owner_principal); +--> statement-breakpoint +ALTER TABLE runners ADD COLUMN diagnostics jsonb; diff --git a/packages/agents-server/drizzle/meta/_journal.json b/packages/agents-server/drizzle/meta/_journal.json index 8331830f37..cd53b818cc 100644 --- a/packages/agents-server/drizzle/meta/_journal.json +++ b/packages/agents-server/drizzle/meta/_journal.json @@ -50,6 +50,13 @@ "when": 1776268800000, "tag": "0006_principals", "breakpoints": true + }, + { + "idx": 7, + "version": "7", + "when": 1778899200000, + "tag": "0007_runner_diagnostics_and_principal", + "breakpoints": true } ] } diff --git a/packages/agents-server/src/db/schema.ts b/packages/agents-server/src/db/schema.ts index a866efa288..d58cbb6aff 100644 --- a/packages/agents-server/src/db/schema.ts +++ b/packages/agents-server/src/db/schema.ts @@ -106,7 +106,7 @@ export const runners = pgTable( { tenantId: text(`tenant_id`).notNull().default(`default`), id: text(`id`).notNull(), - ownerUserId: text(`owner_user_id`).notNull(), + ownerPrincipal: text(`owner_principal`).notNull(), label: text(`label`).notNull(), kind: text(`kind`).notNull().default(`local`), adminStatus: text(`admin_status`).notNull().default(`enabled`), @@ -116,6 +116,7 @@ export const runners = pgTable( livenessLeaseExpiresAt: timestamp(`liveness_lease_expires_at`, { withTimezone: true, }), + diagnostics: jsonb(`diagnostics`), createdAt: timestamp(`created_at`, { withTimezone: true }) .notNull() .defaultNow(), @@ -126,7 +127,10 @@ export const runners = pgTable( (table) => [ primaryKey({ columns: [table.tenantId, table.id] }), unique(`uq_runners_wake_stream`).on(table.tenantId, table.wakeStream), - index(`idx_runners_owner_user_id`).on(table.tenantId, table.ownerUserId), + index(`idx_runners_owner_principal`).on( + table.tenantId, + table.ownerPrincipal + ), index(`idx_runners_admin_status`).on(table.tenantId, table.adminStatus), index(`idx_runners_liveness_lease_expires_at`).on( table.tenantId, diff --git a/packages/agents-server/src/electric-agents-types.ts b/packages/agents-server/src/electric-agents-types.ts index e32d771a85..8e0e98bdff 100644 --- a/packages/agents-server/src/electric-agents-types.ts +++ b/packages/agents-server/src/electric-agents-types.ts @@ -105,7 +105,7 @@ export interface RunnerActiveClaim { export interface ElectricAgentsRunner { id: string - owner_user_id: string + owner_principal: string label: string kind: RunnerKind admin_status: RunnerAdminStatus @@ -115,13 +115,14 @@ export interface ElectricAgentsRunner { active_claims?: Array wake_stream: string wake_stream_offset?: string + diagnostics?: Record created_at: string updated_at: string } export interface RegisterRunnerRequest { id: string - owner_user_id: string + owner_principal: string label: string kind?: RunnerKind admin_status?: RunnerAdminStatus @@ -133,6 +134,45 @@ export interface RunnerHeartbeatRequest { wake_stream_offset?: string wakeStreamOffset?: string liveness_lease_expires_at?: string + diagnostics?: Record +} + +export type RunnerHealthStatus = `healthy` | `degraded` | `unhealthy` + +export interface RunnerHealthResponse { + runner: { + id: string + admin_status: RunnerAdminStatus + liveness_status: RunnerLiveness | `expired` + lease_expires_at: string | null + lease_remaining_ms: number | null + wake_stream: string + wake_stream_offset: string | null + last_seen_at: string | null + created_at: string + } + client: Record | null + claims: { + active_count: number + active: Array<{ + consumer_id: string + epoch: number + entity_url: string + stream_path: string + claimed_at: string + last_heartbeat_at: string | null + lease_expires_at: string | null + }> + } + dispatch: { + entities_with_active_claim: number + entities_with_outstanding_wake: number + entities_with_pending_work: number + } + health: { + status: RunnerHealthStatus + issues: Array + } } export interface EntityDispatchState { diff --git a/packages/agents-server/src/entity-registry.ts b/packages/agents-server/src/entity-registry.ts index 46ccc2ccdf..8d7732d72a 100644 --- a/packages/agents-server/src/entity-registry.ts +++ b/packages/agents-server/src/entity-registry.ts @@ -73,7 +73,7 @@ export interface TagStreamOutboxRow { export interface RegisterRunnerInput { id: string - ownerUserId: string + ownerPrincipal: string label: string kind?: RunnerKind adminStatus?: RunnerAdminStatus @@ -86,6 +86,7 @@ export interface HeartbeatRunnerInput { livenessLeaseExpiresAt?: Date leaseMs?: number wakeStreamOffset?: string + diagnostics?: Record } export interface MaterializeActiveClaimInput { @@ -140,7 +141,7 @@ export class PostgresRegistry { .values({ tenantId: this.tenantId, id: input.id, - ownerUserId: input.ownerUserId, + ownerPrincipal: input.ownerPrincipal, label: input.label, kind: input.kind ?? `local`, adminStatus: input.adminStatus ?? `enabled`, @@ -150,7 +151,7 @@ export class PostgresRegistry { .onConflictDoUpdate({ target: [runners.tenantId, runners.id], set: { - ownerUserId: input.ownerUserId, + ownerPrincipal: input.ownerPrincipal, label: input.label, kind: input.kind ?? `local`, adminStatus: input.adminStatus ?? `enabled`, @@ -176,11 +177,11 @@ export class PostgresRegistry { } async listRunners(filter?: { - ownerUserId?: string + ownerPrincipal?: string }): Promise> { const conditions = [eq(runners.tenantId, this.tenantId)] - if (filter?.ownerUserId) { - conditions.push(eq(runners.ownerUserId, filter.ownerUserId)) + if (filter?.ownerPrincipal) { + conditions.push(eq(runners.ownerPrincipal, filter.ownerPrincipal)) } const rows = await this.db .select() @@ -206,6 +207,9 @@ export class PostgresRegistry { ...(input.wakeStreamOffset !== undefined ? { wakeStreamOffset: input.wakeStreamOffset } : {}), + ...(input.diagnostics !== undefined + ? { diagnostics: input.diagnostics } + : {}), updatedAt: now, }) .where( @@ -366,6 +370,54 @@ export class PostgresRegistry { return claim } + async getActiveClaimsForRunner( + runnerId: string + ): Promise> { + const rows = await this.db + .select() + .from(consumerClaims) + .where( + and( + eq(consumerClaims.tenantId, this.tenantId), + eq(consumerClaims.runnerId, runnerId), + eq(consumerClaims.status, `active`) + ) + ) + return rows.map((row) => this.rowToConsumerClaim(row)) + } + + async getDispatchStatsForRunner(runnerId: string): Promise<{ + entities_with_active_claim: number + entities_with_outstanding_wake: number + entities_with_pending_work: number + }> { + const rows = await this.db + .select() + .from(entityDispatchState) + .where( + and( + eq(entityDispatchState.tenantId, this.tenantId), + eq(entityDispatchState.activeRunnerId, runnerId) + ) + ) + + let activeClaim = 0 + let outstandingWake = 0 + let pendingWork = 0 + for (const row of rows) { + if (row.activeConsumerId) activeClaim++ + if (row.outstandingWakeId && !row.activeConsumerId) outstandingWake++ + const pending = row.pendingSourceStreams as Array | null + if (pending && pending.length > 0) pendingWork++ + } + + return { + entities_with_active_claim: activeClaim, + entities_with_outstanding_wake: outstandingWake, + entities_with_pending_work: pendingWork, + } + } + private entityTypeWhere(name: string) { return and( eq(entityTypes.tenantId, this.tenantId), @@ -1150,7 +1202,7 @@ export class PostgresRegistry { const livenessExpiry = row.livenessLeaseExpiresAt?.getTime() return { id: row.id, - owner_user_id: row.ownerUserId, + owner_principal: row.ownerPrincipal, label: row.label, kind: assertRunnerKind(row.kind), admin_status: assertRunnerAdminStatus(row.adminStatus), @@ -1162,6 +1214,7 @@ export class PostgresRegistry { liveness_lease_expires_at: row.livenessLeaseExpiresAt?.toISOString(), wake_stream: row.wakeStream, wake_stream_offset: row.wakeStreamOffset ?? undefined, + diagnostics: (row.diagnostics as Record) ?? undefined, created_at: row.createdAt.toISOString(), updated_at: row.updatedAt.toISOString(), } diff --git a/packages/agents-server/src/routing/dispatch-policy.ts b/packages/agents-server/src/routing/dispatch-policy.ts index afbe38273f..560129044a 100644 --- a/packages/agents-server/src/routing/dispatch-policy.ts +++ b/packages/agents-server/src/routing/dispatch-policy.ts @@ -124,7 +124,7 @@ export async function assertDispatchPolicyAllowed( 404 ) } - if (ctx.principal && runner.owner_user_id !== ctx.principal.key) { + if (ctx.principal && runner.owner_principal !== ctx.principal.url) { throw new ElectricAgentsError( ErrCodeUnauthorized, `Runner dispatch requires the authenticated owner`, diff --git a/packages/agents-server/src/routing/runners-router.ts b/packages/agents-server/src/routing/runners-router.ts index 06d4c73973..d74da413c3 100644 --- a/packages/agents-server/src/routing/runners-router.ts +++ b/packages/agents-server/src/routing/runners-router.ts @@ -13,7 +13,7 @@ import { import { routeBody, withSchema } from './schema.js' import { subscriptionIdForDispatchTarget } from './dispatch-policy.js' import { withLeadingSlash } from './tenant-stream-paths.js' -import { principalFromCreatedBy } from '../principal.js' +import { principalFromCreatedBy, principalKeyFromUrl } from '../principal.js' import type { JsonRouteRequest } from './schema.js' import type { RouterType } from 'itty-router' import type { TenantContext } from './context.js' @@ -35,7 +35,7 @@ export type RunnersRoutes = RouterType< const registerRunnerBodySchema = Type.Object({ id: Type.String(), - owner_user_id: Type.Optional(Type.String()), + owner_principal: Type.Optional(Type.String()), label: Type.String(), kind: Type.Optional( Type.Union([ @@ -57,6 +57,7 @@ const heartbeatBodySchema = Type.Object({ wake_stream_offset: Type.Optional(Type.String()), wakeStreamOffset: Type.Optional(Type.String()), liveness_lease_expires_at: Type.Optional(Type.String()), + diagnostics: Type.Optional(Type.Record(Type.String(), Type.Unknown())), }) const claimBodySchema = Type.Object( @@ -83,6 +84,7 @@ export const runnersRouter: RunnersRoutes = Router< runnersRouter.post(`/`, withSchema(registerRunnerBodySchema), registerRunner) runnersRouter.get(`/`, listRunners) +runnersRouter.get(`/:id/health`, runnerHealth) runnersRouter.get(`/:id`, getRunner) runnersRouter.post(`/:id/heartbeat`, withSchema(heartbeatBodySchema), heartbeat) runnersRouter.post(`/:id/enable`, setEnabled) @@ -105,25 +107,32 @@ async function registerRunner( ctx: TenantContext ): Promise { const parsed = routeBody(request) - const ownerUserId = parsed.owner_user_id ?? ctx.principal?.key - if (!ownerUserId) { + const ownerPrincipal = parsed.owner_principal ?? ctx.principal?.url + if (!ownerPrincipal) { throw new ElectricAgentsError( ErrCodeInvalidRequest, - `owner_user_id is required when no authenticated user is present`, + `owner_principal is required when no authenticated principal is present`, 400 ) } - if (ctx.principal && ownerUserId !== ctx.principal.key) { + if (!principalKeyFromUrl(ownerPrincipal)) { + throw new ElectricAgentsError( + ErrCodeInvalidRequest, + `owner_principal must be a valid principal URL accepted by principalKeyFromUrl() (e.g. /principal/user%3Aalice), got: ${ownerPrincipal}`, + 400 + ) + } + if (ctx.principal && ownerPrincipal !== ctx.principal.url) { throw new ElectricAgentsError( ErrCodeUnauthorized, - `owner_user_id must match the authenticated user`, + `owner_principal must match the authenticated principal`, 403 ) } const runner = await ctx.entityManager.registry.createRunner({ id: parsed.id, - ownerUserId, + ownerPrincipal, label: parsed.label, kind: parsed.kind, adminStatus: parsed.admin_status, @@ -139,16 +148,23 @@ async function listRunners( request: RunnersRouteRequest, ctx: TenantContext ): Promise { - const requestedOwner = firstQueryValue(request.query.owner_user_id) - if (ctx.principal && requestedOwner && requestedOwner !== ctx.principal.key) { + const requestedOwner = firstQueryValue(request.query.owner_principal) + if (requestedOwner && !principalKeyFromUrl(requestedOwner)) { + throw new ElectricAgentsError( + ErrCodeInvalidRequest, + `owner_principal must be a valid principal URL (e.g. /principal/user%3Aalice), got: ${requestedOwner}`, + 400 + ) + } + if (ctx.principal && requestedOwner && requestedOwner !== ctx.principal.url) { throw new ElectricAgentsError( ErrCodeUnauthorized, - `owner_user_id must match the authenticated user`, + `owner_principal must match the authenticated principal`, 403 ) } const runners = await ctx.entityManager.registry.listRunners({ - ownerUserId: ctx.principal?.key ?? requestedOwner, + ownerPrincipal: ctx.principal?.url ?? requestedOwner, }) return json(runners) } @@ -158,7 +174,7 @@ async function getRunner( ctx: TenantContext ): Promise { const runner = await requireRunner(ctx, routeParam(request, `id`)) - assertRunnerOwnerIfAuthenticated(ctx, runner.owner_user_id) + assertRunnerOwnerIfAuthenticated(ctx, runner.owner_principal) return json(runner) } @@ -168,7 +184,7 @@ async function heartbeat( ): Promise { const runnerId = routeParam(request, `id`) const existing = await requireRunner(ctx, runnerId) - assertRunnerOwnerIfAuthenticated(ctx, existing.owner_user_id) + assertRunnerOwnerIfAuthenticated(ctx, existing.owner_principal) const parsed = routeBody(request) const runner = await ctx.entityManager.registry.heartbeatRunner({ runnerId, @@ -177,6 +193,7 @@ async function heartbeat( livenessLeaseExpiresAt: parsed.liveness_lease_expires_at ? new Date(parsed.liveness_lease_expires_at) : undefined, + diagnostics: parsed.diagnostics, }) if (!runner) { throw new ElectricAgentsError(ErrCodeNotFound, `Runner not found`, 404) @@ -205,7 +222,7 @@ async function setRunnerStatus( ): Promise { const runnerId = routeParam(request, `id`) const existing = await requireRunner(ctx, runnerId) - assertRunnerOwnerIfAuthenticated(ctx, existing.owner_user_id) + assertRunnerOwnerIfAuthenticated(ctx, existing.owner_principal) const runner = await ctx.entityManager.registry.setRunnerAdminStatus( runnerId, adminStatus @@ -222,7 +239,7 @@ async function claimWake( ): Promise { const runnerId = routeParam(request, `id`) const runner = await requireRunner(ctx, runnerId) - if (ctx.principal && runner.owner_user_id !== ctx.principal.key) { + if (ctx.principal && runner.owner_principal !== ctx.principal.url) { throw new ElectricAgentsError( ErrCodeUnauthorized, `Runner claim requires the authenticated owner`, @@ -296,10 +313,10 @@ async function requireRunner(ctx: TenantContext, runnerId: string) { function assertRunnerOwnerIfAuthenticated( ctx: TenantContext, - ownerUserId: string + ownerPrincipal: string ): void { if (!ctx.principal) return - if (ownerUserId === ctx.principal.key) return + if (ownerPrincipal === ctx.principal.url) return throw new ElectricAgentsError( ErrCodeUnauthorized, `Runner access requires the authenticated owner`, @@ -307,6 +324,104 @@ function assertRunnerOwnerIfAuthenticated( ) } +async function runnerHealth( + request: RunnersRouteRequest, + ctx: TenantContext +): Promise { + const runnerId = routeParam(request, `id`) + const runner = await requireRunner(ctx, runnerId) + assertRunnerOwnerIfAuthenticated(ctx, runner.owner_principal) + + const now = Date.now() + const leaseExpiresAt = runner.liveness_lease_expires_at + ? new Date(runner.liveness_lease_expires_at).getTime() + : null + + const livenessStatus = + runner.admin_status === `disabled` + ? `offline` + : leaseExpiresAt !== null && leaseExpiresAt > now + ? `online` + : leaseExpiresAt !== null + ? `expired` + : `offline` + + const [activeClaims, dispatchStats] = await Promise.all([ + ctx.entityManager.registry.getActiveClaimsForRunner(runnerId), + ctx.entityManager.registry.getDispatchStatsForRunner(runnerId), + ]) + + const clientDiagnostics = runner.diagnostics ?? null + const issues: Array = [] + let healthStatus: `healthy` | `degraded` | `unhealthy` = `healthy` + + if (runner.admin_status === `disabled`) { + healthStatus = `unhealthy` + issues.push(`Runner is disabled`) + } + if (livenessStatus === `expired`) { + healthStatus = `unhealthy` + const ago = leaseExpiresAt ? Math.round((now - leaseExpiresAt) / 1000) : 0 + issues.push(`Heartbeat lease expired ${ago}s ago`) + } + if (livenessStatus === `offline` && runner.admin_status === `enabled`) { + healthStatus = healthStatus === `unhealthy` ? `unhealthy` : `degraded` + issues.push(`Runner has never sent a heartbeat`) + } + if (clientDiagnostics) { + if (clientDiagnostics.stream_connected === false) { + if (healthStatus === `healthy`) healthStatus = `degraded` + issues.push(`Client reports stream disconnected`) + } + if (clientDiagnostics.last_heartbeat_ok === false) { + if (healthStatus === `healthy`) healthStatus = `degraded` + issues.push(`Client reports last heartbeat failed`) + } + if ( + typeof clientDiagnostics.reconnect_count === `number` && + clientDiagnostics.reconnect_count > 5 + ) { + if (healthStatus === `healthy`) healthStatus = `degraded` + issues.push( + `Client has reconnected ${clientDiagnostics.reconnect_count} times` + ) + } + } else if (runner.last_seen_at) { + if (healthStatus === `healthy`) healthStatus = `degraded` + issues.push(`No client diagnostics available`) + } + + return json({ + runner: { + id: runner.id, + admin_status: runner.admin_status, + liveness_status: livenessStatus, + lease_expires_at: runner.liveness_lease_expires_at ?? null, + lease_remaining_ms: + leaseExpiresAt !== null ? Math.max(0, leaseExpiresAt - now) : null, + wake_stream: runner.wake_stream, + wake_stream_offset: runner.wake_stream_offset ?? null, + last_seen_at: runner.last_seen_at ?? null, + created_at: runner.created_at, + }, + client: clientDiagnostics, + claims: { + active_count: activeClaims.length, + active: activeClaims.map((c) => ({ + consumer_id: c.consumer_id, + epoch: c.epoch, + entity_url: c.entity_url, + stream_path: c.stream_path, + claimed_at: c.claimed_at, + last_heartbeat_at: c.last_heartbeat_at ?? null, + lease_expires_at: c.lease_expires_at ?? null, + })), + }, + dispatch: dispatchStats, + health: { status: healthStatus, issues }, + }) +} + async function notificationFromClaim( ctx: TenantContext, input: { diff --git a/packages/agents-server/src/utils/server-utils.ts b/packages/agents-server/src/utils/server-utils.ts index 2b0c852f2e..e78805ed81 100644 --- a/packages/agents-server/src/utils/server-utils.ts +++ b/packages/agents-server/src/utils/server-utils.ts @@ -130,7 +130,7 @@ export function buildElectricProxyTarget(options: { } else if (table === `runners`) { target.searchParams.set( `columns`, - `"tenant_id","id","owner_user_id","label","kind","admin_status","wake_stream","wake_stream_offset","last_seen_at","liveness_lease_expires_at","created_at","updated_at"` + `"tenant_id","id","owner_principal","label","kind","admin_status","wake_stream","wake_stream_offset","last_seen_at","liveness_lease_expires_at","diagnostics","created_at","updated_at"` ) applyTenantShapeWhere(target, options.tenantId) } else if (table === `entity_dispatch_state`) { diff --git a/packages/agents-server/test/dispatch-policy-routing.test.ts b/packages/agents-server/test/dispatch-policy-routing.test.ts index 927e1f783a..2ee56b96cf 100644 --- a/packages/agents-server/test/dispatch-policy-routing.test.ts +++ b/packages/agents-server/test/dispatch-policy-routing.test.ts @@ -45,7 +45,7 @@ function buildContext(overrides: Partial = {}): TenantContext { kind: `user`, id: `owner@example.com`, key: `user:owner@example.com`, - url: `/principal/user:owner@example.com`, + url: `/principal/user%3Aowner%40example.com`, }, publicUrl: `http://server`, durableStreamsUrl: `http://durable.local`, @@ -68,7 +68,7 @@ function buildContext(overrides: Partial = {}): TenantContext { ), getRunner: vi.fn(async () => ({ id: `runner-1`, - owner_user_id: `user:owner@example.com`, + owner_principal: `/principal/user%3Aowner%40example.com`, label: `Local runner`, kind: `local`, admin_status: `enabled`, @@ -175,7 +175,7 @@ describe(`dispatch policy routing`, () => { }) ) expect(ctx.entityManager.send).toHaveBeenCalledWith(`/chat/one`, { - from: `/principal/user:owner@example.com`, + from: `/principal/user%3Aowner%40example.com`, payload: `hello`, }) expect(ctx.streamClient.putSubscription).toHaveBeenCalledWith( diff --git a/packages/agents-server/test/horton-pull-wake-e2e.test.ts b/packages/agents-server/test/horton-pull-wake-e2e.test.ts index b631d57b3e..c141ea04f8 100644 --- a/packages/agents-server/test/horton-pull-wake-e2e.test.ts +++ b/packages/agents-server/test/horton-pull-wake-e2e.test.ts @@ -130,7 +130,7 @@ describe(`pull-wake Horton e2e with mocked LLM`, () => { pullWake: { runnerId, registerRunner: true, - ownerUserId: testPrincipal.key, + ownerPrincipal: testPrincipal.url, headers: authHeaders, claimHeaders: authHeaders, claimTokenHeader: `electric-claim-token`, diff --git a/packages/agents-server/test/horton-spawn-worker.test.ts b/packages/agents-server/test/horton-spawn-worker.test.ts index 630b33d58d..0e441e7148 100644 --- a/packages/agents-server/test/horton-spawn-worker.test.ts +++ b/packages/agents-server/test/horton-spawn-worker.test.ts @@ -36,7 +36,7 @@ describe.skipIf(!process.env.ANTHROPIC_API_KEY)( pullWake: { runnerId: `horton-spawn-worker-test`, registerRunner: true, - ownerUserId: `test-user`, + ownerPrincipal: `/principal/system%3Atest-user`, }, }) await builtinAgentsServer.start() diff --git a/packages/agents-server/test/horton-title-generation.test.ts b/packages/agents-server/test/horton-title-generation.test.ts index 00a8811f92..625bb9c897 100644 --- a/packages/agents-server/test/horton-title-generation.test.ts +++ b/packages/agents-server/test/horton-title-generation.test.ts @@ -36,7 +36,7 @@ describe.skipIf(!process.env.ANTHROPIC_API_KEY)( pullWake: { runnerId: `horton-title-generation-test`, registerRunner: true, - ownerUserId: `test-user`, + ownerPrincipal: `/principal/system%3Atest-user`, }, }) await builtinAgentsServer.start() diff --git a/packages/agents-server/test/runners-router.test.ts b/packages/agents-server/test/runners-router.test.ts index 78101b5fea..6ebad77d9c 100644 --- a/packages/agents-server/test/runners-router.test.ts +++ b/packages/agents-server/test/runners-router.test.ts @@ -15,11 +15,11 @@ function request(method: string, path: string, body?: unknown): Request { function runner(overrides: Record = {}) { return { id: `runner-1`, - owner_user_id: `user:owner@example.com`, + owner_principal: `/principal/user%3Aowner%40example.com`, label: `Local runner`, - kind: `local`, - admin_status: `enabled`, - liveness: `offline`, + kind: `local` as const, + admin_status: `enabled` as const, + liveness: `offline` as const, wake_stream: `/runners/runner-1/wake`, created_at: new Date(0).toISOString(), updated_at: new Date(0).toISOString(), @@ -32,7 +32,7 @@ function buildContext(overrides: Partial = {}): TenantContext { createRunner: vi.fn(async (input) => runner({ id: input.id, - owner_user_id: input.ownerUserId, + owner_principal: input.ownerPrincipal, label: input.label, wake_stream: input.wakeStream ?? `/runners/${input.id}/wake`, }) @@ -48,6 +48,12 @@ function buildContext(overrides: Partial = {}): TenantContext { getEntityByStream: vi.fn(), materializeActiveClaim: vi.fn(), updateStatus: vi.fn(), + getActiveClaimsForRunner: vi.fn(async () => []), + getDispatchStatsForRunner: vi.fn(async () => ({ + entities_with_active_claim: 0, + entities_with_outstanding_wake: 0, + entities_with_pending_work: 0, + })), } const insertChain = { values: vi.fn(() => ({ @@ -60,7 +66,7 @@ function buildContext(overrides: Partial = {}): TenantContext { kind: `user`, id: `owner@example.com`, key: `user:owner@example.com`, - url: `/principal/user:owner@example.com`, + url: `/principal/user%3Aowner%40example.com`, }, publicUrl: `http://server`, durableStreamsUrl: `http://durable.local`, @@ -86,7 +92,7 @@ describe(`runner routes`, () => { const response = await globalRouter.fetch( request(`POST`, `/_electric/runners`, { id: `runner-1`, - owner_user_id: `other@example.com`, + owner_principal: `/principal/user%3Aother%40example.com`, label: `Local runner`, }), buildContext({ @@ -94,7 +100,7 @@ describe(`runner routes`, () => { kind: `user`, id: `owner@example.com`, key: `user:owner@example.com`, - url: `/principal/user:owner@example.com`, + url: `/principal/user%3Aowner%40example.com`, }, }) ) @@ -108,14 +114,14 @@ describe(`runner routes`, () => { kind: `user`, id: `owner@example.com`, key: `user:owner@example.com`, - url: `/principal/user:owner@example.com`, + url: `/principal/user%3Aowner%40example.com`, }, }) const response = await globalRouter.fetch( request(`POST`, `/_electric/runners`, { id: `runner-1`, - owner_user_id: `user:owner@example.com`, + owner_principal: `/principal/user%3Aowner%40example.com`, label: `Local runner`, }), ctx @@ -125,7 +131,7 @@ describe(`runner routes`, () => { expect(ctx.entityManager.registry.createRunner).toHaveBeenCalledWith( expect.objectContaining({ id: `runner-1`, - ownerUserId: `user:owner@example.com`, + ownerPrincipal: `/principal/user%3Aowner%40example.com`, }) ) expect(ctx.streamClient.ensure).toHaveBeenCalledWith( @@ -140,7 +146,7 @@ describe(`runner routes`, () => { kind: `user`, id: `owner@example.com`, key: `user:owner@example.com`, - url: `/principal/user:owner@example.com`, + url: `/principal/user%3Aowner%40example.com`, }, }) @@ -155,9 +161,59 @@ describe(`runner routes`, () => { expect(response.status).toBe(201) expect(ctx.entityManager.registry.createRunner).toHaveBeenCalledWith( expect.objectContaining({ - ownerUserId: `user:owner@example.com`, + ownerPrincipal: `/principal/user%3Aowner%40example.com`, + }) + ) + }) + + it(`returns runner health with diagnostics and claim state`, async () => { + const ctx = buildContext() + vi.mocked(ctx.entityManager.registry.getRunner).mockResolvedValue( + runner({ + liveness_lease_expires_at: new Date(Date.now() + 30_000).toISOString(), + last_seen_at: new Date().toISOString(), + diagnostics: { + stream_connected: true, + reconnect_count: 0, + last_heartbeat_ok: true, + }, }) ) + + const response = await globalRouter.fetch( + request(`GET`, `/_electric/runners/runner-1/health`), + ctx + ) + + expect(response.status).toBe(200) + const body = (await response.json()) as Record + expect(body.runner).toMatchObject({ + id: `runner-1`, + liveness_status: `online`, + }) + expect(body.client).toMatchObject({ stream_connected: true }) + expect(body.claims).toMatchObject({ active_count: 0 }) + expect(body.health).toMatchObject({ status: `healthy`, issues: [] }) + }) + + it(`returns unhealthy when runner lease is expired`, async () => { + const ctx = buildContext() + vi.mocked(ctx.entityManager.registry.getRunner).mockResolvedValue( + runner({ + liveness_lease_expires_at: new Date(Date.now() - 10_000).toISOString(), + last_seen_at: new Date(Date.now() - 15_000).toISOString(), + }) + ) + + const response = await globalRouter.fetch( + request(`GET`, `/_electric/runners/runner-1/health`), + ctx + ) + + expect(response.status).toBe(200) + const body = (await response.json()) as Record + expect(body.health.status).toBe(`unhealthy`) + expect(body.health.issues.length).toBeGreaterThan(0) }) it(`allows unauthenticated runner claims when no server auth is configured`, async () => { @@ -181,7 +237,7 @@ describe(`runner routes`, () => { kind: `user`, id: `owner@example.com`, key: `user:owner@example.com`, - url: `/principal/user:owner@example.com`, + url: `/principal/user%3Aowner%40example.com`, }, }) vi.mocked(ctx.streamClient.claimSubscription).mockRejectedValue( @@ -224,7 +280,7 @@ describe(`runner routes`, () => { kind: `user`, id: `owner@example.com`, key: `user:owner@example.com`, - url: `/principal/user:owner@example.com`, + url: `/principal/user%3Aowner%40example.com`, }, }) vi.mocked(ctx.streamClient.claimSubscription).mockResolvedValue({ @@ -280,7 +336,7 @@ describe(`runner routes`, () => { kind: `user`, id: `owner@example.com`, key: `user:owner@example.com`, - url: `/principal/user:owner@example.com`, + url: `/principal/user%3Aowner%40example.com`, }, }) vi.mocked(ctx.streamClient.claimSubscription).mockResolvedValue({ diff --git a/packages/agents/src/server.ts b/packages/agents/src/server.ts index 7584ea6e2b..db59faaf3f 100644 --- a/packages/agents/src/server.ts +++ b/packages/agents/src/server.ts @@ -40,7 +40,7 @@ export interface BuiltinAgentsServerOptions { /** Pull-wake runner configuration for built-in agents. */ pullWake: { runnerId: string - ownerUserId?: string + ownerPrincipal?: string label?: string registerRunner?: boolean headers?: PullWakeRunnerConfig[`headers`] @@ -406,7 +406,7 @@ export class BuiltinAgentsServer { headers, body: JSON.stringify({ id: pullWake.runnerId, - owner_user_id: pullWake.ownerUserId, + owner_principal: pullWake.ownerPrincipal, label: pullWake.label ?? `Built-in agents`, kind: `local`, admin_status: `enabled`, diff --git a/packages/electric-ax/src/start.ts b/packages/electric-ax/src/start.ts index 678da3944c..db7f9aeeed 100644 --- a/packages/electric-ax/src/start.ts +++ b/packages/electric-ax/src/start.ts @@ -18,7 +18,7 @@ export { readDotEnvFile, resolveAnthropicApiKey } from './env.js' const DEFAULT_ELECTRIC_AGENTS_PORT = 4437 const DEFAULT_COMPOSE_PROJECT_NAME = `electric-agents` const DEFAULT_PULL_WAKE_RUNNER_ID = `builtin-agents` -const DEFAULT_PULL_WAKE_OWNER_ID = `builtin-agents` +const DEFAULT_PULL_WAKE_OWNER_PRINCIPAL = `/principal/system%3Abuiltin-agents` const DOCKER_COMPOSE_FILE = fileURLToPath( new URL(`../docker-compose.full.yml`, import.meta.url) ) @@ -128,14 +128,13 @@ export function resolvePullWakeRunnerId( ) } -export function resolvePullWakeOwnerId( +export function resolvePullWakeOwnerPrincipal( env: NodeJS.ProcessEnv = process.env, fileEnv: Record = readDotEnvFile() ): string { - return ( - readConfigValue(env, fileEnv, [`ELECTRIC_AGENTS_IDENTITY`]) ?? - DEFAULT_PULL_WAKE_OWNER_ID - ) + const identity = readConfigValue(env, fileEnv, [`ELECTRIC_AGENTS_IDENTITY`]) + if (identity) return `/principal/${encodeURIComponent(identity)}` + return DEFAULT_PULL_WAKE_OWNER_PRINCIPAL } function parseAdditionalServerHeaders( @@ -376,7 +375,7 @@ export async function startBuiltinAgentsServer( const fileEnv = readDotEnvFile(cwd) const anthropicApiKey = resolveAnthropicApiKey(options, env, fileEnv) const runnerId = resolvePullWakeRunnerId(env, fileEnv) - const ownerUserId = resolvePullWakeOwnerId(env, fileEnv) + const ownerPrincipal = resolvePullWakeOwnerPrincipal(env, fileEnv) const serverHeaders = mergeHeaders(parseAdditionalServerHeaders(env, fileEnv)) const agentServerUrl = params.agentServerUrl ?? @@ -392,7 +391,7 @@ export async function startBuiltinAgentsServer( loadProjectMcpConfig: true, pullWake: { runnerId, - ownerUserId, + ownerPrincipal, registerRunner: true, headers: serverHeaders, claimHeaders: serverHeaders, diff --git a/packages/electric-ax/test/start.test.ts b/packages/electric-ax/test/start.test.ts index 1af137e320..228ec33e5d 100644 --- a/packages/electric-ax/test/start.test.ts +++ b/packages/electric-ax/test/start.test.ts @@ -6,7 +6,7 @@ import { resolveAnthropicApiKey, resolveComposeProjectName, resolveElectricAgentsPort, - resolvePullWakeOwnerId, + resolvePullWakeOwnerPrincipal, resolvePullWakeRunnerId, waitForElectricAgentsServer, } from '../src/start' @@ -100,15 +100,20 @@ describe(`resolvePullWakeRunnerId`, () => { }) }) -describe(`resolvePullWakeOwnerId`, () => { +describe(`resolvePullWakeOwnerPrincipal`, () => { it(`uses the agents identity when present`, () => { expect( - resolvePullWakeOwnerId({ ELECTRIC_AGENTS_IDENTITY: `a@example.com` }, {}) - ).toBe(`a@example.com`) + resolvePullWakeOwnerPrincipal( + { ELECTRIC_AGENTS_IDENTITY: `user:a@example.com` }, + {} + ) + ).toBe(`/principal/user%3Aa%40example.com`) }) it(`falls back to the local builtin owner`, () => { - expect(resolvePullWakeOwnerId({}, {})).toBe(`builtin-agents`) + expect(resolvePullWakeOwnerPrincipal({}, {})).toBe( + `/principal/system%3Abuiltin-agents` + ) }) }) From 15aef190fc5c96879dde6c557bd2d13d3ec4b7a9 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Sat, 16 May 2026 14:30:16 +0100 Subject: [PATCH 10/37] fix(agents): address pull-wake health review findings --- .../agents-runtime/src/pull-wake-runner.ts | 35 +++++----- .../src/electric-agents-types.ts | 7 +- .../src/routing/runners-router.ts | 48 +++++++------ .../agents-server/test/runners-router.test.ts | 67 +++++++++++++++++++ 4 files changed, 118 insertions(+), 39 deletions(-) diff --git a/packages/agents-runtime/src/pull-wake-runner.ts b/packages/agents-runtime/src/pull-wake-runner.ts index ccedaaf5d2..67f9662a74 100644 --- a/packages/agents-runtime/src/pull-wake-runner.ts +++ b/packages/agents-runtime/src/pull-wake-runner.ts @@ -231,6 +231,17 @@ export function createPullWakeRunner( })) as PullWakeStreamResponse }) + const recordClaimSkipped = (): null => { + lastClaimResult = `no_work` + claimsSkipped++ + return null + } + + const recordClaimError = (): void => { + lastClaimResult = `error` + claimsFailed++ + } + const claimWake = async ( event: PullWakeEvent, signal: AbortSignal @@ -246,23 +257,16 @@ export function createPullWakeRunner( signal, body: JSON.stringify(event), }) - if (response.status === 204) { - lastClaimResult = `no_work` - claimsSkipped++ - return null - } + if (response.status === 204) return recordClaimSkipped() if (!response.ok) { const text = await response.text() if ( response.status === 409 && (text.includes(`ALREADY_CLAIMED`) || text.includes(`NO_PENDING_WORK`)) ) { - lastClaimResult = `no_work` - claimsSkipped++ - return null + return recordClaimSkipped() } - lastClaimResult = `error` - claimsFailed++ + recordClaimError() throw new Error( `Pull-wake claim failed for ${config.runnerId}: ${response.status} ${text}` ) @@ -270,18 +274,13 @@ export function createPullWakeRunner( const notification = (await response.json()) as WakeNotification & { done?: boolean } - if (notification.done) { - lastClaimResult = `no_work` - claimsSkipped++ - return null - } + if (notification.done) return recordClaimSkipped() lastClaimResult = `claimed` claimsSucceeded++ return notification } catch (err) { - if (lastClaimResult !== `no_work` && lastClaimResult !== `error`) { - lastClaimResult = `error` - claimsFailed++ + if (lastClaimResult === null || lastClaimResult === `claimed`) { + recordClaimError() } throw err } diff --git a/packages/agents-server/src/electric-agents-types.ts b/packages/agents-server/src/electric-agents-types.ts index 8e0e98bdff..621c4fb7b3 100644 --- a/packages/agents-server/src/electric-agents-types.ts +++ b/packages/agents-server/src/electric-agents-types.ts @@ -2,7 +2,10 @@ * Types for the Electric Agents entity runtime. */ -import type { WebhookNotification } from '@electric-ax/agents-runtime' +import type { + PullWakeRunnerHealth, + WebhookNotification, +} from '@electric-ax/agents-runtime' import type { Principal } from './principal.js' type WakeNotification = WebhookNotification @@ -151,7 +154,7 @@ export interface RunnerHealthResponse { last_seen_at: string | null created_at: string } - client: Record | null + client: Omit | null claims: { active_count: number active: Array<{ diff --git a/packages/agents-server/src/routing/runners-router.ts b/packages/agents-server/src/routing/runners-router.ts index d74da413c3..71c88f3e07 100644 --- a/packages/agents-server/src/routing/runners-router.ts +++ b/packages/agents-server/src/routing/runners-router.ts @@ -9,6 +9,7 @@ import { ErrCodeNotFound, ErrCodeNotRunning, ErrCodeUnauthorized, + type RunnerHealthResponse, } from '../electric-agents-types.js' import { routeBody, withSchema } from './schema.js' import { subscriptionIdForDispatchTarget } from './dispatch-policy.js' @@ -118,7 +119,7 @@ async function registerRunner( if (!principalKeyFromUrl(ownerPrincipal)) { throw new ElectricAgentsError( ErrCodeInvalidRequest, - `owner_principal must be a valid principal URL accepted by principalKeyFromUrl() (e.g. /principal/user%3Aalice), got: ${ownerPrincipal}`, + `owner_principal must be a valid principal URL (e.g. /principal/user%3Aalice), got: ${ownerPrincipal}`, 400 ) } @@ -337,61 +338,69 @@ async function runnerHealth( ? new Date(runner.liveness_lease_expires_at).getTime() : null - const livenessStatus = - runner.admin_status === `disabled` - ? `offline` - : leaseExpiresAt !== null && leaseExpiresAt > now - ? `online` - : leaseExpiresAt !== null - ? `expired` - : `offline` + let livenessStatus: `online` | `offline` | `expired` + if (runner.admin_status === `disabled`) { + livenessStatus = `offline` + } else if (leaseExpiresAt !== null && leaseExpiresAt > now) { + livenessStatus = `online` + } else if (leaseExpiresAt !== null) { + livenessStatus = `expired` + } else { + livenessStatus = `offline` + } const [activeClaims, dispatchStats] = await Promise.all([ ctx.entityManager.registry.getActiveClaimsForRunner(runnerId), ctx.entityManager.registry.getDispatchStatsForRunner(runnerId), ]) - const clientDiagnostics = runner.diagnostics ?? null + const clientDiagnostics = + (runner.diagnostics as RunnerHealthResponse[`client`]) ?? null const issues: Array = [] let healthStatus: `healthy` | `degraded` | `unhealthy` = `healthy` + const escalate = (floor: `degraded` | `unhealthy`): void => { + if (floor === `unhealthy`) healthStatus = `unhealthy` + else if (healthStatus === `healthy`) healthStatus = `degraded` + } + if (runner.admin_status === `disabled`) { - healthStatus = `unhealthy` + escalate(`unhealthy`) issues.push(`Runner is disabled`) } if (livenessStatus === `expired`) { - healthStatus = `unhealthy` + escalate(`unhealthy`) const ago = leaseExpiresAt ? Math.round((now - leaseExpiresAt) / 1000) : 0 issues.push(`Heartbeat lease expired ${ago}s ago`) } if (livenessStatus === `offline` && runner.admin_status === `enabled`) { - healthStatus = healthStatus === `unhealthy` ? `unhealthy` : `degraded` + escalate(`degraded`) issues.push(`Runner has never sent a heartbeat`) } if (clientDiagnostics) { if (clientDiagnostics.stream_connected === false) { - if (healthStatus === `healthy`) healthStatus = `degraded` + escalate(`degraded`) issues.push(`Client reports stream disconnected`) } if (clientDiagnostics.last_heartbeat_ok === false) { - if (healthStatus === `healthy`) healthStatus = `degraded` + escalate(`degraded`) issues.push(`Client reports last heartbeat failed`) } if ( typeof clientDiagnostics.reconnect_count === `number` && clientDiagnostics.reconnect_count > 5 ) { - if (healthStatus === `healthy`) healthStatus = `degraded` + escalate(`degraded`) issues.push( `Client has reconnected ${clientDiagnostics.reconnect_count} times` ) } } else if (runner.last_seen_at) { - if (healthStatus === `healthy`) healthStatus = `degraded` + escalate(`degraded`) issues.push(`No client diagnostics available`) } - return json({ + const body: RunnerHealthResponse = { runner: { id: runner.id, admin_status: runner.admin_status, @@ -419,7 +428,8 @@ async function runnerHealth( }, dispatch: dispatchStats, health: { status: healthStatus, issues }, - }) + } + return json(body) } async function notificationFromClaim( diff --git a/packages/agents-server/test/runners-router.test.ts b/packages/agents-server/test/runners-router.test.ts index 6ebad77d9c..994843970d 100644 --- a/packages/agents-server/test/runners-router.test.ts +++ b/packages/agents-server/test/runners-router.test.ts @@ -330,6 +330,73 @@ describe(`runner routes`, () => { ) }) + it(`rejects invalid owner_principal with 400`, async () => { + const response = await globalRouter.fetch( + request(`POST`, `/_electric/runners`, { + id: `runner-1`, + owner_principal: `/principal/not-a-valid-key`, + label: `Local runner`, + }), + buildContext({ + principal: { + kind: `user`, + id: `owner@example.com`, + key: `user:owner@example.com`, + url: `/principal/user%3Aowner%40example.com`, + }, + }) + ) + + expect(response.status).toBe(400) + }) + + it(`returns unhealthy when runner is disabled`, async () => { + const ctx = buildContext() + vi.mocked(ctx.entityManager.registry.getRunner).mockResolvedValue( + runner({ + admin_status: `disabled`, + liveness_lease_expires_at: new Date(Date.now() + 30_000).toISOString(), + last_seen_at: new Date().toISOString(), + }) + ) + + const response = await globalRouter.fetch( + request(`GET`, `/_electric/runners/runner-1/health`), + ctx + ) + + expect(response.status).toBe(200) + const body = (await response.json()) as Record + expect(body.health.status).toBe(`unhealthy`) + expect(body.health.issues).toContain(`Runner is disabled`) + expect(body.runner.liveness_status).toBe(`offline`) + }) + + it(`returns degraded when stream is disconnected`, async () => { + const ctx = buildContext() + vi.mocked(ctx.entityManager.registry.getRunner).mockResolvedValue( + runner({ + liveness_lease_expires_at: new Date(Date.now() + 30_000).toISOString(), + last_seen_at: new Date().toISOString(), + diagnostics: { + stream_connected: false, + reconnect_count: 2, + last_heartbeat_ok: true, + }, + }) + ) + + const response = await globalRouter.fetch( + request(`GET`, `/_electric/runners/runner-1/health`), + ctx + ) + + expect(response.status).toBe(200) + const body = (await response.json()) as Record + expect(body.health.status).toBe(`degraded`) + expect(body.health.issues).toContain(`Client reports stream disconnected`) + }) + it(`uses the pending stream from multi-stream claim responses`, async () => { const ctx = buildContext({ principal: { From 83fb0395d9a2029b2ff98699bf18e3cc3c921acd Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Sat, 16 May 2026 14:32:29 +0100 Subject: [PATCH 11/37] chore: add changeset for pull-wake health diagnostics Co-Authored-By: Claude Opus 4.6 --- .changeset/pull-wake-health-diagnostics.md | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 .changeset/pull-wake-health-diagnostics.md diff --git a/.changeset/pull-wake-health-diagnostics.md b/.changeset/pull-wake-health-diagnostics.md new file mode 100644 index 0000000000..bc28013e91 --- /dev/null +++ b/.changeset/pull-wake-health-diagnostics.md @@ -0,0 +1,9 @@ +--- +'@electric-ax/agents-server': patch +'@electric-ax/agents-runtime': patch +'@electric-ax/agents-desktop': patch +'@electric-ax/agents': patch +'electric-ax': patch +--- + +Add pull-wake runner health check endpoint and rename `owner_user_id` to `owner_principal` across the runners system. The `GET /_electric/runners/:id/health` endpoint returns comprehensive diagnostics including runner state, client-reported stream/heartbeat/claim metrics, active claims, and dispatch stats with a derived health status (healthy/degraded/unhealthy). The `PullWakeRunner` now tracks internal diagnostics and reports them to the server via heartbeats, stored in a new `diagnostics` JSONB column on the runners table. The `owner_user_id` → `owner_principal` rename stores canonical principal URLs instead of keys, with strict validation via `principalKeyFromUrl()`. This is a breaking change with no backward compatibility — all callers must send principal URLs. From 7c3a0fb6735a998abad344ed405993a714ec8aa4 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Sat, 16 May 2026 17:51:45 +0100 Subject: [PATCH 12/37] feat(agents): surface pull-wake runtime diagnostics --- docs/agents-principals-implementation-plan.md | 5 +- .../src/components/SettingsMenu.tsx | 4 +- .../src/components/UserMessage.tsx | 14 +- .../components/settings/SettingsScreen.tsx | 2 +- .../components/settings/SettingsSidebar.tsx | 8 + .../settings/pages/LocalRuntimePage.tsx | 220 +++++++++++++++++- .../components/settings/pages/ServersPage.tsx | 2 +- .../src/hooks/useServerConnection.tsx | 19 +- .../src/lib/ElectricAgentsProvider.tsx | 64 ++++- .../src/lib/auth-fetch.test.ts | 16 +- .../agents-server-ui/src/lib/auth-fetch.ts | 6 +- packages/agents-server-ui/src/router.tsx | 4 + .../2026-05-16-pull-wake-health-check.md | 25 +- packages/agents-server/src/principal.ts | 21 +- .../src/routing/electric-proxy-router.ts | 1 + .../src/routing/runners-router.ts | 6 +- .../agents-server/src/utils/server-utils.ts | 16 +- packages/agents-server/test/principal.test.ts | 8 +- .../agents-server/test/server-utils.test.ts | 34 +++ packages/agents/src/bootstrap.ts | 2 +- 20 files changed, 417 insertions(+), 60 deletions(-) create mode 100644 packages/agents-server/test/server-utils.test.ts diff --git a/docs/agents-principals-implementation-plan.md b/docs/agents-principals-implementation-plan.md index 81bce2700a..92b5f11a3d 100644 --- a/docs/agents-principals-implementation-plan.md +++ b/docs/agents-principals-implementation-plan.md @@ -82,7 +82,8 @@ Helpers: ```ts export function parsePrincipalKey(input: string): Principal export function principalUrl(key: string): string -export function principalKeyFromUrl(url: string): string | null +export function parsePrincipalUrl(url: string): Principal | null +export function isPrincipalUrl(url: string): boolean export function getPrincipalFromRequest(request: Request): Principal | null export function getDevPrincipal(): Principal ``` @@ -575,7 +576,7 @@ entity: { principal: entity.created_by ? { url: entity.created_by, - key: principalKeyFromUrl(entity.created_by), + key: parsePrincipalUrl(entity.created_by)?.key ?? null, } : undefined, ``` diff --git a/packages/agents-server-ui/src/components/SettingsMenu.tsx b/packages/agents-server-ui/src/components/SettingsMenu.tsx index 98c8971eac..ff516b52b2 100644 --- a/packages/agents-server-ui/src/components/SettingsMenu.tsx +++ b/packages/agents-server-ui/src/components/SettingsMenu.tsx @@ -171,13 +171,13 @@ export function SettingsMenu(): React.ReactElement { truncate className={styles.runtimeUrl} > - {runtimeUrl} + Pull-wake ) : ( {localRuntimeDisabled ? `Disabled for this server` - : `No runtime URL yet`} + : `Runtime not started`} )} {runtimeError && ( diff --git a/packages/agents-server-ui/src/components/UserMessage.tsx b/packages/agents-server-ui/src/components/UserMessage.tsx index 650bc1ca52..2dced64cd4 100644 --- a/packages/agents-server-ui/src/components/UserMessage.tsx +++ b/packages/agents-server-ui/src/components/UserMessage.tsx @@ -9,12 +9,24 @@ type UserMessageSection = Extract< { kind: `user_message` } > +function formatSender(value: string | null | undefined): string { + if (!value) return `user` + if (!value.startsWith(`/principal/`)) return value + const segment = value.slice(`/principal/`.length) + if (!segment || segment.includes(`/`)) return value + try { + return decodeURIComponent(segment) + } catch { + return value + } +} + export const UserMessage = memo(function UserMessage({ section, }: { section: UserMessageSection }): React.ReactElement { - const sender = section.from ?? `user` + const sender = formatSender(section.from) return ( diff --git a/packages/agents-server-ui/src/components/settings/SettingsScreen.tsx b/packages/agents-server-ui/src/components/settings/SettingsScreen.tsx index 269ea82d9d..4e13d47e30 100644 --- a/packages/agents-server-ui/src/components/settings/SettingsScreen.tsx +++ b/packages/agents-server-ui/src/components/settings/SettingsScreen.tsx @@ -58,7 +58,7 @@ export function SettingsScreen({ )} -
+

{title}

{children} diff --git a/packages/agents-server-ui/src/components/settings/SettingsSidebar.tsx b/packages/agents-server-ui/src/components/settings/SettingsSidebar.tsx index 359a9b73a3..ff4d8d9755 100644 --- a/packages/agents-server-ui/src/components/settings/SettingsSidebar.tsx +++ b/packages/agents-server-ui/src/components/settings/SettingsSidebar.tsx @@ -2,6 +2,7 @@ import { useCallback } from 'react' import { useNavigate } from '@tanstack/react-router' import { ArrowLeft, + Brain, KeyRound, Palette, Plug, @@ -18,6 +19,7 @@ export type SettingsCategoryId = | `servers` | `credentials` | `appearance` + | `local-runtime` | `mcp-servers` interface CategoryDef { @@ -84,6 +86,12 @@ export function SettingsSidebar({ icon: , visible: true, }, + { + id: `local-runtime`, + label: `Local Runtime`, + icon: , + visible: isDesktop, + }, { id: `mcp-servers`, label: `MCP Servers`, diff --git a/packages/agents-server-ui/src/components/settings/pages/LocalRuntimePage.tsx b/packages/agents-server-ui/src/components/settings/pages/LocalRuntimePage.tsx index d74dc03c07..16edade0b0 100644 --- a/packages/agents-server-ui/src/components/settings/pages/LocalRuntimePage.tsx +++ b/packages/agents-server-ui/src/components/settings/pages/LocalRuntimePage.tsx @@ -1,10 +1,17 @@ import { useEffect, useState } from 'react' +import { eq, useLiveQuery } from '@tanstack/react-db' +import { appendPathToUrl } from '@electric-ax/agents-runtime/client' import { Play, RefreshCw, Square } from 'lucide-react' import { loadDesktopState, onDesktopStateChanged, type DesktopState, } from '../../../lib/server-connection' +import { + useElectricAgents, + type ElectricRunner, +} from '../../../lib/ElectricAgentsProvider' +import { formatRelativeTime } from '../../../lib/formatTime' import { Badge, Button, Icon, Stack, Text } from '../../../ui' import { SettingsRow, SettingsScreen, SettingsSection } from '../SettingsScreen' @@ -18,6 +25,97 @@ const STATUS_TONES: Record< error: { label: `Error`, tone: `danger` }, } +const RUNNER_HEALTH_TONES: Record< + `healthy` | `degraded` | `unhealthy` | `unknown`, + { label: string; tone: `success` | `warning` | `danger` | `neutral` } +> = { + healthy: { label: `Healthy`, tone: `success` }, + degraded: { label: `Degraded`, tone: `warning` }, + unhealthy: { label: `Unhealthy`, tone: `danger` }, + unknown: { label: `Unknown`, tone: `neutral` }, +} + +function parseTime(value: string | null | undefined): number | null { + if (!value) return null + const parsed = Date.parse(value) + return Number.isFinite(parsed) ? parsed : null +} + +function runnerHealth( + runner: ElectricRunner | null, + now: number = Date.now() +): { status: keyof typeof RUNNER_HEALTH_TONES; issues: Array } { + if (!runner) return { status: `unknown`, issues: [`Runner not synced`] } + const issues: Array = [] + let status: keyof typeof RUNNER_HEALTH_TONES = `healthy` + const escalate = (floor: `degraded` | `unhealthy`) => { + if (floor === `unhealthy`) status = `unhealthy` + else if (status === `healthy`) status = `degraded` + } + + if (runner.admin_status === `disabled`) { + escalate(`unhealthy`) + issues.push(`Disabled`) + } + + const leaseExpiresAt = parseTime(runner.liveness_lease_expires_at) + if (leaseExpiresAt === null) { + escalate(`degraded`) + issues.push(`No heartbeat`) + } else if (leaseExpiresAt <= now) { + escalate(`unhealthy`) + issues.push(`Lease expired`) + } + + const diagnostics = runner.diagnostics + if (!diagnostics) { + if (runner.last_seen_at) { + escalate(`degraded`) + issues.push(`No diagnostics`) + } + } else { + if (diagnostics.stream_connected === false) { + escalate(`degraded`) + issues.push(`Stream disconnected`) + } + if (diagnostics.last_heartbeat_ok === false) { + escalate(`degraded`) + issues.push(`Heartbeat failed`) + } + if ((diagnostics.reconnect_count ?? 0) > 5) { + escalate(`degraded`) + issues.push(`${diagnostics.reconnect_count} reconnects`) + } + } + + return { status, issues } +} + +function timeLabel(value: string | null | undefined): string { + const ts = parseTime(value) + return ts === null ? `-` : formatRelativeTime(ts) +} + +function countLabel(value: number | undefined): string { + return String(value ?? 0) +} + +function runtimeConnectionLabel(value: string | null | undefined): string { + if (!value) return `-` + return `Pull-wake` +} + +function runnerHealthEndpoint( + baseUrl: string | null | undefined, + runnerId: string | null | undefined +): string | null { + if (!baseUrl || !runnerId) return null + return appendPathToUrl( + baseUrl, + `/_electric/runners/${encodeURIComponent(runnerId)}/health` + ) +} + /** * Settings → Local Runtime. Shows the lifecycle state of the bundled * Horton runtime managed by the Electron main process and exposes @@ -31,6 +129,32 @@ const STATUS_TONES: Record< export function LocalRuntimePage(): React.ReactElement { const isDesktop = typeof window !== `undefined` && Boolean(window.electronAPI) const [state, setState] = useState(null) + const [now, setNow] = useState(() => Date.now()) + const { runnersCollection } = useElectricAgents() + const runnerId = state?.pullWakeRunnerId ?? null + const { data: runnerRows = [] } = useLiveQuery( + (query) => { + if (!runnersCollection || !runnerId) return undefined + return query + .from({ runner: runnersCollection }) + .where(({ runner }) => eq(runner.id, runnerId)) + }, + [runnersCollection, runnerId] + ) + const runner = runnerRows[0] ?? null + const health = runnerHealth(runner, now) + const healthTone = RUNNER_HEALTH_TONES[health.status] + const diagnostics = runner?.diagnostics ?? null + const healthEndpoint = runnerHealthEndpoint( + state?.activeServer?.url, + runnerId + ) + + useEffect(() => { + if (!isDesktop) return + const interval = window.setInterval(() => setNow(Date.now()), 5000) + return () => window.clearInterval(interval) + }, [isDesktop]) useEffect(() => { if (!isDesktop) return @@ -82,11 +206,11 @@ export function LocalRuntimePage(): React.ReactElement { control={{statusInfo.label}} /> - {state?.runtimeUrl ?? `—`} + {runtimeConnectionLabel(state?.runtimeUrl)} } /> @@ -102,6 +226,96 @@ export function LocalRuntimePage(): React.ReactElement { )} + + + {runnerId ?? `-`} + + } + /> + 0 ? health.issues.join(`, `) : `No issues` + } + control={{healthTone.label}} + /> + + {healthEndpoint ?? `-`} + + } + /> + + {diagnostics?.stream_connected === false + ? `Disconnected` + : diagnostics?.stream_connected === true + ? `Connected` + : `Unknown`} + + } + /> + + {timeLabel(runner?.last_seen_at)} + + } + /> + + {countLabel(diagnostics?.events_received)} events + + } + /> + + {diagnostics?.last_claim_result ?? `none`} + + } + /> + {diagnostics?.last_error && ( + + {diagnostics.last_error} + + } + /> + )} + + {runtimeUrl && ( - Runtime: {runtimeUrl} + Runtime: Pull-wake )} {isDesktop ? ( diff --git a/packages/agents-server-ui/src/hooks/useServerConnection.tsx b/packages/agents-server-ui/src/hooks/useServerConnection.tsx index cb4e11023f..37a31db18b 100644 --- a/packages/agents-server-ui/src/hooks/useServerConnection.tsx +++ b/packages/agents-server-ui/src/hooks/useServerConnection.tsx @@ -130,16 +130,13 @@ export function ServerConnectionProvider({ : isDesktop ? [] : [currentServer()] - const active = desktopState?.selectedServerId - ? (next.find( - (server) => server.id === desktopState.selectedServerId - ) ?? null) - : desktopState?.activeServer && - next.some( - (server) => server.url === desktopState.activeServer?.url - ) - ? desktopState.activeServer - : (next[0] ?? null) + const active = + desktopState?.activeServer ?? + (desktopState?.selectedServerId + ? (next.find( + (server) => server.id === desktopState.selectedServerId + ) ?? null) + : (next[0] ?? null)) registerActiveBaseUrl(active?.url ?? null) registerActiveServerHeaders(active) setServers(next) @@ -172,8 +169,8 @@ export function ServerConnectionProvider({ const nextServers = state.servers ?? servers setServers(nextServers) const active = - nextServers.find((server) => server.id === state.selectedServerId) ?? state.activeServer ?? + nextServers.find((server) => server.id === state.selectedServerId) ?? null registerActiveBaseUrl(active?.url ?? null) registerActiveServerHeaders(active) diff --git a/packages/agents-server-ui/src/lib/ElectricAgentsProvider.tsx b/packages/agents-server-ui/src/lib/ElectricAgentsProvider.tsx index f59993ebb0..1d41260a32 100644 --- a/packages/agents-server-ui/src/lib/ElectricAgentsProvider.tsx +++ b/packages/agents-server-ui/src/lib/ElectricAgentsProvider.tsx @@ -44,8 +44,45 @@ const entityTypeSchema = z.object({ updated_at: z.string(), }) +const runnerDiagnosticsSchema = z.object({ + started_at: z.string().nullable().optional(), + stream_connected: z.boolean().optional(), + stream_connected_since: z.string().nullable().optional(), + reconnect_count: z.number().optional(), + last_error: z.string().nullable().optional(), + last_error_at: z.string().nullable().optional(), + last_heartbeat_at: z.string().nullable().optional(), + last_heartbeat_ok: z.boolean().optional(), + last_claim_at: z.string().nullable().optional(), + last_claim_result: z + .enum([`claimed`, `no_work`, `error`]) + .nullable() + .optional(), + last_dispatch_at: z.string().nullable().optional(), + events_received: z.number().optional(), + claims_succeeded: z.number().optional(), + claims_skipped: z.number().optional(), + claims_failed: z.number().optional(), +}) + +const runnerSchema = z.object({ + id: z.string(), + owner_principal: z.string(), + label: z.string(), + kind: z.string(), + admin_status: z.enum([`enabled`, `disabled`]), + wake_stream: z.string(), + wake_stream_offset: z.string().nullable().optional(), + last_seen_at: z.string().nullable().optional(), + liveness_lease_expires_at: z.string().nullable().optional(), + diagnostics: runnerDiagnosticsSchema.nullable().optional(), + created_at: z.string(), + updated_at: z.string(), +}) + export type ElectricEntity = z.infer export type ElectricEntityType = z.infer +export type ElectricRunner = z.infer // --- Collection factories --- @@ -97,12 +134,29 @@ function createEntityTypesCollection(baseUrl: string) { ) } +function createRunnersCollection(baseUrl: string) { + return createCollection( + electricCollectionOptions({ + id: `runners`, + schema: runnerSchema, + shapeOptions: { + url: appendPathToUrl(baseUrl, `/_electric/electric/v1/shape`), + params: { table: `runners` }, + fetchClient: serverFetch, + }, + getKey: (item) => item.id, + }) + ) +} + type EntitiesCollection = ReturnType type EntityTypesCollection = ReturnType +type RunnersCollection = ReturnType type AppCollections = { entities: EntitiesCollection entityTypes: EntityTypesCollection + runners: RunnersCollection } const appCollectionsCache = new Map() @@ -113,6 +167,7 @@ function getOrCreateAppCollections(baseUrl: string): AppCollections { const collections = { entities: createEntitiesCollection(baseUrl), entityTypes: createEntityTypesCollection(baseUrl), + runners: createRunnersCollection(baseUrl), } appCollectionsCache.set(baseUrl, collections) return collections @@ -123,6 +178,7 @@ function cleanupAppCollections(baseUrl: string): void { if (!collections) return collections.entities.cleanup() collections.entityTypes.cleanup() + collections.runners.cleanup() appCollectionsCache.delete(baseUrl) } @@ -139,6 +195,7 @@ export async function preloadAppCollections( await Promise.all([ collections.entities.preload(), collections.entityTypes.preload(), + collections.runners.preload(), ]) return collections } @@ -280,6 +337,7 @@ function createForkEntity(baseUrl: string) { interface ElectricAgentsState { entitiesCollection: EntitiesCollection | null entityTypesCollection: EntityTypesCollection | null + runnersCollection: RunnersCollection | null spawnEntity: ReturnType | null killEntity: ReturnType | null forkEntity: ReturnType | null @@ -288,6 +346,7 @@ interface ElectricAgentsState { const ElectricAgentsContext = createContext({ entitiesCollection: null, entityTypesCollection: null, + runnersCollection: null, spawnEntity: null, killEntity: null, forkEntity: null, @@ -316,16 +375,19 @@ export function ElectricAgentsProvider({ return { entitiesCollection: null, entityTypesCollection: null, + runnersCollection: null, spawnEntity: null, killEntity: null, forkEntity: null, } } - const { entities, entityTypes } = getOrCreateAppCollections(baseUrl) + const { entities, entityTypes, runners } = + getOrCreateAppCollections(baseUrl) return { entitiesCollection: entities, entityTypesCollection: entityTypes, + runnersCollection: runners, spawnEntity: createSpawnAction(baseUrl, entities), killEntity: createKillAction(baseUrl, entities), forkEntity: createForkEntity(baseUrl), diff --git a/packages/agents-server-ui/src/lib/auth-fetch.test.ts b/packages/agents-server-ui/src/lib/auth-fetch.test.ts index a3b21b23d2..497460196c 100644 --- a/packages/agents-server-ui/src/lib/auth-fetch.test.ts +++ b/packages/agents-server-ui/src/lib/auth-fetch.test.ts @@ -1,5 +1,9 @@ import { afterEach, describe, expect, it, vi } from 'vitest' -import { registerActiveServerHeaders, serverFetch } from './auth-fetch' +import { + getActivePrincipal, + registerActiveServerHeaders, + serverFetch, +} from './auth-fetch' describe(`server fetch helpers`, () => { afterEach(() => { @@ -81,4 +85,14 @@ describe(`server fetch helpers`, () => { const headers = new Headers(fetchMock.mock.calls[0][1]?.headers) expect(headers.has(`authorization`)).toBe(false) }) + + it(`returns the active principal as a canonical principal URL`, () => { + registerActiveServerHeaders({ + name: `Tenant`, + url: `https://agents.example.test`, + headers: { 'electric-principal': `system:dev-local` }, + }) + + expect(getActivePrincipal()).toBe(`/principal/system%3Adev-local`) + }) }) diff --git a/packages/agents-server-ui/src/lib/auth-fetch.ts b/packages/agents-server-ui/src/lib/auth-fetch.ts index 77674bcc78..c8ecbb2dca 100644 --- a/packages/agents-server-ui/src/lib/auth-fetch.ts +++ b/packages/agents-server-ui/src/lib/auth-fetch.ts @@ -86,7 +86,11 @@ export function getConfiguredServerHeaders( } export function getActivePrincipal(): string { - return activeServerHeaders?.headers[`electric-principal`] ?? `unknown` + const principal = activeServerHeaders?.headers[`electric-principal`] + if (!principal) return `unknown` + return principal.startsWith(`/principal/`) + ? principal + : `/principal/${encodeURIComponent(principal)}` } export async function serverFetch( diff --git a/packages/agents-server-ui/src/router.tsx b/packages/agents-server-ui/src/router.tsx index 9c64ac6140..169ff5b4e4 100644 --- a/packages/agents-server-ui/src/router.tsx +++ b/packages/agents-server-ui/src/router.tsx @@ -47,6 +47,7 @@ import { AppearancePage } from './components/settings/pages/AppearancePage' import { CredentialsPage } from './components/settings/pages/CredentialsPage' import { ServersPage } from './components/settings/pages/ServersPage' import { McpServersPage } from './components/settings/pages/McpServersPage' +import { LocalRuntimePage } from './components/settings/pages/LocalRuntimePage' import styles from './router.module.css' const SETTINGS_CATEGORY_IDS: ReadonlyArray = [ @@ -54,6 +55,7 @@ const SETTINGS_CATEGORY_IDS: ReadonlyArray = [ `servers`, `credentials`, `appearance`, + `local-runtime`, `mcp-servers`, ] @@ -426,6 +428,8 @@ function SettingsCategoryPage(): React.ReactElement { return case `credentials`: return + case `local-runtime`: + return case `mcp-servers`: return case `general`: diff --git a/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md b/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md index 6d2f10b910..4a51231515 100644 --- a/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md +++ b/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md @@ -549,17 +549,17 @@ After the existing routes (line 90), add: runnersRouter.get(`/:id/health`, runnerHealth) ``` -- [x] **Step 4: Add `principalKeyFromUrl` import** +- [x] **Step 4: Add principal URL validation import** Add to the imports at the top of `runners-router.ts`: ```ts -import { principalKeyFromUrl } from '../principal.js' +import { isPrincipalUrl } from '../principal.js' ``` - [x] **Step 5: Update `registerRunner` handler to use `owner_principal` with strict URL validation** -No backwards compatibility for key-form principals. If `owner_principal` is provided it must be a valid principal URL accepted by `principalKeyFromUrl()` (e.g., `/principal/user%3Aalice`); otherwise the server derives it from `ctx.principal.url`. Callers must send URLs. +No backwards compatibility for key-form principals. If `owner_principal` is provided it must be a valid principal URL (e.g., `/principal/user%3Aalice`); otherwise the server derives it from `ctx.principal.url`. Callers must send URLs. In `registerRunner` (line 103-136): @@ -613,10 +613,10 @@ async function registerRunner( 400 ) } - if (!principalKeyFromUrl(ownerPrincipal)) { + if (!isPrincipalUrl(ownerPrincipal)) { throw new ElectricAgentsError( ErrCodeInvalidRequest, - `owner_principal must be a valid principal URL accepted by principalKeyFromUrl() (e.g. /principal/user%3Aalice), got: ${ownerPrincipal}`, + `owner_principal must be a valid principal URL (e.g. /principal/user%3Aalice), got: ${ownerPrincipal}`, 400 ) } @@ -672,7 +672,7 @@ async function listRunners( ctx: TenantContext ): Promise { const requestedOwner = firstQueryValue(request.query.owner_principal) - if (requestedOwner && !principalKeyFromUrl(requestedOwner)) { + if (requestedOwner && !isPrincipalUrl(requestedOwner)) { throw new ElectricAgentsError( ErrCodeInvalidRequest, `owner_principal must be a valid principal URL (e.g. /principal/user%3Aalice), got: ${requestedOwner}`, @@ -1277,7 +1277,7 @@ const DEFAULT_PULL_WAKE_OWNER_ID = `builtin-agents` const DEFAULT_PULL_WAKE_OWNER_PRINCIPAL = `/principal/system%3Abuiltin-agents` ``` -Then rename the function (line 131-139). `ELECTRIC_AGENTS_IDENTITY` is a principal key (`kind:id`), so convert it to a URL. The default is already a URL: +Then rename the function (line 131-139). `ELECTRIC_AGENTS_IDENTITY` is a principal identifier (`kind:id`), so convert it to a URL. The default is already a URL: ```ts // REPLACE: @@ -1323,20 +1323,15 @@ Update the `BuiltinAgentsServer` call (line 395): In `packages/agents-desktop/src/main.ts`: -Rename the constant (line 227-229). No backwards-compat fallback — clean break. Store a principal URL directly: +Replace the previous owner-user constant/env var. No backwards-compat fallback — clean break. Store a principal URL directly: ```ts -// REPLACE: -const PULL_WAKE_OWNER_USER_ID = - process.env.ELECTRIC_DESKTOP_PULL_WAKE_OWNER_USER_ID?.trim() || - `local-desktop` -// WITH: const PULL_WAKE_OWNER_PRINCIPAL = process.env.ELECTRIC_DESKTOP_PULL_WAKE_OWNER_PRINCIPAL?.trim() || `/principal/system%3Alocal-desktop` ``` -Rename the helper function (line 265-274). Do NOT use the `authorization` header as a principal source — that's a bearer token, not a principal key. When the request has auth headers, the server middleware extracts `ctx.principal` from them and uses `ctx.principal.url` as the owner. So when only auth is present (no explicit `electric-principal` header), return `undefined` to let the server derive the owner: +Rename the helper function (line 265-274). Do NOT use the `authorization` header as a principal source — that's a bearer token, not a principal identifier. When the request has auth headers, the server middleware extracts `ctx.principal` from them and uses `ctx.principal.url` as the owner. So when only auth is present (no explicit `electric-principal` header), return `undefined` to let the server derive the owner: ```ts // REPLACE: @@ -1347,7 +1342,7 @@ function runnerOwnerUserIdFromHeaders( return ( normalized.get(`authorization`)?.trim() || normalized.get(ELECTRIC_PRINCIPAL_HEADER)?.trim() || - PULL_WAKE_OWNER_USER_ID + PULL_WAKE_OWNER_PRINCIPAL ) } // WITH: diff --git a/packages/agents-server/src/principal.ts b/packages/agents-server/src/principal.ts index 767be23828..c873e8afc7 100644 --- a/packages/agents-server/src/principal.ts +++ b/packages/agents-server/src/principal.ts @@ -20,7 +20,7 @@ const PRINCIPAL_KINDS = new Set([ export function parsePrincipalKey(input: string): Principal { const colon = input.indexOf(`:`) - if (colon <= 0) throw new Error(`Invalid principal key`) + if (colon <= 0) throw new Error(`Invalid principal identifier`) const kind = input.slice(0, colon) as PrincipalKind const id = input.slice(colon + 1) if (!PRINCIPAL_KINDS.has(kind)) throw new Error(`Invalid principal kind`) @@ -33,21 +33,24 @@ export function principalUrl(key: string): string { return parsePrincipalKey(key).url } -export function principalKeyFromUrl(url: string): string | null { +export function parsePrincipalUrl(url: string): Principal | null { if (!url.startsWith(`/principal/`)) return null const segment = url.slice(`/principal/`.length) if (!segment || segment.includes(`/`)) return null try { - const key = decodeURIComponent(segment) // Principal URLs produced by parsePrincipalKey/principalUrl are canonical // encoded single path segments, but accept legacy unencoded single-segment // URLs here so callers can canonicalize them via parsePrincipalKey(key).url. - return parsePrincipalKey(key).key + return parsePrincipalKey(decodeURIComponent(segment)) } catch { return null } } +export function isPrincipalUrl(url: string): boolean { + return parsePrincipalUrl(url) !== null +} + export function getPrincipalFromRequest(request: Request): Principal | null { const value = request.headers.get(ELECTRIC_PRINCIPAL_HEADER) return value ? parsePrincipalKey(value) : null @@ -66,9 +69,8 @@ const BUILT_IN_SYSTEM_PRINCIPAL_IDS = new Set([ export function isBuiltInSystemPrincipalUrl(url: string | undefined): boolean { if (!url?.startsWith(`/principal/`)) return false try { - const key = principalKeyFromUrl(url) - if (!key) return false - const principal = parsePrincipalKey(key) + const principal = parsePrincipalUrl(url) + if (!principal) return false return ( principal.kind === `system` && BUILT_IN_SYSTEM_PRINCIPAL_IDS.has(principal.id) @@ -84,9 +86,8 @@ export function principalFromCreatedBy( | { url: string; key?: string | null; kind?: string; id?: string } | undefined { if (!createdBy) return undefined - const key = principalKeyFromUrl(createdBy) - if (!key) return { url: createdBy, key: null } - const principal = parsePrincipalKey(key) + const principal = parsePrincipalUrl(createdBy) + if (!principal) return { url: createdBy, key: null } return { url: principal.url, key: principal.key, diff --git a/packages/agents-server/src/routing/electric-proxy-router.ts b/packages/agents-server/src/routing/electric-proxy-router.ts index 0e658f4d8c..5142d1fc39 100644 --- a/packages/agents-server/src/routing/electric-proxy-router.ts +++ b/packages/agents-server/src/routing/electric-proxy-router.ts @@ -38,6 +38,7 @@ async function proxyElectric( electricUrl: ctx.electricUrl, electricSecret: ctx.electricSecret, tenantId: ctx.service, + principalUrl: ctx.principal.url, }) const headers = new Headers(request.headers) headers.delete(`host`) diff --git a/packages/agents-server/src/routing/runners-router.ts b/packages/agents-server/src/routing/runners-router.ts index 71c88f3e07..ab08a6188e 100644 --- a/packages/agents-server/src/routing/runners-router.ts +++ b/packages/agents-server/src/routing/runners-router.ts @@ -14,7 +14,7 @@ import { import { routeBody, withSchema } from './schema.js' import { subscriptionIdForDispatchTarget } from './dispatch-policy.js' import { withLeadingSlash } from './tenant-stream-paths.js' -import { principalFromCreatedBy, principalKeyFromUrl } from '../principal.js' +import { isPrincipalUrl, principalFromCreatedBy } from '../principal.js' import type { JsonRouteRequest } from './schema.js' import type { RouterType } from 'itty-router' import type { TenantContext } from './context.js' @@ -116,7 +116,7 @@ async function registerRunner( 400 ) } - if (!principalKeyFromUrl(ownerPrincipal)) { + if (!isPrincipalUrl(ownerPrincipal)) { throw new ElectricAgentsError( ErrCodeInvalidRequest, `owner_principal must be a valid principal URL (e.g. /principal/user%3Aalice), got: ${ownerPrincipal}`, @@ -150,7 +150,7 @@ async function listRunners( ctx: TenantContext ): Promise { const requestedOwner = firstQueryValue(request.query.owner_principal) - if (requestedOwner && !principalKeyFromUrl(requestedOwner)) { + if (requestedOwner && !isPrincipalUrl(requestedOwner)) { throw new ElectricAgentsError( ErrCodeInvalidRequest, `owner_principal must be a valid principal URL (e.g. /principal/user%3Aalice), got: ${requestedOwner}`, diff --git a/packages/agents-server/src/utils/server-utils.ts b/packages/agents-server/src/utils/server-utils.ts index e78805ed81..32a48a8e36 100644 --- a/packages/agents-server/src/utils/server-utils.ts +++ b/packages/agents-server/src/utils/server-utils.ts @@ -95,6 +95,7 @@ export function buildElectricProxyTarget(options: { electricUrl: string electricSecret?: string tenantId: string + principalUrl?: string }): URL { const targetPath = options.incomingUrl.pathname.replace( `/_electric/electric`, @@ -132,7 +133,9 @@ export function buildElectricProxyTarget(options: { `columns`, `"tenant_id","id","owner_principal","label","kind","admin_status","wake_stream","wake_stream_offset","last_seen_at","liveness_lease_expires_at","diagnostics","created_at","updated_at"` ) - applyTenantShapeWhere(target, options.tenantId) + applyTenantShapeWhere(target, options.tenantId, [ + `owner_principal = ${sqlStringLiteral(options.principalUrl ?? ``)}`, + ]) } else if (table === `entity_dispatch_state`) { target.searchParams.set( `columns`, @@ -231,8 +234,15 @@ export function decodeJsonObject( return null } -function applyTenantShapeWhere(target: URL, tenantId: string): void { - const tenantWhere = `tenant_id = ${sqlStringLiteral(tenantId)}` +function applyTenantShapeWhere( + target: URL, + tenantId: string, + extraConditions: Array = [] +): void { + const tenantWhere = [ + `tenant_id = ${sqlStringLiteral(tenantId)}`, + ...extraConditions, + ].join(` AND `) const existingWhere = target.searchParams.get(`where`) target.searchParams.set( `where`, diff --git a/packages/agents-server/test/principal.test.ts b/packages/agents-server/test/principal.test.ts index e5075e059a..ad424e6de4 100644 --- a/packages/agents-server/test/principal.test.ts +++ b/packages/agents-server/test/principal.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from 'vitest' import { + parsePrincipalUrl, parsePrincipalKey, - principalKeyFromUrl, principalUrl, } from '../src/principal.js' @@ -19,7 +19,7 @@ describe(`principal parser`, () => { const url = `/principal/${encodeURIComponent(key)}` expect(principal.url).toBe(url) expect(principalUrl(key)).toBe(url) - expect(principalKeyFromUrl(url)).toBe(key) + expect(parsePrincipalUrl(url)?.key).toBe(key) }) } @@ -32,8 +32,8 @@ describe(`principal parser`, () => { it(`encodes URL-unsafe principal ids canonically`, () => { const principal = parsePrincipalKey(`user:alice@example.com`) expect(principal.url).toBe(`/principal/user%3Aalice%40example.com`) - expect(principalKeyFromUrl(principal.url)).toBe(`user:alice@example.com`) - expect(principalKeyFromUrl(`/principal/user:alice@example.com`)).toBe( + expect(parsePrincipalUrl(principal.url)?.key).toBe(`user:alice@example.com`) + expect(parsePrincipalUrl(`/principal/user:alice@example.com`)?.key).toBe( `user:alice@example.com` ) }) diff --git a/packages/agents-server/test/server-utils.test.ts b/packages/agents-server/test/server-utils.test.ts new file mode 100644 index 0000000000..de7796e55c --- /dev/null +++ b/packages/agents-server/test/server-utils.test.ts @@ -0,0 +1,34 @@ +import { describe, expect, it } from 'vitest' +import { buildElectricProxyTarget } from '../src/utils/server-utils' + +function shapeTarget(query: string): URL { + return buildElectricProxyTarget({ + incomingUrl: new URL(`http://server/_electric/electric/v1/shape?${query}`), + electricUrl: `http://electric.local`, + tenantId: `tenant-test`, + principalUrl: `/principal/user%3Aowner%40example.com`, + }) +} + +describe(`server utils`, () => { + it(`owner-scopes runner shapes to the authenticated principal`, () => { + const target = shapeTarget(`table=runners`) + + expect(target.pathname).toBe(`/v1/shape`) + expect(target.searchParams.get(`table`)).toBe(`runners`) + expect(target.searchParams.get(`columns`)).toContain(`"owner_principal"`) + expect(target.searchParams.get(`where`)).toBe( + `tenant_id = 'tenant-test' AND owner_principal = '/principal/user%3Aowner%40example.com'` + ) + }) + + it(`combines runner owner scoping with Electric protocol where clauses`, () => { + const target = shapeTarget( + `table=runners&where=${encodeURIComponent(`kind = 'local'`)}` + ) + + expect(target.searchParams.get(`where`)).toBe( + `tenant_id = 'tenant-test' AND owner_principal = '/principal/user%3Aowner%40example.com' AND (kind = 'local')` + ) + }) +}) diff --git a/packages/agents/src/bootstrap.ts b/packages/agents/src/bootstrap.ts index 382866aacb..9ff077aadc 100644 --- a/packages/agents/src/bootstrap.ts +++ b/packages/agents/src/bootstrap.ts @@ -143,7 +143,7 @@ export async function createBuiltinAgentHandler( subscriptionPathForType: (name) => `/${name}/*/main`, defaultDispatchPolicyForType, serverHeaders, - idleTimeout: 5_000, + idleTimeout: 5 * 60_000, createElectricTools, publicUrl, name: runtimeName ?? `builtin-agents`, From 944c272090482d5dcf0d9972272f883c88a71b6a Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Sun, 17 May 2026 06:51:43 -0600 Subject: [PATCH 13/37] fix(agents): harden pull-wake runner lifecycle and error handling State machine, concurrent claim limits, exponential reconnect backoff, and granular health status. onError is now reporting-only with fallback console.error logging. stop() rethrows drainWakes errors to callers. Co-Authored-By: Claude Opus 4.6 --- packages/agents-runtime/src/create-handler.ts | 10 +- packages/agents-runtime/src/index.ts | 3 +- packages/agents-runtime/src/process-wake.ts | 4 +- .../agents-runtime/src/pull-wake-runner.ts | 327 +++++++- .../test/create-handler.test.ts | 34 +- .../test/pull-wake-runner.test.ts | 717 +++++++++++++++--- packages/agents-server/src/routing/hooks.ts | 9 +- .../agents-server/test/routing-hooks.test.ts | 3 + packages/agents/src/server.ts | 3 +- 9 files changed, 926 insertions(+), 184 deletions(-) diff --git a/packages/agents-runtime/src/create-handler.ts b/packages/agents-runtime/src/create-handler.ts index 4317ed8ccd..8128cb82bc 100644 --- a/packages/agents-runtime/src/create-handler.ts +++ b/packages/agents-runtime/src/create-handler.ts @@ -4,7 +4,7 @@ */ import { zodToJsonSchema } from 'zod-to-json-schema' -import { processWebhookWake } from './process-wake' +import { processWake } from './process-wake' import { getEntityType, listEntityTypes } from './define-entity' import { DEFAULT_OUTPUT_SCHEMAS } from './default-output-schemas' import { passthrough } from './entity-schema' @@ -118,13 +118,11 @@ export interface RuntimeRouter { options?: Pick ) => void - /** - * Dispatch an already-parsed webhook wake notification. - */ + /** Dispatch an already-parsed webhook wake notification. */ dispatchWebhookWake: (notification: WebhookNotification) => void /** - * Wait for all in-flight webhook wake handlers to settle. + * Wait for all in-flight wake handlers to settle. * Throws any wake errors instead of hiding them behind logs. */ drainWakes: () => Promise @@ -240,7 +238,7 @@ export function createRuntimeRouter( const wakeLabel = notification.entity?.url ?? notification.streamPath const controller = new AbortController() const wake: Promise = Promise.resolve( - processWebhookWake(notification, { + processWake(notification, { ...wakeConfig, ...options, shutdownSignal: controller.signal, diff --git a/packages/agents-runtime/src/index.ts b/packages/agents-runtime/src/index.ts index e5f9a1d1d6..5668a1f3b2 100644 --- a/packages/agents-runtime/src/index.ts +++ b/packages/agents-runtime/src/index.ts @@ -205,7 +205,7 @@ export type { TaggedQuery, } from './observation-sources' -export { processWake, processWebhookWake } from './process-wake' +export { processWake } from './process-wake' export type { ProcessWakeConfig } from './types' export { DEFAULT_OUTPUT_SCHEMAS } from './default-output-schemas' @@ -240,6 +240,7 @@ export type { PullWakeRunner, PullWakeRunnerConfig, PullWakeRunnerHealth, + PullWakeRunnerStatus, PullWakeStreamResponse, } from './pull-wake-runner' diff --git a/packages/agents-runtime/src/process-wake.ts b/packages/agents-runtime/src/process-wake.ts index f74caeabb9..f909a547ea 100644 --- a/packages/agents-runtime/src/process-wake.ts +++ b/packages/agents-runtime/src/process-wake.ts @@ -297,7 +297,7 @@ function createInFlightTracker() { } } -export async function processWebhookWake( +export async function processWake( notification: WebhookNotification, config: ProcessWakeConfig ): Promise { @@ -1820,8 +1820,6 @@ export async function processWebhookWake( return result } -export const processWake: typeof processWebhookWake = processWebhookWake - async function sendDone( callback: string, token: string, diff --git a/packages/agents-runtime/src/pull-wake-runner.ts b/packages/agents-runtime/src/pull-wake-runner.ts index 67f9662a74..d9615550c9 100644 --- a/packages/agents-runtime/src/pull-wake-runner.ts +++ b/packages/agents-runtime/src/pull-wake-runner.ts @@ -26,10 +26,12 @@ export interface PullWakeRunnerConfig { claimTokenHeader?: ProcessWakeConfig[`claimTokenHeader`] wakeStreamPath?: string heartbeatIntervalMs?: number + eventHeartbeatThrottleMs?: number leaseMs?: number + maxConcurrentClaims?: number heartbeatPath?: string claimPath?: string - onError?: (error: Error) => boolean | void + onError?: (error: Error) => void streamFactory?: (opts: { url: string headers?: Record @@ -54,8 +56,17 @@ export interface PullWakeRunner { getHealth: () => PullWakeRunnerHealth } +export type PullWakeRunnerStatus = + | `stopped` + | `starting` + | `connecting` + | `streaming` + | `reconnecting` + | `stopping` + export interface PullWakeRunnerHealth { running: boolean + status: PullWakeRunnerStatus offset: string | undefined started_at: string | null stream_connected: boolean @@ -74,13 +85,29 @@ export interface PullWakeRunnerHealth { claims_failed: number } +type PullWakeRunnerState = + | `stopped` + | `starting` + | `running.connecting` + | `running.streaming` + | `running.reconnecting` + | `stopping` + +const DEFAULT_MAX_CONCURRENT_CLAIMS = 10 +const INITIAL_RECONNECT_BACKOFF_MS = 1_000 +const MAX_RECONNECT_BACKOFF_MS = 30_000 +const CLAIM_ACTOR_STOP_GRACE_MS = 1_000 +const DEFAULT_EVENT_HEARTBEAT_THROTTLE_MS = 2_000 + export function createPullWakeRunner( config: PullWakeRunnerConfig ): PullWakeRunner { + let state: PullWakeRunnerState = `stopped` let controller: AbortController | null = null let loop: Promise | null = null let response: PullWakeStreamResponse | null = null let heartbeatTimer: ReturnType | null = null + let eventHeartbeatTimer: ReturnType | null = null let currentOffset = config.offset let startedAt: string | null = null let streamConnected = false @@ -97,6 +124,11 @@ export function createPullWakeRunner( let claimsSucceeded = 0 let claimsSkipped = 0 let claimsFailed = 0 + let acceptingClaims = false + let activeClaimCount = 0 + let runGeneration = 0 + let nextReconnectBackoffMs = INITIAL_RECONNECT_BACKOFF_MS + const claimActors = new Map, number>() const wakePath = config.wakeStreamPath ?? @@ -104,6 +136,10 @@ export function createPullWakeRunner( const wakeUrl = appendPathToUrl(config.baseUrl, wakePath) const heartbeatIntervalMs = config.heartbeatIntervalMs ?? DEFAULT_RUNNER_HEARTBEAT_INTERVAL_MS + const eventHeartbeatThrottleMs = Math.max( + 0, + config.eventHeartbeatThrottleMs ?? DEFAULT_EVENT_HEARTBEAT_THROTTLE_MS + ) const leaseMs = config.leaseMs ?? heartbeatIntervalMs * 3 const heartbeatPath = config.heartbeatPath ?? @@ -113,11 +149,33 @@ export function createPullWakeRunner( config.claimPath ?? `/_electric/runners/${encodeURIComponent(config.runnerId)}/claim` const claimUrl = appendPathToUrl(config.baseUrl, claimPath) + const maxConcurrentClaims = Math.max( + 1, + Math.floor(config.maxConcurrentClaims ?? DEFAULT_MAX_CONCURRENT_CLAIMS) + ) + + const toStatus = (): PullWakeRunnerStatus => { + switch (state) { + case `stopped`: + return `stopped` + case `starting`: + return `starting` + case `running.connecting`: + return `connecting` + case `running.streaming`: + return `streaming` + case `running.reconnecting`: + return `reconnecting` + case `stopping`: + return `stopping` + } + } const buildDiagnostics = (): Omit< PullWakeRunnerHealth, `running` | `offset` > => ({ + status: toStatus(), started_at: startedAt, stream_connected: streamConnected, stream_connected_since: streamConnectedSince, @@ -159,7 +217,22 @@ export function createPullWakeRunner( const error = err instanceof Error ? err : new Error(String(err)) lastError = error.message lastErrorAt = new Date().toISOString() - if (config.onError?.(error) !== true) throw error + try { + config.onError?.(error) + } catch (reporterError) { + // onError is reporting-only; reporters must not control runner lifecycle. + console.error(`Pull-wake runner onError callback failed`, reporterError) + } + } + + const notifyHeartbeatChange = (): void => { + const signal = controller?.signal + if (!signal || signal.aborted || heartbeatIntervalMs <= 0) return + if (eventHeartbeatTimer) return + eventHeartbeatTimer = setTimeout(() => { + eventHeartbeatTimer = null + void heartbeat(signal) + }, eventHeartbeatThrottleMs) } const heartbeat = async (signal: AbortSignal): Promise => { @@ -189,7 +262,7 @@ export function createPullWakeRunner( } catch (err) { if (!signal.aborted) { lastHeartbeatOk = false - config.onError?.(err instanceof Error ? err : new Error(String(err))) + reportError(err) } } } @@ -207,6 +280,10 @@ export function createPullWakeRunner( clearInterval(heartbeatTimer) heartbeatTimer = null } + if (eventHeartbeatTimer) { + clearTimeout(eventHeartbeatTimer) + eventHeartbeatTimer = null + } } const streamFactory = @@ -225,7 +302,7 @@ export function createPullWakeRunner( offset: opts.offset, signal: opts.signal, onError: (error) => { - config.onError?.(error) + reportError(error) return {} }, })) as PullWakeStreamResponse @@ -234,12 +311,14 @@ export function createPullWakeRunner( const recordClaimSkipped = (): null => { lastClaimResult = `no_work` claimsSkipped++ + notifyHeartbeatChange() return null } const recordClaimError = (): void => { lastClaimResult = `error` claimsFailed++ + notifyHeartbeatChange() } const claimWake = async ( @@ -248,9 +327,11 @@ export function createPullWakeRunner( ): Promise => { lastClaimAt = new Date().toISOString() lastClaimResult = null - const headers = new Headers(await resolveHeaders()) - headers.set(`content-type`, `application/json`) + notifyHeartbeatChange() + let claimErrorRecorded = false try { + const headers = new Headers(await resolveHeaders()) + headers.set(`content-type`, `application/json`) const response = await fetch(claimUrl, { method: `POST`, headers, @@ -267,6 +348,7 @@ export function createPullWakeRunner( return recordClaimSkipped() } recordClaimError() + claimErrorRecorded = true throw new Error( `Pull-wake claim failed for ${config.runnerId}: ${response.status} ${text}` ) @@ -277,89 +359,264 @@ export function createPullWakeRunner( if (notification.done) return recordClaimSkipped() lastClaimResult = `claimed` claimsSucceeded++ + notifyHeartbeatChange() return notification } catch (err) { - if (lastClaimResult === null || lastClaimResult === `claimed`) { + if (signal.aborted) { + throw err + } + if (!claimErrorRecorded) { recordClaimError() } throw err } } - const run = async (): Promise => { - const signal = controller!.signal + const isRunningState = (): boolean => + state === `starting` || state.startsWith(`running.`) + + const waitForClaimCapacity = async ( + signal: AbortSignal + ): Promise => { + const abortPromise = new Promise((resolve) => { + if (signal.aborted) { + resolve() + return + } + signal.addEventListener(`abort`, () => resolve(), { once: true }) + }) + + while ( + acceptingClaims && + !signal.aborted && + activeClaimCount >= maxConcurrentClaims + ) { + const inFlight = [...claimActors.keys()] + if (inFlight.length === 0) return true + await Promise.race([...inFlight, abortPromise]).catch(() => undefined) + } + return acceptingClaims && !signal.aborted + } + + const claimAndDispatch = async ( + event: PullWakeEvent, + signal: AbortSignal + ): Promise => { try { - response = await streamFactory({ - url: wakeUrl, - headers: await resolveHeaders(), - offset: currentOffset, - signal, + const notification = await claimWake(event, signal) + if (!notification) return + if (!acceptingClaims || signal.aborted) { + return + } + try { + config.runtime.dispatchWake(notification, { + claimHeaders: resolveClaimHeaders, + claimTokenHeader: config.claimTokenHeader, + }) + } catch (err) { + reportError(err) + notifyHeartbeatChange() + return + } + lastDispatchAt = new Date().toISOString() + notifyHeartbeatChange() + } catch (err) { + if (!signal.aborted) { + reportError(err) + } + } + } + + const spawnClaimActor = ( + event: PullWakeEvent, + signal: AbortSignal, + generation: number + ): void => { + activeClaimCount++ + let actor: Promise + actor = claimAndDispatch(event, signal).finally(() => { + if (claimActors.get(actor) === generation) { + activeClaimCount-- + } + claimActors.delete(actor) + }) + claimActors.set(actor, generation) + } + + const waitForClaimActors = async ( + timeoutMs = CLAIM_ACTOR_STOP_GRACE_MS + ): Promise => { + const deadline = Date.now() + timeoutMs + while (claimActors.size > 0) { + const remainingMs = deadline - Date.now() + if (remainingMs <= 0) return false + const result = await new Promise<`settled` | `timeout`>((resolve) => { + const timer = setTimeout(() => resolve(`timeout`), remainingMs) + void Promise.allSettled([...claimActors.keys()]).then(() => { + clearTimeout(timer) + resolve(`settled`) + }) }) - streamConnected = true - streamConnectedSince = new Date().toISOString() + if (result === `timeout`) return false + } + return true + } + + const sleep = async (ms: number, signal: AbortSignal): Promise => { + if (ms <= 0 || signal.aborted) return + await new Promise((resolve) => { + const timer = setTimeout(resolve, ms) + signal.addEventListener( + `abort`, + () => { + clearTimeout(timer) + resolve() + }, + { once: true } + ) + }) + } + + const consumeWakeStream = async ( + signal: AbortSignal, + generation: number + ): Promise => { + response = await streamFactory({ + url: wakeUrl, + headers: await resolveHeaders(), + offset: currentOffset, + signal, + }) + state = `running.streaming` + streamConnected = true + streamConnectedSince = new Date().toISOString() + nextReconnectBackoffMs = INITIAL_RECONNECT_BACKOFF_MS + notifyHeartbeatChange() + + try { for await (const event of response.jsonStream()) { if (signal.aborted) break - if (event?.type !== `wake`) continue - eventsReceived++ - const notification = await claimWake(event, signal) - if (notification) { - config.runtime.dispatchWake(notification, { - claimHeaders: resolveClaimHeaders, - claimTokenHeader: config.claimTokenHeader, - }) - lastDispatchAt = new Date().toISOString() - await config.runtime.drainWakes() + if (event?.type === `wake`) { + eventsReceived++ + notifyHeartbeatChange() + if (await waitForClaimCapacity(signal)) { + spawnClaimActor(event, signal, generation) + } else { + claimsSkipped++ + notifyHeartbeatChange() + } + } + if (response.offset !== undefined) { + currentOffset = response.offset + notifyHeartbeatChange() } - if (response.offset !== undefined) currentOffset = response.offset } await response.closed?.catch((err) => { if (!signal.aborted) throw err }) - } catch (err) { - if (!signal.aborted) { - reconnectCount++ - reportError(err) + } finally { + streamConnected = false + streamConnectedSince = null + response = null + if (!signal.aborted) notifyHeartbeatChange() + } + } + + const run = async (): Promise => { + const signal = controller!.signal + acceptingClaims = true + try { + while (!signal.aborted) { + state = `running.connecting` + notifyHeartbeatChange() + try { + await consumeWakeStream(signal, runGeneration) + if (!signal.aborted) { + state = `running.reconnecting` + notifyHeartbeatChange() + const backoffMs = nextReconnectBackoffMs + nextReconnectBackoffMs = Math.min( + nextReconnectBackoffMs * 2, + MAX_RECONNECT_BACKOFF_MS + ) + await sleep(backoffMs, signal) + } + } catch (err) { + if (!signal.aborted) { + reconnectCount++ + reportError(err) + state = `running.reconnecting` + notifyHeartbeatChange() + const backoffMs = nextReconnectBackoffMs + nextReconnectBackoffMs = Math.min( + nextReconnectBackoffMs * 2, + MAX_RECONNECT_BACKOFF_MS + ) + await sleep(backoffMs, signal) + } + } } } finally { + acceptingClaims = false streamConnected = false - stopHeartbeat() + streamConnectedSince = null response = null controller = null + if (state !== `stopping`) state = `stopped` } } return { start() { if (loop) return + state = `starting` controller = new AbortController() + runGeneration++ startedAt = new Date().toISOString() startHeartbeat(controller.signal) loop = run().finally(() => { loop = null + stopHeartbeat() }) }, async stop() { + if (state === `stopped`) return + state = `stopping` + acceptingClaims = false controller?.abort() stopHeartbeat() response?.cancel?.(new Error(`pull wake runner stopped`)) + if (!(await waitForClaimActors())) { + claimActors.clear() + activeClaimCount = 0 + } config.runtime.abortWakes() await loop?.catch((err) => { if (!(err instanceof Error && err.name === `AbortError`)) throw err }) - await config.runtime.drainWakes() + let drainError: unknown + try { + await config.runtime.drainWakes() + } catch (err) { + reportError(err) + drainError = err + } finally { + state = `stopped` + } + if (drainError) throw drainError }, async waitForStopped() { await loop }, get running() { - return loop !== null + return isRunningState() }, get offset() { return currentOffset }, getHealth(): PullWakeRunnerHealth { return { - running: loop !== null, + running: isRunningState(), offset: currentOffset, ...buildDiagnostics(), } diff --git a/packages/agents-runtime/test/create-handler.test.ts b/packages/agents-runtime/test/create-handler.test.ts index a4dbba30cb..7907b2d522 100644 --- a/packages/agents-runtime/test/create-handler.test.ts +++ b/packages/agents-runtime/test/create-handler.test.ts @@ -11,13 +11,12 @@ import type { StandardSchemaV1, } from '@standard-schema/spec' -const { processWebhookWakeMock } = vi.hoisted(() => ({ - processWebhookWakeMock: vi.fn(), +const { processWakeMock } = vi.hoisted(() => ({ + processWakeMock: vi.fn(), })) vi.mock(`../src/process-wake`, () => ({ - processWebhookWake: processWebhookWakeMock, - processWake: processWebhookWakeMock, + processWake: processWakeMock, })) function makeStandardSchema( @@ -63,10 +62,14 @@ function makeResponse() { } } +function flushAsyncWork(): Promise { + return new Promise((resolve) => setTimeout(resolve, 0)) +} + describe(`createRuntimeHandler`, () => { beforeEach(() => { clearRegistry() - processWebhookWakeMock.mockReset() + processWakeMock.mockReset() }) afterEach(() => { @@ -77,7 +80,7 @@ describe(`createRuntimeHandler`, () => { defineEntity(`test-agent`, { handler: async () => {} }) let resolveWake!: () => void - processWebhookWakeMock.mockImplementation( + processWakeMock.mockImplementation( () => new Promise((resolve) => { resolveWake = resolve @@ -118,7 +121,7 @@ describe(`createRuntimeHandler`, () => { 'content-type': `application/json`, }) expect(res.end).toHaveBeenCalledWith(JSON.stringify({ ok: true })) - expect(processWebhookWakeMock).toHaveBeenCalledWith( + expect(processWakeMock).toHaveBeenCalledWith( notification, expect.objectContaining({ baseUrl: `http://localhost:3000`, @@ -135,7 +138,7 @@ describe(`createRuntimeHandler`, () => { defineEntity(`test-agent`, { handler: async () => {} }) let resolveWake!: () => void - processWebhookWakeMock.mockImplementation( + processWakeMock.mockImplementation( () => new Promise((resolve) => { resolveWake = resolve @@ -195,7 +198,7 @@ describe(`createRuntimeHandler`, () => { it(`records wake errors in debugState() until drained`, async () => { defineEntity(`test-agent`, { handler: async () => {} }) - processWebhookWakeMock.mockRejectedValueOnce(new Error(`wake failed`)) + processWakeMock.mockRejectedValueOnce(new Error(`wake failed`)) const handler = createRuntimeHandler({ baseUrl: `http://localhost:3000`, @@ -230,8 +233,7 @@ describe(`createRuntimeHandler`, () => { ) expect(response.status).toBe(200) - await Promise.resolve() - await Promise.resolve() + await flushAsyncWork() expect(handler.debugState()).toMatchObject({ pendingWakeCount: 0, @@ -260,7 +262,7 @@ describe(`createRuntimeHandler`, () => { await handler.onEnter(req, res) - expect(processWebhookWakeMock).not.toHaveBeenCalled() + expect(processWakeMock).not.toHaveBeenCalled() expect(res.writeHead).toHaveBeenCalledWith(400, { 'content-type': `application/json`, }) @@ -293,7 +295,7 @@ describe(`createRuntimeHandler`, () => { await handler.onEnter(req, res) - expect(processWebhookWakeMock).not.toHaveBeenCalled() + expect(processWakeMock).not.toHaveBeenCalled() expect(res.writeHead).toHaveBeenCalledWith(400, { 'content-type': `application/json`, }) @@ -314,7 +316,7 @@ describe(`createRuntimeHandler`, () => { ) expect(response).toBeNull() - expect(processWebhookWakeMock).not.toHaveBeenCalled() + expect(processWakeMock).not.toHaveBeenCalled() }) it(`returns 503 for unknown entity types`, async () => { @@ -354,7 +356,7 @@ describe(`createRuntimeHandler`, () => { await expect(response.json()).resolves.toMatchObject({ error: expect.stringContaining(`nonexistent-agent`), }) - expect(processWebhookWakeMock).not.toHaveBeenCalled() + expect(processWakeMock).not.toHaveBeenCalled() }) it(`routes matching fetch requests through handleRequest`, async () => { @@ -396,7 +398,7 @@ describe(`createRuntimeHandler`, () => { expect(response).toBeInstanceOf(Response) expect(response?.status).toBe(200) await expect(response?.json()).resolves.toEqual({ ok: true }) - expect(processWebhookWakeMock).toHaveBeenCalledWith( + expect(processWakeMock).toHaveBeenCalledWith( notification, expect.objectContaining({ baseUrl: `http://localhost:3000`, diff --git a/packages/agents-runtime/test/pull-wake-runner.test.ts b/packages/agents-runtime/test/pull-wake-runner.test.ts index ee73839096..a88d3150d2 100644 --- a/packages/agents-runtime/test/pull-wake-runner.test.ts +++ b/packages/agents-runtime/test/pull-wake-runner.test.ts @@ -15,42 +15,90 @@ vi.mock(`@durable-streams/client`, () => ({ DurableStream: durableStreamMocks.DurableStream, })) +function wakeEvent(id: string): PullWakeEvent { + return { + type: `wake`, + subscription_id: `runner:runner-1`, + stream: `chat/${id}/main`, + generation: 7, + ts: 123, + } +} + +function notification(id: string): WakeNotification { + return { + consumerId: `wake-${id}`, + epoch: 7, + wakeId: `wake-${id}`, + streamPath: `/chat/${id}/main`, + streams: [{ path: `/chat/${id}/main`, offset: `12` }], + callback: `http://server/_electric/callback-forward/wake-${id}`, + claimToken: `claim-token-${id}`, + entity: { + type: `chat`, + status: `idle`, + url: `/chat/${id}`, + streams: { main: `/chat/${id}/main`, error: `/chat/${id}/error` }, + }, + } +} + +function deferred(): { + promise: Promise + resolve: (value: T | PromiseLike) => void + reject: (reason?: unknown) => void +} { + let resolve!: (value: T | PromiseLike) => void + let reject!: (reason?: unknown) => void + const promise = new Promise((res, rej) => { + resolve = res + reject = rej + }) + return { promise, resolve, reject } +} + +async function waitFor( + assertion: () => void, + timeoutMs = 1_000 +): Promise { + const started = Date.now() + let lastError: unknown + while (Date.now() - started < timeoutMs) { + try { + assertion() + return + } catch (err) { + lastError = err + await new Promise((resolve) => setTimeout(resolve, 5)) + } + } + throw lastError +} + +function runtime() { + return { + dispatchWake: vi.fn(), + drainWakes: vi.fn(async () => undefined), + abortWakes: vi.fn(), + } +} + describe(`createPullWakeRunner`, () => { afterEach(() => { durableStreamMocks.DurableStream.mockClear() durableStreamMocks.stream.mockReset() + vi.useRealTimers() vi.unstubAllGlobals() }) it(`claims compact DS wake events before dispatching runtime wakes`, async () => { - const event: PullWakeEvent = { - type: `wake`, - subscription_id: `runner:runner-1`, - stream: `chat/one/main`, - generation: 7, - ts: 123, - } - const notification: WakeNotification = { - consumerId: `wake-1`, - epoch: 7, - wakeId: `wake-1`, - streamPath: `/chat/one/main`, - streams: [{ path: `/chat/one/main`, offset: `12` }], - callback: `http://server/_electric/callback-forward/wake-1`, - claimToken: `claim-token`, - entity: { - type: `chat`, - status: `idle`, - url: `/chat/one`, - streams: { main: `/chat/one/main`, error: `/chat/one/error` }, - }, - } + const event = wakeEvent(`one`) + const claimed = notification(`one`) const fetchMock = vi.fn(async (_input: RequestInfo | URL) => - Response.json(notification) + Response.json(claimed) ) vi.stubGlobal(`fetch`, fetchMock) - const dispatchWake = vi.fn() - const drainWakes = vi.fn(async () => undefined) + const testRuntime = runtime() const streamFactory = vi.fn(async () => ({ offset: `42`, async *jsonStream() { @@ -62,11 +110,7 @@ describe(`createPullWakeRunner`, () => { const runner = createPullWakeRunner({ baseUrl: `http://server`, runnerId: `runner-1`, - runtime: { - dispatchWake, - drainWakes, - abortWakes: vi.fn(), - }, + runtime: testRuntime, headers: { 'x-test-runner': `runner-1` }, claimHeaders: { authorization: `Bearer session-token` }, claimTokenHeader: `electric-claim-token`, @@ -75,7 +119,9 @@ describe(`createPullWakeRunner`, () => { }) runner.start() - await runner.waitForStopped() + await waitFor(() => { + expect(testRuntime.dispatchWake).toHaveBeenCalledTimes(1) + }) expect(streamFactory).toHaveBeenCalledWith( expect.objectContaining({ @@ -89,22 +135,19 @@ describe(`createPullWakeRunner`, () => { body: JSON.stringify(event), }) ) - expect(dispatchWake).toHaveBeenCalledWith(notification, { + expect(testRuntime.dispatchWake).toHaveBeenCalledWith(claimed, { claimHeaders: expect.any(Function), claimTokenHeader: `electric-claim-token`, }) - expect(drainWakes).toHaveBeenCalledTimes(1) + expect(testRuntime.drainWakes).not.toHaveBeenCalled() expect(runner.offset).toBe(`42`) + + await runner.stop() + expect(testRuntime.drainWakes).toHaveBeenCalledTimes(1) }) it(`skips stale wake events when claim returns no pending work`, async () => { - const event: PullWakeEvent = { - type: `wake`, - subscription_id: `runner:runner-1`, - stream: `chat/one/main`, - generation: 7, - ts: 123, - } + const event = wakeEvent(`one`) const fetchMock = vi.fn(async (_input: RequestInfo | URL) => Response.json( { @@ -117,8 +160,7 @@ describe(`createPullWakeRunner`, () => { ) ) vi.stubGlobal(`fetch`, fetchMock) - const dispatchWake = vi.fn() - const drainWakes = vi.fn(async () => undefined) + const testRuntime = runtime() const onError = vi.fn() const streamFactory = vi.fn(async () => ({ offset: `42`, @@ -131,11 +173,7 @@ describe(`createPullWakeRunner`, () => { const runner = createPullWakeRunner({ baseUrl: `http://server`, runnerId: `runner-1`, - runtime: { - dispatchWake, - drainWakes, - abortWakes: vi.fn(), - }, + runtime: testRuntime, headers: { 'x-test-runner': `runner-1` }, heartbeatIntervalMs: 0, streamFactory, @@ -143,42 +181,25 @@ describe(`createPullWakeRunner`, () => { }) runner.start() - await runner.waitForStopped() + await waitFor(() => { + expect(fetchMock).toHaveBeenCalledTimes(1) + }) - expect(fetchMock).toHaveBeenCalled() - expect(dispatchWake).not.toHaveBeenCalled() - expect(drainWakes).not.toHaveBeenCalled() + expect(testRuntime.dispatchWake).not.toHaveBeenCalled() + expect(testRuntime.drainWakes).not.toHaveBeenCalled() expect(onError).not.toHaveBeenCalled() expect(runner.offset).toBe(`42`) + + await runner.stop() }) it(`exposes diagnostics via getHealth()`, async () => { - const event: PullWakeEvent = { - type: `wake`, - subscription_id: `runner:runner-1`, - stream: `chat/one/main`, - generation: 7, - ts: 123, - } - const notification: WakeNotification = { - consumerId: `wake-1`, - epoch: 7, - wakeId: `wake-1`, - streamPath: `/chat/one/main`, - streams: [{ path: `/chat/one/main`, offset: `12` }], - callback: `http://server/_electric/callback-forward/wake-1`, - claimToken: `claim-token`, - entity: { - type: `chat`, - status: `idle`, - url: `/chat/one`, - streams: { main: `/chat/one/main`, error: `/chat/one/error` }, - }, - } + const event = wakeEvent(`one`) const fetchMock = vi.fn(async (_input: RequestInfo | URL) => - Response.json(notification) + Response.json(notification(`one`)) ) vi.stubGlobal(`fetch`, fetchMock) + const testRuntime = runtime() const streamFactory = vi.fn(async () => ({ offset: `42`, async *jsonStream() { @@ -190,11 +211,7 @@ describe(`createPullWakeRunner`, () => { const runner = createPullWakeRunner({ baseUrl: `http://server`, runnerId: `runner-1`, - runtime: { - dispatchWake: vi.fn(), - drainWakes: vi.fn(async () => undefined), - abortWakes: vi.fn(), - }, + runtime: testRuntime, heartbeatIntervalMs: 0, streamFactory, }) @@ -205,32 +222,34 @@ describe(`createPullWakeRunner`, () => { expect(healthBefore.events_received).toBe(0) runner.start() - await runner.waitForStopped() - - const healthAfter = runner.getHealth() - expect(healthAfter.running).toBe(false) - expect(healthAfter.started_at).not.toBeNull() - expect(healthAfter.events_received).toBe(1) - expect(healthAfter.claims_succeeded).toBe(1) - expect(healthAfter.last_claim_result).toBe(`claimed`) - expect(healthAfter.last_dispatch_at).not.toBeNull() - expect(healthAfter.offset).toBe(`42`) + await waitFor(() => { + expect(testRuntime.dispatchWake).toHaveBeenCalledTimes(1) + }) + + const healthDuring = runner.getHealth() + expect(healthDuring.running).toBe(true) + expect(healthDuring.started_at).not.toBeNull() + expect(healthDuring.events_received).toBe(1) + expect(healthDuring.claims_succeeded).toBe(1) + expect(healthDuring.last_claim_result).toBe(`claimed`) + expect(healthDuring.last_dispatch_at).not.toBeNull() + expect(healthDuring.offset).toBe(`42`) + + await runner.stop() + expect(runner.getHealth().running).toBe(false) }) it(`preserves base URL query parameters on stream, claim, and heartbeat requests`, async () => { - const fetchMock = vi.fn(async (_input: RequestInfo | URL) => { + const fetchMock = vi.fn(async (input: RequestInfo | URL) => { + const url = String(input) + if (url.includes(`/heartbeat`)) return Response.json({}) return new Response(null, { status: 204 }) }) vi.stubGlobal(`fetch`, fetchMock) const streamFactory = vi.fn(async () => ({ offset: `42`, async *jsonStream() { - yield { - type: `wake`, - subscription_id: `runner:runner-1`, - stream: `chat/one/main`, - generation: 7, - } satisfies PullWakeEvent + yield wakeEvent(`one`) }, closed: Promise.resolve(), })) @@ -238,31 +257,78 @@ describe(`createPullWakeRunner`, () => { const runner = createPullWakeRunner({ baseUrl: `http://server/root?secret=s1`, runnerId: `runner-1`, - runtime: { - dispatchWake: vi.fn(), - drainWakes: vi.fn(), - abortWakes: vi.fn(), - }, - heartbeatIntervalMs: 1, + runtime: runtime(), + heartbeatIntervalMs: 5, streamFactory, }) runner.start() - await runner.waitForStopped() + await waitFor(() => { + expect(streamFactory).toHaveBeenCalledWith( + expect.objectContaining({ + url: `http://server/root/runners/runner-1/wake?secret=s1`, + }) + ) + expect(fetchMock).toHaveBeenCalledWith( + `http://server/root/_electric/runners/runner-1/heartbeat?secret=s1`, + expect.any(Object) + ) + expect(fetchMock).toHaveBeenCalledWith( + `http://server/root/_electric/runners/runner-1/claim?secret=s1`, + expect.objectContaining({ method: `POST` }) + ) + }) - expect(streamFactory).toHaveBeenCalledWith( - expect.objectContaining({ - url: `http://server/root/runners/runner-1/wake?secret=s1`, - }) - ) - expect(fetchMock).toHaveBeenCalledWith( - `http://server/root/_electric/runners/runner-1/heartbeat?secret=s1`, - expect.any(Object) - ) - expect(fetchMock).toHaveBeenCalledWith( - `http://server/root/_electric/runners/runner-1/claim?secret=s1`, - expect.objectContaining({ method: `POST` }) + await runner.stop() + }) + + it(`sends a throttled heartbeat when runner diagnostics change`, async () => { + const heartbeatBodies: Array> = [] + const fetchMock = vi.fn( + async (input: RequestInfo | URL, init?: RequestInit) => { + if (String(input).includes(`/heartbeat`)) { + heartbeatBodies.push(JSON.parse(String(init?.body))) + return Response.json({}) + } + return Response.json(notification(`one`)) + } ) + vi.stubGlobal(`fetch`, fetchMock) + const testRuntime = runtime() + const streamFactory = vi.fn(async () => ({ + offset: `42`, + async *jsonStream() { + yield wakeEvent(`one`) + }, + closed: Promise.resolve(), + })) + + const runner = createPullWakeRunner({ + baseUrl: `http://server`, + runnerId: `runner-1`, + runtime: testRuntime, + heartbeatIntervalMs: 60_000, + eventHeartbeatThrottleMs: 20, + streamFactory, + }) + + runner.start() + await waitFor(() => { + expect(testRuntime.dispatchWake).toHaveBeenCalledTimes(1) + }) + await waitFor(() => { + expect(heartbeatBodies.length).toBe(2) + }) + + const diagnostics = heartbeatBodies[1]!.diagnostics as Record< + string, + unknown + > + expect(diagnostics.events_received).toBe(1) + expect(diagnostics.claims_succeeded).toBe(1) + expect(heartbeatBodies[1]!.wake_stream_offset).toBe(`42`) + + await runner.stop() }) it(`resolves async headers before opening the durable stream`, async () => { @@ -277,11 +343,7 @@ describe(`createPullWakeRunner`, () => { const runner = createPullWakeRunner({ baseUrl: `http://server`, runnerId: `runner-1`, - runtime: { - dispatchWake: vi.fn(), - drainWakes: vi.fn(), - abortWakes: vi.fn(), - }, + runtime: runtime(), headers: async () => ({ Authorization: `Bearer tenant-token`, 'X-Tenant': `tenant-a`, @@ -290,7 +352,10 @@ describe(`createPullWakeRunner`, () => { }) runner.start() - await runner.waitForStopped() + await waitFor(() => { + expect(durableStreamMocks.DurableStream).toHaveBeenCalledTimes(1) + }) + await runner.stop() expect(durableStreamMocks.DurableStream).toHaveBeenCalledWith( expect.objectContaining({ @@ -309,4 +374,414 @@ describe(`createPullWakeRunner`, () => { ) expect(fetchMock).not.toHaveBeenCalled() }) + + it(`continues reading and claiming while runtime wakes are pending`, async () => { + const events = [wakeEvent(`one`), wakeEvent(`two`)] + const fetchMock = vi.fn( + async (_input: RequestInfo | URL, init?: RequestInit) => + Response.json( + notification(JSON.parse(String(init?.body)).stream.split(`/`)[1]) + ) + ) + vi.stubGlobal(`fetch`, fetchMock) + const testRuntime = runtime() + const streamFactory = vi.fn(async () => ({ + offset: `84`, + async *jsonStream() { + yield events[0]! + yield events[1]! + }, + closed: Promise.resolve(), + })) + + const runner = createPullWakeRunner({ + baseUrl: `http://server`, + runnerId: `runner-1`, + runtime: testRuntime, + heartbeatIntervalMs: 0, + streamFactory, + }) + + runner.start() + await waitFor(() => { + expect(testRuntime.dispatchWake).toHaveBeenCalledTimes(2) + }) + + expect(fetchMock).toHaveBeenCalledTimes(2) + expect(testRuntime.drainWakes).not.toHaveBeenCalled() + await runner.stop() + }) + + it(`pauses claim spawning at maxConcurrentClaims without unbounded queuing`, async () => { + const firstClaim = deferred() + const fetchMock = vi + .fn() + .mockImplementationOnce(async () => firstClaim.promise) + .mockImplementationOnce(async () => Response.json(notification(`two`))) + vi.stubGlobal(`fetch`, fetchMock) + const testRuntime = runtime() + const streamFactory = vi.fn(async () => ({ + offset: `84`, + async *jsonStream() { + yield wakeEvent(`one`) + yield wakeEvent(`two`) + }, + closed: Promise.resolve(), + })) + + const runner = createPullWakeRunner({ + baseUrl: `http://server`, + runnerId: `runner-1`, + runtime: testRuntime, + heartbeatIntervalMs: 0, + maxConcurrentClaims: 1, + streamFactory, + }) + + runner.start() + await waitFor(() => { + expect(fetchMock).toHaveBeenCalledTimes(1) + }) + await new Promise((resolve) => setTimeout(resolve, 20)) + expect(fetchMock).toHaveBeenCalledTimes(1) + + firstClaim.resolve(Response.json(notification(`one`))) + await waitFor(() => { + expect(fetchMock).toHaveBeenCalledTimes(2) + expect(testRuntime.dispatchWake).toHaveBeenCalledTimes(2) + }) + + await runner.stop() + }) + + it(`skips dispatch from a claim actor after shutdown begins`, async () => { + const claimResponse = deferred() + const fetchMock = vi.fn(async () => claimResponse.promise) + vi.stubGlobal(`fetch`, fetchMock) + const calls: Array = [] + const testRuntime = { + dispatchWake: vi.fn(() => calls.push(`dispatch`)), + abortWakes: vi.fn(() => calls.push(`abort`)), + drainWakes: vi.fn(async () => { + calls.push(`drain`) + }), + } + const streamFactory = vi.fn(async () => ({ + offset: `42`, + async *jsonStream() { + yield wakeEvent(`one`) + }, + closed: Promise.resolve(), + })) + + const runner = createPullWakeRunner({ + baseUrl: `http://server`, + runnerId: `runner-1`, + runtime: testRuntime, + heartbeatIntervalMs: 0, + streamFactory, + }) + + runner.start() + await waitFor(() => { + expect(fetchMock).toHaveBeenCalledTimes(1) + }) + const stopped = runner.stop() + claimResponse.resolve(Response.json(notification(`one`))) + await stopped + + expect(testRuntime.dispatchWake).not.toHaveBeenCalled() + expect(calls).toEqual([`abort`, `drain`]) + expect(runner.getHealth().claims_succeeded).toBe(1) + expect(runner.getHealth().claims_skipped).toBe(0) + }) + + it(`keeps heartbeating degraded diagnostics while reconnecting`, async () => { + const heartbeatBodies: Array> = [] + const fetchMock = vi.fn( + async (input: RequestInfo | URL, init?: RequestInit) => { + if (String(input).includes(`/heartbeat`)) { + heartbeatBodies.push(JSON.parse(String(init?.body))) + return Response.json({}) + } + return new Response(null, { status: 204 }) + } + ) + vi.stubGlobal(`fetch`, fetchMock) + const onError = vi.fn() + const streamFactory = vi.fn(async () => { + throw new Error(`stream failed`) + }) + + const runner = createPullWakeRunner({ + baseUrl: `http://server`, + runnerId: `runner-1`, + runtime: runtime(), + heartbeatIntervalMs: 5, + streamFactory, + onError, + }) + + runner.start() + await waitFor(() => { + expect(onError).toHaveBeenCalledWith(expect.any(Error)) + expect( + heartbeatBodies.some((body) => { + const diagnostics = body.diagnostics as + | Record + | undefined + return ( + diagnostics?.stream_connected === false && + diagnostics?.reconnect_count === 1 + ) + }) + ).toBe(true) + }) + + await runner.stop() + }) + + it(`marks heartbeat unhealthy before reporting heartbeat errors`, async () => { + const observedHeartbeatOk: Array = [] + let runner: ReturnType + const fetchMock = vi.fn(async (input: RequestInfo | URL) => { + if (!String(input).includes(`/heartbeat`)) { + return new Response(null, { status: 204 }) + } + return fetchMock.mock.calls.length === 1 + ? Response.json({}) + : new Response(`heartbeat failed`, { status: 500 }) + }) + vi.stubGlobal(`fetch`, fetchMock) + const streamFactory = vi.fn(async () => ({ + async *jsonStream() {}, + closed: Promise.resolve(), + })) + + runner = createPullWakeRunner({ + baseUrl: `http://server`, + runnerId: `runner-1`, + runtime: runtime(), + heartbeatIntervalMs: 5, + streamFactory, + onError: () => { + observedHeartbeatOk.push(runner.getHealth().last_heartbeat_ok) + }, + }) + + runner.start() + await waitFor(() => { + expect(observedHeartbeatOk).toContain(false) + }) + + await runner.stop() + }) + + it(`keeps onError reporting-only when the reporter throws`, async () => { + durableStreamMocks.stream.mockImplementationOnce( + async (opts: { onError: (error: Error) => unknown }) => { + opts.onError(new Error(`durable stream failed`)) + return { + async *jsonStream() {}, + closed: Promise.resolve(), + } + } + ) + const fetchMock = vi.fn(async (input: RequestInfo | URL) => { + if (String(input).includes(`/heartbeat`)) { + return new Response(`heartbeat failed`, { status: 500 }) + } + return new Response(null, { status: 204 }) + }) + vi.stubGlobal(`fetch`, fetchMock) + const onError = vi.fn(() => { + throw new Error(`reporter failed`) + }) + const consoleError = vi + .spyOn(console, `error`) + .mockImplementation(() => undefined) + + const runner = createPullWakeRunner({ + baseUrl: `http://server`, + runnerId: `runner-1`, + runtime: runtime(), + heartbeatIntervalMs: 5, + onError, + }) + + runner.start() + await waitFor(() => { + expect(onError).toHaveBeenCalledWith(expect.any(Error)) + expect(runner.running).toBe(true) + }) + + expect(runner.getHealth().last_error).toMatch(/failed/) + expect(consoleError).toHaveBeenCalledWith( + `Pull-wake runner onError callback failed`, + expect.any(Error) + ) + await expect(runner.stop()).resolves.toBeUndefined() + consoleError.mockRestore() + }) + + it(`does not let a stuck claim actor block stop or later claim capacity`, async () => { + vi.useFakeTimers() + const claimStarted = deferred() + const secondClaimStarted = deferred() + const fetchMock = vi + .fn() + .mockImplementationOnce(async () => { + claimStarted.resolve() + return new Promise(() => {}) + }) + .mockImplementationOnce(async () => { + secondClaimStarted.resolve() + return new Promise(() => {}) + }) + vi.stubGlobal(`fetch`, fetchMock) + const calls: Array = [] + const testRuntime = { + dispatchWake: vi.fn(() => calls.push(`dispatch`)), + abortWakes: vi.fn(() => calls.push(`abort`)), + drainWakes: vi.fn(async () => { + calls.push(`drain`) + }), + } + const streamFactory = vi.fn(async () => ({ + offset: `42`, + async *jsonStream() { + yield wakeEvent(`one`) + }, + closed: Promise.resolve(), + })) + + const runner = createPullWakeRunner({ + baseUrl: `http://server`, + runnerId: `runner-1`, + runtime: testRuntime, + heartbeatIntervalMs: 0, + maxConcurrentClaims: 1, + streamFactory, + }) + + runner.start() + await claimStarted.promise + const stopped = runner.stop() + await vi.advanceTimersByTimeAsync(1_000) + await stopped + + expect(testRuntime.dispatchWake).not.toHaveBeenCalled() + expect(calls).toEqual([`abort`, `drain`]) + + runner.start() + await secondClaimStarted.promise + const secondStop = runner.stop() + await vi.advanceTimersByTimeAsync(1_000) + await secondStop + expect(fetchMock).toHaveBeenCalledTimes(2) + }) + + it(`records a skipped claim when stop aborts a capacity wait`, async () => { + vi.useFakeTimers() + const claimStarted = deferred() + const secondEventYielded = deferred() + const fetchMock = vi.fn(async () => { + claimStarted.resolve() + return new Promise(() => {}) + }) + vi.stubGlobal(`fetch`, fetchMock) + const streamFactory = vi.fn(async () => ({ + offset: `84`, + async *jsonStream() { + yield wakeEvent(`one`) + secondEventYielded.resolve() + yield wakeEvent(`two`) + }, + closed: Promise.resolve(), + })) + const runner = createPullWakeRunner({ + baseUrl: `http://server`, + runnerId: `runner-1`, + runtime: runtime(), + heartbeatIntervalMs: 0, + maxConcurrentClaims: 1, + streamFactory, + }) + + runner.start() + await claimStarted.promise + await secondEventYielded.promise + const stopped = runner.stop() + await vi.advanceTimersByTimeAsync(1_000) + await stopped + + expect(runner.getHealth().events_received).toBe(2) + expect(runner.getHealth().claims_skipped).toBe(1) + }) + + it(`throws drain errors after recording them and marking the runner stopped`, async () => { + const drainError = new Error(`drain failed`) + const onError = vi.fn() + const testRuntime = { + dispatchWake: vi.fn(), + abortWakes: vi.fn(), + drainWakes: vi.fn(async () => { + throw drainError + }), + } + const streamFactory = vi.fn(async () => ({ + async *jsonStream() {}, + closed: Promise.resolve(), + })) + const runner = createPullWakeRunner({ + baseUrl: `http://server`, + runnerId: `runner-1`, + runtime: testRuntime, + heartbeatIntervalMs: 0, + streamFactory, + onError, + }) + + runner.start() + await waitFor(() => { + expect(streamFactory).toHaveBeenCalledTimes(1) + }) + await expect(runner.stop()).rejects.toThrow(`drain failed`) + + expect(runner.running).toBe(false) + expect(onError).toHaveBeenCalledWith(drainError) + expect(runner.getHealth().last_error).toBe(`drain failed`) + }) + + it(`uses exponential reconnect backoff between failed connection attempts`, async () => { + vi.useFakeTimers() + const attempts = [deferred(), deferred(), deferred()] + const streamFactory = vi.fn(async () => { + attempts[streamFactory.mock.calls.length - 1]?.resolve() + throw new Error(`stream failed`) + }) + const runner = createPullWakeRunner({ + baseUrl: `http://server`, + runnerId: `runner-1`, + runtime: runtime(), + heartbeatIntervalMs: 0, + streamFactory, + }) + + runner.start() + await attempts[0]!.promise + await vi.advanceTimersByTimeAsync(999) + expect(streamFactory).toHaveBeenCalledTimes(1) + await vi.advanceTimersByTimeAsync(1) + await attempts[1]!.promise + expect(streamFactory).toHaveBeenCalledTimes(2) + + await vi.advanceTimersByTimeAsync(1_999) + expect(streamFactory).toHaveBeenCalledTimes(2) + await vi.advanceTimersByTimeAsync(1) + await attempts[2]!.promise + expect(streamFactory).toHaveBeenCalledTimes(3) + + await runner.stop() + }) }) diff --git a/packages/agents-server/src/routing/hooks.ts b/packages/agents-server/src/routing/hooks.ts index 9c6e22dcb0..fa25c5ec18 100644 --- a/packages/agents-server/src/routing/hooks.ts +++ b/packages/agents-server/src/routing/hooks.ts @@ -1,6 +1,7 @@ import { SpanKind, SpanStatusCode } from '@opentelemetry/api' import { apiError } from '../electric-agents-http.js' import { ElectricAgentsError } from '../entity-manager.js' +import { ELECTRIC_PRINCIPAL_HEADER } from '../principal.js' import { ATTR, extractTraceContext, tracer } from '../tracing.js' import { serverLog } from '../utils/log.js' import type { Span } from '@opentelemetry/api' @@ -80,7 +81,13 @@ export function applyCors( ) headers.set( `access-control-allow-headers`, - `content-type, authorization, electric-claim-token, ngrok-skip-browser-warning` + [ + `content-type`, + `authorization`, + `electric-claim-token`, + ELECTRIC_PRINCIPAL_HEADER, + `ngrok-skip-browser-warning`, + ].join(`, `) ) headers.set(`access-control-expose-headers`, `*`) return new Response(response.body, { diff --git a/packages/agents-server/test/routing-hooks.test.ts b/packages/agents-server/test/routing-hooks.test.ts index 1eda54afdf..90cb4aac29 100644 --- a/packages/agents-server/test/routing-hooks.test.ts +++ b/packages/agents-server/test/routing-hooks.test.ts @@ -41,6 +41,9 @@ describe(`routing/hooks`, () => { expect(wrapped?.headers.get(`access-control-allow-methods`)).toContain( `GET` ) + expect(wrapped?.headers.get(`access-control-allow-headers`)).toContain( + `electric-principal` + ) }) it(`errorMapper converts ElectricAgentsError to API error JSON`, async () => { diff --git a/packages/agents/src/server.ts b/packages/agents/src/server.ts index db59faaf3f..71b3aaaf77 100644 --- a/packages/agents/src/server.ts +++ b/packages/agents/src/server.ts @@ -47,6 +47,7 @@ export interface BuiltinAgentsServerOptions { claimHeaders?: PullWakeRunnerConfig[`claimHeaders`] claimTokenHeader?: PullWakeRunnerConfig[`claimTokenHeader`] heartbeatIntervalMs?: PullWakeRunnerConfig[`heartbeatIntervalMs`] + eventHeartbeatThrottleMs?: PullWakeRunnerConfig[`eventHeartbeatThrottleMs`] leaseMs?: PullWakeRunnerConfig[`leaseMs`] } /** Invoked when an `authorizationCode` server needs user consent. */ @@ -317,11 +318,11 @@ export class BuiltinAgentsServer { claimHeaders: pullWake.claimHeaders, claimTokenHeader: pullWake.claimTokenHeader, heartbeatIntervalMs: pullWake.heartbeatIntervalMs, + eventHeartbeatThrottleMs: pullWake.eventHeartbeatThrottleMs, leaseMs: pullWake.leaseMs, offset: registeredRunner?.wake_stream_offset, onError: (error) => { serverLog.error(`[builtin-agents] pull-wake runner failed`, error) - return true }, }) this.pullWakeRunner.start() From d6c5d4d9cff04275ef9af623daee567778601e64 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Sun, 17 May 2026 06:52:32 -0600 Subject: [PATCH 14/37] chore: add changeset for pull-wake runner hardening Co-Authored-By: Claude Opus 4.6 --- .changeset/harden-pull-wake-runner.md | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 .changeset/harden-pull-wake-runner.md diff --git a/.changeset/harden-pull-wake-runner.md b/.changeset/harden-pull-wake-runner.md new file mode 100644 index 0000000000..5bdd35029c --- /dev/null +++ b/.changeset/harden-pull-wake-runner.md @@ -0,0 +1,7 @@ +--- +'@electric-ax/agents-runtime': patch +'@electric-ax/agents-server': patch +'@electric-ax/agents': patch +--- + +Harden pull-wake runner lifecycle with a state machine, concurrent claim limits (`maxConcurrentClaims`), and exponential reconnect backoff (1s–30s). Add granular `status` field to `PullWakeRunnerHealth` (`stopped | starting | connecting | streaming | reconnecting | stopping`). The `onError` callback is now reporting-only (`(Error) => void`) — it can no longer control runner lifecycle. `stop()` rethrows `drainWakes` errors so callers observe wake handler failures. Event-driven heartbeat throttling avoids stale diagnostics between fixed-interval heartbeats. From e6254688b19bebdb0f993be9e8b36af0e849480c Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Sun, 17 May 2026 20:47:20 -0600 Subject: [PATCH 15/37] fix(agents): avoid delayed pull-wake session startup --- .../src/components/views/NewSessionView.tsx | 37 +------ .../src/routing/dispatch-policy.ts | 34 ++++++- .../src/routing/entities-router.ts | 4 +- .../test/dispatch-policy-routing.test.ts | 98 ++++++++++++++++--- 4 files changed, 126 insertions(+), 47 deletions(-) diff --git a/packages/agents-server-ui/src/components/views/NewSessionView.tsx b/packages/agents-server-ui/src/components/views/NewSessionView.tsx index 02b2d45998..154671f430 100644 --- a/packages/agents-server-ui/src/components/views/NewSessionView.tsx +++ b/packages/agents-server-ui/src/components/views/NewSessionView.tsx @@ -11,11 +11,8 @@ import { useLiveQuery } from '@tanstack/react-db' import { eq, not } from '@tanstack/db' import { nanoid } from 'nanoid' import { useElectricAgents } from '../../lib/ElectricAgentsProvider' -import { useServerConnection } from '../../hooks/useServerConnection' import { useWorkspace } from '../../hooks/useWorkspace' import { useRecentWorkingDirectories } from '../../hooks/useRecentWorkingDirectories' -import { connectEntityStream } from '../../lib/entity-connection' -import { createSendMessageAction } from '../../lib/sendMessage' import { Icon, Select, Stack, Text } from '../../ui' import { SchemaForm, hasSchemaProperties, isObjectSchema } from '../SchemaForm' import { WorkingDirectoryPicker } from '../WorkingDirectoryPicker' @@ -96,7 +93,6 @@ export function NewSessionView({ setToolbarTitle, }: StandaloneViewProps): React.ReactElement { const { entityTypesCollection, spawnEntity } = useElectricAgents() - const { activeServer } = useServerConnection() const { helpers } = useWorkspace() const [selected, setSelected] = useState(null) const [error, setError] = useState(null) @@ -131,18 +127,10 @@ export function NewSessionView({ [entityTypes] ) - const baseUrl = activeServer?.url ?? null - /** - * Spawn an entity, optionally followed by a `/send` of an initial - * user message. We prefer this two-step over `initialMessage` on - * spawn so the message goes through the same path as the regular - * MessageInput (which is the proven path that wakes horton). - * - * On success we *replace this tile* with the freshly-created entity. - * That keeps the workspace layout intact (other tiles around us - * stay in place) and feels like opening a file in VS Code's - * "untitled" tab — the placeholder turns into the new content. + * Spawn an entity and let the server enqueue any initial user message. + * The server links dispatch before writing that message, avoiding a + * client-side stream preload on the critical path to the first wake. */ const doSpawn = useCallback( async ( @@ -166,6 +154,7 @@ export function NewSessionView({ }, } : {}), + ...(initialUserText ? { initialMessage: initialUserText } : {}), }) const entityUrl = `/${typeName}/${name}` try { @@ -173,29 +162,13 @@ export function NewSessionView({ helpers.openEntity(entityUrl, { target: { tileId, position: `replace` }, }) - if (initialUserText && baseUrl) { - const connection = await connectEntityStream({ baseUrl, entityUrl }) - try { - const sendInitialMessage = createSendMessageAction({ - db: connection.db, - baseUrl, - entityUrl, - }) - await sendInitialMessage({ - text: initialUserText, - mode: `immediate`, - }).isPersisted.promise - } finally { - connection.close() - } - } } catch (err) { setError( `Could not start session: ${err instanceof Error ? err.message : String(err)}.` ) } }, - [helpers, spawnEntity, baseUrl, tileId] + [helpers, spawnEntity, tileId] ) const handleSelectType = useCallback( diff --git a/packages/agents-server/src/routing/dispatch-policy.ts b/packages/agents-server/src/routing/dispatch-policy.ts index 560129044a..2d34b52f27 100644 --- a/packages/agents-server/src/routing/dispatch-policy.ts +++ b/packages/agents-server/src/routing/dispatch-policy.ts @@ -31,7 +31,7 @@ function subscriptionIdForEntityDispatchTarget( entityUrl: string ): string { const base = subscriptionIdForDispatchTarget(target) - if (!target.subscription_id) return base + if (!target.subscription_id && target.type !== `runner`) return base const digest = createHash(`sha256`).update(entityUrl).digest(`hex`) return `${base}:${digest.slice(0, 16)}` } @@ -203,9 +203,21 @@ async function linkStreamToTargetSubscription( wake_stream: wakeStream, description: `Electric Agents runner ${target.runnerId}`, }) + await removeLegacyRunnerSubscriptionStream( + ctx, + target, + subscriptionId, + streamPath + ) return } await ctx.streamClient.addSubscriptionStreams(subscriptionId, [streamPath]) + await removeLegacyRunnerSubscriptionStream( + ctx, + target, + subscriptionId, + streamPath + ) return } @@ -246,3 +258,23 @@ async function linkStreamToTargetSubscription( set: { webhookUrl }, }) } + +async function removeLegacyRunnerSubscriptionStream( + ctx: TenantContext, + target: Extract, + subscriptionId: string, + streamPath: string +): Promise { + const legacySubscriptionId = subscriptionIdForDispatchTarget(target) + if (legacySubscriptionId === subscriptionId) return + + await ctx.streamClient + .removeSubscriptionStream(legacySubscriptionId, streamPath) + .catch((err) => { + serverLog.warn( + `[dispatch-policy] failed to remove legacy runner stream from subscription`, + { subscriptionId: legacySubscriptionId, stream: streamPath }, + err + ) + }) +} diff --git a/packages/agents-server/src/routing/entities-router.ts b/packages/agents-server/src/routing/entities-router.ts index b7ec945fa7..5de7793c33 100644 --- a/packages/agents-server/src/routing/entities-router.ts +++ b/packages/agents-server/src/routing/entities-router.ts @@ -533,6 +533,8 @@ async function sendEntity( if (!entity.dispatch_policy) { const updatedEntity = await backfillEntityDispatchPolicy(ctx, entity) await linkEntityDispatchSubscription(ctx, updatedEntity) + } else if (entity.dispatch_policy.targets[0]?.type === `runner`) { + await linkEntityDispatchSubscription(ctx, entity) } if (parsed.afterMs && parsed.afterMs > 0) { @@ -614,13 +616,13 @@ async function spawnEntity( wake: parsed.wake, created_by: principal.url, }) + await linkEntityDispatchSubscription(ctx, entity) if (parsed.initialMessage !== undefined) { await ctx.entityManager.send(entity.url, { from: principal.url, payload: parsed.initialMessage, }) } - await linkEntityDispatchSubscription(ctx, entity) return json( { ...toPublicEntity(entity), txid: entity.txid }, diff --git a/packages/agents-server/test/dispatch-policy-routing.test.ts b/packages/agents-server/test/dispatch-policy-routing.test.ts index 2ee56b96cf..e8e4c7d394 100644 --- a/packages/agents-server/test/dispatch-policy-routing.test.ts +++ b/packages/agents-server/test/dispatch-policy-routing.test.ts @@ -15,14 +15,15 @@ function request(method: string, path: string, body?: unknown): Request { }) } -function entity(dispatchPolicy?: DispatchPolicy): ElectricAgentsEntity & { - txid: number -} { +function entity( + dispatchPolicy?: DispatchPolicy, + id = `one` +): ElectricAgentsEntity & { txid: number } { return { - url: `/chat/one`, + url: `/chat/${id}`, type: `chat`, status: `idle`, - streams: { main: `/chat/one/main`, error: `/chat/one/error` }, + streams: { main: `/chat/${id}/main`, error: `/chat/${id}/error` }, subscription_id: `chat-handler`, dispatch_policy: dispatchPolicy, write_token: `write-token`, @@ -78,12 +79,15 @@ function buildContext(overrides: Partial = {}): TenantContext { })), }, ensurePrincipal: vi.fn(async () => undefined), - spawn: vi.fn(async (_type, req) => entity(req.dispatch_policy)), + spawn: vi.fn(async (_type, req) => + entity(req.dispatch_policy, req.instance_id ?? `one`) + ), } as any, streamClient: { getSubscription: vi.fn(async () => null), putSubscription: vi.fn(async () => ({})), addSubscriptionStreams: vi.fn(async () => ({})), + removeSubscriptionStream: vi.fn(async () => ({})), ensure: vi.fn(async () => undefined), } as any, runtime: undefined as any, @@ -113,13 +117,47 @@ describe(`dispatch policy routing`, () => { { contentType: `application/json` } ) expect(ctx.streamClient.putSubscription).toHaveBeenCalledWith( - `runner:runner-1`, + expect.stringMatching(/^runner:runner-1:/), expect.objectContaining({ type: `pull-wake`, streams: [`/chat/one/main`], wake_stream: `/runners/runner-1/wake`, }) ) + expect(ctx.streamClient.removeSubscriptionStream).toHaveBeenCalledWith( + `runner:runner-1`, + `/chat/one/main` + ) + }) + + it(`uses separate pull-wake subscriptions for separate runner-targeted entities`, async () => { + const dispatchPolicy: DispatchPolicy = { + targets: [{ type: `runner`, runnerId: `runner-1` }], + } + const ctx = buildContext() + + const first = await globalRouter.fetch( + request(`PUT`, `/_electric/entities/chat/one`, { + dispatch_policy: dispatchPolicy, + }), + ctx + ) + const second = await globalRouter.fetch( + request(`PUT`, `/_electric/entities/chat/two`, { + dispatch_policy: dispatchPolicy, + }), + ctx + ) + + expect(first.status).toBe(201) + expect(second.status).toBe(201) + const subscriptionIds = vi + .mocked(ctx.streamClient.putSubscription) + .mock.calls.map(([subscriptionId]) => subscriptionId) + expect(subscriptionIds).toHaveLength(2) + expect(subscriptionIds[0]).toMatch(/^runner:runner-1:/) + expect(subscriptionIds[1]).toMatch(/^runner:runner-1:/) + expect(subscriptionIds[0]).not.toBe(subscriptionIds[1]) }) it(`creates webhook subscriptions and stores the original target`, async () => { @@ -151,7 +189,7 @@ describe(`dispatch policy routing`, () => { expect(ctx.pgDb.insert).toHaveBeenCalled() }) - it(`sends spawn initialMessage before linking pull-wake dispatch`, async () => { + it(`links pull-wake dispatch before sending spawn initialMessage`, async () => { const dispatchPolicy: DispatchPolicy = { targets: [{ type: `runner`, runnerId: `runner-1` }], } @@ -179,7 +217,7 @@ describe(`dispatch policy routing`, () => { payload: `hello`, }) expect(ctx.streamClient.putSubscription).toHaveBeenCalledWith( - `runner:runner-1`, + expect.stringMatching(/^runner:runner-1:/), expect.objectContaining({ type: `pull-wake`, streams: [`/chat/one/main`], @@ -187,10 +225,8 @@ describe(`dispatch policy routing`, () => { }) ) expect( - (ctx.entityManager.send as any).mock.invocationCallOrder[0] - ).toBeLessThan( (ctx.streamClient.putSubscription as any).mock.invocationCallOrder[0] - ) + ).toBeLessThan((ctx.entityManager.send as any).mock.invocationCallOrder[0]) }) it(`links legacy entities through the type default before sending`, async () => { @@ -220,7 +256,7 @@ describe(`dispatch policy routing`, () => { expect(response.status).toBe(204) expect(ctx.streamClient.putSubscription).toHaveBeenCalledWith( - `runner:runner-1`, + expect.stringMatching(/^runner:runner-1:/), expect.objectContaining({ type: `pull-wake`, streams: [`/chat/one/main`], @@ -235,4 +271,40 @@ describe(`dispatch policy routing`, () => { expect.objectContaining({ payload: `hello` }) ) }) + + it(`relinks existing runner-dispatched entities before sending`, async () => { + const dispatchPolicy: DispatchPolicy = { + targets: [{ type: `runner`, runnerId: `runner-1` }], + } + const ctx = buildContext() + ;(ctx.entityManager.registry.getEntity as any).mockResolvedValue( + entity(dispatchPolicy) + ) + ctx.entityManager.send = vi.fn(async () => undefined) + + const response = await globalRouter.fetch( + request(`POST`, `/_electric/entities/chat/one/send`, { + payload: `hello`, + }), + ctx + ) + + expect(response.status).toBe(204) + expect(ctx.streamClient.putSubscription).toHaveBeenCalledWith( + expect.stringMatching(/^runner:runner-1:/), + expect.objectContaining({ + type: `pull-wake`, + streams: [`/chat/one/main`], + wake_stream: `/runners/runner-1/wake`, + }) + ) + expect(ctx.streamClient.removeSubscriptionStream).toHaveBeenCalledWith( + `runner:runner-1`, + `/chat/one/main` + ) + expect(ctx.entityManager.send).toHaveBeenCalledWith( + `/chat/one`, + expect.objectContaining({ payload: `hello` }) + ) + }) }) From 897b7d445a1d2ce838b1888f98c6b50d0c60393f Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Sun, 17 May 2026 20:49:14 -0600 Subject: [PATCH 16/37] chore: add changeset for pull-wake startup UI --- .changeset/pull-wake-session-startup-ui.md | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 .changeset/pull-wake-session-startup-ui.md diff --git a/.changeset/pull-wake-session-startup-ui.md b/.changeset/pull-wake-session-startup-ui.md new file mode 100644 index 0000000000..b9cafee602 --- /dev/null +++ b/.changeset/pull-wake-session-startup-ui.md @@ -0,0 +1,5 @@ +--- +'@electric-ax/agents-server-ui': patch +--- + +Send new-session initial messages through the spawn request so pull-wake sessions can start without waiting for the UI to preload the entity stream. From 8c841aab11e067fe7f1cc5cbc112445decd2bfd9 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Mon, 18 May 2026 06:20:18 -0600 Subject: [PATCH 17/37] fix(agents): address pull-wake review blockers --- .changeset/harden-pull-wake-runner.md | 2 +- .changeset/pull-wake-health-diagnostics.md | 2 +- .../agents-runtime/src/pull-wake-runner.ts | 23 +++ .../test/pull-wake-runner.test.ts | 61 ++++++++ .../src/components/UserMessage.tsx | 12 -- .../settings/pages/LocalRuntimePage.tsx | 50 +++++-- .../src/lib/ElectricAgentsProvider.tsx | 35 +++++ .../0008_runner_runtime_diagnostics.sql | 50 +++++++ .../agents-server/drizzle/meta/_journal.json | 7 + packages/agents-server/src/db/schema.ts | 39 +++-- packages/agents-server/src/entity-registry.ts | 101 ++++++++++--- packages/agents-server/src/principal.ts | 7 +- .../src/routing/dispatch-policy.ts | 9 +- .../src/routing/runners-router.ts | 61 +++++--- .../agents-server/src/utils/server-utils.ts | 10 +- packages/agents-server/test/principal.test.ts | 9 ++ .../agents-server/test/runners-router.test.ts | 140 +++++++++++++++--- .../agents-server/test/server-utils.test.ts | 17 ++- 18 files changed, 536 insertions(+), 99 deletions(-) create mode 100644 packages/agents-server/drizzle/0008_runner_runtime_diagnostics.sql diff --git a/.changeset/harden-pull-wake-runner.md b/.changeset/harden-pull-wake-runner.md index 5bdd35029c..addda03d10 100644 --- a/.changeset/harden-pull-wake-runner.md +++ b/.changeset/harden-pull-wake-runner.md @@ -4,4 +4,4 @@ '@electric-ax/agents': patch --- -Harden pull-wake runner lifecycle with a state machine, concurrent claim limits (`maxConcurrentClaims`), and exponential reconnect backoff (1s–30s). Add granular `status` field to `PullWakeRunnerHealth` (`stopped | starting | connecting | streaming | reconnecting | stopping`). The `onError` callback is now reporting-only (`(Error) => void`) — it can no longer control runner lifecycle. `stop()` rethrows `drainWakes` errors so callers observe wake handler failures. Event-driven heartbeat throttling avoids stale diagnostics between fixed-interval heartbeats. +Harden pull-wake runner lifecycle with a state machine, concurrent claim limits (`maxConcurrentClaims`), heartbeat-driven stream resets, and exponential reconnect backoff (1s–30s). Add granular `status` field to `PullWakeRunnerHealth` (`stopped | starting | connecting | streaming | reconnecting | stopping`). The `onError` callback is now reporting-only (`(Error) => void`) — it can no longer control runner lifecycle. `stop()` rethrows `drainWakes` errors so callers observe wake handler failures. Event-driven heartbeat throttling avoids stale diagnostics between fixed-interval heartbeats. diff --git a/.changeset/pull-wake-health-diagnostics.md b/.changeset/pull-wake-health-diagnostics.md index bc28013e91..b54dbe5f5b 100644 --- a/.changeset/pull-wake-health-diagnostics.md +++ b/.changeset/pull-wake-health-diagnostics.md @@ -6,4 +6,4 @@ 'electric-ax': patch --- -Add pull-wake runner health check endpoint and rename `owner_user_id` to `owner_principal` across the runners system. The `GET /_electric/runners/:id/health` endpoint returns comprehensive diagnostics including runner state, client-reported stream/heartbeat/claim metrics, active claims, and dispatch stats with a derived health status (healthy/degraded/unhealthy). The `PullWakeRunner` now tracks internal diagnostics and reports them to the server via heartbeats, stored in a new `diagnostics` JSONB column on the runners table. The `owner_user_id` → `owner_principal` rename stores canonical principal URLs instead of keys, with strict validation via `principalKeyFromUrl()`. This is a breaking change with no backward compatibility — all callers must send principal URLs. +Add pull-wake runner health check endpoint and rename `owner_user_id` to `owner_principal` across the runners system. The `GET /_electric/runners/:id/health` endpoint returns comprehensive diagnostics including runner state, client-reported stream/heartbeat/claim metrics, active claims, and dispatch stats with a derived health status (healthy/degraded/unhealthy). The `PullWakeRunner` now tracks internal diagnostics and reports them to the server via heartbeats, stored in a separate `runner_runtime_diagnostics` table so the main `runners` shape stays stable for normal UI sync. The `owner_user_id` → `owner_principal` rename stores canonical principal URLs instead of keys, with strict validation and canonicalization at route boundaries. This is a breaking change with no backward compatibility — all callers must send principal URLs. diff --git a/packages/agents-runtime/src/pull-wake-runner.ts b/packages/agents-runtime/src/pull-wake-runner.ts index d9615550c9..b8f348fd55 100644 --- a/packages/agents-runtime/src/pull-wake-runner.ts +++ b/packages/agents-runtime/src/pull-wake-runner.ts @@ -98,6 +98,7 @@ const INITIAL_RECONNECT_BACKOFF_MS = 1_000 const MAX_RECONNECT_BACKOFF_MS = 30_000 const CLAIM_ACTOR_STOP_GRACE_MS = 1_000 const DEFAULT_EVENT_HEARTBEAT_THROTTLE_MS = 2_000 +const HEARTBEAT_FAILURE_STREAM_RESET_THRESHOLD = 2 export function createPullWakeRunner( config: PullWakeRunnerConfig @@ -124,10 +125,12 @@ export function createPullWakeRunner( let claimsSucceeded = 0 let claimsSkipped = 0 let claimsFailed = 0 + let consecutiveHeartbeatFailures = 0 let acceptingClaims = false let activeClaimCount = 0 let runGeneration = 0 let nextReconnectBackoffMs = INITIAL_RECONNECT_BACKOFF_MS + let streamResetError: Error | null = null const claimActors = new Map, number>() const wakePath = @@ -225,6 +228,12 @@ export function createPullWakeRunner( } } + const requestStreamReconnect = (error: Error): void => { + if (!streamConnected || streamResetError) return + streamResetError = error + response?.cancel?.(error) + } + const notifyHeartbeatChange = (): void => { const signal = controller?.signal if (!signal || signal.aborted || heartbeatIntervalMs <= 0) return @@ -259,10 +268,20 @@ export function createPullWakeRunner( ) } lastHeartbeatOk = true + consecutiveHeartbeatFailures = 0 } catch (err) { if (!signal.aborted) { lastHeartbeatOk = false + consecutiveHeartbeatFailures++ reportError(err) + if ( + consecutiveHeartbeatFailures >= + HEARTBEAT_FAILURE_STREAM_RESET_THRESHOLD + ) { + requestStreamReconnect( + err instanceof Error ? err : new Error(String(err)) + ) + } } } } @@ -481,6 +500,7 @@ export function createPullWakeRunner( signal: AbortSignal, generation: number ): Promise => { + streamResetError = null response = await streamFactory({ url: wakeUrl, headers: await resolveHeaders(), @@ -514,6 +534,9 @@ export function createPullWakeRunner( await response.closed?.catch((err) => { if (!signal.aborted) throw err }) + if (streamResetError && !signal.aborted) { + throw streamResetError + } } finally { streamConnected = false streamConnectedSince = null diff --git a/packages/agents-runtime/test/pull-wake-runner.test.ts b/packages/agents-runtime/test/pull-wake-runner.test.ts index a88d3150d2..57cfe15bd6 100644 --- a/packages/agents-runtime/test/pull-wake-runner.test.ts +++ b/packages/agents-runtime/test/pull-wake-runner.test.ts @@ -541,6 +541,67 @@ describe(`createPullWakeRunner`, () => { await runner.stop() }) + it(`forces the stream to reconnect after repeated heartbeat failures`, async () => { + vi.useFakeTimers() + const firstStreamOpened = deferred() + const secondStreamOpened = deferred() + const firstStreamClosed = deferred() + const secondStreamClosed = deferred() + const firstCancel = vi.fn(() => firstStreamClosed.resolve()) + const streamFactory = vi.fn(async () => { + if (streamFactory.mock.calls.length === 1) { + firstStreamOpened.resolve() + return { + async *jsonStream() { + await firstStreamClosed.promise + }, + cancel: firstCancel, + closed: firstStreamClosed.promise, + } + } + secondStreamOpened.resolve() + return { + async *jsonStream() { + await secondStreamClosed.promise + }, + cancel: () => secondStreamClosed.resolve(), + closed: secondStreamClosed.promise, + } + }) + let heartbeatCalls = 0 + const fetchMock = vi.fn(async () => { + heartbeatCalls++ + if (heartbeatCalls <= 2) { + throw new Error(`connect ECONNREFUSED 127.0.0.1:4437`) + } + return Response.json({}) + }) + vi.stubGlobal(`fetch`, fetchMock) + + const runner = createPullWakeRunner({ + baseUrl: `http://server`, + runnerId: `runner-1`, + runtime: runtime(), + heartbeatIntervalMs: 10, + eventHeartbeatThrottleMs: 0, + streamFactory, + }) + + runner.start() + await firstStreamOpened.promise + await vi.advanceTimersByTimeAsync(20) + + expect(firstCancel).toHaveBeenCalledWith(expect.any(Error)) + + await vi.advanceTimersByTimeAsync(1_000) + await secondStreamOpened.promise + + expect(streamFactory).toHaveBeenCalledTimes(2) + expect(runner.getHealth().reconnect_count).toBe(1) + + await runner.stop() + }) + it(`marks heartbeat unhealthy before reporting heartbeat errors`, async () => { const observedHeartbeatOk: Array = [] let runner: ReturnType diff --git a/packages/agents-server-ui/src/components/UserMessage.tsx b/packages/agents-server-ui/src/components/UserMessage.tsx index c16a322500..3113ddd26c 100644 --- a/packages/agents-server-ui/src/components/UserMessage.tsx +++ b/packages/agents-server-ui/src/components/UserMessage.tsx @@ -9,18 +9,6 @@ type UserMessageSection = Extract< { kind: `user_message` } > -function formatSender(value: string | null | undefined): string { - if (!value) return `user` - if (!value.startsWith(`/principal/`)) return value - const segment = value.slice(`/principal/`.length) - if (!segment || segment.includes(`/`)) return value - try { - return decodeURIComponent(segment) - } catch { - return value - } -} - export const UserMessage = memo(function UserMessage({ section, }: { diff --git a/packages/agents-server-ui/src/components/settings/pages/LocalRuntimePage.tsx b/packages/agents-server-ui/src/components/settings/pages/LocalRuntimePage.tsx index 16edade0b0..5d78849ead 100644 --- a/packages/agents-server-ui/src/components/settings/pages/LocalRuntimePage.tsx +++ b/packages/agents-server-ui/src/components/settings/pages/LocalRuntimePage.tsx @@ -1,4 +1,4 @@ -import { useEffect, useState } from 'react' +import { useEffect, useMemo, useState } from 'react' import { eq, useLiveQuery } from '@tanstack/react-db' import { appendPathToUrl } from '@electric-ax/agents-runtime/client' import { Play, RefreshCw, Square } from 'lucide-react' @@ -8,8 +8,10 @@ import { type DesktopState, } from '../../../lib/server-connection' import { + createRunnerRuntimeDiagnosticsCollection, useElectricAgents, type ElectricRunner, + type ElectricRunnerRuntimeDiagnostics, } from '../../../lib/ElectricAgentsProvider' import { formatRelativeTime } from '../../../lib/formatTime' import { Badge, Button, Icon, Stack, Text } from '../../../ui' @@ -43,6 +45,7 @@ function parseTime(value: string | null | undefined): number | null { function runnerHealth( runner: ElectricRunner | null, + runtimeDiagnostics: ElectricRunnerRuntimeDiagnostics | null, now: number = Date.now() ): { status: keyof typeof RUNNER_HEALTH_TONES; issues: Array } { if (!runner) return { status: `unknown`, issues: [`Runner not synced`] } @@ -58,7 +61,10 @@ function runnerHealth( issues.push(`Disabled`) } - const leaseExpiresAt = parseTime(runner.liveness_lease_expires_at) + const leaseExpiresAt = parseTime( + runtimeDiagnostics?.liveness_lease_expires_at ?? + runner.liveness_lease_expires_at + ) if (leaseExpiresAt === null) { escalate(`degraded`) issues.push(`No heartbeat`) @@ -67,9 +73,9 @@ function runnerHealth( issues.push(`Lease expired`) } - const diagnostics = runner.diagnostics + const diagnostics = runtimeDiagnostics?.diagnostics ?? runner.diagnostics if (!diagnostics) { - if (runner.last_seen_at) { + if (runtimeDiagnostics?.last_seen_at ?? runner.last_seen_at) { escalate(`degraded`) issues.push(`No diagnostics`) } @@ -100,6 +106,8 @@ function countLabel(value: number | undefined): string { return String(value ?? 0) } +type RunnerDiagnostics = NonNullable + function runtimeConnectionLabel(value: string | null | undefined): string { if (!value) return `-` return `Pull-wake` @@ -142,13 +150,29 @@ export function LocalRuntimePage(): React.ReactElement { [runnersCollection, runnerId] ) const runner = runnerRows[0] ?? null - const health = runnerHealth(runner, now) - const healthTone = RUNNER_HEALTH_TONES[health.status] - const diagnostics = runner?.diagnostics ?? null const healthEndpoint = runnerHealthEndpoint( state?.activeServer?.url, runnerId ) + const diagnosticsCollection = useMemo(() => { + if (!state?.activeServer?.url || !runnerId) return null + return createRunnerRuntimeDiagnosticsCollection( + state.activeServer.url, + runnerId + ) + }, [state?.activeServer?.url, runnerId]) + const { data: runtimeDiagnosticsRows = [] } = useLiveQuery( + (query) => { + if (!diagnosticsCollection) return undefined + return query.from({ diagnostics: diagnosticsCollection }) + }, + [diagnosticsCollection] + ) + const runnerTelemetry = runtimeDiagnosticsRows[0] ?? null + const health = runnerHealth(runner, runnerTelemetry, now) + const healthTone = RUNNER_HEALTH_TONES[health.status] + const diagnostics: RunnerDiagnostics | null = + runnerTelemetry?.diagnostics ?? runner?.diagnostics ?? null useEffect(() => { if (!isDesktop) return @@ -156,6 +180,12 @@ export function LocalRuntimePage(): React.ReactElement { return () => window.clearInterval(interval) }, [isDesktop]) + useEffect(() => { + return () => { + diagnosticsCollection?.cleanup() + } + }, [diagnosticsCollection]) + useEffect(() => { if (!isDesktop) return let cancelled = false @@ -257,7 +287,7 @@ export function LocalRuntimePage(): React.ReactElement { /> - {timeLabel(runner?.last_seen_at)} + {timeLabel(runnerTelemetry?.last_seen_at ?? runner?.last_seen_at)} } /> diff --git a/packages/agents-server-ui/src/lib/ElectricAgentsProvider.tsx b/packages/agents-server-ui/src/lib/ElectricAgentsProvider.tsx index 1d41260a32..d2a2bc59f3 100644 --- a/packages/agents-server-ui/src/lib/ElectricAgentsProvider.tsx +++ b/packages/agents-server-ui/src/lib/ElectricAgentsProvider.tsx @@ -80,9 +80,22 @@ const runnerSchema = z.object({ updated_at: z.string(), }) +const runnerRuntimeDiagnosticsSchema = z.object({ + runner_id: z.string(), + owner_principal: z.string(), + wake_stream_offset: z.string().nullable().optional(), + last_seen_at: z.string(), + liveness_lease_expires_at: z.string(), + diagnostics: runnerDiagnosticsSchema.nullable().optional(), + updated_at: z.string(), +}) + export type ElectricEntity = z.infer export type ElectricEntityType = z.infer export type ElectricRunner = z.infer +export type ElectricRunnerRuntimeDiagnostics = z.infer< + typeof runnerRuntimeDiagnosticsSchema +> // --- Collection factories --- @@ -149,6 +162,28 @@ function createRunnersCollection(baseUrl: string) { ) } +export function createRunnerRuntimeDiagnosticsCollection( + baseUrl: string, + runnerId: string +) { + return createCollection( + electricCollectionOptions({ + id: `runner-runtime-diagnostics:${baseUrl}:${runnerId}`, + schema: runnerRuntimeDiagnosticsSchema, + shapeOptions: { + url: appendPathToUrl(baseUrl, `/_electric/electric/v1/shape`), + params: { + table: `runner_runtime_diagnostics`, + where: `runner_id = $1`, + params: { '1': runnerId }, + }, + fetchClient: serverFetch, + }, + getKey: (item) => item.runner_id, + }) + ) +} + type EntitiesCollection = ReturnType type EntityTypesCollection = ReturnType type RunnersCollection = ReturnType diff --git a/packages/agents-server/drizzle/0008_runner_runtime_diagnostics.sql b/packages/agents-server/drizzle/0008_runner_runtime_diagnostics.sql new file mode 100644 index 0000000000..4ab187475b --- /dev/null +++ b/packages/agents-server/drizzle/0008_runner_runtime_diagnostics.sql @@ -0,0 +1,50 @@ +CREATE TABLE runner_runtime_diagnostics ( + tenant_id text NOT NULL DEFAULT 'default', + runner_id text NOT NULL, + owner_principal text NOT NULL, + wake_stream_offset text, + last_seen_at timestamp with time zone NOT NULL, + liveness_lease_expires_at timestamp with time zone NOT NULL, + diagnostics jsonb, + updated_at timestamp with time zone NOT NULL DEFAULT NOW(), + PRIMARY KEY (tenant_id, runner_id) +); +--> statement-breakpoint +CREATE INDEX idx_runner_runtime_diagnostics_owner + ON runner_runtime_diagnostics (tenant_id, owner_principal); +--> statement-breakpoint +CREATE INDEX idx_runner_runtime_diagnostics_liveness + ON runner_runtime_diagnostics (tenant_id, liveness_lease_expires_at); +--> statement-breakpoint +INSERT INTO runner_runtime_diagnostics ( + tenant_id, + runner_id, + owner_principal, + wake_stream_offset, + last_seen_at, + liveness_lease_expires_at, + diagnostics, + updated_at +) +SELECT + tenant_id, + id, + owner_principal, + wake_stream_offset, + COALESCE(last_seen_at, updated_at), + COALESCE(liveness_lease_expires_at, updated_at), + diagnostics, + updated_at +FROM runners +WHERE last_seen_at IS NOT NULL + OR liveness_lease_expires_at IS NOT NULL + OR wake_stream_offset IS NOT NULL + OR diagnostics IS NOT NULL; +--> statement-breakpoint +ALTER TABLE runners DROP COLUMN diagnostics; +--> statement-breakpoint +ALTER TABLE runners DROP COLUMN wake_stream_offset; +--> statement-breakpoint +ALTER TABLE runners DROP COLUMN last_seen_at; +--> statement-breakpoint +ALTER TABLE runners DROP COLUMN liveness_lease_expires_at; diff --git a/packages/agents-server/drizzle/meta/_journal.json b/packages/agents-server/drizzle/meta/_journal.json index cd53b818cc..4574ed99be 100644 --- a/packages/agents-server/drizzle/meta/_journal.json +++ b/packages/agents-server/drizzle/meta/_journal.json @@ -57,6 +57,13 @@ "when": 1778899200000, "tag": "0007_runner_diagnostics_and_principal", "breakpoints": true + }, + { + "idx": 8, + "version": "7", + "when": 1778976000000, + "tag": "0008_runner_runtime_diagnostics", + "breakpoints": true } ] } diff --git a/packages/agents-server/src/db/schema.ts b/packages/agents-server/src/db/schema.ts index d58cbb6aff..15a1d234bc 100644 --- a/packages/agents-server/src/db/schema.ts +++ b/packages/agents-server/src/db/schema.ts @@ -111,12 +111,6 @@ export const runners = pgTable( kind: text(`kind`).notNull().default(`local`), adminStatus: text(`admin_status`).notNull().default(`enabled`), wakeStream: text(`wake_stream`).notNull(), - wakeStreamOffset: text(`wake_stream_offset`), - lastSeenAt: timestamp(`last_seen_at`, { withTimezone: true }), - livenessLeaseExpiresAt: timestamp(`liveness_lease_expires_at`, { - withTimezone: true, - }), - diagnostics: jsonb(`diagnostics`), createdAt: timestamp(`created_at`, { withTimezone: true }) .notNull() .defaultNow(), @@ -132,10 +126,6 @@ export const runners = pgTable( table.ownerPrincipal ), index(`idx_runners_admin_status`).on(table.tenantId, table.adminStatus), - index(`idx_runners_liveness_lease_expires_at`).on( - table.tenantId, - table.livenessLeaseExpiresAt - ), check( `chk_runners_kind`, sql`${table.kind} IN ('local', 'cloud-worker', 'sandbox', 'ci', 'server')` @@ -147,6 +137,35 @@ export const runners = pgTable( ] ) +export const runnerRuntimeDiagnostics = pgTable( + `runner_runtime_diagnostics`, + { + tenantId: text(`tenant_id`).notNull().default(`default`), + runnerId: text(`runner_id`).notNull(), + ownerPrincipal: text(`owner_principal`).notNull(), + wakeStreamOffset: text(`wake_stream_offset`), + lastSeenAt: timestamp(`last_seen_at`, { withTimezone: true }).notNull(), + livenessLeaseExpiresAt: timestamp(`liveness_lease_expires_at`, { + withTimezone: true, + }).notNull(), + diagnostics: jsonb(`diagnostics`), + updatedAt: timestamp(`updated_at`, { withTimezone: true }) + .notNull() + .defaultNow(), + }, + (table) => [ + primaryKey({ columns: [table.tenantId, table.runnerId] }), + index(`idx_runner_runtime_diagnostics_owner`).on( + table.tenantId, + table.ownerPrincipal + ), + index(`idx_runner_runtime_diagnostics_liveness`).on( + table.tenantId, + table.livenessLeaseExpiresAt + ), + ] +) + export const entityDispatchState = pgTable( `entity_dispatch_state`, { diff --git a/packages/agents-server/src/entity-registry.ts b/packages/agents-server/src/entity-registry.ts index 8d7732d72a..a0c5b23426 100644 --- a/packages/agents-server/src/entity-registry.ts +++ b/packages/agents-server/src/entity-registry.ts @@ -7,6 +7,7 @@ import { entityDispatchState, entityManifestSources, entityTypes, + runnerRuntimeDiagnostics, runners, tagStreamOutbox, } from './db/schema.js' @@ -82,6 +83,7 @@ export interface RegisterRunnerInput { export interface HeartbeatRunnerInput { runnerId: string + ownerPrincipal: string heartbeatAt?: Date livenessLeaseExpiresAt?: Date leaseMs?: number @@ -89,6 +91,16 @@ export interface HeartbeatRunnerInput { diagnostics?: Record } +export interface RunnerRuntimeDiagnostics { + runner_id: string + owner_principal: string + wake_stream_offset?: string + last_seen_at: string + liveness_lease_expires_at: string + diagnostics?: Record + updated_at: string +} + export interface MaterializeActiveClaimInput { consumerId: string epoch: number @@ -199,25 +211,66 @@ export class PostgresRegistry { input.livenessLeaseExpiresAt ?? new Date(now.getTime() + (input.leaseMs ?? DEFAULT_RUNNER_LEASE_MS)) - const rows = await this.db - .update(runners) - .set({ + await this.db + .insert(runnerRuntimeDiagnostics) + .values({ + tenantId: this.tenantId, + runnerId: input.runnerId, + ownerPrincipal: input.ownerPrincipal, lastSeenAt: now, livenessLeaseExpiresAt: leaseExpiresAt, - ...(input.wakeStreamOffset !== undefined - ? { wakeStreamOffset: input.wakeStreamOffset } - : {}), - ...(input.diagnostics !== undefined - ? { diagnostics: input.diagnostics } - : {}), + wakeStreamOffset: input.wakeStreamOffset, + diagnostics: input.diagnostics, updatedAt: now, }) + .onConflictDoUpdate({ + target: [ + runnerRuntimeDiagnostics.tenantId, + runnerRuntimeDiagnostics.runnerId, + ], + set: { + lastSeenAt: now, + ownerPrincipal: input.ownerPrincipal, + livenessLeaseExpiresAt: leaseExpiresAt, + ...(input.wakeStreamOffset !== undefined + ? { wakeStreamOffset: input.wakeStreamOffset } + : {}), + ...(input.diagnostics !== undefined + ? { diagnostics: input.diagnostics } + : {}), + updatedAt: now, + }, + }) + + const runner = await this.getRunner(input.runnerId) + if (!runner) return null + return { + ...runner, + last_seen_at: now.toISOString(), + liveness_lease_expires_at: leaseExpiresAt.toISOString(), + ...(input.wakeStreamOffset !== undefined + ? { wake_stream_offset: input.wakeStreamOffset } + : {}), + ...(input.diagnostics !== undefined + ? { diagnostics: input.diagnostics } + : {}), + } + } + + async getRunnerDiagnostics( + runnerId: string + ): Promise { + const rows = await this.db + .select() + .from(runnerRuntimeDiagnostics) .where( - and(eq(runners.tenantId, this.tenantId), eq(runners.id, input.runnerId)) + and( + eq(runnerRuntimeDiagnostics.tenantId, this.tenantId), + eq(runnerRuntimeDiagnostics.runnerId, runnerId) + ) ) - .returning() - - return rows[0] ? this.rowToRunner(rows[0]) : null + .limit(1) + return rows[0] ? this.rowToRunnerRuntimeDiagnostics(rows[0]) : null } async setRunnerAdminStatus( @@ -1198,24 +1251,28 @@ export class PostgresRegistry { } private rowToRunner(row: typeof runners.$inferSelect): ElectricAgentsRunner { - const now = Date.now() - const livenessExpiry = row.livenessLeaseExpiresAt?.getTime() return { id: row.id, owner_principal: row.ownerPrincipal, label: row.label, kind: assertRunnerKind(row.kind), admin_status: assertRunnerAdminStatus(row.adminStatus), - liveness: - livenessExpiry !== undefined && livenessExpiry > now - ? `online` - : `offline`, - last_seen_at: row.lastSeenAt?.toISOString(), - liveness_lease_expires_at: row.livenessLeaseExpiresAt?.toISOString(), wake_stream: row.wakeStream, + created_at: row.createdAt.toISOString(), + updated_at: row.updatedAt.toISOString(), + } + } + + private rowToRunnerRuntimeDiagnostics( + row: typeof runnerRuntimeDiagnostics.$inferSelect + ): RunnerRuntimeDiagnostics { + return { + runner_id: row.runnerId, + owner_principal: row.ownerPrincipal, wake_stream_offset: row.wakeStreamOffset ?? undefined, + last_seen_at: row.lastSeenAt.toISOString(), + liveness_lease_expires_at: row.livenessLeaseExpiresAt.toISOString(), diagnostics: (row.diagnostics as Record) ?? undefined, - created_at: row.createdAt.toISOString(), updated_at: row.updatedAt.toISOString(), } } diff --git a/packages/agents-server/src/principal.ts b/packages/agents-server/src/principal.ts index c873e8afc7..996ccdeb5a 100644 --- a/packages/agents-server/src/principal.ts +++ b/packages/agents-server/src/principal.ts @@ -53,7 +53,12 @@ export function isPrincipalUrl(url: string): boolean { export function getPrincipalFromRequest(request: Request): Principal | null { const value = request.headers.get(ELECTRIC_PRINCIPAL_HEADER) - return value ? parsePrincipalKey(value) : null + if (!value) return null + try { + return parsePrincipalKey(value) + } catch { + return null + } } export function getDevPrincipal(): Principal { diff --git a/packages/agents-server/src/routing/dispatch-policy.ts b/packages/agents-server/src/routing/dispatch-policy.ts index 2d34b52f27..459bd517ee 100644 --- a/packages/agents-server/src/routing/dispatch-policy.ts +++ b/packages/agents-server/src/routing/dispatch-policy.ts @@ -115,6 +115,13 @@ export async function assertDispatchPolicyAllowed( ): Promise { const target = policy?.targets[0] if (!target || target.type !== `runner`) return + if (!ctx.principal) { + throw new ElectricAgentsError( + ErrCodeUnauthorized, + `Runner dispatch requires an authenticated owner`, + 401 + ) + } const runner = await ctx.entityManager.registry.getRunner(target.runnerId) if (!runner) { @@ -124,7 +131,7 @@ export async function assertDispatchPolicyAllowed( 404 ) } - if (ctx.principal && runner.owner_principal !== ctx.principal.url) { + if (runner.owner_principal !== ctx.principal.url) { throw new ElectricAgentsError( ErrCodeUnauthorized, `Runner dispatch requires the authenticated owner`, diff --git a/packages/agents-server/src/routing/runners-router.ts b/packages/agents-server/src/routing/runners-router.ts index ab08a6188e..d1d83c93b5 100644 --- a/packages/agents-server/src/routing/runners-router.ts +++ b/packages/agents-server/src/routing/runners-router.ts @@ -14,7 +14,7 @@ import { import { routeBody, withSchema } from './schema.js' import { subscriptionIdForDispatchTarget } from './dispatch-policy.js' import { withLeadingSlash } from './tenant-stream-paths.js' -import { isPrincipalUrl, principalFromCreatedBy } from '../principal.js' +import { parsePrincipalUrl, principalFromCreatedBy } from '../principal.js' import type { JsonRouteRequest } from './schema.js' import type { RouterType } from 'itty-router' import type { TenantContext } from './context.js' @@ -103,12 +103,28 @@ function firstQueryValue( return Array.isArray(value) ? value[0] : value } +function requireAuthenticatedPrincipal( + ctx: TenantContext +): NonNullable { + if (ctx.principal) return ctx.principal + throw new ElectricAgentsError( + ErrCodeUnauthorized, + `Runner route requires an authenticated principal`, + 401 + ) +} + +function canonicalOwnerPrincipal(input: string): string | null { + return parsePrincipalUrl(input)?.url ?? null +} + async function registerRunner( request: RunnersRouteRequest, ctx: TenantContext ): Promise { const parsed = routeBody(request) - const ownerPrincipal = parsed.owner_principal ?? ctx.principal?.url + const principal = requireAuthenticatedPrincipal(ctx) + const ownerPrincipal = parsed.owner_principal ?? principal.url if (!ownerPrincipal) { throw new ElectricAgentsError( ErrCodeInvalidRequest, @@ -116,14 +132,15 @@ async function registerRunner( 400 ) } - if (!isPrincipalUrl(ownerPrincipal)) { + const canonicalOwner = canonicalOwnerPrincipal(ownerPrincipal) + if (!canonicalOwner) { throw new ElectricAgentsError( ErrCodeInvalidRequest, `owner_principal must be a valid principal URL (e.g. /principal/user%3Aalice), got: ${ownerPrincipal}`, 400 ) } - if (ctx.principal && ownerPrincipal !== ctx.principal.url) { + if (canonicalOwner !== principal.url) { throw new ElectricAgentsError( ErrCodeUnauthorized, `owner_principal must match the authenticated principal`, @@ -133,7 +150,7 @@ async function registerRunner( const runner = await ctx.entityManager.registry.createRunner({ id: parsed.id, - ownerPrincipal, + ownerPrincipal: canonicalOwner, label: parsed.label, kind: parsed.kind, adminStatus: parsed.admin_status, @@ -149,15 +166,19 @@ async function listRunners( request: RunnersRouteRequest, ctx: TenantContext ): Promise { + const principal = requireAuthenticatedPrincipal(ctx) const requestedOwner = firstQueryValue(request.query.owner_principal) - if (requestedOwner && !isPrincipalUrl(requestedOwner)) { + const canonicalRequestedOwner = requestedOwner + ? canonicalOwnerPrincipal(requestedOwner) + : undefined + if (requestedOwner && !canonicalRequestedOwner) { throw new ElectricAgentsError( ErrCodeInvalidRequest, `owner_principal must be a valid principal URL (e.g. /principal/user%3Aalice), got: ${requestedOwner}`, 400 ) } - if (ctx.principal && requestedOwner && requestedOwner !== ctx.principal.url) { + if (canonicalRequestedOwner && canonicalRequestedOwner !== principal.url) { throw new ElectricAgentsError( ErrCodeUnauthorized, `owner_principal must match the authenticated principal`, @@ -165,7 +186,7 @@ async function listRunners( ) } const runners = await ctx.entityManager.registry.listRunners({ - ownerPrincipal: ctx.principal?.url ?? requestedOwner, + ownerPrincipal: principal.url, }) return json(runners) } @@ -184,11 +205,13 @@ async function heartbeat( ctx: TenantContext ): Promise { const runnerId = routeParam(request, `id`) + requireAuthenticatedPrincipal(ctx) const existing = await requireRunner(ctx, runnerId) assertRunnerOwnerIfAuthenticated(ctx, existing.owner_principal) const parsed = routeBody(request) const runner = await ctx.entityManager.registry.heartbeatRunner({ runnerId, + ownerPrincipal: existing.owner_principal, leaseMs: parsed.lease_ms, wakeStreamOffset: parsed.wake_stream_offset ?? parsed.wakeStreamOffset, livenessLeaseExpiresAt: parsed.liveness_lease_expires_at @@ -222,6 +245,7 @@ async function setRunnerStatus( adminStatus: `enabled` | `disabled` ): Promise { const runnerId = routeParam(request, `id`) + requireAuthenticatedPrincipal(ctx) const existing = await requireRunner(ctx, runnerId) assertRunnerOwnerIfAuthenticated(ctx, existing.owner_principal) const runner = await ctx.entityManager.registry.setRunnerAdminStatus( @@ -239,8 +263,9 @@ async function claimWake( ctx: TenantContext ): Promise { const runnerId = routeParam(request, `id`) + const principal = requireAuthenticatedPrincipal(ctx) const runner = await requireRunner(ctx, runnerId) - if (ctx.principal && runner.owner_principal !== ctx.principal.url) { + if (runner.owner_principal !== principal.url) { throw new ElectricAgentsError( ErrCodeUnauthorized, `Runner claim requires the authenticated owner`, @@ -316,7 +341,7 @@ function assertRunnerOwnerIfAuthenticated( ctx: TenantContext, ownerPrincipal: string ): void { - if (!ctx.principal) return + requireAuthenticatedPrincipal(ctx) if (ownerPrincipal === ctx.principal.url) return throw new ElectricAgentsError( ErrCodeUnauthorized, @@ -332,10 +357,12 @@ async function runnerHealth( const runnerId = routeParam(request, `id`) const runner = await requireRunner(ctx, runnerId) assertRunnerOwnerIfAuthenticated(ctx, runner.owner_principal) + const runtimeDiagnostics = + await ctx.entityManager.registry.getRunnerDiagnostics(runnerId) const now = Date.now() - const leaseExpiresAt = runner.liveness_lease_expires_at - ? new Date(runner.liveness_lease_expires_at).getTime() + const leaseExpiresAt = runtimeDiagnostics?.liveness_lease_expires_at + ? new Date(runtimeDiagnostics.liveness_lease_expires_at).getTime() : null let livenessStatus: `online` | `offline` | `expired` @@ -355,7 +382,7 @@ async function runnerHealth( ]) const clientDiagnostics = - (runner.diagnostics as RunnerHealthResponse[`client`]) ?? null + (runtimeDiagnostics?.diagnostics as RunnerHealthResponse[`client`]) ?? null const issues: Array = [] let healthStatus: `healthy` | `degraded` | `unhealthy` = `healthy` @@ -395,7 +422,7 @@ async function runnerHealth( `Client has reconnected ${clientDiagnostics.reconnect_count} times` ) } - } else if (runner.last_seen_at) { + } else if (runtimeDiagnostics?.last_seen_at) { escalate(`degraded`) issues.push(`No client diagnostics available`) } @@ -405,12 +432,12 @@ async function runnerHealth( id: runner.id, admin_status: runner.admin_status, liveness_status: livenessStatus, - lease_expires_at: runner.liveness_lease_expires_at ?? null, + lease_expires_at: runtimeDiagnostics?.liveness_lease_expires_at ?? null, lease_remaining_ms: leaseExpiresAt !== null ? Math.max(0, leaseExpiresAt - now) : null, wake_stream: runner.wake_stream, - wake_stream_offset: runner.wake_stream_offset ?? null, - last_seen_at: runner.last_seen_at ?? null, + wake_stream_offset: runtimeDiagnostics?.wake_stream_offset ?? null, + last_seen_at: runtimeDiagnostics?.last_seen_at ?? null, created_at: runner.created_at, }, client: clientDiagnostics, diff --git a/packages/agents-server/src/utils/server-utils.ts b/packages/agents-server/src/utils/server-utils.ts index 32a48a8e36..0c1e23d00c 100644 --- a/packages/agents-server/src/utils/server-utils.ts +++ b/packages/agents-server/src/utils/server-utils.ts @@ -131,7 +131,15 @@ export function buildElectricProxyTarget(options: { } else if (table === `runners`) { target.searchParams.set( `columns`, - `"tenant_id","id","owner_principal","label","kind","admin_status","wake_stream","wake_stream_offset","last_seen_at","liveness_lease_expires_at","diagnostics","created_at","updated_at"` + `"tenant_id","id","owner_principal","label","kind","admin_status","wake_stream","created_at","updated_at"` + ) + applyTenantShapeWhere(target, options.tenantId, [ + `owner_principal = ${sqlStringLiteral(options.principalUrl ?? ``)}`, + ]) + } else if (table === `runner_runtime_diagnostics`) { + target.searchParams.set( + `columns`, + `"tenant_id","runner_id","owner_principal","wake_stream_offset","last_seen_at","liveness_lease_expires_at","diagnostics","updated_at"` ) applyTenantShapeWhere(target, options.tenantId, [ `owner_principal = ${sqlStringLiteral(options.principalUrl ?? ``)}`, diff --git a/packages/agents-server/test/principal.test.ts b/packages/agents-server/test/principal.test.ts index ad424e6de4..1c4f5af9b0 100644 --- a/packages/agents-server/test/principal.test.ts +++ b/packages/agents-server/test/principal.test.ts @@ -1,5 +1,6 @@ import { describe, expect, it } from 'vitest' import { + getPrincipalFromRequest, parsePrincipalUrl, parsePrincipalKey, principalUrl, @@ -43,4 +44,12 @@ describe(`principal parser`, () => { expect(() => parsePrincipalKey(key)).toThrow() } }) + + it(`ignores malformed principal request headers`, () => { + const request = new Request(`http://server`, { + headers: { 'electric-principal': `not-a-principal` }, + }) + + expect(getPrincipalFromRequest(request)).toBeNull() + }) }) diff --git a/packages/agents-server/test/runners-router.test.ts b/packages/agents-server/test/runners-router.test.ts index 994843970d..79eaa680e9 100644 --- a/packages/agents-server/test/runners-router.test.ts +++ b/packages/agents-server/test/runners-router.test.ts @@ -49,6 +49,7 @@ function buildContext(overrides: Partial = {}): TenantContext { materializeActiveClaim: vi.fn(), updateStatus: vi.fn(), getActiveClaimsForRunner: vi.fn(async () => []), + getRunnerDiagnostics: vi.fn(async () => null), getDispatchStatsForRunner: vi.fn(async () => ({ entities_with_active_claim: 0, entities_with_outstanding_wake: 0, @@ -140,6 +141,26 @@ describe(`runner routes`, () => { ) }) + it(`canonicalizes legacy owner_principal URLs on registration`, async () => { + const ctx = buildContext() + + const response = await globalRouter.fetch( + request(`POST`, `/_electric/runners`, { + id: `runner-1`, + owner_principal: `/principal/user:owner@example.com`, + label: `Local runner`, + }), + ctx + ) + + expect(response.status).toBe(201) + expect(ctx.entityManager.registry.createRunner).toHaveBeenCalledWith( + expect.objectContaining({ + ownerPrincipal: `/principal/user%3Aowner%40example.com`, + }) + ) + }) + it(`infers runner owner from the authenticated user when omitted`, async () => { const ctx = buildContext({ principal: { @@ -166,19 +187,56 @@ describe(`runner routes`, () => { ) }) + it(`canonicalizes legacy owner_principal URLs when listing runners`, async () => { + const ctx = buildContext() + + const response = await globalRouter.fetch( + request( + `GET`, + `/_electric/runners?owner_principal=${encodeURIComponent(`/principal/user:owner@example.com`)}` + ), + ctx + ) + + expect(response.status).toBe(200) + expect(ctx.entityManager.registry.listRunners).toHaveBeenCalledWith({ + ownerPrincipal: `/principal/user%3Aowner%40example.com`, + }) + }) + + it(`rejects unauthenticated runner listing`, async () => { + const ctx = buildContext({ principal: undefined as any }) + + const response = await globalRouter.fetch( + request(`GET`, `/_electric/runners`), + ctx + ) + + expect(response.status).toBe(401) + expect(ctx.entityManager.registry.listRunners).not.toHaveBeenCalled() + }) + it(`returns runner health with diagnostics and claim state`, async () => { const ctx = buildContext() vi.mocked(ctx.entityManager.registry.getRunner).mockResolvedValue( runner({ - liveness_lease_expires_at: new Date(Date.now() + 30_000).toISOString(), - last_seen_at: new Date().toISOString(), - diagnostics: { - stream_connected: true, - reconnect_count: 0, - last_heartbeat_ok: true, - }, + admin_status: `enabled`, }) ) + vi.mocked( + ctx.entityManager.registry.getRunnerDiagnostics + ).mockResolvedValue({ + runner_id: `runner-1`, + owner_principal: `/principal/user%3Aowner%40example.com`, + liveness_lease_expires_at: new Date(Date.now() + 30_000).toISOString(), + last_seen_at: new Date().toISOString(), + diagnostics: { + stream_connected: true, + reconnect_count: 0, + last_heartbeat_ok: true, + }, + updated_at: new Date().toISOString(), + }) const response = await globalRouter.fetch( request(`GET`, `/_electric/runners/runner-1/health`), @@ -200,10 +258,18 @@ describe(`runner routes`, () => { const ctx = buildContext() vi.mocked(ctx.entityManager.registry.getRunner).mockResolvedValue( runner({ - liveness_lease_expires_at: new Date(Date.now() - 10_000).toISOString(), - last_seen_at: new Date(Date.now() - 15_000).toISOString(), + admin_status: `enabled`, }) ) + vi.mocked( + ctx.entityManager.registry.getRunnerDiagnostics + ).mockResolvedValue({ + runner_id: `runner-1`, + owner_principal: `/principal/user%3Aowner%40example.com`, + liveness_lease_expires_at: new Date(Date.now() - 10_000).toISOString(), + last_seen_at: new Date(Date.now() - 15_000).toISOString(), + updated_at: new Date().toISOString(), + }) const response = await globalRouter.fetch( request(`GET`, `/_electric/runners/runner-1/health`), @@ -216,8 +282,8 @@ describe(`runner routes`, () => { expect(body.health.issues.length).toBeGreaterThan(0) }) - it(`allows unauthenticated runner claims when no server auth is configured`, async () => { - const ctx = buildContext() + it(`rejects unauthenticated runner claims`, async () => { + const ctx = buildContext({ principal: undefined as any }) const response = await globalRouter.fetch( request(`POST`, `/_electric/runners/runner-1/claim`, { subscription_id: `runner:runner-1`, @@ -227,8 +293,23 @@ describe(`runner routes`, () => { ctx ) - expect(response.status).toBe(204) - expect(ctx.streamClient.claimSubscription).toHaveBeenCalled() + expect(response.status).toBe(401) + expect(ctx.streamClient.claimSubscription).not.toHaveBeenCalled() + }) + + it(`rejects unauthenticated runner registration for an explicit owner`, async () => { + const ctx = buildContext({ principal: undefined as any }) + const response = await globalRouter.fetch( + request(`POST`, `/_electric/runners`, { + id: `runner-1`, + owner_principal: `/principal/user%3Aowner%40example.com`, + label: `Local runner`, + }), + ctx + ) + + expect(response.status).toBe(401) + expect(ctx.entityManager.registry.createRunner).not.toHaveBeenCalled() }) it(`returns DS claim conflicts as 409 responses`, async () => { @@ -355,10 +436,17 @@ describe(`runner routes`, () => { vi.mocked(ctx.entityManager.registry.getRunner).mockResolvedValue( runner({ admin_status: `disabled`, - liveness_lease_expires_at: new Date(Date.now() + 30_000).toISOString(), - last_seen_at: new Date().toISOString(), }) ) + vi.mocked( + ctx.entityManager.registry.getRunnerDiagnostics + ).mockResolvedValue({ + runner_id: `runner-1`, + owner_principal: `/principal/user%3Aowner%40example.com`, + liveness_lease_expires_at: new Date(Date.now() + 30_000).toISOString(), + last_seen_at: new Date().toISOString(), + updated_at: new Date().toISOString(), + }) const response = await globalRouter.fetch( request(`GET`, `/_electric/runners/runner-1/health`), @@ -376,15 +464,23 @@ describe(`runner routes`, () => { const ctx = buildContext() vi.mocked(ctx.entityManager.registry.getRunner).mockResolvedValue( runner({ - liveness_lease_expires_at: new Date(Date.now() + 30_000).toISOString(), - last_seen_at: new Date().toISOString(), - diagnostics: { - stream_connected: false, - reconnect_count: 2, - last_heartbeat_ok: true, - }, + admin_status: `enabled`, }) ) + vi.mocked( + ctx.entityManager.registry.getRunnerDiagnostics + ).mockResolvedValue({ + runner_id: `runner-1`, + owner_principal: `/principal/user%3Aowner%40example.com`, + liveness_lease_expires_at: new Date(Date.now() + 30_000).toISOString(), + last_seen_at: new Date().toISOString(), + diagnostics: { + stream_connected: false, + reconnect_count: 2, + last_heartbeat_ok: true, + }, + updated_at: new Date().toISOString(), + }) const response = await globalRouter.fetch( request(`GET`, `/_electric/runners/runner-1/health`), diff --git a/packages/agents-server/test/server-utils.test.ts b/packages/agents-server/test/server-utils.test.ts index de7796e55c..b81883a2ce 100644 --- a/packages/agents-server/test/server-utils.test.ts +++ b/packages/agents-server/test/server-utils.test.ts @@ -16,7 +16,10 @@ describe(`server utils`, () => { expect(target.pathname).toBe(`/v1/shape`) expect(target.searchParams.get(`table`)).toBe(`runners`) - expect(target.searchParams.get(`columns`)).toContain(`"owner_principal"`) + const columns = target.searchParams.get(`columns`) + expect(columns).toContain(`"owner_principal"`) + expect(columns).not.toContain(`"diagnostics"`) + expect(columns).not.toContain(`"last_seen_at"`) expect(target.searchParams.get(`where`)).toBe( `tenant_id = 'tenant-test' AND owner_principal = '/principal/user%3Aowner%40example.com'` ) @@ -31,4 +34,16 @@ describe(`server utils`, () => { `tenant_id = 'tenant-test' AND owner_principal = '/principal/user%3Aowner%40example.com' AND (kind = 'local')` ) }) + + it(`owner-scopes runner runtime diagnostics shapes and preserves runner filters`, () => { + const target = shapeTarget( + `table=runner_runtime_diagnostics&where=${encodeURIComponent(`runner_id = 'runner-1'`)}` + ) + + expect(target.searchParams.get(`table`)).toBe(`runner_runtime_diagnostics`) + expect(target.searchParams.get(`columns`)).toContain(`"diagnostics"`) + expect(target.searchParams.get(`where`)).toBe( + `tenant_id = 'tenant-test' AND owner_principal = '/principal/user%3Aowner%40example.com' AND (runner_id = 'runner-1')` + ) + }) }) From 365366be01e45063c38899f07412f159cc24eac9 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Mon, 18 May 2026 06:36:43 -0600 Subject: [PATCH 18/37] fix(agents): tighten runner lifecycle diagnostics --- .../agents-runtime/src/pull-wake-runner.ts | 89 +++++++++++------- .../test/pull-wake-runner.test.ts | 93 +++++++++++++++++++ .../src/routing/runners-router.ts | 83 ++++++++++++++++- .../agents-server/test/runners-router.test.ts | 82 ++++++++++++++++ 4 files changed, 310 insertions(+), 37 deletions(-) diff --git a/packages/agents-runtime/src/pull-wake-runner.ts b/packages/agents-runtime/src/pull-wake-runner.ts index b8f348fd55..6f34d7db45 100644 --- a/packages/agents-runtime/src/pull-wake-runner.ts +++ b/packages/agents-runtime/src/pull-wake-runner.ts @@ -131,6 +131,7 @@ export function createPullWakeRunner( let runGeneration = 0 let nextReconnectBackoffMs = INITIAL_RECONNECT_BACKOFF_MS let streamResetError: Error | null = null + let stopPromise: Promise | null = null const claimActors = new Map, number>() const wakePath = @@ -397,24 +398,32 @@ export function createPullWakeRunner( const waitForClaimCapacity = async ( signal: AbortSignal ): Promise => { + let abortListener: (() => void) | null = null const abortPromise = new Promise((resolve) => { if (signal.aborted) { resolve() return } - signal.addEventListener(`abort`, () => resolve(), { once: true }) + abortListener = () => resolve() + signal.addEventListener(`abort`, abortListener, { once: true }) }) - while ( - acceptingClaims && - !signal.aborted && - activeClaimCount >= maxConcurrentClaims - ) { - const inFlight = [...claimActors.keys()] - if (inFlight.length === 0) return true - await Promise.race([...inFlight, abortPromise]).catch(() => undefined) + try { + while ( + acceptingClaims && + !signal.aborted && + activeClaimCount >= maxConcurrentClaims + ) { + const inFlight = [...claimActors.keys()] + if (inFlight.length === 0) return true + await Promise.race([...inFlight, abortPromise]).catch(() => undefined) + } + return acceptingClaims && !signal.aborted + } finally { + if (abortListener) { + signal.removeEventListener(`abort`, abortListener) + } } - return acceptingClaims && !signal.aborted } const claimAndDispatch = async ( @@ -526,7 +535,10 @@ export function createPullWakeRunner( notifyHeartbeatChange() } } - if (response.offset !== undefined) { + if ( + response.offset !== undefined && + response.offset !== currentOffset + ) { currentOffset = response.offset notifyHeartbeatChange() } @@ -589,9 +601,36 @@ export function createPullWakeRunner( } } + const stopRunner = async (): Promise => { + if (state === `stopped`) return + state = `stopping` + acceptingClaims = false + controller?.abort() + stopHeartbeat() + response?.cancel?.(new Error(`pull wake runner stopped`)) + if (!(await waitForClaimActors())) { + claimActors.clear() + activeClaimCount = 0 + } + config.runtime.abortWakes() + await loop?.catch((err) => { + if (!(err instanceof Error && err.name === `AbortError`)) throw err + }) + let drainError: unknown + try { + await config.runtime.drainWakes() + } catch (err) { + reportError(err) + drainError = err + } finally { + state = `stopped` + } + if (drainError) throw drainError + } + return { start() { - if (loop) return + if (loop || stopPromise) return state = `starting` controller = new AbortController() runGeneration++ @@ -603,30 +642,10 @@ export function createPullWakeRunner( }) }, async stop() { - if (state === `stopped`) return - state = `stopping` - acceptingClaims = false - controller?.abort() - stopHeartbeat() - response?.cancel?.(new Error(`pull wake runner stopped`)) - if (!(await waitForClaimActors())) { - claimActors.clear() - activeClaimCount = 0 - } - config.runtime.abortWakes() - await loop?.catch((err) => { - if (!(err instanceof Error && err.name === `AbortError`)) throw err + stopPromise ??= stopRunner().finally(() => { + stopPromise = null }) - let drainError: unknown - try { - await config.runtime.drainWakes() - } catch (err) { - reportError(err) - drainError = err - } finally { - state = `stopped` - } - if (drainError) throw drainError + await stopPromise }, async waitForStopped() { await loop diff --git a/packages/agents-runtime/test/pull-wake-runner.test.ts b/packages/agents-runtime/test/pull-wake-runner.test.ts index 57cfe15bd6..95ac427dac 100644 --- a/packages/agents-runtime/test/pull-wake-runner.test.ts +++ b/packages/agents-runtime/test/pull-wake-runner.test.ts @@ -331,6 +331,50 @@ describe(`createPullWakeRunner`, () => { await runner.stop() }) + it(`does not schedule event heartbeats for unchanged stream offsets`, async () => { + const heartbeatBodies: Array> = [] + const yieldEvent = deferred() + const streamClosed = deferred() + const fetchMock = vi.fn( + async (_input: RequestInfo | URL, init?: RequestInit) => { + heartbeatBodies.push(JSON.parse(String(init?.body))) + return Response.json({}) + } + ) + vi.stubGlobal(`fetch`, fetchMock) + const streamFactory = vi.fn(async () => ({ + offset: `42`, + async *jsonStream() { + await yieldEvent.promise + yield { type: `noop` } as unknown as PullWakeEvent + await streamClosed.promise + }, + cancel: () => streamClosed.resolve(), + closed: streamClosed.promise, + })) + + const runner = createPullWakeRunner({ + baseUrl: `http://server`, + runnerId: `runner-1`, + runtime: runtime(), + offset: `42`, + heartbeatIntervalMs: 60_000, + eventHeartbeatThrottleMs: 5, + streamFactory, + }) + + runner.start() + await waitFor(() => { + expect(heartbeatBodies.length).toBe(2) + }) + + yieldEvent.resolve() + await new Promise((resolve) => setTimeout(resolve, 20)) + + expect(heartbeatBodies.length).toBe(2) + await runner.stop() + }) + it(`resolves async headers before opening the durable stream`, async () => { durableStreamMocks.stream.mockResolvedValueOnce({ offset: `42`, @@ -814,6 +858,55 @@ describe(`createPullWakeRunner`, () => { expect(runner.getHealth().last_error).toBe(`drain failed`) }) + it(`shares one shutdown sequence across concurrent stop calls`, async () => { + const streamClosed = deferred() + const drainStarted = deferred() + const drainReleased = deferred() + const testRuntime = { + dispatchWake: vi.fn(), + abortWakes: vi.fn(), + drainWakes: vi.fn(async () => { + drainStarted.resolve() + await drainReleased.promise + }), + } + const streamFactory = vi.fn(async () => ({ + async *jsonStream() { + await streamClosed.promise + }, + cancel: () => streamClosed.resolve(), + closed: streamClosed.promise, + })) + const runner = createPullWakeRunner({ + baseUrl: `http://server`, + runnerId: `runner-1`, + runtime: testRuntime, + heartbeatIntervalMs: 0, + streamFactory, + }) + + runner.start() + await waitFor(() => { + expect(streamFactory).toHaveBeenCalledTimes(1) + }) + + const firstStop = runner.stop() + const secondStop = runner.stop() + await drainStarted.promise + + expect(testRuntime.abortWakes).toHaveBeenCalledTimes(1) + expect(testRuntime.drainWakes).toHaveBeenCalledTimes(1) + + runner.start() + expect(streamFactory).toHaveBeenCalledTimes(1) + + drainReleased.resolve() + await Promise.all([firstStop, secondStop]) + + expect(testRuntime.abortWakes).toHaveBeenCalledTimes(1) + expect(testRuntime.drainWakes).toHaveBeenCalledTimes(1) + }) + it(`uses exponential reconnect backoff between failed connection attempts`, async () => { vi.useFakeTimers() const attempts = [deferred(), deferred(), deferred()] diff --git a/packages/agents-server/src/routing/runners-router.ts b/packages/agents-server/src/routing/runners-router.ts index d1d83c93b5..c7386a01a8 100644 --- a/packages/agents-server/src/routing/runners-router.ts +++ b/packages/agents-server/src/routing/runners-router.ts @@ -74,6 +74,35 @@ const claimBodySchema = Type.Object( type RegisterRunnerBody = Static type HeartbeatBody = Static type ClaimBody = Static +type RunnerClientDiagnostics = NonNullable + +const runnerClientStatuses = new Set([ + `stopped`, + `starting`, + `connecting`, + `streaming`, + `reconnecting`, + `stopping`, +]) +const runnerLastClaimResults = new Set< + NonNullable +>([`claimed`, `no_work`, `error`]) +const runnerStringOrNullDiagnostics = [ + `started_at`, + `stream_connected_since`, + `last_error`, + `last_error_at`, + `last_heartbeat_at`, + `last_claim_at`, + `last_dispatch_at`, +] as const +const runnerNumberDiagnostics = [ + `reconnect_count`, + `events_received`, + `claims_succeeded`, + `claims_skipped`, + `claims_failed`, +] as const export const runnersRouter: RunnersRoutes = Router< RunnersRouteRequest, @@ -118,6 +147,56 @@ function canonicalOwnerPrincipal(input: string): string | null { return parsePrincipalUrl(input)?.url ?? null } +function sanitizeRunnerDiagnostics( + diagnostics: Record | null | undefined +): RunnerClientDiagnostics | undefined { + if (!diagnostics) return undefined + const sanitized: Record = {} + + if ( + typeof diagnostics.status === `string` && + runnerClientStatuses.has( + diagnostics.status as RunnerClientDiagnostics[`status`] + ) + ) { + sanitized.status = diagnostics.status + } + if (typeof diagnostics.stream_connected === `boolean`) { + sanitized.stream_connected = diagnostics.stream_connected + } + if (typeof diagnostics.last_heartbeat_ok === `boolean`) { + sanitized.last_heartbeat_ok = diagnostics.last_heartbeat_ok + } + if ( + diagnostics.last_claim_result === null || + (typeof diagnostics.last_claim_result === `string` && + runnerLastClaimResults.has( + diagnostics.last_claim_result as NonNullable< + RunnerClientDiagnostics[`last_claim_result`] + > + )) + ) { + sanitized.last_claim_result = diagnostics.last_claim_result + } + + for (const key of runnerStringOrNullDiagnostics) { + const value = diagnostics[key] + if (typeof value === `string` || value === null) { + sanitized[key] = value + } + } + for (const key of runnerNumberDiagnostics) { + const value = diagnostics[key] + if (typeof value === `number` && Number.isFinite(value) && value >= 0) { + sanitized[key] = value + } + } + + return Object.keys(sanitized).length > 0 + ? (sanitized as RunnerClientDiagnostics) + : undefined +} + async function registerRunner( request: RunnersRouteRequest, ctx: TenantContext @@ -217,7 +296,7 @@ async function heartbeat( livenessLeaseExpiresAt: parsed.liveness_lease_expires_at ? new Date(parsed.liveness_lease_expires_at) : undefined, - diagnostics: parsed.diagnostics, + diagnostics: sanitizeRunnerDiagnostics(parsed.diagnostics), }) if (!runner) { throw new ElectricAgentsError(ErrCodeNotFound, `Runner not found`, 404) @@ -382,7 +461,7 @@ async function runnerHealth( ]) const clientDiagnostics = - (runtimeDiagnostics?.diagnostics as RunnerHealthResponse[`client`]) ?? null + sanitizeRunnerDiagnostics(runtimeDiagnostics?.diagnostics) ?? null const issues: Array = [] let healthStatus: `healthy` | `degraded` | `unhealthy` = `healthy` diff --git a/packages/agents-server/test/runners-router.test.ts b/packages/agents-server/test/runners-router.test.ts index 79eaa680e9..4e23641e02 100644 --- a/packages/agents-server/test/runners-router.test.ts +++ b/packages/agents-server/test/runners-router.test.ts @@ -254,6 +254,88 @@ describe(`runner routes`, () => { expect(body.health).toMatchObject({ status: `healthy`, issues: [] }) }) + it(`sanitizes heartbeat diagnostics before storing them`, async () => { + const ctx = buildContext() + + const response = await globalRouter.fetch( + request(`POST`, `/_electric/runners/runner-1/heartbeat`, { + lease_ms: 30_000, + wake_stream_offset: `123`, + diagnostics: { + status: `streaming`, + stream_connected: `yes`, + stream_connected_since: null, + reconnect_count: `2`, + last_heartbeat_ok: false, + last_claim_result: `invalid`, + last_error: `heartbeat failed`, + claims_failed: 1, + events_received: -1, + extra: { noisy: true }, + }, + }), + ctx + ) + + expect(response.status).toBe(200) + const heartbeatInput = vi.mocked(ctx.entityManager.registry.heartbeatRunner) + .mock.calls[0]![0] + expect(heartbeatInput).toMatchObject({ + runnerId: `runner-1`, + wakeStreamOffset: `123`, + diagnostics: { + status: `streaming`, + stream_connected_since: null, + last_heartbeat_ok: false, + last_error: `heartbeat failed`, + claims_failed: 1, + }, + }) + expect(heartbeatInput.diagnostics).not.toHaveProperty(`stream_connected`) + expect(heartbeatInput.diagnostics).not.toHaveProperty(`reconnect_count`) + expect(heartbeatInput.diagnostics).not.toHaveProperty(`last_claim_result`) + expect(heartbeatInput.diagnostics).not.toHaveProperty(`events_received`) + expect(heartbeatInput.diagnostics).not.toHaveProperty(`extra`) + }) + + it(`sanitizes stored runner diagnostics before returning health`, async () => { + const ctx = buildContext() + vi.mocked(ctx.entityManager.registry.getRunner).mockResolvedValue( + runner({ + admin_status: `enabled`, + }) + ) + vi.mocked( + ctx.entityManager.registry.getRunnerDiagnostics + ).mockResolvedValue({ + runner_id: `runner-1`, + owner_principal: `/principal/user%3Aowner%40example.com`, + liveness_lease_expires_at: new Date(Date.now() + 30_000).toISOString(), + last_seen_at: new Date().toISOString(), + diagnostics: { + stream_connected: `yes`, + reconnect_count: 6, + last_heartbeat_ok: false, + last_error: 500, + }, + updated_at: new Date().toISOString(), + }) + + const response = await globalRouter.fetch( + request(`GET`, `/_electric/runners/runner-1/health`), + ctx + ) + + expect(response.status).toBe(200) + const body = (await response.json()) as Record + expect(body.client).toEqual({ + reconnect_count: 6, + last_heartbeat_ok: false, + }) + expect(body.health.issues).toContain(`Client reports last heartbeat failed`) + expect(body.health.issues).toContain(`Client has reconnected 6 times`) + }) + it(`returns unhealthy when runner lease is expired`, async () => { const ctx = buildContext() vi.mocked(ctx.entityManager.registry.getRunner).mockResolvedValue( From 6ba5063e4b1f652c0bb3663160c38ca7800866da Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Mon, 18 May 2026 06:54:02 -0600 Subject: [PATCH 19/37] fix(agents-server): avoid duplicate dispatch subscription links --- .../src/routing/dispatch-policy.ts | 57 +++++++--------- .../test/dispatch-policy-routing.test.ts | 35 ++++++++-- .../agents-server/test/wake-registry.test.ts | 65 ------------------- 3 files changed, 51 insertions(+), 106 deletions(-) diff --git a/packages/agents-server/src/routing/dispatch-policy.ts b/packages/agents-server/src/routing/dispatch-policy.ts index 459bd517ee..fd493d5d60 100644 --- a/packages/agents-server/src/routing/dispatch-policy.ts +++ b/packages/agents-server/src/routing/dispatch-policy.ts @@ -109,6 +109,23 @@ function sameDispatchDestination( return false } +function subscriptionHasStream( + ctx: TenantContext, + existing: { streams?: Array }, + streamPath: string +): boolean { + const normalizedStream = streamPath.replace(/^\/+/, ``) + const backendStream = `${ctx.service}/${normalizedStream}` + return ( + existing.streams?.some((stream) => { + const path = typeof stream === `string` ? stream : stream.path + if (!path) return false + const normalized = path.replace(/^\/+/, ``) + return normalized === normalizedStream || normalized === backendStream + }) ?? false + ) +} + export async function assertDispatchPolicyAllowed( ctx: TenantContext, policy: DispatchPolicy | undefined @@ -210,21 +227,13 @@ async function linkStreamToTargetSubscription( wake_stream: wakeStream, description: `Electric Agents runner ${target.runnerId}`, }) - await removeLegacyRunnerSubscriptionStream( - ctx, - target, - subscriptionId, - streamPath - ) return } - await ctx.streamClient.addSubscriptionStreams(subscriptionId, [streamPath]) - await removeLegacyRunnerSubscriptionStream( - ctx, - target, - subscriptionId, - streamPath - ) + if (!subscriptionHasStream(ctx, existing, streamPath)) { + await ctx.streamClient.addSubscriptionStreams(subscriptionId, [ + streamPath, + ]) + } return } @@ -247,7 +256,7 @@ async function linkStreamToTargetSubscription( webhook: { url: forwardUrl }, description: `Electric Agents webhook ${subscriptionId}`, }) - } else { + } else if (!subscriptionHasStream(ctx, existing, streamPath)) { await ctx.streamClient.addSubscriptionStreams(subscriptionId, [streamPath]) } await ctx.pgDb @@ -265,23 +274,3 @@ async function linkStreamToTargetSubscription( set: { webhookUrl }, }) } - -async function removeLegacyRunnerSubscriptionStream( - ctx: TenantContext, - target: Extract, - subscriptionId: string, - streamPath: string -): Promise { - const legacySubscriptionId = subscriptionIdForDispatchTarget(target) - if (legacySubscriptionId === subscriptionId) return - - await ctx.streamClient - .removeSubscriptionStream(legacySubscriptionId, streamPath) - .catch((err) => { - serverLog.warn( - `[dispatch-policy] failed to remove legacy runner stream from subscription`, - { subscriptionId: legacySubscriptionId, stream: streamPath }, - err - ) - }) -} diff --git a/packages/agents-server/test/dispatch-policy-routing.test.ts b/packages/agents-server/test/dispatch-policy-routing.test.ts index e8e4c7d394..5ce3e0f338 100644 --- a/packages/agents-server/test/dispatch-policy-routing.test.ts +++ b/packages/agents-server/test/dispatch-policy-routing.test.ts @@ -124,10 +124,6 @@ describe(`dispatch policy routing`, () => { wake_stream: `/runners/runner-1/wake`, }) ) - expect(ctx.streamClient.removeSubscriptionStream).toHaveBeenCalledWith( - `runner:runner-1`, - `/chat/one/main` - ) }) it(`uses separate pull-wake subscriptions for separate runner-targeted entities`, async () => { @@ -298,10 +294,35 @@ describe(`dispatch policy routing`, () => { wake_stream: `/runners/runner-1/wake`, }) ) - expect(ctx.streamClient.removeSubscriptionStream).toHaveBeenCalledWith( - `runner:runner-1`, - `/chat/one/main` + expect(ctx.entityManager.send).toHaveBeenCalledWith( + `/chat/one`, + expect.objectContaining({ payload: `hello` }) + ) + }) + + it(`does not re-add already linked runner streams before sending`, async () => { + const dispatchPolicy: DispatchPolicy = { + targets: [{ type: `runner`, runnerId: `runner-1` }], + } + const ctx = buildContext() + ;(ctx.entityManager.registry.getEntity as any).mockResolvedValue( + entity(dispatchPolicy) + ) + ;(ctx.streamClient.getSubscription as any).mockResolvedValue({ + streams: [{ path: `tenant-test/chat/one/main` }], + }) + ctx.entityManager.send = vi.fn(async () => undefined) + + const response = await globalRouter.fetch( + request(`POST`, `/_electric/entities/chat/one/send`, { + payload: `hello`, + }), + ctx ) + + expect(response.status).toBe(204) + expect(ctx.streamClient.addSubscriptionStreams).not.toHaveBeenCalled() + expect(ctx.streamClient.removeSubscriptionStream).not.toHaveBeenCalled() expect(ctx.entityManager.send).toHaveBeenCalledWith( `/chat/one`, expect.objectContaining({ payload: `hello` }) diff --git a/packages/agents-server/test/wake-registry.test.ts b/packages/agents-server/test/wake-registry.test.ts index b804a2c497..f1577ba729 100644 --- a/packages/agents-server/test/wake-registry.test.ts +++ b/packages/agents-server/test/wake-registry.test.ts @@ -761,8 +761,6 @@ describe(`Wake Registry Integration`, () => { let baseUrl: string let receiver: Server let receiverUrl: string - let wakeCount = 0 - let wakeResolvers: Array<() => void> = [] function getElectricAgentsManager(): EntityManager { return (electricAgentsServer as any).electricAgentsManager as EntityManager @@ -774,12 +772,8 @@ describe(`Wake Registry Integration`, () => { const chunks: Array = [] req.on(`data`, (c: Buffer) => chunks.push(c)) req.on(`end`, () => { - wakeCount++ res.writeHead(200, { 'content-type': `application/json` }) res.end(JSON.stringify({ done: true })) - const resolvers = wakeResolvers - wakeResolvers = [] - for (const resolve of resolvers) resolve() }) }) @@ -823,36 +817,6 @@ describe(`Wake Registry Integration`, () => { }) }, 120_000) - function waitForWakes( - targetCount: number, - timeoutMs = 10_000 - ): Promise { - return new Promise((resolve, reject) => { - if (wakeCount >= targetCount) { - resolve() - return - } - const timeout = setTimeout( - () => - reject( - new Error( - `Timed out waiting for ${targetCount} wakes (got ${wakeCount})` - ) - ), - timeoutMs - ) - const check = (): void => { - if (wakeCount >= targetCount) { - clearTimeout(timeout) - resolve() - } else { - wakeResolvers.push(check) - } - } - wakeResolvers.push(check) - }) - } - async function waitForWakeEvents( streamPath: string, count: number, @@ -898,7 +862,6 @@ describe(`Wake Registry Integration`, () => { } it(`spawn with wake registers condition and delivers wake on child run completion`, async () => { - const startCount = wakeCount const ts = Date.now() const typeName = `wakerf${ts}` @@ -931,16 +894,6 @@ describe(`Wake Registry Integration`, () => { streams: { main: string } } - // Send a message to trigger the initial webhook wake for parent - await fetch(`${baseUrl}/_electric/entities${parent.url}/send`, { - method: `POST`, - headers: { 'content-type': `application/json` }, - body: JSON.stringify({ payload: `init` }), - }) - - // Wait for the parent's webhook - await waitForWakes(startCount + 1) - // Spawn child entity const childRes = await fetch( `${baseUrl}/_electric/entities/${typeName}/child`, @@ -1451,15 +1404,6 @@ describe(`Wake Registry Integration`, () => { oneShot: false, }) - // Send a message to watcher to trigger initial webhook (transition consumer to idle) - await fetch(`${baseUrl}/_electric/entities${watcher.url}/send`, { - method: `POST`, - headers: { 'content-type': `application/json` }, - body: JSON.stringify({ payload: `init` }), - }) - const afterSendTarget = wakeCount + 1 - await waitForWakes(afterSendTarget) - // Trigger wake evaluation directly on the manager await manager.evaluateWakes(source.url, { type: `texts`, @@ -1528,15 +1472,6 @@ describe(`Wake Registry Integration`, () => { oneShot: false, }) - // Trigger initial webhook for subscriber - await fetch(`${baseUrl}/_electric/entities${subscriber.url}/send`, { - method: `POST`, - headers: { 'content-type': `application/json` }, - body: JSON.stringify({ payload: `init` }), - }) - const afterSendTarget = wakeCount + 1 - await waitForWakes(afterSendTarget) - // Trigger wake evaluation directly await manager.evaluateWakes(observed.url, { type: `run`, From a5fdad5bca546306ca8eabcb71e6e0a744d4c4f8 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Mon, 18 May 2026 07:09:55 -0600 Subject: [PATCH 20/37] fix(agents-desktop): default local runner owner to dev principal --- packages/agents-desktop/README.md | 17 ++++++++++------- packages/agents-desktop/src/main.ts | 4 ++-- .../plans/2026-05-16-pull-wake-health-check.md | 4 ++-- 3 files changed, 14 insertions(+), 11 deletions(-) diff --git a/packages/agents-desktop/README.md b/packages/agents-desktop/README.md index 6eac3be5be..5f2698e89c 100644 --- a/packages/agents-desktop/README.md +++ b/packages/agents-desktop/README.md @@ -11,19 +11,22 @@ Desktop app for Electric Agents, built with Electron. ### Running the dev server ```bash -ELECTRIC_DESKTOP_PRINCIPAL="system:dev-local" pnpm dev +pnpm dev ``` This starts both the UI dev server (with HMR) and the Electron main process. +For a local unauthenticated agents-server, desktop defaults the pull-wake +runner owner to the same `system:dev-local` principal that agents-server uses in +dev fallback mode. ### Environment variables -| Variable | Default | Description | -| -------------------------------------------- | ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `ELECTRIC_DESKTOP_PRINCIPAL` | _(none)_ | Sets the `electric-principal` header on all requests to the agents-server. Use `system:dev-local` for local development without auth. | -| `ELECTRIC_DESKTOP_PULL_WAKE_OWNER_PRINCIPAL` | `/principal/system%3Alocal-desktop` | Override the `owner_principal` used when registering the pull-wake runner. When `ELECTRIC_DESKTOP_PRINCIPAL` is set, this is derived from it automatically. | -| `ELECTRIC_DESKTOP_PULL_WAKE_RUNNER_ID` | _(auto-generated)_ | Fixed runner ID for the pull-wake runner. | -| `ELECTRIC_DESKTOP_PULL_WAKE_REGISTER_RUNNER` | `true` | Set to `false` to skip runner registration (runner must already exist on the server). | +| Variable | Default | Description | +| -------------------------------------------- | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `ELECTRIC_DESKTOP_PRINCIPAL` | _(none)_ | Sets the `electric-principal` header on all requests to the agents-server. Usually unnecessary for local development because agents-server falls back to `system:dev-local`. | +| `ELECTRIC_DESKTOP_PULL_WAKE_OWNER_PRINCIPAL` | `/principal/system%3Adev-local` | Override the `owner_principal` used when registering the pull-wake runner. When `ELECTRIC_DESKTOP_PRINCIPAL` is set, this is derived from it automatically. | +| `ELECTRIC_DESKTOP_PULL_WAKE_RUNNER_ID` | _(auto-generated)_ | Fixed runner ID for the pull-wake runner. | +| `ELECTRIC_DESKTOP_PULL_WAKE_REGISTER_RUNNER` | `true` | Set to `false` to skip runner registration (runner must already exist on the server). | ### Settings diff --git a/packages/agents-desktop/src/main.ts b/packages/agents-desktop/src/main.ts index e0ca10fca9..6444b8aecf 100644 --- a/packages/agents-desktop/src/main.ts +++ b/packages/agents-desktop/src/main.ts @@ -255,7 +255,7 @@ const PULL_WAKE_REGISTER_RUNNER = ) const PULL_WAKE_OWNER_PRINCIPAL = process.env.ELECTRIC_DESKTOP_PULL_WAKE_OWNER_PRINCIPAL?.trim() || - `/principal/system%3Alocal-desktop` + `/principal/system%3Adev-local` const DEV_PRINCIPAL = ((): string | null => { const raw = process.env.ELECTRIC_DESKTOP_PRINCIPAL?.trim() || null if (!raw) return null @@ -1912,7 +1912,7 @@ async function startRuntime(serverId: string): Promise { // For `electric-cloud` source servers, the cloud-agents-server // authenticates each request via `x-electric-asserted-user-id` // headers (injected by the undici / webRequest hooks). Register the - // runner under that user principal instead of the `local-desktop` + // runner under that user principal instead of the dev-local // fallback used for unauthenticated local servers. const cloudAuthUserId = activeServer.source === `electric-cloud` diff --git a/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md b/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md index 4a51231515..a1cf1e3bbd 100644 --- a/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md +++ b/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md @@ -18,7 +18,7 @@ - [x] **Step 1: Write the migration SQL** -Existing `owner_user_id` values are key-form strings (e.g., `local-desktop`). The new column expects principal URLs (e.g., `/principal/system%3Alocal-desktop`). Since we have no backwards compatibility, the migration deletes existing runner rows — runners are ephemeral and will re-register on next startup. Must also clean up dependent tables (`consumer_claims` and `entity_dispatch_state`) since there are no FK constraints to cascade the deletes. +Existing `owner_user_id` values are key-form strings (e.g., `dev-local`). The new column expects principal URLs (e.g., `/principal/system%3Adev-local`). Since we have no backwards compatibility, the migration deletes existing runner rows — runners are ephemeral and will re-register on next startup. Must also clean up dependent tables (`consumer_claims` and `entity_dispatch_state`) since there are no FK constraints to cascade the deletes. ```sql UPDATE consumer_claims SET status = 'expired', updated_at = NOW() WHERE status = 'active' AND runner_id IS NOT NULL; @@ -1328,7 +1328,7 @@ Replace the previous owner-user constant/env var. No backwards-compat fallback ```ts const PULL_WAKE_OWNER_PRINCIPAL = process.env.ELECTRIC_DESKTOP_PULL_WAKE_OWNER_PRINCIPAL?.trim() || - `/principal/system%3Alocal-desktop` + `/principal/system%3Adev-local` ``` Rename the helper function (line 265-274). Do NOT use the `authorization` header as a principal source — that's a bearer token, not a principal identifier. When the request has auth headers, the server middleware extracts `ctx.principal` from them and uses `ctx.principal.url` as the owner. So when only auth is present (no explicit `electric-principal` header), return `undefined` to let the server derive the owner: From 450e8e5d56fe52bba86f5fe3fcfc8efceb5707fe Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Mon, 18 May 2026 07:31:02 -0600 Subject: [PATCH 21/37] fix(agents-server): tolerate dispatch subscription races --- .../src/routing/dispatch-policy.ts | 73 +++++++++++++++---- .../test/dispatch-policy-routing.test.ts | 43 +++++++++++ 2 files changed, 100 insertions(+), 16 deletions(-) diff --git a/packages/agents-server/src/routing/dispatch-policy.ts b/packages/agents-server/src/routing/dispatch-policy.ts index fd493d5d60..cd8e8799ed 100644 --- a/packages/agents-server/src/routing/dispatch-policy.ts +++ b/packages/agents-server/src/routing/dispatch-policy.ts @@ -8,6 +8,7 @@ import { ErrCodeUnauthorized, } from '../electric-agents-types.js' import { runnerWakeStream } from '../entity-registry.js' +import { DurableStreamsSubscriptionError } from '../stream-client.js' import { rewriteLoopbackWebhookUrl } from '../utils/webhook-url.js' import { serverLog } from '../utils/log.js' import type { @@ -16,6 +17,7 @@ import type { ElectricAgentsEntity, } from '../electric-agents-types.js' import type { TenantContext } from './context.js' +import type { SubscriptionCreateInput } from '../stream-client.js' export function subscriptionIdForDispatchTarget( target: DispatchTarget @@ -126,6 +128,45 @@ function subscriptionHasStream( ) } +function isSubscriptionAlreadyExistsError(err: unknown): boolean { + if (!(err instanceof DurableStreamsSubscriptionError)) return false + if (err.status === 409) return true + return ( + err.code === `SUBSCRIPTION_ALREADY_EXISTS` || + err.code === `ALREADY_EXISTS` || + /already exists/i.test(err.errorMessage ?? err.body ?? err.message) + ) +} + +async function ensureSubscriptionIncludesStream( + ctx: TenantContext, + subscriptionId: string, + streamPath: string, + input: SubscriptionCreateInput, + existing: { streams?: Array } | null +): Promise { + if (!existing) { + try { + await ctx.streamClient.putSubscription(subscriptionId, input) + return + } catch (err) { + if (!isSubscriptionAlreadyExistsError(err)) throw err + existing = await ctx.streamClient.getSubscription(subscriptionId) + if (!existing) { + serverLog.warn( + `[dispatch-policy] subscription create raced with existing subscription but it could not be read`, + { subscriptionId, stream: streamPath } + ) + return + } + } + } + + if (!subscriptionHasStream(ctx, existing, streamPath)) { + await ctx.streamClient.addSubscriptionStreams(subscriptionId, [streamPath]) + } +} + export async function assertDispatchPolicyAllowed( ctx: TenantContext, policy: DispatchPolicy | undefined @@ -220,20 +261,18 @@ async function linkStreamToTargetSubscription( await ctx.streamClient.ensure(wakeStream, { contentType: `application/json`, }) - if (!existing) { - await ctx.streamClient.putSubscription(subscriptionId, { + await ensureSubscriptionIncludesStream( + ctx, + subscriptionId, + streamPath, + { type: `pull-wake`, streams: [streamPath], wake_stream: wakeStream, description: `Electric Agents runner ${target.runnerId}`, - }) - return - } - if (!subscriptionHasStream(ctx, existing, streamPath)) { - await ctx.streamClient.addSubscriptionStreams(subscriptionId, [ - streamPath, - ]) - } + }, + existing + ) return } @@ -249,16 +288,18 @@ async function linkStreamToTargetSubscription( ctx.publicUrl, `/_electric/webhook-forward/${encodeURIComponent(subscriptionId)}` ) - if (!existing) { - await ctx.streamClient.putSubscription(subscriptionId, { + await ensureSubscriptionIncludesStream( + ctx, + subscriptionId, + streamPath, + { type: `webhook`, streams: [streamPath], webhook: { url: forwardUrl }, description: `Electric Agents webhook ${subscriptionId}`, - }) - } else if (!subscriptionHasStream(ctx, existing, streamPath)) { - await ctx.streamClient.addSubscriptionStreams(subscriptionId, [streamPath]) - } + }, + existing + ) await ctx.pgDb .insert(subscriptionWebhooks) .values({ diff --git a/packages/agents-server/test/dispatch-policy-routing.test.ts b/packages/agents-server/test/dispatch-policy-routing.test.ts index 5ce3e0f338..2f54c87540 100644 --- a/packages/agents-server/test/dispatch-policy-routing.test.ts +++ b/packages/agents-server/test/dispatch-policy-routing.test.ts @@ -1,5 +1,6 @@ import { describe, expect, it, vi } from 'vitest' import { globalRouter } from '../src/routing/global-router' +import { DurableStreamsSubscriptionError } from '../src/stream-client' import type { TenantContext } from '../src/routing/context' import type { DispatchPolicy, @@ -328,4 +329,46 @@ describe(`dispatch policy routing`, () => { expect.objectContaining({ payload: `hello` }) ) }) + + it(`treats runner subscription create conflicts as an idempotent relink`, async () => { + const dispatchPolicy: DispatchPolicy = { + targets: [{ type: `runner`, runnerId: `runner-1` }], + } + const ctx = buildContext() + ;(ctx.entityManager.registry.getEntity as any).mockResolvedValue( + entity(dispatchPolicy) + ) + ;(ctx.streamClient.getSubscription as any) + .mockResolvedValueOnce(null) + .mockResolvedValueOnce({ + streams: [{ path: `tenant-test/chat/one/main` }], + }) + ;(ctx.streamClient.putSubscription as any).mockRejectedValueOnce( + new DurableStreamsSubscriptionError( + `Subscription creation failed`, + 409, + JSON.stringify({ + error: { + code: `SUBSCRIPTION_ALREADY_EXISTS`, + message: `Subscription already exists`, + }, + }) + ) + ) + ctx.entityManager.send = vi.fn(async () => undefined) + + const response = await globalRouter.fetch( + request(`POST`, `/_electric/entities/chat/one/send`, { + payload: `hello`, + }), + ctx + ) + + expect(response.status).toBe(204) + expect(ctx.streamClient.addSubscriptionStreams).not.toHaveBeenCalled() + expect(ctx.entityManager.send).toHaveBeenCalledWith( + `/chat/one`, + expect.objectContaining({ payload: `hello` }) + ) + }) }) From 12a03ffdf9463a5393a98fef2dce1a1600c6a16a Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Mon, 18 May 2026 07:59:08 -0600 Subject: [PATCH 22/37] test(agents-server): add Horton send failure diagnostics --- .../test/horton-pull-wake-e2e.test.ts | 115 +++++++++++++++++- 1 file changed, 113 insertions(+), 2 deletions(-) diff --git a/packages/agents-server/test/horton-pull-wake-e2e.test.ts b/packages/agents-server/test/horton-pull-wake-e2e.test.ts index c141ea04f8..fb570ab8ec 100644 --- a/packages/agents-server/test/horton-pull-wake-e2e.test.ts +++ b/packages/agents-server/test/horton-pull-wake-e2e.test.ts @@ -1,3 +1,4 @@ +import { createHash } from 'node:crypto' import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest' import { DurableStreamTestServer } from '@durable-streams/server' import { BuiltinAgentsServer } from '../../agents/src/server' @@ -64,6 +65,99 @@ function eventType(event: any): unknown { return event.type ?? event.value?.type ?? event.value?.value?.type } +function runnerEntitySubscriptionId( + runnerId: string, + entityUrl: string +): string { + const digest = createHash(`sha256`).update(entityUrl).digest(`hex`) + return `runner:${runnerId}:${digest.slice(0, 16)}` +} + +function subscriptionUrl( + streamBaseUrl: string, + subscriptionId: string +): string { + const url = new URL(streamBaseUrl) + const match = /^(.*)\/v1\/stream\/([^/]+)\/?$/.exec(url.pathname) + if (match) { + const [, prefix = ``, serviceId] = match + url.pathname = `${prefix}/v1/stream-meta/subscriptions/${encodeURIComponent(subscriptionId)}` + url.searchParams.set(`service`, decodeURIComponent(serviceId!)) + return url.toString() + } + + url.pathname = `${url.pathname.replace(/\/+$/, ``)}/v1/stream-meta/subscriptions/${encodeURIComponent(subscriptionId)}` + return url.toString() +} + +function truncateDiagnostic(value: string, max = 4_000): string { + return value.length > max ? `${value.slice(0, max)}...` : value +} + +async function responseDiagnostic( + label: string, + input: RequestInfo | URL, + init?: RequestInit +): Promise { + try { + const res = await fetch(input, init) + const body = truncateDiagnostic(await res.text()) + return `${label}: ${res.status} ${res.statusText}\n${body}` + } catch (err) { + return `${label}: fetch failed\n${err instanceof Error ? err.stack : String(err)}` + } +} + +async function expectNoContentWithDiagnostics( + res: Response, + opts: { + phase: string + baseUrl: string + streamBaseUrl: string + entityApiUrl: string + entityUrl: string + runnerId: string + authHeaders: Record + } +): Promise { + if (res.status === 204) return + + const body = truncateDiagnostic(await res.text()) + const subscriptionId = runnerEntitySubscriptionId( + opts.runnerId, + opts.entityUrl + ) + const diagnostics = await Promise.all([ + responseDiagnostic(`entity`, opts.entityApiUrl, { + headers: opts.authHeaders, + }), + responseDiagnostic( + `runner`, + `${opts.baseUrl}/_electric/runners/${opts.runnerId}`, + { + headers: opts.authHeaders, + } + ), + responseDiagnostic( + `runner health`, + `${opts.baseUrl}/_electric/runners/${opts.runnerId}/health`, + { headers: opts.authHeaders } + ), + responseDiagnostic( + `subscription ${subscriptionId}`, + subscriptionUrl(opts.streamBaseUrl, subscriptionId) + ), + ]) + + throw new Error( + [ + `${opts.phase} returned ${res.status} ${res.statusText}; expected 204`, + `response body:\n${body}`, + ...diagnostics, + ].join(`\n\n`) + ) +} + function assertCompleteResponses( events: Array, responseText: string, @@ -149,6 +243,7 @@ describe(`pull-wake Horton e2e with mocked LLM`, () => { it(`dispatches explicit runner-policy wakes and Horton writes mocked responses`, async () => { const id = `pull-wake-horton-${Date.now()}` + const entityUrl = `/horton/${id}` const entityApiUrl = `${baseUrl}/_electric/entities/horton/${id}` const dispatch_policy = { targets: [{ type: `runner`, runnerId }] } @@ -180,7 +275,15 @@ describe(`pull-wake Horton e2e with mocked LLM`, () => { payload: `Please answer via pull-wake.`, }), }) - expect(sendRes.status).toBe(204) + await expectNoContentWithDiagnostics(sendRes, { + phase: `initial send`, + baseUrl, + streamBaseUrl, + entityApiUrl, + entityUrl, + runnerId, + authHeaders, + }) await waitFor(async () => mockStreamFn.mock.calls.length > 0, 20_000, 50) @@ -203,7 +306,15 @@ describe(`pull-wake Horton e2e with mocked LLM`, () => { payload: `Please answer via pull-wake again after idle.`, }), }) - expect(secondSendRes.status).toBe(204) + await expectNoContentWithDiagnostics(secondSendRes, { + phase: `second send`, + baseUrl, + streamBaseUrl, + entityApiUrl, + entityUrl, + runnerId, + authHeaders, + }) await waitFor( async () => mockStreamFn.mock.calls.length > firstCallCount, From c3a7fc61e51095a346f9e043472c7b97dc998511 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Mon, 18 May 2026 08:36:35 -0600 Subject: [PATCH 23/37] fix(agents-server): avoid send-time dispatch relinks --- .../src/routing/entities-router.ts | 2 -- .../test/dispatch-policy-routing.test.ts | 29 ++++++------------- 2 files changed, 9 insertions(+), 22 deletions(-) diff --git a/packages/agents-server/src/routing/entities-router.ts b/packages/agents-server/src/routing/entities-router.ts index 5de7793c33..3e05f559c7 100644 --- a/packages/agents-server/src/routing/entities-router.ts +++ b/packages/agents-server/src/routing/entities-router.ts @@ -533,8 +533,6 @@ async function sendEntity( if (!entity.dispatch_policy) { const updatedEntity = await backfillEntityDispatchPolicy(ctx, entity) await linkEntityDispatchSubscription(ctx, updatedEntity) - } else if (entity.dispatch_policy.targets[0]?.type === `runner`) { - await linkEntityDispatchSubscription(ctx, entity) } if (parsed.afterMs && parsed.afterMs > 0) { diff --git a/packages/agents-server/test/dispatch-policy-routing.test.ts b/packages/agents-server/test/dispatch-policy-routing.test.ts index 2f54c87540..66a2beacda 100644 --- a/packages/agents-server/test/dispatch-policy-routing.test.ts +++ b/packages/agents-server/test/dispatch-policy-routing.test.ts @@ -269,7 +269,7 @@ describe(`dispatch policy routing`, () => { ) }) - it(`relinks existing runner-dispatched entities before sending`, async () => { + it(`does not relink existing runner-dispatched entities before sending`, async () => { const dispatchPolicy: DispatchPolicy = { targets: [{ type: `runner`, runnerId: `runner-1` }], } @@ -287,14 +287,9 @@ describe(`dispatch policy routing`, () => { ) expect(response.status).toBe(204) - expect(ctx.streamClient.putSubscription).toHaveBeenCalledWith( - expect.stringMatching(/^runner:runner-1:/), - expect.objectContaining({ - type: `pull-wake`, - streams: [`/chat/one/main`], - wake_stream: `/runners/runner-1/wake`, - }) - ) + expect(ctx.streamClient.getSubscription).not.toHaveBeenCalled() + expect(ctx.streamClient.putSubscription).not.toHaveBeenCalled() + expect(ctx.streamClient.addSubscriptionStreams).not.toHaveBeenCalled() expect(ctx.entityManager.send).toHaveBeenCalledWith( `/chat/one`, expect.objectContaining({ payload: `hello` }) @@ -322,6 +317,7 @@ describe(`dispatch policy routing`, () => { ) expect(response.status).toBe(204) + expect(ctx.streamClient.getSubscription).not.toHaveBeenCalled() expect(ctx.streamClient.addSubscriptionStreams).not.toHaveBeenCalled() expect(ctx.streamClient.removeSubscriptionStream).not.toHaveBeenCalled() expect(ctx.entityManager.send).toHaveBeenCalledWith( @@ -330,14 +326,11 @@ describe(`dispatch policy routing`, () => { ) }) - it(`treats runner subscription create conflicts as an idempotent relink`, async () => { + it(`treats runner subscription create conflicts as an idempotent spawn link`, async () => { const dispatchPolicy: DispatchPolicy = { targets: [{ type: `runner`, runnerId: `runner-1` }], } const ctx = buildContext() - ;(ctx.entityManager.registry.getEntity as any).mockResolvedValue( - entity(dispatchPolicy) - ) ;(ctx.streamClient.getSubscription as any) .mockResolvedValueOnce(null) .mockResolvedValueOnce({ @@ -358,17 +351,13 @@ describe(`dispatch policy routing`, () => { ctx.entityManager.send = vi.fn(async () => undefined) const response = await globalRouter.fetch( - request(`POST`, `/_electric/entities/chat/one/send`, { - payload: `hello`, + request(`PUT`, `/_electric/entities/chat/one`, { + dispatch_policy: dispatchPolicy, }), ctx ) - expect(response.status).toBe(204) + expect(response.status).toBe(201) expect(ctx.streamClient.addSubscriptionStreams).not.toHaveBeenCalled() - expect(ctx.entityManager.send).toHaveBeenCalledWith( - `/chat/one`, - expect.objectContaining({ payload: `hello` }) - ) }) }) From f6c4d41bf948471c5a1c2ca10cd51ee38eb4159a Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Mon, 18 May 2026 09:17:31 -0600 Subject: [PATCH 24/37] fix(agents): recover pull-wake dispatch races --- .../agents-runtime/src/pull-wake-runner.ts | 2 +- .../test/pull-wake-runner.test.ts | 25 +++++++++ .../src/routing/dispatch-policy.ts | 31 ++++++++++- .../src/routing/entities-router.ts | 5 ++ .../src/routing/runners-router.ts | 43 +++++++++++++++- .../test/dispatch-policy-routing.test.ts | 13 +++++ .../agents-server/test/runners-router.test.ts | 51 +++++++++++++++++++ 7 files changed, 167 insertions(+), 3 deletions(-) diff --git a/packages/agents-runtime/src/pull-wake-runner.ts b/packages/agents-runtime/src/pull-wake-runner.ts index 6f34d7db45..82f6a9958b 100644 --- a/packages/agents-runtime/src/pull-wake-runner.ts +++ b/packages/agents-runtime/src/pull-wake-runner.ts @@ -109,7 +109,7 @@ export function createPullWakeRunner( let response: PullWakeStreamResponse | null = null let heartbeatTimer: ReturnType | null = null let eventHeartbeatTimer: ReturnType | null = null - let currentOffset = config.offset + let currentOffset = config.offset ?? `-1` let startedAt: string | null = null let streamConnected = false let streamConnectedSince: string | null = null diff --git a/packages/agents-runtime/test/pull-wake-runner.test.ts b/packages/agents-runtime/test/pull-wake-runner.test.ts index 95ac427dac..194190a5eb 100644 --- a/packages/agents-runtime/test/pull-wake-runner.test.ts +++ b/packages/agents-runtime/test/pull-wake-runner.test.ts @@ -91,6 +91,31 @@ describe(`createPullWakeRunner`, () => { vi.unstubAllGlobals() }) + it(`starts from the beginning when no wake stream offset is committed`, async () => { + const streamFactory = vi.fn(async () => ({ + offset: `42`, + async *jsonStream() {}, + closed: Promise.resolve(), + })) + + const runner = createPullWakeRunner({ + baseUrl: `http://server`, + runnerId: `runner-1`, + runtime: runtime(), + heartbeatIntervalMs: 0, + streamFactory, + }) + + runner.start() + await waitFor(() => { + expect(streamFactory).toHaveBeenCalledWith( + expect.objectContaining({ offset: `-1` }) + ) + }) + + await runner.stop() + }) + it(`claims compact DS wake events before dispatching runtime wakes`, async () => { const event = wakeEvent(`one`) const claimed = notification(`one`) diff --git a/packages/agents-server/src/routing/dispatch-policy.ts b/packages/agents-server/src/routing/dispatch-policy.ts index cd8e8799ed..016e5ee837 100644 --- a/packages/agents-server/src/routing/dispatch-policy.ts +++ b/packages/agents-server/src/routing/dispatch-policy.ts @@ -28,7 +28,7 @@ export function subscriptionIdForDispatchTarget( return `webhook:${digest.slice(0, 16)}` } -function subscriptionIdForEntityDispatchTarget( +export function subscriptionIdForEntityDispatchTarget( target: DispatchTarget, entityUrl: string ): string { @@ -211,6 +211,35 @@ export async function linkEntityDispatchSubscription( await linkStreamToTargetSubscription(ctx, target, entity) } +export async function notifyEntityDispatchWake( + ctx: TenantContext, + entity: ElectricAgentsEntity +): Promise { + const dispatchPolicy = await resolveEffectiveDispatchPolicyForEntity( + ctx, + entity + ) + const target = dispatchPolicy?.targets[0] + if (!target || target.type !== `runner`) return + + const runner = await ctx.entityManager.registry.getRunner(target.runnerId) + if (!runner) return + + await ctx.streamClient.append( + runner.wake_stream || runnerWakeStream(target.runnerId), + JSON.stringify({ + type: `wake`, + subscription_id: subscriptionIdForEntityDispatchTarget( + target, + entity.url + ), + stream: entity.streams.main.replace(/^\/+/, ``), + generation: Date.now(), + ts: Date.now(), + }) + ) +} + export async function unlinkEntityDispatchSubscription( ctx: TenantContext, entity: ElectricAgentsEntity diff --git a/packages/agents-server/src/routing/entities-router.ts b/packages/agents-server/src/routing/entities-router.ts index 3e05f559c7..a39dc6f087 100644 --- a/packages/agents-server/src/routing/entities-router.ts +++ b/packages/agents-server/src/routing/entities-router.ts @@ -17,6 +17,7 @@ import { assertDispatchPolicyAllowed, backfillEntityDispatchPolicy, linkEntityDispatchSubscription, + notifyEntityDispatchWake, resolveEffectiveDispatchPolicyForSpawn, unlinkEntityDispatchSubscription, } from './dispatch-policy.js' @@ -529,10 +530,12 @@ async function sendEntity( } await ctx.entityManager.ensurePrincipal(principal) const { entityUrl, entity } = requireExistingEntityRoute(request) + let dispatchEntity = entity if (!entity.dispatch_policy) { const updatedEntity = await backfillEntityDispatchPolicy(ctx, entity) await linkEntityDispatchSubscription(ctx, updatedEntity) + dispatchEntity = updatedEntity } if (parsed.afterMs && parsed.afterMs > 0) { @@ -557,6 +560,7 @@ async function sendEntity( mode: parsed.mode, position: parsed.position, }) + await notifyEntityDispatchWake(ctx, dispatchEntity) } return status(204) @@ -620,6 +624,7 @@ async function spawnEntity( from: principal.url, payload: parsed.initialMessage, }) + await notifyEntityDispatchWake(ctx, entity) } return json( diff --git a/packages/agents-server/src/routing/runners-router.ts b/packages/agents-server/src/routing/runners-router.ts index c7386a01a8..d31a3abcf9 100644 --- a/packages/agents-server/src/routing/runners-router.ts +++ b/packages/agents-server/src/routing/runners-router.ts @@ -1,3 +1,4 @@ +import { randomUUID } from 'node:crypto' import { appendPathToUrl } from '@electric-ax/agents-runtime' import { Type, type Static } from '@sinclair/typebox' import { Router, json, status } from 'itty-router' @@ -389,15 +390,55 @@ async function claimWake( } if (!claim) return status(204) + const effectiveClaim = isSubscriptionClaimResponse(claim) + ? claim + : fallbackClaimFromWakeEvent(parsed) + if (!effectiveClaim) return status(204) + const notification = await notificationFromClaim(ctx, { runnerId, runnerWakeStream: runner.wake_stream, subscriptionId, - claim, + claim: effectiveClaim, }) return json(notification) } +function isSubscriptionClaimResponse( + claim: SubscriptionClaimResponse +): boolean { + return ( + typeof claim.wake_id === `string` && + typeof claim.generation === `number` && + typeof claim.token === `string` && + Array.isArray(claim.streams) + ) +} + +function fallbackClaimFromWakeEvent( + event: ClaimBody +): SubscriptionClaimResponse | null { + if (typeof event.stream !== `string` || event.stream.trim().length === 0) { + return null + } + const generation = + typeof event.generation === `number` && Number.isFinite(event.generation) + ? event.generation + : Date.now() + return { + wake_id: `fallback-${randomUUID()}`, + generation, + token: randomUUID(), + streams: [ + { + path: event.stream, + tail_offset: `-1`, + has_pending: true, + }, + ], + } +} + function isExpectedClaimConflict( err: unknown ): err is DurableStreamsSubscriptionError { diff --git a/packages/agents-server/test/dispatch-policy-routing.test.ts b/packages/agents-server/test/dispatch-policy-routing.test.ts index 66a2beacda..a8fd31a45f 100644 --- a/packages/agents-server/test/dispatch-policy-routing.test.ts +++ b/packages/agents-server/test/dispatch-policy-routing.test.ts @@ -90,6 +90,7 @@ function buildContext(overrides: Partial = {}): TenantContext { addSubscriptionStreams: vi.fn(async () => ({})), removeSubscriptionStream: vi.fn(async () => ({})), ensure: vi.fn(async () => undefined), + append: vi.fn(async () => ({ offset: `1` })), } as any, runtime: undefined as any, entityBridgeManager: undefined as any, @@ -267,6 +268,18 @@ describe(`dispatch policy routing`, () => { `/chat/one`, expect.objectContaining({ payload: `hello` }) ) + expect(ctx.streamClient.append).toHaveBeenCalledWith( + `/runners/runner-1/wake`, + expect.any(String) + ) + const wakeEvent = JSON.parse( + vi.mocked(ctx.streamClient.append).mock.calls[0]![1] as string + ) + expect(wakeEvent).toMatchObject({ + type: `wake`, + subscription_id: expect.stringMatching(/^runner:runner-1:/), + stream: `chat/one/main`, + }) }) it(`does not relink existing runner-dispatched entities before sending`, async () => { diff --git a/packages/agents-server/test/runners-router.test.ts b/packages/agents-server/test/runners-router.test.ts index 4e23641e02..e7292c89f9 100644 --- a/packages/agents-server/test/runners-router.test.ts +++ b/packages/agents-server/test/runners-router.test.ts @@ -493,6 +493,57 @@ describe(`runner routes`, () => { ) }) + it(`falls back to wake event details when subscription claims are unavailable`, async () => { + const ctx = buildContext({ + principal: { + kind: `user`, + id: `owner@example.com`, + key: `user:owner@example.com`, + url: `/principal/user%3Aowner%40example.com`, + }, + }) + vi.mocked(ctx.streamClient.claimSubscription).mockResolvedValue({} as any) + vi.mocked(ctx.entityManager.registry.getEntityByStream).mockResolvedValue({ + url: `/chat/one`, + type: `chat`, + status: `idle`, + streams: { main: `/chat/one/main`, error: `/chat/one/error` }, + subscription_id: `runner:runner-1`, + write_token: `entity-token`, + tags: {}, + created_at: 1, + updated_at: 1, + }) + + const response = await globalRouter.fetch( + request(`POST`, `/_electric/runners/runner-1/claim`, { + subscription_id: `runner:runner-1:fallback`, + stream: `chat/one/main`, + generation: 7, + }), + ctx + ) + + expect(response.status).toBe(200) + const body = (await response.json()) as Record + expect(body).toMatchObject({ + consumerId: expect.stringMatching(/^fallback-/), + epoch: 7, + wakeId: expect.stringMatching(/^fallback-/), + streamPath: `/chat/one/main`, + callback: expect.stringMatching( + /^http:\/\/server\/_electric\/callback-forward\/fallback-/ + ), + claimToken: expect.any(String), + }) + expect(body.streams).toEqual([{ path: `/chat/one/main`, offset: `-1` }]) + expect(ctx.entityManager.registry.materializeActiveClaim).toHaveBeenCalled() + expect(ctx.entityManager.registry.updateStatus).toHaveBeenCalledWith( + `/chat/one`, + `running` + ) + }) + it(`rejects invalid owner_principal with 400`, async () => { const response = await globalRouter.fetch( request(`POST`, `/_electric/runners`, { From 9f0a2ede42f87b21d322ef3a71d0194175aac5dc Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Mon, 18 May 2026 10:09:24 -0600 Subject: [PATCH 25/37] test(agents-server): cover pull-wake subscription stack --- .../src/routing/dispatch-policy.ts | 31 +--- .../src/routing/entities-router.ts | 5 - .../src/routing/runners-router.ts | 43 +---- .../test/dispatch-policy-routing.test.ts | 13 -- .../test/horton-pull-wake-e2e.test.ts | 83 ++++++++- .../test/pull-wake-subscription-stack.test.ts | 161 ++++++++++++++++++ .../agents-server/test/runners-router.test.ts | 51 ------ 7 files changed, 241 insertions(+), 146 deletions(-) create mode 100644 packages/agents-server/test/pull-wake-subscription-stack.test.ts diff --git a/packages/agents-server/src/routing/dispatch-policy.ts b/packages/agents-server/src/routing/dispatch-policy.ts index 016e5ee837..cd8e8799ed 100644 --- a/packages/agents-server/src/routing/dispatch-policy.ts +++ b/packages/agents-server/src/routing/dispatch-policy.ts @@ -28,7 +28,7 @@ export function subscriptionIdForDispatchTarget( return `webhook:${digest.slice(0, 16)}` } -export function subscriptionIdForEntityDispatchTarget( +function subscriptionIdForEntityDispatchTarget( target: DispatchTarget, entityUrl: string ): string { @@ -211,35 +211,6 @@ export async function linkEntityDispatchSubscription( await linkStreamToTargetSubscription(ctx, target, entity) } -export async function notifyEntityDispatchWake( - ctx: TenantContext, - entity: ElectricAgentsEntity -): Promise { - const dispatchPolicy = await resolveEffectiveDispatchPolicyForEntity( - ctx, - entity - ) - const target = dispatchPolicy?.targets[0] - if (!target || target.type !== `runner`) return - - const runner = await ctx.entityManager.registry.getRunner(target.runnerId) - if (!runner) return - - await ctx.streamClient.append( - runner.wake_stream || runnerWakeStream(target.runnerId), - JSON.stringify({ - type: `wake`, - subscription_id: subscriptionIdForEntityDispatchTarget( - target, - entity.url - ), - stream: entity.streams.main.replace(/^\/+/, ``), - generation: Date.now(), - ts: Date.now(), - }) - ) -} - export async function unlinkEntityDispatchSubscription( ctx: TenantContext, entity: ElectricAgentsEntity diff --git a/packages/agents-server/src/routing/entities-router.ts b/packages/agents-server/src/routing/entities-router.ts index a39dc6f087..3e05f559c7 100644 --- a/packages/agents-server/src/routing/entities-router.ts +++ b/packages/agents-server/src/routing/entities-router.ts @@ -17,7 +17,6 @@ import { assertDispatchPolicyAllowed, backfillEntityDispatchPolicy, linkEntityDispatchSubscription, - notifyEntityDispatchWake, resolveEffectiveDispatchPolicyForSpawn, unlinkEntityDispatchSubscription, } from './dispatch-policy.js' @@ -530,12 +529,10 @@ async function sendEntity( } await ctx.entityManager.ensurePrincipal(principal) const { entityUrl, entity } = requireExistingEntityRoute(request) - let dispatchEntity = entity if (!entity.dispatch_policy) { const updatedEntity = await backfillEntityDispatchPolicy(ctx, entity) await linkEntityDispatchSubscription(ctx, updatedEntity) - dispatchEntity = updatedEntity } if (parsed.afterMs && parsed.afterMs > 0) { @@ -560,7 +557,6 @@ async function sendEntity( mode: parsed.mode, position: parsed.position, }) - await notifyEntityDispatchWake(ctx, dispatchEntity) } return status(204) @@ -624,7 +620,6 @@ async function spawnEntity( from: principal.url, payload: parsed.initialMessage, }) - await notifyEntityDispatchWake(ctx, entity) } return json( diff --git a/packages/agents-server/src/routing/runners-router.ts b/packages/agents-server/src/routing/runners-router.ts index d31a3abcf9..c7386a01a8 100644 --- a/packages/agents-server/src/routing/runners-router.ts +++ b/packages/agents-server/src/routing/runners-router.ts @@ -1,4 +1,3 @@ -import { randomUUID } from 'node:crypto' import { appendPathToUrl } from '@electric-ax/agents-runtime' import { Type, type Static } from '@sinclair/typebox' import { Router, json, status } from 'itty-router' @@ -390,55 +389,15 @@ async function claimWake( } if (!claim) return status(204) - const effectiveClaim = isSubscriptionClaimResponse(claim) - ? claim - : fallbackClaimFromWakeEvent(parsed) - if (!effectiveClaim) return status(204) - const notification = await notificationFromClaim(ctx, { runnerId, runnerWakeStream: runner.wake_stream, subscriptionId, - claim: effectiveClaim, + claim, }) return json(notification) } -function isSubscriptionClaimResponse( - claim: SubscriptionClaimResponse -): boolean { - return ( - typeof claim.wake_id === `string` && - typeof claim.generation === `number` && - typeof claim.token === `string` && - Array.isArray(claim.streams) - ) -} - -function fallbackClaimFromWakeEvent( - event: ClaimBody -): SubscriptionClaimResponse | null { - if (typeof event.stream !== `string` || event.stream.trim().length === 0) { - return null - } - const generation = - typeof event.generation === `number` && Number.isFinite(event.generation) - ? event.generation - : Date.now() - return { - wake_id: `fallback-${randomUUID()}`, - generation, - token: randomUUID(), - streams: [ - { - path: event.stream, - tail_offset: `-1`, - has_pending: true, - }, - ], - } -} - function isExpectedClaimConflict( err: unknown ): err is DurableStreamsSubscriptionError { diff --git a/packages/agents-server/test/dispatch-policy-routing.test.ts b/packages/agents-server/test/dispatch-policy-routing.test.ts index a8fd31a45f..66a2beacda 100644 --- a/packages/agents-server/test/dispatch-policy-routing.test.ts +++ b/packages/agents-server/test/dispatch-policy-routing.test.ts @@ -90,7 +90,6 @@ function buildContext(overrides: Partial = {}): TenantContext { addSubscriptionStreams: vi.fn(async () => ({})), removeSubscriptionStream: vi.fn(async () => ({})), ensure: vi.fn(async () => undefined), - append: vi.fn(async () => ({ offset: `1` })), } as any, runtime: undefined as any, entityBridgeManager: undefined as any, @@ -268,18 +267,6 @@ describe(`dispatch policy routing`, () => { `/chat/one`, expect.objectContaining({ payload: `hello` }) ) - expect(ctx.streamClient.append).toHaveBeenCalledWith( - `/runners/runner-1/wake`, - expect.any(String) - ) - const wakeEvent = JSON.parse( - vi.mocked(ctx.streamClient.append).mock.calls[0]![1] as string - ) - expect(wakeEvent).toMatchObject({ - type: `wake`, - subscription_id: expect.stringMatching(/^runner:runner-1:/), - stream: `chat/one/main`, - }) }) it(`does not relink existing runner-dispatched entities before sending`, async () => { diff --git a/packages/agents-server/test/horton-pull-wake-e2e.test.ts b/packages/agents-server/test/horton-pull-wake-e2e.test.ts index fb570ab8ec..f806bf9512 100644 --- a/packages/agents-server/test/horton-pull-wake-e2e.test.ts +++ b/packages/agents-server/test/horton-pull-wake-e2e.test.ts @@ -158,6 +158,59 @@ async function expectNoContentWithDiagnostics( ) } +async function waitForMockCallWithDiagnostics( + predicate: () => boolean, + opts: { + phase: string + baseUrl: string + streamBaseUrl: string + entityApiUrl: string + entityUrl: string + entityStream: string + runnerId: string + authHeaders: Record + } +): Promise { + try { + await waitFor(async () => predicate(), 20_000, 50) + } catch (err) { + const subscriptionId = runnerEntitySubscriptionId( + opts.runnerId, + opts.entityUrl + ) + const diagnostics = await Promise.all([ + responseDiagnostic(`entity`, opts.entityApiUrl, { + headers: opts.authHeaders, + }), + responseDiagnostic( + `runner health`, + `${opts.baseUrl}/_electric/runners/${opts.runnerId}/health`, + { headers: opts.authHeaders } + ), + responseDiagnostic( + `subscription ${subscriptionId}`, + subscriptionUrl(opts.streamBaseUrl, subscriptionId) + ), + responseDiagnostic( + `runner wake stream`, + `${opts.streamBaseUrl}/runners/${opts.runnerId}/wake?offset=-1&live=false` + ), + responseDiagnostic( + `entity main stream`, + `${opts.streamBaseUrl}${opts.entityStream}?offset=-1&live=false` + ), + ]) + + throw new Error( + [ + `${opts.phase} did not reach Horton within 20000ms`, + err instanceof Error ? err.message : String(err), + ...diagnostics, + ].join(`\n\n`) + ) + } +} + function assertCompleteResponses( events: Array, responseText: string, @@ -285,7 +338,19 @@ describe(`pull-wake Horton e2e with mocked LLM`, () => { authHeaders, }) - await waitFor(async () => mockStreamFn.mock.calls.length > 0, 20_000, 50) + await waitForMockCallWithDiagnostics( + () => mockStreamFn.mock.calls.length > 0, + { + phase: `initial send`, + baseUrl, + streamBaseUrl, + entityApiUrl, + entityUrl, + entityStream: spawned.streams.main, + runnerId, + authHeaders, + } + ) await waitFor(async () => { const events = await readStreamEvents(streamBaseUrl, spawned.streams.main) @@ -316,10 +381,18 @@ describe(`pull-wake Horton e2e with mocked LLM`, () => { authHeaders, }) - await waitFor( - async () => mockStreamFn.mock.calls.length > firstCallCount, - 20_000, - 50 + await waitForMockCallWithDiagnostics( + () => mockStreamFn.mock.calls.length > firstCallCount, + { + phase: `second send`, + baseUrl, + streamBaseUrl, + entityApiUrl, + entityUrl, + entityStream: spawned.streams.main, + runnerId, + authHeaders, + } ) await waitFor(async () => { diff --git a/packages/agents-server/test/pull-wake-subscription-stack.test.ts b/packages/agents-server/test/pull-wake-subscription-stack.test.ts new file mode 100644 index 0000000000..dc51929efb --- /dev/null +++ b/packages/agents-server/test/pull-wake-subscription-stack.test.ts @@ -0,0 +1,161 @@ +import { createServer } from 'node:http' +import { createServerAdapter } from '@whatwg-node/server' +import { stream } from '@durable-streams/client' +import { DurableStreamTestServer } from '@durable-streams/server' +import { afterEach, describe, expect, it } from 'vitest' +import { globalRouter } from '../src/routing/global-router' +import { StreamClient, durableStreamsServiceUrl } from '../src/stream-client' +import type { Server } from 'node:http' +import type { TenantContext } from '../src/routing/context' + +async function closeServer(server: Server): Promise { + await new Promise((resolve, reject) => { + server.close((err) => { + if (err) reject(err) + else resolve() + }) + }) +} + +async function readJsonStream( + baseUrl: string, + path: string +): Promise> { + const res = await stream({ + url: `${baseUrl}${path}`, + offset: `-1`, + live: false, + }) + return await res.json() +} + +describe(`pull-wake subscription stack`, () => { + let dsServer: DurableStreamTestServer | undefined + let proxyServer: Server | undefined + + afterEach(async () => { + await Promise.allSettled([ + proxyServer ? closeServer(proxyServer) : undefined, + dsServer?.stop(), + ]) + proxyServer = undefined + dsServer = undefined + }) + + it(`emits and claims runner wakes through Durable Streams subscriptions`, async () => { + dsServer = new DurableStreamTestServer({ + port: 0, + longPollTimeout: 100, + webhooks: true, + }) + await dsServer.start() + const streamBaseUrl = durableStreamsServiceUrl(dsServer.url, `default`) + const client = new StreamClient(streamBaseUrl) + + await client.ensure(`/runners/runner-1/wake`, { + contentType: `application/json`, + }) + await client.ensure(`/horton/one/main`, { + contentType: `application/json`, + }) + await client.putSubscription(`runner:runner-1:one`, { + type: `pull-wake`, + streams: [`/horton/one/main`], + wake_stream: `/runners/runner-1/wake`, + }) + + await client.append( + `/horton/one/main`, + JSON.stringify({ type: `message`, value: `hello` }) + ) + + const wakes = await readJsonStream>( + streamBaseUrl, + `/runners/runner-1/wake` + ) + expect(wakes).toEqual([ + expect.objectContaining({ + type: `wake`, + subscription_id: `runner:runner-1:one`, + stream: `default/horton/one/main`, + generation: 1, + }), + ]) + + await expect( + client.claimSubscription(`runner:runner-1:one`, `worker-1`) + ).resolves.toMatchObject({ + wake_id: expect.any(String), + generation: 1, + token: expect.any(String), + streams: [ + expect.objectContaining({ + path: `horton/one/main`, + has_pending: true, + }), + ], + }) + }) + + it(`proxies pre-existing runner wake events to pull-wake runners`, async () => { + dsServer = new DurableStreamTestServer({ + port: 0, + longPollTimeout: 100, + webhooks: true, + }) + await dsServer.start() + const streamBaseUrl = durableStreamsServiceUrl(dsServer.url, `default`) + const client = new StreamClient(streamBaseUrl) + await client.ensure(`/runners/runner-1/wake`, { + contentType: `application/json`, + }) + await client.append( + `/runners/runner-1/wake`, + JSON.stringify({ + type: `wake`, + subscription_id: `runner:runner-1:one`, + stream: `horton/one/main`, + generation: 1, + }) + ) + + const ctx = { + service: `default`, + principal: { + kind: `user`, + id: `owner@example.com`, + key: `user:owner@example.com`, + url: `/principal/user%3Aowner%40example.com`, + }, + publicUrl: `http://agents.local`, + durableStreamsUrl: dsServer.url, + entityBridgeManager: { + beginClientRead: async () => null, + touchByStreamPath: async () => undefined, + }, + isShuttingDown: () => false, + } as unknown as TenantContext + const adapter = createServerAdapter((request) => + globalRouter.fetch(request as any, ctx) + ) + proxyServer = createServer(adapter) + await new Promise((resolve) => + proxyServer!.listen(0, `127.0.0.1`, resolve) + ) + const address = proxyServer.address() + if (!address || typeof address === `string`) { + throw new Error(`Expected TCP test server address`) + } + + const wakes = await readJsonStream>( + `http://127.0.0.1:${address.port}`, + `/runners/runner-1/wake` + ) + expect(wakes).toEqual([ + expect.objectContaining({ + type: `wake`, + subscription_id: `runner:runner-1:one`, + }), + ]) + }) +}) diff --git a/packages/agents-server/test/runners-router.test.ts b/packages/agents-server/test/runners-router.test.ts index e7292c89f9..4e23641e02 100644 --- a/packages/agents-server/test/runners-router.test.ts +++ b/packages/agents-server/test/runners-router.test.ts @@ -493,57 +493,6 @@ describe(`runner routes`, () => { ) }) - it(`falls back to wake event details when subscription claims are unavailable`, async () => { - const ctx = buildContext({ - principal: { - kind: `user`, - id: `owner@example.com`, - key: `user:owner@example.com`, - url: `/principal/user%3Aowner%40example.com`, - }, - }) - vi.mocked(ctx.streamClient.claimSubscription).mockResolvedValue({} as any) - vi.mocked(ctx.entityManager.registry.getEntityByStream).mockResolvedValue({ - url: `/chat/one`, - type: `chat`, - status: `idle`, - streams: { main: `/chat/one/main`, error: `/chat/one/error` }, - subscription_id: `runner:runner-1`, - write_token: `entity-token`, - tags: {}, - created_at: 1, - updated_at: 1, - }) - - const response = await globalRouter.fetch( - request(`POST`, `/_electric/runners/runner-1/claim`, { - subscription_id: `runner:runner-1:fallback`, - stream: `chat/one/main`, - generation: 7, - }), - ctx - ) - - expect(response.status).toBe(200) - const body = (await response.json()) as Record - expect(body).toMatchObject({ - consumerId: expect.stringMatching(/^fallback-/), - epoch: 7, - wakeId: expect.stringMatching(/^fallback-/), - streamPath: `/chat/one/main`, - callback: expect.stringMatching( - /^http:\/\/server\/_electric\/callback-forward\/fallback-/ - ), - claimToken: expect.any(String), - }) - expect(body.streams).toEqual([{ path: `/chat/one/main`, offset: `-1` }]) - expect(ctx.entityManager.registry.materializeActiveClaim).toHaveBeenCalled() - expect(ctx.entityManager.registry.updateStatus).toHaveBeenCalledWith( - `/chat/one`, - `running` - ) - }) - it(`rejects invalid owner_principal with 400`, async () => { const response = await globalRouter.fetch( request(`POST`, `/_electric/runners`, { From 696607709d36bf2e101aa4d3fadb958f27502fb8 Mon Sep 17 00:00:00 2001 From: Ilia Borovitinov Date: Mon, 18 May 2026 20:13:49 +0300 Subject: [PATCH 26/37] fix: restore service-scoped pull-wake subscriptions --- .changeset/harden-pull-wake-runner.md | 2 +- .../2026-05-16-pull-wake-health-check.md | 1685 ----------------- ...026-05-16-pull-wake-health-check-design.md | 214 --- packages/agents-server/src/stream-client.ts | 29 +- .../test/horton-pull-wake-e2e.test.ts | 4 +- .../test/pull-wake-subscription-stack.test.ts | 2 +- .../agents-server/test/stream-client.test.ts | 4 +- 7 files changed, 33 insertions(+), 1907 deletions(-) delete mode 100644 packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md delete mode 100644 packages/agents-server/docs/superpowers/specs/2026-05-16-pull-wake-health-check-design.md diff --git a/.changeset/harden-pull-wake-runner.md b/.changeset/harden-pull-wake-runner.md index addda03d10..04888c3b5a 100644 --- a/.changeset/harden-pull-wake-runner.md +++ b/.changeset/harden-pull-wake-runner.md @@ -4,4 +4,4 @@ '@electric-ax/agents': patch --- -Harden pull-wake runner lifecycle with a state machine, concurrent claim limits (`maxConcurrentClaims`), heartbeat-driven stream resets, and exponential reconnect backoff (1s–30s). Add granular `status` field to `PullWakeRunnerHealth` (`stopped | starting | connecting | streaming | reconnecting | stopping`). The `onError` callback is now reporting-only (`(Error) => void`) — it can no longer control runner lifecycle. `stop()` rethrows `drainWakes` errors so callers observe wake handler failures. Event-driven heartbeat throttling avoids stale diagnostics between fixed-interval heartbeats. +Harden pull-wake runner lifecycle with a state machine, concurrent claim limits (`maxConcurrentClaims`), heartbeat-driven stream resets, and exponential reconnect backoff (1s–30s). Add granular `status` field to `PullWakeRunnerHealth` (`stopped | starting | connecting | streaming | reconnecting | stopping`). The `onError` callback is now reporting-only (`(Error) => void`) — it can no longer control runner lifecycle. `stop()` rethrows `drainWakes` errors so callers observe wake handler failures. Event-driven heartbeat throttling avoids stale diagnostics between fixed-interval heartbeats. Service-scoped Durable Streams clients now route subscription control through `__ds` while preserving tenant-prefixed stream names, so pull-wake subscriptions emit runner wake events correctly. diff --git a/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md b/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md deleted file mode 100644 index a1cf1e3bbd..0000000000 --- a/packages/agents-server/docs/superpowers/plans/2026-05-16-pull-wake-health-check.md +++ /dev/null @@ -1,1685 +0,0 @@ -# Pull-Wake Runner Health Check Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]` / `- [x]`) syntax for tracking. - -**Goal:** Add comprehensive diagnostics to the pull-wake runner system: client-side state tracking reported via heartbeats, server-side storage + aggregation, and a `GET /_electric/runners/:id/health` endpoint. Also rename `owner_user_id` → `owner_principal` throughout the runners system, storing principal URLs instead of keys. - -**Architecture:** Three layers — (1) `PullWakeRunner` tracks 16 diagnostic fields internally and reports them to the server in each heartbeat, (2) the server stores client diagnostics in a new `diagnostics` JSONB column on the `runners` table, (3) a new health endpoint aggregates runner state, client diagnostics, active claims, and dispatch stats into a single response with derived health status. - -**Tech Stack:** TypeScript, Drizzle ORM, itty-router, Vitest, PostgreSQL - ---- - -### Task 1: Migration — rename `owner_user_id` and add `diagnostics` column - -**Files:** - -- Create: `packages/agents-server/drizzle/0007_runner_diagnostics_and_principal.sql` - -- [x] **Step 1: Write the migration SQL** - -Existing `owner_user_id` values are key-form strings (e.g., `dev-local`). The new column expects principal URLs (e.g., `/principal/system%3Adev-local`). Since we have no backwards compatibility, the migration deletes existing runner rows — runners are ephemeral and will re-register on next startup. Must also clean up dependent tables (`consumer_claims` and `entity_dispatch_state`) since there are no FK constraints to cascade the deletes. - -```sql -UPDATE consumer_claims SET status = 'expired', updated_at = NOW() WHERE status = 'active' AND runner_id IS NOT NULL; ---> statement-breakpoint -UPDATE entity_dispatch_state SET active_runner_id = NULL, active_consumer_id = NULL, active_epoch = NULL, active_claimed_at = NULL, active_lease_expires_at = NULL, updated_at = NOW() WHERE active_runner_id IS NOT NULL; ---> statement-breakpoint -DELETE FROM runners; ---> statement-breakpoint -ALTER TABLE runners RENAME COLUMN owner_user_id TO owner_principal; ---> statement-breakpoint -DROP INDEX IF EXISTS idx_runners_owner_user_id; ---> statement-breakpoint -CREATE INDEX idx_runners_owner_principal ON runners (tenant_id, owner_principal); ---> statement-breakpoint -ALTER TABLE runners ADD COLUMN diagnostics jsonb; -``` - -- [x] **Step 2: Commit** - -```bash -git add packages/agents-server/drizzle/0007_runner_diagnostics_and_principal.sql -git commit -m "feat(agents-server): add migration for runner diagnostics and principal rename" -``` - ---- - -### Task 2: Update Drizzle schema and types for principal rename + diagnostics - -**Files:** - -- Modify: `packages/agents-server/src/db/schema.ts:104-144` -- Modify: `packages/agents-server/src/electric-agents-types.ts:99-136` - -- [x] **Step 1: Update the `runners` table in Drizzle schema** - -In `packages/agents-server/src/db/schema.ts`, change the `runners` table definition: - -```ts -// In the runners column definitions (line 109): -// REPLACE: - ownerUserId: text(`owner_user_id`).notNull(), -// WITH: - ownerPrincipal: text(`owner_principal`).notNull(), - -// After livenessLeaseExpiresAt (line 118), ADD: - diagnostics: jsonb(`diagnostics`), - -// In the table constraints (line 129): -// REPLACE: - index(`idx_runners_owner_user_id`).on(table.tenantId, table.ownerUserId), -// WITH: - index(`idx_runners_owner_principal`).on(table.tenantId, table.ownerPrincipal), -``` - -- [x] **Step 2: Update the `ElectricAgentsRunner` type** - -In `packages/agents-server/src/electric-agents-types.ts`, update the runner types: - -```ts -// In ElectricAgentsRunner (line 106-120): -// REPLACE: -export interface ElectricAgentsRunner { - id: string - owner_user_id: string - label: string - kind: RunnerKind - admin_status: RunnerAdminStatus - liveness?: RunnerLiveness - last_seen_at?: string - liveness_lease_expires_at?: string - active_claims?: Array - wake_stream: string - wake_stream_offset?: string - created_at: string - updated_at: string -} -// WITH: -export interface ElectricAgentsRunner { - id: string - owner_principal: string - label: string - kind: RunnerKind - admin_status: RunnerAdminStatus - liveness?: RunnerLiveness - last_seen_at?: string - liveness_lease_expires_at?: string - active_claims?: Array - wake_stream: string - wake_stream_offset?: string - diagnostics?: Record - created_at: string - updated_at: string -} - -// In RegisterRunnerRequest (line 122-129): -// REPLACE: -export interface RegisterRunnerRequest { - id: string - owner_user_id: string - label: string - kind?: RunnerKind - admin_status?: RunnerAdminStatus - wake_stream?: string -} -// WITH: -export interface RegisterRunnerRequest { - id: string - owner_principal: string - label: string - kind?: RunnerKind - admin_status?: RunnerAdminStatus - wake_stream?: string -} -``` - -- [x] **Step 3: Add `RunnerHealthResponse` and `RunnerHealthStatus` types** - -Append to `packages/agents-server/src/electric-agents-types.ts`: - -```ts -export type RunnerHealthStatus = `healthy` | `degraded` | `unhealthy` - -export interface RunnerHealthResponse { - runner: { - id: string - admin_status: RunnerAdminStatus - liveness_status: RunnerLiveness | `expired` - lease_expires_at: string | null - lease_remaining_ms: number | null - wake_stream: string - wake_stream_offset: string | null - last_seen_at: string | null - created_at: string - } - client: Record | null - claims: { - active_count: number - active: Array<{ - consumer_id: string - epoch: number - entity_url: string - stream_path: string - claimed_at: string - last_heartbeat_at: string | null - lease_expires_at: string | null - }> - } - dispatch: { - entities_with_active_claim: number - entities_with_outstanding_wake: number - entities_with_pending_work: number - } - health: { - status: RunnerHealthStatus - issues: Array - } -} -``` - -- [x] **Step 4: Commit** - -```bash -git add packages/agents-server/src/db/schema.ts packages/agents-server/src/electric-agents-types.ts -git commit -m "feat(agents-server): rename owner_user_id to owner_principal in schema and types, add diagnostics" -``` - ---- - -### Task 3: Update entity registry — principal rename, diagnostics storage, health queries - -**Files:** - -- Modify: `packages/agents-server/src/entity-registry.ts:74-81, 132-190, 193-217, 1148-1168` - -- [x] **Step 1: Rename `RegisterRunnerInput.ownerUserId` → `ownerPrincipal`** - -In `packages/agents-server/src/entity-registry.ts` (line 74-81): - -```ts -// REPLACE: -export interface RegisterRunnerInput { - id: string - ownerUserId: string - label: string - kind?: RunnerKind - adminStatus?: RunnerAdminStatus - wakeStream?: string -} -// WITH: -export interface RegisterRunnerInput { - id: string - ownerPrincipal: string - label: string - kind?: RunnerKind - adminStatus?: RunnerAdminStatus - wakeStream?: string -} -``` - -- [x] **Step 2: Add `diagnostics` to `HeartbeatRunnerInput`** - -In `packages/agents-server/src/entity-registry.ts` (line 83-89): - -```ts -// REPLACE: -export interface HeartbeatRunnerInput { - runnerId: string - heartbeatAt?: Date - livenessLeaseExpiresAt?: Date - leaseMs?: number - wakeStreamOffset?: string -} -// WITH: -export interface HeartbeatRunnerInput { - runnerId: string - heartbeatAt?: Date - livenessLeaseExpiresAt?: Date - leaseMs?: number - wakeStreamOffset?: string - diagnostics?: Record -} -``` - -- [x] **Step 3: Update `createRunner` to use `ownerPrincipal`** - -In the `createRunner` method (line 132-167), replace all `ownerUserId` → `ownerPrincipal` references: - -```ts - async createRunner( - input: RegisterRunnerInput - ): Promise { - const now = new Date() - const wakeStream = input.wakeStream ?? runnerWakeStream(input.id) - - await this.db - .insert(runners) - .values({ - tenantId: this.tenantId, - id: input.id, - ownerPrincipal: input.ownerPrincipal, - label: input.label, - kind: input.kind ?? `local`, - adminStatus: input.adminStatus ?? `enabled`, - wakeStream, - updatedAt: now, - }) - .onConflictDoUpdate({ - target: [runners.tenantId, runners.id], - set: { - ownerPrincipal: input.ownerPrincipal, - label: input.label, - kind: input.kind ?? `local`, - adminStatus: input.adminStatus ?? `enabled`, - wakeStream, - updatedAt: now, - }, - }) - - const runner = await this.getRunner(input.id) - if (!runner) { - throw new Error(`Failed to read back runner "${input.id}"`) - } - return runner - } -``` - -- [x] **Step 4: Update `listRunners` filter** - -In `listRunners` (line 178-191): - -```ts -// REPLACE: - async listRunners(filter?: { - ownerUserId?: string - }): Promise> { - const conditions = [eq(runners.tenantId, this.tenantId)] - if (filter?.ownerUserId) { - conditions.push(eq(runners.ownerUserId, filter.ownerUserId)) - } -// WITH: - async listRunners(filter?: { - ownerPrincipal?: string - }): Promise> { - const conditions = [eq(runners.tenantId, this.tenantId)] - if (filter?.ownerPrincipal) { - conditions.push(eq(runners.ownerPrincipal, filter.ownerPrincipal)) - } -``` - -- [x] **Step 5: Update `heartbeatRunner` to store diagnostics** - -In `heartbeatRunner` (line 193-217): - -```ts - async heartbeatRunner( - input: HeartbeatRunnerInput - ): Promise { - const now = input.heartbeatAt ?? new Date() - const leaseExpiresAt = - input.livenessLeaseExpiresAt ?? - new Date(now.getTime() + (input.leaseMs ?? DEFAULT_RUNNER_LEASE_MS)) - - const rows = await this.db - .update(runners) - .set({ - lastSeenAt: now, - livenessLeaseExpiresAt: leaseExpiresAt, - ...(input.wakeStreamOffset !== undefined - ? { wakeStreamOffset: input.wakeStreamOffset } - : {}), - ...(input.diagnostics !== undefined - ? { diagnostics: input.diagnostics } - : {}), - updatedAt: now, - }) - .where( - and(eq(runners.tenantId, this.tenantId), eq(runners.id, input.runnerId)) - ) - .returning() - - return rows[0] ? this.rowToRunner(rows[0]) : null - } -``` - -- [x] **Step 6: Add `getActiveClaimsForRunner` query** - -Add after `materializeReleasedClaim` (around line 367): - -```ts - async getActiveClaimsForRunner( - runnerId: string - ): Promise> { - const rows = await this.db - .select() - .from(consumerClaims) - .where( - and( - eq(consumerClaims.tenantId, this.tenantId), - eq(consumerClaims.runnerId, runnerId), - eq(consumerClaims.status, `active`) - ) - ) - return rows.map((row) => this.rowToConsumerClaim(row)) - } -``` - -- [x] **Step 7: Add `getDispatchStatsForRunner` query** - -Add right after `getActiveClaimsForRunner`: - -```ts - async getDispatchStatsForRunner( - runnerId: string - ): Promise<{ - entities_with_active_claim: number - entities_with_outstanding_wake: number - entities_with_pending_work: number - }> { - const rows = await this.db - .select() - .from(entityDispatchState) - .where( - and( - eq(entityDispatchState.tenantId, this.tenantId), - eq(entityDispatchState.activeRunnerId, runnerId) - ) - ) - - let activeClaim = 0 - let outstandingWake = 0 - let pendingWork = 0 - for (const row of rows) { - if (row.activeConsumerId) activeClaim++ - if (row.outstandingWakeId && !row.activeConsumerId) outstandingWake++ - const pending = row.pendingSourceStreams as Array | null - if (pending && pending.length > 0) pendingWork++ - } - - return { - entities_with_active_claim: activeClaim, - entities_with_outstanding_wake: outstandingWake, - entities_with_pending_work: pendingWork, - } - } -``` - -- [x] **Step 8: Update `rowToRunner` to include `owner_principal` and `diagnostics`** - -In `rowToRunner` (line 1148-1168): - -```ts -// REPLACE: - private rowToRunner(row: typeof runners.$inferSelect): ElectricAgentsRunner { - const now = Date.now() - const livenessExpiry = row.livenessLeaseExpiresAt?.getTime() - return { - id: row.id, - owner_user_id: row.ownerUserId, - label: row.label, - kind: assertRunnerKind(row.kind), - admin_status: assertRunnerAdminStatus(row.adminStatus), - liveness: - livenessExpiry !== undefined && livenessExpiry > now - ? `online` - : `offline`, - last_seen_at: row.lastSeenAt?.toISOString(), - liveness_lease_expires_at: row.livenessLeaseExpiresAt?.toISOString(), - wake_stream: row.wakeStream, - wake_stream_offset: row.wakeStreamOffset ?? undefined, - created_at: row.createdAt.toISOString(), - updated_at: row.updatedAt.toISOString(), - } - } -// WITH: - private rowToRunner(row: typeof runners.$inferSelect): ElectricAgentsRunner { - const now = Date.now() - const livenessExpiry = row.livenessLeaseExpiresAt?.getTime() - return { - id: row.id, - owner_principal: row.ownerPrincipal, - label: row.label, - kind: assertRunnerKind(row.kind), - admin_status: assertRunnerAdminStatus(row.adminStatus), - liveness: - livenessExpiry !== undefined && livenessExpiry > now - ? `online` - : `offline`, - last_seen_at: row.lastSeenAt?.toISOString(), - liveness_lease_expires_at: row.livenessLeaseExpiresAt?.toISOString(), - wake_stream: row.wakeStream, - wake_stream_offset: row.wakeStreamOffset ?? undefined, - diagnostics: (row.diagnostics as Record) ?? undefined, - created_at: row.createdAt.toISOString(), - updated_at: row.updatedAt.toISOString(), - } - } -``` - -- [x] **Step 9: Commit** - -```bash -git add packages/agents-server/src/entity-registry.ts -git commit -m "feat(agents-server): update entity registry for principal rename, diagnostics, and health queries" -``` - ---- - -### Task 4: Update runners router, dispatch policy, and shape columns — principal rename, diagnostics in heartbeat, health endpoint - -**Files:** - -- Modify: `packages/agents-server/src/routing/runners-router.ts` -- Modify: `packages/agents-server/src/routing/dispatch-policy.ts:127` -- Modify: `packages/agents-server/src/utils/server-utils.ts:130-134` - -- [x] **Step 1: Update the registration body schema** - -In `packages/agents-server/src/routing/runners-router.ts` (line 36-53): - -```ts -// REPLACE: -const registerRunnerBodySchema = Type.Object({ - id: Type.String(), - owner_user_id: Type.Optional(Type.String()), - label: Type.String(), - kind: Type.Optional( - Type.Union([ - Type.Literal(`local`), - Type.Literal(`cloud-worker`), - Type.Literal(`sandbox`), - Type.Literal(`ci`), - Type.Literal(`server`), - ]) - ), - admin_status: Type.Optional( - Type.Union([Type.Literal(`enabled`), Type.Literal(`disabled`)]) - ), - wake_stream: Type.Optional(Type.String()), -}) -// WITH: -const registerRunnerBodySchema = Type.Object({ - id: Type.String(), - owner_principal: Type.Optional(Type.String()), - label: Type.String(), - kind: Type.Optional( - Type.Union([ - Type.Literal(`local`), - Type.Literal(`cloud-worker`), - Type.Literal(`sandbox`), - Type.Literal(`ci`), - Type.Literal(`server`), - ]) - ), - admin_status: Type.Optional( - Type.Union([Type.Literal(`enabled`), Type.Literal(`disabled`)]) - ), - wake_stream: Type.Optional(Type.String()), -}) -``` - -- [x] **Step 2: Add `diagnostics` to heartbeat body schema** - -In the `heartbeatBodySchema` (line 55-60): - -```ts -// REPLACE: -const heartbeatBodySchema = Type.Object({ - lease_ms: Type.Optional(Type.Number()), - wake_stream_offset: Type.Optional(Type.String()), - wakeStreamOffset: Type.Optional(Type.String()), - liveness_lease_expires_at: Type.Optional(Type.String()), -}) -// WITH: -const heartbeatBodySchema = Type.Object({ - lease_ms: Type.Optional(Type.Number()), - wake_stream_offset: Type.Optional(Type.String()), - wakeStreamOffset: Type.Optional(Type.String()), - liveness_lease_expires_at: Type.Optional(Type.String()), - diagnostics: Type.Optional(Type.Record(Type.String(), Type.Unknown())), -}) -``` - -- [x] **Step 3: Add the health route** - -After the existing routes (line 90), add: - -```ts -runnersRouter.get(`/:id/health`, runnerHealth) -``` - -- [x] **Step 4: Add principal URL validation import** - -Add to the imports at the top of `runners-router.ts`: - -```ts -import { isPrincipalUrl } from '../principal.js' -``` - -- [x] **Step 5: Update `registerRunner` handler to use `owner_principal` with strict URL validation** - -No backwards compatibility for key-form principals. If `owner_principal` is provided it must be a valid principal URL (e.g., `/principal/user%3Aalice`); otherwise the server derives it from `ctx.principal.url`. Callers must send URLs. - -In `registerRunner` (line 103-136): - -```ts -// REPLACE: -async function registerRunner( - request: RunnersRouteRequest, - ctx: TenantContext -): Promise { - const parsed = routeBody(request) - const ownerUserId = parsed.owner_user_id ?? ctx.principal?.key - if (!ownerUserId) { - throw new ElectricAgentsError( - ErrCodeInvalidRequest, - `owner_user_id is required when no authenticated user is present`, - 400 - ) - } - if (ctx.principal && ownerUserId !== ctx.principal.key) { - throw new ElectricAgentsError( - ErrCodeUnauthorized, - `owner_user_id must match the authenticated user`, - 403 - ) - } - - const runner = await ctx.entityManager.registry.createRunner({ - id: parsed.id, - ownerUserId, - label: parsed.label, - kind: parsed.kind, - adminStatus: parsed.admin_status, - wakeStream: parsed.wake_stream, - }) - await ctx.streamClient.ensure(runner.wake_stream, { - contentType: `application/json`, - }) - return json(runner, { status: 201 }) -} -// WITH: -async function registerRunner( - request: RunnersRouteRequest, - ctx: TenantContext -): Promise { - const parsed = routeBody(request) - const ownerPrincipal = parsed.owner_principal ?? ctx.principal?.url - if (!ownerPrincipal) { - throw new ElectricAgentsError( - ErrCodeInvalidRequest, - `owner_principal is required when no authenticated principal is present`, - 400 - ) - } - if (!isPrincipalUrl(ownerPrincipal)) { - throw new ElectricAgentsError( - ErrCodeInvalidRequest, - `owner_principal must be a valid principal URL (e.g. /principal/user%3Aalice), got: ${ownerPrincipal}`, - 400 - ) - } - if (ctx.principal && ownerPrincipal !== ctx.principal.url) { - throw new ElectricAgentsError( - ErrCodeUnauthorized, - `owner_principal must match the authenticated principal`, - 403 - ) - } - - const runner = await ctx.entityManager.registry.createRunner({ - id: parsed.id, - ownerPrincipal, - label: parsed.label, - kind: parsed.kind, - adminStatus: parsed.admin_status, - wakeStream: parsed.wake_stream, - }) - await ctx.streamClient.ensure(runner.wake_stream, { - contentType: `application/json`, - }) - return json(runner, { status: 201 }) -} -``` - -- [x] **Step 6: Update `listRunners` handler** - -In `listRunners` (line 138-154): - -```ts -// REPLACE: -async function listRunners( - request: RunnersRouteRequest, - ctx: TenantContext -): Promise { - const requestedOwner = firstQueryValue(request.query.owner_user_id) - if (ctx.principal && requestedOwner && requestedOwner !== ctx.principal.key) { - throw new ElectricAgentsError( - ErrCodeUnauthorized, - `owner_user_id must match the authenticated user`, - 403 - ) - } - const runners = await ctx.entityManager.registry.listRunners({ - ownerUserId: ctx.principal?.key ?? requestedOwner, - }) - return json(runners) -} -// WITH: -async function listRunners( - request: RunnersRouteRequest, - ctx: TenantContext -): Promise { - const requestedOwner = firstQueryValue(request.query.owner_principal) - if (requestedOwner && !isPrincipalUrl(requestedOwner)) { - throw new ElectricAgentsError( - ErrCodeInvalidRequest, - `owner_principal must be a valid principal URL (e.g. /principal/user%3Aalice), got: ${requestedOwner}`, - 400 - ) - } - if (ctx.principal && requestedOwner && requestedOwner !== ctx.principal.url) { - throw new ElectricAgentsError( - ErrCodeUnauthorized, - `owner_principal must match the authenticated principal`, - 403 - ) - } - const runners = await ctx.entityManager.registry.listRunners({ - ownerPrincipal: ctx.principal?.url ?? requestedOwner, - }) - return json(runners) -} -``` - -- [x] **Step 7: Update heartbeat handler to pass diagnostics** - -In `heartbeat` (line 165-185), add `diagnostics` to the `heartbeatRunner` call: - -```ts -const runner = await ctx.entityManager.registry.heartbeatRunner({ - runnerId, - leaseMs: parsed.lease_ms, - wakeStreamOffset: parsed.wake_stream_offset ?? parsed.wakeStreamOffset, - livenessLeaseExpiresAt: parsed.liveness_lease_expires_at - ? new Date(parsed.liveness_lease_expires_at) - : undefined, - diagnostics: parsed.diagnostics, -}) -``` - -- [x] **Step 8: Update `assertRunnerOwnerIfAuthenticated` to use `principal.url`** - -In `assertRunnerOwnerIfAuthenticated` (line 297-308): - -```ts -// REPLACE: -function assertRunnerOwnerIfAuthenticated( - ctx: TenantContext, - ownerUserId: string -): void { - if (!ctx.principal) return - if (ownerUserId === ctx.principal.key) return - throw new ElectricAgentsError( - ErrCodeUnauthorized, - `Runner access requires the authenticated owner`, - 403 - ) -} -// WITH: -function assertRunnerOwnerIfAuthenticated( - ctx: TenantContext, - ownerPrincipal: string -): void { - if (!ctx.principal) return - if (ownerPrincipal === ctx.principal.url) return - throw new ElectricAgentsError( - ErrCodeUnauthorized, - `Runner access requires the authenticated owner`, - 403 - ) -} -``` - -- [x] **Step 9: Update all callers of `assertRunnerOwnerIfAuthenticated`** - -Change all calls from `runner.owner_user_id` → `runner.owner_principal`: - -In `getRunner` (line 161): `assertRunnerOwnerIfAuthenticated(ctx, runner.owner_principal)` - -In `heartbeat` (line 171): `assertRunnerOwnerIfAuthenticated(ctx, existing.owner_principal)` - -In `setRunnerStatus` (line 208): `assertRunnerOwnerIfAuthenticated(ctx, existing.owner_principal)` - -- [x] **Step 10: Update claim auth check** - -In `claimWake` (line 225): - -```ts -// REPLACE: - if (ctx.principal && runner.owner_user_id !== ctx.principal.key) { -// WITH: - if (ctx.principal && runner.owner_principal !== ctx.principal.url) { -``` - -- [x] **Step 11: Update `assertDispatchPolicyAllowed` in dispatch-policy.ts** - -In `packages/agents-server/src/routing/dispatch-policy.ts` (line 127): - -```ts -// REPLACE: - if (ctx.principal && runner.owner_user_id !== ctx.principal.key) { -// WITH: - if (ctx.principal && runner.owner_principal !== ctx.principal.url) { -``` - -- [x] **Step 12: Update runners Shape column allowlist in server-utils.ts** - -In `packages/agents-server/src/utils/server-utils.ts` (line 131-133): - -```ts -// REPLACE: -;`"tenant_id","id","owner_user_id","label","kind","admin_status","wake_stream","wake_stream_offset","last_seen_at","liveness_lease_expires_at","created_at","updated_at"` -// WITH: -`"tenant_id","id","owner_principal","label","kind","admin_status","wake_stream","wake_stream_offset","last_seen_at","liveness_lease_expires_at","diagnostics","created_at","updated_at"` -``` - -- [x] **Step 13: Implement `runnerHealth` handler** - -Add at the bottom of the file, before `notificationFromClaim`: - -```ts -async function runnerHealth( - request: RunnersRouteRequest, - ctx: TenantContext -): Promise { - const runnerId = routeParam(request, `id`) - const runner = await requireRunner(ctx, runnerId) - assertRunnerOwnerIfAuthenticated(ctx, runner.owner_principal) - - const now = Date.now() - const leaseExpiresAt = runner.liveness_lease_expires_at - ? new Date(runner.liveness_lease_expires_at).getTime() - : null - - const livenessStatus = - runner.admin_status === `disabled` - ? `offline` - : leaseExpiresAt !== null && leaseExpiresAt > now - ? `online` - : leaseExpiresAt !== null - ? `expired` - : `offline` - - const [activeClaims, dispatchStats] = await Promise.all([ - ctx.entityManager.registry.getActiveClaimsForRunner(runnerId), - ctx.entityManager.registry.getDispatchStatsForRunner(runnerId), - ]) - - const clientDiagnostics = runner.diagnostics ?? null - - const issues: Array = [] - let healthStatus: `healthy` | `degraded` | `unhealthy` = `healthy` - - if (runner.admin_status === `disabled`) { - healthStatus = `unhealthy` - issues.push(`Runner is disabled`) - } - if (livenessStatus === `expired`) { - healthStatus = `unhealthy` - const ago = leaseExpiresAt ? Math.round((now - leaseExpiresAt) / 1000) : 0 - issues.push(`Heartbeat lease expired ${ago}s ago`) - } - if (livenessStatus === `offline` && runner.admin_status === `enabled`) { - healthStatus = healthStatus === `unhealthy` ? `unhealthy` : `degraded` - issues.push(`Runner has never sent a heartbeat`) - } - if (clientDiagnostics) { - if (clientDiagnostics.stream_connected === false) { - if (healthStatus === `healthy`) healthStatus = `degraded` - issues.push(`Client reports stream disconnected`) - } - if (clientDiagnostics.last_heartbeat_ok === false) { - if (healthStatus === `healthy`) healthStatus = `degraded` - issues.push(`Client reports last heartbeat failed`) - } - if ( - typeof clientDiagnostics.reconnect_count === `number` && - clientDiagnostics.reconnect_count > 5 - ) { - if (healthStatus === `healthy`) healthStatus = `degraded` - issues.push( - `Client has reconnected ${clientDiagnostics.reconnect_count} times` - ) - } - } else if (runner.last_seen_at) { - if (healthStatus === `healthy`) healthStatus = `degraded` - issues.push(`No client diagnostics available`) - } - - return json({ - runner: { - id: runner.id, - admin_status: runner.admin_status, - liveness_status: livenessStatus, - lease_expires_at: runner.liveness_lease_expires_at ?? null, - lease_remaining_ms: - leaseExpiresAt !== null ? Math.max(0, leaseExpiresAt - now) : null, - wake_stream: runner.wake_stream, - wake_stream_offset: runner.wake_stream_offset ?? null, - last_seen_at: runner.last_seen_at ?? null, - created_at: runner.created_at, - }, - client: clientDiagnostics, - claims: { - active_count: activeClaims.length, - active: activeClaims.map((c) => ({ - consumer_id: c.consumer_id, - epoch: c.epoch, - entity_url: c.entity_url, - stream_path: c.stream_path, - claimed_at: c.claimed_at, - last_heartbeat_at: c.last_heartbeat_at ?? null, - lease_expires_at: c.lease_expires_at ?? null, - })), - }, - dispatch: dispatchStats, - health: { status: healthStatus, issues }, - }) -} -``` - -- [x] **Step 14: Commit** - -```bash -git add packages/agents-server/src/routing/runners-router.ts packages/agents-server/src/routing/dispatch-policy.ts packages/agents-server/src/utils/server-utils.ts -git commit -m "feat(agents-server): update runners router, dispatch policy, and shape columns for principal rename, diagnostics, and health endpoint" -``` - ---- - -### Task 5: Client-side diagnostics in PullWakeRunner - -**Files:** - -- Modify: `packages/agents-runtime/src/pull-wake-runner.ts` - -- [x] **Step 1: Add `PullWakeRunnerHealth` interface and diagnostics tracking** - -In `packages/agents-runtime/src/pull-wake-runner.ts`, after the existing `PullWakeRunner` interface (line 48-54), add: - -```ts -export interface PullWakeRunnerHealth { - running: boolean - offset: string | undefined - started_at: string | null - stream_connected: boolean - stream_connected_since: string | null - reconnect_count: number - last_error: string | null - last_error_at: string | null - last_heartbeat_at: string | null - last_heartbeat_ok: boolean - last_claim_at: string | null - last_claim_result: `claimed` | `no_work` | `error` | null - last_dispatch_at: string | null - events_received: number - claims_succeeded: number - claims_skipped: number - claims_failed: number -} -``` - -Add `getHealth` to the `PullWakeRunner` interface: - -```ts -export interface PullWakeRunner { - start: () => void - stop: () => Promise - waitForStopped: () => Promise - readonly running: boolean - readonly offset: string | undefined - getHealth: () => PullWakeRunnerHealth -} -``` - -- [x] **Step 2: Add diagnostic state variables inside `createPullWakeRunner`** - -After the existing `let currentOffset = config.offset` (line 63), add: - -```ts -let startedAt: string | null = null -let streamConnected = false -let streamConnectedSince: string | null = null -let reconnectCount = 0 -let lastError: string | null = null -let lastErrorAt: string | null = null -let lastHeartbeatAt: string | null = null -let lastHeartbeatOk = false -let lastClaimAt: string | null = null -let lastClaimResult: PullWakeRunnerHealth[`last_claim_result`] = null -let lastDispatchAt: string | null = null -let eventsReceived = 0 -let claimsSucceeded = 0 -let claimsSkipped = 0 -let claimsFailed = 0 -``` - -- [x] **Step 3: Build the diagnostics snapshot function** - -Add after the diagnostic variables: - -```ts -const buildDiagnostics = (): Omit< - PullWakeRunnerHealth, - `running` | `offset` -> => ({ - started_at: startedAt, - stream_connected: streamConnected, - stream_connected_since: streamConnectedSince, - reconnect_count: reconnectCount, - last_error: lastError, - last_error_at: lastErrorAt, - last_heartbeat_at: lastHeartbeatAt, - last_heartbeat_ok: lastHeartbeatOk, - last_claim_at: lastClaimAt, - last_claim_result: lastClaimResult, - last_dispatch_at: lastDispatchAt, - events_received: eventsReceived, - claims_succeeded: claimsSucceeded, - claims_skipped: claimsSkipped, - claims_failed: claimsFailed, -}) -``` - -- [x] **Step 4: Update `heartbeat` to report diagnostics and track heartbeat state** - -Replace the existing `heartbeat` function (line 106-131): - -```ts -const heartbeat = async (signal: AbortSignal): Promise => { - try { - const headers = new Headers(await resolveHeaders()) - headers.set(`content-type`, `application/json`) - const res = await fetch(heartbeatUrl, { - method: `POST`, - headers, - body: JSON.stringify({ - lease_ms: leaseMs, - ...(currentOffset !== undefined - ? { wake_stream_offset: currentOffset } - : {}), - diagnostics: buildDiagnostics(), - }), - signal, - }) - lastHeartbeatAt = new Date().toISOString() - if (!res.ok) { - lastHeartbeatOk = false - throw new Error( - `Pull-wake runner heartbeat failed for ${config.runnerId}: ${res.status} ${await res.text()}` - ) - } - lastHeartbeatOk = true - } catch (err) { - if (!signal.aborted) { - lastHeartbeatOk = false - config.onError?.(err instanceof Error ? err : new Error(String(err))) - } - } -} -``` - -- [x] **Step 5: Update `reportError` to track errors** - -Replace the existing `reportError` (line 101-104): - -```ts -const reportError = (err: unknown): void => { - const error = err instanceof Error ? err : new Error(String(err)) - lastError = error.message - lastErrorAt = new Date().toISOString() - if (config.onError?.(error) !== true) throw error -} -``` - -- [x] **Step 6: Update `claimWake` to track claim results** - -Replace the existing `claimWake` (line 170-200): - -```ts -const claimWake = async ( - event: PullWakeEvent, - signal: AbortSignal -): Promise => { - lastClaimAt = new Date().toISOString() - const headers = new Headers(await resolveHeaders()) - headers.set(`content-type`, `application/json`) - try { - const response = await fetch(claimUrl, { - method: `POST`, - headers, - signal, - body: JSON.stringify(event), - }) - if (response.status === 204) { - lastClaimResult = `no_work` - claimsSkipped++ - return null - } - if (!response.ok) { - const text = await response.text() - if ( - response.status === 409 && - (text.includes(`ALREADY_CLAIMED`) || text.includes(`NO_PENDING_WORK`)) - ) { - lastClaimResult = `no_work` - claimsSkipped++ - return null - } - lastClaimResult = `error` - claimsFailed++ - throw new Error( - `Pull-wake claim failed for ${config.runnerId}: ${response.status} ${text}` - ) - } - const notification = (await response.json()) as WakeNotification & { - done?: boolean - } - if (notification.done) { - lastClaimResult = `no_work` - claimsSkipped++ - return null - } - lastClaimResult = `claimed` - claimsSucceeded++ - return notification - } catch (err) { - if (lastClaimResult !== `no_work` && lastClaimResult !== `error`) { - lastClaimResult = `error` - claimsFailed++ - } - throw err - } -} -``` - -- [x] **Step 7: Update the `run` function to track stream and event state** - -Replace the existing `run` function (line 202-236): - -```ts -const run = async (): Promise => { - const signal = controller!.signal - try { - response = await streamFactory({ - url: wakeUrl, - headers: await resolveHeaders(), - offset: currentOffset, - signal, - }) - streamConnected = true - streamConnectedSince = new Date().toISOString() - for await (const event of response.jsonStream()) { - if (signal.aborted) break - if (event?.type !== `wake`) continue - eventsReceived++ - const notification = await claimWake(event, signal) - if (notification) { - config.runtime.dispatchWake(notification, { - claimHeaders: resolveClaimHeaders, - claimTokenHeader: config.claimTokenHeader, - }) - lastDispatchAt = new Date().toISOString() - await config.runtime.drainWakes() - } - if (response.offset !== undefined) currentOffset = response.offset - } - await response.closed?.catch((err) => { - if (!signal.aborted) throw err - }) - } catch (err) { - if (!signal.aborted) { - reconnectCount++ - reportError(err) - } - } finally { - streamConnected = false - stopHeartbeat() - response = null - controller = null - } -} -``` - -- [x] **Step 8: Update `start()` to record `startedAt`** - -In the returned object's `start()` method (line 239-244): - -```ts - start() { - if (loop) return - controller = new AbortController() - startedAt = new Date().toISOString() - startHeartbeat(controller.signal) - loop = run().finally(() => { - loop = null - }) - }, -``` - -- [x] **Step 9: Add `getHealth()` to the returned object** - -Add after the `offset` getter: - -```ts - getHealth(): PullWakeRunnerHealth { - return { - running: loop !== null, - offset: currentOffset, - ...buildDiagnostics(), - } - }, -``` - -- [x] **Step 10: Update the runtime index exports** - -In `packages/agents-runtime/src/index.ts`, add `PullWakeRunnerHealth` to the exports (line 238-243): - -```ts -// REPLACE: -export type { - PullWakeEvent, - PullWakeRunner, - PullWakeRunnerConfig, - PullWakeStreamResponse, -} from './pull-wake-runner' -// WITH: -export type { - PullWakeEvent, - PullWakeRunner, - PullWakeRunnerConfig, - PullWakeRunnerHealth, - PullWakeStreamResponse, -} from './pull-wake-runner' -``` - -- [x] **Step 11: Commit** - -```bash -git add packages/agents-runtime/src/pull-wake-runner.ts packages/agents-runtime/src/index.ts -git commit -m "feat(agents-runtime): add diagnostics tracking and getHealth() to PullWakeRunner" -``` - ---- - -### Task 6: Update BuiltinAgentsServer, electric-ax, and desktop app for principal rename - -**Files:** - -- Modify: `packages/agents/src/server.ts:40-51, 393-422` -- Modify: `packages/electric-ax/src/start.ts:131-139, 379, 395` -- Modify: `packages/agents-desktop/src/main.ts:219-274, 1544-1582` - -- [x] **Step 1: Update `BuiltinAgentsServerOptions` in agents/server.ts** - -In `packages/agents/src/server.ts` (line 40-51): - -```ts -// REPLACE: - pullWake: { - runnerId: string - ownerUserId?: string - label?: string - registerRunner?: boolean -// WITH: - pullWake: { - runnerId: string - ownerPrincipal?: string - label?: string - registerRunner?: boolean -``` - -- [x] **Step 2: Update `registerPullWakeRunner` to use `owner_principal`** - -In `packages/agents/src/server.ts` (line 393-422): - -```ts -// REPLACE: - body: JSON.stringify({ - id: pullWake.runnerId, - owner_user_id: pullWake.ownerUserId, - label: pullWake.label ?? `Built-in agents`, - kind: `local`, - admin_status: `enabled`, - }), -// WITH: - body: JSON.stringify({ - id: pullWake.runnerId, - owner_principal: pullWake.ownerPrincipal, - label: pullWake.label ?? `Built-in agents`, - kind: `local`, - admin_status: `enabled`, - }), -``` - -- [x] **Step 3: Update electric-ax/src/start.ts** - -In `packages/electric-ax/src/start.ts`: - -First, rename the default constant (line 21). Store a principal URL directly: - -```ts -// REPLACE: -const DEFAULT_PULL_WAKE_OWNER_ID = `builtin-agents` -// WITH: -const DEFAULT_PULL_WAKE_OWNER_PRINCIPAL = `/principal/system%3Abuiltin-agents` -``` - -Then rename the function (line 131-139). `ELECTRIC_AGENTS_IDENTITY` is a principal identifier (`kind:id`), so convert it to a URL. The default is already a URL: - -```ts -// REPLACE: -export function resolvePullWakeOwnerId( - env: NodeJS.ProcessEnv = process.env, - fileEnv: Record = readDotEnvFile() -): string { - return ( - readConfigValue(env, fileEnv, [`ELECTRIC_AGENTS_IDENTITY`]) ?? - DEFAULT_PULL_WAKE_OWNER_ID - ) -} -// WITH: -export function resolvePullWakeOwnerPrincipal( - env: NodeJS.ProcessEnv = process.env, - fileEnv: Record = readDotEnvFile() -): string { - const identity = readConfigValue(env, fileEnv, [`ELECTRIC_AGENTS_IDENTITY`]) - if (identity) return `/principal/${encodeURIComponent(identity)}` - return DEFAULT_PULL_WAKE_OWNER_PRINCIPAL -} -``` - -Update the usage (line 379): - -```ts -// REPLACE: -const ownerUserId = resolvePullWakeOwnerId(env, fileEnv) -// WITH: -const ownerPrincipal = resolvePullWakeOwnerPrincipal(env, fileEnv) -``` - -Update the `BuiltinAgentsServer` call (line 395): - -```ts -// REPLACE: - ownerUserId, -// WITH: - ownerPrincipal, -``` - -- [x] **Step 4: Update desktop env var and function names** - -In `packages/agents-desktop/src/main.ts`: - -Replace the previous owner-user constant/env var. No backwards-compat fallback — clean break. Store a principal URL directly: - -```ts -const PULL_WAKE_OWNER_PRINCIPAL = - process.env.ELECTRIC_DESKTOP_PULL_WAKE_OWNER_PRINCIPAL?.trim() || - `/principal/system%3Adev-local` -``` - -Rename the helper function (line 265-274). Do NOT use the `authorization` header as a principal source — that's a bearer token, not a principal identifier. When the request has auth headers, the server middleware extracts `ctx.principal` from them and uses `ctx.principal.url` as the owner. So when only auth is present (no explicit `electric-principal` header), return `undefined` to let the server derive the owner: - -```ts -// REPLACE: -function runnerOwnerUserIdFromHeaders( - headers: Record | undefined -): string { - const normalized = new Headers(headers) - return ( - normalized.get(`authorization`)?.trim() || - normalized.get(ELECTRIC_PRINCIPAL_HEADER)?.trim() || - PULL_WAKE_OWNER_PRINCIPAL - ) -} -// WITH: -function runnerOwnerPrincipalFromHeaders( - headers: Record | undefined -): string | undefined { - const normalized = new Headers(headers) - const principalKey = normalized.get(ELECTRIC_PRINCIPAL_HEADER)?.trim() - if (principalKey) { - return principalKey.startsWith(`/principal/`) - ? principalKey - : `/principal/${encodeURIComponent(principalKey)}` - } - if (normalized.has(`authorization`)) return undefined - return PULL_WAKE_OWNER_PRINCIPAL -} -``` - -Update usage (line 1544, 1551, 1575): - -```ts -// REPLACE: -const runnerOwnerUserId = runnerOwnerUserIdFromHeaders(runtimeHeaders) -// WITH: -const runnerOwnerPrincipal = runnerOwnerPrincipalFromHeaders(runtimeHeaders) -``` - -```ts -// REPLACE: - ownerUserId: PULL_WAKE_REGISTER_RUNNER ? runnerOwnerUserId : undefined, -// WITH: - ownerPrincipal: PULL_WAKE_REGISTER_RUNNER ? runnerOwnerPrincipal : undefined, -``` - -Update log messages referencing `owner user id` → `owner principal`. - -- [x] **Step 5: Commit** - -```bash -git add packages/agents/src/server.ts packages/electric-ax/src/start.ts packages/agents-desktop/src/main.ts -git commit -m "feat(agents, electric-ax, agents-desktop): rename ownerUserId to ownerPrincipal for runner registration" -``` - ---- - -### Task 7: Update tests for principal rename and health endpoint - -**Files:** - -- Modify: `packages/agents-server/test/runners-router.test.ts` -- Modify: `packages/agents-runtime/test/pull-wake-runner.test.ts` -- Modify: `packages/agents-server/test/horton-pull-wake-e2e.test.ts` -- Modify: `packages/agents-server/test/horton-title-generation.test.ts` -- Modify: `packages/agents-server/test/horton-spawn-worker.test.ts` -- Modify: `packages/agents-server/test/dispatch-policy-routing.test.ts` - -- [x] **Step 1: Update runners-router.test.ts — principal rename and context** - -In `packages/agents-server/test/runners-router.test.ts`: - -Update the `runner()` helper (line 15-28): - -```ts -// REPLACE: - owner_user_id: `user:owner@example.com`, -// WITH: - owner_principal: `/principal/user%3Aowner%40example.com`, -``` - -Update `buildContext` registry mock (line 33-35): - -```ts -// REPLACE: - createRunner: vi.fn(async (input) => - runner({ - id: input.id, - owner_user_id: input.ownerUserId, -// WITH: - createRunner: vi.fn(async (input) => - runner({ - id: input.id, - owner_principal: input.ownerPrincipal, -``` - -Update all test assertions that reference `owner_user_id`: - -- Line 89: `owner_user_id: `other@example.com``→`owner_principal: `/principal/other` -- Line 118: `owner_user_id: `user:owner@example.com`` → `owner_principal: `/principal/user%3Aowner%40example.com`` -- Line 128-129: `ownerUserId: `user:owner@example.com`` → `ownerPrincipal: `/principal/user%3Aowner%40example.com`` -- Line 158-159: same replacement - -- [x] **Step 2: Add health endpoint test to runners-router.test.ts** - -Add to the `runner routes` describe block: - -```ts -it(`returns runner health with diagnostics and claim state`, async () => { - const ctx = buildContext({ - principal: { - kind: `user`, - id: `owner@example.com`, - key: `user:owner@example.com`, - url: `/principal/user%3Aowner%40example.com`, - }, - }) - vi.mocked(ctx.entityManager.registry.getRunner).mockResolvedValue( - runner({ - owner_principal: `/principal/user%3Aowner%40example.com`, - liveness_lease_expires_at: new Date(Date.now() + 30_000).toISOString(), - last_seen_at: new Date().toISOString(), - diagnostics: { - stream_connected: true, - reconnect_count: 0, - last_heartbeat_ok: true, - }, - }) - ) - ctx.entityManager.registry.getActiveClaimsForRunner = vi.fn(async () => []) - ctx.entityManager.registry.getDispatchStatsForRunner = vi.fn(async () => ({ - entities_with_active_claim: 0, - entities_with_outstanding_wake: 0, - entities_with_pending_work: 0, - })) - - const response = await globalRouter.fetch( - request(`GET`, `/_electric/runners/runner-1/health`), - ctx - ) - - expect(response.status).toBe(200) - const body = (await response.json()) as Record - expect(body.runner).toMatchObject({ - id: `runner-1`, - liveness_status: `online`, - }) - expect(body.client).toMatchObject({ stream_connected: true }) - expect(body.claims).toMatchObject({ active_count: 0 }) - expect(body.health).toMatchObject({ status: `healthy`, issues: [] }) -}) - -it(`returns unhealthy when runner lease is expired`, async () => { - const ctx = buildContext({ - principal: { - kind: `user`, - id: `owner@example.com`, - key: `user:owner@example.com`, - url: `/principal/user%3Aowner%40example.com`, - }, - }) - vi.mocked(ctx.entityManager.registry.getRunner).mockResolvedValue( - runner({ - owner_principal: `/principal/user%3Aowner%40example.com`, - liveness_lease_expires_at: new Date(Date.now() - 10_000).toISOString(), - last_seen_at: new Date(Date.now() - 15_000).toISOString(), - }) - ) - ctx.entityManager.registry.getActiveClaimsForRunner = vi.fn(async () => []) - ctx.entityManager.registry.getDispatchStatsForRunner = vi.fn(async () => ({ - entities_with_active_claim: 0, - entities_with_outstanding_wake: 0, - entities_with_pending_work: 0, - })) - - const response = await globalRouter.fetch( - request(`GET`, `/_electric/runners/runner-1/health`), - ctx - ) - - expect(response.status).toBe(200) - const body = (await response.json()) as Record - expect((body.health as any).status).toBe(`unhealthy`) - expect((body.health as any).issues.length).toBeGreaterThan(0) -}) -``` - -- [x] **Step 3: Add `getHealth()` test to pull-wake-runner.test.ts** - -Add to the `createPullWakeRunner` describe block in `packages/agents-runtime/test/pull-wake-runner.test.ts`: - -```ts -it(`exposes diagnostics via getHealth()`, async () => { - const event: PullWakeEvent = { - type: `wake`, - subscription_id: `runner:runner-1`, - stream: `chat/one/main`, - generation: 7, - ts: 123, - } - const notification: WakeNotification = { - consumerId: `wake-1`, - epoch: 7, - wakeId: `wake-1`, - streamPath: `/chat/one/main`, - streams: [{ path: `/chat/one/main`, offset: `12` }], - callback: `http://server/_electric/callback-forward/wake-1`, - claimToken: `claim-token`, - entity: { - type: `chat`, - status: `idle`, - url: `/chat/one`, - streams: { main: `/chat/one/main`, error: `/chat/one/error` }, - }, - } - const fetchMock = vi.fn(async (_input: RequestInfo | URL) => - Response.json(notification) - ) - vi.stubGlobal(`fetch`, fetchMock) - const streamFactory = vi.fn(async () => ({ - offset: `42`, - async *jsonStream() { - yield event - }, - closed: Promise.resolve(), - })) - - const runner = createPullWakeRunner({ - baseUrl: `http://server`, - runnerId: `runner-1`, - runtime: { - dispatchWake: vi.fn(), - drainWakes: vi.fn(async () => undefined), - abortWakes: vi.fn(), - }, - heartbeatIntervalMs: 0, - streamFactory, - }) - - const healthBefore = runner.getHealth() - expect(healthBefore.running).toBe(false) - expect(healthBefore.started_at).toBeNull() - expect(healthBefore.events_received).toBe(0) - - runner.start() - await runner.waitForStopped() - - const healthAfter = runner.getHealth() - expect(healthAfter.running).toBe(false) - expect(healthAfter.started_at).not.toBeNull() - expect(healthAfter.events_received).toBe(1) - expect(healthAfter.claims_succeeded).toBe(1) - expect(healthAfter.last_claim_result).toBe(`claimed`) - expect(healthAfter.last_dispatch_at).not.toBeNull() - expect(healthAfter.offset).toBe(`42`) -}) -``` - -- [x] **Step 4: Update horton-pull-wake-e2e.test.ts for principal rename** - -In `packages/agents-server/test/horton-pull-wake-e2e.test.ts` (line 133): - -```ts -// REPLACE: - ownerUserId: testPrincipal.key, -// WITH: - ownerPrincipal: testPrincipal.url, -``` - -- [x] **Step 5: Update horton-title-generation.test.ts and horton-spawn-worker.test.ts** - -In `packages/agents-server/test/horton-title-generation.test.ts` (line 39): - -```ts -// REPLACE: - ownerUserId: `test-user`, -// WITH: - ownerPrincipal: `/principal/system%3Atest-user`, -``` - -In `packages/agents-server/test/horton-spawn-worker.test.ts` (line 39): - -```ts -// REPLACE: - ownerUserId: `test-user`, -// WITH: - ownerPrincipal: `/principal/system%3Atest-user`, -``` - -- [x] **Step 6: Update dispatch-policy-routing.test.ts** - -In `packages/agents-server/test/dispatch-policy-routing.test.ts` (line 71): - -```ts -// REPLACE: - owner_user_id: `user:owner@example.com`, -// WITH: - owner_principal: `/principal/user%3Aowner%40example.com`, -``` - -- [x] **Step 7: Run all tests** - -Run: `cd packages/agents-runtime && pnpm vitest run test/pull-wake-runner.test.ts --reporter=dot` - -Run: `cd packages/agents-server && pnpm vitest run test/runners-router.test.ts --reporter=dot` - -Expected: All tests PASS - -- [x] **Step 8: Commit** - -```bash -git add packages/agents-server/test/ packages/agents-runtime/test/ -git commit -m "test: update all tests for principal rename and add health endpoint tests" -``` - ---- - -### Task 8: Typecheck and final verification - -- [x] **Step 1: Typecheck agents-runtime** - -Run: `pnpm -C packages/agents-runtime build` -Expected: No errors - -- [x] **Step 2: Typecheck agents-server** - -Run: `pnpm --filter @electric-ax/agents-server typecheck` -Expected: No errors - -- [x] **Step 3: Typecheck agents** - -Run: `pnpm --filter @electric-ax/agents typecheck` -Expected: No errors - -- [x] **Step 4: Typecheck agents-desktop** - -Run: `pnpm --filter @electric-ax/agents-desktop typecheck` -Expected: No errors - -- [x] **Step 5: Run unit tests** - -Run: `cd packages/agents-runtime && pnpm vitest run test/pull-wake-runner.test.ts --reporter=dot` -Run: `cd packages/agents-server && pnpm vitest run test/runners-router.test.ts --reporter=dot` -Expected: All PASS - -- [x] **Step 6: Fix any issues and commit** - -If any typecheck or test failures, fix and commit: - -```bash -git commit -m "fix: address typecheck and test issues from health check implementation" -``` diff --git a/packages/agents-server/docs/superpowers/specs/2026-05-16-pull-wake-health-check-design.md b/packages/agents-server/docs/superpowers/specs/2026-05-16-pull-wake-health-check-design.md deleted file mode 100644 index e9d0373597..0000000000 --- a/packages/agents-server/docs/superpowers/specs/2026-05-16-pull-wake-health-check-design.md +++ /dev/null @@ -1,214 +0,0 @@ -# Pull-Wake Runner Health Check - -## Problem - -The pull-wake dispatch system is unreliable and there's no way to diagnose what's going wrong. We need: - -1. A **health check endpoint** (`GET /_electric/runners/:id/health`) for deep debugging — curl it to see comprehensive diagnostics about a runner's dispatch pipeline. -2. **Rich runner state in Postgres** so that apps can sync the `runners` table via an Electric Shape and show runner status on any device (e.g. see your laptop runner's status from your phone). - -The `diagnostics` JSONB column on the `runners` table serves both purposes: the health endpoint reads it for the detailed response, and Shape sync delivers it reactively to any connected client. - -## Design - -### Layer 1: Client-Side Diagnostics (PullWakeRunner) - -Add internal state tracking to `createPullWakeRunner` in `packages/agents-runtime/src/pull-wake-runner.ts`. - -**Tracked state:** - -| Field | Type | Description | -| ------------------------ | ------------------------------------- | -------------------------------------------------- | -| `started_at` | ISO string | When `start()` was called | -| `stream_connected` | boolean | Whether the stream iterator is actively yielding | -| `stream_connected_since` | ISO string | When the current stream connection was established | -| `reconnect_count` | number | Total reconnection attempts since start | -| `last_error` | string | Most recent error message | -| `last_error_at` | ISO string | When the last error occurred | -| `last_heartbeat_at` | ISO string | When the last heartbeat was sent | -| `last_heartbeat_ok` | boolean | Whether the last heartbeat succeeded | -| `last_claim_at` | ISO string | When the last claim attempt was made | -| `last_claim_result` | `"claimed"` / `"no_work"` / `"error"` | Result of the last claim | -| `last_dispatch_at` | ISO string | When the last wake was dispatched to the runtime | -| `events_received` | number | Total wake events received from the stream | -| `claims_succeeded` | number | Total successful claims | -| `claims_skipped` | number | Claims that returned no work / already claimed | -| `claims_failed` | number | Claims that errored | - -**New interface method:** - -```ts -export interface PullWakeRunner { - // ... existing - getHealth: () => PullWakeRunnerHealth -} - -export interface PullWakeRunnerHealth { - running: boolean - offset: string | undefined - started_at: string | null - stream_connected: boolean - stream_connected_since: string | null - reconnect_count: number - last_error: string | null - last_error_at: string | null - last_heartbeat_at: string | null - last_heartbeat_ok: boolean - last_claim_at: string | null - last_claim_result: 'claimed' | 'no_work' | 'error' | null - last_dispatch_at: string | null - events_received: number - claims_succeeded: number - claims_skipped: number - claims_failed: number -} -``` - -**Reporting to server:** The heartbeat POST body already sends `lease_ms` and `wake_stream_offset`. Extend it with a `diagnostics` field containing the tracked state above. The server persists this in the runners table. - -### Layer 2: Server-Side Storage - -Add a `diagnostics` JSONB column to the `runners` table via migration `0007_runner_diagnostics.sql`. - -The `heartbeatRunner` method in `PostgresRegistry` stores the diagnostics payload from the heartbeat request. - -The `ElectricAgentsRunner` type gains an optional `diagnostics` field. - -### Principal-Based Ownership Rename - -The existing `owner_user_id` column and field name is a misnomer — principals aren't limited to users. They include `agent:`, `service:`, and `system:` actors. The canonical identifier is the principal URL (e.g. `/principal/user%3Aalice`), not the key (`user:alice`). - -As part of this work, rename across the runners system: - -- **DB column**: `owner_user_id` → `owner_principal` (migration) -- **Drizzle schema**: `ownerUserId` → `ownerPrincipal` -- **Types**: `ElectricAgentsRunner.owner_user_id` → `ElectricAgentsRunner.owner_principal` -- **API body/query**: `owner_user_id` → `owner_principal` -- **Auth checks**: compare against `ctx.principal.url` instead of `ctx.principal.key` -- **Registry methods**: `ownerUserId` param → `ownerPrincipal` - -The stored value is the principal URL (`/principal/user%3Aalice`), which is the primary identifier for everything in the system. - -### Multi-Device Runner Status via Shape Sync - -Apps sync the `runners` table via an Electric Shape scoped to `owner_principal`. This gives reactive runner status on any device without polling. The table row provides everything a UI needs: - -- **Online/offline**: derive from `liveness_lease_expires_at` vs current time -- **Admin status**: `admin_status` column (`enabled`/`disabled`) -- **Owner**: `owner_principal` — the principal URL, for Shape scoping (`WHERE owner_principal = :my_principal_url`) -- **Stream connected**: `diagnostics.stream_connected` -- **Last error**: `diagnostics.last_error` + `diagnostics.last_error_at` -- **Activity**: `diagnostics.last_claim_at`, `diagnostics.last_dispatch_at` -- **Counters**: `diagnostics.events_received`, `diagnostics.claims_succeeded`, etc. - -The health endpoint aggregates additional server-side state (active claims, dispatch stats) that isn't in the runners table — it's the "explain what's happening right now" view for debugging. The Shape is the "show me my runners" view for apps. - -### Layer 3: Health Endpoint - -**Route:** `GET /_electric/runners/:id/health` - -Added to `runners-router.ts` alongside the existing runner CRUD routes. Same auth as `getRunner` — owner must match authenticated principal. - -**Response shape:** - -```json -{ - "runner": { - "id": "desktop-abc123", - "admin_status": "enabled", - "liveness_status": "online", - "lease_expires_at": "2026-05-16T12:00:30Z", - "lease_remaining_ms": 12345, - "wake_stream": "/runners/desktop-abc123/wake", - "wake_stream_offset": "0_3", - "last_seen_at": "2026-05-16T12:00:00Z", - "created_at": "2026-05-16T11:00:00Z" - }, - "client": { - "started_at": "2026-05-16T11:00:01Z", - "stream_connected": true, - "stream_connected_since": "2026-05-16T11:00:02Z", - "reconnect_count": 0, - "last_error": null, - "last_error_at": null, - "last_heartbeat_at": "2026-05-16T12:00:00Z", - "last_heartbeat_ok": true, - "last_claim_at": "2026-05-16T11:55:00Z", - "last_claim_result": "claimed", - "last_dispatch_at": "2026-05-16T11:55:01Z", - "events_received": 14, - "claims_succeeded": 10, - "claims_skipped": 3, - "claims_failed": 1 - }, - "claims": { - "active_count": 1, - "active": [ - { - "consumer_id": "wake-001", - "epoch": 3, - "entity_url": "/entities/coder/session-42", - "stream_path": "/coder/session-42/main", - "claimed_at": "2026-05-16T11:55:00Z", - "last_heartbeat_at": "2026-05-16T12:00:00Z", - "lease_expires_at": "2026-05-16T12:00:30Z" - } - ] - }, - "dispatch": { - "entities_with_active_claim": 1, - "entities_with_outstanding_wake": 0, - "entities_with_pending_work": 2 - }, - "health": { - "status": "healthy", - "issues": [] - } -} -``` - -**Health status derivation rules:** - -| Condition | Status | -| ----------------------------------------------- | ----------- | -| Lease expired (liveness_lease_expires_at < now) | `unhealthy` | -| admin_status is `disabled` | `unhealthy` | -| Client reports stream_connected = false | `degraded` | -| Client reports last_heartbeat_ok = false | `degraded` | -| reconnect_count > 5 (since last check) | `degraded` | -| No client diagnostics available | `degraded` | -| Otherwise | `healthy` | - -Each failing condition adds a human-readable string to the `issues` array. - -### Data Sources for the Endpoint - -| Section | Source | -| ---------- | --------------------------------------------------------------------- | -| `runner` | `runners` table row | -| `client` | `runners.diagnostics` JSONB (from last heartbeat) | -| `claims` | `consumer_claims` table where `runner_id = :id AND status = 'active'` | -| `dispatch` | `entity_dispatch_state` table where `active_runner_id = :id` | -| `health` | Derived from above | - -### Files Changed - -**New:** - -- `packages/agents-server/drizzle/0007_runner_diagnostics_and_principal.sql` — adds `diagnostics` JSONB column, renames `owner_user_id` → `owner_principal` on runners table - -**Modified:** - -- `packages/agents-runtime/src/pull-wake-runner.ts` — add diagnostics tracking, `getHealth()` method, report diagnostics in heartbeat -- `packages/agents-server/src/db/schema.ts` — rename `ownerUserId` → `ownerPrincipal`, add `diagnostics` column -- `packages/agents-server/src/entity-registry.ts` — rename `ownerUserId` params → `ownerPrincipal`; extend `heartbeatRunner` to store diagnostics; add `getActiveClaimsForRunner` and `getDispatchStatsForRunner` queries -- `packages/agents-server/src/electric-agents-types.ts` — rename `owner_user_id` → `owner_principal` on `ElectricAgentsRunner`, add `diagnostics` field, add health response types -- `packages/agents-server/src/routing/runners-router.ts` — rename all `owner_user_id` references → `owner_principal`; switch auth checks to use `ctx.principal.url`; add `GET /:id/health` route; extend heartbeat body schema with optional `diagnostics` -- `packages/agents/src/server.ts` — update `BuiltinAgentsServer` registration to use `ownerPrincipal` -- `packages/agents-desktop/src/main.ts` — update runner registration to use `owner_principal` - -### Testing - -- Unit test for health status derivation logic (pure function) -- Unit test for `getHealth()` on the PullWakeRunner -- Integration test extending the existing `horton-pull-wake-e2e.test.ts` to call the health endpoint after dispatch and verify diagnostics are populated diff --git a/packages/agents-server/src/stream-client.ts b/packages/agents-server/src/stream-client.ts index 482bbcefa6..20e8612540 100644 --- a/packages/agents-server/src/stream-client.ts +++ b/packages/agents-server/src/stream-client.ts @@ -219,16 +219,41 @@ export class StreamClient { return headers } + private subscriptionServiceId(): string | null { + const url = new URL(this.baseUrl) + const match = /^(.*)\/v1\/stream\/([^/]+)\/?$/.exec(url.pathname) + return match ? decodeURIComponent(match[2]!) : null + } + private backendSubscriptionPath(path: string): string { - return normalizeSubscriptionPath(path) + const normalized = normalizeSubscriptionPath(path) + const serviceId = this.subscriptionServiceId() + if (!serviceId) return normalized + if (normalized === serviceId || normalized.startsWith(`${serviceId}/`)) { + return normalized + } + return `${serviceId}/${normalized}` } private runtimeSubscriptionPath(path: string): string { - return normalizeSubscriptionPath(path) + const normalized = normalizeSubscriptionPath(path) + const serviceId = this.subscriptionServiceId() + if (!serviceId) return normalized + return normalized.startsWith(`${serviceId}/`) + ? normalized.slice(serviceId.length + 1) + : normalized } private subscriptionUrl(subscriptionId: string): string { const url = new URL(this.baseUrl) + const match = /^(.*)\/v1\/stream\/([^/]+)\/?$/.exec(url.pathname) + if (match) { + const [, prefix = ``, serviceId] = match + url.pathname = `${prefix}/v1/stream/__ds/subscriptions/${encodeURIComponent(subscriptionId)}` + url.searchParams.set(`service`, decodeURIComponent(serviceId!)) + return url.toString() + } + url.pathname = `${url.pathname.replace(/\/+$/, ``)}/__ds/subscriptions/${encodeURIComponent(subscriptionId)}` return url.toString() } diff --git a/packages/agents-server/test/horton-pull-wake-e2e.test.ts b/packages/agents-server/test/horton-pull-wake-e2e.test.ts index f806bf9512..367b83716d 100644 --- a/packages/agents-server/test/horton-pull-wake-e2e.test.ts +++ b/packages/agents-server/test/horton-pull-wake-e2e.test.ts @@ -81,12 +81,12 @@ function subscriptionUrl( const match = /^(.*)\/v1\/stream\/([^/]+)\/?$/.exec(url.pathname) if (match) { const [, prefix = ``, serviceId] = match - url.pathname = `${prefix}/v1/stream-meta/subscriptions/${encodeURIComponent(subscriptionId)}` + url.pathname = `${prefix}/v1/stream/__ds/subscriptions/${encodeURIComponent(subscriptionId)}` url.searchParams.set(`service`, decodeURIComponent(serviceId!)) return url.toString() } - url.pathname = `${url.pathname.replace(/\/+$/, ``)}/v1/stream-meta/subscriptions/${encodeURIComponent(subscriptionId)}` + url.pathname = `${url.pathname.replace(/\/+$/, ``)}/__ds/subscriptions/${encodeURIComponent(subscriptionId)}` return url.toString() } diff --git a/packages/agents-server/test/pull-wake-subscription-stack.test.ts b/packages/agents-server/test/pull-wake-subscription-stack.test.ts index dc51929efb..4302649f82 100644 --- a/packages/agents-server/test/pull-wake-subscription-stack.test.ts +++ b/packages/agents-server/test/pull-wake-subscription-stack.test.ts @@ -128,7 +128,7 @@ describe(`pull-wake subscription stack`, () => { url: `/principal/user%3Aowner%40example.com`, }, publicUrl: `http://agents.local`, - durableStreamsUrl: dsServer.url, + durableStreamsUrl: streamBaseUrl, entityBridgeManager: { beginClientRead: async () => null, touchByStreamPath: async () => undefined, diff --git a/packages/agents-server/test/stream-client.test.ts b/packages/agents-server/test/stream-client.test.ts index 5378b82666..ba9004a7e1 100644 --- a/packages/agents-server/test/stream-client.test.ts +++ b/packages/agents-server/test/stream-client.test.ts @@ -114,13 +114,13 @@ describe(`StreamClient`, () => { ) expect(fetchMock).toHaveBeenCalledWith( - `http://127.0.0.1:4545/v1/stream/tenant-a/__ds/subscriptions/sub-1`, + `http://127.0.0.1:4545/v1/stream/__ds/subscriptions/sub-1?service=tenant-a`, expect.objectContaining({ method: `PUT` }) ) const [, init] = fetchMock.mock.calls[0]! expect(JSON.parse(init?.body as string)).toEqual({ type: `webhook`, - pattern: `chat/**`, + pattern: `tenant-a/chat/**`, webhook: { url: `http://agent.local/webhook` }, description: `test subscription`, }) From 761876de8e480da7334f251d54a24ed8b0e1e05b Mon Sep 17 00:00:00 2001 From: Ilia Borovitinov Date: Mon, 18 May 2026 21:00:27 +0300 Subject: [PATCH 27/37] fix: treat durable streams urls as opaque prefixes --- .changeset/harden-pull-wake-runner.md | 2 +- packages/agents-server/src/host.ts | 9 +-- packages/agents-server/src/routing/context.ts | 2 +- .../durable-streams-routing-adapter.ts | 13 ++- packages/agents-server/src/runtime.ts | 11 +-- packages/agents-server/src/server.ts | 35 ++++---- .../agents-server/src/standalone-runtime.ts | 11 +-- packages/agents-server/src/stream-client.ts | 79 +++++-------------- .../agents-server/test/conformance.test.ts | 9 ++- .../test/electric-agents-routes.test.ts | 8 +- .../test/entity-lifecycle.test.ts | 3 +- .../test/horton-pull-wake-e2e.test.ts | 8 +- .../test/horton-spawn-worker.test.ts | 6 +- .../test/horton-title-generation.test.ts | 4 +- .../test/pull-wake-subscription-stack.test.ts | 9 ++- .../test/scheduler-integration.test.ts | 8 +- .../test/server-claim-write-token.test.ts | 3 +- .../agents-server/test/server-start.test.ts | 2 - .../test/stream-client-fork.test.ts | 3 +- .../agents-server/test/stream-client.test.ts | 38 ++------- packages/agents-server/test/test-utils.ts | 6 ++ .../agents-server/test/wake-registry.test.ts | 8 +- 22 files changed, 114 insertions(+), 163 deletions(-) diff --git a/.changeset/harden-pull-wake-runner.md b/.changeset/harden-pull-wake-runner.md index 04888c3b5a..5888dffdf7 100644 --- a/.changeset/harden-pull-wake-runner.md +++ b/.changeset/harden-pull-wake-runner.md @@ -4,4 +4,4 @@ '@electric-ax/agents': patch --- -Harden pull-wake runner lifecycle with a state machine, concurrent claim limits (`maxConcurrentClaims`), heartbeat-driven stream resets, and exponential reconnect backoff (1s–30s). Add granular `status` field to `PullWakeRunnerHealth` (`stopped | starting | connecting | streaming | reconnecting | stopping`). The `onError` callback is now reporting-only (`(Error) => void`) — it can no longer control runner lifecycle. `stop()` rethrows `drainWakes` errors so callers observe wake handler failures. Event-driven heartbeat throttling avoids stale diagnostics between fixed-interval heartbeats. Service-scoped Durable Streams clients now route subscription control through `__ds` while preserving tenant-prefixed stream names, so pull-wake subscriptions emit runner wake events correctly. +Harden pull-wake runner lifecycle with a state machine, concurrent claim limits (`maxConcurrentClaims`), heartbeat-driven stream resets, and exponential reconnect backoff (1s–30s). Add granular `status` field to `PullWakeRunnerHealth` (`stopped | starting | connecting | streaming | reconnecting | stopping`). The `onError` callback is now reporting-only (`(Error) => void`) — it can no longer control runner lifecycle. `stop()` rethrows `drainWakes` errors so callers observe wake handler failures. Event-driven heartbeat throttling avoids stale diagnostics between fixed-interval heartbeats. Durable Streams clients now append stream and `__ds` subscription control paths to the configured backend URL prefix without inferring a `/v1/stream` layout, so pull-wake subscriptions work behind arbitrary DS backend prefixes. diff --git a/packages/agents-server/src/host.ts b/packages/agents-server/src/host.ts index b7e569aeb5..de556f8bdd 100644 --- a/packages/agents-server/src/host.ts +++ b/packages/agents-server/src/host.ts @@ -3,7 +3,7 @@ import { PostgresRegistry } from './entity-registry.js' import { EntityProjector } from './entity-projector.js' import { ElectricAgentsTenantRuntime } from './runtime.js' import { PostgresSchedulerClient, Scheduler } from './scheduler.js' -import { StreamClient, durableStreamsServiceUrl } from './stream-client.js' +import { StreamClient } from './stream-client.js' import { TagStreamOutboxDrainer } from './tag-stream-outbox-drainer.js' import { DEFAULT_TENANT_ID, UnregisteredTenantError } from './tenant.js' import { WakeRegistry } from './wake-registry.js' @@ -313,10 +313,9 @@ export class AgentsHost { private createStreamClient(config: AgentsHostTenantConfig): StreamClient { if (config.streamClient) return config.streamClient if (config.durableStreamsUrl) { - return new StreamClient( - durableStreamsServiceUrl(config.durableStreamsUrl, config.serviceId), - { bearer: config.durableStreamsBearer } - ) + return new StreamClient(config.durableStreamsUrl, { + bearer: config.durableStreamsBearer, + }) } throw new Error( `AgentsHost tenant "${config.serviceId}" must provide a streamClient or durableStreamsUrl` diff --git a/packages/agents-server/src/routing/context.ts b/packages/agents-server/src/routing/context.ts index 2a0b1971f9..5b5b6b948a 100644 --- a/packages/agents-server/src/routing/context.ts +++ b/packages/agents-server/src/routing/context.ts @@ -19,7 +19,7 @@ export interface TenantContext { principal: Principal publicUrl: string localUrl?: string - /** Resolved Durable Streams root URL for this tenant. */ + /** Durable Streams backend URL prefix. Stream and control paths are appended as-is. */ durableStreamsUrl: string durableStreamsBearer?: DurableStreamsBearerProvider durableStreamsRouting?: DurableStreamsRoutingAdapter diff --git a/packages/agents-server/src/routing/durable-streams-routing-adapter.ts b/packages/agents-server/src/routing/durable-streams-routing-adapter.ts index d557b2e6e1..38d258a339 100644 --- a/packages/agents-server/src/routing/durable-streams-routing-adapter.ts +++ b/packages/agents-server/src/routing/durable-streams-routing-adapter.ts @@ -12,12 +12,11 @@ export interface DurableStreamsRoutingAdapter { } function appendSearch(target: URL, source: URL): URL { - target.search = source.search - return target -} - -function removeServiceQuery(target: URL): URL { - target.searchParams.delete(`service`) + source.searchParams.forEach((value, key) => { + if (key !== `service`) { + target.searchParams.append(key, value) + } + }) return target } @@ -32,7 +31,7 @@ function appendRequestPathToStreamRoot(input: DurableStreamsRoutingInput): URL { target.pathname = path ? `${withoutTrailingSlash(target.pathname)}/${path}` : withoutTrailingSlash(target.pathname) - return removeServiceQuery(appendSearch(target, incomingUrl)) + return appendSearch(target, incomingUrl) } export const streamRootDurableStreamsRoutingAdapter: DurableStreamsRoutingAdapter = diff --git a/packages/agents-server/src/runtime.ts b/packages/agents-server/src/runtime.ts index 1715aa3aa1..29038e5b6c 100644 --- a/packages/agents-server/src/runtime.ts +++ b/packages/agents-server/src/runtime.ts @@ -11,7 +11,7 @@ import { import { SchemaValidator } from './electric-agents/schema-validator.js' import { serverLog } from './utils/log.js' import { isPermanentElectricAgentsError } from './scheduler.js' -import { StreamClient, durableStreamsServiceUrl } from './stream-client.js' +import { StreamClient } from './stream-client.js' import { DEFAULT_TENANT_ID } from './tenant.js' import type { DrizzleDB } from './db/index.js' import type { EntityBridgeCoordinator } from './entity-bridge-manager.js' @@ -63,12 +63,9 @@ export class ElectricAgentsTenantRuntime { if (options.streamClient) { this.streamClient = options.streamClient } else if (options.durableStreamsUrl) { - this.streamClient = new StreamClient( - durableStreamsServiceUrl(options.durableStreamsUrl, this.serviceId, { - scope: `stream-root`, - }), - { bearer: options.durableStreamsBearer } - ) + this.streamClient = new StreamClient(options.durableStreamsUrl, { + bearer: options.durableStreamsBearer, + }) } else { throw new Error(`Either durableStreamsUrl or streamClient is required`) } diff --git a/packages/agents-server/src/server.ts b/packages/agents-server/src/server.ts index 43714c342e..dfd9da2ba4 100644 --- a/packages/agents-server/src/server.ts +++ b/packages/agents-server/src/server.ts @@ -9,7 +9,7 @@ import { import { createDb, runMigrations } from './db/index.js' import { ossServerRouter } from './routing/oss-server-router.js' import { startStandaloneAgentsRuntime } from './standalone-runtime.js' -import { StreamClient, durableStreamsServiceUrl } from './stream-client.js' +import { StreamClient } from './stream-client.js' import { DEFAULT_TENANT_ID } from './tenant.js' import { getDevPrincipal, getPrincipalFromRequest } from './principal.js' import { apiError } from './electric-agents-http.js' @@ -120,6 +120,16 @@ function createMockAgentBootstrap(options: { return { runtime, registry } } +function durableStreamTestServerBackendUrl(origin: string): string { + // DurableStreamTestServer.start() returns the HTTP origin, while the + // reference server's stream backend is mounted under /v1/stream. + // User-provided durableStreamsUrl values are already backend prefixes and + // are passed through unchanged. + const url = new URL(origin) + url.pathname = `${url.pathname.replace(/\/+$/, ``)}/v1/stream` + return url.toString().replace(/\/+$/, ``) +} + export class ElectricAgentsServer { private server?: Server private electricAgentsManager?: StartedStandaloneAgentsRuntime[`manager`] @@ -143,12 +153,9 @@ export class ElectricAgentsServer { } this.options = options this.streamClient = options.durableStreamsUrl - ? new StreamClient( - durableStreamsServiceUrl(options.durableStreamsUrl, this.tenantId, { - scope: `stream-root`, - }), - { bearer: options.durableStreamsBearer } - ) + ? new StreamClient(options.durableStreamsUrl, { + bearer: options.durableStreamsBearer, + }) : null! } @@ -185,13 +192,11 @@ export class ElectricAgentsServer { serverLog.info( `[agent-server] durable streams server started at ${streamsUrl}` ) - this.options.durableStreamsUrl = streamsUrl - this.streamClient = new StreamClient( - durableStreamsServiceUrl(streamsUrl, this.tenantId, { - scope: `stream-root`, - }), - { bearer: this.options.durableStreamsBearer } - ) + this.options.durableStreamsUrl = + durableStreamTestServerBackendUrl(streamsUrl) + this.streamClient = new StreamClient(this.options.durableStreamsUrl, { + bearer: this.options.durableStreamsBearer, + }) } this.streamsAgent = new Agent({ @@ -404,7 +409,7 @@ export class ElectricAgentsServer { principal, publicUrl: this.publicUrl, localUrl: this._url, - durableStreamsUrl: this.streamClient.baseUrl, + durableStreamsUrl: this.options.durableStreamsUrl!, durableStreamsBearer: this.options.durableStreamsBearer, durableStreamsRouting: this.options.durableStreamsRouting, durableStreamsDispatcher: this.streamsAgent, diff --git a/packages/agents-server/src/standalone-runtime.ts b/packages/agents-server/src/standalone-runtime.ts index 3e97160b79..5f94c5b856 100644 --- a/packages/agents-server/src/standalone-runtime.ts +++ b/packages/agents-server/src/standalone-runtime.ts @@ -4,7 +4,7 @@ import { EntityBridgeManager } from './entity-bridge-manager.js' import { serverLog } from './utils/log.js' import { ElectricAgentsTenantRuntime } from './runtime.js' import { Scheduler } from './scheduler.js' -import { StreamClient, durableStreamsServiceUrl } from './stream-client.js' +import { StreamClient } from './stream-client.js' import { TagStreamOutboxDrainer } from './tag-stream-outbox-drainer.js' import { DEFAULT_TENANT_ID } from './tenant.js' import { WakeRegistry } from './wake-registry.js' @@ -57,12 +57,9 @@ export async function startStandaloneAgentsRuntime( const streamClient = options.streamClient ?? (options.durableStreamsUrl - ? new StreamClient( - durableStreamsServiceUrl(options.durableStreamsUrl, serviceId, { - scope: `stream-root`, - }), - { bearer: options.durableStreamsBearer } - ) + ? new StreamClient(options.durableStreamsUrl, { + bearer: options.durableStreamsBearer, + }) : undefined) if (!streamClient) { throw new Error(`Either durableStreamsUrl or streamClient is required`) diff --git a/packages/agents-server/src/stream-client.ts b/packages/agents-server/src/stream-client.ts index 20e8612540..264413d331 100644 --- a/packages/agents-server/src/stream-client.ts +++ b/packages/agents-server/src/stream-client.ts @@ -14,8 +14,6 @@ export interface StreamClientOptions { bearer?: DurableStreamsBearerProvider } -type DurableStreamsUrlScope = `service` | `stream-root` - export interface StreamAppendResult { offset: string } @@ -131,6 +129,16 @@ export async function applyDurableStreamsBearer( } } +function appendPathToBaseUrl(baseUrl: string, path: string): string { + const url = new URL(baseUrl) + const basePath = url.pathname.replace(/\/+$/, ``) + const childPath = path.replace(/^\/+/, ``) + url.pathname = childPath + ? `${basePath === `/` ? `` : basePath}/${childPath}` + : basePath || `/` + return url.toString().replace(/\/+$/, ``) +} + function durableStreamsBearerHeaders( bearer: DurableStreamsBearerProvider | undefined ): HeadersRecord | undefined { @@ -141,33 +149,6 @@ function durableStreamsBearerHeaders( } } -export function durableStreamsServiceUrl( - baseUrl: string, - serviceId: string, - options: { scope?: DurableStreamsUrlScope } = {} -): string { - const url = new URL(baseUrl) - if (/\/v1\/streams\/[^/]+\/?$/.test(url.pathname)) { - return baseUrl.replace(/\/+$/, ``) - } - if (/\/v1\/stream\/[^/]+\/?$/.test(url.pathname)) { - return baseUrl.replace(/\/+$/, ``) - } - const scope = options.scope ?? `service` - const encodedServiceId = encodeURIComponent(serviceId) - const path = url.pathname.replace(/\/+$/, ``) || `/` - if (path.endsWith(`/v1/streams`)) { - url.pathname = `${path}/${encodedServiceId}` - } else if (path.endsWith(`/v1/stream`)) { - url.pathname = scope === `service` ? `${path}/${encodedServiceId}` : path - } else if (scope === `stream-root`) { - url.pathname = `${path === `/` ? `` : path}/v1/stream` - } else { - url.pathname = `${path === `/` ? `` : path}/v1/stream/${encodedServiceId}` - } - return url.toString().replace(/\/+$/, ``) -} - function isNotFoundError(err: unknown): boolean { return ( (err instanceof DurableStreamError && err.code === ErrCodeNotFound) || @@ -201,7 +182,7 @@ export class StreamClient { ) {} private streamUrl(path: string): string { - return `${this.baseUrl}${path}` + return appendPathToBaseUrl(this.baseUrl, path) } private streamHeaders(): HeadersRecord | undefined { @@ -219,43 +200,19 @@ export class StreamClient { return headers } - private subscriptionServiceId(): string | null { - const url = new URL(this.baseUrl) - const match = /^(.*)\/v1\/stream\/([^/]+)\/?$/.exec(url.pathname) - return match ? decodeURIComponent(match[2]!) : null - } - private backendSubscriptionPath(path: string): string { - const normalized = normalizeSubscriptionPath(path) - const serviceId = this.subscriptionServiceId() - if (!serviceId) return normalized - if (normalized === serviceId || normalized.startsWith(`${serviceId}/`)) { - return normalized - } - return `${serviceId}/${normalized}` + return normalizeSubscriptionPath(path) } private runtimeSubscriptionPath(path: string): string { - const normalized = normalizeSubscriptionPath(path) - const serviceId = this.subscriptionServiceId() - if (!serviceId) return normalized - return normalized.startsWith(`${serviceId}/`) - ? normalized.slice(serviceId.length + 1) - : normalized + return normalizeSubscriptionPath(path) } private subscriptionUrl(subscriptionId: string): string { - const url = new URL(this.baseUrl) - const match = /^(.*)\/v1\/stream\/([^/]+)\/?$/.exec(url.pathname) - if (match) { - const [, prefix = ``, serviceId] = match - url.pathname = `${prefix}/v1/stream/__ds/subscriptions/${encodeURIComponent(subscriptionId)}` - url.searchParams.set(`service`, decodeURIComponent(serviceId!)) - return url.toString() - } - - url.pathname = `${url.pathname.replace(/\/+$/, ``)}/__ds/subscriptions/${encodeURIComponent(subscriptionId)}` - return url.toString() + return appendPathToBaseUrl( + this.baseUrl, + `/__ds/subscriptions/${encodeURIComponent(subscriptionId)}` + ) } private subscriptionChildUrl( @@ -295,7 +252,7 @@ export class StreamClient { }) const headers: Record = { 'content-type': `application/json`, - 'Stream-Forked-From': sourcePath, + 'Stream-Forked-From': new URL(this.streamUrl(sourcePath)).pathname, } injectTraceHeaders(headers) diff --git a/packages/agents-server/test/conformance.test.ts b/packages/agents-server/test/conformance.test.ts index 868ed4b475..2c4f879fde 100644 --- a/packages/agents-server/test/conformance.test.ts +++ b/packages/agents-server/test/conformance.test.ts @@ -15,6 +15,7 @@ import { TEST_POSTGRES_URL, resetElectricAgentsTestBackend, } from './test-backend' +import { durableStreamTestServerUrl } from './test-utils' const CLI_BIN = path.resolve( __dirname, @@ -39,7 +40,7 @@ describe(`Electric Agents Entity Runtime`, () => { await Promise.all([resetElectricAgentsTestBackend(), dsServer.start()]) electricAgentsServer = new ElectricAgentsServer({ - durableStreamsUrl: dsServer.url, + durableStreamsUrl: durableStreamTestServerUrl(dsServer.url), port: 0, postgresUrl: TEST_POSTGRES_URL, electricUrl: TEST_ELECTRIC_URL, @@ -68,7 +69,7 @@ describeCli(`Electric Agents CLI`, () => { await Promise.all([resetElectricAgentsTestBackend(), dsServer.start()]) electricAgentsServer = new ElectricAgentsServer({ - durableStreamsUrl: dsServer.url, + durableStreamsUrl: durableStreamTestServerUrl(dsServer.url), port: 0, postgresUrl: TEST_POSTGRES_URL, electricUrl: TEST_ELECTRIC_URL, @@ -97,7 +98,7 @@ describe(`Electric Agents Mock Agent`, () => { await Promise.all([resetElectricAgentsTestBackend(), dsServer.start()]) electricAgentsServer = new ElectricAgentsServer({ - durableStreamsUrl: dsServer.url, + durableStreamsUrl: durableStreamTestServerUrl(dsServer.url), port: 0, mockStreamFn: MOCK_STREAM_FN, postgresUrl: TEST_POSTGRES_URL, @@ -127,7 +128,7 @@ describeCli(`Electric Agents CLI with Mock Agent`, () => { await Promise.all([resetElectricAgentsTestBackend(), dsServer.start()]) electricAgentsServer = new ElectricAgentsServer({ - durableStreamsUrl: dsServer.url, + durableStreamsUrl: durableStreamTestServerUrl(dsServer.url), port: 0, mockStreamFn: MOCK_STREAM_FN, postgresUrl: TEST_POSTGRES_URL, diff --git a/packages/agents-server/test/electric-agents-routes.test.ts b/packages/agents-server/test/electric-agents-routes.test.ts index 9d5afd46ae..8153b578f4 100644 --- a/packages/agents-server/test/electric-agents-routes.test.ts +++ b/packages/agents-server/test/electric-agents-routes.test.ts @@ -267,7 +267,7 @@ describe(`ElectricAgentsRoutes shared-state streams`, () => { createRequest(`PUT`, `/_electric/shared-state/board-1`), { service: `test`, - durableStreamsUrl: `http://durable.local/v1/stream/test`, + durableStreamsUrl: `http://durable.local/custom/ds-prefix`, isShuttingDown: () => false, } as unknown as TenantContext ) @@ -276,7 +276,7 @@ describe(`ElectricAgentsRoutes shared-state streams`, () => { expect(fetchSpy).toHaveBeenCalledOnce() const [url, init] = fetchSpy.mock.calls[0]! expect(String(url)).toBe( - `http://durable.local/v1/stream/test/_electric/shared-state/board-1` + `http://durable.local/custom/ds-prefix/_electric/shared-state/board-1` ) expect(init).toMatchObject({ method: `PUT` }) } finally { @@ -294,7 +294,7 @@ describe(`ElectricAgentsRoutes shared-state streams`, () => { createRequest(`GET`, `/__ds/subscriptions/sub-1`), { service: `test`, - durableStreamsUrl: `http://durable.local/v1/stream/test`, + durableStreamsUrl: `http://durable.local/custom/ds-prefix`, isShuttingDown: () => false, } as unknown as TenantContext ) @@ -303,7 +303,7 @@ describe(`ElectricAgentsRoutes shared-state streams`, () => { expect(fetchSpy).toHaveBeenCalledOnce() const [url, init] = fetchSpy.mock.calls[0]! expect(String(url)).toBe( - `http://durable.local/v1/stream/test/__ds/subscriptions/sub-1` + `http://durable.local/custom/ds-prefix/__ds/subscriptions/sub-1` ) expect(init).toMatchObject({ method: `GET` }) } finally { diff --git a/packages/agents-server/test/entity-lifecycle.test.ts b/packages/agents-server/test/entity-lifecycle.test.ts index f4ff58bc27..c637ad64af 100644 --- a/packages/agents-server/test/entity-lifecycle.test.ts +++ b/packages/agents-server/test/entity-lifecycle.test.ts @@ -6,6 +6,7 @@ import { TEST_POSTGRES_URL, resetElectricAgentsTestBackend, } from './test-backend' +import { durableStreamTestServerUrl } from './test-utils' describe(`entity lifecycle`, () => { let dsServer: DurableStreamTestServer | null = null @@ -21,7 +22,7 @@ describe(`entity lifecycle`, () => { await Promise.all([resetElectricAgentsTestBackend(), dsServer.start()]) electricAgentsServer = new ElectricAgentsServer({ - durableStreamsUrl: dsServer.url, + durableStreamsUrl: durableStreamTestServerUrl(dsServer.url), port: 0, postgresUrl: TEST_POSTGRES_URL, electricUrl: TEST_ELECTRIC_URL, diff --git a/packages/agents-server/test/horton-pull-wake-e2e.test.ts b/packages/agents-server/test/horton-pull-wake-e2e.test.ts index 367b83716d..c7bc259d92 100644 --- a/packages/agents-server/test/horton-pull-wake-e2e.test.ts +++ b/packages/agents-server/test/horton-pull-wake-e2e.test.ts @@ -4,7 +4,11 @@ import { DurableStreamTestServer } from '@durable-streams/server' import { BuiltinAgentsServer } from '../../agents/src/server' import { ElectricAgentsServer } from '../src/server' import { parsePrincipalKey } from '../src/principal' -import { readStreamEvents, waitFor } from './test-utils' +import { + durableStreamTestServerUrl, + readStreamEvents, + waitFor, +} from './test-utils' import { TEST_POSTGRES_URL, resetElectricAgentsTestBackend, @@ -260,7 +264,7 @@ describe(`pull-wake Horton e2e with mocked LLM`, () => { }) await Promise.all([resetElectricAgentsTestBackend(), dsServer.start()]) electricAgentsServer = new ElectricAgentsServer({ - durableStreamsUrl: dsServer.url, + durableStreamsUrl: durableStreamTestServerUrl(dsServer.url), port: 0, postgresUrl: TEST_POSTGRES_URL, electricUrl: undefined, diff --git a/packages/agents-server/test/horton-spawn-worker.test.ts b/packages/agents-server/test/horton-spawn-worker.test.ts index 0e441e7148..cba4d8d173 100644 --- a/packages/agents-server/test/horton-spawn-worker.test.ts +++ b/packages/agents-server/test/horton-spawn-worker.test.ts @@ -2,7 +2,7 @@ import { afterAll, beforeAll, describe, expect, it } from 'vitest' import { DurableStreamTestServer } from '@durable-streams/server' import { BuiltinAgentsServer } from '../../agents/src/server' import { ElectricAgentsServer } from '../src/server' -import { waitForStreamEvents } from './test-utils' +import { durableStreamTestServerUrl, waitForStreamEvents } from './test-utils' import { TEST_ELECTRIC_URL, TEST_POSTGRES_URL, @@ -25,7 +25,7 @@ describe.skipIf(!process.env.ANTHROPIC_API_KEY)( }) await Promise.all([resetElectricAgentsTestBackend(), dsServer.start()]) electricAgentsServer = new ElectricAgentsServer({ - durableStreamsUrl: dsServer.url, + durableStreamsUrl: durableStreamTestServerUrl(dsServer.url), port: 0, postgresUrl: TEST_POSTGRES_URL, electricUrl: TEST_ELECTRIC_URL, @@ -79,7 +79,7 @@ describe.skipIf(!process.env.ANTHROPIC_API_KEY)( expect(sendRes.status).toBe(204) const events = await waitForStreamEvents( - dsServer.url, + durableStreamTestServerUrl(dsServer.url), horton.streams.main, (currentEvents) => currentEvents.some((event) => { diff --git a/packages/agents-server/test/horton-title-generation.test.ts b/packages/agents-server/test/horton-title-generation.test.ts index 625bb9c897..19454171df 100644 --- a/packages/agents-server/test/horton-title-generation.test.ts +++ b/packages/agents-server/test/horton-title-generation.test.ts @@ -2,7 +2,7 @@ import { afterAll, beforeAll, describe, expect, it } from 'vitest' import { DurableStreamTestServer } from '@durable-streams/server' import { BuiltinAgentsServer } from '../../agents/src/server' import { ElectricAgentsServer } from '../src/server' -import { waitFor } from './test-utils' +import { durableStreamTestServerUrl, waitFor } from './test-utils' import { TEST_ELECTRIC_URL, TEST_POSTGRES_URL, @@ -25,7 +25,7 @@ describe.skipIf(!process.env.ANTHROPIC_API_KEY)( }) await Promise.all([resetElectricAgentsTestBackend(), dsServer.start()]) electricAgentsServer = new ElectricAgentsServer({ - durableStreamsUrl: dsServer.url, + durableStreamsUrl: durableStreamTestServerUrl(dsServer.url), port: 0, postgresUrl: TEST_POSTGRES_URL, electricUrl: TEST_ELECTRIC_URL, diff --git a/packages/agents-server/test/pull-wake-subscription-stack.test.ts b/packages/agents-server/test/pull-wake-subscription-stack.test.ts index 4302649f82..241cde77db 100644 --- a/packages/agents-server/test/pull-wake-subscription-stack.test.ts +++ b/packages/agents-server/test/pull-wake-subscription-stack.test.ts @@ -4,7 +4,8 @@ import { stream } from '@durable-streams/client' import { DurableStreamTestServer } from '@durable-streams/server' import { afterEach, describe, expect, it } from 'vitest' import { globalRouter } from '../src/routing/global-router' -import { StreamClient, durableStreamsServiceUrl } from '../src/stream-client' +import { StreamClient } from '../src/stream-client' +import { durableStreamTestServerUrl } from './test-utils' import type { Server } from 'node:http' import type { TenantContext } from '../src/routing/context' @@ -49,7 +50,7 @@ describe(`pull-wake subscription stack`, () => { webhooks: true, }) await dsServer.start() - const streamBaseUrl = durableStreamsServiceUrl(dsServer.url, `default`) + const streamBaseUrl = durableStreamTestServerUrl(dsServer.url) const client = new StreamClient(streamBaseUrl) await client.ensure(`/runners/runner-1/wake`, { @@ -77,7 +78,7 @@ describe(`pull-wake subscription stack`, () => { expect.objectContaining({ type: `wake`, subscription_id: `runner:runner-1:one`, - stream: `default/horton/one/main`, + stream: `horton/one/main`, generation: 1, }), ]) @@ -104,7 +105,7 @@ describe(`pull-wake subscription stack`, () => { webhooks: true, }) await dsServer.start() - const streamBaseUrl = durableStreamsServiceUrl(dsServer.url, `default`) + const streamBaseUrl = durableStreamTestServerUrl(dsServer.url) const client = new StreamClient(streamBaseUrl) await client.ensure(`/runners/runner-1/wake`, { contentType: `application/json`, diff --git a/packages/agents-server/test/scheduler-integration.test.ts b/packages/agents-server/test/scheduler-integration.test.ts index e4971d7f8b..af5ba79fbb 100644 --- a/packages/agents-server/test/scheduler-integration.test.ts +++ b/packages/agents-server/test/scheduler-integration.test.ts @@ -2,7 +2,11 @@ import { afterAll, beforeAll, describe, expect, it } from 'vitest' import { getCronStreamPath } from '@electric-ax/agents-runtime' import { DurableStreamTestServer } from '@durable-streams/server' import { ElectricAgentsServer } from '../src/server' -import { readStreamEvents, waitFor } from './test-utils' +import { + durableStreamTestServerUrl, + readStreamEvents, + waitFor, +} from './test-utils' import { TEST_ELECTRIC_URL, TEST_POSTGRES_URL, @@ -17,7 +21,7 @@ describe(`Scheduler Integration`, () => { async function startElectricAgentsServer(): Promise { electricAgentsServer = new ElectricAgentsServer({ - durableStreamsUrl: dsServer.url, + durableStreamsUrl: durableStreamTestServerUrl(dsServer.url), port: 0, postgresUrl: TEST_POSTGRES_URL, electricUrl: TEST_ELECTRIC_URL, diff --git a/packages/agents-server/test/server-claim-write-token.test.ts b/packages/agents-server/test/server-claim-write-token.test.ts index f2bb3d868d..87d3b9f200 100644 --- a/packages/agents-server/test/server-claim-write-token.test.ts +++ b/packages/agents-server/test/server-claim-write-token.test.ts @@ -8,6 +8,7 @@ import { TEST_POSTGRES_URL, resetElectricAgentsTestBackend, } from './test-backend' +import { durableStreamTestServerUrl } from './test-utils' import type { Server } from 'node:http' describe(`Claim-scoped write tokens`, () => { @@ -19,7 +20,7 @@ describe(`Claim-scoped write tokens`, () => { async function startElectricAgentsServer(): Promise { electricAgentsServer = new ElectricAgentsServer({ - durableStreamsUrl: dsServer.url, + durableStreamsUrl: durableStreamTestServerUrl(dsServer.url), port: 0, postgresUrl: TEST_POSTGRES_URL, electricUrl: TEST_ELECTRIC_URL, diff --git a/packages/agents-server/test/server-start.test.ts b/packages/agents-server/test/server-start.test.ts index 4bc6feb898..6f56d9c063 100644 --- a/packages/agents-server/test/server-start.test.ts +++ b/packages/agents-server/test/server-start.test.ts @@ -179,8 +179,6 @@ vi.mock(`drizzle-orm`, () => ({ })) vi.mock(`../src/stream-client`, () => ({ - durableStreamsServiceUrl: (baseUrl: string, serviceId: string) => - `${baseUrl.replace(/\/+$/, ``)}/v1/stream/${encodeURIComponent(serviceId)}`, StreamClient: class MockStreamClient { exists(): Promise { return streamExistsMock() diff --git a/packages/agents-server/test/stream-client-fork.test.ts b/packages/agents-server/test/stream-client-fork.test.ts index 9622d7e6cc..d50acac5a1 100644 --- a/packages/agents-server/test/stream-client-fork.test.ts +++ b/packages/agents-server/test/stream-client-fork.test.ts @@ -2,6 +2,7 @@ import { afterAll, beforeAll, describe, expect, it } from 'vitest' import { DurableStreamTestServer } from '@durable-streams/server' import { StreamClient } from '../src/stream-client' +import { durableStreamTestServerUrl } from './test-utils' describe(`StreamClient.fork`, () => { let dsServer: DurableStreamTestServer | null = null @@ -14,7 +15,7 @@ describe(`StreamClient.fork`, () => { webhooks: true, }) const baseUrl = await dsServer.start() - client = new StreamClient(baseUrl) + client = new StreamClient(durableStreamTestServerUrl(baseUrl)) }) afterAll(async () => { diff --git a/packages/agents-server/test/stream-client.test.ts b/packages/agents-server/test/stream-client.test.ts index ba9004a7e1..6daa4aa173 100644 --- a/packages/agents-server/test/stream-client.test.ts +++ b/packages/agents-server/test/stream-client.test.ts @@ -1,6 +1,6 @@ import { beforeEach, describe, expect, it, vi } from 'vitest' -import { StreamClient, durableStreamsServiceUrl } from '../src/stream-client' +import { StreamClient } from '../src/stream-client' const { appendMock, @@ -97,13 +97,15 @@ describe(`StreamClient`, () => { await expect(client.exists(`/_cron/test`)).rejects.toBe(error) }) - it(`createSubscription uses the reserved __ds subscription contract`, async () => { + it(`createSubscription appends reserved __ds control paths to the opaque backend URL`, async () => { const fetchMock = vi.spyOn(globalThis, `fetch`).mockResolvedValueOnce( new Response(JSON.stringify({ subscription_id: `sub-1` }), { headers: { 'content-type': `application/json` }, }) ) - const client = new StreamClient(`http://127.0.0.1:4545/v1/stream/tenant-a`) + const client = new StreamClient( + `http://127.0.0.1:4545/custom/ds-prefix?tenant=tenant-a` + ) try { await client.createSubscription( @@ -114,13 +116,13 @@ describe(`StreamClient`, () => { ) expect(fetchMock).toHaveBeenCalledWith( - `http://127.0.0.1:4545/v1/stream/__ds/subscriptions/sub-1?service=tenant-a`, + `http://127.0.0.1:4545/custom/ds-prefix/__ds/subscriptions/sub-1?tenant=tenant-a`, expect.objectContaining({ method: `PUT` }) ) const [, init] = fetchMock.mock.calls[0]! expect(JSON.parse(init?.body as string)).toEqual({ type: `webhook`, - pattern: `tenant-a/chat/**`, + pattern: `chat/**`, webhook: { url: `http://agent.local/webhook` }, description: `test subscription`, }) @@ -245,29 +247,3 @@ describe(`StreamClient`, () => { } }) }) - -describe(`durableStreamsServiceUrl`, () => { - it(`derives a single-tenant stream root from a bare server origin`, () => { - expect( - durableStreamsServiceUrl(`http://127.0.0.1:4545`, `tenant-a`, { - scope: `stream-root`, - }) - ).toBe(`http://127.0.0.1:4545/v1/stream`) - }) - - it(`derives a service-scoped stream root for host tenant registrations`, () => { - expect(durableStreamsServiceUrl(`http://127.0.0.1:4545`, `tenant-a`)).toBe( - `http://127.0.0.1:4545/v1/stream/tenant-a` - ) - }) - - it(`preserves explicitly scoped stream roots`, () => { - expect( - durableStreamsServiceUrl( - `https://streams.test/v1/streams/tenant-a`, - `tenant-a`, - { scope: `stream-root` } - ) - ).toBe(`https://streams.test/v1/streams/tenant-a`) - }) -}) diff --git a/packages/agents-server/test/test-utils.ts b/packages/agents-server/test/test-utils.ts index 5cbd7a4368..12fdb01a5f 100644 --- a/packages/agents-server/test/test-utils.ts +++ b/packages/agents-server/test/test-utils.ts @@ -2,6 +2,12 @@ import { stream } from '@durable-streams/client' const debugTestTiming = process.env.ELECTRIC_AGENTS_DEBUG_TEST_TIMING === `1` +export function durableStreamTestServerUrl(origin: string): string { + const url = new URL(origin) + url.pathname = `${url.pathname.replace(/\/+$/, ``)}/v1/stream` + return url.toString().replace(/\/+$/, ``) +} + export async function timeStep( label: string, fn: () => Promise diff --git a/packages/agents-server/test/wake-registry.test.ts b/packages/agents-server/test/wake-registry.test.ts index 5de85f2adb..f7e05fefbd 100644 --- a/packages/agents-server/test/wake-registry.test.ts +++ b/packages/agents-server/test/wake-registry.test.ts @@ -16,7 +16,11 @@ import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest' import { EntityManager } from '../src/entity-manager' import { ElectricAgentsServer } from '../src/server' import { WakeRegistry } from '../src/wake-registry' -import { timeStep, waitForStreamEvents } from './test-utils' +import { + durableStreamTestServerUrl, + timeStep, + waitForStreamEvents, +} from './test-utils' import { TEST_ELECTRIC_URL, TEST_POSTGRES_URL, @@ -795,7 +799,7 @@ describe(`Wake Registry Integration`, () => { receiverUrl = `http://127.0.0.1:${addr.port}` electricAgentsServer = new ElectricAgentsServer({ - durableStreamsUrl: dsServer.url, + durableStreamsUrl: durableStreamTestServerUrl(dsServer.url), port: 0, postgresUrl: TEST_POSTGRES_URL, electricUrl: TEST_ELECTRIC_URL, From f2999190339a58dfdadd712a1421ed5232b27980 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Mon, 18 May 2026 12:14:16 -0600 Subject: [PATCH 28/37] fix(agents-server): route subscription control to stream-meta --- .../durable-streams-routing-adapter.ts | 24 ++++++++++++++++- packages/agents-server/src/stream-client.ts | 11 ++++++++ .../test/electric-agents-routes.test.ts | 27 ++++++++++++++++++- .../agents-server/test/stream-client.test.ts | 24 +++++++++++++++++ 4 files changed, 84 insertions(+), 2 deletions(-) diff --git a/packages/agents-server/src/routing/durable-streams-routing-adapter.ts b/packages/agents-server/src/routing/durable-streams-routing-adapter.ts index 38d258a339..e3b853bb64 100644 --- a/packages/agents-server/src/routing/durable-streams-routing-adapter.ts +++ b/packages/agents-server/src/routing/durable-streams-routing-adapter.ts @@ -34,11 +34,33 @@ function appendRequestPathToStreamRoot(input: DurableStreamsRoutingInput): URL { return appendSearch(target, incomingUrl) } +function appendControlPathToBackend(input: DurableStreamsRoutingInput): URL { + const incomingUrl = new URL(input.requestUrl, `http://localhost`) + if (!incomingUrl.pathname.startsWith(`/__ds/subscriptions`)) { + return appendRequestPathToStreamRoot(input) + } + const match = /^(.*)\/v1\/stream(?:\/(.+))?\/?$/.exec( + new URL(input.durableStreamsUrl).pathname + ) + if (!match) { + return appendRequestPathToStreamRoot(input) + } + + const [, prefix = ``, serviceId] = match + const target = new URL(input.durableStreamsUrl) + target.pathname = `${prefix}/v1/stream-meta${incomingUrl.pathname.replace(/^\/__ds/, ``)}` + appendSearch(target, incomingUrl) + if (serviceId) { + target.searchParams.set(`service`, decodeURIComponent(serviceId)) + } + return target +} + export const streamRootDurableStreamsRoutingAdapter: DurableStreamsRoutingAdapter = { streamUrl: appendRequestPathToStreamRoot, - controlUrl: appendRequestPathToStreamRoot, + controlUrl: appendControlPathToBackend, toBackendStreamPath(_serviceId, streamPath) { return streamPath.replace(/^\/+/, ``) diff --git a/packages/agents-server/src/stream-client.ts b/packages/agents-server/src/stream-client.ts index 264413d331..fba225fb2f 100644 --- a/packages/agents-server/src/stream-client.ts +++ b/packages/agents-server/src/stream-client.ts @@ -209,6 +209,17 @@ export class StreamClient { } private subscriptionUrl(subscriptionId: string): string { + const url = new URL(this.baseUrl) + const match = /^(.*)\/v1\/stream(?:\/(.+))?\/?$/.exec(url.pathname) + if (match) { + const [, prefix = ``, serviceId] = match + url.pathname = `${prefix}/v1/stream-meta/subscriptions/${encodeURIComponent(subscriptionId)}` + if (serviceId) { + url.searchParams.set(`service`, decodeURIComponent(serviceId)) + } + return url.toString() + } + return appendPathToBaseUrl( this.baseUrl, `/__ds/subscriptions/${encodeURIComponent(subscriptionId)}` diff --git a/packages/agents-server/test/electric-agents-routes.test.ts b/packages/agents-server/test/electric-agents-routes.test.ts index 8153b578f4..f79f4cdb91 100644 --- a/packages/agents-server/test/electric-agents-routes.test.ts +++ b/packages/agents-server/test/electric-agents-routes.test.ts @@ -462,7 +462,7 @@ describe(`ElectricAgentsRoutes shared-state streams`, () => { expect(result.status).toBe(201) const [url, init] = fetchSpy.mock.calls[0]! expect(String(url)).toBe( - `http://durable.local/v1/stream/tenant-a/__ds/subscriptions/horton-handler` + `http://durable.local/v1/stream-meta/subscriptions/horton-handler?service=tenant-a` ) expect(JSON.parse(requestBodyText(init?.body))).toEqual({ type: `webhook`, @@ -481,6 +481,31 @@ describe(`ElectricAgentsRoutes shared-state streams`, () => { } }) + it(`routes __ds subscription control to stream-meta for reference stream URLs`, async () => { + const fetchSpy = vi + .spyOn(globalThis, `fetch`) + .mockResolvedValue(new Response(JSON.stringify({ ok: true }))) + + try { + const result = await globalRouter.fetch( + createRequest(`GET`, `/__ds/subscriptions/horton-handler`), + { + service: `tenant-a`, + durableStreamsUrl: `http://durable.local/v1/stream/tenant-a`, + isShuttingDown: () => false, + } as unknown as TenantContext + ) + + expect(result.status).toBe(200) + const [url] = fetchSpy.mock.calls[0]! + expect(String(url)).toBe( + `http://durable.local/v1/stream-meta/subscriptions/horton-handler?service=tenant-a` + ) + } finally { + fetchSpy.mockRestore() + } + }) + it(`lets a routing adapter own service-routed subscription URLs and stream names`, async () => { const fetchSpy = vi.spyOn(globalThis, `fetch`).mockResolvedValue( new Response( diff --git a/packages/agents-server/test/stream-client.test.ts b/packages/agents-server/test/stream-client.test.ts index 6daa4aa173..18ed908b6f 100644 --- a/packages/agents-server/test/stream-client.test.ts +++ b/packages/agents-server/test/stream-client.test.ts @@ -162,6 +162,30 @@ describe(`StreamClient`, () => { } }) + it(`uses stream-meta control endpoints for reference stream URLs`, async () => { + const fetchMock = vi.spyOn(globalThis, `fetch`).mockResolvedValueOnce( + new Response(JSON.stringify({ subscription_id: `sub-1` }), { + headers: { 'content-type': `application/json` }, + }) + ) + const client = new StreamClient(`http://127.0.0.1:4545/v1/stream/tenant-a`) + + try { + await client.putSubscription(`sub-1`, { + type: `pull-wake`, + streams: [`/chat/one/main`], + wake_stream: `/runners/runner-1/wake`, + }) + + expect(fetchMock).toHaveBeenCalledWith( + `http://127.0.0.1:4545/v1/stream-meta/subscriptions/sub-1?service=tenant-a`, + expect.objectContaining({ method: `PUT` }) + ) + } finally { + fetchMock.mockRestore() + } + }) + it(`sends configured durable streams bearer auth on subscription requests`, async () => { const fetchMock = vi.spyOn(globalThis, `fetch`).mockResolvedValueOnce( new Response(JSON.stringify({ subscription_id: `sub-1` }), { From 673627630b1a5df87a6e45717e1457b467946960 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Mon, 18 May 2026 12:28:21 -0600 Subject: [PATCH 29/37] fix(agents-desktop): default local send principal --- .changeset/local-desktop-principal.md | 6 +++++ packages/agents-desktop/src/main.ts | 14 +++++++++--- .../src/lib/auth-fetch.test.ts | 12 ++++++++++ .../agents-server-ui/src/lib/auth-fetch.ts | 22 ++++++++++++++----- .../agents-server-ui/src/lib/sendMessage.ts | 11 +++++++--- 5 files changed, 54 insertions(+), 11 deletions(-) create mode 100644 .changeset/local-desktop-principal.md diff --git a/.changeset/local-desktop-principal.md b/.changeset/local-desktop-principal.md new file mode 100644 index 0000000000..ac23cf9885 --- /dev/null +++ b/.changeset/local-desktop-principal.md @@ -0,0 +1,6 @@ +--- +'@electric-ax/agents-desktop': patch +'@electric-ax/agents-server-ui': patch +--- + +Default unauthenticated local desktop sessions to the `system:dev-local` principal and resolve optimistic send principals at mutation time so pending messages do not render as `unknown`. diff --git a/packages/agents-desktop/src/main.ts b/packages/agents-desktop/src/main.ts index ed8c8ed54b..f68c71b4e2 100644 --- a/packages/agents-desktop/src/main.ts +++ b/packages/agents-desktop/src/main.ts @@ -256,7 +256,8 @@ const PULL_WAKE_REGISTER_RUNNER = const PULL_WAKE_OWNER_PRINCIPAL = process.env.ELECTRIC_DESKTOP_PULL_WAKE_OWNER_PRINCIPAL?.trim() || `/principal/system%3Adev-local` -const DEV_PRINCIPAL = ((): string | null => { +const DEFAULT_LOCAL_DEV_PRINCIPAL = `system:dev-local` +const EXPLICIT_DEV_PRINCIPAL = ((): string | null => { const raw = process.env.ELECTRIC_DESKTOP_PRINCIPAL?.trim() || null if (!raw) return null const colon = raw.indexOf(`:`) @@ -1216,10 +1217,17 @@ function localRuntimeStatusLabel(status: LocalRuntimeStatus): string { } function injectDevPrincipalHeaders(server: ServerConfig): ServerConfig { - if (!DEV_PRINCIPAL) return server + if (server.source === `electric-cloud`) return server + const principal = + EXPLICIT_DEV_PRINCIPAL ?? + (hasHeader(server.headers, ELECTRIC_PRINCIPAL_HEADER) || + hasHeader(server.headers, `authorization`) + ? null + : DEFAULT_LOCAL_DEV_PRINCIPAL) + if (!principal) return server return { ...server, - headers: { ...server.headers, [ELECTRIC_PRINCIPAL_HEADER]: DEV_PRINCIPAL }, + headers: { ...server.headers, [ELECTRIC_PRINCIPAL_HEADER]: principal }, } } diff --git a/packages/agents-server-ui/src/lib/auth-fetch.test.ts b/packages/agents-server-ui/src/lib/auth-fetch.test.ts index 497460196c..14e5b93be5 100644 --- a/packages/agents-server-ui/src/lib/auth-fetch.test.ts +++ b/packages/agents-server-ui/src/lib/auth-fetch.test.ts @@ -1,6 +1,7 @@ import { afterEach, describe, expect, it, vi } from 'vitest' import { getActivePrincipal, + getConfiguredActivePrincipal, registerActiveServerHeaders, serverFetch, } from './auth-fetch' @@ -94,5 +95,16 @@ describe(`server fetch helpers`, () => { }) expect(getActivePrincipal()).toBe(`/principal/system%3Adev-local`) + expect(getConfiguredActivePrincipal()).toBe(`/principal/system%3Adev-local`) + }) + + it(`uses the local dev principal when no active principal is configured`, () => { + registerActiveServerHeaders({ + name: `Local`, + url: `http://127.0.0.1:4437`, + }) + + expect(getConfiguredActivePrincipal()).toBe(null) + expect(getActivePrincipal()).toBe(`/principal/system%3Adev-local`) }) }) diff --git a/packages/agents-server-ui/src/lib/auth-fetch.ts b/packages/agents-server-ui/src/lib/auth-fetch.ts index c8ecbb2dca..31cf0e11c7 100644 --- a/packages/agents-server-ui/src/lib/auth-fetch.ts +++ b/packages/agents-server-ui/src/lib/auth-fetch.ts @@ -9,8 +9,17 @@ type ActiveServerHeaders = { headers: Record } +const DEFAULT_ACTIVE_PRINCIPAL = `system:dev-local` + let activeServerHeaders: ActiveServerHeaders | null = null +function principalUrl(principal: string): string { + const trimmed = principal.trim() + return trimmed.startsWith(`/principal/`) + ? trimmed + : `/principal/${encodeURIComponent(trimmed)}` +} + function normalizeHeaders( headers: Record | undefined ): Record { @@ -85,12 +94,15 @@ export function getConfiguredServerHeaders( return matchesActiveServer(input) ? (activeServerHeaders?.headers ?? {}) : {} } -export function getActivePrincipal(): string { +export function getConfiguredActivePrincipal(): string | null { const principal = activeServerHeaders?.headers[`electric-principal`] - if (!principal) return `unknown` - return principal.startsWith(`/principal/`) - ? principal - : `/principal/${encodeURIComponent(principal)}` + return principal ? principalUrl(principal) : null +} + +export function getActivePrincipal(): string { + return ( + getConfiguredActivePrincipal() ?? principalUrl(DEFAULT_ACTIVE_PRINCIPAL) + ) } export async function serverFetch( diff --git a/packages/agents-server-ui/src/lib/sendMessage.ts b/packages/agents-server-ui/src/lib/sendMessage.ts index 018d7152fe..a4553e4396 100644 --- a/packages/agents-server-ui/src/lib/sendMessage.ts +++ b/packages/agents-server-ui/src/lib/sendMessage.ts @@ -2,6 +2,7 @@ import { createOptimisticAction } from '@tanstack/db' import { generateKeyBetween } from 'fractional-indexing' import { getActivePrincipal, + getConfiguredActivePrincipal, getConfiguredServerHeaders, serverFetch, } from './auth-fetch' @@ -180,7 +181,7 @@ export function createSendMessageAction({ db, baseUrl, entityUrl, - from = getActivePrincipal(), + from, onOptimisticMessage, }: { db: EntityStreamDBWithActions @@ -191,11 +192,12 @@ export function createSendMessageAction({ }) { const action = createOptimisticAction({ onMutate: ({ text, mode, key, seq, position }) => { + const sender = from ?? getActivePrincipal() const now = new Date().toISOString() const message: OptimisticInboxMessage = { key, _seq: seq, - from, + from: sender, payload: { text }, timestamp: now, mode, @@ -211,7 +213,10 @@ export function createSendMessageAction({ }, mutationFn: async ({ text, key, mode, position }) => { const url = entityApiUrl(baseUrl, entityUrl, `/send`) - const sender = await resolveSenderPrincipalUrl(url, from) + const sender = await resolveSenderPrincipalUrl( + url, + from ?? getConfiguredActivePrincipal() ?? `` + ) const res = await serverFetch(url, { method: `POST`, headers: { 'content-type': `application/json` }, From d266b296a6c4de336bd633d429780ff66e1de4b6 Mon Sep 17 00:00:00 2001 From: Ilia Borovitinov Date: Mon, 18 May 2026 21:38:17 +0300 Subject: [PATCH 30/37] fix: remove stale durable streams consumer API --- .changeset/harden-pull-wake-runner.md | 2 +- packages/agents-server/src/stream-client.ts | 25 ------------------- .../agents-server/test/server-start.test.ts | 4 --- 3 files changed, 1 insertion(+), 30 deletions(-) diff --git a/.changeset/harden-pull-wake-runner.md b/.changeset/harden-pull-wake-runner.md index 5888dffdf7..8370a12b69 100644 --- a/.changeset/harden-pull-wake-runner.md +++ b/.changeset/harden-pull-wake-runner.md @@ -4,4 +4,4 @@ '@electric-ax/agents': patch --- -Harden pull-wake runner lifecycle with a state machine, concurrent claim limits (`maxConcurrentClaims`), heartbeat-driven stream resets, and exponential reconnect backoff (1s–30s). Add granular `status` field to `PullWakeRunnerHealth` (`stopped | starting | connecting | streaming | reconnecting | stopping`). The `onError` callback is now reporting-only (`(Error) => void`) — it can no longer control runner lifecycle. `stop()` rethrows `drainWakes` errors so callers observe wake handler failures. Event-driven heartbeat throttling avoids stale diagnostics between fixed-interval heartbeats. Durable Streams clients now append stream and `__ds` subscription control paths to the configured backend URL prefix without inferring a `/v1/stream` layout, so pull-wake subscriptions work behind arbitrary DS backend prefixes. +Harden pull-wake runner lifecycle with a state machine, concurrent claim limits (`maxConcurrentClaims`), heartbeat-driven stream resets, and exponential reconnect backoff (1s-30s). Add granular `status` field to `PullWakeRunnerHealth` (`stopped | starting | connecting | streaming | reconnecting | stopping`). The `onError` callback is now reporting-only (`(Error) => void`) - it can no longer control runner lifecycle. `stop()` rethrows `drainWakes` errors so callers observe wake handler failures. Event-driven heartbeat throttling avoids stale diagnostics between fixed-interval heartbeats. Durable Streams clients now append stream and `__ds` subscription control paths to the configured backend URL prefix without inferring a `/v1/stream` layout, so pull-wake subscriptions work behind arbitrary DS backend prefixes. Remove the stale `StreamClient.getConsumerState()` helper for the old Durable Streams `/consumers` endpoint. diff --git a/packages/agents-server/src/stream-client.ts b/packages/agents-server/src/stream-client.ts index fba225fb2f..823d934766 100644 --- a/packages/agents-server/src/stream-client.ts +++ b/packages/agents-server/src/stream-client.ts @@ -32,15 +32,6 @@ export interface WaitForMessagesResult { timedOut: boolean } -export interface ConsumerStateResponse { - state: string - wake_id?: string | null - webhook?: { - wake_id?: string | null - subscription_id?: string - } -} - export interface SubscriptionStreamInfo { path: string tail_offset?: string @@ -808,20 +799,4 @@ export class StreamClient { JSON.parse(text) as SubscriptionResponse ) } - - async getConsumerState( - consumerId: string - ): Promise { - const res = await fetch( - `${this.baseUrl}/consumers/${encodeURIComponent(consumerId)}`, - { method: `GET`, headers: await this.requestHeaders() } - ) - if (res.status === 404) return null - if (!res.ok) { - throw new Error( - `Consumer query failed: ${res.status} ${await res.text()}` - ) - } - return res.json() as Promise - } } diff --git a/packages/agents-server/test/server-start.test.ts b/packages/agents-server/test/server-start.test.ts index 6f56d9c063..85d5b8d5cb 100644 --- a/packages/agents-server/test/server-start.test.ts +++ b/packages/agents-server/test/server-start.test.ts @@ -191,10 +191,6 @@ vi.mock(`../src/stream-client`, () => ({ readJson(): Promise>> { return streamReadJsonMock() } - - getConsumerState(): Promise { - return Promise.resolve(null) - } }, })) From c6eda8cb168fa63cf2955188ea5ac3d6410ea129 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Mon, 18 May 2026 13:06:41 -0600 Subject: [PATCH 31/37] fix(agents-server): keep durable streams control urls opaque --- .../durable-streams-routing-adapter.ts | 24 +---------------- packages/agents-server/src/stream-client.ts | 11 -------- .../test/electric-agents-routes.test.ts | 27 +------------------ .../agents-server/test/stream-client.test.ts | 24 ----------------- 4 files changed, 2 insertions(+), 84 deletions(-) diff --git a/packages/agents-server/src/routing/durable-streams-routing-adapter.ts b/packages/agents-server/src/routing/durable-streams-routing-adapter.ts index e3b853bb64..38d258a339 100644 --- a/packages/agents-server/src/routing/durable-streams-routing-adapter.ts +++ b/packages/agents-server/src/routing/durable-streams-routing-adapter.ts @@ -34,33 +34,11 @@ function appendRequestPathToStreamRoot(input: DurableStreamsRoutingInput): URL { return appendSearch(target, incomingUrl) } -function appendControlPathToBackend(input: DurableStreamsRoutingInput): URL { - const incomingUrl = new URL(input.requestUrl, `http://localhost`) - if (!incomingUrl.pathname.startsWith(`/__ds/subscriptions`)) { - return appendRequestPathToStreamRoot(input) - } - const match = /^(.*)\/v1\/stream(?:\/(.+))?\/?$/.exec( - new URL(input.durableStreamsUrl).pathname - ) - if (!match) { - return appendRequestPathToStreamRoot(input) - } - - const [, prefix = ``, serviceId] = match - const target = new URL(input.durableStreamsUrl) - target.pathname = `${prefix}/v1/stream-meta${incomingUrl.pathname.replace(/^\/__ds/, ``)}` - appendSearch(target, incomingUrl) - if (serviceId) { - target.searchParams.set(`service`, decodeURIComponent(serviceId)) - } - return target -} - export const streamRootDurableStreamsRoutingAdapter: DurableStreamsRoutingAdapter = { streamUrl: appendRequestPathToStreamRoot, - controlUrl: appendControlPathToBackend, + controlUrl: appendRequestPathToStreamRoot, toBackendStreamPath(_serviceId, streamPath) { return streamPath.replace(/^\/+/, ``) diff --git a/packages/agents-server/src/stream-client.ts b/packages/agents-server/src/stream-client.ts index 823d934766..3d5b511212 100644 --- a/packages/agents-server/src/stream-client.ts +++ b/packages/agents-server/src/stream-client.ts @@ -200,17 +200,6 @@ export class StreamClient { } private subscriptionUrl(subscriptionId: string): string { - const url = new URL(this.baseUrl) - const match = /^(.*)\/v1\/stream(?:\/(.+))?\/?$/.exec(url.pathname) - if (match) { - const [, prefix = ``, serviceId] = match - url.pathname = `${prefix}/v1/stream-meta/subscriptions/${encodeURIComponent(subscriptionId)}` - if (serviceId) { - url.searchParams.set(`service`, decodeURIComponent(serviceId)) - } - return url.toString() - } - return appendPathToBaseUrl( this.baseUrl, `/__ds/subscriptions/${encodeURIComponent(subscriptionId)}` diff --git a/packages/agents-server/test/electric-agents-routes.test.ts b/packages/agents-server/test/electric-agents-routes.test.ts index f79f4cdb91..8153b578f4 100644 --- a/packages/agents-server/test/electric-agents-routes.test.ts +++ b/packages/agents-server/test/electric-agents-routes.test.ts @@ -462,7 +462,7 @@ describe(`ElectricAgentsRoutes shared-state streams`, () => { expect(result.status).toBe(201) const [url, init] = fetchSpy.mock.calls[0]! expect(String(url)).toBe( - `http://durable.local/v1/stream-meta/subscriptions/horton-handler?service=tenant-a` + `http://durable.local/v1/stream/tenant-a/__ds/subscriptions/horton-handler` ) expect(JSON.parse(requestBodyText(init?.body))).toEqual({ type: `webhook`, @@ -481,31 +481,6 @@ describe(`ElectricAgentsRoutes shared-state streams`, () => { } }) - it(`routes __ds subscription control to stream-meta for reference stream URLs`, async () => { - const fetchSpy = vi - .spyOn(globalThis, `fetch`) - .mockResolvedValue(new Response(JSON.stringify({ ok: true }))) - - try { - const result = await globalRouter.fetch( - createRequest(`GET`, `/__ds/subscriptions/horton-handler`), - { - service: `tenant-a`, - durableStreamsUrl: `http://durable.local/v1/stream/tenant-a`, - isShuttingDown: () => false, - } as unknown as TenantContext - ) - - expect(result.status).toBe(200) - const [url] = fetchSpy.mock.calls[0]! - expect(String(url)).toBe( - `http://durable.local/v1/stream-meta/subscriptions/horton-handler?service=tenant-a` - ) - } finally { - fetchSpy.mockRestore() - } - }) - it(`lets a routing adapter own service-routed subscription URLs and stream names`, async () => { const fetchSpy = vi.spyOn(globalThis, `fetch`).mockResolvedValue( new Response( diff --git a/packages/agents-server/test/stream-client.test.ts b/packages/agents-server/test/stream-client.test.ts index 18ed908b6f..6daa4aa173 100644 --- a/packages/agents-server/test/stream-client.test.ts +++ b/packages/agents-server/test/stream-client.test.ts @@ -162,30 +162,6 @@ describe(`StreamClient`, () => { } }) - it(`uses stream-meta control endpoints for reference stream URLs`, async () => { - const fetchMock = vi.spyOn(globalThis, `fetch`).mockResolvedValueOnce( - new Response(JSON.stringify({ subscription_id: `sub-1` }), { - headers: { 'content-type': `application/json` }, - }) - ) - const client = new StreamClient(`http://127.0.0.1:4545/v1/stream/tenant-a`) - - try { - await client.putSubscription(`sub-1`, { - type: `pull-wake`, - streams: [`/chat/one/main`], - wake_stream: `/runners/runner-1/wake`, - }) - - expect(fetchMock).toHaveBeenCalledWith( - `http://127.0.0.1:4545/v1/stream-meta/subscriptions/sub-1?service=tenant-a`, - expect.objectContaining({ method: `PUT` }) - ) - } finally { - fetchMock.mockRestore() - } - }) - it(`sends configured durable streams bearer auth on subscription requests`, async () => { const fetchMock = vi.spyOn(globalThis, `fetch`).mockResolvedValueOnce( new Response(JSON.stringify({ subscription_id: `sub-1` }), { From 2e9544321125b509e9d5e9f269afb68650450e63 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Mon, 18 May 2026 13:21:17 -0600 Subject: [PATCH 32/37] fix(agents-server): relink dispatch subscriptions on send --- .../src/routing/entities-router.ts | 8 ++++---- .../test/dispatch-policy-routing.test.ts | 20 +++++++++++++++---- 2 files changed, 20 insertions(+), 8 deletions(-) diff --git a/packages/agents-server/src/routing/entities-router.ts b/packages/agents-server/src/routing/entities-router.ts index 3e05f559c7..b6c8ef3141 100644 --- a/packages/agents-server/src/routing/entities-router.ts +++ b/packages/agents-server/src/routing/entities-router.ts @@ -530,10 +530,10 @@ async function sendEntity( await ctx.entityManager.ensurePrincipal(principal) const { entityUrl, entity } = requireExistingEntityRoute(request) - if (!entity.dispatch_policy) { - const updatedEntity = await backfillEntityDispatchPolicy(ctx, entity) - await linkEntityDispatchSubscription(ctx, updatedEntity) - } + const dispatchEntity = entity.dispatch_policy + ? entity + : await backfillEntityDispatchPolicy(ctx, entity) + await linkEntityDispatchSubscription(ctx, dispatchEntity) if (parsed.afterMs && parsed.afterMs > 0) { await ctx.entityManager.enqueueDelayedSend( diff --git a/packages/agents-server/test/dispatch-policy-routing.test.ts b/packages/agents-server/test/dispatch-policy-routing.test.ts index 66a2beacda..75b75cfef4 100644 --- a/packages/agents-server/test/dispatch-policy-routing.test.ts +++ b/packages/agents-server/test/dispatch-policy-routing.test.ts @@ -269,7 +269,7 @@ describe(`dispatch policy routing`, () => { ) }) - it(`does not relink existing runner-dispatched entities before sending`, async () => { + it(`recreates missing runner dispatch subscriptions before sending`, async () => { const dispatchPolicy: DispatchPolicy = { targets: [{ type: `runner`, runnerId: `runner-1` }], } @@ -287,8 +287,17 @@ describe(`dispatch policy routing`, () => { ) expect(response.status).toBe(204) - expect(ctx.streamClient.getSubscription).not.toHaveBeenCalled() - expect(ctx.streamClient.putSubscription).not.toHaveBeenCalled() + expect(ctx.streamClient.getSubscription).toHaveBeenCalledWith( + expect.stringMatching(/^runner:runner-1:/) + ) + expect(ctx.streamClient.putSubscription).toHaveBeenCalledWith( + expect.stringMatching(/^runner:runner-1:/), + expect.objectContaining({ + type: `pull-wake`, + streams: [`/chat/one/main`], + wake_stream: `/runners/runner-1/wake`, + }) + ) expect(ctx.streamClient.addSubscriptionStreams).not.toHaveBeenCalled() expect(ctx.entityManager.send).toHaveBeenCalledWith( `/chat/one`, @@ -317,7 +326,10 @@ describe(`dispatch policy routing`, () => { ) expect(response.status).toBe(204) - expect(ctx.streamClient.getSubscription).not.toHaveBeenCalled() + expect(ctx.streamClient.getSubscription).toHaveBeenCalledWith( + expect.stringMatching(/^runner:runner-1:/) + ) + expect(ctx.streamClient.putSubscription).not.toHaveBeenCalled() expect(ctx.streamClient.addSubscriptionStreams).not.toHaveBeenCalled() expect(ctx.streamClient.removeSubscriptionStream).not.toHaveBeenCalled() expect(ctx.entityManager.send).toHaveBeenCalledWith( From dbf7910c4c7419465a86b1df57801d4333006720 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Mon, 18 May 2026 13:33:07 -0600 Subject: [PATCH 33/37] fix(agents): use durable streams backend url in runtime tests --- packages/agents-runtime/test/runtime-dsl.ts | 2 +- packages/agents-server/src/routing/dispatch-policy.ts | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/packages/agents-runtime/test/runtime-dsl.ts b/packages/agents-runtime/test/runtime-dsl.ts index c2086ffa71..3836a2395d 100644 --- a/packages/agents-runtime/test/runtime-dsl.ts +++ b/packages/agents-runtime/test/runtime-dsl.ts @@ -260,7 +260,7 @@ async function startServers(registry: EntityRegistry): Promise { await timeStep(`DurableStreamTestServer.start`, () => dsServer.start()) const electricAgentsServer = new ElectricAgentsServer({ - durableStreamsUrl: dsServer.url, + durableStreamsUrl: `${dsServer.url}/v1/stream`, port: 0, postgresUrl: TEST_POSTGRES_URL, electricUrl: TEST_ELECTRIC_URL, diff --git a/packages/agents-server/src/routing/dispatch-policy.ts b/packages/agents-server/src/routing/dispatch-policy.ts index cd8e8799ed..407942ac44 100644 --- a/packages/agents-server/src/routing/dispatch-policy.ts +++ b/packages/agents-server/src/routing/dispatch-policy.ts @@ -242,6 +242,9 @@ async function linkStreamToTargetSubscription( entity: ElectricAgentsEntity ): Promise { const streamPath = entity.streams.main + await ctx.streamClient.ensure(streamPath, { + contentType: `application/json`, + }) const subscriptionId = subscriptionIdForEntityDispatchTarget( target, entity.url From 756b7dc9fa53e303f04d5b9e78c926f0f20a1647 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Mon, 18 May 2026 20:07:52 -0600 Subject: [PATCH 34/37] Route local desktop writes through main process --- .changeset/desktop-local-fetch.md | 6 + packages/agents-desktop/src/main.ts | 168 ++++++++++++++++-- packages/agents-desktop/src/preload.ts | 19 ++ .../src/lib/auth-fetch.test.ts | 116 ++++++++++++ .../agents-server-ui/src/lib/auth-fetch.ts | 119 ++++++++++++- .../src/lib/server-connection.ts | 18 ++ 6 files changed, 431 insertions(+), 15 deletions(-) create mode 100644 .changeset/desktop-local-fetch.md diff --git a/.changeset/desktop-local-fetch.md b/.changeset/desktop-local-fetch.md new file mode 100644 index 0000000000..f91d3f99ab --- /dev/null +++ b/.changeset/desktop-local-fetch.md @@ -0,0 +1,6 @@ +--- +'@electric-ax/agents-desktop': patch +'@electric-ax/agents-server-ui': patch +--- + +Route local desktop mutating agents-server requests through the Electron main process so CORS preflights cannot stall behind renderer connection limits. diff --git a/packages/agents-desktop/src/main.ts b/packages/agents-desktop/src/main.ts index f68c71b4e2..2ccce37561 100644 --- a/packages/agents-desktop/src/main.ts +++ b/packages/agents-desktop/src/main.ts @@ -98,6 +98,21 @@ type DesktopState = { pullWakeRunnerId: string | null } +type DesktopServerFetchRequest = { + url: string + method: string + headers: Record + body: string | null +} + +type DesktopServerFetchResponse = { + url: string + status: number + statusText: string + headers: Record + body: string +} + type ServerConnectionState = { serverId: string status: ServerConnectionStatus @@ -204,7 +219,7 @@ const APP_ICON_FILE = process.platform === `darwin` ? `icon-mac.png` : `icon.png` const APP_ICON_PATH = path.join(RESOURCE_DIR, `assets`, APP_ICON_FILE) const APP_DISPLAY_NAME = `Electric Agents` -const MAX_CONNECTIONS_PER_HOST = `256` +const IGNORE_CONNECTION_LIMIT_DOMAINS = `localhost,127.0.0.1` const SETTINGS_VERSION = 2 const GLOBAL_API_KEYS_REF = `api-keys:global` const RECONNECT_BASE_MS = 1_000 @@ -229,11 +244,16 @@ if (DESKTOP_USER_DATA_DIR) { const MCP_OAUTH_REDIRECT_BASE = `http://127.0.0.1:53117` // Electric streams can hold many long-polling HTTP requests open to the same -// agents server. Raise Chromium's default per-host connection cap before -// Electron creates its network context so those streams do not queue behind it. +// local agents server. Electron supports bypassing Chromium's connection cap +// for a domain list; this must run before Electron creates its network context. app.commandLine.appendSwitch( - `max-connections-per-host`, - MAX_CONNECTIONS_PER_HOST + `ignore-connections-limit`, + IGNORE_CONNECTION_LIMIT_DOMAINS +) +console.info( + `[agents-desktop] ignore-connections-limit=${app.commandLine.getSwitchValue( + `ignore-connections-limit` + )}` ) /** @@ -537,12 +557,40 @@ function findCloudServerForUrl(requestUrl: string): ServerConfig | null { return fallbackMatches.length === 1 ? fallbackMatches[0]! : null } +function findSavedServerForUrl(requestUrl: string): ServerConfig | null { + let parsed: URL + try { + parsed = new URL(requestUrl) + } catch { + return null + } + + for (const server of settings.servers) { + let base: URL + try { + base = new URL(server.url) + } catch { + continue + } + if (base.origin !== parsed.origin) continue + const basePath = base.pathname.replace(/\/+$/, ``) + if ( + basePath === `` || + parsed.pathname === basePath || + parsed.pathname.startsWith(`${basePath}/`) + ) { + return server + } + } + return null +} + /** - * Decorate outgoing requests bound for a saved cloud agent server - * with `Authorization: Bearer ` and - * `x-electric-service: ` headers. Two injection points, - * both reading from the same in-memory agents-token map - * (`SecretStore`-backed): + * Decorate outgoing requests bound for saved agent servers with the + * configured server headers. Cloud agent servers also receive + * `Authorization: Bearer ` and `x-electric-service: + * ` headers. Two injection points, both reading from the + * same in-memory agents-token map (`SecretStore`-backed): * * 1. Renderer fetches — Electron's * `session.webRequest.onBeforeSendHeaders` hook catches anything @@ -562,7 +610,10 @@ function findCloudServerForUrl(requestUrl: string): ServerConfig | null { */ function installCloudAuthHeaderInjection(): void { session.defaultSession.webRequest.onBeforeSendHeaders((details, callback) => { - const extra = buildCloudAuthHeaders(details.url) + const extra = mergeHeaders( + buildSavedServerHeaders(details.url) ?? undefined, + buildCloudAuthHeaders(details.url) ?? undefined + ) if (!extra) { callback({ requestHeaders: details.requestHeaders }) return @@ -575,6 +626,87 @@ function installCloudAuthHeaderInjection(): void { installCloudAuthUndiciInterceptor() } +function buildSavedServerHeaders(url: string): Record | null { + const server = findSavedServerForUrl(url) + if (!server) return null + return mergeHeaders(injectDevPrincipalHeaders(server).headers) ?? null +} + +function assertDesktopServerFetchAllowed( + request: unknown +): DesktopServerFetchRequest { + if (!request || typeof request !== `object`) { + throw new Error(`Invalid desktop server fetch request`) + } + const raw = request as Partial + if (typeof raw.url !== `string` || raw.url.trim().length === 0) { + throw new Error(`Invalid desktop server fetch URL`) + } + if (typeof raw.method !== `string` || raw.method.trim().length === 0) { + throw new Error(`Invalid desktop server fetch method`) + } + if (!raw.headers || typeof raw.headers !== `object`) { + throw new Error(`Invalid desktop server fetch headers`) + } + if (raw.body !== null && typeof raw.body !== `string`) { + throw new Error(`Invalid desktop server fetch body`) + } + + const url = raw.url.trim() + const method = raw.method.trim().toUpperCase() + if (![`POST`, `PUT`, `PATCH`, `DELETE`].includes(method)) { + throw new Error(`Desktop server fetch only supports mutating requests`) + } + const server = findSavedServerForUrl(url) + if (!server || server.source === `electric-cloud`) { + throw new Error( + `Desktop server fetch is only available for saved local servers` + ) + } + let parsed: URL + try { + parsed = new URL(url) + } catch { + throw new Error(`Invalid desktop server fetch URL`) + } + if ( + parsed.protocol !== `http:` || + !isLocalLoopbackHostname(parsed.hostname) + ) { + throw new Error(`Desktop server fetch only supports local HTTP servers`) + } + + return { + url, + method, + headers: normalizeHeaderRecord(raw.headers) ?? {}, + body: raw.body, + } +} + +async function desktopServerFetch( + request: unknown +): Promise { + const checked = assertDesktopServerFetchAllowed(request) + const headers = mergeHeaders( + buildSavedServerHeaders(checked.url) ?? undefined, + buildCloudAuthHeaders(checked.url) ?? undefined, + checked.headers + ) + const response = await fetch(checked.url, { + method: checked.method, + headers, + body: checked.body, + }) + return { + url: response.url, + status: response.status, + statusText: response.statusText, + headers: headersToRecord(response.headers), + body: await response.text(), + } +} + /** * Build the cloud-auth headers to inject on a request to `url`, or * `null` if the URL doesn't target a saved cloud agent server (or we @@ -801,6 +933,17 @@ function headersToRecord(headers: Headers): Record { return record } +function isLocalLoopbackHostname(hostname: string): boolean { + const normalized = hostname.toLowerCase() + return ( + normalized === `localhost` || + normalized === `127.0.0.1` || + normalized === `0.0.0.0` || + normalized === `[::1]` || + normalized === `::1` + ) +} + function normalizeServers( value: unknown, activeUrl?: string | null @@ -2345,6 +2488,9 @@ function registerIpcHandlers(): void { const win = BrowserWindow.fromWebContents(event.sender) return desktopStateForWindow(win) }) + ipcMain.handle(`desktop:server-fetch`, (_event, request: unknown) => + desktopServerFetch(request) + ) ipcMain.handle( `desktop:set-active-server`, async (_event, server: ServerConfig | null) => { diff --git a/packages/agents-desktop/src/preload.ts b/packages/agents-desktop/src/preload.ts index f017e823a4..5a4565a36c 100644 --- a/packages/agents-desktop/src/preload.ts +++ b/packages/agents-desktop/src/preload.ts @@ -80,6 +80,21 @@ type DesktopState = { pullWakeRunnerId: string | null } +type DesktopServerFetchRequest = { + url: string + method: string + headers: Record + body: string | null +} + +type DesktopServerFetchResponse = { + url: string + status: number + statusText: string + headers: Record + body: string +} + type ServerConnectionState = { serverId: string status: ServerConnectionStatus @@ -263,6 +278,10 @@ const api = { ipcRenderer.invoke(`desktop:save-servers`, servers), getDesktopState: (): Promise => ipcRenderer.invoke(`desktop:get-state`), + serverFetch: ( + request: DesktopServerFetchRequest + ): Promise => + ipcRenderer.invoke(`desktop:server-fetch`, request), setNativeAppearance: (appearance: DesktopAppearance): Promise => ipcRenderer.invoke(`desktop:set-native-appearance`, appearance), setActiveServer: (server: ServerConfig | null): Promise => diff --git a/packages/agents-server-ui/src/lib/auth-fetch.test.ts b/packages/agents-server-ui/src/lib/auth-fetch.test.ts index 14e5b93be5..c661687add 100644 --- a/packages/agents-server-ui/src/lib/auth-fetch.test.ts +++ b/packages/agents-server-ui/src/lib/auth-fetch.test.ts @@ -87,6 +87,122 @@ describe(`server fetch helpers`, () => { expect(headers.has(`authorization`)).toBe(false) }) + it(`leaves configured headers to desktop injection inside Electron`, async () => { + ;(globalThis as { window?: unknown }).window = { + electronAPI: {}, + } + registerActiveServerHeaders({ + name: `Local`, + url: `http://localhost:4437`, + headers: { 'electric-principal': `system:dev-local` }, + }) + + const fetchMock = vi + .spyOn(globalThis, `fetch`) + .mockResolvedValue(new Response(`ok`)) + + await serverFetch( + `http://localhost:4437/_electric/entities/horton/a/send`, + { + method: `POST`, + headers: { 'content-type': `text/plain` }, + } + ) + + expect(fetchMock.mock.calls[0][0]).toBe( + `http://localhost:4437/_electric/entities/horton/a/send` + ) + const headers = new Headers(fetchMock.mock.calls[0][1]?.headers) + expect(headers.get(`content-type`)).toBe(`text/plain`) + expect(headers.has(`electric-principal`)).toBe(false) + }) + + it(`routes local mutating requests through the desktop server fetch transport`, async () => { + const desktopFetch = vi.fn().mockResolvedValue({ + url: `http://127.0.0.1:4437/_electric/entities/horton/a`, + status: 204, + statusText: `No Content`, + headers: {}, + body: ``, + }) + ;(globalThis as { window?: unknown }).window = { + electronAPI: { serverFetch: desktopFetch }, + } + registerActiveServerHeaders({ + name: `Local`, + url: `http://127.0.0.1:4437`, + headers: { 'electric-principal': `system:dev-local` }, + }) + + const fetchMock = vi.spyOn(globalThis, `fetch`) + + const response = await serverFetch( + `http://127.0.0.1:4437/_electric/entities/horton/a`, + { + method: `PUT`, + headers: { 'content-type': `application/json` }, + body: JSON.stringify({}), + } + ) + + expect(response.status).toBe(204) + expect(fetchMock).not.toHaveBeenCalled() + expect(desktopFetch).toHaveBeenCalledWith({ + url: `http://127.0.0.1:4437/_electric/entities/horton/a`, + method: `PUT`, + headers: { 'content-type': `application/json` }, + body: `{}`, + }) + }) + + it(`keeps local GET requests in the browser in Electron`, async () => { + const desktopFetch = vi.fn() + ;(globalThis as { window?: unknown }).window = { + electronAPI: { serverFetch: desktopFetch }, + } + registerActiveServerHeaders({ + name: `Local`, + url: `http://127.0.0.1:4437`, + headers: { 'electric-principal': `system:dev-local` }, + }) + + const fetchMock = vi + .spyOn(globalThis, `fetch`) + .mockResolvedValue(new Response(`ok`)) + + await serverFetch(`http://127.0.0.1:4437/_electric/shape`) + + expect(desktopFetch).not.toHaveBeenCalled() + expect(fetchMock).toHaveBeenCalledOnce() + }) + + it(`keeps non-local mutating requests in the browser in Electron`, async () => { + const desktopFetch = vi.fn() + ;(globalThis as { window?: unknown }).window = { + electronAPI: { serverFetch: desktopFetch }, + } + registerActiveServerHeaders({ + name: `Cloud`, + url: `https://agents.example.test`, + headers: { Authorization: `Bearer tenant-token` }, + }) + + const fetchMock = vi + .spyOn(globalThis, `fetch`) + .mockResolvedValue(new Response(`ok`)) + + await serverFetch( + `https://agents.example.test/_electric/entities/horton/a`, + { + method: `PUT`, + body: JSON.stringify({}), + } + ) + + expect(desktopFetch).not.toHaveBeenCalled() + expect(fetchMock).toHaveBeenCalledOnce() + }) + it(`returns the active principal as a canonical principal URL`, () => { registerActiveServerHeaders({ name: `Tenant`, diff --git a/packages/agents-server-ui/src/lib/auth-fetch.ts b/packages/agents-server-ui/src/lib/auth-fetch.ts index 31cf0e11c7..07f70a6d7a 100644 --- a/packages/agents-server-ui/src/lib/auth-fetch.ts +++ b/packages/agents-server-ui/src/lib/auth-fetch.ts @@ -1,3 +1,8 @@ +import type { + DesktopServerFetchRequest, + DesktopServerFetchResponse, +} from './server-connection' + type ServerHeaderConfig = { name?: string url: string @@ -10,6 +15,8 @@ type ActiveServerHeaders = { } const DEFAULT_ACTIVE_PRINCIPAL = `system:dev-local` +const DESKTOP_SERVER_FETCH_METHODS = new Set([`POST`, `PUT`, `PATCH`, `DELETE`]) +const NULL_BODY_STATUSES = new Set([204, 205, 304]) let activeServerHeaders: ActiveServerHeaders | null = null @@ -78,6 +85,85 @@ function matchesActiveServer(input: RequestInfo | URL): boolean { ) } +function isLocalHttpUrl(url: URL): boolean { + if (url.protocol !== `http:`) return false + const hostname = url.hostname.toLowerCase() + return ( + hostname === `localhost` || + hostname === `127.0.0.1` || + hostname === `0.0.0.0` || + hostname === `[::1]` || + hostname === `::1` + ) +} + +function activeServerIsLocal(): boolean { + if (!activeServerHeaders) return false + try { + return isLocalHttpUrl(new URL(activeServerHeaders.baseUrl)) + } catch { + return false + } +} + +function requestMethod(input: RequestInfo | URL, init: RequestInit): string { + return ( + init.method ?? + (input instanceof Request ? input.method : undefined) ?? + `GET` + ).toUpperCase() +} + +function desktopServerFetchApi(): + | (( + request: DesktopServerFetchRequest + ) => Promise) + | undefined { + if (typeof window === `undefined`) return undefined + return window.electronAPI?.serverFetch +} + +function shouldUseDesktopServerFetch( + input: RequestInfo | URL, + init: RequestInit +): boolean { + const method = requestMethod(input, init) + return ( + DESKTOP_SERVER_FETCH_METHODS.has(method) && + activeServerIsLocal() && + matchesActiveServer(input) && + Boolean(desktopServerFetchApi()) + ) +} + +async function desktopFetchBody( + input: RequestInfo | URL, + init: RequestInit +): Promise { + if (init.body === undefined || init.body === null) { + if (input instanceof Request) { + if (input.bodyUsed) return undefined + return await input.clone().text() + } + return null + } + if (typeof init.body === `string`) return init.body + if (init.body instanceof URLSearchParams) return init.body.toString() + if (init.body instanceof Blob) return await init.body.text() + return undefined +} + +function responseFromDesktopFetch( + response: DesktopServerFetchResponse +): Response { + const body = NULL_BODY_STATUSES.has(response.status) ? null : response.body + return new Response(body, { + status: response.status, + statusText: response.statusText, + headers: response.headers, + }) +} + export function registerActiveServerHeaders( server: ServerHeaderConfig | null ): void { @@ -105,20 +191,45 @@ export function getActivePrincipal(): string { ) } +function hasDesktopHeaderInjection(): boolean { + return ( + typeof window !== `undefined` && + Boolean((window as { electronAPI?: unknown }).electronAPI) + ) +} + export async function serverFetch( input: RequestInfo | URL, init: RequestInit = {} ): Promise { + const method = requestMethod(input, init) const headers = new Headers( input instanceof Request ? input.headers : undefined ) new Headers(init.headers).forEach((value, key) => { headers.set(key, value) }) - for (const [key, value] of Object.entries( - getConfiguredServerHeaders(input) - )) { - if (!headers.has(key)) headers.set(key, value) + if (!hasDesktopHeaderInjection()) { + for (const [key, value] of Object.entries( + getConfiguredServerHeaders(input) + )) { + if (!headers.has(key)) headers.set(key, value) + } + } + if (shouldUseDesktopServerFetch(input, init)) { + const api = desktopServerFetchApi() + const url = urlFromInput(input) + const body = await desktopFetchBody(input, init) + if (api && url && body !== undefined) { + return responseFromDesktopFetch( + await api({ + url: url.toString(), + method, + headers: Object.fromEntries(headers.entries()), + body, + }) + ) + } } return fetch(input, { ...init, headers }) } diff --git a/packages/agents-server-ui/src/lib/server-connection.ts b/packages/agents-server-ui/src/lib/server-connection.ts index 1c99ac7836..0e2ec05e8d 100644 --- a/packages/agents-server-ui/src/lib/server-connection.ts +++ b/packages/agents-server-ui/src/lib/server-connection.ts @@ -41,6 +41,21 @@ export interface DesktopState { pullWakeRunnerId: string | null } +export interface DesktopServerFetchRequest { + url: string + method: string + headers: Record + body: string | null +} + +export interface DesktopServerFetchResponse { + url: string + status: number + statusText: string + headers: Record + body: string +} + export interface ServerConnectionState { serverId: string status: ServerConnectionStatus @@ -227,6 +242,9 @@ declare global { getServers: () => Promise> saveServers: (servers: Array) => Promise getDesktopState?: () => Promise + serverFetch?: ( + request: DesktopServerFetchRequest + ) => Promise setNativeAppearance?: (appearance: DesktopAppearance) => Promise setActiveServer?: (server: ServerConfig | null) => Promise setSelectedServer?: (serverId: string | null) => Promise From 846de02f80cf60d76b87f73138953c4fdd702f93 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Mon, 18 May 2026 20:18:34 -0600 Subject: [PATCH 35/37] Avoid local send preflights --- .../agents-server-ui/src/lib/sendMessage.ts | 2 +- packages/agents-server-ui/src/main.tsx | 30 +++++++++++++++---- 2 files changed, 26 insertions(+), 6 deletions(-) diff --git a/packages/agents-server-ui/src/lib/sendMessage.ts b/packages/agents-server-ui/src/lib/sendMessage.ts index a4553e4396..c97a5552cd 100644 --- a/packages/agents-server-ui/src/lib/sendMessage.ts +++ b/packages/agents-server-ui/src/lib/sendMessage.ts @@ -219,7 +219,7 @@ export function createSendMessageAction({ ) const res = await serverFetch(url, { method: `POST`, - headers: { 'content-type': `application/json` }, + headers: { 'content-type': `text/plain` }, body: JSON.stringify({ from: sender, key, diff --git a/packages/agents-server-ui/src/main.tsx b/packages/agents-server-ui/src/main.tsx index dc494a5c75..b60db2210b 100644 --- a/packages/agents-server-ui/src/main.tsx +++ b/packages/agents-server-ui/src/main.tsx @@ -23,16 +23,36 @@ import { App } from './App' // ngrok's free tier intercepts browser requests with an HTML warning page // (status 200, no CORS header) — every fetch to an ngrok host fails CORS -// as a result. Setting `ngrok-skip-browser-warning` on every outbound -// request makes ngrok pass through to the upstream. No effect on requests -// to non-ngrok hosts. Covers the durable-streams client's internal fetches -// too, since it calls through the global fetch. +// as a result. Set `ngrok-skip-browser-warning` only for ngrok hosts: +// adding a custom header to local sends forces CORS preflights. +function isNgrokHost(input: RequestInfo | URL): boolean { + try { + const url = + input instanceof Request + ? new URL(input.url, window.location.href) + : new URL(input, window.location.href) + return ( + url.hostname === `ngrok-free.app` || + url.hostname.endsWith(`.ngrok-free.app`) || + url.hostname === `ngrok.app` || + url.hostname.endsWith(`.ngrok.app`) || + url.hostname === `ngrok.io` || + url.hostname.endsWith(`.ngrok.io`) + ) + } catch { + return false + } +} + const originalFetch = window.fetch.bind(window) window.fetch = ( input: RequestInfo | URL, init?: RequestInit ): Promise => { - const headers = new Headers(init?.headers ?? {}) + if (!isNgrokHost(input)) return originalFetch(input, init) + const headers = new Headers( + init?.headers ?? (input instanceof Request ? input.headers : undefined) + ) if (!headers.has(`ngrok-skip-browser-warning`)) { headers.set(`ngrok-skip-browser-warning`, `true`) } From e8241f12cdb845d4206c6735c3a97d44ae1323f0 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Mon, 18 May 2026 21:30:45 -0600 Subject: [PATCH 36/37] Harden pull-wake runner invariants --- .changeset/harden-pull-wake-runner.md | 2 +- .changeset/pull-wake-health-diagnostics.md | 2 +- packages/agents-desktop/src/main.ts | 1 - .../agents-runtime/src/pull-wake-runner.ts | 125 ++- .../test/pull-wake-runner.test.ts | 212 +++-- .../agents-server-ui/src/lib/sendMessage.ts | 2 +- packages/agents-server-ui/src/main.tsx | 6 +- ...-05-16-pull-wake-runner-state-machine.html | 746 ++++++++++++++++++ ...26-05-16-pull-wake-runner-state-machine.md | 522 ++++++++++++ .../src/electric-agents-types.ts | 5 +- packages/agents-server/src/principal.ts | 16 +- .../src/routing/dispatch-policy.ts | 43 +- .../src/routing/runners-router.ts | 11 +- .../test/dispatch-policy-routing.test.ts | 15 + packages/agents-server/test/principal.test.ts | 18 + .../agents-server/test/runners-router.test.ts | 24 + 16 files changed, 1575 insertions(+), 175 deletions(-) create mode 100644 packages/agents-server/docs/superpowers/specs/2026-05-16-pull-wake-runner-state-machine.html create mode 100644 packages/agents-server/docs/superpowers/specs/2026-05-16-pull-wake-runner-state-machine.md diff --git a/.changeset/harden-pull-wake-runner.md b/.changeset/harden-pull-wake-runner.md index 8370a12b69..b61498f2f5 100644 --- a/.changeset/harden-pull-wake-runner.md +++ b/.changeset/harden-pull-wake-runner.md @@ -4,4 +4,4 @@ '@electric-ax/agents': patch --- -Harden pull-wake runner lifecycle with a state machine, concurrent claim limits (`maxConcurrentClaims`), heartbeat-driven stream resets, and exponential reconnect backoff (1s-30s). Add granular `status` field to `PullWakeRunnerHealth` (`stopped | starting | connecting | streaming | reconnecting | stopping`). The `onError` callback is now reporting-only (`(Error) => void`) - it can no longer control runner lifecycle. `stop()` rethrows `drainWakes` errors so callers observe wake handler failures. Event-driven heartbeat throttling avoids stale diagnostics between fixed-interval heartbeats. Durable Streams clients now append stream and `__ds` subscription control paths to the configured backend URL prefix without inferring a `/v1/stream` layout, so pull-wake subscriptions work behind arbitrary DS backend prefixes. Remove the stale `StreamClient.getConsumerState()` helper for the old Durable Streams `/consumers` endpoint. +Harden pull-wake runner lifecycle with a state machine, heartbeat-driven stream resets, and exponential reconnect backoff (1s-30s). Add granular `status` field to `PullWakeRunnerHealth` (`stopped | starting | connecting | streaming | reconnecting | stopping`). The `onError` callback is now reporting-only (`(Error) => void`) - it can no longer control runner lifecycle. `stop()` rethrows `drainWakes` errors so callers observe wake handler failures. Event-driven heartbeat throttling avoids stale diagnostics between fixed-interval heartbeats. Durable Streams clients now append stream and `__ds` subscription control paths to the configured backend URL prefix without inferring a `/v1/stream` layout, so pull-wake subscriptions work behind arbitrary DS backend prefixes. Remove the stale `StreamClient.getConsumerState()` helper for the old Durable Streams `/consumers` endpoint. diff --git a/.changeset/pull-wake-health-diagnostics.md b/.changeset/pull-wake-health-diagnostics.md index b54dbe5f5b..c57ab9c354 100644 --- a/.changeset/pull-wake-health-diagnostics.md +++ b/.changeset/pull-wake-health-diagnostics.md @@ -6,4 +6,4 @@ 'electric-ax': patch --- -Add pull-wake runner health check endpoint and rename `owner_user_id` to `owner_principal` across the runners system. The `GET /_electric/runners/:id/health` endpoint returns comprehensive diagnostics including runner state, client-reported stream/heartbeat/claim metrics, active claims, and dispatch stats with a derived health status (healthy/degraded/unhealthy). The `PullWakeRunner` now tracks internal diagnostics and reports them to the server via heartbeats, stored in a separate `runner_runtime_diagnostics` table so the main `runners` shape stays stable for normal UI sync. The `owner_user_id` → `owner_principal` rename stores canonical principal URLs instead of keys, with strict validation and canonicalization at route boundaries. This is a breaking change with no backward compatibility — all callers must send principal URLs. +Add pull-wake runner health check endpoint and rename `owner_user_id` to `owner_principal` across the runners system. The `GET /_electric/runners/:id/health` endpoint returns comprehensive diagnostics including runner state, client-reported stream/heartbeat/claim metrics, active claims, and dispatch stats with a derived health status (healthy/degraded/unhealthy). The `PullWakeRunner` now tracks internal diagnostics and reports them to the server via heartbeats, stored in a separate `runner_runtime_diagnostics` table so the main `runners` shape stays stable for normal UI sync. The `owner_user_id` → `owner_principal` rename stores canonical principal URLs instead of keys, with strict validation and canonicalization at route boundaries. The migration expires active runner claims and deletes existing runner rows as part of the principal rewrite. This is a breaking change with no backward compatibility — all callers must send principal URLs. diff --git a/packages/agents-desktop/src/main.ts b/packages/agents-desktop/src/main.ts index 2ccce37561..4f886b867e 100644 --- a/packages/agents-desktop/src/main.ts +++ b/packages/agents-desktop/src/main.ts @@ -690,7 +690,6 @@ async function desktopServerFetch( const checked = assertDesktopServerFetchAllowed(request) const headers = mergeHeaders( buildSavedServerHeaders(checked.url) ?? undefined, - buildCloudAuthHeaders(checked.url) ?? undefined, checked.headers ) const response = await fetch(checked.url, { diff --git a/packages/agents-runtime/src/pull-wake-runner.ts b/packages/agents-runtime/src/pull-wake-runner.ts index 82f6a9958b..e78c774f53 100644 --- a/packages/agents-runtime/src/pull-wake-runner.ts +++ b/packages/agents-runtime/src/pull-wake-runner.ts @@ -28,7 +28,6 @@ export interface PullWakeRunnerConfig { heartbeatIntervalMs?: number eventHeartbeatThrottleMs?: number leaseMs?: number - maxConcurrentClaims?: number heartbeatPath?: string claimPath?: string onError?: (error: Error) => void @@ -93,7 +92,6 @@ type PullWakeRunnerState = | `running.reconnecting` | `stopping` -const DEFAULT_MAX_CONCURRENT_CLAIMS = 10 const INITIAL_RECONNECT_BACKOFF_MS = 1_000 const MAX_RECONNECT_BACKOFF_MS = 30_000 const CLAIM_ACTOR_STOP_GRACE_MS = 1_000 @@ -109,6 +107,8 @@ export function createPullWakeRunner( let response: PullWakeStreamResponse | null = null let heartbeatTimer: ReturnType | null = null let eventHeartbeatTimer: ReturnType | null = null + let heartbeatInFlight: Promise | null = null + let heartbeatPending = false let currentOffset = config.offset ?? `-1` let startedAt: string | null = null let streamConnected = false @@ -127,12 +127,10 @@ export function createPullWakeRunner( let claimsFailed = 0 let consecutiveHeartbeatFailures = 0 let acceptingClaims = false - let activeClaimCount = 0 - let runGeneration = 0 let nextReconnectBackoffMs = INITIAL_RECONNECT_BACKOFF_MS let streamResetError: Error | null = null let stopPromise: Promise | null = null - const claimActors = new Map, number>() + const claimActors = new Set>() const wakePath = config.wakeStreamPath ?? @@ -153,10 +151,6 @@ export function createPullWakeRunner( config.claimPath ?? `/_electric/runners/${encodeURIComponent(config.runnerId)}/claim` const claimUrl = appendPathToUrl(config.baseUrl, claimPath) - const maxConcurrentClaims = Math.max( - 1, - Math.floor(config.maxConcurrentClaims ?? DEFAULT_MAX_CONCURRENT_CLAIMS) - ) const toStatus = (): PullWakeRunnerStatus => { switch (state) { @@ -237,15 +231,31 @@ export function createPullWakeRunner( const notifyHeartbeatChange = (): void => { const signal = controller?.signal - if (!signal || signal.aborted || heartbeatIntervalMs <= 0) return + if (!signal || signal.aborted || eventHeartbeatThrottleMs <= 0) return if (eventHeartbeatTimer) return eventHeartbeatTimer = setTimeout(() => { eventHeartbeatTimer = null - void heartbeat(signal) + requestHeartbeat(signal) }, eventHeartbeatThrottleMs) } - const heartbeat = async (signal: AbortSignal): Promise => { + const requestHeartbeat = (signal: AbortSignal): void => { + if (signal.aborted) return + heartbeatPending = true + if (heartbeatInFlight) return + heartbeatInFlight = flushHeartbeats(signal).finally(() => { + heartbeatInFlight = null + }) + } + + const flushHeartbeats = async (signal: AbortSignal): Promise => { + while (heartbeatPending && !signal.aborted) { + heartbeatPending = false + await sendHeartbeat(signal) + } + } + + const sendHeartbeat = async (signal: AbortSignal): Promise => { try { const headers = new Headers(await resolveHeaders()) headers.set(`content-type`, `application/json`) @@ -254,16 +264,13 @@ export function createPullWakeRunner( headers, body: JSON.stringify({ lease_ms: leaseMs, - ...(currentOffset !== undefined - ? { wake_stream_offset: currentOffset } - : {}), + wake_stream_offset: currentOffset, diagnostics: buildDiagnostics(), }), signal, }) lastHeartbeatAt = new Date().toISOString() if (!res.ok) { - lastHeartbeatOk = false throw new Error( `Pull-wake runner heartbeat failed for ${config.runnerId}: ${res.status} ${await res.text()}` ) @@ -289,13 +296,14 @@ export function createPullWakeRunner( const startHeartbeat = (signal: AbortSignal): void => { if (heartbeatIntervalMs <= 0) return - void heartbeat(signal) + requestHeartbeat(signal) heartbeatTimer = setInterval(() => { - void heartbeat(signal) + requestHeartbeat(signal) }, heartbeatIntervalMs) } const stopHeartbeat = (): void => { + heartbeatPending = false if (heartbeatTimer) { clearInterval(heartbeatTimer) heartbeatTimer = null @@ -395,37 +403,6 @@ export function createPullWakeRunner( const isRunningState = (): boolean => state === `starting` || state.startsWith(`running.`) - const waitForClaimCapacity = async ( - signal: AbortSignal - ): Promise => { - let abortListener: (() => void) | null = null - const abortPromise = new Promise((resolve) => { - if (signal.aborted) { - resolve() - return - } - abortListener = () => resolve() - signal.addEventListener(`abort`, abortListener, { once: true }) - }) - - try { - while ( - acceptingClaims && - !signal.aborted && - activeClaimCount >= maxConcurrentClaims - ) { - const inFlight = [...claimActors.keys()] - if (inFlight.length === 0) return true - await Promise.race([...inFlight, abortPromise]).catch(() => undefined) - } - return acceptingClaims && !signal.aborted - } finally { - if (abortListener) { - signal.removeEventListener(`abort`, abortListener) - } - } - } - const claimAndDispatch = async ( event: PullWakeEvent, signal: AbortSignal @@ -455,20 +432,12 @@ export function createPullWakeRunner( } } - const spawnClaimActor = ( - event: PullWakeEvent, - signal: AbortSignal, - generation: number - ): void => { - activeClaimCount++ + const spawnClaimActor = (event: PullWakeEvent, signal: AbortSignal): void => { let actor: Promise actor = claimAndDispatch(event, signal).finally(() => { - if (claimActors.get(actor) === generation) { - activeClaimCount-- - } claimActors.delete(actor) }) - claimActors.set(actor, generation) + claimActors.add(actor) } const waitForClaimActors = async ( @@ -480,7 +449,7 @@ export function createPullWakeRunner( if (remainingMs <= 0) return false const result = await new Promise<`settled` | `timeout`>((resolve) => { const timer = setTimeout(() => resolve(`timeout`), remainingMs) - void Promise.allSettled([...claimActors.keys()]).then(() => { + void Promise.allSettled([...claimActors]).then(() => { clearTimeout(timer) resolve(`settled`) }) @@ -505,10 +474,7 @@ export function createPullWakeRunner( }) } - const consumeWakeStream = async ( - signal: AbortSignal, - generation: number - ): Promise => { + const consumeWakeStream = async (signal: AbortSignal): Promise => { streamResetError = null response = await streamFactory({ url: wakeUrl, @@ -528,12 +494,7 @@ export function createPullWakeRunner( if (event?.type === `wake`) { eventsReceived++ notifyHeartbeatChange() - if (await waitForClaimCapacity(signal)) { - spawnClaimActor(event, signal, generation) - } else { - claimsSkipped++ - notifyHeartbeatChange() - } + if (acceptingClaims && !signal.aborted) spawnClaimActor(event, signal) } if ( response.offset !== undefined && @@ -565,7 +526,7 @@ export function createPullWakeRunner( state = `running.connecting` notifyHeartbeatChange() try { - await consumeWakeStream(signal, runGeneration) + await consumeWakeStream(signal) if (!signal.aborted) { state = `running.reconnecting` notifyHeartbeatChange() @@ -610,7 +571,6 @@ export function createPullWakeRunner( response?.cancel?.(new Error(`pull wake runner stopped`)) if (!(await waitForClaimActors())) { claimActors.clear() - activeClaimCount = 0 } config.runtime.abortWakes() await loop?.catch((err) => { @@ -633,7 +593,21 @@ export function createPullWakeRunner( if (loop || stopPromise) return state = `starting` controller = new AbortController() - runGeneration++ + reconnectCount = 0 + lastError = null + lastErrorAt = null + lastHeartbeatAt = null + lastHeartbeatOk = false + lastClaimAt = null + lastClaimResult = null + lastDispatchAt = null + eventsReceived = 0 + claimsSucceeded = 0 + claimsSkipped = 0 + claimsFailed = 0 + consecutiveHeartbeatFailures = 0 + nextReconnectBackoffMs = INITIAL_RECONNECT_BACKOFF_MS + streamResetError = null startedAt = new Date().toISOString() startHeartbeat(controller.signal) loop = run().finally(() => { @@ -648,7 +622,12 @@ export function createPullWakeRunner( await stopPromise }, async waitForStopped() { + if (stopPromise) { + await stopPromise + return + } await loop + if (stopPromise) await stopPromise }, get running() { return isRunningState() diff --git a/packages/agents-runtime/test/pull-wake-runner.test.ts b/packages/agents-runtime/test/pull-wake-runner.test.ts index 194190a5eb..bc382db673 100644 --- a/packages/agents-runtime/test/pull-wake-runner.test.ts +++ b/packages/agents-runtime/test/pull-wake-runner.test.ts @@ -400,6 +400,128 @@ describe(`createPullWakeRunner`, () => { await runner.stop() }) + it(`coalesces event heartbeats while a heartbeat is in flight`, async () => { + const firstHeartbeat = deferred() + const yieldWake = deferred() + const streamClosed = deferred() + let heartbeatCalls = 0 + const fetchMock = vi.fn( + async (input: RequestInfo | URL, init?: RequestInit) => { + if (String(input).includes(`/heartbeat`)) { + heartbeatCalls++ + JSON.parse(String(init?.body)) + return heartbeatCalls === 1 + ? firstHeartbeat.promise + : Response.json({}) + } + return Response.json(notification(`one`)) + } + ) + vi.stubGlobal(`fetch`, fetchMock) + const testRuntime = runtime() + const streamFactory = vi.fn(async () => ({ + offset: `42`, + async *jsonStream() { + await yieldWake.promise + yield wakeEvent(`one`) + await streamClosed.promise + }, + cancel: () => streamClosed.resolve(), + closed: streamClosed.promise, + })) + + const runner = createPullWakeRunner({ + baseUrl: `http://server`, + runnerId: `runner-1`, + runtime: testRuntime, + heartbeatIntervalMs: 0, + eventHeartbeatThrottleMs: 1, + streamFactory, + }) + + runner.start() + await waitFor(() => { + expect(streamFactory).toHaveBeenCalledTimes(1) + expect(heartbeatCalls).toBe(1) + }) + + yieldWake.resolve() + await waitFor(() => { + expect(testRuntime.dispatchWake).toHaveBeenCalledTimes(1) + }) + await new Promise((resolve) => setTimeout(resolve, 10)) + expect(heartbeatCalls).toBe(1) + + firstHeartbeat.resolve(Response.json({})) + await waitFor(() => { + expect(heartbeatCalls).toBe(2) + }) + + await runner.stop() + }) + + it(`resets heartbeat failure counters across restarts`, async () => { + const heartbeatFailures = [deferred(), deferred()] + const streamClosed = [deferred(), deferred()] + const cancel = [ + vi.fn(() => streamClosed[0]!.resolve()), + vi.fn(() => streamClosed[1]!.resolve()), + ] + let heartbeatCalls = 0 + const fetchMock = vi.fn(async (input: RequestInfo | URL) => { + if (String(input).includes(`/heartbeat`)) { + const failure = heartbeatFailures[heartbeatCalls++] + if (failure) return failure.promise + return Response.json({}) + } + return new Response(null, { status: 204 }) + }) + vi.stubGlobal(`fetch`, fetchMock) + const streamFactory = vi.fn(async () => { + const index = streamFactory.mock.calls.length - 1 + return { + async *jsonStream() { + await streamClosed[index]!.promise + }, + cancel: cancel[index], + closed: streamClosed[index]!.promise, + } + }) + + const runner = createPullWakeRunner({ + baseUrl: `http://server`, + runnerId: `runner-1`, + runtime: runtime(), + heartbeatIntervalMs: 60_000, + eventHeartbeatThrottleMs: 0, + streamFactory, + }) + + runner.start() + await waitFor(() => { + expect(streamFactory).toHaveBeenCalledTimes(1) + expect(heartbeatCalls).toBe(1) + }) + heartbeatFailures[0]!.resolve(new Response(`failed`, { status: 500 })) + await waitFor(() => { + expect(runner.getHealth().last_heartbeat_ok).toBe(false) + }) + await runner.stop() + + runner.start() + await waitFor(() => { + expect(streamFactory).toHaveBeenCalledTimes(2) + expect(heartbeatCalls).toBe(2) + }) + heartbeatFailures[1]!.resolve(new Response(`failed`, { status: 500 })) + await waitFor(() => { + expect(runner.getHealth().last_heartbeat_ok).toBe(false) + }) + + expect(cancel[1]).not.toHaveBeenCalled() + await runner.stop() + }) + it(`resolves async headers before opening the durable stream`, async () => { durableStreamMocks.stream.mockResolvedValueOnce({ offset: `42`, @@ -481,48 +603,6 @@ describe(`createPullWakeRunner`, () => { await runner.stop() }) - it(`pauses claim spawning at maxConcurrentClaims without unbounded queuing`, async () => { - const firstClaim = deferred() - const fetchMock = vi - .fn() - .mockImplementationOnce(async () => firstClaim.promise) - .mockImplementationOnce(async () => Response.json(notification(`two`))) - vi.stubGlobal(`fetch`, fetchMock) - const testRuntime = runtime() - const streamFactory = vi.fn(async () => ({ - offset: `84`, - async *jsonStream() { - yield wakeEvent(`one`) - yield wakeEvent(`two`) - }, - closed: Promise.resolve(), - })) - - const runner = createPullWakeRunner({ - baseUrl: `http://server`, - runnerId: `runner-1`, - runtime: testRuntime, - heartbeatIntervalMs: 0, - maxConcurrentClaims: 1, - streamFactory, - }) - - runner.start() - await waitFor(() => { - expect(fetchMock).toHaveBeenCalledTimes(1) - }) - await new Promise((resolve) => setTimeout(resolve, 20)) - expect(fetchMock).toHaveBeenCalledTimes(1) - - firstClaim.resolve(Response.json(notification(`one`))) - await waitFor(() => { - expect(fetchMock).toHaveBeenCalledTimes(2) - expect(testRuntime.dispatchWake).toHaveBeenCalledTimes(2) - }) - - await runner.stop() - }) - it(`skips dispatch from a claim actor after shutdown begins`, async () => { const claimResponse = deferred() const fetchMock = vi.fn(async () => claimResponse.promise) @@ -754,7 +834,7 @@ describe(`createPullWakeRunner`, () => { consoleError.mockRestore() }) - it(`does not let a stuck claim actor block stop or later claim capacity`, async () => { + it(`does not let a stuck claim actor block stop or a later restart`, async () => { vi.useFakeTimers() const claimStarted = deferred() const secondClaimStarted = deferred() @@ -790,7 +870,6 @@ describe(`createPullWakeRunner`, () => { runnerId: `runner-1`, runtime: testRuntime, heartbeatIntervalMs: 0, - maxConcurrentClaims: 1, streamFactory, }) @@ -811,44 +890,6 @@ describe(`createPullWakeRunner`, () => { expect(fetchMock).toHaveBeenCalledTimes(2) }) - it(`records a skipped claim when stop aborts a capacity wait`, async () => { - vi.useFakeTimers() - const claimStarted = deferred() - const secondEventYielded = deferred() - const fetchMock = vi.fn(async () => { - claimStarted.resolve() - return new Promise(() => {}) - }) - vi.stubGlobal(`fetch`, fetchMock) - const streamFactory = vi.fn(async () => ({ - offset: `84`, - async *jsonStream() { - yield wakeEvent(`one`) - secondEventYielded.resolve() - yield wakeEvent(`two`) - }, - closed: Promise.resolve(), - })) - const runner = createPullWakeRunner({ - baseUrl: `http://server`, - runnerId: `runner-1`, - runtime: runtime(), - heartbeatIntervalMs: 0, - maxConcurrentClaims: 1, - streamFactory, - }) - - runner.start() - await claimStarted.promise - await secondEventYielded.promise - const stopped = runner.stop() - await vi.advanceTimersByTimeAsync(1_000) - await stopped - - expect(runner.getHealth().events_received).toBe(2) - expect(runner.getHealth().claims_skipped).toBe(1) - }) - it(`throws drain errors after recording them and marking the runner stopped`, async () => { const drainError = new Error(`drain failed`) const onError = vi.fn() @@ -917,16 +958,21 @@ describe(`createPullWakeRunner`, () => { const firstStop = runner.stop() const secondStop = runner.stop() + let waitForStoppedResolved = false + const stopped = runner.waitForStopped().then(() => { + waitForStoppedResolved = true + }) await drainStarted.promise expect(testRuntime.abortWakes).toHaveBeenCalledTimes(1) expect(testRuntime.drainWakes).toHaveBeenCalledTimes(1) + expect(waitForStoppedResolved).toBe(false) runner.start() expect(streamFactory).toHaveBeenCalledTimes(1) drainReleased.resolve() - await Promise.all([firstStop, secondStop]) + await Promise.all([firstStop, secondStop, stopped]) expect(testRuntime.abortWakes).toHaveBeenCalledTimes(1) expect(testRuntime.drainWakes).toHaveBeenCalledTimes(1) diff --git a/packages/agents-server-ui/src/lib/sendMessage.ts b/packages/agents-server-ui/src/lib/sendMessage.ts index c97a5552cd..a4553e4396 100644 --- a/packages/agents-server-ui/src/lib/sendMessage.ts +++ b/packages/agents-server-ui/src/lib/sendMessage.ts @@ -219,7 +219,7 @@ export function createSendMessageAction({ ) const res = await serverFetch(url, { method: `POST`, - headers: { 'content-type': `text/plain` }, + headers: { 'content-type': `application/json` }, body: JSON.stringify({ from: sender, key, diff --git a/packages/agents-server-ui/src/main.tsx b/packages/agents-server-ui/src/main.tsx index b60db2210b..4f966fe34f 100644 --- a/packages/agents-server-ui/src/main.tsx +++ b/packages/agents-server-ui/src/main.tsx @@ -36,8 +36,12 @@ function isNgrokHost(input: RequestInfo | URL): boolean { url.hostname.endsWith(`.ngrok-free.app`) || url.hostname === `ngrok.app` || url.hostname.endsWith(`.ngrok.app`) || + url.hostname === `ngrok.dev` || + url.hostname.endsWith(`.ngrok.dev`) || url.hostname === `ngrok.io` || - url.hostname.endsWith(`.ngrok.io`) + url.hostname.endsWith(`.ngrok.io`) || + url.hostname === `ngrok-free.dev` || + url.hostname.endsWith(`.ngrok-free.dev`) ) } catch { return false diff --git a/packages/agents-server/docs/superpowers/specs/2026-05-16-pull-wake-runner-state-machine.html b/packages/agents-server/docs/superpowers/specs/2026-05-16-pull-wake-runner-state-machine.html new file mode 100644 index 0000000000..b9355d9105 --- /dev/null +++ b/packages/agents-server/docs/superpowers/specs/2026-05-16-pull-wake-runner-state-machine.html @@ -0,0 +1,746 @@ + + + + + + Pull-Wake Runner State Machine + + + +
+
+
+

Pull-Wake Runner State Machine

+

+ Runner lifecycle visualization for the proposed XState refactor. + Entity execution stays in processWake; this machine + owns transport, heartbeat, claim, dispatch, diagnostics, and + shutdown. +

+
+
Proposed for review
+
+ +
+ + + + + + + + + + + + + +
+ + + + +
+
+ running + START accepted +
+
+ + + +
+
+ + +
+
+ +
+
+
+

Selected State

+ stopped +
+

+ No stream, heartbeat, or active abort controller. +

+
+
+ +
+
+
!
+
+

Key Invariant

+

+ A claimed wake dispatch must not prevent the runner from reading + and claiming subsequent wake events. Entity execution is + independent runtime work. +

+
+
+
+
↗
+
+

Offset Policy

+

+ wake_stream_offset is read-committed. Unclaimed + pending work must be re-emitted after a missed notification; + claimed work recovers through claim lease expiry. +

+
+
+
+
+ +
+

Actors And Invocations

+
+
+

wakeStreamActor

+

+ Owns the live Durable Streams connection for the runner wake + stream. +

+
    +
  1. Resolve headers.
    2. +
  2. Open live JSON stream at current offset.
    3. +
  3. Emit STREAM_OPENED.
    4. +
  4. Iterate response.jsonStream().
    5. +
  5. Emit STREAM_EVENT and STREAM_OFFSET.
    6. +
  6. Exit only on stop, close, or error.
    7. +
+
+ +
+

claimAndDispatch

+

+ Spawned per compact wake event. It never blocks the stream actor. +

+
    +
  1. POST compact wake to the claim endpoint.
    2. +
  2. Record claimed, no-work, or failed diagnostics.
    3. +
  3. Check the shutdown gate before dispatch.
    4. +
  4. Dispatch full notification to runtime.dispatchWake only if shutdown has not begun.
    5. +
  5. Do not call runtime.drainWakes().
    6. +
+
+ +
+

sendHeartbeat

+

+ Short-lived invocation scheduled by the machine's heartbeat + interval. It owns one HTTP request, not the timer loop. +

+
    +
  1. Read current machine snapshot.
    2. +
  2. Build getHealth() diagnostics.
    3. +
  3. POST lease, offset, and diagnostics.
    4. +
  4. Record heartbeat success or failure.
    5. +
+
+
+
+ +
+

getHealth() Mapping

+
+
+ running + starting or running.* +
+
+ offset + Context wake stream offset +
+
+ stream_connected + True after STREAM_OPENED, false on close/error +
+
+ stream_connected_since + Current connection start; null while disconnected +
+
+ reconnect_count + Incremented on stream open/read errors +
+
+ last_heartbeat_ok + Updated by each scheduled heartbeat invocation +
+
+ last_claim_result + claimed, no_work, error, or null +
+
+ claims_succeeded + Incremented on full wake notification claim +
+
+ claims_failed + Incremented on claim request failure +
+
+ last_error + Latest operational error reported to diagnostics +
+
+ last_dispatch_at + Updated after runtime.dispatchWake +
+
+
+
+ + + + diff --git a/packages/agents-server/docs/superpowers/specs/2026-05-16-pull-wake-runner-state-machine.md b/packages/agents-server/docs/superpowers/specs/2026-05-16-pull-wake-runner-state-machine.md new file mode 100644 index 0000000000..01f0949fc7 --- /dev/null +++ b/packages/agents-server/docs/superpowers/specs/2026-05-16-pull-wake-runner-state-machine.md @@ -0,0 +1,522 @@ +# Pull-Wake Runner State Machine + +## Status + +Proposed design for review. + +## Summary + +Refactor the pull-wake runner lifecycle into an explicit state machine. The +machine owns runner transport and liveness concerns only: wake-stream +connection, offset tracking, runner heartbeat, claim attempts, dispatch into the +runtime, diagnostics, and shutdown. + +Entity wake execution remains outside this machine. The runner machine dispatches +claimed wake notifications to the runtime via `runtime.dispatchWake(...)`; the +runtime continues to execute those wakes through the shared `processWake` +workflow. + +The machine context becomes the single source of truth for +`PullWakeRunner.getHealth()`. Heartbeats continue to send that health snapshot +to agents-server as `runners.diagnostics`, which feeds the existing health +endpoint and desktop UI. + +## Goals + +- Make the runner lifecycle legible and testable as explicit states and events. +- Prevent runner stream consumption from blocking on an entity wake's idle + window. +- Preserve the existing health check contract: `getHealth()` returns the + diagnostics that heartbeat persists to the server. +- Keep pull-wake and webhook wake execution unified through `processWake`. +- Make reconnect, heartbeat, and shutdown behavior easier to reason about. + +## Non-Goals + +- Do not rewrite `processWake` as a state machine. +- Do not change Durable Streams claim semantics. +- Do not change runner registration, authorization, or ownership semantics. +- Do not add runner-level scheduling policy beyond claim and dispatch. +- Do not wait for in-flight entity wakes before reading the next runner wake + event. + +## Current Boundary + +### Runner Lifecycle + +Implemented by `createPullWakeRunner` in +`packages/agents-runtime/src/pull-wake-runner.ts`. + +Responsibilities: + +- Open the runner wake stream. +- Track the current wake stream offset. +- Heartbeat runner liveness and diagnostics to agents-server. +- Claim compact wake events. +- Dispatch full `WakeNotification` objects into the runtime. +- Abort stream reading and in-flight wakes during stop. +- Drain in-flight wakes during stop after aborting. + +### Wake Execution + +Implemented by `processWake` in +`packages/agents-runtime/src/process-wake.ts`. + +Responsibilities: + +- Claim callback lifecycle for the specific wake. +- Preload and tail the entity stream. +- Invoke the entity handler. +- Idle and resume inside one claimed wake when fresh entity work arrives. +- Persist manifest changes. +- Ack consumed stream offsets through the done callback. +- Cleanup entity-stream DBs, producers, and secondary streams. + +## Design Principle + +The runner lifecycle must not contain a `processingWake` state that blocks the +wake stream. Claim and dispatch are short side effects. Entity execution is an +independent spawned workflow tracked by the runtime. + +This is the key invariant: + +> A claimed wake dispatch must not prevent the runner from reading and claiming +> subsequent wake events. + +## Offset Commit Policy + +The runner uses read-commit semantics for the runner wake stream offset. + +`wake_stream_offset` is a delivery cursor for compact runner wake events, not a +work-completion cursor. Work ownership and completion are tracked by server-side +wake, subscription, and claim state. + +There are two separate recovery paths: + +1. If a runner crashes after reading a compact wake event but before attempting + to claim it, there is no claim lease yet. Recovery depends on the server + continuing to treat that work as unclaimed pending work and emitting another + compact wake notification for it. +2. If a runner crashes after successfully claiming work but before dispatching + or completing it, recovery depends on the server-side claim lease expiring and + the server re-emitting the pending work. + +Consequences: + +- The runner may update its local `offset` when `response.offset` advances. +- Heartbeat may persist that read-committed `offset` as `wake_stream_offset`. +- The runner does not need a contiguous claim-safe or dispatch-safe offset + commit log. +- Entity wake completion must not gate runner wake-stream offset progress. +- Pre-claim crashes are recovered through server pending-work re-emission, not + by rewinding the runner wake stream cursor. +- Post-claim failures are recovered through server claim lease expiry and + re-emission. + +This policy depends on the server contract that unresolved pending work is +re-emitted both when it remains unclaimed after a missed notification and when a +claim expires. If the server does not provide both guarantees, the runner must +not persist offsets past wake events that have not at least reached a claim +attempt. + +## State Model + +### Top-Level States + +```txt +stopped +starting +running + connecting + streaming + reconnecting +stopping +``` + +### State Descriptions + +| State | Description | +| ---------------------- | -------------------------------------------------------------------------------------------------------------------- | +| `stopped` | No stream, no heartbeat timer, no active abort controller. | +| `starting` | Allocating controller, setting `started_at`, starting heartbeat, preparing the first stream connection. | +| `running.connecting` | Starting the long-running wake stream actor at the current offset. | +| `running.streaming` | Wake stream is connected and being consumed. | +| `running.reconnecting` | Previous stream failed or closed unexpectedly; wait for backoff before reconnect. | +| `stopping` | Abort stream, stop accepting claim actors, abort or gate in-flight claim actors, then abort and drain runtime wakes. | + +There is intentionally no steady-state `failed` state. The runner is a service +process and should keep trying to run until `stop()` is called. Errors are +recorded in diagnostics and reported through `onError`, then the machine +continues or reconnects as appropriate. + +## Events + +### External Events + +| Event | Payload | Description | +| ------- | ------- | -------------------------------------------------- | +| `START` | none | Start the runner. Ignored unless `stopped`. | +| `STOP` | none | Stop the runner. Valid from any non-stopped state. | + +### Stream Events + +| Event | Payload | Description | +| --------------- | -------------- | -------------------------------------------------------- | +| `STREAM_OPENED` | `{ response }` | Durable stream reader is connected. | +| `STREAM_EVENT` | `{ event }` | Compact wake event received from the runner wake stream. | +| `STREAM_OFFSET` | `{ offset }` | Stream response offset advanced. | +| `STREAM_CLOSED` | none | Stream ended without explicit stop. | +| `STREAM_ERROR` | `{ error }` | Stream connection/read failed. | + +### Heartbeat Events + +| Event | Payload | Description | +| -------------------- | --------------- | -------------------------------------------------------------------- | +| `HEARTBEAT_INTERVAL` | none | Machine-owned delayed transition that invokes one heartbeat request. | +| `HEARTBEAT_OK` | `{ at }` | Heartbeat succeeded. | +| `HEARTBEAT_ERROR` | `{ error, at }` | Heartbeat failed. | + +### Claim/Dispatch Events + +| Event | Payload | Description | +| ------------------ | ---------------------- | --------------------------------------------------- | +| `CLAIM_STARTED` | `{ at }` | Claim request started. | +| `CLAIM_EMPTY` | `{ at }` | Claim returned no work/already claimed. | +| `CLAIM_FAILED` | `{ error, at }` | Claim failed. | +| `CLAIMED` | `{ notification, at }` | Claim returned a full wake notification. | +| `DISPATCH_SKIPPED` | `{ reason, at }` | Claim succeeded but shutdown began before dispatch. | +| `DISPATCHED` | `{ at }` | Notification was passed to `runtime.dispatchWake`. | + +## Context + +The machine context should contain every field needed to derive +`PullWakeRunnerHealth`. + +```ts +interface PullWakeRunnerMachineContext { + runnerId: string + baseUrl: string + wakeUrl: string + heartbeatUrl: string + claimUrl: string + + offset?: string + startedAt: string | null + streamConnected: boolean + streamConnectedSince: string | null + reconnectCount: number + lastError: string | null + lastErrorAt: string | null + lastHeartbeatAt: string | null + lastHeartbeatOk: boolean + lastClaimAt: string | null + lastClaimResult: 'claimed' | 'no_work' | 'error' | null + lastDispatchAt: string | null + eventsReceived: number + claimsSucceeded: number + claimsSkipped: number + claimsFailed: number + claimActors: Set> + + response: PullWakeStreamResponse | null + abortController: AbortController | null +} +``` + +Claim actors are tracked for shutdown only. The runner dispatches claimed wakes +to the runtime and does not try to limit entity wake execution concurrency. + +## `getHealth()` Mapping + +`getHealth()` reads machine state and context only. It should not inspect local +variables outside the machine. + +```ts +function getHealth(snapshot: PullWakeRunnerSnapshot): PullWakeRunnerHealth { + return { + running: snapshot.matches('running') || snapshot.matches('starting'), + offset: snapshot.context.offset, + started_at: snapshot.context.startedAt, + stream_connected: snapshot.context.streamConnected, + stream_connected_since: snapshot.context.streamConnectedSince, + reconnect_count: snapshot.context.reconnectCount, + last_error: snapshot.context.lastError, + last_error_at: snapshot.context.lastErrorAt, + last_heartbeat_at: snapshot.context.lastHeartbeatAt, + last_heartbeat_ok: snapshot.context.lastHeartbeatOk, + last_claim_at: snapshot.context.lastClaimAt, + last_claim_result: snapshot.context.lastClaimResult, + last_dispatch_at: snapshot.context.lastDispatchAt, + events_received: snapshot.context.eventsReceived, + claims_succeeded: snapshot.context.claimsSucceeded, + claims_skipped: snapshot.context.claimsSkipped, + claims_failed: snapshot.context.claimsFailed, + } +} +``` + +Heartbeat sends this same snapshot as `diagnostics`. + +## Transition Sketch + +```txt +stopped + START -> starting + +starting + entry: + - create AbortController + - set startedAt + - schedule heartbeat tick + always -> running.connecting + +running.connecting + invoke wakeStreamActor + STREAM_OPENED -> running.streaming / assign response, streamConnected=true + STREAM_ERROR -> running.reconnecting / record error, reconnectCount++ + STREAM_CLOSED -> running.reconnecting / streamConnected=false + +running + after heartbeat interval -> invoke sendHeartbeat + STOP -> stopping + +running.streaming + STREAM_EVENT -> spawn claimAndDispatch(event), eventsReceived++ + STREAM_OFFSET -> assign offset + STREAM_CLOSED -> running.reconnecting / streamConnected=false + STREAM_ERROR -> running.reconnecting / record error, reconnectCount++ + +running.reconnecting + after backoff -> running.connecting + +stopping + entry: + - abort controller + - cancel stream response + - stop accepting new claim actors + - abort in-flight claim actors + - wait for claim actors that can still dispatch to settle or skip dispatch + - runtime.abortWakes() + invoke runtime.drainWakes + done -> stopped / clear response/controller/stream state + error -> stopped / record error, report error +``` + +`claimAndDispatch` is a spawned actor, not a parent state. It sends diagnostic +events back to the runner machine. `sendHeartbeat` is different: it is a +short-lived invocation scheduled by the machine on a heartbeat interval, not a +peer long-running actor. + +## Actor Sketches + +### `wakeStreamActor` + +Input: + +- `wakeUrl` +- headers provider +- current offset +- abort signal + +Output: + +- `PullWakeStreamResponse` + +Behavior: + +- Resolve headers and open `DurableStream.stream({ live: true, json: true, offset })`. +- Emit `STREAM_OPENED` after the stream response is available. +- Iterate `response.jsonStream()`. +- For each wake event, emit `STREAM_EVENT`. +- After each iteration, if `response.offset` is defined, emit `STREAM_OFFSET`. +- On clean close, emit `STREAM_CLOSED`. +- On error, emit `STREAM_ERROR` unless stopped/aborted. +- Return only when the stream loop exits due to stop/abort, normal close, or + error. This is not a one-shot "open connection" actor; it owns consumption for + the lifetime of the connection. + +### `sendHeartbeat` + +`sendHeartbeat` is a short-lived invoked actor. The machine owns the repeated +schedule using an `after` delay or equivalent timer. The actor owns only one HTTP +request. + +Input: + +- heartbeat URL +- headers provider +- lease ms +- current offset +- `getHealth()` snapshot +- abort signal + +Behavior: + +- POST `{ lease_ms, wake_stream_offset, diagnostics }`. +- Emit `HEARTBEAT_OK` or `HEARTBEAT_ERROR`. + +### `claimAndDispatch` + +Input: + +- compact `PullWakeEvent` +- claim URL +- headers provider +- runtime dispatch function +- claim token header config +- abort signal + +Behavior: + +1. Emit `CLAIM_STARTED`. +2. POST compact wake event to claim endpoint. +3. If response is 204, 409 `ALREADY_CLAIMED`, or 409 `NO_PENDING_WORK`, emit + `CLAIM_EMPTY`. +4. If response is an error, emit `CLAIM_FAILED`. +5. If response contains `{ done: true }`, emit `CLAIM_EMPTY`. +6. Otherwise emit `CLAIMED`. +7. Check the shutdown gate. If stop has begun, do not call + `runtime.dispatchWake`; emit `DISPATCH_SKIPPED` and rely on server claim + lease expiry or an explicit release API if one exists. +8. Otherwise call + `runtime.dispatchWake(notification, { claimHeaders, claimTokenHeader })`. +9. Emit `DISPATCHED`. + +It must not call `runtime.drainWakes()`. + +## Diagnostics Updates + +| Event | Context Update | +| ------------------ | -------------------------------------------------------------------------------------------------------- | +| `START` | `startedAt = now` | +| `STREAM_OPENED` | `streamConnected = true`, `streamConnectedSince = now` | +| `STREAM_CLOSED` | `streamConnected = false`, `streamConnectedSince = null` | +| `STREAM_ERROR` | `streamConnected = false`, `streamConnectedSince = null`, `lastError`, `lastErrorAt`, `reconnectCount++` | +| `STREAM_EVENT` | `eventsReceived++` | +| `STREAM_OFFSET` | `offset = event.offset` | +| `HEARTBEAT_OK` | `lastHeartbeatAt = now`, `lastHeartbeatOk = true` | +| `HEARTBEAT_ERROR` | `lastHeartbeatAt = now`, `lastHeartbeatOk = false`, `lastError`, `lastErrorAt` | +| `CLAIM_STARTED` | `lastClaimAt = now`, `lastClaimResult = null` | +| `CLAIM_EMPTY` | `lastClaimResult = 'no_work'`, `claimsSkipped++` | +| `CLAIM_FAILED` | `lastClaimResult = 'error'`, `claimsFailed++`, `lastError`, `lastErrorAt` | +| `CLAIMED` | `lastClaimResult = 'claimed'`, `claimsSucceeded++` | +| `DISPATCH_SKIPPED` | `claimsSkipped++` | +| `DISPATCHED` | `lastDispatchAt = now` | + +## Concurrency Rules + +1. The stream reader may continue while one or more `claimAndDispatch` actors + are in flight. +2. The runner does not wait for entity wake execution after dispatch. +3. The runner does not expose a claim concurrency limit. Backpressure belongs + in runtime wake execution or the Durable Streams lease/claim contract. +4. Stop aborts future stream reads and claim requests. +5. Stop prevents any claim actor from dispatching after shutdown begins. A claim + actor that has already received a notification must either dispatch before + runtime drain begins or skip dispatch and rely on server claim lease expiry. +6. Stop gates and aborts claim actors before calling `runtime.abortWakes()` and + `runtime.drainWakes()`. +7. Claim actors must use the runner abort signal so stop can cancel in-flight + claim requests. +8. If two compact wake events race for the same work, the claim endpoint remains + authoritative. One may return `claimed`; the other may return `no_work`. +9. Offset progress follows the read-commit policy; claim actor completion does + not block `wake_stream_offset` advancement. + +## Error Handling + +`onError` is reporting-only. It exists so a host such as the Electron desktop +process can write errors to its own logs. It must not decide runner lifecycle. + +The runner should always try to stay alive until `stop()` is called. Operational +errors are written to diagnostics, reported through `onError`, and then handled +with the most local recovery action. + +Recommended handling: + +| Error Source | Recovery Behavior | +| ---------------- | ---------------------------------------------------------------------- | +| Stream open/read | Record error, increment reconnect count, transition to `reconnecting`. | +| Heartbeat | Record degraded diagnostics, continue streaming. | +| Claim | Record claim failure, continue streaming. | +| Dispatch | Record error if synchronous dispatch throws, continue streaming. | +| Stop drain | Record/report error and finish stopping. | + +`onError` callback shape: + +```ts +onError?: (error: Error) => void +``` + +The current boolean return value should be removed as part of this refactor. +Desktop introspection comes from `getHealth()` and persisted +`runners.diagnostics`, not from `onError`. + +## Public API + +The `PullWakeRunner` interface can remain source-compatible except for internal +implementation details. + +```ts +export interface PullWakeRunner { + start: () => void + stop: () => Promise + waitForStopped: () => Promise + readonly running: boolean + readonly offset: string | undefined + getHealth: () => PullWakeRunnerHealth +} +``` + +`running` should be derived from machine state: + +- true for `starting` and `running.*` +- false for `stopped` and `stopping` + +`waitForStopped()` should resolve when the interpreter reaches `stopped`. + +## Testing Requirements + +### Unit Tests + +- Starts in `stopped`; `start()` reaches `running.streaming`. +- `getHealth()` reflects state and context after start. +- Stream wake event spawns claim and dispatch without calling + `runtime.drainWakes()`. +- A second wake event can be claimed and dispatched while the first runtime wake + is still pending. +- Claim 204 and 409 no-work update `claimsSkipped`. +- Claim failure updates `claimsFailed` and `lastError`, reports through + `onError`, and does not stop stream consumption. +- Heartbeat success stores `lastHeartbeatOk = true`. +- Heartbeat failure stores `lastHeartbeatOk = false` and continues stream. +- Stream error transitions through `reconnecting` and increments + `reconnectCount`. +- `stop()` aborts stream/claims, calls `runtime.abortWakes()`, then drains. + +### Integration Tests + +- Built-in desktop runtime registers runner, heartbeats diagnostics, and the UI + receives diagnostics through the `runners` Electric shape. +- Sending to one entity while another entity is idling does not wait for the + idling entity's timeout before claim/dispatch. + +## Migration Plan + +1. Introduce the machine behind `createPullWakeRunner` without changing the + public interface. +2. Keep the existing `PullWakeRunnerHealth` shape unchanged. +3. Replace mutable local diagnostics with machine context. +4. Keep the heartbeat request body unchanged except that diagnostics now come + from `getHealth()`. +5. Update stop ordering: abort stream, stop accepting claim actors, abort or + wait for claim actors to dispatch or skip, then call `runtime.abortWakes()` + and `runtime.drainWakes()`. +6. Make `onError` reporting-only; remove the boolean lifecycle contract. +7. Add tests for non-blocking claim/dispatch before refactoring deeper. + +## Open Questions + +- Should `running` return true while `stopping` drains existing wakes? +- Should reconnect backoff remain delegated to Durable Streams, or be modeled + explicitly in this runner machine? diff --git a/packages/agents-server/src/electric-agents-types.ts b/packages/agents-server/src/electric-agents-types.ts index 621c4fb7b3..26482350ed 100644 --- a/packages/agents-server/src/electric-agents-types.ts +++ b/packages/agents-server/src/electric-agents-types.ts @@ -141,6 +141,9 @@ export interface RunnerHeartbeatRequest { } export type RunnerHealthStatus = `healthy` | `degraded` | `unhealthy` +export type RunnerClientDiagnostics = Partial< + Omit +> export interface RunnerHealthResponse { runner: { @@ -154,7 +157,7 @@ export interface RunnerHealthResponse { last_seen_at: string | null created_at: string } - client: Omit | null + client: RunnerClientDiagnostics | null claims: { active_count: number active: Array<{ diff --git a/packages/agents-server/src/principal.ts b/packages/agents-server/src/principal.ts index 996ccdeb5a..98ed98361e 100644 --- a/packages/agents-server/src/principal.ts +++ b/packages/agents-server/src/principal.ts @@ -47,6 +47,16 @@ export function parsePrincipalUrl(url: string): Principal | null { } } +export function parsePrincipalInput(input: string): Principal | null { + const urlPrincipal = parsePrincipalUrl(input) + if (urlPrincipal) return urlPrincipal + try { + return parsePrincipalKey(input) + } catch { + return null + } +} + export function isPrincipalUrl(url: string): boolean { return parsePrincipalUrl(url) !== null } @@ -54,11 +64,7 @@ export function isPrincipalUrl(url: string): boolean { export function getPrincipalFromRequest(request: Request): Principal | null { const value = request.headers.get(ELECTRIC_PRINCIPAL_HEADER) if (!value) return null - try { - return parsePrincipalKey(value) - } catch { - return null - } + return parsePrincipalInput(value) } export function getDevPrincipal(): Principal { diff --git a/packages/agents-server/src/routing/dispatch-policy.ts b/packages/agents-server/src/routing/dispatch-policy.ts index 407942ac44..c3c0985ae8 100644 --- a/packages/agents-server/src/routing/dispatch-policy.ts +++ b/packages/agents-server/src/routing/dispatch-policy.ts @@ -19,6 +19,8 @@ import type { import type { TenantContext } from './context.js' import type { SubscriptionCreateInput } from '../stream-client.js' +const linkedDispatchSubscriptions = new WeakMap>() + export function subscriptionIdForDispatchTarget( target: DispatchTarget ): string { @@ -128,6 +130,23 @@ function subscriptionHasStream( ) } +function dispatchLinkCacheKey( + ctx: TenantContext, + subscriptionId: string, + streamPath: string +): string { + return `${ctx.service}:${subscriptionId}:${streamPath}` +} + +function getDispatchLinkCache(ctx: TenantContext): Set { + let cache = linkedDispatchSubscriptions.get(ctx.streamClient) + if (!cache) { + cache = new Set() + linkedDispatchSubscriptions.set(ctx.streamClient, cache) + } + return cache +} + function isSubscriptionAlreadyExistsError(err: unknown): boolean { if (!(err instanceof DurableStreamsSubscriptionError)) return false if (err.status === 409) return true @@ -208,7 +227,19 @@ export async function linkEntityDispatchSubscription( ) const target = dispatchPolicy?.targets[0] if (!target) return - await linkStreamToTargetSubscription(ctx, target, entity) + const subscriptionId = subscriptionIdForEntityDispatchTarget( + target, + entity.url + ) + const cacheKey = dispatchLinkCacheKey( + ctx, + subscriptionId, + entity.streams.main + ) + const cache = getDispatchLinkCache(ctx) + if (cache.has(cacheKey)) return + await linkStreamToTargetSubscription(ctx, target, entity, subscriptionId) + cache.add(cacheKey) } export async function unlinkEntityDispatchSubscription( @@ -225,6 +256,9 @@ export async function unlinkEntityDispatchSubscription( target, entity.url ) + getDispatchLinkCache(ctx).delete( + dispatchLinkCacheKey(ctx, subscriptionId, entity.streams.main) + ) await ctx.streamClient .removeSubscriptionStream(subscriptionId, entity.streams.main) .catch((err) => { @@ -239,16 +273,13 @@ export async function unlinkEntityDispatchSubscription( async function linkStreamToTargetSubscription( ctx: TenantContext, target: DispatchTarget, - entity: ElectricAgentsEntity + entity: ElectricAgentsEntity, + subscriptionId: string ): Promise { const streamPath = entity.streams.main await ctx.streamClient.ensure(streamPath, { contentType: `application/json`, }) - const subscriptionId = subscriptionIdForEntityDispatchTarget( - target, - entity.url - ) const existing = await ctx.streamClient.getSubscription(subscriptionId) if (target.type === `runner`) { diff --git a/packages/agents-server/src/routing/runners-router.ts b/packages/agents-server/src/routing/runners-router.ts index c7386a01a8..4a5b74a97d 100644 --- a/packages/agents-server/src/routing/runners-router.ts +++ b/packages/agents-server/src/routing/runners-router.ts @@ -440,9 +440,13 @@ async function runnerHealth( await ctx.entityManager.registry.getRunnerDiagnostics(runnerId) const now = Date.now() - const leaseExpiresAt = runtimeDiagnostics?.liveness_lease_expires_at + const parsedLeaseExpiresAt = runtimeDiagnostics?.liveness_lease_expires_at ? new Date(runtimeDiagnostics.liveness_lease_expires_at).getTime() : null + const leaseExpiresAt = + parsedLeaseExpiresAt !== null && Number.isFinite(parsedLeaseExpiresAt) + ? parsedLeaseExpiresAt + : null let livenessStatus: `online` | `offline` | `expired` if (runner.admin_status === `disabled`) { @@ -511,7 +515,10 @@ async function runnerHealth( id: runner.id, admin_status: runner.admin_status, liveness_status: livenessStatus, - lease_expires_at: runtimeDiagnostics?.liveness_lease_expires_at ?? null, + lease_expires_at: + leaseExpiresAt !== null + ? (runtimeDiagnostics?.liveness_lease_expires_at ?? null) + : null, lease_remaining_ms: leaseExpiresAt !== null ? Math.max(0, leaseExpiresAt - now) : null, wake_stream: runner.wake_stream, diff --git a/packages/agents-server/test/dispatch-policy-routing.test.ts b/packages/agents-server/test/dispatch-policy-routing.test.ts index 75b75cfef4..3352e85fbc 100644 --- a/packages/agents-server/test/dispatch-policy-routing.test.ts +++ b/packages/agents-server/test/dispatch-policy-routing.test.ts @@ -336,6 +336,21 @@ describe(`dispatch policy routing`, () => { `/chat/one`, expect.objectContaining({ payload: `hello` }) ) + + const second = await globalRouter.fetch( + request(`POST`, `/_electric/entities/chat/one/send`, { + payload: `again`, + }), + ctx + ) + + expect(second.status).toBe(204) + expect(ctx.streamClient.getSubscription).toHaveBeenCalledTimes(1) + expect(ctx.streamClient.ensure).toHaveBeenCalledTimes(2) + expect(ctx.entityManager.send).toHaveBeenCalledWith( + `/chat/one`, + expect.objectContaining({ payload: `again` }) + ) }) it(`treats runner subscription create conflicts as an idempotent spawn link`, async () => { diff --git a/packages/agents-server/test/principal.test.ts b/packages/agents-server/test/principal.test.ts index 1c4f5af9b0..ddb8687701 100644 --- a/packages/agents-server/test/principal.test.ts +++ b/packages/agents-server/test/principal.test.ts @@ -1,6 +1,7 @@ import { describe, expect, it } from 'vitest' import { getPrincipalFromRequest, + parsePrincipalInput, parsePrincipalUrl, parsePrincipalKey, principalUrl, @@ -39,6 +40,15 @@ describe(`principal parser`, () => { ) }) + it(`parses principal keys and URLs through one canonical input parser`, () => { + expect(parsePrincipalInput(`user:alice@example.com`)?.url).toBe( + `/principal/user%3Aalice%40example.com` + ) + expect(parsePrincipalInput(`/principal/user:alice@example.com`)?.url).toBe( + `/principal/user%3Aalice%40example.com` + ) + }) + it(`rejects invalid keys`, () => { for (const key of [`userkyle`, `user:`, `user:/kyle`, `admin:kyle`]) { expect(() => parsePrincipalKey(key)).toThrow() @@ -52,4 +62,12 @@ describe(`principal parser`, () => { expect(getPrincipalFromRequest(request)).toBeNull() }) + + it(`accepts canonical principal URLs in request headers`, () => { + const request = new Request(`http://server`, { + headers: { 'electric-principal': `/principal/user%3Akyle` }, + }) + + expect(getPrincipalFromRequest(request)?.key).toBe(`user:kyle`) + }) }) diff --git a/packages/agents-server/test/runners-router.test.ts b/packages/agents-server/test/runners-router.test.ts index 4e23641e02..0ddba15ade 100644 --- a/packages/agents-server/test/runners-router.test.ts +++ b/packages/agents-server/test/runners-router.test.ts @@ -575,6 +575,30 @@ describe(`runner routes`, () => { expect(body.health.issues).toContain(`Client reports stream disconnected`) }) + it(`ignores invalid runner lease timestamps in health output`, async () => { + const ctx = buildContext() + vi.mocked( + ctx.entityManager.registry.getRunnerDiagnostics + ).mockResolvedValue({ + runner_id: `runner-1`, + owner_principal: `/principal/user%3Aowner%40example.com`, + liveness_lease_expires_at: `not-a-date`, + last_seen_at: new Date().toISOString(), + updated_at: new Date().toISOString(), + }) + + const response = await globalRouter.fetch( + request(`GET`, `/_electric/runners/runner-1/health`), + ctx + ) + + expect(response.status).toBe(200) + const body = (await response.json()) as Record + expect(body.runner.lease_expires_at).toBeNull() + expect(body.runner.lease_remaining_ms).toBeNull() + expect(body.runner.liveness_status).toBe(`offline`) + }) + it(`uses the pending stream from multi-stream claim responses`, async () => { const ctx = buildContext({ principal: { From 8f057525df0412e7572398065012068dda687246 Mon Sep 17 00:00:00 2001 From: Ilia Borovitinov Date: Tue, 19 May 2026 15:08:34 +0300 Subject: [PATCH 37/37] some additional logging & removed stale docs --- docs/agents-development.md | 205 ----- docs/agents-principals-implementation-plan.md | 790 ------------------ .../agents-principals-implementation-plan.pdf | Bin 264243 -> 0 bytes packages/agents-runtime/src/process-wake.ts | 37 +- ...-05-16-pull-wake-runner-state-machine.html | 746 ----------------- ...26-05-16-pull-wake-runner-state-machine.md | 522 ------------ 6 files changed, 34 insertions(+), 2266 deletions(-) delete mode 100644 docs/agents-development.md delete mode 100644 docs/agents-principals-implementation-plan.md delete mode 100644 docs/agents-principals-implementation-plan.pdf delete mode 100644 packages/agents-server/docs/superpowers/specs/2026-05-16-pull-wake-runner-state-machine.html delete mode 100644 packages/agents-server/docs/superpowers/specs/2026-05-16-pull-wake-runner-state-machine.md diff --git a/docs/agents-development.md b/docs/agents-development.md deleted file mode 100644 index 70e3ca6bc6..0000000000 --- a/docs/agents-development.md +++ /dev/null @@ -1,205 +0,0 @@ -# Electric Agents — Development Guide - -## Package overview - -The agents subsystem lives in seven packages under `packages/`: - -| Package | Description | -| --------------------------------- | ------------------------------------------------------------------------------------- | -| `agents-runtime` | Core runtime — entity definitions, context, handler lifecycle | -| `agents-mcp` | MCP (Model Context Protocol) bridge library used by built-in agents | -| `agents-server` | Orchestration server — wake registry, scheduling, Electric + Postgres integration | -| `agents` | Built-in agents (Horton & Worker) with tools (bash, read, write, edit, fetch, search) | -| `agents-server-ui` | React dashboard for agent monitoring and interaction | -| `agents-desktop` | Electron wrapper around `agents-server-ui` for a native desktop experience | -| `agents-server-conformance-tests` | Conformance test suite for agents-server | - -## Prerequisites - -- **Docker Desktop** running (for Postgres + Electric) -- **Node.js** and **pnpm** (see `.tool-versions` for exact versions) -- **`.env` file** at the project root with at least `ANTHROPIC_API_KEY` (needed by built-in agents). Both entrypoints call `process.loadEnvFile()` on startup, loading from the current working directory — so always run entrypoints from the project root. - -## Quick start: `./scripts/dev.sh` - -For day-to-day development, use the bundled dev script: - -```sh -./scripts/dev.sh build # one-shot install + build of all required packages -./scripts/dev.sh start # docker + 5 dev processes; Ctrl-C to stop -./scripts/dev.sh start --detach # same, but exits after spawning (logs to .dev-logs/) -./scripts/dev.sh start --with-agents # also spawn built-in agents (Horton + Worker) -./scripts/dev.sh desktop # run the Electron desktop app in this terminal -./scripts/dev.sh stop # stop processes + docker compose down -./scripts/dev.sh teardown # stop + remove Postgres volume + .streams-data/ -./scripts/dev.sh status # show which services are running -``` - -`desktop` is a separate command because the Electron app is interactive — it opens a window. Run it in its own terminal after `start` has the rest of the stack up; Ctrl-C in that terminal closes the app without touching the backing services. - -`build` covers `typescript-client`, `agents-runtime`, `agents-mcp`, `agents-server`, and `agents`. Re-run it after any dep change before restarting — entrypoints do not auto-restart on `dist/` rebuilds. - -**Built-in agents (`packages/agents`)** register against `agents-server` at startup and will fail with `Stream not found` if they race ahead of it. Pass `--with-agents` to `start` to spawn them after `agents-server` binds `:4437`. Without the flag, run them manually in a separate terminal once `start` reports the server is up — Ctrl-C in that terminal stops only the built-in agents: - -```sh -ELECTRIC_AGENTS_SERVER_URL=http://localhost:4437 \ - node packages/agents/dist/entrypoint.js -``` - -The rest of this document describes the manual flow that the script automates. - -## Starting the dev environment - -All commands below assume you are in the project root. All `pnpm dev` commands use `tsdown --watch` (or Vite for the UI) — they do an initial build then watch for changes. The build order matters because packages import from each other's `dist/`. - -### Step 1 — Install dependencies and build workspace prerequisites - -In a fresh checkout or worktree, workspace packages have no `dist/` directories. The agent packages depend on `@electric-sql/client` (the typescript-client) and on `@electric-ax/agents-mcp` at runtime, so both must be built before starting any agent server. - -```sh -pnpm install -pnpm -C packages/typescript-client build -pnpm -C packages/agents-mcp build -``` - -### Step 2 — Start backing services (Postgres + Electric + Jaeger) - -```sh -docker compose -f packages/agents-server/docker-compose.dev.yml up -d -``` - -Services will be available at: - -- PostgreSQL: `localhost:5432` (electric_agents/electric_agents) -- Electric API: `http://localhost:3060` -- Jaeger UI: `http://localhost:16686` (tracing) - -### Step 3 — Build agents-runtime - -`agents-server` and `agents` both depend on `agents-runtime`, so it must be built first. - -```sh -pnpm -C packages/agents-runtime dev -# wait for "Build complete" before step 4 -``` - -### Step 4 — Build agents-server and agents - -These can be started in parallel once the runtime is built. - -```sh -# Terminal 2: -pnpm -C packages/agents-server dev - -# Terminal 3: -pnpm -C packages/agents dev -``` - -Wait for both "Build complete" messages before step 5. - -### Step 5 — Start the server processes - -Run entrypoints from the project root so they pick up the root `.env` file. - -```sh -# Terminal 4: agents-server -DATABASE_URL=postgresql://electric_agents:electric_agents@localhost:5432/electric_agents \ - ELECTRIC_AGENTS_ELECTRIC_URL=http://localhost:3060 \ - ELECTRIC_INSECURE=true \ - node packages/agents-server/dist/entrypoint.js -``` - -The agents-server will start on `http://localhost:4437` with an embedded durable streams server. - -```sh -# Terminal 5: built-in agents (Horton + Worker) -ELECTRIC_AGENTS_SERVER_URL=http://localhost:4437 \ - node packages/agents/dist/entrypoint.js -``` - -The built-in agents server starts on `http://localhost:4448` and auto-registers Horton and Worker entity types. - -### Step 6 — Start the agents UI dashboard - -```sh -pnpm -C packages/agents-server-ui dev -``` - -Vite dev server with HMR — changes appear instantly. - -## Environment variables reference - -### agents-server - -| Variable | Default | Description | -| ------------------------------------- | --------- | --------------------------------------------------- | -| `DATABASE_URL` | — | Postgres connection URL (required) | -| `ELECTRIC_AGENTS_ELECTRIC_URL` | — | Electric sync service URL | -| `ELECTRIC_AGENTS_HOST` | `0.0.0.0` | Bind address | -| `ELECTRIC_AGENTS_PORT` | `4437` | Server port | -| `ELECTRIC_AGENTS_BASE_URL` | — | Public webhook base URL | -| `ELECTRIC_AGENTS_STREAMS_DATA_DIR` | — | Local streams data directory | -| `ELECTRIC_AGENTS_DURABLE_STREAMS_URL` | — | External durable streams URL (omit to use embedded) | - -### agents (built-in) - -| Variable | Default | Description | -| ------------------------------ | ----------- | ---------------------------- | -| `ELECTRIC_AGENTS_SERVER_URL` | — | agents-server URL (required) | -| `ANTHROPIC_API_KEY` | — | Claude API key (required) | -| `ELECTRIC_AGENTS_BUILTIN_HOST` | `127.0.0.1` | Bind address | -| `ELECTRIC_AGENTS_BUILTIN_PORT` | `4448` | Server port | - -## Running tests - -```sh -# Runtime unit tests (no services needed) -cd packages/agents-runtime -pnpm test - -# Server tests (requires Postgres + Electric via docker-compose.dev.yml) -cd packages/agents-server -pnpm test - -# Built-in agents tests -cd packages/agents -pnpm test - -# All with coverage -pnpm coverage # in any agent package -``` - -## Iterating on agent packages - -All agent packages use `tsdown` for building. The `pnpm dev` command in each starts a watch-mode rebuild, so changes are picked up automatically. - -- **Runtime changes** (`agents-runtime`): Rebuild propagates to `agents-server` and `agents` since they depend on it via workspace links. -- **Server changes** (`agents-server`): Restart `node dist/entrypoint.js` after rebuild (watch mode rebuilds but does not restart the process). -- **Agent logic changes** (`agents`): Same — restart the entrypoint after rebuild. -- **UI changes** (`agents-server-ui`): Vite HMR — changes appear instantly. - -## Working with examples - -The `examples/deep-survey` example demonstrates a custom agent with its own entity types: - -```sh -cd examples/deep-survey -pnpm install -pnpm dev # starts both server (tsx watch) and UI (vite) in parallel -``` - -It requires the agents-server backing services (Postgres + Electric) to be running. - -## Local state - -- **Postgres** (docker volume) — entity types, entities, wake registrations, scheduling state. -- **Durable streams** — in-memory by default in dev. Data resets on server restart. Set `ELECTRIC_AGENTS_STREAMS_DATA_DIR` to persist streams to disk (uses lmdb + log files). - -To clear all state: stop the servers and run `docker compose down -v` to remove the Postgres volume. - -## Teardown - -```sh -docker compose -f packages/agents-server/docker-compose.dev.yml down # stop services -docker compose -f packages/agents-server/docker-compose.dev.yml down -v # stop + remove volumes -``` diff --git a/docs/agents-principals-implementation-plan.md b/docs/agents-principals-implementation-plan.md deleted file mode 100644 index 92b5f11a3d..0000000000 --- a/docs/agents-principals-implementation-plan.md +++ /dev/null @@ -1,790 +0,0 @@ -# Principals implementation plan - -Issue: - -## Goal - -Add **principals** as a first-class entity type so every action in the agents system traces to an owning identity. - -Principals are entity streams addressed as: - -```txt -/principal/user:kyle -/principal/agent:ci-bot -/principal/service:github -/principal/system:framework -/principal/system:dev-local -``` - -Inbound requests carry the principal in a trusted header set by edge/auth middleware: - -```txt -Electric-Principal: user:kyle -``` - -In local/dev mode, missing headers default to: - -```txt -system:dev-local -``` - -Because agents are pre-release, there is no backwards compatibility path for unauthenticated/no-principal requests. API routes should error if the request has no principal. - -## Request context - -Move request identity from user-centric naming to principal-centric naming: - -```ts -// before -authenticatedUser?: AuthenticatedRequestUser - -// after -principal: Principal -``` - -All routes, including internal routes, should include a principal. Use one principal-aware entry point into the verbs instead of parallel unauthenticated/internal code paths. - -There are no first-class "users" in the agents runtime; there are only principals, one kind of which may be `user`. - -## Creator field - -Use **`created_by`** for the immutable entity creator/owner field. It stores a principal entity URL, e.g. `/principal/user:kyle`. - -## Principal model - -Add a new module: - -```txt -packages/agents-server/src/principal.ts -``` - -Types: - -```ts -export type PrincipalKind = 'user' | 'agent' | 'service' | 'system' - -export interface Principal { - kind: PrincipalKind - id: string - key: string // `${kind}:${id}` - url: string // `/principal/${kind}:${id}` -} -``` - -Header constant: - -```ts -export const ELECTRIC_PRINCIPAL_HEADER = 'electric-principal' -``` - -Helpers: - -```ts -export function parsePrincipalKey(input: string): Principal -export function principalUrl(key: string): string -export function parsePrincipalUrl(url: string): Principal | null -export function isPrincipalUrl(url: string): boolean -export function getPrincipalFromRequest(request: Request): Principal | null -export function getDevPrincipal(): Principal -``` - -Validation rules: - -- Principal key is `{kind}:{id}`. -- Split on the first colon only. -- Additional colons are allowed in the id so principals can use ids from external systems. -- Kind is one of: - - `user` - - `agent` - - `service` - - `system` -- ID must be non-empty. -- ID must not contain `/`. - -Examples: - -```txt -user:kyle ✅ -agent:ci-bot ✅ -service:github ✅ -system:framework ✅ -system:dev-local ✅ -user:clerk:user_123 ✅ id contains additional colon -service:github:installation ✅ id contains additional colon -user:/kyle ❌ slash -admin:kyle ❌ unknown kind -``` - -## Request principal extraction - -Wire request extraction wherever the server builds `TenantContext`. - -Likely files to inspect/change: - -```txt -packages/agents-server/src/host.ts -packages/agents-server/src/routing/global-router.ts -packages/agents-server/src/entrypoint-lib.ts -packages/agents-server/src/dev-asserted-auth.ts -packages/agents-server/src/authenticated-user-format.ts -packages/agents-server/src/electric-agents-types.ts -``` - -The `authenticated-user-format` module may become obsolete or should be renamed/reworked as a principal formatter/parser. - -Desired behavior: - -```ts -const headerValue = request.headers.get('electric-principal') - -const principal = headerValue - ? parsePrincipalKey(headerValue) - : isDevOrInsecure - ? getDevPrincipal() // system:dev-local - : null -``` - -If no principal exists, return an auth/invalid-request error. - -As part of this, replace user-centric request auth names with principal-centric names: - -- `AuthenticatedRequestUser` → `AuthenticatedRequestPrincipal` or just `RequestPrincipal` -- `AuthenticateRequest` should return a `Principal`/principal assertion, not a user object -- `ctx.authenticatedUser` → `ctx.principal` -- fields such as `userId` should become principal fields, e.g. `principal.key`, `principal.url`, `principal.kind`, `principal.id` - -The agents server trusts this header. Auth middleware/proxy is responsible for setting it correctly. - -## Built-in `principal` entity type - -Principals must be normal entities, so ensure a built-in entity type named `principal` exists. - -Add to `PostgresRegistry`: - -```ts -async ensureEntityType(et: ElectricAgentsEntityType): Promise -``` - -Behavior: - -- Insert if missing. -- If present, return existing unchanged. -- Do not bump revisions on every server startup. - -Seed at server/runtime startup: - -```ts -await registry.ensureEntityType({ - name: 'principal', - description: 'built-in principal entity', - inbox_schemas: { - update_identity: principalUpdateIdentityMessageSchema, - }, - state_schemas: { - identity: principalIdentityStateSchema, - }, - revision: 1, - created_at: now, - updated_at: now, -}) -``` - -The `principal` entity type has one built-in state collection: - -- `identity` — trusted profile/identity information for the principal. - -The `principal` entity type has one built-in inbox message: - -- `update_identity` — request to create/update the `identity` state row. - -The `principal` entity type is immutable from user/API code. It is created and modified only by system code. - -## Principal identity state - -Add built-in schema definitions for principal identity. - -Identity state row: - -```ts -const principalIdentityStateSchema = { - type: 'object', - additionalProperties: false, - required: ['kind', 'id', 'key', 'url', 'updated_at'], - properties: { - kind: { enum: ['user', 'agent', 'service', 'system'] }, - id: { type: 'string' }, - key: { type: 'string' }, - url: { type: 'string' }, - display_name: { type: 'string' }, - email: { type: 'string' }, - avatar_url: { type: 'string' }, - auth_provider: { type: 'string' }, - auth_subject: { type: 'string' }, - claims: { - type: 'object', - additionalProperties: true, - }, - created_at: { type: 'string' }, - updated_at: { type: 'string' }, - }, -} -``` - -Identity state uses a single stable key: - -```txt -identity/self -``` - -Update message schema: - -```ts -const principalUpdateIdentityMessageSchema = { - type: 'object', - additionalProperties: false, - required: ['identity'], - properties: { - identity: principalIdentityStateSchema, - }, -} -``` - -The `update_identity` inbox message is how principal identity is created/updated. Anyone can target a principal entity with this message shape at the protocol/schema level, but authorization must restrict who is allowed to send it. - -In Electric Cloud, a built-in system entity should send `update_identity` when: - -- a user logs in via Google/SSO/etc. -- a CI bot principal is created -- a service integration principal is provisioned -- identity/profile data changes in the upstream auth system - -Non-system principals should not be allowed to send `update_identity` unless explicitly authorized by deployment-specific policy. - -## Persistence changes - -Add migration: - -```txt -packages/agents-server/drizzle/0006_principals.sql -``` - -SQL: - -```sql -ALTER TABLE entities - ADD COLUMN created_by text; - -CREATE INDEX idx_entities_created_by - ON entities (tenant_id, created_by); -``` - -Update Drizzle schema in: - -```txt -packages/agents-server/src/db/schema.ts -``` - -Add to `entities`: - -```ts -createdBy: text(`created_by`), -``` - -Update server types in: - -```txt -packages/agents-server/src/electric-agents-types.ts -``` - -Add to `ElectricAgentsEntity`: - -```ts -created_by?: string -``` - -Add to `PublicElectricAgentsEntity`: - -```ts -created_by?: string -``` - -Add to `TypedSpawnRequest`: - -```ts -created_by?: string -``` - -Update `toPublicEntity()` to include `created_by`. - -Update registry in: - -```txt -packages/agents-server/src/entity-registry.ts -``` - -- `createEntity()` writes `createdBy: entity.created_by ?? null` -- `rowToEntity()` reads `created_by` -- `listEntities()` accepts `created_by?: string` -- `listEntities()` filters on `entities.createdBy` - -Update route list filtering in: - -```txt -packages/agents-server/src/routing/entities-router.ts -``` - -Support: - -```txt -GET /_electric/entities?created_by=/principal/user:kyle -``` - -## Lazy principal materialization - -Add to `EntityManager`: - -```ts -async ensurePrincipal(principal: Principal): Promise -``` - -Behavior: - -1. Check `registry.getEntity(principal.url)`. -2. If found, return it. -3. If missing, create a `principal` entity at that URL. - -Principal spawn details: - -```ts -await this.spawn('principal', { - instance_id: principal.key, - args: { - kind: principal.kind, - id: principal.id, - key: principal.key, - }, - tags: { - principal_kind: principal.kind, - principal_id: principal.id, - }, - created_by: principal.url, -}) -``` - -On creation, also initialize `identity/self` with the built-in identity state: - -```ts -{ - kind: principal.kind, - id: principal.id, - key: principal.key, - url: principal.url, - created_at: now, - updated_at: now, -} -``` - -If trusted auth/profile claims are available during materialization, include the mapped fields in `identity/self`. - -Need to avoid recursive principal creation. Either: - -- Add an internal spawn option such as `{ skipPrincipalEnsure: true }`, or -- Implement `ensurePrincipal()` using a lower-level helper that creates the entity without trying to ensure `created_by` first. - -Recommended internal rule for `created_by`: - -```ts -const createdBy = req.created_by ?? parentEntity?.created_by -``` - -This means child/worker agents inherit the initiating principal from their parent unless explicitly overridden. - -For principal entities themselves, `created_by` can be their own URL: - -```txt -/principal/user:kyle created_by=/principal/user:kyle -/principal/system:dev-local created_by=/principal/system:dev-local -``` - -## Route behavior - -File: - -```txt -packages/agents-server/src/routing/entities-router.ts -``` - -### Principal route materialization - -Current `withExistingEntity()` returns 404 if the entity is missing. Adjust for principal URLs: - -```ts -if (!entity && request.params.type === 'principal') { - const principal = parsePrincipalKey(request.params.instanceId) - const materialized = await ctx.entityManager.ensurePrincipal(principal) - request.entityRoute = { entityUrl, entity: materialized } - return undefined -} -``` - -This enables: - -```txt -POST /_electric/entities/principal/user:bob/send -``` - -to create Bob's principal stream on first reference. - -### Spawn - -In `spawnEntity()`: - -1. Require `ctx.principal`. -2. Ensure the inbound principal exists. -3. Pass `created_by: ctx.principal.url` to `entityManager.spawn()`. -4. Use principal as the initial message sender. - -Pseudo-code: - -```ts -const principal = requirePrincipal(ctx) -await ctx.entityManager.ensurePrincipal(principal) - -const entity = await ctx.entityManager.spawn(request.params.type, { - instance_id: request.params.instanceId, - args: parsed.args, - tags: parsed.tags, - parent: parsed.parent, - dispatch_policy: dispatchPolicy, - initialMessage: undefined, - wake: parsed.wake, - created_by: principal.url, -}) - -if (parsed.initialMessage !== undefined) { - await ctx.entityManager.send(entity.url, { - from: principal.url, - payload: parsed.initialMessage, - }) -} -``` - -### Send - -In `sendEntity()`: - -1. Require `ctx.principal`. -2. Ensure the inbound principal exists. -3. Default `from` to `ctx.principal.url`. -4. Reject client-supplied `from` if present and not equal to `ctx.principal.url`. - -Recommended v1 security posture: - -- HTTP `send` should not allow arbitrary `from` spoofing. -- Use `ctx.principal.url` as sender. -- Internal APIs/tools should also pass through the same principal-aware verb entry point. - -So update route behavior to: - -```ts -await ctx.entityManager.send(entityUrl, { - from: principal.url, - payload: parsed.payload, - key: parsed.key, - type: parsed.type, -}) -``` - -Do not allow callers to assert arbitrary principals via body. The request/context principal is the sender. - -### Principal identity updates - -Principal entities accept an `update_identity` inbox message, but ordinary principals must not be able to send it by default. - -Route/send authorization should enforce: - -- `update_identity` to `/principal/*` is allowed from built-in system principals. -- `update_identity` from non-system principals is rejected unless deployment policy explicitly allows it. -- Other messages to `/principal/*` can continue through normal send authorization. - -This preserves the uniform send mechanism while letting Electric Cloud run a built-in system entity that creates/updates principals from trusted auth events. - -### Sharing/authz tags - -Sharing is app-specific and should not be first-class in this PR. Apps can build sharing systems with tags or entity state. - -A future PR may reserve protected tag namespaces such as `share:*`, `acl:*`, `authz:*`, or `system:*`, but this principals PR does not implement protected tag namespaces or tag authorization rules. - -### Schedule/future-send - -Future-send route currently accepts `from` in body. - -For the same anti-spoofing reason, schedule routes should ignore/reject body `from` and use `ctx.principal.url`: - -```ts -from: principal.url -``` - -## Inter-principal messaging - -Once principal entities are lazy-materialized, the existing send mechanism works: - -```http -POST /_electric/entities/principal/user:bob/send -Electric-Principal: user:kyle -Content-Type: application/json - -{ - "payload": { "text": "hello" } -} -``` - -Result: - -- `/principal/user:kyle` is ensured. -- `/principal/user:bob` is ensured. -- Bob's principal inbox receives a message with: - -```ts -from: '/principal/user:kyle' -``` - -## Handler/runtime context - -Issue requirement: - -> Authorization v1: Handler decides which tools/functions to expose based on principal context. - -Handlers need access to principal information. - -Likely files to inspect/change: - -```txt -packages/agents-runtime/src/create-handler.ts -packages/agents-runtime/src/setup-context.ts -packages/agents-runtime/src/types.ts -packages/agents-server/src/entity-manager.ts // enrichPayload() -``` - -`EntityManager.enrichPayload()` currently injects `entity` info into webhook payloads. Add: - -```ts -entity: { - ..., - createdBy: entity.created_by, -}, -principal: entity.created_by - ? { - url: entity.created_by, - key: parsePrincipalUrl(entity.created_by)?.key ?? null, - } - : undefined, -``` - -Then expose this through runtime handler context as: - -```ts -ctx.principal -ctx.entity.created_by -``` - -This enables handler-level authorization: - -```ts -const tools = ctx.principal?.kind === 'user' ? userTools : serviceTools - -await runAgent({ tools }) -``` - -## Authorization v1 - -Keep authorization flexible and handler-level. - -What this implementation should do now: - -- Identify the inbound principal. -- Persist owner/creator on spawned agents. -- Prevent routes from spoofing `from` in request bodies. -- Materialize principal streams lazily. -- Expose principal to handlers. - -What this implementation should **not** do yet: - -- Capability expressions in entity streams. -- Named capability sets. -- Delegation semantics. -- General cross-principal policy engine beyond the built-in `update_identity` restriction. -- First-class sharing system, protected tag namespaces, public sharing links, or signed URLs. -- Principal garbage collection. - -## Tests - -### Principal parser tests - -File: - -```txt -packages/agents-server/test/principal.test.ts -``` - -Cases: - -- `user:kyle` → `/principal/user:kyle` -- `agent:ci-bot` → `/principal/agent:ci-bot` -- `service:github` → `/principal/service:github` -- `system:framework` → `/principal/system:framework` -- `system:dev-local` → `/principal/system:dev-local` -- reject missing colon -- allow additional colons in the id, e.g. `user:clerk:user_123` -- reject slash -- reject empty id -- reject unknown kind - -### Spawn records owner principal - -- Send `PUT /_electric/entities//` with `Electric-Principal: user:kyle`. -- Assert spawned entity has: - -```ts -created_by: '/principal/user:kyle' -``` - -- Assert `/principal/user:kyle` exists. - -### Child spawn inherits `created_by` - -Using `EntityManager.spawn()` directly: - -1. Create parent with `created_by: '/principal/user:kyle'`. -2. Spawn child with `parent` and no explicit `created_by`. -3. Assert child has same `created_by`. - -### Send uses principal as `from` - -- Create an entity. -- `POST /send` with `Electric-Principal: user:kyle` and no body `from`. -- Read stream. -- Assert inbox event value has: - -```ts -from: '/principal/user:kyle' -``` - -### Public send does not allow spoofed `from` - -- `POST /send` with `Electric-Principal: user:kyle` and body `from: '/principal/user:alice'`. -- Assert the route rejects the request with 400/422. - -### Sending to unmaterialized principal creates it - -- `POST /_electric/entities/principal/user:bob/send` -- Header: `Electric-Principal: user:kyle` -- Assert: - - `/principal/user:bob` exists. - - `/principal/user:kyle` exists. - - Bob's inbox has `from: /principal/user:kyle`. - -### Principal identity is initialized - -- Materialize `/principal/user:kyle`. -- Assert its state contains `identity/self` with: - -```ts -{ - kind: 'user', - id: 'kyle', - key: 'user:kyle', - url: '/principal/user:kyle', -} -``` - -### System principal can update identity - -- Send `update_identity` to `/principal/user:kyle` from a built-in system principal. -- Assert `identity/self` is updated with trusted fields such as `email`, `display_name`, `auth_provider`, and `auth_subject`. - -### Non-system principal cannot update identity - -- Send `update_identity` to `/principal/user:kyle` from `/principal/user:alice`. -- Assert the route rejects the request with 401/403. - -### Missing principal in production fails - -- Simulate production/non-dev/non-insecure context. -- Spawn/send without `Electric-Principal`. -- Assert 401/400. - -### Dev fallback - -- Simulate dev/insecure context. -- Spawn/send without header. -- Assert: - -```ts -ctx.principal.url === '/principal/system:dev-local' -created_by === '/principal/system:dev-local' -``` - -### List by owner - -- Spawn two agents under `user:kyle` and one under `user:alice`. -- Request: - -```txt -GET /_electric/entities?created_by=/principal/user:kyle -``` - -- Assert only Kyle's entities are returned. - -## Implementation order - -### Phase 1 — Types and persistence - -1. Add `principal.ts` parser/helpers. -2. Add `created_by` migration. -3. Update Drizzle schema. -4. Update entity types/public types. -5. Update registry create/read/list. - -### Phase 2 — Built-in principal type - -6. Add `PostgresRegistry.ensureEntityType()`. -7. Add built-in `principal` identity state and `update_identity` inbox schemas. -8. Seed built-in `principal` entity type during server startup. - -### Phase 3 — Context and route behavior - -9. Replace user-centric request context with principal-centric context: - - `AuthenticatedRequestUser` → principal-oriented type - - `ctx.authenticatedUser` → `ctx.principal` - - update/rename `authenticated-user-format.ts` if still needed -10. Wire header extraction and dev fallback. -11. Require principal for all API routes. -12. Lazy-materialize `/principal/*` in `withExistingEntity()`. -13. Ensure inbound principal during spawn/send. -14. Persist `created_by` on spawn. -15. Use principal as `from` for send/initial messages/schedules. -16. Enforce the built-in `update_identity` send restriction for principal entities. - -### Phase 4 — Runtime handler context - -17. Include `createdBy`/principal in webhook enrichment. -18. Expose `ctx.principal` in runtime handler context. - -### Phase 5 — Tests/docs - -19. Add parser tests. -20. Add spawn/send/materialization/list tests. -21. Add identity initialization and `update_identity` authorization tests. -22. Update agents development docs with header/trust-boundary/dev fallback notes. - -## Implementation decisions - -1. All API routes require a principal. Missing principal is an error, except local/dev mode where the server supplies `system:dev-local`. -2. Internal routes/code paths should include a principal too. There should be one principal-aware entry point into verbs. -3. Request/body `from` must not spoof principals. Use `ctx.principal.url` as the sender; reject mismatches. -4. `created_by` is immutable. -5. The `principal` entity type is immutable from user/API code and may only be created/modified by system code. -6. Principal identity lives in built-in `identity/self` state. -7. Principal identity updates use the built-in `update_identity` inbox message and are restricted to built-in system principals unless deployment policy explicitly allows otherwise. -8. Sharing is app-specific and out of scope for this PR. Apps may use tags or entity state; protected tag namespaces can be added in a future PR. diff --git a/docs/agents-principals-implementation-plan.pdf b/docs/agents-principals-implementation-plan.pdf deleted file mode 100644 index 2b977d642552a8b258606512706f27ed3c8997a9..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 264243 zcmd?S2b>f|+CT1b21G^i5Rfx8m;eQ~!z4wNEGJ2Vf(e)10alh>+yH_BGiSi8U>3!M zh&hV^(<#PNF{8qn1Ft!9{GY1lsqU(tn(CU7^X~Kiy@zj4_s%@^)KgEYr@mFa%ZUC1 zySqIyt#&y9{;yRl*Qs#oCmh+TSFcu%({uF9X?dsTh{{R%Mt0k;zPYZ+aRc^w|C;JX z=MdKoGVqI$4t&VbhZLj1o}_&7AtgSDFQ>$p)8fl%@#VDma$0;jExw!)U(WC^AJ(dO z?^gM`YQ}FX51NQGKD@c9wx$kwA6V1S*yQlvG4_X4%6HT;nI8LQL~lf8Lms-$8X3uc zj?@Bi%c%vNyGTym-88!yUDH&Xce;#dsHv-}nO0fb=+sP|R-2y+^eUTb>g$|owUu>U z**E*=fhQ0oGAZ?qwNt=L*p*@VqnjKqG^x2k^_8_U(F**6M3g;9Aet8a8_|%T?u5Oi z9*xW&)yz1qs;_GT7L?ZTmr`r&K|*VMxLFr?Yw-^CYsgnN)i*d3Yx1?#N?Z6lsV(*( zp{=aav#_7>v2v!?SLanZ;xDDv*n@=Dxc-Jc3;P+9<20cm4oI`1vWh8((orNMHO(F* zH0@R>Jq`O!ZTD@iscq_B14QKY-r~>YMpXwx4nZ|S zH@wA0!_HHqLn>#@lq5YBdfHG^Sz9xUI|8Ns0w2I>Cv`fMFSZ}b0Ns@vS}^xlZm10C z{<^0m|1q+@xhd~V$WN}EUQ^$ouvVakoNfVYkubP}a^nUaIz*>|w`GzBGA@sX)_cX9 z4DA}l5q6)3W7M?D8H$Awk8(R&zZGvMv}kN6-qTSKqZA1uUve{Amlba&94~Ap?43~Q z(1b%{YMxdNI;LtiGAlQ(^~LsH)bxIpb=9@` zhMowVZ;YM8qPry*Ze-vB9Y{fs(jJJdKu_{g%{zSb?pdm0)pNj%aKNfFY^Uv~uYsoPYMOW&kypYDZ&BfalTR`aKlBDDlu$-OC^I2#nkq}6vd+Ok6LKYM z3H3?RQQkr#8>&}~BtvWrtFPl@(~J&_~;Mww25FV6Dq5wD1(6f%iT_Jl}JDA zHqjpfIkPiiCVEAsfdXI1<3NF2LxSAtM^k3hwXi!hTMu0WPOwAk&?Kyb)k!t@?!1z= zQd6=7w5~FWjhsV2<`a4i958TTMMbKjq9UDz|J;g-j1O0M;K4H=?mhUQ8{mBku9+PC z$G#8WVc%unW4}uefdA-w;osqpoF4lPd^ZDsXTRg$=p_=bnw_G z)d3Tg(90xs5m=PF7%-L@i@n`hF20rlbux_)i2(d6x!0-6?6mcnC zE%5NZ*2Bk#FZ5oh&n1o~AFXIK1I9LtJ{t`&G8xbU_L;F?FsiJ}@O>s8@qPY1_MC}K zXiyqHGx4$SdP=u(sAR3x#*tXJgV0j2+u?wt87GY}?@rY16tK%jSah4Q%0?Kvz`H}; z5DLvklPf#?k^jA@*p#l~P)S|oy)Y;<_AMM?yS{OrrM{7Y31j01%7&OdXIJ1o4*b2w3NbwgO8jnlnmh`_s33}edHe8$*_-{StrXrau?&|*hg+W@PU1%=2&~8 zJj)u)V1qDkgU`H0rAW(5n-Tk1ULMud08w&uLq0#UzP^dMw~_U=`Jt859A1QG?haC4 zKIiHQxp^IYgZZj_1N0qg(+%t=q4VlF0`GumJ;Y<_2Lq;tcQAJFgZ-UDP-=&7o=|CqN;2PJklvodEw7I8+EvIRVNh?NCKLwLNrpJ>=BFu!Mi63jRMa2Qgk1|CJ9y0V@7y zKM0Mg_}~1H%c1hFzbo26E{7(N%b}j;a_F|Bd(@X)4o5DRB|fqk^K)7HewKcp72iWS z<}%byhT6e}Y$ul?x|CPtDA&qSCYGZ-Ek{XObWijt(aTY!&rzVyQFxE8Xm&XY@;Qq7 z(Us_lM1=x`F80Y~mGFH%n0?`AS1d(Ip#RX^3gEpG~~d(26=i z;+2MyqE|NYN<&CdFH6xaOErw}o+=$#Dt=@s`eiA)Whwea_oT8E-Le$LvSNNN?rCzL zr73JSyr+Isw8~QS%2H^`QfMmuO5;x;7#D3#9^y^^1!ls;SXmHY&y^4Y{I*|!(JlKoKyH2a5L$^K}{ zk^RH2WPdcp$%a?54@6{2DJat;OhIYZlcm{CmS#JpUrFRNJIa=PCDGBWDw}wvE>qDn z`@63~R0&s_BI(S^6FonxcF0TwKw#Gc;SvP$4Qqv$YHrq>``1C(Z6)l^gla z5Z{Sc;)iB|8Jf*yXiAi!DOGeY=&z?MO^Grz#mR`(O`59Ey<}=^_R38yL$i+LE3rzm zk_^pKGBhhmzS2<9tR+Khav7@KC126BGBn%C(5xmyHNEJb_@voQhH7vbn%!h*Hj|-Q zPKIVP8Jf*xXjTCL22=sco->pkXDD0CP_~$%Y$ik5MTVj;tgPejDcWQxYGlakXUGc! zP6WS)-q6WV;U^Q`lWLJm%>2i$809kLYBS{OGUPI&d*YW|O@<0f8FCdFT1|=W$+XEO zWd37U!B=_LG!?>B<5x1w3uT)@6<27=XyjsFtWtwV(X^KwKJ&BZ}PrB?Y ziIi%N>5{J`T8e(@lCLCMikj(?uOxbko`Cnlp(u8x8fx15N~T9qHchqHba+oHOf}Fn zMc*_<-|(IajcF?Ur73zw_he!eO{3=|UjZW`WKo(z;vasc(3=jgbSMn!RxK01p zuVg=za-|Ed4DwNu=T5i4J9LX~H6h$lv zig7FRO4)C^>??&bns}#6zLKk_>^q%!CD%>ac{=e*u9LF*bjerJS1J^wOTLnwQDGrX zvxfitmGprM7l2epKk*;FrfBw2{uT8pMYEF9uc%Kcnw3OX)SDE|3R0A{rzrc6?$JI| zltrf~3lHxpWlm8FouU*uMJZ;AQr+kt{Zn{PDPoFJz7!Q?qI=>${5$GTic-H6rFCo1*MC72cD{PNIZqQX&%Qj_qW3P>qRRZ`(SwG-ac5K#(~O1zT% zD3wW3dHz3lrSYJG7i_cPt~EueQ}HYHlTxeH|IC%-O$D+PrBEqKO;UtYPElx2QHV}a z=uJ^bOi_q~ojNEF%WFWcK0uc(nHU>aZejt|h@vY-<$%0*K;Ar{T1r4(J0LF}kk<~V zydIGEj_!#bdGny)iYgY6caN@U0s$2$O248B1{6I4Dlk}IDVhXDujm>AiZZ2NQDpMZ=)rO7f)W7!yAa5FwHw|dQ5YURGPYJe9 zUehPn=u^V$Q-bT0YxKz_`s5;ga)~~zNc!XoeR7RHxj3I(nolmtCl}+BEAYv5eX3FV zWQsnSmQSV^-J|@%ds13(lRU-W zNq>FPU!U~XC;jzFe=V<)iLvZ}o0w0k?vv{Kl)(6;`aY?+PbLuElSs%UqUY2FG6|nd z!KXR8Pp06L33x*FDY5WK`r$pPyhqW~ zBbE2a1fqNNz8;x?M%OO24A2dSr5?UolbiXii!BmE=ToPLHCC zN72M16Atgmw4;0CpG?&w6ZgV5ASKBiTl03=oJ-e&qTG)7U*W%F}!y{MVkt^^hN=El&0^#3jC@5OOu?0LzyA&N=auF^? zMVDNJOD@8tiGoY6BD^P8;Zk&T$z??M*se>i!=(h?rD*Ar%W%n6xP+r|$u+nX9bIx6 z;XSzumrT?pml54#|D${2gG|b$DC&xPG9i~t%%yUROHnX9>y;8UmrN%-6_!lGr6}l< zDbQi3aHcG^Pc;TOKLzDas_#;?bIAl;Qhk?H-KA(3-J?o{_Y~b+G6k2Sn@gtPQf(l* zr{9xFxKs;p#XW^CmrTPYQ;F`;lwDF~SM1THEH1n!y@XQ~u)mbOMfXG}+^CAKh9Veg zC&ew&O|fSYB}xj4dMTL<8-H#xq@kpk6w1T}xe|BC3ng^of&)eRNs&(6;jB^Ip~?wF z7_vutOlnM8OQ%TCow(r7DQ1vfi#y~@EIZ@ai90SasppI-)^quYV?2Xm036H9&Vm>P zM^4mNHM1jn`Oy##Y9;#&hel}6Qy}$~4o$#;f-?><*vaQ0kZwiiAP6lt9)Ca5kAZ@_sq8{v74P;WkqWbUErPZR3Q! zn%L7E1*~WjC@*PRxCf2ASF~XW$0FPiL|!S{ z;Dh}x*w{k>F-tGn6n!XiXfb2%U*~8=^BQG0X@X5!ONGkrDPm z-aJEA5TP4UJ`og=43NevL*BX&8c5%x1R+D-F+<}MLG)0q3L$#bFG}Du6h$-SwbC?B zAwG!4sQ{-!<%TqkTbdHNLOchKQz32xZKM#lLC?v~(iBZ-9o}pO&KNl`2*Nr}0S9c%;Z)Qle5#6Qz`>yo$L#g^5^Y z6b`0AI|wLZgyl~fr?8w!{R^mn0XgP?^e~_-A)xUJ;$N}lCC?`&j2>lOUN~Vxz454@9<}RH7UPLT zpV|o%bjl{eM4QIPqv#)HspO_1l6MvK#6Ng|h#2~;v_i+vpM>GwGBlY2IKzItz*(W$WX#yNw%FbZw4`*9Vt*nK= z_3q8jT7n=-Q)Ehah2KN>Mcz#~13m@{yS)3!><%V=dGFb0XqQG36;&n5#v++$-}>sA zNEZiz=t6!el+ty4508|Z1gH6<>BuKD@$V%z)DKRcr*^oKP&G){7kDDcZUsa{efpXe4r+$dky$Y~N^_{7Z>VM8YPIg#OSq znhuXUiKIjQR(N-fmcu=MGG(29Q;^0SQWdtHIZDFj`34Carsx#+VbU(@E^uCA%BVC} zb}9*O_@iXjNCB}_L1P#z8cJu~@;YKN(a08&F`^5+OL~WStxVi5 z0t;JINatj~R7lw_G1kfg=cG7QRt0RA6HBt-E>6fe?TAXP?4Sx+w=((27?DxYh#cMv zNn2&cp*5na3CeOJ#{#~vfZ2?vsG{bM@7sPahNDe{;3X8htW;D^Z}4XQM?i(T>q;VhaS31I(lU`t%MdP=)!~h=g>H&J zOYVlS6|7{Fp1ESt64%>ewoIiNqsY zQTt%b?r>O=Y3oHVmXdJ+Srt8hmP}Z3Q>Y0x$2N#i@+h(eiI4_COZBT-?r> zeH&lTOe7wO9vuz!A#JuO?wpco7ZE)lV~`Oq8^=Un>hy#Bq+r{euW*1;GTUPA8>4M@ z1l(^Y)7FbbMOvvM;pnSm!jdDv5F$hZuZ*FkZ&>VeY4tm8_6hu3iET=Xi4ZZ-q}eBM zZptTab_!gQiNqtVS7l|kC?2AdX%`t{OxFoq7aQV|!z^YCLf+=s$`{U)*$%Z{p`9{2 z0-n7j)7Fbb#UI8*!kk1hVOi_BJ0X#z%#MJ!oJ>6ui9*}#2zX+bOq($}g1w2JrE~1Y zat_B@SOr?1AKz6hvXl?36gFE|GZTH&Dc)4`!dlhb5V|Ui2Zzm-~;H=-IPm!jh4o zhJ?e4kr>p6FvF(8F@7bnHCg_gWg#PTD#SONl~3I40l5qliA#ekPc8zp2jsSwOuNY7 za_fiX&%y;ce3ndD8eDV(VsORwKC>s}-%4yn8dPMJ8VSs0iQ=F z(-v}%{47)jB7{93Xw|4Vjx(9CEO)syN2vW6O)EoXuhnBQ0g$r z9|Uc4jmZwNR!$x0zxU2h82!NPtu29XI9Aajr)Buo;q+XB{J4a)cQwd4e3m@WLf;kG znM*J~+G(QdrX5sHj^D_h&>@)Z3W%!N#q)0^_CRtq2=){0m~(U9!$jhdt5N#kn_WB~ zmSoyRL{GC_gwe%BUvl1$8bTt0xsKR#1Y34ou}RnL@cFkAJ0p*R91MN4!{?~d@`;;W z9S1}w5|@TpcFVrm=4s!2J512XA0o8aJ__o zE3qMIToJ-3+A&*~CBiO@GD;sjvvt`)>&l5T{ME?OYl#2DFVTj`;+aj)66wc=L|&2o zJ&}br$GEh+%%u%wt^na7>MW+Rc3w%iJH*YC9>w-j#t2(wL2{%~KyjYg_i!{}q5(wi zL8Wa@Ho3TxY3oHo_1IASf4b~*7I@J=555B90U|F3}8s8>{Ea_}E~P6G%D zUUtBrr{R0)0q`GvFZ?_FheZgG$4zn$w5hJ|QL(LwcKhZy0KlL2H;&^9Nmyyax!g}z3m zZT4e)BVi)#$Wn{e2r6gRH`8vu5w(?7dA1kc>DjlozG@2eGOF*>K^cnt@L3`kqNIeI zA4zt`d>rh}s;JCpm64@t^v7qTQoh~Djg$SYMMdp zkqnLD2pdiuoG=$vNgLw?3M-j(5rJ^O?$8M>F@e}wdKBqs+(;m>U5x;b0es?>f5(_q zE%trc*athO|qmKkWdU{a3oz-B^;EUd>U?-44@ zBix^dXgAdt@Mp_WcHT5gFXjD#YC9RP1?l(-#wP!vu*RZ|GtN@t^~La{^+ zNbTlRn4@@TBoKCZtAO>tD0)+>%f!M)Ha50titRyEpduWWb4U| z(H;yB_h7_E)XohEOU2WPXgEDWHqxP@F~W8(K(bV1FH{wCFenk)NPQ29n|-4twy}s% zxTAFFWL8xumgq!DC<;IkDsg&2U&TMSTPnA4RRb^dW zeUsBXt-7))@6=R7F|DR)rf@#F1go*L_Vws{=vXxmPuY)IgPn^IW`-xF3ExBK5Yj=( zDq%|?q-NG$bN z-cYPD56(@ZmK7tOHRmd&f+uNAi1VyDSFr<67UkR=^X<@+MTwh@$r3>sc@0I!%n39% zo5b-9GdR^W5cCCl$!EAR)3EaZl4+m?(_jJ$LL7+>hbDP+Xkf;iN!tM|5>v`-`K<1gH@oWVm7x zJv#ZH*OB45X8vIZe-y=2#A^yQG8?lU_)$w4n1Wve@DX|for3-WE==CFs1v(Y)j-=E zxE)Y47zZ#bP<~KR(4@3E39&;%YMTSs0|%i0?C<-zt=Tql}BislOeGh%9eOZref=9u7)r($3xhn>bosLmLgwL(TQ~a zAH4u1M%ZrkQ3?twsuJcR1P=Ah&e{%ODLHH97(f}7s~aq<`A1YzQTt$?Ay{G>i-?{) zod$)7?`uhpFtX_7bv7s?iC4e)=5WK$I81Cv7QI~XlX1m5(OVh3BzmO}=DgVsl&Fo9 zBcgf@&6Av(MyTD?G}Tnr*38ORi)oX56t1Rwmhi>(=LUQ_tjQy2iH9d~IUa&6RqM5c zKO%Q@E;YNc1l;PN3<$~scBs~Y@WB{cc1}f@4EBui#SzAo1Za?x_$s7CO_qz0D!LK! zRy+*QNm#VB15rvQgWQIFFxKBKQIuh7srJE~w%Q%K7|tGIqUX~TkP*+zVq;qJnuLm+ zq(k9lJg?5&th|XVch0JiyV(y~LMS7-%WhCUs#wFgC5lq7@^tzly&+7MyB%s$R(S{%jBk*&s%hDMjddy+_55;31)5w8i5152b~sM@lKV=v2#iNh*+S5%v) zQ-bZZ0~lAf?r9PYekD;fvTDo48Z%-7yO&5@lDpCYW8#U4Kr-zja>pp@Yk-0HESYeW zcv4P6a;Fy|Dr1Sy5@|+KCdR~28XY&jfiaPIC}ovh8Twr4xbdYOiL|vj=BRvKHCr2O zs&|^}rb11=fpSZ^X5Wy9KS1W$)K&D==!ty}EKKBQJ;J4YI_u9zaARy(t>Uq8ugJWB zA^7yRG9TZf7b9G%?Gt-Kh{lM6Ehh_6MEV$QW5hvmCDSIejEe-bW_stSDiS+*pyUo| zw#=;rC zPBxM6$myUNkxI};n~ zrwV6N0V=>ljxj{q9Umx*QO>#(Owr2FBig^a%1o~mR&C1;AxN1Qz2Hf>8ptu&Daq^< zWf#IJJBSs|d6fi;!jMiN_boAltcCScJoFK=1MdvtqYYPz|9AjnOc*6i&#^P1Q>T}Q z(Tkj8giGCV;3In^X$54U5o&2gz8Gz}I;o*;m+cFuu3#m7I>?mXbsZClRmv|rbp?`@ zlc5nBw%MZ+e3VLCt|Vw^mlpa}kq~^6UJo76E#xr4^zoQG zI1Jxtn06q-$Qw}ygo4UxYN5nRg$>4t%P~Vk+{$Q+axUjI8sc__8|F*sWk@RW`P{go zCvQPO&y=jAdefp8_0sF;RXwtV3`)}&bdrK?vq!MQ3`XH!rENA_I~<@k{_DS-XEtM7c2Xg6iNXAD$ov%LfP3k8;UXlpzeqvA7Mt0PtZ6Y z5sIqHSV9MdF5(LyNnd;d+W80pixIZlQYh&nvL|#A-l>ClJ11-QF?KLQY19Ul(^aTaJWz+acPJZS($^Y-3f;A3?U=304yY%Ip7~IbcPpThLY=Y0>CyZqnqt9v8^Bh4 zf;nc)NkXY`Js%;HBpms?ZH-0Q!S%EuvESsH+G?2nR?et{1>VU}g==bbs(9V4di;bC z9}6fQACaTR4ABnSCt06z-eB5neU{ihCbBZZqGq|)$FYu?lU3vo?NmXw`Ghd^5&st> zY!~$kXkNQr~LLZM2jCI>CQxGW#q7Z3Zee_wNpoQyt_$)c7 zN_wxHa;QM+3Vl0oIhlGmIa6sT=z3|cKoyBNwhxQGFlE62-0HULm%;M zfIu{An;m2?GA^I0)!=O*5T?wQYlpxqO57Y;?T$8#QUg_1=83Ew@J<_9NAZ~2kgtMM zeH!uHg=pH@RScCmD@>D|MRaKGD zTbO-3tuZF+l3fUnoDhlG^H~D>La(aQHaj9qSY8p4uv#3bs|Hwy&q~q~1>hB!T^z@- zB~uSYqS7``|Lp*EVa^%T5?VFIGyhaAu>{jehJ*Glw8ZR~E#Y*LNaX3Vf-o

vl&G z7841d9D=0cCxJrl}$+8AV4? zLbz^1=t0TZ$FmfTiAbqvI$~KgX7=TFXgVnoQJ{JXRJ?F-E#Y+G;3`l(1u3qfU0V9B zx&S=h4WzG$o%~s{-;(rQf!6tiw1(vy9}$7(!DT&5Y)zIwXITuc*#+BS?TQjNJ2Sgu z{h};Dk&*(KE*M-)vJ$1DcV+>dVi#!MPmIr!2}grV`CjPXF`U90Z6HqHW7QS?q6mH^ zv9(D4#4Wa6ySk;Cdqu2M!3k2FK6^2skqit{r+#8eA2xW%QiLN@7=v5m)fl zh1O*Ukc$Rak&=Q>*3h=HT-~~WeBQE7YnW-2Ad%l)$P74WZe&cwSdSCApza0u{$KLd z%BgP}!CB?dTEUt91b>HZBFoxxOthzMT4h6HzQF-taZ_~oka125uZFM@)qWlmkG%FU z+UYV{X~yiS{&3`5fKUh6G|YV-Tp3ViBI>jlCgVN4nMbUmgcb`xn?AY7$s9ASGrVcv{$YWz|n z|5EHy&YLqQopy+~D0C^fF|(hvJSDG)*m)TS1jLhnRqXs(qC6z{A$b%MyV=Wf6j(Cx zP!G{r5oxn$wS>DB5eOM3;^|@nu?ppiKxFGN*8KuIs9U5V=*3AOW>3W(f3mFg0zv2! zXo$uz`Lkrgl3}Xn*C|tF6qN)H4=h1wk&eh#4<{~5X|RU46%b9Ew}9sl2!sY_JDXdv z0;7&1-OA3hWh0>K>t42rc z3_euA7;)JHYKYrK;vz!f)(BGPDj$O-;IhbgaSK%0z%%wS?@-hw1clnc?9wswj?KT8VT81d?@}{H_YzJ z4!IVl`KqBJW(>r{nO1eu4z;F;UtS;rT((){B^oL^uTbLhx~)dyzsFGN^tS+gZ%im9 z8>)gZgBQk|g;JuS#)zA}mmRQ78$MmLf3ZWA=_#9?iyfv+JG1Z8+%&nqp=MSkKg)Q! z+cnf3voEj%kVU?rXo$9ju?2+d3a}IfCxh`BCS18z zgN=Xi;tMD>yHI&nPZ2LPFwR^Pc`96J`d4ZvB72?OzcfK400ACoh0d`&9T=GrKW8KbUkHG0qsC)TH-KED{@$+ z!vV`?4J*kp#b61r@K|_irMbCMyicGu2k;Zv#gGBmkFg0mY@1e+Q2c|cB7;J}rWwHF zoArTfZh2Y<3-Sr23~nCf5r;(4oBJUX>b}vk?%*3s- zkz&LXY{SahBifOe%8k(o7=xp9$;}h~IN-<8ja1#Z-4S&{Nf7);GBrUGQ0=2t*MOJ0W)8rU;>!(j2FU&yop8)p}x& z{I+t$i$<7M)0A)lEk;}R-5T0f`9d*~;Or!&!S&&@(nNyU47e_ES1APZsTBnc`z59c z#Hx_H(QZ>5TA17gQIICp6SU<>p<&$0Clrx8V&Pz0oW{&8v3#;BGFC!0r=tLsR|yre z&%c!@8p)+8h&81+qakipPmuIo?tt>D91U&uD3Nhpd{NH0(Ga#WQb{7*c9^o18A($J zMS|BdV9g-KVPzvhE66GlEeH2lGf3g#fJ(B7&#c9PP)|~D<+VW?d05R3grF75Qr*dc z$**1M&lryEPTSKmqa0eG>890OLQ6R|5>^gDO9-mW;{78=DcIrv^i8mrjHsodLRr`W z|FraBEf3ZTVSQ1&VFZgd@{3N)U9)l-qUq4|cwTHK%(|IqAlj^PFeHf$-J>&_rsf+P zD<=W`KrFEY@-$T=&OhFx@CdBxr5zYhNEQK+?DY4hU!zU6U8bG{mjEf)orC;!0^xL%bwUp$H1z-9vyu<)!h3ZIpUQUP?g> z`tr11L!4DXl5NUovX*DDfI}WtH9241T$}GXvALW-Hf-1~O*?XpGwQLu`OQA+D zVF@UVcPJM5O8_gS2&I)G=yM9so>NWN5{XEVlN2H)Nx@Ti)EnrF#wS?f5TzVbIT{aa zH6>2-RVdsXy^lLHy?yf!wbR>m*E zO-Mt;uoCDoUJEbmfgMCq7Cq;d+EI2&X^2~S07>++7gJutp`mSM=fy-%xiaE^RMA@= zjVEb68fhqcZb@uaB86}|Seh_;6+3XDGz1E2N!jzI5fkGk(FqqL?#k^YwGM z@)1@&1%H-kTS5{Dz9cUJl~$uqu){HGNrR9bSkZ;I1HgRIF~=`ERHL>T4>AY#!gcmu zc=nEX55gIH9XfkUIlQ!7>K$JUlWX)S#~!#)!-702<;} z#fD-+As>lI%$T&T@(e{N+_Q#4f!In&PfvN}f~F5vS%(x^k>9W-vv1&N^4hGo^2Mp$81haGjaFQaZ zrJFOm13Rdqq~XYW*s8ods2NNve^8uyf^}L&{3w#B0#g>k3R$;99u^sB!pEi%533He z9nw%9Xw*j`a-f|e+^q`;w-jM-U1cFl(;usPf;1G!d)TTusM|pfi&K{?ts36#d_uT_ z9TS*YM=g)t6XiwKq6uu6La1SEpsnJ#GSIxV5{ld$Yb_7mi-e*QH=7oBiRHREf(NEF zut0p4Ojru!3N+9K7ndT~p{u;wpipb%bEy;I zIK)y0+mcEe0SHbS7yIGLODAyUb1ArTCgF-{M4BL7X*%L7O^~28!i}6XLA}z1%t|9N z3!W1_JWCGlDJ?-v8;mmV30!H~qCrNc><$zZli~;#$u-IBe@cTms-z*scOkcSm3KjC z#?~r*&~p#dMi*wXF3faYm`1xWNpfKV#@-*~rp=i%7jU9!>zjp|nG3}&7b;IK)Y@FA zySZSXE^IS()5hrlj&GUGLZQlqLX``JDi;b>E)=RT207o3^Pc4{bFm^IB>LmFg}J{`!|f27 zVPPgl+Z+Py(3!iriL}{EahFl9K=5Eb2%&uXs0zdmAE^jLu8}}IhHRdBF>~(Dzm&+o zq~^#g$;Rvtc=r-%ONuCWz-&Mq&X!181Zu8#!ZJZH=nIUsNBmM^3$kFiTcK9W7RoV% ziL~|g7xuO!QGhYqsvWdZD)hdhe_ z45hdzO6odj6@&GnlhjH>xn$5|sJrEgj?Mr~;~9Xk!veKGjZi+gr&<99>QuUBO3CPB%|w+LsK&sAyLj;k*nt_dzNTeVG&=yM zUciM_8c1(IGWNwLGi#-kEn<=@tdKyJgcoSwYAimPSt@0&p|TV!FhG$(TjGG{P~gFH zx1hpOP;r50VIj!Os#QLt^B_=>NaN9z%GOk!Zd*CP$PK6QoKXr<^6ImWxRndk#;E8B z`G)%Dro1z?61FMTRMyfFa3Xri8$dM-*a1zW-u?a{gJ(&q9RoA~# z9AMNrm^jNDP<4c@T%LAA*+@QK6Y`TQr`Oash=G!q<8|#>l~vd4K+*ghk+QJ?E6Hsk z#+EEXUZ&Uevn21MQkIf%9p$o}1t1R6s2gxRozWrjZzYEEXq3{*k(OC$K~dKMD`%ne zcKm=nBja`u&xrBJRLaem5Z0)w6@Qi}J`t!DL?}pucK)^4# ziZl!-FA*YeG^Q(7jzXTgD=<+|;54C(L62CKoz5doo2M3b@KJf_xEmEcGN zl8byGfle+~jzCX2z8+{CT{{S=wjn61LH!FR+F~iv+z>0m+S))`krf~_bZdd;=1NHs zV$EsJvV`?Nb8}Wx4Xqd%iUBGD437hq6Fk}yHUOBLvzlpWo1-mh4(xt|`8`bI%?*__ ziBLw(>=HQ`)m9xMtA|M+WCDa}H3w=t>}lj$(uj{G`%DmvmElLUqYjjrif0I5mmz>% zngDib0@$SqV3$UK9VFS@3B#3&oCLQ_BRmqGUDe zY)Q^YZ5I4`3~*;|sw8(5YbwO~nVYamcl26muJ)RnwMvAFG;3R()!3kwBhxoqh>Mep z#^&CZl-?-g2#$`wU;)~KZp(Y)3JnpxWqF1O4iOv_xJiMHgCaDb^Gw}BCtY515IP~l zEP|86*DaWvE+yZI!V$z%8Gx@7p_Nl9w^X4+A7u!fo*^)L20`aa>W$A5MJ;Bt8N{no zY|jw7J%gzB7-1_HBh6-sJwe-tcRBEK0vx7i9x*FN6Xg)Zt!PJHw=J-1>|wHa1kfYa z2RX6Q(6;g!MMTe0oq)x_4F@sNTX_XV^o$izbCZGk8b!9llZF#~(mo+>c^X?N60Why z;vUSXjeyQGn~K=R6fJ{$GgOiAXUT-4tw>}%#5PpIGGgn8JfqO`z{<=eZA5NGI}$cX zCwi+4Ard`0072XAb4$Z|%KDep_SI!Q=OP)Ft3(>uXvPjUDwIfpK?ZW&L(@Mix1m)K zvsrVPYNFn+SHbZavub^I2+~k++>w*@1?K2)hac4n&K&tUrZiD-<>qUerKJqvr!ojl zgEv_5=msyd;&vJnsFgzra|Z;-l0-9v63rkwJx18di^#bHvKP|EdI-$EGXjLlV)Z13 zb177NJ4|Y%oy4+G4Y@R?A#PQmC`w$eTxp0~%@d+ftVk&^J6k&}YO%rPyn^((5vhaA zpC#+Glm(#yW?-cN65rV1mTbA`XvE+mToc}%1yBNlj~fQJRM;u8EC$!?)a_8B^5AlL z2;ye9z-5?h1mToUk&;o zxTRVyvMl6p_M99(oG2q{a24Y60l6CM;xbGm9$Bs;CHd?-9c?SijTry^IVc{`shT-| zme@m)N+Ir(21Z$XD-3CN_()wUCBLIiBe;?vkCD`1_5&$&2IgMi>)=2{_{f4Td^CFx z{;fm_MdOq~n5GbtIW6Seluz93J~+g$1mcmb%AMtC0FN2|ERhSLVG1l!T6cx>){NN! zKb+LIm8hej6i|rCpq9ZRlDQ40T6xH~|6gX#|Wv&1fjgT;6j z1+#GF8X)vWG1>tpb&i3(0U0L*=7h@*3@Htiiib3FZv?E+vas`Z$Vr`8VRbS?ph;zSg{F-X-AG0OO)5JqG{mjSf9rMhG?R>y;FTGKvQn&6 zb~mCJasgOK~U} zm9p#by%G)}IF;QPnpUiwi!f`Y6;%)=p(Isy#j3s(1Pd9pJ8x6VD5x{&KLCM&efh*b zO_<^k*Rr%ZV|L=DpR6g;xpGq#MDNB1sgz72gScOE1Ip5|W_Z}6@&qSOCNzYt+<zBz#f~B4PGCGNvU~Hm-<7o)0M7lr$o-vTI386xuRYL_^!o zthFM^5=uIYC}~9`_$*o0(jpStJB?(@n9yO%$?O8X)UX zT2pe#sX*-bt;9AWSB*@Gl1xf2K5B@U5}B2V19jkR_?mHiZ>cHb;CD;v2KOg4#(}(c zr|E`Ow2ovH^`y)+CriioW(mWUMbMR#C4^*_&~#ZkzBfyV$t(gKWBPC95=e`(h+!0R zNzi>Occ7BAzJ*qww2B^n#Dg9z#y^4SWcIH1tnid6*HwgQU@?8Qg$ z;vsnN5WIFs*%)PR)T#`iZIqq#3a6>S+l^>rl(`|Rl7O}$+K>cgU3o{8xwTR}i>lPp z`?(xrNXnKEF*IftS&r5#!C9>be^Qt#WbKZZ6a%Ip42{W6mLC?G*gYZ~^$q43zWjb! zTt}1ZtyvB%Or*R%9Wj9nvWPHnvIt~=E9PKXj=UHN_dL^qD~}#<<>3XcxEPov2ve4h zq{|}K2nrfJ9e^v*!xeV8Cwd%+2K4Fl#wynjMoRH^+C;j&`2M?;e(WXkeCnB z^Ffk6NYw`k`yg!}B=0K=>Kc~Vqw;#~K~CvIcIrc>>O)@VLw4rFI))EPjt?={hsfZ5ozJ@`H+$JeY)dFo*EucQt8@t|TlZny5$*J(wnXFq!pW zdg;OZ$b%W1C%^r4@!g{)CfIgvyZ0tC1qMFO8{Z=XOM2jjJa9uExFLA$AN&V{1a1i4 zI0^qj^WcVxy!+D3663NwFX@7@!n>m2KS&9|cevm?*bBp4@E!27F!&F)%7fctugrs2 z%)x)aCU{llRe2hp!lQASEP*A#YhK_#XdWyH-dh9zLGxhn@ERUq71)F_z6-v`1>fU> z?{UHR!0WHfQeufhnJj@NN$>429lZ#D12(lMMQU`Ze9xP3rXi&DAQ>11>4h5b8CZjS zs*Lx7K=*+jI2U@AiCNO+dv@ss7vlIvb5rGecF^{MJ*NeW%CNW$S_U@DAP@jffd4=n z!DcNFYy?m{$Xq#NLZb%d#{#}G2j{TizEvk7;Sve36-#1%PAanJHW1fMXI5uuBv8QWvvsw)x5C@c1?AqbX+-b zP1ABIo>1+Q2W!DNQG$F(SvJx%W98kVxnmY_$D$ABG~5!#%ZQcnVo=TIn=Arl)k+_K zmKamS+EW(s+tk4pp>-+4Q=))aTs*Q51}%Pz*pE2YL>j!gi;a?2^cgR z3OLGQgQf#kroJARZnGS zzJ|NN#8%`H;;d35%E5A)5wZ&Yk@_kfP!5zUq+LENm#qd7E?3$2t7*fk#x4m*0p=>} zKpNs!Q9U9a1*>w`f%rLpQ10VMH+JPr!Ig2s?$NPk5!(nt;%*SGR8B;=DBNR|%o1ug zizruk&grR9q$K2TRzd#q*H9VV3vW14c|{VY5|6+2%G{8ZtI;+jt>@q=cjkty@(OK3 z>M{;gM9mGAG>=h4Cl-#(^h=t&DDMk_J(v^--gB|&>tHHO+N zUy3r*xAHXFUcjqqoD3brXKusF;b_~?f$$4G8uusVnPb9~Ug{V3QOY_FcV}fb3%YE0 zb*H)6@_mBlm6w`|fTpLVc!wxnuw6s0JYmyNwsHsSGnYc_rIRH@aTXE8YVhFC5=S=7 ztq^=H@*jdIXAzT5ZQNy>DMMs(vBQDQ-HN>rqnjC)2UC4{_8p>ADeSPMxh~tcnjg=LPfMz`Z4{Dyq zTBy<7IeaxyM|5(ARfsIYbJaru3nk9KmDolk@hoRbES)N=q#EL-WCn_&09#pvs=}*( z%|fy=f4KvERzh~9yxd090W0&@)*Jga*VHz3uc?y`Y0B3%)ilj?nr2Q5tB#;NSvm|? z+03M2#VRe3`^k701EmzXHN>slg-+A5=J?OizlrL#9^-!wAwaN9!Fvg(WRercqli^Q z=g*S4BQ$Roq0B;a2>6*r9Bz!Tl?RbS9qiu+6kLHLza;vj4VXO9C#BxrS{KhKODkWt*T%qD{$C$`?vB70pHxj%n_d|-)!)M8a zqm-6>v>051lS6edM~HG|Y~KWv5R*(QDjAln`0Lrt3=|FlTg6X%F^gGP(pV z<3!!pc1rAmDe`V0O?Rw3gto1MwG-HeW9(N+cv4k#YA`Md1*#I_s|tm${E01|07Mqh zv4Gh%_owU%fb!~}!a?C%7HkzDvUMm~!`zDYJ+K`;X6{S$DLLiHN^}0s_h2M;V14>4 z+9i^sT_QQ!C6c3EB01V6lEYmhF?A}*iQxVckqVhbT#~O)2%GbB-m66U3aeEJCzxnS z?m!`IPWnqi8s-$j=0u?+Orc8HSP0J99r{T!R*cYWcje;;8Rj~NM_V%7b)Aa?2wAWcoAJm?nXl4n+0UG z)KH{}5cF?uq9jkD_9EJ2U~Iw;f~fCB9RIH|5jzlK5%PSSftmYKvV&YqKzuiYnS3d8 zjTm{{G+`!hl^y7XgL^Z~ejr$G2%i;G!w zR<^x9R~^J!6|RoKJHVKTp-x6nw;Vy+atKHQXUJSbY%a*3CB`FBxXCF)o+IF44v~VY zr0pP$a@7^t6CO1qB}5u^vO8upszbzxOG%NAxRr}2CKRexY5;^eI{=|96ctPc$gLa! z6kWOEs_TQ5qljWL@+Q_^T)7IWqi*FVBJr!;kWb^%5w~&`#l%k$UUM91jz3FQcBz<% zCM0xHF0ktOw(|f26hihH5|<{mI^tGwo!pCLr?Jcm&PRg=q3$^ZJ`HQ0Iw22RR3$PV1PLdS`VKkb1lo6n_oz~@HmT{ z9Yjx-GtUGH@q z>4@6Os{B;IEIK{=*49@|0SWf*&4-4S7%B*q1|N7<2TLggNP`bl>C+(Na{O5$r^25o zpe4&A7);tjyp`ELEQ-6Zama;b*xzxKL;`# zj!>AAnd?bFs%$@Lo=E`ZKi*Fg9vM`d;#&pb+~5h%m6_HH7fPUIc!DkzI4&#{(!&yl zMxF(vjWY~txlG!y1SFwS94Pcb>CrsvfCfwPcr=7U&x!z4%+7;LyUZTKQYtKZ!a^Xd zG0N}I)wu{hyk)inYw)lf4{O-`eAtBkl;VnF0%XS=vUB|4R6*F;i-*J}v}Wa=$nnKm zi+LDs8f6M7+&SvV`&8lBAP8Y_V3R$31rEYX=y2KINIq$Rjxmvrrv%OaFL)!JJK<F@)|n?bLPlwX-A<9;-q}cflhjM@JHKeHXMiYp83R2ByG8FWzrtMGs8y;HrS?FWM-v(!`LIAOc`N5plsvS?!&=O; z?O=_gni}fsCNV}wgyIow%hNQST3ID0+QIMFkgsg2Z*V5oQsCDF7E}$UI*F8)Jt!=*gGdwHg5CSeb?U9mdv%PnXNg0pbvOdfRaIZtls~#j zxE%RSu3Brd@&vlRK!BD{Pt+2(@&oJDR1tO>)4P%|k647}>ZE7o1JB#&ffH*eHbh`AZuAjgUzk%-# zZ>UDZV;9juHZ_7jjB1`nPR?PiL(LQ(GJaUs8)~Mp>J9WR%vt&$#J|_`AmXRc`UNy#oTO%|IPIHZ%zTQm|_R7JJ}d1_}Vs82rny z3ws{}Y}jS{BG|tatV=@a75=3FLJfsd_?Ka?w}2bq9;PpB#tR#6*V{&}YQ?8l&P`O2xSj=ucYbvtg@dV>xAx_ILUTY=fobNIyQwIVf& zXk*k@l})JMu*#`m)CLRYU#zX1)Cjx*<*fK&-@f%nGu`aY-tEH9waQdrupZo0SzA-p zr*2Yh9#yyxv+XAMPQZSPrrrd<3)1YqU*)v@@->qtH-UDf*n?4EX$P{`XY|0ogZ1Sh zcvnRDO4gHsHMMzAx(bfcjQayMY6#K7YnK>(lt#aymDAYWK|`5e;ohIbxJ#!Iqq=jO z?(X*hUz2aFZ*Hi{cjv#@WtaZ-Rm}`RGAy63&R2`SHtq>K+v=+tfm3i7gX?OVTJ5sS z;JT{X=IXroj_xac!Hd*IUtn!G{hAw_>Zii*N7ppf=J#~E-~)&A?&IHzOD=_xznyMzcVyb@*5XfWG2%%4ZuDP>^ak-|1!vnOZ`KdF7_{z z#ouB70_4o4{-ps+f&F)}k7?w<#Xbhq3ii=M&Ryyw{SNz>rdFtr#4GkOK-sv|$3Dmp z^^sb@K4#MVZy+$E#VxC>(ka8;4Pl;@|jGZ~Q?LqgfX?M&4m#i4k?!-k4&)9k3>JD>X?l7;zyghc^ zwcC;|J#K^;dq=-aFs0wEul>657U|&EZZEay?p)n+*E+Y;z}+r$PyhP)?6!k0$Xqhx z*%?>N`*Fd2)w_tbg z6BkZBW7ymao;$vK&Foh{nKNdOXL>LA@rE~cTbp~ef9qRbe{bn8n_c{krO*H^2Spg=4x_H`X@&`_B&_j--cfH_y=dLo|bF;_OLJhdfk$dEvMac|6jj2cJ$cYuHNvQ!h>t=kh!B+#<{{C@rWJI5b*TbIFij@y6k(l3KU zgQn@nFKhZ`c;ms}k7;%Dv^%>#bL=HY+^|{iJI1tr_{YkQTkn0rea9Si_Nzxccb#|3 z{nwp(;gZqIUp@D|D|+np#^5cR?%$`w&U-cvTk`$LG3P)3TU+ce5d!ti_iY-6C zX;t4TJuVu3+rr6X77WW@_-^X(yUyG9vK^n#c7C(}vE!eA_=PJU_~C);zx!e7UazeF zOP7}~Y1MW2hq8O-$IhDZR_E=eZ~EC@d+q&U^`ucN26kN9dF=edI^1;J`&U2tQs&jQ zxBYq4J*%g*_^8zp@4Rs0MyoQ@2i0GH*VIdQ-TQ!LZO1hXKD~4B<5k!G`|bG|*N-uuKUy_dA#Z=dPUZ1Yh6gAUkt@DoQ}a{9l{`2Oj~cKq$gIU6s{AMo|A zd%iik=af!|EShrJG50)l&V7?t`7d7k@yeT4zR_vHF5557ef-P?J3YLh-(5HU>B+&J z-rZ#O+H-&1d&6bjT7LcO4#QU8+rIa~J5-)@_0@|;j_J4T+f3cEvnoIR^v2c?bWcB2 zyZa^g?UL?&@qu4AZ}HZm_vY`lVdw6xTc0?l{`Re#=N^7g>dn{MpS|lIL*}m?*=Nz+ zy)SC_<><>dU)$l~A2(_`dEK2At#@5Edhuqz{POjFha7icCim(AZ+>*$?Q0*{dhXuu z^%(uHuE8bMr!9Kus`Rp8has1}e&0{2;P^G4Ej<6MvwwJCQOC`y@{2Eg;ikrpS1g-w z^3s1fpH5lcx5L_dE}YZ6_tN_Os*w|3TQGn3xohrzb^C>z-?sUnJ}t%_dF1qV_wRPY z?Qc#wviJC*XO5V7>US$X@3C}`_jlcJ!ph4|_~6VtHhX>V(H-_aZNcoWXD#3G!BHp7 zJ@$xiKI=N?pNH-@_?z)vkM4K#^O?3|Tb(-T)pirN>D2d6XSAMv(mBU$eDr%Kymaf9 z*RTHQ%Zso0Ve*1St7?C!`1#H8#~yU>nV-Ef<)z|myK%{llWV8U z9e>dN7fm^Hi(!}Dc39($x1HK;?(pWiT_0bT|N85uS+n|#^v)Y`{(+sYyL9zw6E15w z^@?#%x;6ceeW+J%!k{%9b-ie6w+VNo`nKuV?y=!B-skd=@zR`@?qpo_i$Gcyi zxqXW>S5D}3-hTYl3+_31_!0B|y=LwYOI95+|GiD;_k47-ojZ5h zu*;*jeg4JgC#*hZr+eG>zu=AGw|@1>&%00c=Xc+*%g?EHGd|7yx@5t&!SuB?|JrAh z=U0xn|Epv|kC;vXBoa#X*|J{)n=ps|ftuN*gI#WAA} zKj71EoAx~TlzY1MKJCSgk2r0o_xEbqzrztNPjeY>(8gksD zuMeH{(yP-tuV{17B?IyY-rIh~x&3bF*s)jB>CaxeW?|czH(h(!xgYFs^(OPz-8cBR zM`v7o>Gn-`KDs!ysOuNqE`2m}*EeVFcUgf=Cfcz zi|5-u@$?N{?>X<2*Sg=f)kzOGANSSXf7^2U=3`o)ztz!oZ=Q5;#|Mw!cgCY*Cv^Yr zrcExqs^#tbAM*HHFI6pU*{<8MGe>6sb@GxWCp_4+`RXIuO>g_}&s%r8Z1$<=-u}>_ zYpTaOKP_7@q~~?FUt95~rB9tX`Kz~%JO7Jb6Yt2@+&TBAm)upR~#8C%wF|rpwXC{rt@K`Mw(ab6 zxaaCz#U>B7T66fL!^gXa4B7R9J@2buve)4Y=8oCqoS(n$@Ir@;&$(h^uOGMWbo*2H zuln_)$5;0`^1+qW-uR7%jp#S!n8A0}>~X*wUAMjSsyEwjQ*-XS=d^q9ttEH2|Hsz% z?y=cF7ri!XZueEMI5iJ0Kl&fN()&I2(Bri$AO3nlr@gPNKI+fMZBx1H+{3ROx$z?> z{B+aT4{o+~@MGu42lhH`V!!1d?6l>Czf3sj&gXu;Wz6$^+dOyRc2)a)x@uPC#XJ6b zVT(tOzx&6}2Hn-Kd(W=U_~WVuKYQ8ETR*(P@JA0l`h)%Fx4Phj&t^9F+5WI=zMg-^ zEuU2lIPTPbeV5${oj6RLa^;FFMic`^~3M%yJpW{@BeMjuD>kl`|QT8v-xvZ zZ}!usEh@L)aKb!i$Lm+Uf5OunEj@D3V;AqWb@%b@|2<;d-GBXM_RxdB*y@P2TRySF z^sa+0y7A0cHa%cuX2*$FPFcL?*ska7aqX)8kNk1wGxt2T=*n-NyLfHg1INCXe!X48 zmlt2&Vbi6%cb?UARhyYVpOhUutWEd!U7mmI$Rj&!^KJWen~mA=+um&-JFxNWwNv_5 zA9VT2e|@;cwy&Q3&2L>=KD_aXZ_L@?xcl7ayR4hLY{KIkzVX5M%AX(lb&qpeZ1l#Q z70(6xcRqdE6a9L%edytaF|&g!UOoJ>y@sq9|3!~m7Tz*w+Wo;EJ6*N;IUP^i^yC#Q z&zShY%EM+~efr{?U;FCj1#Ncf^3V1i&*<{nBlcMQ z&^BkC+2*|)Zavwy4m@$?%yPQG@7s&iid>9n)A z-s$|^yPVW&%h#T~V5d*^IR46W)_%8U@6X$v^0)DQ`z%;=$G}U6ZvWoou4_+N_QuK& z7GH8}|1XX?dF2!B7u>Sdx{ZF^>aL&rZ2!dS*HU{gS=D*c)^n$SbKOnnebZvzHK(;3 zH{ylox1D{+&P#VWa*NxR^*DOd6$|!l`NYL1bnO1*mcfO0&kK&*^pIap^jogp|F4%_ zy7b4lkF4o-!IL{b`P8;MZ~4}u#RJ~H=7qB_Iqi;WfA<~zW$MFhkH6e@ z(1?%s8ar~>^n3O`|MF!$-rZ*OHAj#B!coBwGxxi0)ou4(x#-NkkKB99&Uc`(T2YtgMeet4$w0Plf7v4Rf{W*v4 zaqF7ws?!ht+v!Js*ZYS-Z@#g^J6j(A-3JSft*Lu*@TFJuxbK}KcJL0UYI=D7;3J-x zF>Cqkx7NLMXp5>%HmdpUseL~C?ykm1e?Ia1`O7+|KKlHWTbJ*A*_FEvc;)-;M}GR^ zp`-rv@uU|TH@bX-%wC&(d*y`8sy*ip9OSoI^!IDNY(4a%B~Se8{+AAYt@@{z+g882 z^wgvMm(Tj}m6JC6_WOfV3+}pkd6zx5J^r8HRMZc?`SbZx_nmxTzQecAywm^iv5SH! zyXVe)aej~EXaD7+D;{cd=z&x2xNOrq-&(lv)%s7j9(>oSJC2|1=U=|2#i_%4$cyd zm&bKz{HoKKPyg@a7uxx6SKYh!38(#d;lN9F+ir(JJN3yOc<-|V_T2laziqO`@MXE- z=iFTR($>Aw-pQv=@^;?3-#5pNXfySopS&Y-zfM1B-}b#WUfyYg&nw%%cj+l_XZGLy zqjTGQ-njaQf1ZEJ!o%J^ee%a=f41SG8DD;Q-{YGuTlv!tU++5M*DLR9uD`Y8FV~*6 z%)9fJH!t08?zxp6PCK-6%iS*?G4{nxFW9`xH=}3YfA5)f^>v@_J#@R}llRZ=H(>jf zuWt8Yn=|&V-)^ryo8NoS4r{NkeDkx0MWaXE+;e)%O;U$F`(u|j3*Q@h-d}e9V!Nw< z8{75Rxs!hQaK`7S@3HHst7?wj;r^5F`rF)LtKVt4{G<+lsj41wfA7~$oa0Q}^3E^k zo$loito?JFj+;Jv*ZoaB{yDC4@m;UX-gxc!bI+TXzxkyH_PgQjf8G8{#kMOOcbvXs z$K$)q7&qjM>KFf9^>pgu3!nS)uP;6141M_7tB-0mV82dF?>>FXDSh_o@W9=hkAHm7 z8MA9wUGc%0FD;+(-Jci!a$jc9@bUkeamwTOp44#fb*C(8H+I{`T~lkj>^O1ym@)T1 zpQ}lCTY12|w}$Po_@-WmKCS zxYyuzN1XKY-0Cbe^+_eZ9DIJ-svkgy7|Jh&Uj+h zQ!5^y^3%$fUQRuC^qN}+T{v^c3!2*wxMQb2r{)&ie*4aUyYI{W#$9&x%3GG+Fn#H1 z4gCha{?f_r%hz9V{htr(`~IUl^nybwpWFTy)%vx6Iq!_-_c47D+Al)S?-QA6VG)Q-+G}0m6 zsdRTrcZbs5-5}jv0`hI#+xLB*bMEtgHOBet=o&*8E9SiBoC`L4{^qqWXP&kU_)}$h zhZA8Z5OrvICruKe-f;)$lOD+=Mbs!keoS9=E)IK!);j=X+tfB-;x!vCIZ}CRE$CrP+SD)C#xh5BH&VeaA^jaZmQY z^=LJjOi@lmpYIV$*~FR?H(H{pmBtR>Sk)alvn<=ks$ulz%@?=1>WzxY;md0p?tU(J$uyAm=h&+CfG2o$Rpis_?` zj-n#WxGfU5!nK8&X1k!c9A9uMNeSe=r*tYiGo>81f!gE;=fP8qdb8SL^Y}~B#D!DI z!P^dP*#5!fC#{f34abtCS|9uSW^!S*m?EZN3Dh@Ed~NN0^EX4LBjXfTM8tH+ic5~p zH75F0Fb|)lwLF&!ZyvfUgCwJmJd)lZVNiszqOnXGYjpax>Rdpgl|R! zwG7lFy;?uCEpyc)3Y+xq*l~hFczmKx0=v|wxmf^fA&@KkU}bAnYEi#ur3oFVGqqTI zW0=!i)^KXJlrrAH{<*WWC^0dM+AN1pG8NzI%WjqLV~hwnX+65IrEWHNx(6*OGq>M+ zUk|15n&0|U4}JdnXc32{O=6GMzeCM2ZgAX9XO8M>siZcHkbk{er;|y8=ej|BGmrd9 z&t!3Q`*606DBs#JqprGJUCfZQk70|*o6MC~1&zGSX?MZ|>VgKgNsv~Qfqt}zCaY!< z`tp}#5{%HC&u`=1ZH*zNZ$^&li%gLYgT_l~t<9Sq$ zSnFbP&tChA&wn)0rV6BK8Dq+hrWR=&9&SsO&6Cq1dDX3GuJ9*!_Gf7(3AYGhsv( z!fd#h5grBYHk4dDkp%*;?}q1PQ%#aH7Oe(j6E%s6jqDX!w@miy+-gfVjTPR)cBuKl z|K`+thDd4jj#P{nQ+tOZw`8xoOS&`^xufLR)3DEwalsx+6BiNDkit1>=s2y?UGiPt z29M}#ZSYu@=riIJ;zr{?zZ3mWQsZTY@XuW0-*=qUqre8p z1PC+=>;NnFpNs;aQb41?251wIQ2?|8&?vA2gaObfywo3L6aaw(jRHGhAvd6EfQ=n6 zeG4=ST!4Q-Mgia|fJT8GaOW3(0BC7|Dgnxbmlg)f1a`oR@E4f?Xb(Vz0YwA0mv(>2 ziw@WT5dx+}0UPIwt@yLO=!chN=`VEy=1d3N9MFO2hsZ~!>T%NZ{P0FMH`45;wSVy1t4NdH6jqraNqzntj$4^RV601S+y8+Pv>zj^&* z?xX(`)cC)D;sIvlzpxtgfS&!MH2`YjWz9NpLBRiauEGAtf&jS&H#@^0-SP7M=8x9+ z(`tWQ{=aGsP{N{rZ{L46jepyhgX7N)|3_;WnSj*A|7Z=6<^VjV|Ex9GKq3RA89QHG7bSf`roZ%{zGy=(2q1uK!Vtpm&X9Q_@8tq;LHCHbWEfMv@6E; z><0v`_u3UWYMoz3Vf+R~QBmUTg{b``unk|qM(Gl@S-DrITBWBS(-RZ7eUz?-B_@C& zQrYjxVrvS~S3ALZZ2bOQ<4d+TbLg4jSwB2_zvHpXyUTyd-8W@4O@%BcCzlyNLA)VJ zJz-ZgaM%@_L#q1#=md5Ou7RgD#n&G*lp=|fcKcjWa``3V4>}&+j5_G;5KTD<+%ev& zk9~G^y%M-?4G!MjE6lt-yStuaMqf_!!e^a$vim$thRi4@$6?#kS$P+Awr#LsM?FD} z6TXSg5|cxktEl_tp2KMI3a4TuiNk0I9xta;?~TG?V4dXoQm2TLu6j}m8fIo3-%Qsg zGqT**+so47&dzjI*WM#4E2bKS_CS${o9aiN9lMz%%CubQj5kaXZZZ3tpWR&DU0q#E z9Z&agH1l&-@ID>hggun4Eg9v1Zc-sc8tcheA-+Kz zW&1#=*4y^Wb!XEgca6aFER`k}-G{`TvdYx#4NT&AAvFzHQnu0#)fu5QwW z@WY+m<{_o)2QzoOsc=e)FiK|`cW1>XX}NaC1+|$WkD1+~0~3lH_aZUou+Jr@wDXC- zpYY#e`G8{)K^zLC?r*nzC%$FuhwNJSE`8uZf(zjHs0F*CYwXz}b|^Yqy)@5osc`Rb zUq4TlOdBF^R>BpoP>YwoUM^E6!mPJOC*U&8Q7%S1$X`|pLtC)wc#=$8NUnCl?CvTd za6~?KbXFoTq7aDlP>U^Upkf8`HlQIkBo zp`I&Lp@=IXXtsA|{DS>UVF1^dk+50_{1*RYZm|G+xUb#JGOMi{QkV% zeAZlvv&8i>X7l4DA-L5#o7q?~Or$U=sj)Igd>env7s7hQuQ^P%O<;%b+S43Oggy{f z@sse;!hT&*ck~aqeJ(L;XqC8X2eV$Mvvxhz^1JnGgcjC03mB=nhlA9XHsA zJjAL#y4%C3Lr=y5GJrnMTjp8(&``ffwUiXf(^jlquv_nG8v4C+vp@k>Io(5I|Ds%k{P`|t&pu$81w-cyj=&=d2V{mXE$IWz1sC zgL$=bFwGg^fA}kX%OAEE)?Oj9?!3KCXk6?sA6+EPgsoh1S-Ur#G<}|@&xYh6iXT*o zOAcS5i(KcN0n9>f_Wi;polR?$G*cFfE9df;S)!ifg*p=`)V8d~LgkBdl8f4Dg1~Ld zy3Fu-+bQo)1xURA!9sG6>wBJszl>?J_| zaZ4~=UIv8wbV>JhDWm>m@u^}OfTg3?t z`*+XBr!~jz>0b6IQxOdXVjCu^i)%9Hs@8W}RQ2ogP3C)dPv;(i`RW?;>V?#upIsC* zuJdSTQevsp`2;-Nx3sP)j}8vpp?%ZogV`&?1=*SSf0jK?^?gv`efYY_IY!~yuR&v+ zfTwJJcI5&H*4yj07i<=0$hR*T49U4x?X!%inMukk5?u@)`@AQXnkxwKi_KvyYSkgJ zv>DNc`g%V5JoU|k4w0V40_M=i?&9v^=HkT*8UvkIz6&v%v9Xe?7F9aL#{DZIzW4iJ z@AbM4^a(}I@)esF$$aAROZTJKC4Wz?59|ae;Dq77Dxn?U-%{1C;7pC6s8Ok?-g20t z#mhr_gm(Rj)e7Fazx(lZ;K$y+sGv7qCbIrTgo4X0XQv@;ZnGuKonZ%TXl!ZrWliJ4 zXX8%iI%5znZAi7xl=Ej%4UNI))ovVmmxdI}TSKx?iykR0>^xNLFnYDW@cEFwI(a{y zi}!w`qEF7E9%SqrI)uY}5Hw4_7`($Vd-XFj8GJSPCc$e}`hp{jU|0!htll!KX=mnj z$D<@A3Ooh=VNvZu=$91}ab$R}Utk&UE9h@!VZw+={3qwiFn)-^tm`fDzLkBydfErs z{swUsy8zDz5C7738aEX%X&TsQd62Z|l?S&wNHx^l1 zd$Y4*#RiZ;ghXprNH+@*caC)&iR?e=od!;$%}_sn@EukY2uPjr&$h`e4z_d1<#Zt!ik=9S>|!&TALXAy4BrOkO)RCa?kXplaFhJwh0bJW{f~tB_h-7pKBd zOrGqv9zwRf=$gA1g?=5Y{#Aw|US5>}xh@gzl-{Xj_87eMp5yG!+ehRp&#^_Mgp0-x zNQcT;Qr%m;OkS!v+EvuE@rQij#>PQ>QKC4KF z7>I{Q!5(oVXzL%D-9yLy1mm9G#;uLX)*4CC9Dm|iCokKzG@pk_(*miNv4wF=RhP2u zjvBh^!y#ZHa>(tOnHi)&`TggGX$XN#Vpq!{9o^IPGyKu#`O&nEh6alA`9_-#TJAg> zUTK|bC6y4P2BfqhVLCP&G2u9GukBI3YKb&U*{;COtI}~=*x>ch7VHjazTL*z6iKO( z&i3bJm4$k>aiSurata#7T&c%TSN3Od*4#U_G{Li~?nilbK^}Lma&t~F-%p_S4S$Qe zJ}UJ;aNHlv`sa}{H+EoqZ?m)ETDy1L*gwB!0f^bn&br~<#~8z+ekl%$+4e|Czd1}= zhsRLvqSOo4vm!&+2OjxgUS5e+glGr>d)Je8gy_UJeUjykxmCU!{)oeE&Fz9$1m}uJ z#p~Cn4r=sYl5dc~7va*z!d%#1w})g%BhdDTl3$SCV{+i>_<1Wan~=KGxo#H3 zzn$MWmU4=0&)Dj4%||40W|qryv(T@`ZWZ`(8Z22VoUW@CC_?r?lwuo+ln71L%g2Ly%CEZS5$c#W=%9hF%Ukjn!b+ z0pWL7gchkG{Toa*A>#?N*!29>avE;C1A+iqEb+;fcT2up9GQ#K9#ZF6tlXYOaDA^usIx+M3)c5Y?$}!ZVJ)2Z&l(38r`kjx9q7J4HUQt#fC8cW`x@K7j zQuIRRRBfW;jYE@$r-Se5LWRq-LNtW2W;w{+_JfBY9d0Ny?AhKNw}e(3Cg7@`M#Yii7VR}0~g2(boxPM*tzGN4^n$wQO znDpodU3oNvKIQ>wvoPIx5=p#|6sAs?*4bgeUcJ|xt>8Y^EEmsC94*D)dGbTBWw%iy zatIuwE!D?WyUV34@FTj?0KGf)RAmyC6iDT=$?aa}C2!eu!}}Bi#p!nqlVxRy-zC$3 z1{Yo4$!|}bW50{T&iSzsS}gvUhUR1vIlAt2>Kg`)07uXl0*3QSE!i*%bJZ-lE*@XD8uf;N?(LPf*cXwcy(!b911Pd3&}kDERA&n5e>1eb1M@W;Kg`n#nFB z(dgdxV#3ln8<@b$H(fTK#>VSE_5Fo9QMw_`dqoxzsMKRk?&O&g-B+ksRn9SM$hMNzC59=@1v1enpc|Hc_I8<)cLDkQp@p04`(hc~7ZNR1=s| zXHkYIB1B^+ZXbWZW3^1=K4SGQ9- z9vYhR>9MRRe4LGC`kAxeZZZ;I<110HzcYerHtDcsyXrwNc+2GLhoh|O&ueXP%$cHV zoo`&XSGm0M7-fT_pX<7|a&9L^Msc^}nbE5)o?-^a=+eFF3gXeJ^DgTVfR#sRkWvSIV2e;L@buD=utA6J}=Bm+LlMA%Q6^W z`o(joYa-TIqnr~t>EQWjFkwDiEE1K(S9s^y>KAGea1`6S>Jj_jIj1X0VJesg=KqMZ z9pT|!C=?h@`LJt;S*s`#H-YJ>VMMCJL55K#J4rDgjXZoEeTbo%{;ovPvaTDO6E6k7 zSM@kbyiopJ+1M2_on7?V|Wpc-*82eZpJhmPj zMJWk;QE*GHgcNS;=&H1d)qY9^j(#Jm!lX)?$d){Swv6PZo}(o{{Zyp_ScoJ!w?*zr zgh%qm=CQ2Uf&lz{n@1zpMi9Hi!hoV`+6tykZ#c&63d&u(g8fY$*$#Y>45IYN&Xe?V zuXeSY?svH7gL6NZB$nUX1kK@H)$T23u$+Ary?%Cj1m%&T2C9KjV7r%(h%a?t22w zb@`{LdC9kx+szRZLd`A>mOrM~ahVLTeMk20A~Cm}WJ3!O5R=PCr`6z?3YHg|3Zs~4 z5z!#3#A+GVb-rdSPS?}Dez z*{;^E1)Ml10^_@so>{jHJ_`nP3YyW+cSlMQJ1%Bl636s;gw}QOa1xE~YEzGVpeA_^ z#HxCAxa5z#!7DY*!&<~HPtg?JLO!sXAwgmrqF;yon%e!8?IkDOJP5q>dJ9 zWT)WEMsT-gSL%O~qovvWZH4qW?FPE-VkjcO`%Z=iH!?#P?b(F|jAi~>wa?ry@tZxf zpy1wwoEKKkLY*FsN?y~PhxT{zQuLzBxaA&mRLj0i+64ogd`~jg%?AEQdcb=;ggAF0 za?e$!wTEEVeJipnI8t~H3?t|yjP63IODt=4NE5*`fdLmytaCXr1cz0?0BbVWyiT9+ragikoh-6JQ0{puFu+AWzLo5<)^o| z7wycjr&ZP&Y{P-J8*YrI5SQ@JX)^IdSE8&Zau?Ax6Edb!gT$U2uw5t#!HQbn`unNe zQ?&#&za`!bMI_fr8*F;U$|bB4F>hk|7fHwoQyEbm(>N0lBsj+A#l_J(_QRA3Yx%f=H!bWcTt1@i-7ASy7g$$P+a&+hfPvAecqVLF z-((3eIitGM9}j2nf;~1_ypG6kSrsPC z{tftLXB#D?WlBp#gg}M$=a^s^NgdQ;SV1nQXM`$!{+J-wSRZOs{<1)vF0L8gu~kui zagcGnSv7k$bOFQ;>6{X7`s!02?X$&6@g#RJ4&m(jhUsKg)j27yeHGp*+Yo)&*4}O5 zy+pTt^N-GLb7clI8}4Y;_}6%vb>^#rdOfdyAU*HM`QOgBl$kt)TpnHaWJ}XZluYJu z95&9oX=vRLvUVEq*)mo!M`KzV+HE#3P$ojvw-9iuxpV96wZ7WbLO4MU zYl8hB6}gba6_C5r#KI?(YoYj?P$M(xXe>!whwnPfZdAKu)FI(Hue!kB1V4FHeC6&G zpE_Yw84uClF+FiGJL=+q&#qU@Udq`!CbhY{))f_vpsuoS`pO|kI*J`qRh2P zLX45Qb;Yg3nacAR$o_+&O7 zg7TR#mTR(4<0`5Jx|H=lvHVSfzXXg#)QnfvRC}v8aar)jCpE?+5h$GOa)_3Y@P-Vf zX7h}`!C(E%z{vMJ>ZE=J<^0Hm-h-WB8kfj+RLVqiEd}A5u7aCKk(<@r>o&WP$YvrM zwzkOL-6=|U)Vq~f^J*4O+4F$n6;@3uPdrWar;2;N6{4B#;vT7!nBn0(34D4JmrTCw zDbzLZ5uWk2_6I*OzSVjD#2;TsEAky?qxg1?c2Q0m`j9)@1b>N!?w@!*aX&B4WgeeW zZ>ZW=n)e%VUif@86QaV?I&?3wuA%pRIMsVRdFt$STr%Rg?`7Dzh=L$LHTA+Ftn_$T zp}X}7gets~7!(q)znT_PD4!)F9{$<-Q(2o#W+=pACobW1f{i0qhu-O`Nw#sd<;wC*O6k;%1tJJh7y3dB5Tml*^BTB@1(!2nrcen zXjj!vT4Nq^V3F4}7uy%llafsRU(s(Hd5hV_%HGjVNx*2dvLK1m&M#d$He1Eq<~K$* ztsX6SOu?Uo-s`RHwc7cy4UoB`3V07xClzsZ7Iv1LP1X&vjv`%rI_+rY=R$dMFrA4~ zZPGBB7`l;Y?s?Kqe@;-n!uY)t^JcAX>tZH5Ru3afR4_MDpSQ!m%8(L+Ya?x{<{pXl zcQN+mR$Jno8KGuKZw}-Y;UgkH@}a?1*?~W$M6{%7EsW2QU0l*3bat6k@h*jur1D<< zkKi;?4aV)8AZ?c<$FP8L!?f`X!8{_{@g{bFIVT5&c{$S&QLIVpAVnX>fZU?~*y`m`rYq z2>r@nH>pAr%J=!B1HSJG(-4Y)vTq$NZq8Nc@VR8Uk+`bky^4R4VS>77;o z^eV;=ar_pGUhxM}H^p`|5eSHus74nCoSr=Wessao2VkA_K3{5hR}Teg5! zG_&?pp!ex5;^E0s=0PBA=9|gZjbX$itTbPzM>Wp&77D(GX1Y|Wfstk6?x`u(DdBRl zw?5xJ&&gfApH`jNXsAf^VuTVp_!3&1BxfTz=Vul<&|1_zgmX*`8uH%W4GMIVlFXz~ zvEbHK4c^q7rQe;m6KzReBqYFXOs&;mQfjY12hr&_6j>CnGE`2nJC2<0cPhJnZ5y*# zm36EnHtV@g~1 z$1`AM^D07krDlIE{_TkoVIP`4_*8eW$KTuSL$0k(hR3Jq%U|~4{QOi#a@y-$m2uQ~7u}y(L7;uvr zexDa-jkLeDk(3}clG31EBC8h@`lvKS?8n(f25IpLW%@xM#SvEKF~FQwS9p`d{<8L7 z4CmalD_ZB_tHakc^Hy#mpR$mtX0XU>(fLx1boliq+{CJHzD(TrMN9Px$#+}RylrRm z<*OOF&^Z%JMT3y$_ya>Y7KfoDt9poZT*z9lE0-d^x1_=C=tK1Ft77`D;9DVWW08ec zZ!HrPrA8{;X`y3hHyP7Gw;YfC4kv!cYVE=y_!H*Psp*9H;Iuh-UiL~I8?S({ZV5L{ zV+jt{vs}X}-oYe}N)PY|d~|NN+N^Y8A4jY$S^3>Rj?&Vmo4R>|@>_7by37Bcf!#kP zuKQ2I_8)Yxyo9`}{0G|iCKd*&PD%zg4zjj3wlA~y|1VOQ0ifZZ&-epe^{*++{(Su0 ze@I#M=hT17*z{85Kgj@=mjnM>0{@rQ{qF?8FF8T}kiWkG7*M#ft+?cSKw=*v#=nV+ zUMl`4s~JE8CpR|(7a%w99~%B&u}e;NZU#;c_CJ}b{v}lw z69Wq?8|xo%`M(op{p-2^6~hFk%L6p|Uxo?5(oDb)T>KAh{_l?dFT>(7z|u0zAL}%rHSImY6^c6NDsz1nh4LEfC=YaxEqx0pkEcND$!z@l6oJ1Y$7| z!vxXCzYG(EIe`R>6LcMjfPs#I2px=qe~-7CY!nAW;O^y+79hV2^n~=f^aA>K7i8vb5G#-1c=H%dmw@djGYzOe}D{Pe~tm;1BCLj z2O^jt)(Ik*AkqsWnC!rM{w0_okO_>B71%3(2_}eVg8C113^+bPY0N+b6NE!S1d{{A zK|%3>$^*s+N(2Y!xIb7E7z5yDe*h?mZh{CcaJ;Yq;{y!Em-qm~`{hhP4z@pgKw=6Q zD1Y{V&)*B0{ci`J|GseR?+;5>z~>>oyuA4*;rXx2LRLIr9WhiE_gor^56Ii{dDA^D zd5D9K0#IP`U?E{zd{9uL$h#{8h`>=<5s3WhVBbT_7~8$?6&m5{B{bw(X<@!zlsJlG9jN z){sIk0K<_58_zBE*oWZ%nIizzfA`rhNZ_@U}F?W6D2VaU;#@DZW z=0$4;cfBTKA;FC!UwyLD&{k+t=BD`$&7{5j{Oc#V1NQCb<3ff|aM|(VcHHsY zamakIamd4Snjh)kWj~>5f|neo$kQ%{Ac@c(LA|Z*)NBm9+G3A`u*}8*3p2;B6{bH@ zJS`=E4lvoGYK1S{AXOLp9cn*dU?zGhjQO|h0f8pCZ~`7o z9!mDxE7c7ev(C+Itks;4;Oq42fOi*Y`4BPB5~LP1uc*?3{pAsl&$!wll{Sd*gOu2i zxRTiLzaf@)&@B5vuPc5FQj&+~)rZ!Ex0aU#-}mh*EJdX3n)-y|4F1DvzGrcMQ<$(c zvaX8uj1BC)q->0ibGD00)3KEV8>G&H6Q`2Dfip-tyK&qT7JtG|^V6Z}1A{(ym0v%nSSO%;Pn z*vzx&_`sYu6qOO{?r>Y7FpXih-ci4=8mdGhh&uwikosn94d(2sk?@B6jZlU9kk=6d z&BAF@4cE;zwH%Aww9#!>jf-9V9P%U6e9JF8>MqO7-lqtxLm#tT~q z4FfC42n3*hgBZ^ZH{>ym^uhq!HDtB<;kxlQ?(*{Qmuv6n~q zRm8l%6WDaB-hU{O&LaZ@gUsiIH$$?7c$G~E5rl{#3h{+Pxs4Zs?wEd`3cjg7%yezr zsPs74{>S(*wc0SoeRuLHY0BcTA^~LOU_ak3; z_e1JHcBC1-S~uohmBwhDD`0Wpv0=;Ya_X$MrA>36H8^~=x)Cu}kLDCHfr-Xs8R1!0 z2i8|7{0Navy3Nx1eJV`YpFHMBc9V5aZ|U6I&&RW;jVyzg*MdtW+BR)%om-_pdewTU zcpwWQ2SI>2h8RN=KnfL8%9ukMg?mw^MSUVA&qa_h%+=4+<e4-UFcwb0WP!u|Pl|M5Z^NQ~-W_(Q&3|FXx=#9IZ zuoJ~ANJ;it(@j|ZPP7P^Ny(sW=WM9#fZEpZW8D%~PEG<19aY&q9gm3+k;_vprIqQp zbF1yQnO%_tq@RqQ5)z;1t1?DCoOjaXR6cXNUA}UfqJi;elWVNv*=vLe=zG|C06X+ql%Uh+ibU)w%61SMDbA-@RKsccy+r1bWP^w7%1+g+qvf z4Ciy%GlBWa9+YBYh7fOQWj9mRmt*11wp1VdI$IvsbW*yPxJua`p|xxB#QNNgm9?y? z2`6gSzG{VXSn*8(`&@!$-FCs}oJpfeqp8iwO_wg4TD;TQQ;s9ph2v#{W?n0oMss)0 z!5W9saw~NE9!lfelc~2RCE5cEDar3im-*I<)&v{R&aua{MV|DupZYj$9}ju2&Sr7? zP8s+WwLLA~3Db=>+nz7?oDzP(ovAXl9pJAx;AdtX9?n&8GKBxc3JWt4+S zkZ?V^m$c{iZBM?9m?wxQ3zwkq#-i_h9GAAtj#I1QHewRbki_^69&vJ*@c^OT{M6*? z&uceEP3xv6Kc|t&cFY7*hhhtS?rin?r1p9q2a{`0DZYc+<~JCT_<$6KnNdjuk&v`) zQ=e!CPgW>tSp!u$)_}cZL})1&=1c9KFd0=?twvC|+5TlcOafaF16)^atzLboYaqUzSBV28A2sb1 zg(Zb}xF`0PHhGU$kJW*ed&J+wVQm+yF0xk>h=&;s>o`jJiF&805BByfgQ6Y%A$*Ug zg*SW+-q`w%uwdR|=rh`A8?c?9*WH@M4XMKX47n2vEmoIKWW_Z||U^@=(d4f`4~e(q+MJorSSwx$~a4*O>x7jrd10wIY=R(%cKXazX~)E{_x{NTON}X z89_`QvAkJr?dP^+)xye6zEjacJ^v%ig&YrhTD_sG!ygUQ=g=59+4~viXsU52_=Cdn zR?Df^3PC>=TM^dAB6xDB47w#nG?FkFf@%0ANe-!RDWA-dhTLJz7Q8AS7R*{K<*cIIg$;ib7ULG(ML#pN)jz7KuSf=Y&Ku4zu+xI+%%HQQ81gA zRIWdI^W46c#LR12JI>s0ZJeDjRebl!;<9gfBcq$i$nky7m-%9U5l8k3Vs=x-sa)!diIqCRny?5>EHk7DWPgg2!+1d1Q)%$rk8ChU!*K-*g zXCi6Qa?HzQJHq_i+n8v-U$QB`zL{27f`9Aug6@dgfMsXyk-_Z<`#=@x-jmgZK^8t@ z+>kb8dA?m|Y++Z&U(rLWD8_@S(y|Bd#l+l-LWBB*3=Wy@AHBo8_$X(^^LNqYV4i zrXys6ges2ySeHFWd-zfEdor3g5h`iY3G)CEX&_Y`%Tcc@PYNNO_NykTFWJIuS8wz# z*wgn@B{IHO@ydUcym>pL!Ry^Tdhy#h-Qh$KlDHKhLYZDQLgHEz;xOYp7Qo2 z?xM5%$|4rKVJ}?JoZ0+ zd)%8bX4e@}l`h$y8MIijc(M?&mME*?sO2c-XywS^sOBiQWRI89>adbGFHYVSBBt~orL8_N)YPHBdXa@{o0m9ZLu$tI_}$r$tVg-S<@Ww zJ&aBceQXpba`sF`w$mW$>f~=PKrqV_aB(B_HZcRqSph2m~pRoL=PDdS@PP7@*8gWP=Bgi zJ}M{+(raNnho@(tdH*QIp4BuOR}ewy&Z5HqKw|zTd|Gs`3z4}OQsSK?moK@bWVQ6Z z$#9RSdU+h}M{2iT;d}|sX{JstN$Ocy4jJZzpKf_^qehob4v4p#&-z!@_Ryg7$wb@Xm5vd-*|L&y-3A*7?r`Ym z^%Xt?mV>YO{bBYL4qbqkwI1cixyXSwa^U04rLtQrav{occKlZI2m6`VA^R-GXownO z;bW=$mgcB&t}~5TbTa+VJfbJ3Ng)l6bRB&jLmFBTZx_eyvV}HPe(3ftzVp?WxCzK{ zho3&=n5T1xQqoTr@+?Cs8>%ij6HV&(sdb(xAuY+IcG0GAnm1>uZk9hzWS2Wy92U>^ zpF`c@wDq`uEq%yUW4+Q`_LYmlzi9HT7ADJv6mI0}=DeH+4xx7JUfInLy* z+qK^5k@sLNTh|VFE%zRTRuorufcb6r&;zTaH=ixgI`z%ATOWy*GYm>I5ix(s0bV>S&wQ zX_xhByp$0_$K$2-+U!|6?G&0r4W&&@zJ}2j|BLqwZW{cayNNF44D z$UzyfZ#=skfuQ(6kW^pqG3>j*`X!z~K+sjp&Y-hsR`qJ{%K~1X+`e8Kvx&k+RBn^! zS3!1*<6225bm+YE67B6N#CMq=t$l=$0ZE8`@qGIgY{X1Bb7NAOiyhG0ep0w^&7$)+ zGNa@$f|1PSJ{kLe*~idOgqGt;DBx5Vm}OC070u+7zq;@8yQx@;*1R5gU-s)tKS=W% zlvEVBl3^Z?h9aRDUKL5P09ql^r=KD4iFMhpfA88(?AL_zwClJgH1SZmZMd^&_LV>i z){(3NGO-3pB%SV;C>iG@E(f6P6vSajVBiPv95~(l%0hfSSW3(jx)+K`BK!Vrf@B$v zvu7ub zF-luzcFRu2KXZO_FWxXM=%-hucC>9`)}u7V%TZnb{d>(ZOva?$eRl9%2<(xVKt}KQ zwgqDRcIZ)QcFM>S>4%{75JlQ=2KqIKM`2tmAC*@~5t{2z>YB&DT_4U`FE`q(xZ`J3 zd!Fn|>$NIW^Iwk>r_l|5tN$u+G|y%LHz=fv0r(l5A^*_D?j)9fqe%q;qmFBUET)k2 zTDgR?t_`gWb5K5&1~M0HkB0{x;VkVVmxEOG`}BappZ*E~sCI@C2p`^Mxg1Vj50kWj?oG!n6+GuAoifigN9pqDLc%iZtUU(DyxA8~TKnw@1Mx zz9Fwc*-=JQ_nU-9?s3@|kg6(&6zo@}QsF_T2pL2mqYD=eElrH0qojJ0K*gazj6J1@ z)u>XYlT*ctf*ydVzO4W9$%t0>w^JV~Ur~DSv6+cAfyv$Zmo>mb;?pStLl+1U<`hY_ zDE{9WixMuX*n8tFrex?+{SuX<9d zzM*TB427lY>01*RR#ucZoKr2+iQdnBpSEs0m3{uToB!=p2z3a&!_CX5R@!+FJU`=z zsiupl$!P;iJkGbL7nO}O`hCuG(oXLK-*+IFC%AU zv%9RceKS1JVITQw#rrJ#G`UFbq&8|>mfpJkKv^FkYPE3fc}zA_k#@z>P7(hKyg@t| zsf#%5LgJk!>FpuKmt|W;X+to%HR@HzYzargz~goCJjHf!elW^ zCaEz(BFaF;2L6$s2x2D(gj$B~vtmmn{ta20O4_x7Qv7kc2ndSrnGSI*Nr=RGN8W#E zNORb6OHXd+tLSk#w||}2ma<@fzi2YFtyWQVPdj(;R-U*dB^j-ZVyiOv8>)P5`-c*q zkimgbdofaGD%slNE*hM-+E*n#-i%tRUgePS85ug0JiFJNtNf~dy(h`8b4r`Pe>F&p zpQ8u_(a|S+ooZY`>hKvU4n%HJ1?E!_;H&#g;4w!!$>B45USozMJuBJcZoBlR%~x$! zG%p<+v{O>{5}+Eg(W1T^6F<9k!1^Wi$`v*HtT>qrPWYR{JN!BR`!P<|Mbztex0M`G#FIubY!ttu&Gz~*m@hD` zYCp5qG5xIW&7Y9Um>i7N(CWJ8DCmLWLVQQgUf;~$;a)eGe~l)>O{N)ypFUy#0VCU@ zDSZKU`1{kR6omDPZ0$A4xFJ0wYeY^}cD8Z$->V_w6E&^)FiCoT6z{XJ z$EpDQl9?vaeFI5}6d(J^Z&@uPLi)_Z4aZASD7Mol9KBK8N`H>6j$DE5O8>=f#zwIR8lQ7M;d655>o&4jvxJFw8#eIm&o z+=(yho72is(~FDe?ftJ>J`G$OKWd7Xt3tpxZq5+?faB*!D$;ELKNvwva$*@^Gwq*9 zaMbeAc@7y!J^6BMR7cj`RN7HYPEJhA1H-w-<%w>ZkR@pbBRGh6x6sBp{)z3PH*%XU z(sr}tUgVLvDl$|a;=5FWBnOP&o39#+P`t82EGVgv+3r>1QAMG%KQ9YNVR?@@3TeSV z-qvqNnHol}@E!DdZI7i<;KeDc#UJmVT2?_|&>%KF?{Vs(D@yP^SP3$$4aM6f9 z;6+4{KeOO+DN1VB7ZbI&TF$l+1w~9Q6$Q37;5yy__7if4YHV_H8EeN@~Dq^-{2#iB+ zlFW0jSZo3kF@pNRqFp^LeqI-K9RYF%I&^2V?k&)&xmFohJawp*EFLg`{rV=QlufB< zXXF>ErMmShpO7QUy#VDM8zPxONlwCnvd%~Hd{vn+zi~%}3_0X44j}^~9AUigK0GW+ zwvPFHKMXOj=Cx5~JIeYBmsXaems2AI-2)K!rf^~C;oNSdX+)8)C`CnyFb{gD*e`cK z-C%O@e-X&5P?Qj(ij}NlNgP&7_`leD>$obH=wBG5Te_q>q&D4M(%s$N9g>1{cY}0E zcS)hApMj8Dw+JhFddVE zvU_}X8CuJP71eKiW{h;v&9_D96LUw{m++O`3r#>OyHNf$RT0%XRT7n}{7w$*UXZT^ z(UQdZgs)9bz5G(~-qiloA>c`n@fHrhc=UwvUaE@>|Enic1XDx+#9J<-o01{EQ$n{G zml&TIrx-cnJ1#-KR|VUWi`bLwk%704x{+*cC_r$-gzr@}e#S@lFRkxpW-^*o)a@@f zKH7I(zUm;*e~-(-P<)uG=z;B@k+Wg{wMmhCC&)?R%I*$KzmMoMN-omYtOe#m?m)CE z2jh5N8KIY;zr0Tk*w? zS_DeB$VW(#eT2f(d9Clt8d>H$Y&C1yM_Xw)vD|q# zyNuc~GImn^vT?PwPhPGlOv%lupr!bth$%jj@KVoM#5`kv zRcNNoe}1J!&s$9~{>A)x(RwE5%;X2Bc_%If#S;8_p}RPfuU6MqFUU`)Y*F&8MbvC( z-%vb#dNJ(iwr-ITl&;38QDgQl5USjo(6O4}e!A_DpY33k@S3P&V5|Uj=!3bTotK}g zKxcxy1WImu-K^uM^emoP-g;-M`+QRWp_u}SYGi`guJW|%)feAJZ-m;6eKn}PU}kb( zOz2-W_=5>6U#bQ|9EGvwr=*2RFbVdm)Miex4>fq0M4{W7viG!q8^IjDyIr=XDC;jU zYL2PO)$RccC*aUO){){VHvczGen^5M z?aNqb$MNUUj{>dlGApn>W5VV}*@qOWm35R@!LeDw(jpogVoQ;WKRK!;@2fqdG!X{g z(_rxlza-}H47$}uWeT>z3+@?~5FQd5f^HtH9j_74eP(Ge}o z2OU93=35U~67~`eH`w9x)5%aq2;R4c&d=Z;H`M0=6u)T`gWQGSYC?8-(=5nNQjgfR zqU{D5VwQrq*m`v^i@kikP!SVrC@rM}u=?CcaVB^_I8OWqi}> z5a>C0XQ-Y_g?K_P-co0?LJ|p#!V2mOJ5~B8n7?no8lux8Cc7n-K zL;bh*&HB|WF^ncZ;E~3RD!Zps2il`|$(wHZ;jdk>?{_|WFSj+F&dhtGJn`@ujNZ|W z;pZ)CvFX=;$uDX{7B1zpU)$xEG`{BIcG&t3#di7Hz|&N9Ec=DAv4B^pG;ci=tP}Xw zS_Kk0&bpK^seXX=BU1ip%S7%FTCh-Lv~V;?#Nt|)`Oj07vP(H1$s8lnP)B-Ax3cSe zzBKuKnTJmH;nSSP#YqObCyJqtgQ+ys?J3Wqxlhg&hFH(zSZwF>vJ%8?1*jjB)v?(2 zEJhCxobfbP>v#DMUb%PeZZ)50Rj={*z+vV&+;RD*aU6{);?RCZIi3LyY6z_uwUZgY zoH-fw@pWRM%)tR_@$Mmn$sCj5JStg?x2Et7VZOP*MM18OsQNcO;x(}ZS-sW4%i)xX z><){u7=v(Ka)VQ-Qz}hhRC`zr&m}pHo_R?y<&Ul`+3k}<=6mY^J#h%IyviG8AIzfC z+t7d;!gI;seCk-y;(N1PPN&D}OsyL{4aUgQJbkmds_Dp7;NzGTNRc(~r=ImdjvA$a z8y5EAU~Y-!RXpajFyI#3bK0f|f3-~^i82X4)Jfd)S1ELJniBbL5aC7-*SEJ7b=2oM_!xN;=dyVt0-lmxRTzN{iIzIce zF>UwdOcr&E0UpMNTQZ9E+k6uIa;#+{{NxYq*Jt-~kcclX$&KRn9z&{a?S>mYKV^fr zB2UJel5-N~vIJ0Tym+U=5+LiFonO>{@A2ZQCWYO`#Ht4CT~!#m8gmN62Gv9p<6VG1 zm#dX>LY9x3cfrP&wl+0se_$AE>;SE&7+&3AG0&pm%4Y|swNL#EOYj=UuVT4J-Q0ax z$szkkmB`_-!~$n`+vaBK{azW$&%Ozn>5x)vZ!@a>(C@Z9K#>!+vohnNNKl`%*;zgm zvPaf513ux0G1N(53M)?RLYb!H}FMelwY!@zsnp{H4Ym{r5D`)59-Qc;( z1*K7Jd8E zK{!F+z$3!m!j`c zu<-z^HW)>MKC-FRWTta_QuW4=YLAoq;Qfc|OR=RuUTUj>leGhz=FRC+<7(vW zj8pIhMK&kaG6_tht@y>ww_Busq4!BS8@9}jTN53dlczD08LOom6NRHP7M8KaYu zkp}3U8QI2YBJHS=?v1VX*3TRz7O1{`bg;%V@hO@T8OO<9FSmFOvl+r^MQqHb9O~m+ zIpf%gFNp9guTFbQhmAsW(q?$tP)3IwNg<)Q#Iu4^ZQjGehnS!-noocHy!PpA!nJYV z>G7%PYthe<@b9Fe5%zHnqYHYZ<+XK!(M-Z68@R5t_|F@^Y0+0(#jrw)*BBjBO8T9B zZWukhRh^#Op+2DMvuY}ic=cv?dcTe}k>b0Dh5`0M zR;u9m6qz`CeN>u*$KCR`&5Sa@l>YS;5F~zGO`qA&te<1)-qm>$ZP`JpgQc|S$X-xI z?^creZMy5F038uGBvePs(P;c?M(9k(;~gKa3EzvYJL~rtw|-i;)_HPu0Lgb5-bb+M zZ>yeT4gf>S2Ox%+RtBlHhFA*W4N-X~DJ02p z8(w-pTn2!H0E$GDx9}~RvI26QndTyE+M!@ki;q9Wa!!QwWwZ`!6KCLNR6X_Dc}+}VjM2F=@( zgMG2S&TL;Q)J`BQJA{Qg#Eg{)xm$`8Zr6Y#GnlVD6>?UwDvWA+$6=Q@TW z4DvugjhHaxItFPRW&JWHC=h-8_RbhRao)YhpyYXf&7|A zYzX%W#W@dBNeD&6fZ^>6O(wWxA^NugEU8ctD|)V`^lz0`2YY!nAZu%Il6!14U{4H? z-w4C02y(qus$N7k4}=#J#6iNWfu+p_qs@JU)lH7+Pw`fH4B@>J34#;Q|LNeqvZQU8 zQfAH;oqW}T{e&T;e;j4<4Pl)B?QIRBQXU$u0WN(27n6F2d-#YhEq!214MwP25Y`Gi z@fp!GO&xSJjB>L+9SvB_nkVrpY426s)WLW^<}2U7X-9gsYTZr>-o7NL?0GbF4nJlILg|uO{ckWWz@S#^-obkw#U# zTEBcf!GUxkdBIilIJs1THH}`~Hdp5CvdC{3)^~YC5zH?oMBPoPO(w~(+HL`LP?$lt z_276}np%1627{=B`%*@EO|Al7^8^OzoVQEGe&&wRnIJY(ywc1Z??scAF1Ey|s%PT8 z1kR%+g|dOSVt&YM_!GHs&rmZ}N~TrH;!{za4Wr@ZwWk)Nv9AhD1VsX!dwSaCd{u-6 zW_YNs4%4CP@`R%hb-A|)BU;J7u1OLfhdigNs5nk;0N2;NNnp;s_6?+4{h0nrG_z~w z1FNM)oX-cwEoH$EpF_rdTdw9Y1@6#w=|eDSJBq9@9@i+_$4xUX(wV%@Pw)%S9n$&c zM4S(|5R%2P&0C;IYl63;v{lzO19hCzoj+`iORv)9>PUWo8Q(E$AA;9)(JP#k!E)b1 zv1nAXB3DrO8hvp8xb&up*EPnGqDdol<+PaQo2j&2zk8dG-%D}Xmk}y5feAS=JL$=9 z+H$LJu$lt^NmtKal1)IZx#tln3Ry~Js&0hqgc>UI~1Lgig% z4?WZJYaz&QHbAuyJtW@)K};aOX+tkjpM&=0^G=oZBs z)4+j8H)KH!Jlbg>O$DoELLTVTNQD$*f`m5=D@zgD!F1%)I~y2QC=`Bz;9gq8inC@I z3-1m8rmRM+h+F)Y9qpw7BBJ8q(_kY^2@)BFyy%c4+Pw)TWO1Rmm&)bLgWU!=Q0_gC z&pcAKqO86RbZeX`R}fWB<|}fe4!HGs(WyQ<4wf4c7a@s!(Zlq~-}NzuU>v;sIOA6o z)tEt-Nv>)6ZDLjNN3WlA;T!}i?r=DsGvJ`C8PLlptkV#q94YgNuNzWFUYsbEDCJrT z7g#VwhBz{b=`tgR_K7CJ2UzC&7k%SbT$ivgtAFWQ=%rR*Q=vclxf$W*76hbR6f()Z zR8}fg$xTe4Vwnw87rF{!2otQhdoR-5dnU}lT1gMH{?{{7EM;0#T8`Fg}Twa(BtunsavwvN$JPVV`LIgc zk+JIdQ38^`mu$YW9Osd4#1^!eUrWPsxyq4Mre(7q;uAxS~KD#8fO zcs1s?O0!`NpKpSiT>J4*?pu%_wcqeuM${!)VPQ6G@v8RbXUm-R=}yk)xDJa-3TD_P zEx$Tr*hL?|x-CJ*WNB`~U(fHFvO-i;ht0gY@9}vmn{@Fg6~>Y_>hlr|v1(;6w#b4l zX3?C0s=wVG-Ye+4$;0FvI52}CI4jCF^w8eNI0*eraG@d?kpY%Wk0^u0&emy|&<2E1 zWkL+c(O*9o)DL?>7>k771Mhy#)dpkMZ4hfnSt#`Aiy$NQb41BdQKsjp@dm_*&$oA$ z7n{2wC~JZsgo&|PNT5G=BW{nwh>PGfli*$T5eUxQs|OSGmdCAiH*VV~kUI$yJpS^0 zX6GKJFCf?8Y!RBBc&Lm}d)gha2No!^<2W8r9a<}7hXF|!Kup#gKn!8>NXQE_TTZv= zIUH>KjIoq%-?VXAHq{PpMYd2jO{Rk!w;CD|6$fWawv->IaHqahXPa7I=h~OHwbO+r z<+%ntCtJ4LM!o=Fo)DkiynVPY`s!12T~!I?mS4KKT6%{$(!RXT^3j!O*GDqelWUjd z81u55tPYK1WK61I+#M=HJBFgVzu^tMO-L9c8#2$-(Gg^DIM$aKD!zv`&e5Z~hb7CI zj_Y!E%1($8aH@&nx*QVXuOK73dDp|g^2I&_Y0n<{_)hv*PhhffzI)*+&N;LFIy$>! zbePA>DLJ8ui>q#UY#r)OPsy;_?l9hJRCZ&XHtSPPoM?@6%`j( z&oEu)lVux&!Tm3(ajf_2u+|%27$u79x}K2uYs$tDI@T3T4J)GXeWrZD#ib&VZUs}D z!ev=O@CL6sA;k@{HYl@QLl`+RRv=@@4}X4e7~#{D(A}Hilgyh5aXe1X15I>qh6G+~ z(y_gU*XCaF9csrfik0Vx)Z&YkhYq>s`-YV>_xxBQ_Zdj!@80)jBa(I1#ZsM`U{mq9 zL)Kxr=BalbisbO}M#gSoq zF3(d`4)0S~&o{DG<%<<=kLO>{cQ?)rn?aP8y49tlDRm5wlr5ShB-wm1fuP#u^7P0` z;(I&HO?QbzWzFR|q}5HBF8IOW^%eN*omskhS~exTr3iU)z+% zJq09~%EIyzO%{p2KIN>ekCR5x?H%^bq~9nsnt)VOl)y^5Fo~dyJjWwfh*rujte!>Kc3c%I!A^}$!$tE;hhLL1SkIG%1t)^|? zq%1&YwQx?3(wQYZp^db6PB+bjlnR{_cWgL|ZkSVUU||?XPP_nvJv~8DVrVGaL{yO) zKThN&^hXUvYFs^>kx*wAo0=#jIua3J;InbELZ(DSTpME%r=gWcboe%8K^ZlJ1qDz+ zZ!5Z`;nSHsTwG!Y-$17g=Tf?SDk#YIAv;M8D~78)-?Hdb96XL9?}GE>rEnppf>40R zcgXtUIN3$DJt$Sx@O(Wvwi9=?;!rZaPbvq4XSYx4Tym0!FY}C0>h)xwCKgrNheyf7 zQsOVOOoJ0n9paOX%q{cCRk3@OVt~iqd+7V+l+tSiKaLuib_7Z>3fU3JcJ{rC=V}qb z?mciwApaCY;W(L)AS#FU_6>nj_otA;7irC$;T##7yi|_0H1u{9;j$#mWK#7F&&8b4 z`lU=naU#BuKcc56jSO~18<3)=dLR2}Ff|sbGie?MrDN}2@QPog4i!!j)2A5X8m2_yZeayf7G`E9@3{Qw$UFe+h8@7a+E<{4-@$p+A7wPlfGmOh8DR{{b4Ci}_#C z*jyll_#bHOhphi4vp{q+2viH=`42Sqf6g7f#!sO1c8O@XNQ|FlQ^Pf^*N z%q$FCoScCFji_v9c20%|KsQM954YnyNb>CZg>mPOdq+|usv44^uQ zMO>X-?16kxs93t#m~s3 z@Ssb~Kx_~*DggNc3DC^a&cx~eqO$)Ll?@!i|BlN3ak0!#RQ7k&GSd&#@(+~s4^;LK zto3(P@6Ri5exR~{T!8Z(75W2L`xBM@qs$+u*B^N5AE@jfDDdy7Q1&0seq22A1NZ#{ zmHi!e`U92C{vG+r^c@xY6J7ls6$-iz=l?YF^I?qsJzM=4k8A+Ie=9h}I}Ie4#F;d| zg9|_R_{-rNQ!^Hl$7X0~Q1LRU7Lt{pgRvXp39>RqNR-e6=i;Q8Su=&%C&d=v#>5~yN_{K{0cgrPL?vQ=LFPmf+yCyPd72@Ub+y* zx`8Wn8Fo&47@m4}eUaAO4xzH#pOdZcIhSLzFV~(pPAws}k*`X-1LHM|!#Qrj1oX_a&nP^k-KHj3)(;I_3TA5V2w|F1bwk zTcM==!1+`HFTrj&3EYPaOaq#;d|=HVgX?q`3xok2k}yK)b2{XQsmpf}Ds^_ft+NDp ztb2>b2sd~8UUmlWciz#dt5v8fe{1FEZ;4^~d_5g`8o-$GWaz!tYop5Q!?VfwlIclV zih0dvw6o_lxSz{RvX*nQRJBMqgD3g3Ck6o{>}Qu&eVs4%P>QhmOsW}2pfQ3Q^w{2_ z1U-HsMG|p^hgeo5{_?iEv&QWDetuPT(gIy1a@~juWjg^0m5QiE|gb>ySwX zrf-5Srv!HgnVgJ@mFCjW<|3G8)o|3!d#*W!o07tdKGm2y)r5!;5k6p(5n1m^J<6>k zHAmmbq_4x=DvDS96E1qbcPnn5$ANX5CAdul@2Ok0QIliGRzX>pXDKXY4ln%@W)pud-n zJzSiosoBqA&fcU)sK!CCJk4s#0_l%m<&%F9%dK3)nrP=p)l3-LM%4q{jgt15rlU#S zJMWG+J}dMp0tV)VyjVC+y+5}J(Q6bqYLXMn(e2==4^szIG(Kyv9G*g)pYBvB zEl3C|wU!**%FprlA(F53{f)v?qV|TVcI0jEgv0A+?EzzH%t5Ag7#k)W)JB^| zmJ(fpEEvoLB=J^lr=73%xQD_72}vj!=B+UANMFaUDBX`DnQj(u$h|vTziy%?I6xjF z+%xo+X-{U?pTFa3pXfav?f>BE!9PW@Ex!JZ6rI>>rR+RBVd%W$b(t^NTsW zF@l|$Cx6WG&Z_q;PHmknix+gI3~YpvJE z2Lw%(9@p#2ROU|^=%`y=j5ip;L)ith`ca5c>xA{}qM8&@p4j;N#Lz6+sHk}C$(4`# zxh<8hdKL#xHuprF$RR-!goL!bU3I@EU>MWcz?yq<_oz$6Gj4y{Eu|9g+uGNN1^*fq zKS6KfmE-aF&jIFlvWhTs?-Xk8MyPglf)Lzf2KL4=WL`UA%ff5xu6{$_)%zeNMawHa zzRE10V?;!V@dmzQ6RJp&B`r*PIQ(-B&q*L5W$9#T9yz=0wb;txuD-x^kNoU~1x)&3 zdtJlXd!CH#LFx4z=oppR8lMf*1kD5ScWxzJ_v3kYemzk2G&{KaWRGna*YsWq!?ly4 z^QE-hLhYmpz|_a+dCRJZKy|3|E4Cprq`KnMvmaC9)7St)7YQ0WXtIXCat7%|^Z;no z=N=;>qv)9iL6%>qhC@1Y?@4=UZ6Mio^~qy0k863jja}vVIh>zeEj`)Cwn>jC2&s6x z)U4<&&r|LDszX`X_st3#``eihWijO$1|sSub0+LPW?>Ojc91q|_%?9Kklo1!vP@~R zSatJ>aR%%Qh8ujRsWa`%cCt08k|#1MV;_nf=`x1lfkPXcX~R6+Z@t(#XYuN~>P!{Dw_ue#_7x_% z>$cmtD}9+={%I$#?xo%v@9ENYv~qr6!^Z%nqB%h#7`6sA5aOSaDnUA7Y6wj zw&m%oZ-jds)O@ePq|G)~nff)wUWPqEl_Nw1PD%H?9Et|0T&w1OFYb+tg4s~$D#Y*& zlr=-o<612}F1nvPJ;&Hj9q;?fn+rW{M^4I-XhUw+@ErG)`yB?#Mp&To3R>f+LAtx1 zyA&%XL2GHjx)zcJQA#pnm+p0Hk^Fwg?zYGXOiD4`;DM#rDSnE3Sw^&-jfehh8=(R= zu8yRj_|P!wm{*B1BUqV4z>MGd^j8|bvllQ0Bk=j2hw`H)7nVfS>zHi!rCHq0fgSQ0 z(04Zl*V-_5swuutVq3DqZpO(|_Z_GE_V_-?k$oH^P|GM`I->0R2Jg(z$!kFb4?I}Z zM9oY(dkb}0XPU=Mom11Wptw)A|EP|L@BXsd91^tz!G0_M?X(QBSnv8(Xo}_#bn%zQU<#v_r zM}|Z5;Vyim;RX9pzvA1_^N!=ay6UhPx3Kv+pMBYVXW)-4I}sx0bFZ9*C{`ajm!Mfo zK$&A|yiu65b}#=z#%qCjLYNn}ubV6;eYAbz3_B}_;_L+43dd9$3deL@uNTx;1h-zf z2K5EG*=}tj7q6HswFB?jvZto**q5YzYnsQ!KQrhHt)I!bWPY0Fv+X0C zm+nMCD}$ZDZ%_kJub?)dlA)rZ-a~~W=pcNe!8ffuBCk8ucoT{HWjn8+eapUizqSz} zt!E{wtBo1wZMy?QOG9B{76Uzl=QAEx&uS+xiJ6^)(YnN`#qx4Do$MmJop#fCX+@E| zPZRgWNr$S-mKhf?GEph-)~@Z}w&PqA+Tq^C`s3(BLHjDjpucIbPzF1D>kx4cb*~_e zcWtzT8|WKd_2KRD1agg1jZaHn&cJ~JJOT<1okzO$EjA3#3LR2W?xquwlsuz{cs7;%~qzi zGzb72Us=3eYy>IWn7N}R@aSTq-%%UwPdEx2QSx9BXrsDyya=%n;b|;@khKuS$@;n~ zK5fM9151kCG8BZuHit}PCxp(8Dn!Mt5MxzOjKH6mhuJU`fYNUv4)7_#iNlOjKMTtx zB$#{YHBACTnV3uHo`+Xn3&lT88fjIZuUK`Or!{hs2Uqzuk3d@?$c!})i+?%~>1j!_ zGy1xUG>vdCOMM=uZ8Mauh4`X+)k#!^&eJ&YadLA=7_OndOh+`MYvZ=ME>ZcQ%vsH? zNES+sZBd?=j|IV8C(g8Mp}4|wk+jDHw5CTp2;$?O)aMgsX|RfGMdqazSDi&v{(O=&OHjUrShUqP<&wau$Rs5#&36&o! z)71drVqgyR;+r7qCxVo8E!Y=Ng~s`E!0eQ zO6keM%{NQjmrrf%MV_WB5H!fg^k~@#MI@5exJ7@RdQQnIvG#2MHL?It^Z?-|YwP7e z%Q)vqc=J>=a(y1MHu-x+R@cc#tF@HV1`%%nBYi2>k0+r}8Y~8|Xv8)PpA86VZ1|Wb z;;tSZVfBBB5%(~G+{YR={D5;9R=Z+4JzH>zFQl>5l1!jU<_2xDcqBFK#lQ7BBfld0 z;_j?6nz^`OW=*Ea!Dl*3YntNH;!+FGXnvg7+RS=dm`~irl2j>}!WK$3nHP!AGHI8@?%?c^7**r5cBI$vu=8AN zsjW+TJ;UdyH^OZzI$xYB^5OPK<#@yowj`m7ndw3V2E}#vZqT$`IvaB}^`f-M&*9DR z3lvK}ehf3Sl;!lQHkLuc{ID%aO2$ojjD9JdG4P?e)e)7FzJon*mkhVso|JL6x1?^4 zmJRM%XM5Q$FeW=xXR@P{u{$YQ9PVgFc(J5!{Sw@pg7nrxRaYnf1-AG7^{8!B2ak7+ zi|rOtEvB5!w}YL}eg&%(2b(O}K?phm( zMi89s{aI&nEF~PRKU>R2yg`SIe0MW{DMx-!;Es{@LyC5hyB2rEXB`_H!yUCmX}(d# zXxdV9F04FwYlTR7F7;69Z#9oFHKJ#zMc||rNpF?t&ENQ>za3zsPC!dBPD(5qEfK6u z>_FzH(RL|YKemD!QHVxsX~M8BidIsQE*(`QF=gz7rx#0`F1<)y*9a@p2tni0qi<8x z)#pkcwXgL{dy`LA*?Y{aHCTWaPbU%kRD_1M=5dp_d4wavNFi^4{+2%Z;5r#6wpbk2 zv*=(G5@!-M!_sy%clEqSbTvM~ckV zjZ|Gi80_;gy+UDc3e>g;siW`oIYa9d4-vv2IU~k|nW37h*O4B}6Qh=kVuY3BAdn6O zr?`Ca>fGWs^3%bL8q?TFl3(FSmZ){rFLv8{ZPYsGt?-q`V-j^Isqn^nc*a_trg6A+ zmXL3jR6XmHaogZTm5a@N8r!Q6(fg36wCx&IwrMqXr&W46X8`NdG;{U9PX?XT5udB-QhCh-}HS!nf|uqDjuDNwcB*FInz98A;D}lDAu{ zl8Y`+u2xDkKN(eipJc(G2KaI}QszSByzxAf%#oT50@g;g=b4h#G zzZk|VCdRlxj*E3`BBx?EICL^kO%(`!J6o9yr$-dUPg1#6#;@Wq&I#S((K+RfIH3-k z^{Q*-qT%@$Z$SFvSe*VP+o3pJ51vQp+|ozaT|T;(p*L9`pY4aALv)i?^j1>$;L#_m z`asauV$jvF#K9jQ{-w8qQgl@j2Us@BOT(#c0>WZa;Vn;ebUE+W2FX6(QzJe*b3 zXY6p`jq9pOi>8`92|wkFiuUCQAZim`mdz~~jCb2d z;dPA0mp8E{cjX3s=CysudZisjIAzf#&UfavR2M|JxcNE8%VL?#t#eE)W^Q{mROQYg zmhkoX5+Bv2Ueq~nbKD`9l($UZH!mP9+fo|Y3Xg4Fr;8@kqPWT6c;f{L z+&8}>M;*M+I}cS^>I_#WoEKJt5a#n`6Uy^)B;(B7%T22$R>>8gg ztj3jLfA8@(yrPC)p{qtvKDn+Z85ye)lrmrL9;U-P6i)5s~&@f1&wiv2V zQSf{#6iqKQmX2;ZGJ&B$kg-;&j)+f=g^@ehrnq?G^e#^{c?6v?4j_r_9m~TIM<8d1 z6929>`J-DbrktJbGq@x6>{w~6x+ra>VH+~^OkV!?y=%4@)!I1bOh=pu{cs3TWdZmt zJ))t0SYA@=;-7C3^GsCtDU{GSi{j%25#`=1!O0ojRvZNQ+HBnKV&A=NExzH{1!p1s zcc0F`2JmJ1Z|#xRF)=AHDM&J}Oja`~I1Iyp;lM*E{(CLIhw}Zg7yCc2MdhIuWC*2y zujTi^xZiy)Ah*qbzl8uI{3|{+DEUFhKixpi#Js$~ z9Kbw3T{qf{3Zmjb&y}eM(4Qk^YiMpN^5d^CsLjBf8uCU~rp7KmDS;^-OvKF0tl#~# zpe&#yW>C_1e;FtN==1vVi}AaY@Q-$b*!=Ga{g+7H%s_4YaG!!`Wer`NEIo*|nHZRu zm{~w|1AqSh{Gc*%ODAU+5eq{npdSucrHtW^Zx4P_bxRW$3uj_&kd}e}zZ3k{k$|z8 zxj^0R!8Hgf1|+t>g=h{G1QXZKkj}vJ&4FTI;`$kP`#}hQyTpHr{=2e%ddj~GS=7?Z z%+$%$&e#;l4->E;(0^tSN5DdW|A1~4kdp(R`MV7M;qYK*VFR`Ad(r>b0cmLm&&)z> zVhQZF56(CEhu_AwK*ub|d-;pIAMibwiL1kpoDbbu3Ftco`fNet738g@AvQ5J`>rT? zClgaAOFMI7>Yv`yUmhyEf~FdfmlW7%KqNr5J^W<^spR`W_~CH{J$*?1;btXf{yXD? z7QYwp(~11>koh}{e}sJohGPZBWEKBc=i@&jLIZQMZ~((Q1LL)VVz;tzfFk(*ANJWX zGl5QifV!|Yqm+mUC}_S3aQ0w&I2)n+Yghky#^LvT4-?OC`F=mK@bkUH_7I+34(PuI zjtnJ3J9AUe(0UkD|7Ap2*O*%AJ_Q_c`N#&%xC0m?!Rr>30M!CNF8(an;&XhnHCQ`1Cw6?7s zaL;4gz()f5D`IJv%Iv1W5-n)bD8WU1_>I!8YraGb?Jn(Ywq})YwqWKE{wqcSJiVg;&kde3MW%p;pN~K5E=nlOrbM??CLY;)R>flotiiuJd z_iK?znP?QiXrRu(-uwV#W<%^=9u!JWoEz?Y(wSw zof*`|ZZ2Vc$27Cfe&3MD#}ie962uvN<)+hDMUQmnkxFv<1RsUk_jH^it$o(7{dYZq z-h~h23;0fcco_lT;+k5-z%yU$983%x0HE>k+xYou9cuq0KPxjc0~03)D{zGUM)A|q z`zHkp0C91BW12`F&SU5qa zseYwk23Y_<+RgTGy6i6s05c~8Gw|FGa7Ou!;;)&AnVIcj{`rf7m7SS^g_Y$Grhspi zzpHpKcm5&(aIrG5uyC>g8~R%nKPM+(GuVDt1}hVQfs>gP@Mj%AhZ%_C_lC1@F);wI zi}+Ji%)cm{{r84*fH?k*pI?;yM_%Be2?j0z=Rex~i?Z2&7ZN)w3rI-7Du1iu7hV5B zzzXbtZ0v0RVColL|IuUsEAW;M06UO}U#npGMb&>)0SscxzzSeu{zDHSCVzI0KPZ5z zXW(Gr{IiN*RL%aos#!SL7+AQt0Dp*<^3cHqPcVEUtPu>PXyKPcFNvppv}+n+6G z{YBLrzfWZUPy^_V_fxKayhHz)_osRvR(Om*>`&HTj)eYLLxT*_zZU1>z=b|2;omMN z02!=5&Qt!q+WddLWCy)A9+tAdt3=YyUu}|yy#dgy^8a@kf%-aVx&Qx#jDC*g?=t$nm>06M zvv>KeMX8*?eEfe(x-xB+woh6KIb4rT+laL3>rdP?`ZLnK=H=2Fzgw+BV|&yXS$D z9_|GLO)P(vZw3HuE`c=oOZnd=ENg0FX(((D+6)6eW(S&A99%39YW`K|z#UJJEL04Q zoPqNY=;IG%JiL-X-~Unw(B%Hf4iF;)?18{-DPY>e1oAr);d{ov<^FT^4dD8I37e&j z3$Wuc0$~6yrlO{xy^CKC#Cv2eYA@Q8v!VItboC;5*YxQ=QBM+BKod5?mg9y*D}KiZ zjxt}4~d@NnbpZM5-dexGuUcU4Pnu1xlirOI!n}!Cm+Sbm+j6oQq=J&s*p2Av(O9* zvDDk9yl>ZbZZ}?F3(yw%slbO)Ap2z_j6)xggXI@zVVaszrVR&M%E6HZkG&!>BCX!Lq{`2Zttmiz4Q_R zdlN^wj7c$<(|+ZE2_oHOpy9(j)yT(qm)GpbncV?Kc$`AxlMu8e6++<_JN(bdKO_>@ zFfPn>2VzOJ3ZiLl4?P{1?yUqvL?e;TvCUwUg&h8$S>jihEj? zj5*JiX<@C--h@zISZDTuA-%KPE(9>?I+8ZzZKgQG+_D@9rNkm%!|-ko8=+iH4TblW zWh$;3iMkIDf#Z|w0P40+={;Q~c)HrmGL((^Yg44Q{ZN_TR9-PVksZmjB?cN^v|PBq zLrwU~6@*ph(8W+qY@9L`HbiKlh|l`c@x}Gx0wU&ih$fwma!G!04n+a(puAv->1x%A z0xHEelp+6P(GyuSx=4YLL#$QZ00*c^h)mgU65%CW+KXEsjcFTk77Ea6mT8RRC+Gya z3rsq+?2emRrfJrC802Y|^#Tp#9AB@<%0A2FXOx@DQ&Fu5$WK_L-G)vOdyKCO92sTn1m#%00E+O4 zSrooKRVA7t$*5eFk)_9Lp47$@4o}X6tMrGusD@bCzMwN%bf0h=$W`3?Ovi`BB{*b{ z_ePpMDz^p;I1Dn5Q-0sO6!Dcz)B3SXvNM#l26wGFPZ9lWkend$n)nTCpd@02daVIE zS-|ke$b*R2@8fhoyHUpxmKVxZgbbEF?k~7W=B(^iza3H-#hTj|)H)u8$1g?TKB9Sb zr0r-}bg`sOWb5}CFJ5?Qm|f1&PY?cbe(-j)g0mpw)0znYCU0B+4HkdnT(s z3(+(!iLR&lx|jXJ;gP)3Yb9kf`|nLk)F?8aS*sXdp-5%Sn#dy~F45;V4rQBVfg?~b zILVd0R|$|_d@g|0R@oT&;X@gq_0ifz&Mf^4O%FJC(GxmWN2QWD`zvRBgQRnA1-}zc zw-cDX;_$j!1#9v(at{Fu#!r^j)8gJxOD|82%_l6K-B2q%hrpeSc!B5E?33=n3gw;{ zD2zYW*N|S6v#C|kCaWwuF-cDHcrn0M$FE#@E03>7?=cW)Gn66AziV&Dr%oCmCuM3o zh|45dHwF2n2AhNw_Z06cW^r+I{&S?HH(k}8asiX~t9(Fw#la~IvEwW$Q_7Pd^+~P0 z8y1R}w9EM4RNC>Rlm!Qg)~>+=)J) zkcyII533+^zs^4PUhBVGTO?fRxzU@$w6c5Z{VgH0p1|r0 z5AWcW_T7uu;F<-eS(A21@C$b@oKP`bs3u&^NU-y{C&`|@9)H^XUW#~gFMlc?(I>i0 z0J~9uaA$l+x$&gixlcEK8sR&a@tqeR=h1TBm1i&d-_NoXrID@p zNUkG##ATh}2PM#W5Kx$l)P{L*+%6N!TpRDe#26|zSKR=ENFzy5$iAla{zfj*k5hp` z>96_5~GtBQ{plIK)`D?_uw2)x1~{X3B7-iuBLX+e|B2d#Sfxkht~A1>OK3Ln^3Bdu6IgJ8!LjWvAov zv+Wck%AkAOZpND6QOe-j*}JagJ9@unA5U-IrOx}(`}4Cy`{?`Yv&!b1>;3(s{k;2~ zorC+sYJbnBF8a!%0xQ{r7d)&z1SFYmEQR zR`PGQ?sqHsr^WeWi2yV`|IJDU-q!om*ahbJkF4YeU-<7<@-OD~gMs`%vyxf<3)7hG zyAA%^>f<5pA675`!2JDu!GC21>n=LuH=+7(>O0y^vRYYLxF2P;+SyWNS(>v<$(>?t z_9#_;$g0hx-V?dMMwVPU0y_YwA_SK{gODe!G^Ds?(X{Szn4h@OHjj#M!aNE->5bde4)m-3QVR zN5<}FvolTVaFM)!YLlTgHh z&6ctMPCG4Uu5{Ef!<(u+tmq6976v71RX7?&=x`811Hm&^721!ABG+qCgo~-Nk8LM~ zjEKTG5YV{quM-Rg!#Icm{M@I&8fLeP6H02UZ_~Vwf|w&OEH+T0qN9>%?apT5>d}!& zBmgN2W~TY+D8wBE3H9G&Q=p?2CG`{O6Z1{Bxlwij*g4sP5={?C7YBg|mv?*PflOwr zJ{>3ybvDTChC^e_=f~wVa&!OH>5Q9(*w2?0Af?lhi0$WgMW|cfptipi@}M3~XZwoc zaINtRav9J0<neH$B|0e*Q_91DrvH`k30ac>S_pVeX))>{FO(+n_&S)q`VY3hpi;yJ}}zYQCAJpDr?C`-_%i!PD9M@7-u||(m+K- z<4~E27Ge3db7GIsOB2g4{YHEp@+RbgF%}P`6*)>pRM9RU95Cn!>_bw_(bxJ+p*>(A z65Q28@BXQoL?qWWRC+cuTgP-DfP^qR!3C)3JM*EwACg&#Yxeira0JR>)d@)8X?{8& zBsLEMs7Jp(+no8dK5N7#ydZJXkHHRrS~HsN1r4qUSxdIRzSJ7rMsZ!)UPMYzPI6%s z2+IBdq_8bE91jhhWE}N@25*s*rVqW*4$|6DR#2EPtd4?LOnLTVPlS87k}0~@7}#GC zh!&IAzj%He_zgCt5P#DY^<#&_|Fa1qb+X3sEY|s0kFNjxcS|Mb4937?MU_dhFEdo! z`?s^=pn$BKJkV~XvQ1X$viotH7sL5za|ASt4O9*@)*L2IX*1|Z8$kV9^VOYC;=qYw zyKJs=CTtu_+;av_f|)}G*u7lCo5Bd6Ba59@H`w{L3!c2Sc|UR>@^}KD59^Jp}0>vo)rAr!KiGb3BMM;&FIbX*BYmEVfvF zL|Z7U7GtVpuaq&5Z#26nC}x5``@IT-bEfA=8@(qEeP(kAV6knAjoZ<_?s$44Zm0bd zhc|k^9hkmgLci@WH)#V`^=!CXQLB%6By z6@p;YsiRgiKxzYK-yy@+ZCtgQ$p9tI8^3y+trpEM?^ssGp0X%sm%r|j5i=8-G?Oc3AW%pqn9^;p!_m&^DB9(@FKpS0;;WuY$0x?h}vT8k)`{ zi#yDsE-Y%)@8! z^=tr4X~_;vQY4#t)eh59LTymVnka5*9|xXBI;XSKb)q~Gof3J`+74bvmbKiTa4MdJ zCMa3WG0QtA8rIQL_8JD8>+`dskO>8H^yq@?2O|&|r0Hg=EFqHQyI5ygLDD(5UV@x# zEE}MT07%|Jlq4aFH_P4VnotUAb(a8UwMZcPOf?m-y>xTH>t6WSh@9eRpwtu1KkV=> zIL?B08bZHQBP%GVyLkufx_##$Vk%mMy2%IlN>%1~%*0rm%?2ng*z1^@nFshDR@mwn zK-rpm2KX8pF(!$^4YfI<9He6fc4gQ)7R9!$`Fb9LP^J&3|F8=!n8}x^ue2FzLw1)5 z`^t5}N?2cqd`dooi3r6Ktv*h^46&Kt%u>Sl``OF;l{V<&i1F~|NYS2F(8&GX ztf|E4H_fqTAF3}I30%?|8DmRRzy?H#u0R+!=f^J#Vap=>H(LaXH><`2?nV`hbAI;x5doVd4wb2Rzl(%`t0JjN~6tk30xS!#4@ z?yC5#rA&d^VrNj3N|dCN1yyz1SmpaA^!H(D>}(t4@4~y0+4pxDw=&2H30fQ9)IP%; zT_;=L6KgLJ*=ag{k!C`PI698}Ag*#uh);!-O^*q zE+h=DB^=0lTK`FQ2te{q=FC!U7;9^buHk{6lRz8r`9j04+7zk+Ax;$>Yn`(KzJk)1 z9ME*Z(UaNYdLkjJ(6~A5?GBLqMAvrl<~GsTRIuHyvar;#-D3OwDLMa@ezpD4=IWbS zNsA!=h^w4S^|{F7m(Y>uFrB^f_l5Oa$&2$7-_2i6Gl(MQB7}map{=2IJczhT!JE7e zSvVjT^jw$k+%sWg*5r`{Bt(KBk3oaYH7CJN9|Nh-ltbShYR9>)`||x@rZj}BwwNcB zUcg&8JW`UNy#)fR5{EV?EWjRhLX-L5*1}PFQTZ!@MG@nFn8w{c1{%iGa_HruM?+Wz z=aY#Qw}hYb@fOQ1lepowERK`g%U~qXX+d7dADOY1QJGqv(k<(fqpm$V>ln1=_v0o{ zdz_i*qH{#EN?>Q0Ig%7h+D;OT$GZohuUajcDA4ZSq*2c@PSrprFO z5*P*RA%VAv+MvXz6!ord;TbGS{T7QZ{UsLl;o@6McXuH?0KRmlB^E9#N(@!Xx~i$K z$YzbQc?9B40GXea|I}`ZW4RFJh<_HUBc?UIq^DyooEYC~>enozW4#eaP(91C+)Ao< zwHsMkcN5Z^pWw9Y_U?&~fE^ZbSC9d?Dx@gOKnTU^WSI_})z*y-ZT?b#mkv@*axkuE zSBLd?1vzOP+7e6$8kDsUXI3+R)O|rsVMFJ8ck6K)S5ID{>Lt$`lrgc;oHdVRoTtGNr%GyjR8EPQXzs`u9+DI5u%W_HF4Cm z2dlo$|6zsY07q^9K!#-PQ+BFC#;G)8BM;g5eXl7m$op-T{atVnMu~9VuJhv)TI})JxtGh^4enzt~k%y;K25keJfH9&3ix2gP*M^5M@CH zQ#pp*=GvLNTzHMpoqvv>NsM1sX=-+r5)N1jAieT=?AXC;@X>^9DAfzYKQ{oa)Vmsb ziDQM?A$Sd|(hTyirQu*lsYCc=5c-JR3qYh7P?OiY2C^-AeYfzKfF}RTb)dMx?4(jc z05_r+9@*S=wo}eM3zc4%7n^5jhRq}iZxDS)IyIx zgj16DVf?=((Olx{{{B0B`{@H}!|wmM zvov{efqn)Sb_rtc4?tVc!U6v?MVS(TQ?(lY^BGW{B(@`K^Dz8iO!g8yH-;#+fw|$H zxA2Ee-CZ|2z1plYek7XjS6SeWnr=(IF88kYXE<#0Ja>OtzhN-7X{GzGFy32_=H*i0}4f5AKkCi$d#&V4^HhSkZ* zfj{-pP*Qu zdS%?6DG;N*7#+VM=s->9H0Sipi>Vuf zcxz%Xhe-s?oR7TU2^2tjQGK{Na@!s)5}sm6fP20fb&Yd_S@jN{E}hU|N1O3#gj7U- z%9xJRnW8`jLtYefEQ2zHQ5sbmy1zz#A!)=Sw<>8E&nbfsnwW8ddRkXGlE_pCyWk;_ z&z^SU2X&}f=wghmAhNEYZ|(aKnmaYNUeQ8o$jb_1S)L~i$5o(;!#t07bFG7<5Adn z2+5~zG65X)Yq2-+LIx(Uk`#YBkZ`URMhq0%05=93h&^NS$}WN-2-Y9_q)iT`paq=S|FCaCHhM=+Qq5gLNeZSY%Ix`tCCLdXP`L`SKC4gm z@{!L-u$hp3<_Ni&fKr8K)Es)_k0UH9v;!ba;^?8+v9Pojq>Xt);T4ZY(gRX^^Ik7^ z11bG96Lu5b`qHrgLh9uiYU@RmPsDVQFN~kzajIsIa8XQ!e%LgWA5#@=j!*;NMpJ$| z)~0AEXw;p5d@7+e+jTe|bV2_zSO!3y{~Cwnrik7o!p-u;-~q+m61`@usKuNhkczRF zRRArSLNphRhlGAIBOIE;C~+7V8ZXJjxRuee{!0e$V0svqXHDLt`wUTr2D+#+%p_RC zX-Nnv-FPwSXa;r4YSPd=7Rb5Ckl?I{U^1WgyT{~9Y64~^hv=ptN445vkc?DTw&Jpu z*|NcCBIcrMEq>rL#dxCk8ox$R|C%+{I9x!~ADcirO&7gtk&0k`g~ikfNu|QjzJhKY zI04x{3`VffKb?#_(*sFjQRVr{<}6Af1_JOA$0LBXcmrYAnJao5ZVh!>Wt@ zKo3)Ve%%GA&BF6`KvSaxrw*^G>me{BjFj>RsVEOSqu*HL*z*#1hYMvowgETI8_L^e}!$ zkn|)dsiq*K4&e`&M?3E9uvO2>KWa+X{hH+^<-Sl|`5U3eqG=MRNKdGmMfh&?T?`dx zPQ^@ZYCH9|P5sICq*;|_0s@{BkK$IukjKdF``U$etUbdNBcss35>alP~C@AxAI&Ys#($pNl-C4Tn0!<+-AE zV0dIy5-p5OX{W0MmBPKFnx369KGo*Xs%+=&%X=nb+H86}+oFJ+>C2X8ZAcJA_01Sb zb=@Kujt$vS4t-GPbMw9;KtFG6!SP9&cM;2};5ImqCS)fy?$CcZFL4r$s0Jn!p}Ayf zm{poC5p7|ZdIfFTX6UoJ_Y>atnP5@Ylue3C*LuUqP*%Hu)n;}6?gi;K-@;50_0jNR zUJI2cT~p)Ft!ln{-9+7)<$7BGstlOL6IuM@&Xz$Di+vbm9SiNq?3Xcjxs(r>nW5{%?-C)x{vNvN`nFl^aVsewYwcad zpatzNEwYuoR2SMXW{en3PD;vS`Vj$?peDtckfj1Ol^W{LpZjqCG3`fH&-Glc0HV|S zhJg~Ub?jY%4Pa`<>Xr)Ke)x-n{`FUEc3U;HG)z{sk5cTnnxK!0>}()vemi-u85A`5u``4*q4_P=#$pGXKi>pn&B#f9zQiwtt5-)!WMZ8mZumMu9 z4ET^SS}q1feQWZ54&%au!4ialt|4#sD3@#Lsm|()CDWYh@KtwFM?Ba7l_Q~*guvUB9cT9E(+Ym9?YNCiS99h^n*@+6R_ z3z!5}Ja@#`Q1VtwqSJQK!66$vcw@WCSUQm%mHFk7O*=*wFUoY%egonDGlJD2P!Q@- z(~xs-VS}cqd@;TOWRxScyl^otVTeJbElAm#=RE2H{-lExIiAQL2W7}}$BuDcr^r*K zY)D{@3rLMX_v6^V0tnnFVJO&xs(Eec+A=YDk-+B5y)q*v@HmF~@>u!qY%ypfDH;kO zd7ln3)!_ypb*AOfAK)`yrg-BAP?AR4IylW%T^UhXQ5_;k!Q7$MhSxxxpj(#NYjY7( zsK3jd(4DjRRP3!%fnHvF;oOiXN@Ns{^f_&0SAJVMO-(ghL*oE#=;K@a`qmw~4gUQz z>T;9`h;LWH1(-z6yhU!G(#dO+m&tKkEEj^EKSBlvOTMI>PH>cknVsI#<2Ox3vWj#c z{IHhLlFF4_MDa2xiB5!j&Wbwa(AvOw`hqKYV?Dgx6Te64m`1YD7lxL+&`6rN3Y19W z=E)+#2HS~B4Ye9<5DyoiQ?eS`7HIrgY-GgyF5KJuM?13kMKL^% zaJI_7N9z-GagO!O9|raiXB4)`Him^|b$bMhihoT3VeVE}9}5B~Ue&K3#)m4cfJ08 z-MyarAz;3%XMEqUx~{iR-|FY@&qHCy*UOREcX-#!;);Rip6>S%`IhS^e(z3H`*Kv~ z^x?Fp8JmH?uZ%ebeaya3Y{neJ|1_5Q-de3?~@+-10BVrtFVJdg+;J2;ftoj3Sx&4M~*D!_Sd&k z_2&U_d>z<__O`@8bf4C)%x~Kdhxh#@_UoP`rS7CZr8O(ta4o)FZx>Hz*Xp_6 zk7G+ySV2Sr6qLb`Y9VAA2au60Jzk(3-b1(N*CXH2J<0Uf@+)f#KMxh9=Xz2cpE~Wo zs_A$!7356=XXUt-13&L{TvrAzdzf3q&z|I<&AOOjulT2J@cc^<+$f8hx&i&3pkBw#fhFUIg`NFd^TCJ=f#pbG z(m)b+Z|xk}*s?SQ3awj@9zZR7afSRD-&oCYgCZ-A% zUfX0QgtEiB&PRx7%}3%etD=Z(=Wf{#rjP~E!-z-_{1Z@@5gL~p3Z_ps64TAZS+6%6 zKgM2oosvh4slyVsQ|v->l6;EJ#Ai<5>kbSHkhEeJ^m2^EGYQ6U1(2)*({j@DAjHiOt)V!l0R7Mdnt?&8J}oBRK&g@w+__h{ z+Sz4~S3G~Jvt0B7K$SIpEUB3In)PRb<=0^u#ulXz_`$%fJ{9lvVLk$E}HM zRYxBLgkA>NGkfv-?^zQgMF77k#7D;CIL5&9V#ncVQ%1qOF;aEOd>9qL4R1#s4DxO}ef%jI&r0OsSVyMA-^~6xuitl$V#zDsr z=ACP#?8N4oV@XY`@3+U#vSE-oKy8UtMC?HR5Zqh z!4?1@7DER~j2sxtqRwC9jPD7Oi6@`2k6dXPuGl2%hUuv_&F?ItXlp z@7tm{YB*$dI=9;JWWv1KKAcF%lLMbDX*4;AOtllNsVHGQs;uI0@tSDUD;f0wBz|MV zTyaDVK(h$tJ3amVq&DQZeO5)xe(C6NR@v@0g(_kELu7CVPr^*j|GWt^5c`}r@-v8B z1HUaBP1@`2@>(v{nBQ(|To0bU5`qu}o2VzP1!*NPp&`(eu-t^h?P&HF8r8`q%3~cP zyp2_L40A4aPIy`R#9SrkfibUUq%TKD$-x>nDDXNLA#bgY2drDuj4n3o$XP^eTfFAd zz8t<6TLWaE5a1fQYVyR5LtqEZrfz+D*^%*G=#Y>@yJ?1X^2g`?9#`tec{PU^_JwXB zavo_UB7qJ()buLmRl=W0w#JyH3K-Fkj)ulCO|LIqQ9j_ad#sDeAxm^OLAhfgx~BM~ zm3b(7cdI}{vZ8gni80bF>!PYt^7cc#R~D&*=zF5F>jqFxf?}0tv2Z@M4l9fOtgPM(-ZXFul}`B7`ue-#%=gtj`B5 zi0KC1bRqFD1?3?x2Zr$Oib7{DpO8Njrd8OSK7dr@(L*UepL!jMok;9}Vi?IkOV0d*<)8!Iwz!gT` zQ&Kj-GREYPkgDdCUf6z(0s6FMf*wE)?FLGkdH@8mNi}I$Mn>b2q)!5pXW@nBB~B%t z;)oD70Y9V3AdgMWK`(;6SG^#|Q|8iBNG7-!g6g|L8~#ZkSzs5$z+oT2y~rs#Q-h8j ze$YA}v*mUye?5^T^1LN+KiRzeve$f7Z1kwNfd?T)h`RsX=a(?yUE2?ogIufIP93j{ zGThIiu=UL3Y1t-z3i7ALe>X1L{YVb^bJFKmaS_q`cOI)yXr0H^i+i3S2p6Re#^aNh z_NOu;AS%i1S{>Gs)2;gI_(>iz0)Z2knUaFUX`^2QpV=1mLOI(y@DK?2vBB9qPs9~G ziZYHD2XNYG7!Ze?FHP&lVyt!U!UQ)n=oe64OhXof;HDGMFrL?bh$D-ZVKLf$)hZ7 zM9QQaCnVv6aC+h{^OEynBXS6924-?ZSJn^M(S-|0lLf$WUCs$vYkm8XYsTwaPx3CC zk_jff7uyHW%XG#DIj>=M_IZ-l57aR2(t zWD`J@7M<txTq3HwyC!yW9{c69WFI*S$Bc zKtmsdYe&X3axH~;ossZq7w79B#sRDEcXC|6}8_TlkQ<7Q-? z8O4VlLp8nDcAyt_6e2>iTC4_B+M2C>5~+TN?1&AU~hEwO2nij6Vqk@ zRoh|`mF7wI?gYNDdypn zIx!Cbwjiy5aR#|01kL)nuzEWh#nKkdr;`E$|2V*st7b6o?-2jF!PF|igapSIe(j2M zuQJJ19Ic?$=70~hndO=cw|QVtNhHPgE*U*)9YzxReo{Ho=bftbZ(# zR`-;ALJhAP9VY~ka5k+6a?Giqg8(R-%0k!Y3&5Ibcr{6ZFA!1pLs6MVP0k+fGlM42 zOnbeDgr3b9=L~VGRnt=B90{je5IHA17N`T6<4psiP0ZRK^?A`+K+8i-t|Pg9$zN_O z7?e9QuTKv5LJ-{(qM|zP7vT#aawKeuA$F`I=(GpV9tWVbmkE~!c00yZK;KfEOamwj zkZ}NoiAt{E`sXui5L7)8YJy`OKy2ao02Jd$Z!y9OZEJ&Sm@h$SEMnD5{Zg_S<&Md? zDdZ*4kT1aaW%_9p7k0O)GH}Wu8I7$uVB9;Bno!yVt}zOC17V6gI3*I!XCq_afBZsF zobzzRBe;-wD!x#SLQK23v^PUup(XF?B18$D8<+hHIK5tk1m~Po=-ygZm10}voX6oQ1aF~oTE`sZ z@lOLQSw*u2qpRRo|0oQfUH0VMfun>qf@qGAG_g@i7;N6CvIOP+sPZgyS+pAXEsD}$ z+FcUbiyY5^{i9_WIeFsivWV5=$=bWkxS&6Z)UBcu3qZ?Q-A!@Xw`GrCCt3!|6LQ>6(w*Lvfy5#PD=lvGEo+FsB z6zTD*+lxTx$F3B*6-ja8*eevaNPJ&h`&e5WFuq`Xzjap*Rxd>4&%DsVhTE94^!~

l8MqP`FvwRWE3!O6{&(8yGhuc+3neA)8%2kDrYus<@YbsEux@skJ zu7uqDY&FPL<5?5JF9Z?RK`MGNM>NGZI~Q_~yXcwhpLEl0o3hBAFvCLQRLF@(y%nq$ ziR>19L0E~+?umK00y24~;TFN0jy+{1PVNXZ?;B3PMIrK&2~b6wkG3H;GelQgi9fvaJ-xX;);L_4qJ zVoFF@bkm!}@4n;GXJ7WOJ1%Cr`V&E?70$)%g4W%BiSqd+XA%sDR0|q(cM%l5R!30@ z18!3qrxH_=xB#$)a=G%G*Y;zXf<*L>;}Gt#;@PqSiRz!aof!DM&Hy4>j%Y|bzL0q} zXbY2*r0*+ch*!&JG_lVQUX=Oki>M(N-it*OtIM}ZMZM*o!P)E;;-8(({q}RGY5IkT zmM1#cMI4@mRx-M?ER=G-d?5|l%!uL^xu~f?c|Ozo8;Mg;37RVbxe!F+;fcUS=uBAI zWgg&HuDTWxC5!AR}qNmWNK)Ix7LEz@q-fn|WI`K*QmzEZAeJjI|r6)(#l(lRhgc zfx3Fj%I47X20@})uwPY-*r%UD9Sg&hS)CCBrxuA>A`0jlX%tW%O^7ABdGMk@B%d{{4`Y{R)LK5JiZka=ye`J-H=v=x_rZT|CHeOlh5vLBgk!U6u{LE3+8NdNcNUJk~8CTsmq57Klr9dTL^eYSrwMo?4Q zkb=yaXHjZ4(+#JVCopZ3@W(}^QJ^ZqbyZ7!d0V=Gr-3#DPSTlDbWDcgI)npad3$;9 zaD0D_RCHX8C91Au_xRjCPQ=vY@LlHgc-6hlKyhu|UcqpUa9^Sb$IWhfX=HETTpg97 zUi-ASY4i5z-2S+&oqZ;_C#+rFxw}{R>G&(Rczh-}eJ9%f@*=v(_wIT&c|zf}-1`RN zb#HgMqIY@oHnN`Mw@>_5tD#f()$a26G<439gJD8uPS7lDqnU)^^6MXT$<1_jHElO2 z=a7Ltz;EU;R3pkm zY~PoGQ0bRE@on1JUYZ- zj_MPj>Zu;Yf&ikdpXH9=Jxc-_qB$$AwV2Ymt!~bYVgrQfcK|D}}3qP!-&X>)e9M2zVGTfH7Pp zmK7M#UIKKgt-#QTBr7$bKk-*HMlLLdId~&?tFjGEjEWPZwe(VFn-e9Y7r*GO_8|Qm;jm+*k}ltFol+Q8D{XM zw^tm8kVldz>c$l*o}@44yGC6}#(?3~cfJE&w>&DJCt@dq(xlbeaFXQ@N)kOHz$Deu zhCDN%a}i3?0D?XS1xS2ctppY!Og=E7QDq5#QVSwddJ1nXOGB;c9uPAbTnOgEF(bU; zzEFyd7o25h+vVi>99^(nLh2vMSYXwDNq8bq#}w%Wu@@0ylCGde8H|cS92A*ap>ah& zp^_*yN4vg72mQG-&mXUR=@h1Nt_t&Waa zfkf5>S&eE%dS?V=w92%oE`(IUIzq~1guNC;Ypw8lVxG;%{g!2f zs!{?_wz#(_$^$^NZG*5g77I=px#A zVR zvUfNBRxJ8$p9N`Kl_rS9JJZ1&ETfS$UlX|vcfV$^^x4i0JpJ}cr0#R7bK}h3m>OJ- zg*I~^6wEh@D6R<#g^#-ayP%gnr~TJM=jQKI%B|!%Y8n|+)(6k7mEjeB+4Ig%psHSH zGhk+tt^xm5(A^bJ`!_zwa#@WYqHq!Vr+m~5XR|cNX^WOpZTrIdkXLu`TqYz1=9)=U z!tB3FqGC{uN0Rkb%i54a4wv}U&?Vc#RY(9gjH+aMiKgmGUm`>v z%H+f{k>FGe0>)LA0>_OINs+bW=i`5AP79DuBE?}Yn&M^VF_0z8dm~rOV>Ym+*sD%k zJCfK^4oVqTj3%u!F3z&JY5j$u%3556XA?2U0uPY;X?ekNK>wsB+R>y8kr=@uw230+ zgU6p4p0FxPHZwaSu z#8QonCzawo;36O+7<9@K<3m=v?W8(IAVe4hvNraXL~~sdl~=>XJ5SlG+giDY|a-cRtCbY@jG$z67C z93Vn=kR&)Ced^;gGfj!tKL>(0m8L}~XiWlY;hOZIP{&jJx!y0V>5G|r78V3E2yjgR(H zYJzabZDU4LFIHNQ5qm?R56Q<23Q2d7XHO(Upu*?G34kmPkCVXKqK>Kf!ZvMQ&Y~da!*rlW(dk5~fIWA!0b`8aXt5TSSXfKfyCAP16ne zp9J97Y23_BLdSjAiCshCb(2js`5y5Wx+mdy3&%r6(2wyq%=TLR6SW~KZfm@>_61kf z`6}mx0@Q2D>~HI|Yh=k-kd_%>|4w#Xx>9e^n#8+uLPJ)jwnII`;n0YTD4@RxkzwLl z?Yog)2$lV2V5xRnCfJ>4D%~x>^k!ofX*r-oYJ@57XS=n9e6Mz46&T)!rcJY7rX8+p zs2k_7rC>s&Q+FpHx5Jn*$ge=Hbu;&|XR~D&{v8ee?Ecz5&aFAv4{nf6W#dlHxzYd? zb;a58DVh<(yujaP%Pm{Ws<8#O_nyDS^P-+?X!@#3F+B^(zJOWk_YOJU!Z7}vfEINE zd^Cn&ta@h1h~9|%-cioL)X^DiH%WPlr8J0K(^a3oK7xLD7_{%29sP-)XtS=Pu#oE2 zJtgF&YoRv;!p>YX^(x*{yw zN8Cr>F1m%WodEq$g%%wo?l#MwPJhlvQSjsKErMPj902qt4>nMnz;=4`-3sNnL6@`} z+@?b%M&h2I-fqx)Qd(RUj|oO8j5VeTC+Ly|IJzTVZuUA3 zE=4d!CcL0pxMQkfVe0+OT(AIJCN)7weeDg{>{KHfN)hyGc*D>b_Z9bGp$DftHT{(P zq;kIl72r04-Z*+CqxxsA*sp)r*s}RF44XPS-_N-EUT4YzmB|jR7}0cx-NDT$H++Ka}}q}1wHlBg|7U;nf5j#LX0!i~T=I{#e--iHxrCGPpXT`g8JAq@YR3M9 zOLBkBt#gE+`?snNaoa^{ggW*K0$pcbf>#yDq6koH=kOeCgY#}|E*zl6YC4#b5cnuu z8CYtW`F=3;n^OM;JEO<5{n6P62!@ffw`0Jd2OSO- zx^m5?USr_)IRO^(OK;mINB6gW>sM`CdvG5lyUzC2H8%avJGOmqfdM`M^h+u_d-j|_ zJ$*?bwo^IuO4-9p;HM5_2Oe|T>1)`1w>v+cUXQ5o0GWsdMEkiF*svh3z=*OEC}`6G zLE;w=P-A@CH@DkevKAPM+i_TRKrg_g5Ju z-$XeyjoPbsVIWXfaFgQV+bFhnrLR}l0M0CzG`ae($DwIm(MI%2GTChQwgT1 zZB*4^S0SMX^1!T93mz!&=cr$QZ|cFxUjo~zdWmU@OoUJ?^?PW7ww)X{h)D8rAy<#e zjhtV}>f$h>KS6ZPf>mARml*qXxHhMw^s{&$F^2`3bWye4NUTFgDsa}u2G@r}dm>3< z#EQsrWq<_?8Hy0ySWpK)`W+S7@^s7|?i^_!!ILX)gN2V-d)zs=MR>SJOlcYhpRWeV zG@{_4uujG8Cs_i$5_+*`EXh^ErhTdzUP-t5GHYN?T zKyu<7B@%RQ)qgO?)<|?FNb5mV0x*c>N^*0ryh!iRGdmCQX%+WIR@c#pFeRQkz}}Bs zO7XIbnke)O&Oz5+xxrDH3=2}@>9P5z|Af)>TLQ_AJV-SM>1Pp?QG=_ef86$YFe1Cx znmJG+dQdk$Y%V%vT*#FuQ7iPjaTMroMCq#YbdvOD>Qq*<9#hnGaQC+A@*S{*u{XyB zyA=jm0rix$30r4L!$2r<><^zT2#RKF=0=KbQbAJH9aZu`Ksxd8E2cdqW`z14_!Ll_ zlhf{b?~rq;AEgm*ZV3}9rfJP6S}Ou~2{XUt|Vrb%b8rdQm=&TzWvjk{ML7i1I<#3{(?qQxNbCWIzm*QB0(UE{y z)6|Le4wN*PvJHT>5jj>qPn?Pkfz?V2>-I)KKbQCwU|1QXzRsh#)i9agCjK}a!G#;> zLfY1LRYM4e91Y*IL<)WR&rCu2QL1n?4xTzLSo9waB0ug1Li!m&1Y`b2a^$M8;ppt9v z?K{PU+Bwxg7ox9TrT~`9rYM!al$v0b9=**8R5`u%Mbrq!+1XT$LE1qva_;u;PT$L$owsk{(!CV`F2!X*-&dZ2J;! zc}n&H7d?+cvh_4yY5m2ZidKHMQhlrsaaiKIEnd6G;xuLj<)6^tQ^AMq&BRG8C7Pl=-wp^AeTcQ@sG^Q;m zbqVJj!adYBiPYn<*K-Q}SZ2;0A5ta9s|;5HevC2b_-^3y$gm7-2~pqC2_V917CaDt zk4f#kaF&Fyk{u%HPUKKcaq&%y0ZhhnbU&=^HMj_qQ!YzhWlwLdb=4&e7CDZT5gLpp z+QuPkYQ=wAMjt)~sfgoIwJ2>epbVs!wg1-4B=GpnZuwmYk~vhlmE+eSfpPk9BT{sDSF{8?B_wUoRiDQCuGGH`RP9j1 zee+51k9d;OTYpYLqw%R76JoO%#9Hpc-CJa-MYX(c@}eGTs!Z* z3l5xqM1MN>w0f7Y7&??Cv{bYT=IK~gHa z(1_J@H~DGP`^@sdNcV}{8W(m5jdLoH_%b#InRA4l<}nqn(t-7I5|64qMR z0fj(m4RUhcPYzcb2l1eq7E9JaHH}H16#jUBQVQ7xedP+vY@1>>>S~!JNU>xVHuWQO zIMXHEfhL^!UGGR@-GTF7S`>h)kSf=ab5uzb;0jGK1|&`GwPVc8rAP2uS%{)c{kW7+ zJt^k-g1Hl;&#hhUJ2WD)lDIY@qS3ID`DH1A)_{Ybn!?anxEXpV$NlL;%y(UqFH8YU zL`kUD_D?+^rq7Wr0{E*)7M!~#_btX1qTOC~HI)=>YdQKBUhJ?Y1ryf*6w#IoBx9U#^d)Q0Lu%%$;S8k4X)1blEP^1^r3 zjd)i6AMBk|kY;V0rPH=ED{b4hZQHhOqtdpm%FIgJwr$&)Q~x*7Gt)ia^xx43gNWHD zZ^W~Y-lP4jd#(Fgr`6zHz94QRA6Yz&M%yRz%ng!?O0S-hT!B!kN}#6S7%19H#go4- zo1_{htZ7?@egLxD8ANVUbd&9xYM_qBZocsT#EG598$hlWBkJb!;d?X>P-25NZYixb z4VZ)&`z`*_jWHAhJZ8~UIu|l_UI3n@g!4XaOSqk@xy_t1HqvKHr1w_NiC-2@eZhL; zBc+0ujLX$<^FBPB2-qBo`!085tq-G@ud~(n%cswFykWQQ_vy)} zZoW>;nl6uT-}j^bm!-$6$8Fuu`^(MB%k^isiy;w;Btjl@nks^koC4jIKSj1ZpM0HO zSx|zax8C|AsO3*V+Fh zdzjc5|3kUvMI9+S;ugfN>DrS=Xte0!5lUXb{s^~ENgyJ)s?5uNOCu6EeH%hKxYuVu zvHmO4cJZh1i>*go=D7=Jw_BsdVMCSrqqldz?%#O4pAYmf%iCQa^XR~Q-cPGHdr_%j z-`E2^;Jn=ELHj|jKQUqZ&-LJ!YhQ2PT7dCza;{XRw%_`2@yE9v8E{ndhSWOF|foIJj6_RlX*k2ha~ynPG*2=HvW9f0F! zg07Q~ws!H9_Cukc?^#=)LexK&)%al9{CK@RoL<&<$L;iRYn1Ro0f`}|7r2}3KAp*i4=KyL2h3giB9QLXiv6j@~0B3DneOIsprH^V~*}9G#|q6ht>Kg zyP@P#(#fp!=j>+-x_SrXmqKg*#2f!%qv#Muv5)oz^8-u%4q$>T>yfzWwxm~#fF|(~ zFQrP;k+GE-g$=~C>K!9Rz$0m|oX8;nrH|H%S=oK{y&6MOmjnU{RV_Fz5DnO!X6;HV znME?Wdli%Ftd=P#6xHY0*Iajn*sk0mD;rc-ACw;#MYUYC6`-}#W-5lc&fC;m%nN{F zqu}E}dfFuoFma3wUiHs1)#X=Fj+_WVkY-2FO~>-r8WsyQcFa$A#;ywkaS=sgQN7CJ z)pGiYAECo#XJ3LdNJ)`UW!o`s7E|DYn}yP6eu=vIO*q+;ep{<;S9Vt;om(xw$PVS` zMZqjh1?`FV2X;kTuAXtJT~;TAxLA$MA3~uCpe?5Ak6K`koamk*}p<+JI>aG{_HlB`jSdEuBFB<7SRw?xTW~sJ2cK z*!^4_QY0iPgQUy}NiaQ;(|l$rDOkOvNEO8@I=Ep({1QU9Vx>Y_tlE8H1;QmX285of zc_S=lCTW!%$uzN3TmHN*7UYpl0tEdyK`+O*j8k^bPi-V=c8oco>*t{Onp%nmkxSU| zGKcg}=^7Gbu%@#DHzN#QX;l~9BJu`7Dry#@q&9!3V`<5*M8q*kei9^8p;Umq14%RF zqz3Ci*!Wr|f$~Vwm5&cp(z0>V6@l{{{lF6Q!0I|>8T$QXDl1H}@|=$yIZJrgba1PE zSh;57J6W_l0rlOQ&|^`0n%Q!vUu?vCeTHGmXVFrA1|#rskQ{riM9Uz@WDS$=Z3$u= z#|*XDNnmKNTZfH97cO~G(R5-qy)P~yDk?*TiVzbRU(TvN66MOWy@WG+oN4?dQdrVd zc+3)t_uybsQZGd0=Q?kBMao7+2Z5;DlYDLDkEcrV%t~GJm6S<&0|UPb;)P5oIg$WE zA>@`sswfCGhIcV6XYVd%+ruga_$xfhucEyGeh-n>982WEE+3G7Rv9N)au-5RP0jmM zQ3*QUTWH}P@liZs;>1PdSiSLk?CSg9x*#hN?W_#8k|BScuH6m9Yt00oRE(@k7GRa* zA*4YbFGsIe&6o0)?@3%z5R;p>3NO~2Ni-Y8W7bh(G}qdD$4}9d9lD0v-G^@9T&%R< zb>C!cJ zU^N+w1|sZhN>Fb+fZ=yK67%nU2Py(|V@>h)GZ_ZJ1^dyCXPwTuV;%}ip93=FIJfcE zqSjw0Q6G-K6G!JO2tS+ygMW2ralQ41|ifYLN{j=AcoLrql#i&fL7`EsGgBGNPvP1L3}~2g@h?{0HB`QJ?$KiIidqH|ssSBi2$2F_ z5$`V>1ZXlwOa9qj1zBjMFdSv*M;|GH;z@SLhVmM8IN@(q93qmSiS#l>!trxOo#0hL zXhm@xNOCmZ>cm((?*3C(1;x^XpF|@p+1HG#CEYp(M1M%4IJYaNEfSQ4P8LYn{06=3)&@5Sve!G_>^tZx+Vg3r$d1Q0A=|sWycAOMPSV~SoG{!L7 zom#r3d{%LaD{@RMT@XQq{D0>3;>QcDN<@VIu=h=ay@mmN+Y{+{j>}ngXJo|J zy=+gQkP^;M6InfLCiW_L>~zOY4?O^WnOB_4;1zS&A1erZ*{qYySv==e@O@LVSRo*1 zR59Q%37d=2agV9N+K#DYotNw}sxl~7h|OGrwi4;zX27bG40YGO9t2QfAwZ@B2#SVg zF5Y*nYp5$3GbSm4od;$z`(?Wf2|LYKk3vQ|!M_mM+W!-%LM%YrlnHE%9<;*T(gf~> zs2rQBrH!Thl;}ML(n2jl7k<)&8c^^$$GniS_OlXe`wippjgnOCIG6-^gY$wubM?+x zl4D-jqd^OX>J`HA#VaQm55BR=T)@p^zE#AX-mvMxFG@tyfr3e=?A_vaMWziq?_AWc zZ4oQEzQQp5zJKSPJg>3Uq(QreCvIo#I11UIMIxG=k!2dSQ5I@dAhZIYW+TLMoPrDT zW!ZAPSu!<@G@22#6m}H9cNC_x2_{;lE>05?ZV@{n+*1$B}?4Sp|&rV=_75pF@z$7Y4Ta9I%zgnDA5$4Kc}{FgKhy|Nth>k>j+Fo zQ}o^oELIau>}~HFM#hfu-j+S-n|{j>_h?C`5@7$6_S5b|;Z{hZb^ob1xj1G!x}hM9t}+oCutEC0IzQA1=4A#&T$4IG)O3;h+5w{jK9Vt+5s@De;DOGRa}))EBr3 zf6AqZrb>(I9M$+3J_RSRtbN;1!D`qt)WoSVTzX|LPv{1((rv0&Qn~P@%LVq-t4WT_ zP(z2UQBE^~r{^rqd=`PwD-R`d<5#F-N^9AT{^TZ4=NPR1l)jKvMgYi zE<+UC)Uaa|m!G%qha+_B`Py z)@)Fz=enbKSy>9N3EPG4s^Dm=d=v6T-6LsN;cO8{uQaC(siRL$T6NBtrC6y&KzyxK zS+m*S{*D^Ct+h0Dkm7nXpF2IPKnKCLRoLl@=7O9W^Q!o({m$6sj| z1Qgy`D}lji7R!i;s}1tv`csY;o_Q{qtMG@OQ=B=6iMQ46H4aoKHGYYy?C85R?DfXT zo$24L;DW{oQW&Htk*>wHWD+A?9soT>NQPuEBb^`^rbTkAm$vSWzjd|S_D&=TI}r!Z z*o1%5^q0^Acuf$lo?4+HY&oHw-;0!th8U=xxHeU6!$~}y2;Nku9LNp`J=JCu%w9G= zx)!h*b5S`5a#%exf-v4+!T?vUAOCv#1Pip5IDEnXV55T2xftA>e=TZPjF~bcoMP=w zH-1NB5zaxW({!dexl%T2YJ^-{^b~n6#roXG0pKaw&dTbsVicoY%A{M}AET;Okd6wy zRC#~%UH@YOHlCThMEak!ZMp1ff1cfWG+ zjb-_g4Jxk{Z%8Sv47~DCy`sTs%+Uk_mx6pQoF=aMHg#-xP0Q6LNKZyV03VajYz9x@T>!`~h8^89`J-PHfLEI1{lD84 z{T~*Z{~apz|1dH5PmBlt3RwPoWP|J)V*Z{qmEV;NR`1q zixN9**gfJv=SPnQZjF-IJ(_0K*mu{!u^FEkmOq({TTt-9Etm__PYqRWD-B(I4kqEd zySlD*ySSYWL*I^FE?MxZ|vX4cZcQlBq-W%&Xbc${F-)eMZSe~J{LB>PH!GB zcWc9HyT9-38&RNCgBx_o>mc*-DHf)!MVt^1@qo@VX`Vt zlZ$TLUS>a?viOEy{8;?ph7EA^7TN+3+32Z4=8Y+taZsykf3N$I($=aAcwXl43s9p@t>vNfH7^9nX85+~4H$ zP`MqVheyMmj7B&-n?88%H0_)CBh3UXbPn2ah(E;OEg)dDwOGsH59j&GzFzDje}Tql zjD{*CT;I@Lsit+WaP#P;>E(L!B8pGjWf@;2Et>do=gZNznH*F^%+vb|!Nj0TJutgt ztvitrWN!|z_P6g<`KjAuVJvW;Vt5lZak(iv|l#&tH#!y@st)xrucK;IdH1^7iJt+%j356$>6dAx&Ir8#gmfx0+ zDPXkpR4*W!>@zGcD6?$WSgav(&4J#(>r&Q8?Xv)OUJ-9>?;_eC77&o)x~zfF4nY6UL|iq2gu}{Jiyp?*MJQ z7fo8Nn~}RRe zCe$7mNX(}KW(7Yp93SEw`!^>h7>`ZI96zfLgDyGMC9mc|>JEV^du7YRU}13X!N5Kc z4?9H>gDoCteUx)4D5lGvNoQc7Xir97-ag)#tK!NH`Q$1s`=7PxiMdu#dsqoFn~P@c zG%+ergmlkBW17C2>3$VR=|T^_>A^;*T_99Xgy1z+`+02~=w-qP0)lL?W)pG^(yFKf zVmxVbbm^nQ71InQ+KdGVhkPx>A2=gW`|<|Bo!&`GP(QFypstG=Y@8cz1*s~HTuC7D zvz7=z>}ri@yYSTkGj;|>et=-?IQE2)Q7aL2=1yv2Rg;b>+cu+ zY&91igC;^{nK7Wnn$=_`B%47xGwY~RXj5m#>u0SOKQvT`K{(#GRW-9tnot|UnvAcwm{R3L|yhXkA^R#;%{99q{N#+qwG-l2F)?%a*;^ZIl}yjO-;7S6f+L5h{Vny00VcjNP1aJUA<@JNbiP=$l>hUe_o^EHqAJ z68;ZPHdn~g(0*4DT@2`kkmrs!X0l$%mV8l~ta_D}hW$&fv0vO+U9<8~ICqt#ebi{} zz*sfG=kC`J5YSQna`8BFo+|PTS;Mm%hgBI8@0FnB_Lt5}+kFL>f;k4`d~*=h)yj`= zV*#!8r-u5_HY`x?p)#+ikcF{q2CQnRnYUDA9YBu<;tooY2&yILKFyXBB7E(k@BD(8 z_>J&4>o7Mh@usJ@pGef_rzwn!by&M&_8SfCv8v)`;hN&{;Ev-!9QpXABdm;w@UqY2 zIYU*W(wtW+=Tu>`OQeNciNBe^H+G8Y!E_Flse-ba?>ayTnpHjVMjRMnIcSPBz@ij? zv}pr&iMBG1IHZMUc|zXB@00&|CA|RUs7<<9v!n*+sLeT>qvzKT)U(=UJLj%k-HMIN z0aw|1k>u1z-i?mkMvUEzU`q4IM`LXk@;DlyL1c}|>5+aRU5<_qFN^!Lli~-~g|+ov z+C_cUk}6yVcX0;!aq4zKP<{ILyK|##Goh3G;Uko?29bQimPPZC3|3`^IMXpm)y4-V zCfDbEA;saxqz~QDj~s4!ez7Kr=W0wFg0(hdXnSBtsGTXY{)fbMZ^6tmW;XFB*-C}J zge97V?93y4wF<_ucza{mE2e3Il5s|v0gh8%tqobcFFpcf-|-~3>t)-zW3luSeC3!w zle!@3IT$SQ_}m@*dEuZC1TE&1ff6~Mupo?5A~E+{wh2 zJr-)}wV{I5N#Ewb-4CB?KggcFogcqY^(gxGkB~r+Rd3US;-%`bUK3(+ z=SsCB0@WL9$7{A45s7yqctEOSgCQVzvQvtf=lo$eZc42F*3{Fe+dut9*~HEn6SMGm zAOe~)&BciHl#WdufIbUV$-M{Xes_Fxp15DzF+{$q@GEghxuV({9AK*ptPj~299OI} zr|~w%WHnaCj!PrMY)vy@CljPrdz__?TZ4|vQJ1_;Ehg1uJXXP*bb`s`FhDZZiV{Wh zLY7je+CqPxiNm93-Y|?^t1`(tOD}(5h{O>%hMR#1US0JPW4~p&PBzrd_Uza3K23}~ zLV!BjC4#ZlVoGj>b!w+7&?r@7t5wT{a>kUJ0QT7tm`9>eeco5E>Yc+3! z3Ss|Gj!I6u8QkR@$#R>ZSN!D|KY~2{B`%W@L=idM*ucc7)sUe$$z- z#7Aa~QfAr45G;GBDbX4!XDh7B{J}`i&7)Al$V@i6Or{c*`!d;{GNu9Pht;uK3Qh9+ zZ8O$$ArGF##Ab8)`5B$R0}ok?kqPr(C5ph zb~ulos<=f#s@8pfj+Kk%_eg1yky!dUquz8Ze%(|crPI~~S-V-IRv~CN&uCgH6`z%z zcFfm0GZrNF1&K6&zN+*jYP|QOoq?`?!BFY1*4?61{zS~uV&fgU$ySx19(Ls(tr+4T z6br1=UfU@df{5wr%o$r^?~S_9@at zV`I4)c0z@ZWmGc=nDCaMvfcz0VYJh*+m9F&g<=M&iSVG~<&8YV#-kvM?P*cibmE?l zGU320R~dWf%uE6gP7Xo-iK>dMiescU#kfx~;n1Zv&R9gddrOi!{E){r@w8fr2#t2i zH;LnWipT5_3I63(f)q_aSWELOTe6_;AgZaEWUlVRQWz-*A72J|I0Nh;?zLwO)Z$2Unc_@XuYb#xZ4E!m>al5MFEFm=>C6DO#12`y{pMhv5K_{E98>p*3f5@&J zC}Xde2#^|oJhh_8%HvqI$;#ksUI#%gJ>u1GL(jmA96<7Hdd>9)hg8m&v4o~foUh0W zUBKSa7-W0{DKfc`&B)5y4H1otB*wgZNe~Q}Kqd6(S4>%;qf8_WtS+=4A&teYLOmJB zHdMOF+&Atiy)>YjiZM}?Rb*z^xmh6_W!_lJXK}UN?65slsC%)m)6@;QdE*75b!h8t z_Jc52GOyk``AYr`eHeY?(^g+Z=K?})L0TRU89J-Mt)Cg@4P&twKA0@98P!ZKaOyO)XY0g9chF{ zuD2MPf>~$BkP}7XH^GdpV>g7xPZN1xBR;Sph(@lk0}ntbWMGw3b6Z$PWqg3ZahB}{ zM>*RKM#K-x)QoD(U{@~%MbA!oF(z!E85-n%KqM5(|4~O`;l+8NtH0-U##I$ODu?JqMKas5bn4_1}nBfp4skxpnXHuo z?AW*#R(B@kryewvDd6AFA^xTH`TuW?{-dSW37_MiRAI6H4Swyfv;RKK`ETiLrvFen zdr^DS?gtx^-?Co4%v`}xFS8l@2T~1s zE@HK{n@9jv&z~G^z9DqPdCt*KYt8T zW+5g@6%*gwcDuI5*2U?4Sdz_eW2;(MyVlvsr~9|&IceF9cJ1?ZGv4yGTDR}3rq!qB zk25|xU#psKJs+>mH;czhcsez;EC_^ffa>HJ*OI${YPyW`Zxjf6LO|WqooEgHFI8R>DDi@xJjQ@%Jz?%n_NHsz$*21obVV`pg2f z(>MRCpxyzYY#h$FH_JUInr2?`TV!6uX|;ULDuy)z@ZI2sAIiu0rd-x};^I&$C{$78 z>RyBi*NI1^5HxfV0NI!u9JClXkZ08;+NR`KuolisOLDP5lF8Iuw9?;ublo7vSii_= zahLKCX^4z+kuQiAkr1af=qvpyGob$&Ib?;(g>ye&YgHN3j4^s%f#Zi9JhEJhyr)sp zqba?02@4z1cRC|;JshhtpO)Rhr`hvaXr}YgqptvRl_bz_h_Yz#>(ZO{&{@{g0>mG}Ur49>*v0VA=B;XH1h`0H0gI24_LxLv{NjmAT8|_|^+lQ~@M00$DFabF*Ng;y zGdaEJ+RS-wc6^gD8EbC3EuOe!1{2|~%z>L3EKwj?z-Is=eV2^a%{ICuEXEkwLhBWUN%xskV`f~@ls`5e<(b?h} zow2N_5KKO1PmWIPYjTS&5EQvBpe3u%d_gJcPXX@z-XwHlm8K}CL z063_FtRXG)ePrmKI?X!0@w1fR4k?Cs3K=QNW{h1!*j-@q<~z|j^e7nYMt3;Cdc+xr zOrRcamR&TCU+xRU`DzeFS_vaDS}P&r*) z2u5#e;GnmMY4cS2weI_S`$Ur5=ez^Pl20hjce4OFw5QJ7B5STUWzkHw%7$1Xk{06^ zdA5s5x(K9WcJ|S@hYYSXraKep5&O!I0`3;qn;_xzGt<=J4!`*q&3*+M0 zSirfm;11o@-?5X6m;nP18GX^G}EswI?BDuA$f6L5u@$&DtS#*%tps2IOU31PZmx9NrN};6X5OZNyp)!(gStTOt%0adE~(_8o{*Yx>zNrHVKf`| z$}cZb9jg)}eTq^|^N^Fu@bKvsuY4+5g~4YuPa+YP+;hU87d?5J!MfUGYU&QXynZ?{ zD$x3=_lxgq_Df#HFdr#ZsbCrzjYd`Q9!S`{4AFqH-UAMi>-9wV{2j-sW}SL6Gj@Om zmFPZ*em14>-Wyl;8ttsxZF*Z}l}xp|hpO%}cCY_NPLM6UYy@Z`X;%nPmU!tKQ$rvmMH>PZ7>N&UEJ#-OJX0DgdjY%s{x6d>bVm~+Y@XSE!B zPKWl|ra##MY#LMjmN|ElK7*t7l!GA@v@*JoH8>)Aj=>zco(5XI@pJi38c&TNEhUJH z5;vBQ$fs(&+P3p~6mnDGN(E9Jkp2wJNPf#4d8(xF^Y*8-^MuG`wq=Eq9hbc!OV6+4 z^sGnk)#I{r_IiB%AgG)TB#?#MO6b8#M&nhwilslFM%xWDRt_;Ps1BKWH!-cvm#Z1o zr#NL6qQuieHcG)dCMX*{)q`CoObkwWw3)ZCwus@>+C75>YWIlIl(}3hX{1D28xE}pT4}`@^Vr2LCTcvnfxk;1YEG(- z6XjiCyDlp5f+6#hO8u;PVg_NpK5PjH(ZZ@4z-&1Vij0!c~wFwE9NK#b{LU zvO5iZVgi5O45g#UQ`c%Jn(q==J-FoM}f6l<$uxMd6s`gDcN;ZKqWoKRGouADsr0T-fX%MVN?HX(OCdC2Q%`I z8T-AnVRQi>UB4056V4|V3Zj|1j)0bz52x@)J;Dp1?K{$cMNtN*cL={nKlo(uLp7y6 zVHGlaJjC(mr4#AuR|)17bS-OQ@J(xn<71PqN`y7LH8P=}VlFl*)$_5e9?!_0<#4wW z7j-AM(sobe;PNZuWecFcuUPYTd{=*-)A?;#e?7miADu0R!DqHSU*^`dZ2NiN*Sfjcv0n3TNKAsR zkr!NUs*Z6j3~mJyBE9h{JbVM^oF^dty%hR?383@8BeBgu|If70-wB6j+&W0hYsPJ1M0WZ>#B}-chH|{?-K^*~bo@}j`_O$YPe=iz z4Zw|9)h0iYU_4Ud2{>~G4^6~S?5DXvp1>!s8)?8z+~qx*vW4;(Sz zfvaK4yn^=E2QK;bqqa*k!ppRxdAr|KvAMV8X;-xFJph7i+TgH#zdUUJ;qTSH3xMan z4dd0iU7jZb>>8wO(t7|TkP!^zXZx{z?o&S-afj-OnY+4ehjFuiT7E#@OJ8aJwM94*0rhGd&E~d?+w7OM5_wRxt!hDBBj6LO%;hu z$X^jMVumDiV(mSy$(ApW#fEvwKKA7^ja#YHH^j^!>O?g*^*lya^#KgcGfCA=uTRpa z&bQyno>X@zII|e&w;e*EB*mY+_eT3d+}q>{05f-Q{EY-09sS5-FEP9mmRGXOYhFFP@0ZGK^nK_}f z$#d9v8wP)nyZc(2jSLZgFW{Gb!6ks#=3w-OHv*iw6R_Cl8S>`$6EY#|*}hznO&aDDch8y^%EII!9{Y5R5Y=>3u1nmxxqP172QXk$PwHHF+a{NfU5&3CX6 zAw{!P*4tQ3lH+pZa@u+13Q*_RoTzd~277)*t}Qk--|0jF(51whfKv#Ic>vvyGX znoz3~6^=mF^Q|yf9+FTTCUh*2e^#?u$jPZFj9r>t`!r0>MzneQ1FKY@@uLNfsQ{i@ zgV8GII?*mO&DK{+yL^)KztD!BFC5t|jN%B$B#$RlV8+^6I zrB(FW$rh8;r^0|WTZ2wDfR-;<#=Kq?#pC7{^YT$L6vkjMDEyPEiKvUM9F&mm61+hz zpA9%<6A05=7@#3C%nVVEWF`!O#^x%??F=Zx<#~V$m6v*`Ln@8+lBU*Eq11IoI>?s{ z$jHPkG~tX#?NliIbT|xVG(Mqc)ob$VDdWBH?wr8Ry>%^U@-1?f=Hiz%$0^7wX|m;7 zg_)geB+jG6tu&VpZbVnXf^BN`X>q=zS!rZD>}pBtnNX9dCYv3wobq2Cbx|<~SJ4+F z@jXn=s$M+7M4iCEvJw54mi#M?4VldhXQTwmsw;t)L#(5ZtYIifAYhhy6jSRCs$eMS zV8|=zeMiSmg?;=9i4bP1jSeEMZ$s=&09_g#O~H`r@FB@!+U2dSjCp^-mTM5S(xhty zz0SBoYCBC<I$nu|2;hFty)BqYoH}ON1Zx?a%bhjui?SfPps%{Tnu1FHzEh^;hQ5j zLv5?6v2*0ee-;AFnwzVIQ|WD6Q*cp*4Ye{5fHH?4eb)?wZRbsDT01}d+&J}_WyAHx zc`^SSlx&_xIDf*wG-_Vu*wkbmVBJSS0Av7L-EhpLV7|+FRk@un-aeYDBjTh!?DUH@ zmrGk%IBww}d`n$@tdvk&w^JL0G?zw8Q)vT^*jG1iT7Z_l6+$?+IZYJOV)leH@PQw% z&%hSXl!KC_|Af39Q3?EKGLPVJ304FMc@**}AJY#*)5UVmY#w)opd>Hc0Y~Snm2Aqw zV4`ZeqH&zk@!7z57;dVxfoVB9kydTi03Xa&Z`QEMeb6mH4{NG9W}L6)ncn`$c{mha zn)qXm-si_U1`jco4 zeWFUBp{TL^!xs8&+Hb_fuC+4~1f#WXSLR6(0KCAq|$7 zVC$X!kzqIEH;SyvTG6(cnoF%1+qiH|sI6Mu#1K^`^)~Izu9csRcP5Urr$|U=2Rg)c zw;=!6$CjcIWR;IsWUsagmn?_k2l_!GPE|x2F%C(t5+6K9%{twf*}aM3m?YFg%9QLtrcF?fTgO>T z5?%ThYhiz8;^wLj>$~0E>X%#-6@BToAR&+t$Zivv2dfhg*I^5M4G;v|=`&g~@s^ zJKgw%C{etu@-7~^B(m%!Gj4+)rNUBtn&AfEaql+w+JWXR_>iQRLi@P+h(_qol39%q z$m^yJlReF^sgEnxbJn_yZ)K+f4S1%l8--`7r;Z#hlQ21GEGJ5Rl}a5 z_swL?GZx#K$j%FWX-zyNu!G}Op$eIIJ@57y?;7+U_;m?!O=Wa~^C?mzVGq@^nmvBD z@!5V2H(yS;QC?*?N&G1hqc;20CCsOu6zsyzF3QkKjxhS+(zN(CWPVBoA==@mAl+ftX7UPFP z`jh*yAgg+=#m>)-u`Tebt0cKF^0g1zLnyg#JdMf5Nb|L4zpl3TdXJ~8tJiH^J7f|_ z*>sN)qfxlWE^RydNa6+!F5terDgqcLLiu(ZYwON}jC)KSK;+?{#NxX=22W-Eg0P27 zSeBzVVjzC#;f`f;SD1%g&}jZNIFTb58D$pV9pCYUyFkUjC&cB)T7n!J8G4)7+t;?X z&p7*CYu9$R-?vxSlh0~&e1EET_&#QQ-`=+Ep1)sDb(|jmQMKdtd?~gZ*8S~xzyIEz z`T55f(yj;cx0hwpZ25kN+Ijun6^6HNBDvS|R&l`ndxgXHuP8GAmxoUO&$P`l{H0U= zy&&q};CWd89?!$V{1=1tp9N75wWSi*hY-8&)%bZ^o4_BRjB+9_8i>a|l-npNu4V27 z*36NMC8^9N|0GeB@EP#U*x};=h&g5rC=Bc8gTCH2d0M|tQ`w$3=Swqd+P`iGbGxqX z?=M?6@2~Z&znb&mxGr+!kL#Iq16VdTeOI)%4c*?)!P@oguGyyTSXVZ=t!u99h|X-U z;QSWZr9NTGG;L{*xQM71M0a*yvWS;Y);%`3ZYCAnICk)ju=m1dpt1P`c~IhLuIJY9C%#{p%SnHgV z8=AG9n|M2-UnQA#V-2H!?#S@uwt?yj$34l1l=|p(srs%q6G=SB3;`jAT>hMh)XCIy z-l~}%sw8mCoLpLAaE_?uz2U~uDaIg8mb+_m$55&g$mZ8e4U%1k$+RUM$0uqD3w#U8 z?*m`MIqk-#NPc*=o!#f>#s-P%8OD&&Eo9|1<-tGNE9Kts>}!My6gx=K(o~_BEpOL~ zlZZ~tyuz5nA*gr<)WihjL5I=alMCe!F{Vux(j*;l8Q66i4){U+% z+v;tusDD5kmS%(Nfo5u@FujEIETh0@Ys0@ePu#v=T=!+H8!3lcJ(C4|czlo#1gufY z6QVFdNgsg1#&N@L43q*mm3onM3hM0S*3gJb%-%{+uha_}D^Y(OMyZ#Awgi zcQ6^AC*`3rog81^tZ!06`JKqzmK>EVuE9?tX-ZE6$M1*N#S9JARu&KqWG8Y9l%=iQ zJ+!FWAJ%va<2vT0zij43HOl0qkyDCbw@*L02X>uS@|`5i*ZX9cSlAXG0272>IY~XF zOL@EaW(q%pJgJ_&}JYCnnSd!`=9ZaFHZ3W4=6()*Jat8lP81OPV$o zj@t~mm^PpkFd<&1w9P|P3e)ict(N4tc9a7DNp;%rHRLxDhuGAJzvo#jZ|#-0#0gpj ztmF^;J1Wbc73OmwdsOC%no!3pmBS*3iD%-8^S^7vm@s_Hanpa}()jnG7m86CX^qSz z6V?e*St^i-3F1S`2DkYnxG4zl4SlE7A?d9`@|jP z8!XONj(ZS<9O_CP29Qc+@x&9kdVCN(jxZm}27}lB@+wV+U=On(^<)U4X<$$-k15jaj>SU@!yloz*D}bJg`swWe}bS3{3$a{s`Ornrt8EyJr^`9_gA+mIS;3` z4|6B|@8E2bD50HtVmu93}zIG%GgEwn7rehS-PPd9L@9 zb6a0aVxrK8~-L}@##zqjAJsKuU8=H1cKYI?ccz=M(HmT;~P47sQjMiVT_ zx8m(>^qk4}@YzCImy(@F5W_^3hMe!09|1m-QZ@K4Ystrw-U^{5o~%=s?C zghrU6NZ`p?K{LChJzgI4Sh84TDk)V$iW$F6^9?v`KnPj%h{H8j8bMM-Slzmpp6jQEX$Jn7aZH| zw~qmqo}`&(bd{DeH$V@N?H-}CSi8Q>5Ru<0x!;yE7LEKa+ZPhkQ&z8AS5G`{QPxh)jVc$pR8( zISiQQbNNZRaq^s@H7Noujn!f1dOthr+;|V}8^%&lx>r`LG!u~WiCE(L=g05}t7c01mNhZ)v-Ucf^-0xgE%+Vs{ znqJs^!2#Jk4o`#RE~lhEKT#^@nN+CF(xQsNwSx2HNQL&HZ31Nvcbk4X65+&K+U&Gw z!ABZxV?#6XmLNK_RW=ZQ#6hH>-u_Y!!}HYcinb-<7@v9WKeB+ zn$Nc7&((ZG@AGuquI2mnP{-}_PrvNC8~kT+=H_#E$Lst1JKOp*-gTo;9mLeB?D^MZ z%}8puxHEbNiNuW`AWWtB-%~PK|DVFe{MTJ9{J$PL(|_P%IQ}cWT+>|A`;L_BRNbzs~+HoAsBu@z1X2fB6wZuz&oB-UXRGD@$zu&#Bhafhn>1 z^ogcKFQ+?wJ)deWBaXH*_|I!D6utqSfOhVLg-w?|@LS>pO=|y7z$nn2aXeq&&gvY# zZRN$oXS(k9ioQv7J!QHs&$d?+Ayid$oZ~*nQC%=!UK4SOu+{&<9yD*R)@uJvdb}g98vul zA)?q1GIK!BcoSlFP9~4GxY*5*Pj0#ZFg6zus}zc5R_#o4mfXESGvifZKB@*HPM}f? z+N>^6JCv7)_|-s*ME_&P<=D8UR$I3b_fJl*`C)LIJ5ptX@|KCjJ!a^Z&W}U|H-sbm zh6DDXCdjcAUVrJX;#}~+H%SlgFE*(0Pb1Asgs6MLvR?h>*|9&x3VYQk&*c#yD9Up| zZe9q_xGp3c9`15nzkn;2Nz-r#uk+lD2m;_OPIxZ7M5L6dNhGxte0ZnQi4wv&rlqVJjew#yIPe@@gB@ z^!fBJ!})5`mg?jLF?KUK?|%@}kh7~2NLOWQ7yNt~+)Bjqm35SRSc%!WNQa1K)aJh0 z610-7koKTgyP2-TP>YNuGxZ`hfB#9#8#v2kN6mAk{RpXgyEW87=(IDFb2QIj88Snm*L(Y<~5Yh}dIlYZa zMCO*Is-~xNTdo^Ar2|378r%>z%)#i6ZaNqWt8UDvAk*qCJ$|+R)_k_ITl!d|LD0F~ zGky*zYF7@VOw4+1##C9iukLs|5-Bd{$3+KP_-Kk>{j-=5FE#;)xk>Uqf5Wq)eHVXr zcwo#0ME%aJ4UN9d`F5thA4+@G z*${y88?69zFqIukKle&8$S}!`?l?ETv7YjfNzGDK5hr^w=U^>f*=+{63+1XhF$~{& zj?svoBg|N;d1TL2brBzRF&|0vrp@{ONXwR%Vg$@`m(&6w?A!`JFS(v)n>A}o(?>Mg zDh-DCEwsf3FfJu(x?6&|blqc^d8?YTeWNo!v}ojj4>X&sI+aa8;|QoSvg?~f^Nn<) zQn1PXtQX0J1>C*hbatI54TMIhFHHqBqKm#V2htuiXBMZ!uOpfK!DPYSwZJsWnWg$w z@7H5NU_(^#DL@R+L8m^0&uPl0xOX&W+?6Y;N6yYPKipsI_7^xN?OSBVLp#~r_kaio ztR9%@#{-?Q$&%iFGuaFx!dza=%+Jm>E8@56=v*#G5YUv>dDLaL!zs$%Yt%xKYpwMZ zQ_{C0D{M8rYasXe*fca&DTytk!{QoXLki5sx(RPEt};;OBGKp`me4T_cX@@LisdeC zG}h|~k~QdvdL}Z^@MOcn!H|6Q^w{6dL3O9K862_BL|spQbw2~iP>77D^9M@iCf}Dh z03oCi*cpvNRD2!piF6Xa4HgQ}!$!Nhds-JoU-MXu#xG+E;EweBv|q=Fj(X%lb~;?< zTCr{KH@0&1g!#fl{?2P#95RERRg1h)m8sPZ+X+K>5JP5@I~F?(xymE?%7PP+B}}vx zgARlcN1@=Pw2Q1v{Y3ff>MRMD#8;vcRd)AUA)8wTQ>D230 z!6>I;1Me@m+DxW#RtXJM0GR)l6TnI!jWdVJVfJ^Or=4sAESfCR0W5goED+=8*dK#* z71W4q8^&;Siu2Svql zDnBxscbeVUw4E}K^KTJ8GRt_+I64$&GW5*4vNb!4*dN#{L$8e*W6!}fR$+&-HF~Es z7+Q%u_(}-CiGtD<*F>1aqqx>ZbXBOdRUuyeVQS|wM@l-yl}S3K%&j8VNV@^JCn^Ex zCO)1JYW(w{$K6f>e6MzumhFn)ciEBCo=)t8TY-I!XwV7!ltDE}so73lhs65@o3?km zYl2d`dhRC*$IS^hcOrHYM8WVYl5!{>o3DIOuc;;snP<_Z$`D-(3X&PM{35l#FJ=u8Vx-+ah|?G z8ryU+%=R%g&cJGKK-xeY@K(EE;5oLmDEWOHM`6YfBl++W&*G!d_LZB$mini-{leHD z1Plgk+iXdr!~=a(ha6EE{w6yI;wtskF!Gb25}&%aEtVWoOcR$$ftF(Xn&6;<59J zh+)jUSd?LvGNRFoXi-<%*9dqrZxj@s`b2a1V@uN^PUacBX%L@skl2N-_WL1-4c0d~ z3*ZINL$7TvsBC4+l~#i!6E8A)(IDx8<00rJbFXZLF#INkF^-@P?r8;VT50$8?qqq zClq5RFg}{QsEjYy=^c!s)!Q5zosiUr2Z;l%ZJfd`i2-B9muTI_8)bapX)|#q4R}s5hkcuW~>!+K&uy&6^dpO11ZxY@qRV+40 zP{y`%i@T~3<4O99-Wnb$6lkXf#8#0%UBoqApS7Hc2-RVln->fvi)LnI%i@+(ClYEm zaS)z@#C8?s7-~SmvuXTOiqg{^2w52VJpqBRijN=6LN=mE4z=?4Fc2}X`Qo!g=Cca66AMULC^(Q0srr*rZ5!$F z*i=f~V42G-=>8yTcIYSZjAjnB&BeZOrpD;Y=zUcQ_=@rirl(#wMm}p6>)R_}@;zjG zeKogYEpu~sBw!HZ=`9ex!cPs-v1!ly(ZLS0I(2C}A##}0U9`95XvB9Z`9EmX4%;1H zj&F9xDKt=5%(=yx#wj8$Q<&#_T*`et5je(9byD=HCR6`}+nV`l)DZ1Dm{}bKJRbsh zmG&T}_e3!V-J9=JUqtWP8(vuhn>bDf^Ar(NrgJ#eM>AWo3b3~L{HyjXqnw}7McTi9 zs}TDY21iC%q;@l@Oy*%UO9P^-W3#i})w9za;@z&<-SeZm`Kv_hch_^xx~I#R!pAAx zmuvmo)wFf_xHY%ON@r&!Pup7e*ZP@v*Lv4h=UYvpQ2_W>`WU)RlCALCx|VA8u*V0U zYskLtzja%%{Kd8WuTS{@&2Ec-HY$JNB&@9*@c+q@`Mb%XF{1t6&;0KI#`q72|Nj?< z#D5C$E$4p~UHGT%zozV887ZvH|DTXC7B$7jnz za)%3vMps)0S2qpd9zF%xVS3^v^a&EGduKOm4csW@Ly`m%sn~yMEi3+lC$J^vZ2%8N zf@;#SdmD-1l+rWe6uHZ%+)E0@Ttv)QSFnkan62C_bV9XTs%7?HIxM^|fR75)!7oUB z7gH1}4V9$VZLDI9tkod`xB6{O$tql`$4Z-uUw|TeNsSU(ilagaY%)$z0Gz${|g znpCCs5QOPQ)U~XLGC4L$v8Nyhr|5Bc76sG~Uo9;tFj!a_-4F264NAE4toDFCQIq86 zY5yu7h0gKBMg?hm(A!IsZy1g(TcmY+@IG)R>vk60J1*EvAwa}w>PCrL%z?$W_gu9w2)zfi8vprvOF=7B*1j}9SA1rPqN7E zL3;NH?6GbrY@u$Lm=y_k*o*tn67+tpC`3bO(ISdG(tWfhB9(YuJbgj!tJ}!@rIflV)n*6BfkX~1Z`-@vM>$*1!C%b6jp|>&*L*cG8-~Gq;7QXaI z90AJ+$Q=HM_}Czts{ka?%DG@Z`XM6GS$-`Tg{rzHuIRjrsxd^+yOc)WQe`)o+-O0O z(QW9_eRGj1HsSf~;W;5I_!)sjXJiZcyn-J@WQoT1AEpctRR%Yk4)^Jt#USz4In`Eo zXZVBUc<8ix{bPKET0#pyaA*x6G68zAsW1%nc#fg#JRONlj!bzI5=%IwisWq_V_@Ph+nHsYb7#bIR z7UC!HUyy$T2@_qy``n|KZgU*B%9tQ@OqFDd!waKH02-!tTM_prIBZJ|zqE{(Q-;T95`Sq`RUJ8V2DpdVWxX{4sv}?1pEn4aBTX1 z!VOscGtiv zjTE2R=D^M%1*j4%Eqs3|2=o2ylI)yl9$FPE5)Lti9ftS8TY{3WPD~gFXjG`MZ$l%h ze_0=52=Rbe0V0W$EoKHX5Op6dymDQ*YR^d$n54jNPhQTCXIo4o3ii8ZCwv+^6Ml4^ zA%~hzDGG=>aH6);XY4b%9h~rSFsaZy#L3TnDi?=h4a`M8bV333Itk}CG1lo) zfHYBRs(}%-HuMaBr56`UnI=?~!N!<`@uewq_?qVpVyzUFUKw;~@k`pI?2#n#LL6e_RsQ?D3nTup**AE7c#NmsIvVZ&_aEm${xOw0zBEv-+ct@bIiJ{c9v}k@z z{L+EXYOcbRl!6pYHIh}NbYb?^;>Sc(enpHW}T)%G%nP=7fC9R}%{3_NBB*yM+nO=3l3!k zV%jON*lV^DmD$_q;BLY!Kw4uveLAU^#>SFAv1yy>sYE(cov9$b&x`pv)EHdqS}1Xs zoVrN$(m)0Z ztj>}@vB=GxKH+`y)4%A5&7OZ{c=B~@0U2ajM@?pq4LpC=_#;po@{;Yc5&R9ey&`O}7*?5>a zgg@IM|&I&6<z^eKNSQL{AFR7HdR`*mM z2I}FO96N3L;8|IWSLO!23GQMT#hz$W-EQaV=Mxomkt)utHkKOCRc`*;g>#dc9nzTG zJVd-^V;hj)$zqn|Kiq>)pjR2v=U4J@lh2{1GFK6#&d@+zX4E59_{<_NYDt1E6M)SL z8#fozkHmBp`j|Hl1Xql7N$*MlI#4p1zf0dq0b(>L98j8Eq#fL@r!~~0!|x6OA_W_E zuEVDF^SP!Ax6$cWtv}vsvguSdrDV8c^ZGZ@4`NWGv+kC-x0(J-OD7h;R;}~q339r* zhogx?9?T2RZ(+CfDwg*<0P*4 zMve1y7XNEPT3DJ*mdfpSjVt6@nje(YJUwugts>1Il^M+~g+jDI$2fn5y3if%n`ghl z8VqA-%I37noTxe$s`a%I$Y^NDfvDIs_#!c(H0ewhC{-4lWHzQS46umxs(DJ80c1+| zfk@O5UFPWdFRwTq@~jE0E0Mc(6{1)^6-lU>xJP2+Nua)}nb$wXg|@+KrXs}!rdZr9 z>k;<9iy8(CfT~=;p0cyp4|X6AE4}Q#K|0kFJ(|tT>vvUSgFMeE(xb*wqGEI!&skk{ zXfvJK$s$yr=I2aUI^VElDyK3{uqiAe9Zr@+OZjY;2#>dwTZeM7ezgMbr|g!aFXa*9 z!X*%T&`5W%iQSmLcofwRvx2y?S{B^NRzAizn&ehD12P@sdD(3tEUh$PHf2VMsBbkJ z_?-T_7GyL?d~ev~D4qhota<4+U{Tam<*q*F<~g-@ZJ4;6WAJ0dki{9b#6^AkWBc{_ z`Dn$n_O9~wak0s>{gL_Z@^P!%?frZ`zdMijdFS-CoxL4<#>35dzRiWp-G;ICb@7OI z<=p(8Y0q(w4+*(Mo`;M+Nyf7+=3zij_wWV8e&tW~?{_W#M>FgHkMCL-|5v`6f0IYg z^0)2(v1_4YV)>_b!2ilu)2O~_z0M5(*H_a3!VJXV`HmW#!y9k30*Y)R*UjV^(%~}h z!59<5#At*Bep88MVXQp z7y>@;QG_b~8YH>7lydZ<9*Dt1C*ZATV`)_qO3Y!CdWA17ADdcL^!5er`0$_c=w^Ux zqbGYPN$jkgr)i_FcGxY)FW$@S^~-FCPe;WR*dr_0uV-Bcu3z>7kKx;ZB|GEH62+OG zMC04#>4p!bZ7(*zsr0C71Ck zGHl5MX5}dmCA}9Xbyr(8Cqys4cx#}fYQO+IIEi9DI*)d>s%PPDe;R5#5A_4x<}nZ$ zx#e4mPLa)}V+nr-_dvK@M=T@Kj1fySA5m8GHEZdmtp`VmZE!pJkKqbyqm4|88iUfu z%)2}3m_m%(8No5oFE6N@H$GV7^opwC+=J8U1RS90w+==f&43Vl9Y|7k~x~Ha6_WS0gYQ5ew^}1 z12fi~ykhSQ-`l??r>56|&$l2W%0`tkB$%FY>6{Pku`+8)Uq27nGj1{96uijmtXAQ} z=P?|Knuonh^l){3L+gbg#ZIHV#Fhddx5O{)8%i5!7L(%w)o1ytE01;1>q9zz%zf(Y z_2x)$-DWmX+(;C-3S4EMBmW1YZO@1j3feB6IX3ns#EuYjuLx|o1?jN%#pH+3Y;9fQ zX&Sw_PyA}T;t*3NYxrQf6j_Gh9 zN%X6DYLJffyPEe=9LktLj=Y#%s3WT#T6nj0;fblVVS+S zoy7%aiT#!{$O{QsDqz5J_WivADI2d@{1xl7wlPGI*oGL3Mq%h~j8LtZ#UsNe#t%5k zzQrk4@3K>ELDMcmd1NVb$SH9~ZQT8grzbWaa@&0n`eH~aQ_OyP7{pIPY)Gl`^g*|; z?|Sz}lt7%x%uJcImVO;er$i#h!xj{);(!x*6Hv1b87wlj> zWRpQczu0Ru;y*bv5D8g6SdqMg9HNjErtC`zS6Nz*=*LURB%$_SjTKk{FlIP9*Xi*Q zSeD+HUUck`yX(@~NrycbvtLRyABr7)7B}S3$VSlF#aKufr+&v((BBW`NLGdN9w&!;N~pu< z%n^dMfn3-hr`HRz*t9gQa}jIJLN>!O%5oN94cTgb-t#hYe`bPeF~vl3cLP1e_RF3x z^}7aJ{})76nCzW#lxjdI3#-#1G*<-zLq;aW<9;g*I4mFqV#}c-l+`n-1nCb!ED(_- z3JP8FB4m)EF*sdTL;M3)Uy(*1`KEHH-}~3pz*Z*Q{vnfdr7N2SJ(oYBOAhDe@6Ycy zGYHyG7G^wv=A$sBjEj7JIvM&n#U(|l`j-d%JFK+U5dvC+~c(*H_{-1SKT{12< zC&dAE(Fbpm@l`-wA?$w;fLN_7&ch%(AT^PVRzh#`HA>r5xA#e1gb|-0=0!_|n-rQ- zWud>#iKeqM+DF{A`|;3(?dwgQZNOt#*S0LPa&y6b&Ago2>RWq`j<;-&Qh|Y~L?^{e zE%Mc5glnsS_f-*9V1Wj@I6^hLOALBi?jEV62pLJ&S^;zv) zsC>s*`zddF_318qUj(QP!d;p5;-dnQCxe-*HT}=$RIV4HeB})$GY-dmWgdDn5fb4G zLPoRPSNuQyC2;xN5wEoWg9mTNp66G~b=}z$pI%QxmkZLI|cCkzOnO!R~!46raLFDoEV%p{EA<^PULo(Yi-kLtvh3%J^N`@=;Z zK-8faCtk+dz5gSi?5n-I?P>GP`w9gB_Ko0K&^1yfl!?5znEd@p{gq}|J^7#w+@@Do z?;&)*6_VEOq|l@^Vy2P2m%UMk3!w-xpcDivsAiT312dQw5&@~)(M-7w(f=jXj4@&x zffsy<)K>RQsbv9%yq7o;Uz2J@(n2>>Clp&U0LrFb;K;9kfLt>&cb!*PbDFS)s&Bu) zSX{L{;DLN4$4`nJT&4FLkG+II9!6skzA(rVGh~T30#lEXxv8C?#b=;|PH>$Oy{Zzq z4V)>6E`z^~FL3v_L@usGrHL|#F8NA9kd3-b0cak2I%dcsZwBgKV@wW6Co1BFD5hlH zlv2xO&xL**?|?a-NiA}Z38Ya$kYG98h%PxaE7FUcpzgRDW4Rv{`Fn?uoSW4$Z^w`k zgBlc>y68NA8|s{6Os>CV6j{vSt$}naY78_^5NT##B7d$#K|zS|JZD&(8x#mOmHkiQ z9~DF5OBIk!rMm5D4QAI2sp9+$4KaoO7B~kc1R^kO;v=yFQOI_LqE!npezP%%c95}{ z{fU+RGlsJ03S^TH2~6vY=zl~Z--uDjdB#|6&1oM^3XTx@yVV_ev{P1`vv9B z>!8<@6_~O^TgRaabpF<_??{~*KT9ATBv>i7L{5z~Mq>7FIFk0wRM07gbQ;T%t)$HE zoT+<{V9@bjH4!kj8#P-w_s@JDnlAjOvqz0fRx%u|9d9?Fa5XCjmUYh~9I_QixO z*qdv|ZQy)aulm|@`N`~x&vLg>HDTXAH+@u`u^H<@%rZI&A2)qe^nK#RjHmDDq;ku9 zIfPW(eerq)MkNy_apea{28v3huqj zs8JUGfjYCRWVh7WeVG{ApfGz}hC3aLCkBBdrpXiE@-4Mth2hl7s`rrry=29=;n_Fh zpuY9=_!bGYZtLQAd&w@@)!eCp+PHcVGo_gSPrU_L*$%$YMXz#5Jw+Et(V|B(5hxCa z8gs~VdA?Zsl@6E1-01InzlFEJ6J2MHnTYetuw{TwQLW8g=Z;D8U?$PUH~*&bpA63x z4b9zR0_o2ZfQ&nhz!osOc9@X30R^#8-lMQk@zsJM!yH-QuR^_blr8b!r0>3MS8n(s zzHQL*CZ;IkABx)_i9GTxWBGL`#y zz3y7M{#?2Kk;LGXaTsFH0L;w#h_K(cf`_$*hP7B7jU(!wPoij^0VE_3?d1zAWMC>A zrWNUjKB!A*#hRlDA}Du6TxaKOqMHmG)+qb4efgHgo4d_xop)}i>!ha3Qukw47l_gr z)P5=I>@zvsK77Qc_k8%;>O6dlIp+R2|Ln4s6PSozu|^`89epq~VNi}-kr{jN37p97XQUx}8jYbLM-?x9f$XXld)$ z66<-9k6mWab5@4ujK<;9_RWll8}`TBdm_s2s&8cF2o5R=&uw zpNS<`sCNCxHA12LN}Thuvuw`Eu;N9$sr*KepPpCl=F%boS(UeH;A8GRl-B+G6Mvy`DAmY80OL;}em}z-RB{b*6b}S=wz4;07 z#eRfL1|gy@y^m-w^OAuTWlX>b5Eh^vw@*^WRP#6 zGThv)r>j-lu1Ye2^eAOyGkHbOOyuDTxm`B$Y{(T8kRfvXd607pG5Nk8RLG{f(WvCe za2LNnw?di9qrF@H41kqOd)53|_!5wA$3RLk$$r;a424t1`?~H#GRIwO$0kQEDH07& zvtulwlnZ?Qc}mE!VBG@PM20U7L1s3h*bfVXY?iACbqJYg_;?n)t>yUc&l1)7Hgxl} z#d@=aMWQC%iecCyo9zPU?|r_hG|c1jalKAHZb7P&lfzFnHl~>0!b&8Loz|Cr0H5n~ zAwEMv@EfMpjC4jPyGewqVve)c`cOmU>wPVDu^Pwyt#N+)?%Sj4=vr)Nh9jF^p4+4H;BB|V8N4F(W8FmR`MiouGvwqv{!lqcZqJoy?%LirBi>yu39Ls(89IC32Rz#6o*#?- z)>Nn92#V#o83rgNa++WZnOupmoXyl8yhKwTm%xZTn{Es3Nnpd@N>`Q#c5jLtw>pB9-qgN(_G7s zV+>qh&ZD{7&}UNZa$|Eq1ksLMmF6ihplsw6a=BfaL}d%(a%ZUIQ*`vA34B3-w+S^G zAcf=;&iS{OM<5dH-d!c;t9d}$z2)w9dJFI1IN2*$xOnB+@w~s8U{u2Sibak+?%*hp z(kb}zH^*;~k`0XF3&F11ji~U~B91A^Rw6H&;jX;R_D*?8^FMG$TsHtK(`H@5=jmIMd2%qVxA1 z+Qn~WMGcS71#N2Wu8C>u1MR({hUv7$mw67Cvk$m0m+gLgcHGwP2NK*y^%7#+LCAy< zFT)Zhalf$r!WiOyVs8WEr#MV<9T+Y?bY@x4c2e+iO1ketJG*O7)WH5Ji$=5??K26vZ(_? z$cx7FS7AGi-fa0mg1j=B7d20`R)6%muN`mC(|u)mT3Pv&-T4+u$to1 z^)|tw#rt+{YFS@=m6>l2`$${fE4n9+LP{qS#s6@?LJWS7I1@_} zA}!Y(X6Z7s)H1!)(%BxT*9%9%Ti3IzvFl-cy1T>f0TQk?+xuxS@6!9ua`tr1|H|cu zB}1dT@gH^nL@5SAR1|bE%~EWlqfyzCtb-w7zMn6f1pVnH7~RD86xMAji|hicZryd0 zlMld6U!u$G6r%}lk0YrP$<#LzTTI5x?u-;DmQOdD!e<|4OBlCB?z7dQ5A7O3AST7{H41q>S+22wF%VY>h`Q-Z;{3(Fi>AC|7K1X&K$)bX&ion5 zI?ZDq{0(CdW$xtd^ymF4x%-&lshwh*%l&|5<^*qrJ@ZRky$FO^=FU8WoK{~vuZ3QG z4o4n2P1H^wkZ<}qkvA7vyik@5C|^=1Dg4Z_OG?2bt@yd2rFM6nez08#ifWE0ZHsHm zE8`tU6D}#tE8)mwC90A{ znA>y`-8&X|JwSXqczQp48gNJ=S#n=-GW0?}_0o~DYZFB`Ml!Cn%@z12QLeOtvk0s= zxW}8U(5)+Ns0Z9Wqe0Q-`GR{B#jpxUJ}Pnn+d{;I6tn<-Br&T99zsxFVf+^uzd8Jp z1RpuTMnqpFVv`uCv;#jQ7QuLIb{BxuM9}rCvc5++Upaa%;OQ8H-@lBxph|xpF8{=DUSAH+oeCKnf8M#!Cy#FNLq_ z*MkWRbI(pK9Z$RRjz1}=eVra!Q)#9w(1}KznXkOa>qp5)I?lT%4k)Lm1pv7iui3o4 zKK_8h^jygl^niDM%!2g0&5o?#iLJ)RAB|bF+v{K%&*?=#3_uashrs6@)Q<@s6R@_R zgm2g2oxCuY0O_p?KYp@Qz~ z%x_WFShA-&`1W^~(eArqF2P*AYR2Kj-*fx$pg(1i0FnS|KzRiK7~3d6XMicBXq=j_ zvqSmcxbw!&kDm{jV(=vZ#^_^O5WN#pj)VChx=2~p!0x$YE}>kz4wm7_-+Okzm_BuG zz{+_TpZ9g!e*TytTv*r?VweT-Vvt*Bxrj4DSqQhtMjf8}mh|(Vi69lb^N=gq0y`8A zAFYNvc!|;0F1ZHrr&)OR9fYg!n%NP_=2Gd$!}QFNyDqX_4E5wR{%KJT%Qa4XMLP4% z#p_>OL`h#!enD`@909d*%kp~XY@n5S_8x#4@s!;8ySd?@6(IPTAE!+aZ!yR{26E;u zA{pn&sq1n<(7)wYd|R`a$A1xayqe_Y!4I2edHvz!o~DpA%wcK1m4}+s23RWPV>hP_8}+hWpSk|(&Wia1!Il+sCFa$cGX@o*C3^;fN>k$|x0rVC z!Vg5rOm*m)8M0QU%}*8IpbjvkTO`_=P zH;ZJ()w>7L0<;gv=;Qqxe@alqbxLqSIkblyyPZixOqOc2?5FKXvSLx(l4ZcUbtl-A z;Lx$l^f$?*(R9bu0gE{J4ZkP~e-rqvkSw7eJ%VCb556BJLW+=2_UNcImx-Ap2=OO_ z+)3$EExksde%m=aqM|?@xtJz`*aT|sj0?e7=TD;&vV<_-%|4)&U|CRE2pxcT$uwVO zFL#eWo<>GwMO#`Mdjt+hV7n2f|E%?< zD#$P-1q{Wv7A+`?(F#tTJ5%st^Fs2X_DK6|I(bRmFH_yKHxEL0DQ}lmmsM#E(m#D1 zj&Ey!dZlP~dAy`scd^xe9VFd?&os|K54QiSX`RQQCeeA43}+vEB?Z$~&K~3pCor0X9Z_ZrssU z^Ymci>HhBP^ZoP=B`8cs|GI>tiNA(y#bB2WNmrxM85MPiUFB9nNnMkq>e(EGXYlL! zm`B<Gynqv0kPHEw!7)DiAY5DUrsk345G;2wx{TeSQNF0UQD-oud-vr{IC6v^!sl zmOH$JH|b+w<{7lat(!E1oKU^W*T?))An+ zazCz{HinYoU-yJXTCfKGAVk0i!bi&i$U)bEyHHV?-(t#lIat_%VlNVB2xf*BWMV{e zyh{z8@|LhvYV?QoW-KA#TO>4x3#Gsh#UOOx7{y`eW(*BC$CfL+X$J{kS@5J0UVU0a?%Doy%X0H~oOPlW94^w~NUu2kXZhFYT4 ziCgXaV0u><#8$-~8`7?o(j6t|BWCfipkJ9Ns=d9%4>3uSTslH$&G)qHK@~il8!p_$ z-lpgq9l!GLN4uh|6gef=9;KbOHc%9+dD72LTz&1jb(ofua5acad*zA4@FnKUv$n8R0F ze7AMdp2lOmU0nOT;_6jtwK@khL8%N-|3`N!>Aedn?v)8%1=K53REzkNr?RG+hxVu^ zP4ahvz~%1MFv~(DZIblwrRYaZpy!2{;W%E-}t=sQM44fbqdv_TI^^SzI=dlp@s#s84aI| zc||@)b(AwkwoE@)+bPA3^+ahAww_)UFFu>I+p*4ZdR8 zxTJ8XThTZVyJ#ZC7}OVIo3PyL-lyAWDzimx$_xsXY}`CDpmVqXowS*J9KcbyZT8Mv z#ogiNuwKTsZ);-J4V{>;T3B0I7Q0g9zI>CuC+P)~(r1r_vn{btb@s+j>k1RUY#o-l zEp;FN{rg$x2({v8I~Aqy&5AsPX4I#&H5PMkHBO)SvWlG&b5<^$98s@VpT^xe){-y;42PYpc86%0s0wt#71y=; zs3~J?Qd$~aQC@wcB9WRH+KZ`ZBaZV25;&N4(**(eg%*D#D(!Buo7>o#q^Pv=5- zSFTffC4QT0YAQ46Kx2FKs<%E!b`6MWAM^zK&^3ZGhjWLl$`Dn;{C3mYU>v1x%Q~Q#fr+K`$kk+2&9+zW$Dm0#E zx>jV&VQNr-cw4ZY;%&M2!p;Vco>sx8b~A_ZGWaOyW?b<|?;DBJ)xvju75z^##jGTn z<+hF{|-?^QH*Pa;JLZt+P1i@yQP>BZU+;DEml8779AG2{k7MBO6^~O4`I!^Yl~( zn)r@%Yl!$hTPJrM%|s*Qb>rG}x<`$%u|sl(=Hl<9i>u1jiA%>5)08^vh8C^0N*C-y zyYE}eD)6nJe+n6$rLx}eiOEI@NrOI| zPc9QGn-wR{-CK}cq34=TlI)tJpxp=6&Rbh{h?Cf6Bg`+4t*m@KbHuInntZs9jo1{a zi&NOpif7p}G18pYGGoJ4XEV&*_pPj;&=`q>Bsp8_FQlE2#|S|t*p}t&sueQCxm=k& zlblu#?FiDRGE6S9FOHUMcJ=a_)0xH(&S4=PnFF~T9S*DtA34&3OpK_di>Ko{4?Q@( z`<8A_*XSQ8n~^sE)TvKJwlSKD{DmNwFTiN&M^qDPVa;Y;V-ob6a$;|Dr5_|13ayyQ zx<-xI5{dLVpCnlUTL7prYw9-Q5SsnUc%JR&^c0gwS31GUaIkHf$d8F49itkfc(R3B z<`7~hyLp$ausr6ctpe^>Kq}<{M zT8vDyVh(%S3Aq;fWsm@*u>q>ozLMgYA$b@53?Smn{gU?i`63fUi8>rI>S#~K7I-50 znY~qXcIh)E8}t_PAXU^URuR8(Bx^7vJZeo%0G#13uwJ~mf`1>P@!tU)|1(U9mGz%s z+yChZjsJgPN}Zb$(m-^`f^s5ypirZ*ylQF#0nvzc>;*Lh5kBy@we7#Jl$2!V{TVJb zlO;^K;m*?aJgk=BX5Pl5+bD*NRd|wm&c>hA&ze@UEGr*8LvJE_tZ@N2uPu7Cw`i|N z7=C=Y`1c@Xfjm!R0cPDm_b6-uT>FuLW!)M1s&LX>$9-G!>w+HhEy9{nBtVT02%TUO zpeEPfI>{t}k1jYnz%qd6cWFIhSpckq<^aYn4m*rjkgVgsfR0_zdVb)duugLT!QO=Q zAZ$U{#E1Yz-9+{1;zF>FaRB_l1!0|f`%(4S^9GFHSMJ#D7O*7)M5k35?aV;}e|T{@ zwl@hT=xQA4K3Hhz6GCB9{}1 z288MQV`n}-{{O1&+QX`>vOa=oYTh98o<|(fyuf+y?~8!utyD-;5lbu&a3GXBM?poU zG6%d>Hk6k#wMj-T3@tmRnWIN3GwrU1mV%a}soC`9jG2AE7tE3UJ2TIGe|$YXkGj~G zwbx$vwf0N$5Z7&%L@BAmXtRNMmJemRnuiNGaN5f-PXJ?cFp{% zpoN9~)-0}S)O_Y0YyR|aSAF$QM@Gk9`RBRM%YJ;~>Lag3{nTT@74Pu*-!)!P7&GOW z?*bRhynD(M-vz8MoHb=xd6V@sA9g%m-gsCD76 z#Z`?*7DlZpsR}$$*k#Q#RRJSs&U74@J$GB+se$K5-8*nj#)~_Love=S(qUBBjNOlw zx&rqPY1_j2S!IXf?FWaqIyhla(v+C7%K|&LENk=rh@B~eCOjS%_+y~e?3rViniU`4 z`An;nBTuI|1|IrzV7m*w+x}(5r1U}YGe-rU+cTy4m}C7ziobd-vgP=MZ$_+%89BO9 z*N=(<&yKjdchHg2CmkObI6?z=maQK5Vc^?uPwNp8koe>;>apgxH@=u~bl2#Zu}>7A z4QTty+Q8zo#UZ^ecWSaOXyR*unQcm1m0S+o-rW(od;j-Wn&5|BjffX-TNhOP{ngwi zC9Qh@SbK1UBXZD&$L{NeAC`w@UI_f8(fIh8)s5B#ZF$G(Z8o%NVcChippsS{rfogf z*AW@;dd02?4BZwpa$wN9pwmlJuQx7f<(<9kdXF|;cNe^5ngO4*Xk64ZbZ&9Lr7Lr9 zubsK%w!(o!@40gE7wzJJfpZ$IJ=1Tye~sb)%cPQK+O3BF-zww((9~sXHleBO#Hj4g zCf?cfhpp*%eSdCb**oXfwU|+IaYN0pVJkbd>Ab7)EB6j+Q4^m&yyl~WMU7Mc+I~>c zqGNY1-}giBF@I=0=gb%Sp}mJbe(=z~6(2vNzqNVJ=EKqRx^^EtVph@IceaIW4=Wo)^%KtH?{MEs0qm5p*{}VfUc2w%9oJ{hPELNwl`3EV>CMecy^GGbNbL8`mgQ#iitLAeYdx;( z<<_|?CWclFu9=f#{~4V0`1$EiZ+~)OY3u0+yB7cQy4*3MZB2Mq-{;aBeg9}|RMf@C z_h#%HUDW${_C8<=C7oLN_AAftoOHcyrJ?5T?0aNqe3#k-(bxK9 zoyqI(%S@457JE0PKh(cE>yw@r?+D2a>5v*!I^dh!fXdyOVNZ=(Z9aIs*lQI}Y!>cH zy3PA!>92$DZ5KWw$?g3nW2}SERjXAcZ(^nxz*K$|&^y%Kfv*+7*-}Sv0RCKocg!q*q1-bD{Z(kAPIQFng8tiR@M%8sdskCH`4|sj+)VK zQQ4%=0g{9P)%5PZSrM1B11|mKn2{V-F<|MYrWXfv znf;nK^zA2%xJw&C&Oh0A!o|VYyt9I)B9E)l?ujW1tvBrZ2U&-Qxm%#cX#>z%%zB}BVvqZuP3!ksCYPW zq;K)6(3-uw(u1>~^6Z}cW$S|0m*ubXGCNNhH950K<$x}gi@ep(zq-ZMWu^J;)RO8^ z*LuhMuAbcWMQ3A)Z}F(*6T2mTRPf=|kRw-1CRc5cDi%K-^L0)C?b`x2o}1dDblP9D z|FEH5bFKR8E&a;+pPKmfxy#*$dynrv)NSgJozH#r&Uc5le|P9Y%GQ-9V|;B-6u&v# zNV@X*J@|ir&rwCE-~aO29*aANT<+Ymq^x$=+NY{VANMVJy~Bj~<2lQ`pbD_L_m7c$#*RJ29D<|ypw#|=Au8H;@8Gm?TKyF;;nqYmecWhqAlTo`j z2OPe9wdWb{u#8%3o9FK8+M$Oddh{6mZDN{%Ln(`bnN@DPM>bFdwr9) z8oi-g+gEO@DD=X8Zo5w*PH) zpNA*x+qP-rrTbnw_t>HjeV^b*8~1xB*YpXwZ{qfqFOF-uvE?KCT)9SDqupB{_&m8Q zmP3`ao;SE`?D25v<*crSgUimukXw~X92SudV=vSQWzzEe*GBqis@IrJ_s=#^pKnojqJt^ebWr7vuo z>+5rBNZ9)MlWr?49k})9fZxWtlQwua1#Fz!Qv0omx5W=3UrsKXxG&kp1B*)jE@M+3%xF|~Mh=ab*9%^g-zGcK#CZ_~ML>$Y~kKk?HA-qYr= z>PH}Z!-{90`C(FSPsiM~jV`ZrxBDN*m(8y{F!A^!H3MQ}+a}B}ZC1O`DY+8c?k!EP z?esuKU`3^G<;mx3t1k2kAJ;29^x{VMoF-%99}V{X+Wm6->DAXe4!qj?s~J+0nsv1q zQ;R3scq*~{_W*}mw#CB_rt>6_;h@maeKaaeB#NNR;eorM|^|Uzm7T#hpp}%SQEUZ-h#<2YN0I zGSs?1TN{n*{_K2Zj6Kfmc)WXQ$As|QW!|R2N=0JL>D(*bQxmuQHs&;Kt5n=sv#bg9 zVRfImTUL*qbt(9b*Be0*h7`XSa8atbqh_{wuve#(3l9eM3rO7C;^z)VQpL#*n@`>U z%jUJspPk$>VoN|y+uu$PI{kjvZ|4oT`oSvyzEi_*at_IJyFA%>j^6N^@;we`xTJ>5 z_tE(3Rjj5ZRo}Kld2R8VKi)nw(lAl95 zopsMi>V3C3x5t&@^0;g?Zazzy7#Dj>dwkTuLOe5>=Jq&}+-d1q(V>@iycg<7Pl*mq z(BhnNIYZo!rQbg*&pmF|1LKotO-r^?Lh=5!FtRW+CljkX3NtgZ@*@kQLtQD^N$yDe z-3CX5I_idarbUNhWdDM^bdNhPG$tx9B{g#F(BU_x;a}0Ak9j;fkr5FE1qI;+N_ci& zT7+a-R)kZIkmddu(LaA?md91tKP$h-jRl6f^ON(^b37oJ!(Pypls(-O9g02ckV#3t zxoFPxyo|afQ<5Xx8SYH`mpDgA;S%us9rMhbo1?Sx!|P5Fo}8T-QRvFSNN0nQadH27 zY-VPI;p4}o{pazXh@7;4&B(ch8Vyjv4}gk-k)V8LPUH<<8=sw>5pyfMMMa1|jY>(5 z%y4C;O~*^5dg2durPco>BR$5Q)jxh*RD?}J{Hy*ibw7`2FwI^ROq`xN(`L^Yn~D27 zwf>SkUeY6-YGfTJM>?JL+ut(sU&y-uo0)aTsvl|31oa{x7z4!fV#B`EXywNO*9D zb8P02+PF{945Uj^a z_*oQij~p{}_yl)eJ|@IS;YO5wo?Bt{S9A;e-?%hL6#jNoC}aL5)Yv~dG!G){SBIJY zzMVA$3L_&s&DI%p)W`3Xu!ApKlig4!cxF?led>Vrm{?bmJ0str-h87I3!ZcIuH&yd z{Wi{%H$B-iKF{qQo1N`(sP^mk`Xya?(?(>aX8-=}tcW-O+$j!>x1Y|{>8<=6ysXQ2 zq`ETlZCKso^@s`S?gDq-m^^o?8^4-8*`0rLg`w$r7;o!_h*;U)q<+Ha>7I=AER3s1 z@)liMh((&?gIy9H^g(^z_|KthnxZ*UTi%38cmier=g7JVlbs-r2(zgwz*L!Gnv>Vl z{p?}Lc=#@?XDSS{G(^b_$4O57qFoUtJ87<@Fn*G%!omsb!EO=Eka(Oa^S+i$=apd$ z#r0%K!xOHC`^vKY8!ZKxV$r^`CegmKuESCaa}E0!Q3^2AqO&PZroAF*w4NfHgrA}+ zgo~o-bT-8RM+vwnrY%Yh@l!0Of$B7g2C8Jz*;GX(-cU7-=Ne1{&8gBIX|h3kXsSx| z(=@(c&ET`?l9TA7YZlSKRCqm0wM5#(!6Ldyic=TylT^hN>4MysMB@y@67LdJCccs^ zmIbgl-G?k0q&H+)lSMuN+uI)~Z@>q#ig(8ZGf4Jjm3c^3E#d=Nb1EYJAVxC7WRfje zgD{A=Xe#M7SyL^MUt|r0rnx%P3g7n-???to8D?oj7u(qu_s}KcKUtR*o~uZ-hpze= zr;)CM${`+-b;Bw05MZP?WZh)C=oa&V;UqmN8~7TEcwU$fhA9f&hoNc=(S(tj95 zdRK937Vn{x4+093i#kz}Eb^}uxQ-&9!V4qb0F3+`MP`3PkrlEF3PgtY&~#A`DY8y_Ly--p zm2458DzZhgui)ZnUl2y*dq^+wsRGp`+8zb&3&S+#XGJIdq9_LQ0sJevBZY0FqB>RL z0~OA)$h)e{_p3r0Xq;j(Ol7*L2KkPPib6iEXBl*tnv-dzflaJPClGs7_7q#lVUZ+uvo8w1SG2_ zevOZ~FYGVj0?{I#G&SaNxNvkwFabno(`0>bTI7Q$meV9$EO^i&&K5K|!Bjq*rLnHF zG}d)6Q6w`K$5D!fgV4S()B+5i0MSZy%Cbm%)u~Xt1$AYA{-gn)!W$=xgptyx+mFJWk{TOHd!U#R)t$4>P{8*mvJ%t>}!$jftexOqbd^dI3gFmU)y+# zI4i)0U>fNSRe>_5aR^EnW)WTBPSU|1>FdiO!ntXD<6WNENWmXq-n0 zk=G21b(8628HCy;oJ~KpH)V>!R8t|FhqMp*{Hm#utyPh~BH1zx)=j2K`A*feDF3X& zR1ggi6BCakCZ>Dy6B8p<#dq!}CRSmSiPtQhd@j{8Shg%bXG85#Jcs6lkrM7#gH0CC zfm9mlNjrTi;-@*;w?H+3WCpH0`K-2j5%nox6eDV|(<1L`NKw$)B$?zAS#-*6B7;IP zgoc<{8xR7m`&C$Dn;>g>*CNhv8i+2cAHIV7NBF_0@LaZe8WbSoY>|JeLAYof7>n)~sUSKp za@=$_EJ*T(5)02YSl$p56Cc1%6D~UEzTizVU686GUel2Y5${9iSPc~_=3QM^h*rA6 zIO_=X#61kIyC8EzF{0)tPpu&hLi?h8#Ct&cX6Ty)bljtgnKqBAUNgjep%*EbQK9%+Sg#2h6^Y92fClywho7!=nUtUXbST|>!G&IFl5+7 z{Ltb+w9;6I=?GIujx+>Uv>xY`(P_Z+gMiREz-&Z65SsQyYZILVArsFvScmBr<)3kD ztecQxr1jwXFbo-Tkp_sxd7MW17i5sh=hx9LK=*-KG|?IMoM>P=SyoNS&p6KA=zi+X z-~!g;e(KJs;8IRiN8OodfD|(6A=4uN59t!h4Pyt22Xzb8caippyNO>cL>4p-j72bm z>RTw2P_9(B*q1^&hwdC*G9>$mW++Diua5W`EJ-#1*-5eia8JpGqKrd%9|Lw;qzlqb z6x$mplZd*`Kx-kxP@NU%2azG20~qO8xX%>Npv*-66(Vu+>i{D?ZlD$|>Tv^oc%oby zNO_2SV8ABxTx1l*v%$S08X)h@Fs>QGN|R4*$jtZffGPJ681W>mCBbGANYx}u><+VcA1m-Pt#aO07Q3$V2xpHw+z&niB@o+=^U`NL}%Dq z!cRvcMC5xN&2$7q#K16?OXR|dH=x`}-k>}Q7bqt>8-$->XjBz(F}TmoFu0e~FxUpb z{F2Xr=#B3eb!WoaEM zB0mDvJhlOF$7v5gHW}Q^`oQ9z9k?{CpAmx5dHv+LQSzrgDAURQfeBkH@(Zqw@=dTj zWEW7GCA)1=e+^EqTFe1X@HTH<>{tSj=ViC9fEySVX zeOQP?#d||_f#Oix&EdHw<#KW5q{GlIKzbafC0&eGKhimt9~+8pNAevlJa`fL6tC5Y z$1RjJc^vnxTQc{pqfEkgiKcB)rY(d*4C5XG3n>q}OQgpc#_tCqGJHpPUna^CQb8nJ z7MkwpE}^oCE-LrsT6iKx{EYNH!~D=5Z7sYnya7QE*)1YO1LT1j#&4YPf{0=WMCat| zS*ZWg9U(ko80&ku^2`SiebIi{r7=1$3Ret+po=s>L50WhyAyP;Q62r(rsy z{!6rn=#xHx`%L)RNjDK^B(RA8&?n8~_{|^0@-Q6SSKRb zp!1@fNBj)umSOzv1no!^tD$p&dYSByj_w!k0i6v-ieU(vL>gG!zhS}F(tSX2(0#zx z@_ML#(jFW~*=b;sOSCNs>q&M4s6ao!5D+%p12F2V2aISB7#w$+OMH)UR1XD=dX533 z-f0^q$~j=5s<J%E{_p92`ZQ-b28w=#giff4sb9afMd zjH5no!01gdVAQV%7%Ygm2Y!>PAX}*TQ|t#A`M`kT^#iS^iFFN(LzRlgQSTAPk&ljt zf|P3k3`qrXJ)|Ru|8SLbmuRvf9zs2oXph>bpd*mvr&<1_~TRKo>~Yy}?i;fBS1(MCc0!Zjlrpb$ZKg!oeMudoNzodE-3XkY5zz&Of100xgz zoQq^1?TgeG@fs3X#0N9}Acv`uBjLIg0Q@Q;`V6b0(<+i07~cSi}#^OLi5E zAo`QQ4-dC#U+P`JI7{@eBzv-W9{`H}A3zA2#66Hk;<+dm(Kvcrfc2=)5-{rR0Spb~ z;(GjD5}-%@41m%58NkRt1iF8Iil+Fg*NN2Mjwuy7`4wvYM^f-@G#C*Gi z);H26aGU(Bhi)+1m;FDaMnr!Xd!X?_oQp<%!VeAZq7AU0pNl$HLd?nQAvsKYP)-I} zAIbp$Ml}q;peV)l>;_7Lp|OBqGT$%K5rngjhHskdhxT|cM0|jje8vw|Dq4@;7C__p zkq?+g+5-&(45Pjb`xniNc>%!aO&nn8CK2&NYY*X!{4S5TX(_JFJ%X{GAtax6iE5Z-SLz;_}2jPN*7~x`34$oHOV(tboiuVEY zqlXZ9^B(j)0126roE`~Xx`#@=t;vN|eph$aEFvu^# zOHiUIK0QG`rVPtP>j5O{aTz2K>1U^C#k+(FB|b$tlwt#XACveBEyCol;WJO9$MIa3 z_)i8=2^SzJ@+7=irUBj%@i@F95_sHxe#9_T2x%P0Rx^VYbiVb9iYBq2| zsAdBgI(|f4kj|mA!T!>nqaTUx7qKbLMVKPWG_vv(=RwNJ&&E?}iWl(t67qfQ?^+S9 z?8k!Q*^u*P7%V@Hqkcl$U1R+4eKS#a+AVD)3wS_7G(cKUw4wNhm7j6Qkc;@C$)D!h z4f8Y(U%X)$*$PyS#ac8nOOz`GjK0rcXQ0TRw7*wHyn)6|q6_M)L>Ig9P{bMWFP#nX zFR#b(3{pXSHoN(V_Mn~^B(E&yXEd?UzGxC5zCyDGkHe!qabKt_IvY$6&!wJJBoF-1 z9&dR_7P!6&?M!bEZ5ZnpyPzlHjAEM~m{`NLEda&rfce=2uhMA`KX-|+jOdIQn`p}4 z-vBQ3P7^TB71*By6xV~5CfjJ2?nInXx+8vprlb3XiX|TtuTqHy=sqRBQY^gs5om?5 zhkPlhKgt2%U9{LkV232+qod`7Y%Sh46I~Fd(49lmkATL!SrvdbVj4@6AGLN!0DAvy&dO zpR|hp0or#(IY)R&`A$2$6nO|*obsIr5vitRhkjz*0~pmq5z^rmZ^JtWjQnvsd=uk5 zz^I=Pp&B}Qg!K?#ivAzEeZWTIIDTh{u#Ebe0i*Y|2$?9RgN3KQaAbFJbHcu;zKMRZ z4D)d-pL+6K=^5_4=221hht$WV&vMs)Zyf)q`y#rXn*RNp=yl&iN8$+AUjLEw5kno5 ldjGpG*Wcs#_dl9k|BD`6d7ip&oC7I5ifZ1o=djVk{} { @@ -146,6 +146,17 @@ function applyClaimTokenHeader( } } +function parseClaimCallbackResponseBody( + body: string +): RawClaimCallbackResponse { + if (!body) return {} + try { + return JSON.parse(body) as RawClaimCallbackResponse + } catch { + return { error: { message: `Non-JSON claim callback response` } } + } +} + function constructWakeEvent( notification: WebhookNotification, catchUpEvents?: Array @@ -327,6 +338,8 @@ export async function processWake( } const serverHeaders = await resolveHeadersProvider(config.claimHeaders) const debugWakeTypes = process.env.ELECTRIC_AGENTS_DEBUG_WAKE_TYPES === `1` + let claimCallbackStatus: number | null = null + let claimCallbackResponseBody = `` if (!typeName) { // Don't ack — let the server's own timeout reclaim the wake. @@ -814,7 +827,11 @@ export async function processWake( }) ) .then(async (response) => { - const rawClaimData = (await response.json()) as RawClaimCallbackResponse + claimCallbackStatus = response.status + claimCallbackResponseBody = await response.text() + const rawClaimData = parseClaimCallbackResponseBody( + claimCallbackResponseBody + ) claimData = { ...rawClaimData, ok: @@ -825,6 +842,13 @@ export async function processWake( if (claimData.claimToken) activeClaimToken = claimData.claimToken if (claimData.token) activeClaimToken = claimData.token claimMs = +(performance.now() - claimT0).toFixed(2) + if (!claimData.ok) { + const logClaimCallbackReturned = + response.status === 401 ? log.error : log.warn + logClaimCallbackReturned( + `claim callback returned status=${response.status} ok=false claimMs=${claimMs} hasWriteToken=${Boolean(claimData.writeToken)} errorCode=${claimData.error?.code ?? `(none)`} errorMessage=${claimData.error?.message ?? `(none)`} callback=${callback} responseBody=${claimCallbackResponseBody || `(empty)`}` + ) + } return claimData }) @@ -838,7 +862,14 @@ export async function processWake( lastCatchUpOffset = db.offset } - if (!claimed.ok) return null + if (!claimed.ok) { + const logClaimCallbackRejected = + claimCallbackStatus === 401 ? log.error : log.warn + logClaimCallbackRejected( + `claim callback rejected wake status=${claimCallbackStatus ?? `(unknown)`} errorCode=${claimed.error?.code ?? `(none)`} errorMessage=${claimed.error?.message ?? `(none)`} callback=${callback} responseBody=${claimCallbackResponseBody || `(empty)`}` + ) + return null + } claimedWake = true writeToken = claimed.writeToken ?? `` diff --git a/packages/agents-server/docs/superpowers/specs/2026-05-16-pull-wake-runner-state-machine.html b/packages/agents-server/docs/superpowers/specs/2026-05-16-pull-wake-runner-state-machine.html deleted file mode 100644 index b9355d9105..0000000000 --- a/packages/agents-server/docs/superpowers/specs/2026-05-16-pull-wake-runner-state-machine.html +++ /dev/null @@ -1,746 +0,0 @@ - - - - - - Pull-Wake Runner State Machine - - - -

-
-
-

Pull-Wake Runner State Machine

-

- Runner lifecycle visualization for the proposed XState refactor. - Entity execution stays in processWake; this machine - owns transport, heartbeat, claim, dispatch, diagnostics, and - shutdown. -

-
-
Proposed for review
-
- -
- - - - - - - - - - - - - -
- - - - -
-
- running - START accepted -
-
- - - -
-
- - -
-
- -
-
-
-

Selected State

- stopped -
-

- No stream, heartbeat, or active abort controller. -

-
-
- -
-
-
!
-
-

Key Invariant

-

- A claimed wake dispatch must not prevent the runner from reading - and claiming subsequent wake events. Entity execution is - independent runtime work. -

-
-
-
-
↗
-
-

Offset Policy

-

- wake_stream_offset is read-committed. Unclaimed - pending work must be re-emitted after a missed notification; - claimed work recovers through claim lease expiry. -

-
-
-
-
- -
-

Actors And Invocations

-
-
-

wakeStreamActor

-

- Owns the live Durable Streams connection for the runner wake - stream. -

-
    -
  1. Resolve headers.
    2. -
  2. Open live JSON stream at current offset.
    3. -
  3. Emit STREAM_OPENED.
    4. -
  4. Iterate response.jsonStream().
    5. -
  5. Emit STREAM_EVENT and STREAM_OFFSET.
    6. -
  6. Exit only on stop, close, or error.
    7. -
-
- -
-

claimAndDispatch

-

- Spawned per compact wake event. It never blocks the stream actor. -

-
    -
  1. POST compact wake to the claim endpoint.
    2. -
  2. Record claimed, no-work, or failed diagnostics.
    3. -
  3. Check the shutdown gate before dispatch.
    4. -
  4. Dispatch full notification to runtime.dispatchWake only if shutdown has not begun.
    5. -
  5. Do not call runtime.drainWakes().
    6. -
-
- -
-

sendHeartbeat

-

- Short-lived invocation scheduled by the machine's heartbeat - interval. It owns one HTTP request, not the timer loop. -

-
    -
  1. Read current machine snapshot.
    2. -
  2. Build getHealth() diagnostics.
    3. -
  3. POST lease, offset, and diagnostics.
    4. -
  4. Record heartbeat success or failure.
    5. -
-
-
-
- -
-

getHealth() Mapping

-
-
- running - starting or running.* -
-
- offset - Context wake stream offset -
-
- stream_connected - True after STREAM_OPENED, false on close/error -
-
- stream_connected_since - Current connection start; null while disconnected -
-
- reconnect_count - Incremented on stream open/read errors -
-
- last_heartbeat_ok - Updated by each scheduled heartbeat invocation -
-
- last_claim_result - claimed, no_work, error, or null -
-
- claims_succeeded - Incremented on full wake notification claim -
-
- claims_failed - Incremented on claim request failure -
-
- last_error - Latest operational error reported to diagnostics -
-
- last_dispatch_at - Updated after runtime.dispatchWake -
-
-
-
- - - - diff --git a/packages/agents-server/docs/superpowers/specs/2026-05-16-pull-wake-runner-state-machine.md b/packages/agents-server/docs/superpowers/specs/2026-05-16-pull-wake-runner-state-machine.md deleted file mode 100644 index 01f0949fc7..0000000000 --- a/packages/agents-server/docs/superpowers/specs/2026-05-16-pull-wake-runner-state-machine.md +++ /dev/null @@ -1,522 +0,0 @@ -# Pull-Wake Runner State Machine - -## Status - -Proposed design for review. - -## Summary - -Refactor the pull-wake runner lifecycle into an explicit state machine. The -machine owns runner transport and liveness concerns only: wake-stream -connection, offset tracking, runner heartbeat, claim attempts, dispatch into the -runtime, diagnostics, and shutdown. - -Entity wake execution remains outside this machine. The runner machine dispatches -claimed wake notifications to the runtime via `runtime.dispatchWake(...)`; the -runtime continues to execute those wakes through the shared `processWake` -workflow. - -The machine context becomes the single source of truth for -`PullWakeRunner.getHealth()`. Heartbeats continue to send that health snapshot -to agents-server as `runners.diagnostics`, which feeds the existing health -endpoint and desktop UI. - -## Goals - -- Make the runner lifecycle legible and testable as explicit states and events. -- Prevent runner stream consumption from blocking on an entity wake's idle - window. -- Preserve the existing health check contract: `getHealth()` returns the - diagnostics that heartbeat persists to the server. -- Keep pull-wake and webhook wake execution unified through `processWake`. -- Make reconnect, heartbeat, and shutdown behavior easier to reason about. - -## Non-Goals - -- Do not rewrite `processWake` as a state machine. -- Do not change Durable Streams claim semantics. -- Do not change runner registration, authorization, or ownership semantics. -- Do not add runner-level scheduling policy beyond claim and dispatch. -- Do not wait for in-flight entity wakes before reading the next runner wake - event. - -## Current Boundary - -### Runner Lifecycle - -Implemented by `createPullWakeRunner` in -`packages/agents-runtime/src/pull-wake-runner.ts`. - -Responsibilities: - -- Open the runner wake stream. -- Track the current wake stream offset. -- Heartbeat runner liveness and diagnostics to agents-server. -- Claim compact wake events. -- Dispatch full `WakeNotification` objects into the runtime. -- Abort stream reading and in-flight wakes during stop. -- Drain in-flight wakes during stop after aborting. - -### Wake Execution - -Implemented by `processWake` in -`packages/agents-runtime/src/process-wake.ts`. - -Responsibilities: - -- Claim callback lifecycle for the specific wake. -- Preload and tail the entity stream. -- Invoke the entity handler. -- Idle and resume inside one claimed wake when fresh entity work arrives. -- Persist manifest changes. -- Ack consumed stream offsets through the done callback. -- Cleanup entity-stream DBs, producers, and secondary streams. - -## Design Principle - -The runner lifecycle must not contain a `processingWake` state that blocks the -wake stream. Claim and dispatch are short side effects. Entity execution is an -independent spawned workflow tracked by the runtime. - -This is the key invariant: - -> A claimed wake dispatch must not prevent the runner from reading and claiming -> subsequent wake events. - -## Offset Commit Policy - -The runner uses read-commit semantics for the runner wake stream offset. - -`wake_stream_offset` is a delivery cursor for compact runner wake events, not a -work-completion cursor. Work ownership and completion are tracked by server-side -wake, subscription, and claim state. - -There are two separate recovery paths: - -1. If a runner crashes after reading a compact wake event but before attempting - to claim it, there is no claim lease yet. Recovery depends on the server - continuing to treat that work as unclaimed pending work and emitting another - compact wake notification for it. -2. If a runner crashes after successfully claiming work but before dispatching - or completing it, recovery depends on the server-side claim lease expiring and - the server re-emitting the pending work. - -Consequences: - -- The runner may update its local `offset` when `response.offset` advances. -- Heartbeat may persist that read-committed `offset` as `wake_stream_offset`. -- The runner does not need a contiguous claim-safe or dispatch-safe offset - commit log. -- Entity wake completion must not gate runner wake-stream offset progress. -- Pre-claim crashes are recovered through server pending-work re-emission, not - by rewinding the runner wake stream cursor. -- Post-claim failures are recovered through server claim lease expiry and - re-emission. - -This policy depends on the server contract that unresolved pending work is -re-emitted both when it remains unclaimed after a missed notification and when a -claim expires. If the server does not provide both guarantees, the runner must -not persist offsets past wake events that have not at least reached a claim -attempt. - -## State Model - -### Top-Level States - -```txt -stopped -starting -running - connecting - streaming - reconnecting -stopping -``` - -### State Descriptions - -| State | Description | -| ---------------------- | -------------------------------------------------------------------------------------------------------------------- | -| `stopped` | No stream, no heartbeat timer, no active abort controller. | -| `starting` | Allocating controller, setting `started_at`, starting heartbeat, preparing the first stream connection. | -| `running.connecting` | Starting the long-running wake stream actor at the current offset. | -| `running.streaming` | Wake stream is connected and being consumed. | -| `running.reconnecting` | Previous stream failed or closed unexpectedly; wait for backoff before reconnect. | -| `stopping` | Abort stream, stop accepting claim actors, abort or gate in-flight claim actors, then abort and drain runtime wakes. | - -There is intentionally no steady-state `failed` state. The runner is a service -process and should keep trying to run until `stop()` is called. Errors are -recorded in diagnostics and reported through `onError`, then the machine -continues or reconnects as appropriate. - -## Events - -### External Events - -| Event | Payload | Description | -| ------- | ------- | -------------------------------------------------- | -| `START` | none | Start the runner. Ignored unless `stopped`. | -| `STOP` | none | Stop the runner. Valid from any non-stopped state. | - -### Stream Events - -| Event | Payload | Description | -| --------------- | -------------- | -------------------------------------------------------- | -| `STREAM_OPENED` | `{ response }` | Durable stream reader is connected. | -| `STREAM_EVENT` | `{ event }` | Compact wake event received from the runner wake stream. | -| `STREAM_OFFSET` | `{ offset }` | Stream response offset advanced. | -| `STREAM_CLOSED` | none | Stream ended without explicit stop. | -| `STREAM_ERROR` | `{ error }` | Stream connection/read failed. | - -### Heartbeat Events - -| Event | Payload | Description | -| -------------------- | --------------- | -------------------------------------------------------------------- | -| `HEARTBEAT_INTERVAL` | none | Machine-owned delayed transition that invokes one heartbeat request. | -| `HEARTBEAT_OK` | `{ at }` | Heartbeat succeeded. | -| `HEARTBEAT_ERROR` | `{ error, at }` | Heartbeat failed. | - -### Claim/Dispatch Events - -| Event | Payload | Description | -| ------------------ | ---------------------- | --------------------------------------------------- | -| `CLAIM_STARTED` | `{ at }` | Claim request started. | -| `CLAIM_EMPTY` | `{ at }` | Claim returned no work/already claimed. | -| `CLAIM_FAILED` | `{ error, at }` | Claim failed. | -| `CLAIMED` | `{ notification, at }` | Claim returned a full wake notification. | -| `DISPATCH_SKIPPED` | `{ reason, at }` | Claim succeeded but shutdown began before dispatch. | -| `DISPATCHED` | `{ at }` | Notification was passed to `runtime.dispatchWake`. | - -## Context - -The machine context should contain every field needed to derive -`PullWakeRunnerHealth`. - -```ts -interface PullWakeRunnerMachineContext { - runnerId: string - baseUrl: string - wakeUrl: string - heartbeatUrl: string - claimUrl: string - - offset?: string - startedAt: string | null - streamConnected: boolean - streamConnectedSince: string | null - reconnectCount: number - lastError: string | null - lastErrorAt: string | null - lastHeartbeatAt: string | null - lastHeartbeatOk: boolean - lastClaimAt: string | null - lastClaimResult: 'claimed' | 'no_work' | 'error' | null - lastDispatchAt: string | null - eventsReceived: number - claimsSucceeded: number - claimsSkipped: number - claimsFailed: number - claimActors: Set> - - response: PullWakeStreamResponse | null - abortController: AbortController | null -} -``` - -Claim actors are tracked for shutdown only. The runner dispatches claimed wakes -to the runtime and does not try to limit entity wake execution concurrency. - -## `getHealth()` Mapping - -`getHealth()` reads machine state and context only. It should not inspect local -variables outside the machine. - -```ts -function getHealth(snapshot: PullWakeRunnerSnapshot): PullWakeRunnerHealth { - return { - running: snapshot.matches('running') || snapshot.matches('starting'), - offset: snapshot.context.offset, - started_at: snapshot.context.startedAt, - stream_connected: snapshot.context.streamConnected, - stream_connected_since: snapshot.context.streamConnectedSince, - reconnect_count: snapshot.context.reconnectCount, - last_error: snapshot.context.lastError, - last_error_at: snapshot.context.lastErrorAt, - last_heartbeat_at: snapshot.context.lastHeartbeatAt, - last_heartbeat_ok: snapshot.context.lastHeartbeatOk, - last_claim_at: snapshot.context.lastClaimAt, - last_claim_result: snapshot.context.lastClaimResult, - last_dispatch_at: snapshot.context.lastDispatchAt, - events_received: snapshot.context.eventsReceived, - claims_succeeded: snapshot.context.claimsSucceeded, - claims_skipped: snapshot.context.claimsSkipped, - claims_failed: snapshot.context.claimsFailed, - } -} -``` - -Heartbeat sends this same snapshot as `diagnostics`. - -## Transition Sketch - -```txt -stopped - START -> starting - -starting - entry: - - create AbortController - - set startedAt - - schedule heartbeat tick - always -> running.connecting - -running.connecting - invoke wakeStreamActor - STREAM_OPENED -> running.streaming / assign response, streamConnected=true - STREAM_ERROR -> running.reconnecting / record error, reconnectCount++ - STREAM_CLOSED -> running.reconnecting / streamConnected=false - -running - after heartbeat interval -> invoke sendHeartbeat - STOP -> stopping - -running.streaming - STREAM_EVENT -> spawn claimAndDispatch(event), eventsReceived++ - STREAM_OFFSET -> assign offset - STREAM_CLOSED -> running.reconnecting / streamConnected=false - STREAM_ERROR -> running.reconnecting / record error, reconnectCount++ - -running.reconnecting - after backoff -> running.connecting - -stopping - entry: - - abort controller - - cancel stream response - - stop accepting new claim actors - - abort in-flight claim actors - - wait for claim actors that can still dispatch to settle or skip dispatch - - runtime.abortWakes() - invoke runtime.drainWakes - done -> stopped / clear response/controller/stream state - error -> stopped / record error, report error -``` - -`claimAndDispatch` is a spawned actor, not a parent state. It sends diagnostic -events back to the runner machine. `sendHeartbeat` is different: it is a -short-lived invocation scheduled by the machine on a heartbeat interval, not a -peer long-running actor. - -## Actor Sketches - -### `wakeStreamActor` - -Input: - -- `wakeUrl` -- headers provider -- current offset -- abort signal - -Output: - -- `PullWakeStreamResponse` - -Behavior: - -- Resolve headers and open `DurableStream.stream({ live: true, json: true, offset })`. -- Emit `STREAM_OPENED` after the stream response is available. -- Iterate `response.jsonStream()`. -- For each wake event, emit `STREAM_EVENT`. -- After each iteration, if `response.offset` is defined, emit `STREAM_OFFSET`. -- On clean close, emit `STREAM_CLOSED`. -- On error, emit `STREAM_ERROR` unless stopped/aborted. -- Return only when the stream loop exits due to stop/abort, normal close, or - error. This is not a one-shot "open connection" actor; it owns consumption for - the lifetime of the connection. - -### `sendHeartbeat` - -`sendHeartbeat` is a short-lived invoked actor. The machine owns the repeated -schedule using an `after` delay or equivalent timer. The actor owns only one HTTP -request. - -Input: - -- heartbeat URL -- headers provider -- lease ms -- current offset -- `getHealth()` snapshot -- abort signal - -Behavior: - -- POST `{ lease_ms, wake_stream_offset, diagnostics }`. -- Emit `HEARTBEAT_OK` or `HEARTBEAT_ERROR`. - -### `claimAndDispatch` - -Input: - -- compact `PullWakeEvent` -- claim URL -- headers provider -- runtime dispatch function -- claim token header config -- abort signal - -Behavior: - -1. Emit `CLAIM_STARTED`. -2. POST compact wake event to claim endpoint. -3. If response is 204, 409 `ALREADY_CLAIMED`, or 409 `NO_PENDING_WORK`, emit - `CLAIM_EMPTY`. -4. If response is an error, emit `CLAIM_FAILED`. -5. If response contains `{ done: true }`, emit `CLAIM_EMPTY`. -6. Otherwise emit `CLAIMED`. -7. Check the shutdown gate. If stop has begun, do not call - `runtime.dispatchWake`; emit `DISPATCH_SKIPPED` and rely on server claim - lease expiry or an explicit release API if one exists. -8. Otherwise call - `runtime.dispatchWake(notification, { claimHeaders, claimTokenHeader })`. -9. Emit `DISPATCHED`. - -It must not call `runtime.drainWakes()`. - -## Diagnostics Updates - -| Event | Context Update | -| ------------------ | -------------------------------------------------------------------------------------------------------- | -| `START` | `startedAt = now` | -| `STREAM_OPENED` | `streamConnected = true`, `streamConnectedSince = now` | -| `STREAM_CLOSED` | `streamConnected = false`, `streamConnectedSince = null` | -| `STREAM_ERROR` | `streamConnected = false`, `streamConnectedSince = null`, `lastError`, `lastErrorAt`, `reconnectCount++` | -| `STREAM_EVENT` | `eventsReceived++` | -| `STREAM_OFFSET` | `offset = event.offset` | -| `HEARTBEAT_OK` | `lastHeartbeatAt = now`, `lastHeartbeatOk = true` | -| `HEARTBEAT_ERROR` | `lastHeartbeatAt = now`, `lastHeartbeatOk = false`, `lastError`, `lastErrorAt` | -| `CLAIM_STARTED` | `lastClaimAt = now`, `lastClaimResult = null` | -| `CLAIM_EMPTY` | `lastClaimResult = 'no_work'`, `claimsSkipped++` | -| `CLAIM_FAILED` | `lastClaimResult = 'error'`, `claimsFailed++`, `lastError`, `lastErrorAt` | -| `CLAIMED` | `lastClaimResult = 'claimed'`, `claimsSucceeded++` | -| `DISPATCH_SKIPPED` | `claimsSkipped++` | -| `DISPATCHED` | `lastDispatchAt = now` | - -## Concurrency Rules - -1. The stream reader may continue while one or more `claimAndDispatch` actors - are in flight. -2. The runner does not wait for entity wake execution after dispatch. -3. The runner does not expose a claim concurrency limit. Backpressure belongs - in runtime wake execution or the Durable Streams lease/claim contract. -4. Stop aborts future stream reads and claim requests. -5. Stop prevents any claim actor from dispatching after shutdown begins. A claim - actor that has already received a notification must either dispatch before - runtime drain begins or skip dispatch and rely on server claim lease expiry. -6. Stop gates and aborts claim actors before calling `runtime.abortWakes()` and - `runtime.drainWakes()`. -7. Claim actors must use the runner abort signal so stop can cancel in-flight - claim requests. -8. If two compact wake events race for the same work, the claim endpoint remains - authoritative. One may return `claimed`; the other may return `no_work`. -9. Offset progress follows the read-commit policy; claim actor completion does - not block `wake_stream_offset` advancement. - -## Error Handling - -`onError` is reporting-only. It exists so a host such as the Electron desktop -process can write errors to its own logs. It must not decide runner lifecycle. - -The runner should always try to stay alive until `stop()` is called. Operational -errors are written to diagnostics, reported through `onError`, and then handled -with the most local recovery action. - -Recommended handling: - -| Error Source | Recovery Behavior | -| ---------------- | ---------------------------------------------------------------------- | -| Stream open/read | Record error, increment reconnect count, transition to `reconnecting`. | -| Heartbeat | Record degraded diagnostics, continue streaming. | -| Claim | Record claim failure, continue streaming. | -| Dispatch | Record error if synchronous dispatch throws, continue streaming. | -| Stop drain | Record/report error and finish stopping. | - -`onError` callback shape: - -```ts -onError?: (error: Error) => void -``` - -The current boolean return value should be removed as part of this refactor. -Desktop introspection comes from `getHealth()` and persisted -`runners.diagnostics`, not from `onError`. - -## Public API - -The `PullWakeRunner` interface can remain source-compatible except for internal -implementation details. - -```ts -export interface PullWakeRunner { - start: () => void - stop: () => Promise - waitForStopped: () => Promise - readonly running: boolean - readonly offset: string | undefined - getHealth: () => PullWakeRunnerHealth -} -``` - -`running` should be derived from machine state: - -- true for `starting` and `running.*` -- false for `stopped` and `stopping` - -`waitForStopped()` should resolve when the interpreter reaches `stopped`. - -## Testing Requirements - -### Unit Tests - -- Starts in `stopped`; `start()` reaches `running.streaming`. -- `getHealth()` reflects state and context after start. -- Stream wake event spawns claim and dispatch without calling - `runtime.drainWakes()`. -- A second wake event can be claimed and dispatched while the first runtime wake - is still pending. -- Claim 204 and 409 no-work update `claimsSkipped`. -- Claim failure updates `claimsFailed` and `lastError`, reports through - `onError`, and does not stop stream consumption. -- Heartbeat success stores `lastHeartbeatOk = true`. -- Heartbeat failure stores `lastHeartbeatOk = false` and continues stream. -- Stream error transitions through `reconnecting` and increments - `reconnectCount`. -- `stop()` aborts stream/claims, calls `runtime.abortWakes()`, then drains. - -### Integration Tests - -- Built-in desktop runtime registers runner, heartbeats diagnostics, and the UI - receives diagnostics through the `runners` Electric shape. -- Sending to one entity while another entity is idling does not wait for the - idling entity's timeout before claim/dispatch. - -## Migration Plan - -1. Introduce the machine behind `createPullWakeRunner` without changing the - public interface. -2. Keep the existing `PullWakeRunnerHealth` shape unchanged. -3. Replace mutable local diagnostics with machine context. -4. Keep the heartbeat request body unchanged except that diagnostics now come - from `getHealth()`. -5. Update stop ordering: abort stream, stop accepting claim actors, abort or - wait for claim actors to dispatch or skip, then call `runtime.abortWakes()` - and `runtime.drainWakes()`. -6. Make `onError` reporting-only; remove the boolean lifecycle contract. -7. Add tests for non-blocking claim/dispatch before refactoring deeper. - -## Open Questions - -- Should `running` return true while `stopping` drains existing wakes? -- Should reconnect backoff remain delegated to Durable Streams, or be modeled - explicitly in this runner machine?