Stabilize startup persona backpressure

docs/planning/ALPHA-GAP-ANALYSIS.md

@@ -34,6 +34,7 @@ The non-negotiable gates:
 | Docker | Too much historical bulk and mixed responsibility; several open Docker issues remain | Docker can mask failures and slow iteration |
 | Rust core | Strong core exists, but GPU lifecycle, paging, and persona runtime boundaries are still incomplete | Core instability can make UI/Node fixes irrelevant |
 | Node/TS | Still owns too much cognition/command behavior | Adds latency, GC/IPC complexity, and harder cross-platform reuse |
+| Config/secrets | `$HOME/.continuum/config.env` is the local source of truth, but empty placeholders and per-process loading have caused false provider availability | Cloud providers can steal local turns and fail; grid nodes cannot yet receive encrypted config consistently |
 | Tests | Many tests exist, but the alpha loop still overuses `npm start`/browser/Docker as proof | Slow tests hide root causes and discourage TDD |
 
 ## Issue-Driven Workstreams
@@ -75,6 +76,30 @@ Implementation posture:
 - If build is unavoidable, make it explicit and resumable.
 - Install health must distinguish: network unavailable, Docker unavailable, GPU unavailable, model unavailable, Rust core unavailable, UI unavailable.
 
+### 1A. Config, Secrets, And Grid Propagation
+
+**Goal**: one authoritative config path per node, explicit encrypted propagation across trusted grid nodes, and no false "configured" state from empty placeholders.
+
+| Issue | Priority | Direction | Test gate |
+|---|---:|---|---|
+| file: config single-source issue | P0 | `SecretManager` and Rust `secrets.rs` must treat only non-empty values as configured and must lazy-load `$HOME/.continuum/config.env` before any provider check | provider status shows cloud unavailable for empty placeholders; local chat still works |
+| file: `grid/config/sync` command issue | P0 | create a command pair for encrypted config sharing over trusted grid/Tailscale nodes; no loose file copying and no browser exposure | two-node test shares selected keys, decrypts only on trusted target, and never logs values |
+| #860 config.env as directory | P1 | keep setup file/dir creation idempotent and typed | setup test catches file-vs-dir mismatch |
+
+Command shape:
+
+- `grid/config/status`: list configured key names, source path, empty placeholders, and target-node drift without values.
+- `grid/config/export`: encrypt selected config keys for a specific trusted node identity.
+- `grid/config/import`: decrypt and merge selected keys into the target node's `$HOME/.continuum/config.env`.
+- `grid/config/sync`: orchestrate export/import across trusted grid nodes and report per-node success.
+
+Rules:
+
+- Empty placeholders such as `DEEPSEEK_API_KEY=` are documentation, not availability.
+- Local mode must work with zero API keys.
+- Cloud personas are eligible only when their required key is non-empty and the provider health check is not expired/failed.
+- Config sharing is an owner/trusted-node command. It should use grid identity plus transport encryption, then persist through `SecretManager` so all runtimes see one source.
+
 ### 2. GPU Runtime Stability
 
 **Goal**: GPU resource failures degrade or recover; they do not brick the session.
@@ -141,6 +166,31 @@ Near-term PR sequence:
 | #944 embedding loop/cache misses | P1 | migrate embedding cache to shared paging primitive | repeated index pass has cache hits and bounded memory |
 | #911 16GB MacBook Air | P1 | define reduced alpha profile with strict budgets | 16GB profile starts and reports disabled features honestly |
 
+Model selection contract:
+
+- Callers request capabilities, not model IDs.
+- Discovery and admission are separate: discovery builds the catalog of model
+  artifacts, modalities, context windows, templates, quantizations, and backend
+  requirements; admission chooses the best viable candidate for the current
+  machine state and request.
+- The catalog is a curated whitelist, not arbitrary Hugging Face passthrough.
+  Candidate discovery may crawl/search HF offline or through foundry commands,
+  but runtime selection only admits vetted rows with known templates, license,
+  backend compatibility, memory estimates, modality metadata, and forge status.
+- Foundry output flows back into the same registry: `candidate` -> `vetted` ->
+  `forged` -> `published`, with Sentinel/foundry jobs updating metadata rather
+  than TS code hardcoding new model names.
+- Provider identity must be typed. Runtime local chat is `LocalRuntime`
+  (llama.cpp/Qwen through our adapter stack), cloud providers are explicit
+  external identities, and Candle is not an inference provider for persona chat.
+  Export this with `ts-rs` so TS seed/config/user paths cannot invent free-form
+  provider strings.
+- Request fields should be typed: `taskKind`, `minIntelligence`, `modalities`, `toolSupport`, `minContextTokens`, `latencyClass`, `qualityClass`, `memoryBudget`, `gpuRequired`, `familyAllowlist`, `familyPreference`, and `explicitOverride`.
+- Constraint syntax should feel like semver where it helps: exact pins for repro, `>=` for minimum intelligence/capability, `~qwen3.5` for near-family preference, ranges for context/latency/memory, and hard allow/deny lists for safety.
+- Rust registry/admission returns the selected provider/model/artifact plus explanation: why selected, why alternatives were rejected, projected VRAM/RAM/KV/LoRA footprint, and whether the choice is degraded.
+- Persona seed stores intent (`local-default`, `vision-default`, future typed capability refs), not hardcoded model strings.
+- TS may display selection state; it must not invent fallback models.
+
 Implementation order:
 
 1. PressureBroker admission gate.
@@ -219,12 +269,13 @@ Design rule:
 |---:|---|---|---|---|---|
 | 1 | `codex/alpha-gap-stability-plan` | `canary` | planning doc | this document; shared execution map | docs lint/readability, AIRC review |
 | 2 | `fix/gpu-backend-lifecycle` | `canary` | #1048, #1050, #960, #964 | mutex + backend state/recovery | Rust tests with injected failure; GPU provider evidence |
-| 3 | `fix/docker-alpha-profiles` | `canary` | #892, #955, #834, #776, #796 | modular Docker profile cleanup | compose profile smoke; image size report |
-| 4 | `feature/persona-rust-replay` | `canary` | #969, #909 | Rust persona replay/tool-loop foundation | `cargo test`; net-negative TS cognition lines |
-| 5 | `feature/pressure-broker-gate` | `canary` | #1049, #1051, #945, #944 | admission gate + first resource consumer | memory/load tests; no Node required |
-| 6 | `fix/realtime-core-reconnect` | `canary` | #793, #794, #773 | core restart + realtime browser recovery | kill core, command recovers, browser receives AI message |
-| 7 | `feature/airc-persona-peer` | `canary` | #967, PR #1046 | Continuum persona as AIRC participant | AIRC -> Continuum -> AIRC round trip |
-| 8 | `test/fresh-install-e2e` | `canary` | #770, #1006-#1008, #983 | install validation matrix | Mac + Windows logs; no silent waits |
+| 3 | `feature/grid-config-sync` | `canary` | config single-source, grid config sync | encrypted config status/export/import/sync commands | two-node encrypted config sync; provider status remains truthful |
+| 4 | `fix/docker-alpha-profiles` | `canary` | #892, #955, #834, #776, #796 | modular Docker profile cleanup | compose profile smoke; image size report |
+| 5 | `feature/persona-rust-replay` | `canary` | #969, #909 | Rust persona replay/tool-loop foundation | `cargo test`; net-negative TS cognition lines |
+| 6 | `feature/pressure-broker-gate` | `canary` | #1049, #1051, #945, #944 | admission gate + first resource consumer | memory/load tests; no Node required |
+| 7 | `fix/realtime-core-reconnect` | `canary` | #793, #794, #773 | core restart + realtime browser recovery | kill core, command recovers, browser receives AI message |
+| 8 | `feature/airc-persona-peer` | `canary` | #967, PR #1046 | Continuum persona as AIRC participant | AIRC -> Continuum -> AIRC round trip |
+| 9 | `test/fresh-install-e2e` | `canary` | #770, #1006-#1008, #983 | install validation matrix | Mac + Windows logs; no silent waits |
 
 This order can change when a blocker is discovered, but changes must be made in this document and on the issue/PR thread, not only in chat.
 

src/commands/ai/providers/status/server/AIProvidersStatusServerCommand.ts

@@ -146,8 +146,8 @@ export class AIProvidersStatusServerCommand extends AIProvidersStatusCommand {
       // positive isConfigured=true for every fresh install, leading users to
       // attempt chat and hit an opaque 401. Check the actual value length
       // instead. (#980 Bug 5.)
-      const rawKey = config.category === 'local' ? undefined : secrets.get(config.key);
-      const isConfigured = config.category === 'local' ? true : (rawKey?.length ?? 0) > 0;
+      const rawKey = config.category === 'local' ? undefined : secrets.get(config.key, 'AIProvidersStatusServerCommand');
+      const isConfigured = config.category === 'local' ? true : (rawKey?.trim().length ?? 0) > 0;
 
       return {
         provider: config.provider,

src/commands/collaboration/chat/poll/server/ChatPollServerCommand.ts

@@ -1,5 +1,5 @@
 /**
- * Chat Poll Server Command - Get messages after a specific messageId
+ * Chat Poll Server Command - Get recent messages or messages after a marker
  */
 
 import type { JTAGContext } from '@system/core/types/JTAGTypes';
@@ -29,48 +29,52 @@ export class ChatPollServerCommand extends ChatPollCommand {
         }
       }
 
-      // Get the original message to find its timestamp
-      const originalMessageResult = await ORM.query<ChatMessageEntity>({
-        collection: 'chat_messages',
-        filter: { id: params.afterMessageId },
-        limit: 1
-      }, 'default');
+      const filter: {timestamp?: {$gt: string}, roomId?: UUID} = {};
 
-      if (!originalMessageResult.success || !originalMessageResult.data || originalMessageResult.data.length === 0) {
-        return {
-          context: params.context,
-          sessionId: params.sessionId,
-          success: false,
-          messages: [],
-          count: 0,
-          afterMessageId: params.afterMessageId,
-          timestamp: new Date().toISOString(),
-          error: `Message not found: ${params.afterMessageId}`
-        };
-      }
+      if (params.afterMessageId) {
+        // Get the original message to find its timestamp.
+        const originalMessageResult = await ORM.query<ChatMessageEntity>({
+          collection: 'chat_messages',
+          filter: { id: params.afterMessageId },
+          limit: 1
+        }, 'default');
+
+        if (!originalMessageResult.success || !originalMessageResult.data || originalMessageResult.data.length === 0) {
+          return {
+            context: params.context,
+            sessionId: params.sessionId,
+            success: false,
+            messages: [],
+            count: 0,
+            afterMessageId: params.afterMessageId,
+            timestamp: new Date().toISOString(),
+            error: `Message not found: ${params.afterMessageId}`
+          };
+        }
 
-      const originalMessage = originalMessageResult.data[0];
+        const originalMessage = originalMessageResult.data[0];
 
-      // Build filter for messages after this one
-      // Convert Date to ISO string for query comparison
-      const afterTimestamp = originalMessage.data.timestamp instanceof Date
-        ? originalMessage.data.timestamp.toISOString()
-        : originalMessage.data.timestamp;
+        // Build filter for messages after this one.
+        const afterTimestamp = originalMessage.data.timestamp instanceof Date
+          ? originalMessage.data.timestamp.toISOString()
+          : originalMessage.data.timestamp;
 
-      const filter: {timestamp: {$gt: string}, roomId?: UUID} = {
-        timestamp: { $gt: afterTimestamp }
-      };
+        filter.timestamp = { $gt: afterTimestamp };
+      }
 
       // Optional room filter (from roomId or resolved room name)
       if (roomId) {
         filter.roomId = roomId;
       }
 
-      // Query messages
+      const sortDirection = params.afterMessageId ? 'asc' : 'desc';
+
+      // Query messages. No afterMessageId means "latest messages"; this is
+      // the ergonomic smoke-test/default read path for CLI and agents.
       const result = await ORM.query<ChatMessageEntity>({
         collection: 'chat_messages',
         filter,
-        sort: [{ field: 'timestamp', direction: 'asc' }],
+        sort: [{ field: 'timestamp', direction: sortDirection }],
         limit: params.limit || 50
       }, 'default');
 
@@ -87,8 +91,15 @@ export class ChatPollServerCommand extends ChatPollCommand {
         };
       }
 
-      // Extract entity data from DataRecord<ChatMessageEntity>[]
-      const messages = result.data.map(record => record.data);
+      // Extract entity data from DataRecord<ChatMessageEntity>[] and normalize
+      // latest-mode back to chronological order for display/readability.
+      const messages = result.data
+        .map(record => record.data)
+        .sort((a, b) => {
+          const aTime = new Date(a.timestamp).getTime();
+          const bTime = new Date(b.timestamp).getTime();
+          return aTime - bTime;
+        });
 
       return {
         context: params.context,

src/commands/collaboration/chat/poll/shared/ChatPollTypes.ts

@@ -1,10 +1,11 @@
 /**
- * Chat Poll Command Types - Get messages after a specific messageId
+ * Chat Poll Command Types - Get recent messages or messages after a marker
  *
  * Simple command for conversational research workflow:
  * 1. Send a question and get messageId
- * 2. Wait for responses (sleep)
- * 3. Poll for all messages after your question
+ * 2. Wait for responses
+ * 3. Poll for all messages after your question, or omit afterMessageId to
+ *    inspect the latest messages in a room.
  */
 
 import type { JTAGContext, CommandParams, JTAGPayload, CommandInput} from '@system/core/types/JTAGTypes';
@@ -21,8 +22,9 @@ export interface ChatPollParams extends CommandParams {
   readonly context: JTAGContext;
   readonly sessionId: UUID;
 
-  // Message ID to poll from (returns all messages after this one)
-  readonly afterMessageId: UUID;
+  // Optional message ID to poll from (returns messages after this one).
+  // When omitted, returns latest messages in the room.
+  readonly afterMessageId?: UUID;
 
   // Optional: limit number of messages returned
   readonly limit?: number;
@@ -41,7 +43,7 @@ export interface ChatPollResult extends JTAGPayload {
   readonly success: boolean;
   readonly messages: ReadonlyArray<ChatMessageEntity>;
   readonly count: number;
-  readonly afterMessageId: UUID;
+  readonly afterMessageId?: UUID;
   readonly timestamp: string;
   readonly error?: string;
 }
@@ -92,4 +94,3 @@ export const createCollaborationChatPollResultFromParams = (
   params: ChatPollParams,
   differences: Omit<ChatPollResult, 'context' | 'sessionId' | 'userId'>
 ): ChatPollResult => transformPayload(params, differences);
-

src/commands/data/list/server/DataListServerCommand.ts

@@ -99,10 +99,22 @@ export class DataListServerCommand<T extends BaseEntity> extends CommandBase<Dat
       };
 
       // Push column projection down to Rust when fields are specified —
-      // avoids SELECT * → IPC → TS discard pattern (DMA principle: don't move data you don't need)
-      const selectColumns = params.fields?.length ? params.fields
-        : params.select?.length ? params.select
-        : undefined;
+      // avoids SELECT * → IPC → TS discard pattern (DMA principle: don't move data you don't need).
+      // CLI callers commonly pass `--select=id`, which arrives as a string at
+      // this wire boundary despite the TypeScript type. Normalize here so
+      // readiness probes and scripts can use the cheap path without depending
+      // on fragile CLI array syntax.
+      const normalizeProjection = (value: unknown): readonly string[] | undefined => {
+        if (Array.isArray(value)) {
+          const fields = value.filter((field): field is string => typeof field === 'string' && field.length > 0);
+          return fields.length > 0 ? fields : undefined;
+        }
+        if (typeof value === 'string' && value.length > 0) {
+          return value.split(',').map(field => field.trim()).filter(Boolean);
+        }
+        return undefined;
+      };
+      const selectColumns = normalizeProjection(params.fields) ?? normalizeProjection(params.select);
 
       const storageQuery = {
         collection,
@@ -190,4 +202,4 @@ export class DataListServerCommand<T extends BaseEntity> extends CommandBase<Dat
       });
     }
   }
-}
+}

src/commands/user/create/server/UserCreateServerCommand.ts

@@ -18,8 +18,6 @@ import type { UserEntity } from '../../../../system/data/entities/UserEntity';
 import { COLLECTIONS } from '../../../../system/data/config/DatabaseConfig';
 import type { DataListParams, DataListResult } from '../../../data/list/shared/DataListTypes';
 import { createDataListParams } from '../../../data/list/shared/DataListTypes';
-import { Events } from '../../../../system/core/shared/Events';
-import { DATA_EVENTS } from '../../../../system/core/shared/EventConstants';
 
 export class UserCreateServerCommand extends UserCreateCommand {
   constructor(context: JTAGContext, subpath: string, commander: ICommandDaemon) {
@@ -71,29 +69,6 @@ export class UserCreateServerCommand extends UserCreateCommand {
           // data/list command returns items array with UserEntity objects directly
           const existingUser = existingResult.items[0];
 
-          // ON RECREATE: re-emit data:users:created so listeners (UserDaemon)
-          // re-spin runtime instances. Without this, PersonaLifecycleManager
-          // calls user/create on every boot for already-seeded personas, gets
-          // existing-user-found, the create path silently returns success, and
-          // UserDaemon's data:users:created subscription never fires — so no
-          // PersonaUser instance is constructed, no .initialize() runs, no
-          // chat subscriptions wire, and personas sit dead in the DB while
-          // PersonaLifecycleManager logs "✅ activated."
-          //
-          // Empirical regression on Linux/CUDA Carl recreate (2026-04-24):
-          // probe message stored cleanly via ORM, data:chat_messages:created
-          // fired, ZERO persona handlers triggered. Logs showed
-          // "🎭 Allocator returned 4 persona(s)" + "✅ 4 activated" but no
-          // "📢 Subscribing to chat events for N room(s)" — because the chat
-          // subscription path runs in PersonaUser.initialize() which only
-          // runs from UserDaemon.handleUserCreated.
-          //
-          // Re-emitting on existing-user-found makes the recreate path
-          // identical to the fresh-create path from UserDaemon's POV. Other
-          // listeners (RoomMembershipDaemon auto-add) are idempotent
-          // because membership checks gate on already-member.
-          Events.emit(DATA_EVENTS.USERS.CREATED, existingUser);
-
           return createUserCreateResult(params, {
             success: true,
             user: existingUser