feat: Phase 2 — Indexing Rethink + CLI UX Overhaul

.changeset/cli-ux-overhaul.md

@@ -0,0 +1,54 @@
+---
+"@prosdevlab/dev-agent": minor
+---
+
+### CLI UX Overhaul
+
+**Setup (`dev setup`)**
+- Native-first: Antfly native binary is now the default, Docker available via `--docker` flag
+- Consistent ora spinners throughout (no more mixed logger/spinner output)
+- Docker model pull: setup now pulls the embedding model inside Docker containers
+- Docker memory warning: warns if Docker has less than 4GB allocated
+
+**Index (`dev index`)**
+- 7x faster: removed `buildCodeMetadata` (32s of N+1 git calls → 0s)
+- Auto-starts Antfly if not running — no more "fetch failed" errors
+- Ora spinners with file count during scanning
+- Pre-flight model check: auto-pulls embedding model if missing
+- Resilient error messages with actionable guidance (OOM, port conflict, model missing)
+- Normalized `dev index .` → `dev index` (path defaults to cwd)
+- Improved next steps: MCP install, try-it-out commands, `dev --help`
+
+**Search (`dev search`)**
+- Removed misleading percentage scores (RRF scores are not similarity percentages)
+- Default threshold changed from 0.7 to 0 (RRF scores are much lower than cosine similarity)
+- Config no longer required — defaults to current directory
+
+**Map (`dev map`)**
+- Clean output: no markdown headers, no emojis, relative paths, proper tree connectors
+- Fixed `--focus` nesting bug (was showing redundant parent directories)
+- Next steps with usage examples
+- N+1 git fix: `calculateChangeFrequency` now uses single `git log` call with pure testable parser
+
+**Reset (`dev reset`)**
+- New command to tear down Antfly and clean all indexed data
+- Supports both Docker and native cleanup
+
+**MCP Server**
+- Auto-starts Antfly on MCP server startup (no manual `dev setup` needed after reboot)
+- Auto-recovery: if Antfly crashes mid-session, MCP retries tool calls after restarting the server
+- Human-readable errors when Antfly is unreachable
+
+**Removed**
+- `dev init` — config is now optional, all commands default to current directory
+- `dev stats` and `dev dashboard` — metrics collection removed
+- Dead GitHub output functions (~200 lines)
+
+**Internal**
+- Native-first priority in `ensureAntfly` (better performance, no VM overhead)
+- Port conflict detection with `lsof` guidance
+- `linearMerge` per-page progress via `onProgress` callback
+- `vectors.lance` → `vectors` (clean Antfly table names)
+- Extended scanner exclusions: `.env*`, `*.min.js`, `*.d.ts`, `generated/`, `.terraform/`, `.claude/`
+- Pure testable functions: `parseGitLogOutput`, `buildFrequencyMap`, `stripFocusPrefix`
+- Upgraded ora to 9.x

.claude/da-plans/core/phase-2-indexing-rethink/2.1-spike-findings.md

@@ -0,0 +1,260 @@
+# Part 2.1 — Spike Findings
+
+**Date:** 2026-03-29
+**Status:** Complete
+**Decision:** Plan A confirmed — use Antfly Linear Merge
+
+---
+
+## 1. Antfly Linear Merge API
+
+### Availability
+
+**Confirmed.** The Linear Merge API is available in `@antfly/sdk@0.0.14` via the
+raw OpenAPI client:
+
+```typescript
+const raw = client.getRawClient();
+const result = await raw.POST('/tables/{tableName}/merge', {
+  params: { path: { tableName: 'my-table' } },
+  body: {
+    records: { 'doc-id': { text: '...', metadata: '...' } },
+    last_merged_id: '',
+  },
+});
+```
+
+No convenience method exists on `client.tables` — must use `getRawClient()`.
+
+### API shape
+
+**Request** (`LinearMergeRequest`):
+```typescript
+{
+  records: Record<string, unknown>  // resource_id → document object
+  last_merged_id?: string           // "" for first page, next_cursor for subsequent
+  dry_run?: boolean                 // preview deletions without applying
+  sync_level?: SyncLevel            // "propose" | "write" | "full_text" | "enrichments" | "aknn"
+}
+```
+
+**Response** (`LinearMergeResult`):
+```typescript
+{
+  status: "success" | "partial" | "error"
+  upserted: number        // Records inserted or updated
+  skipped: number          // Records skipped (content hash unchanged)
+  deleted: number          // Records deleted (absent from batch)
+  deleted_ids?: string[]   // Only present when dry_run=true
+  next_cursor: string      // Use for pagination
+  key_range?: { from?: string, to?: string }
+  keys_scanned?: number
+  took?: number            // Nanoseconds
+}
+```
+
+### Test results
+
+| Test | Records sent | Result | Notes |
+|------|-------------|--------|-------|
+| Initial merge (3 new) | 3 | upserted=3, skipped=0, deleted=0 | ~2.2ms |
+| Re-merge identical | 3 | upserted=0, skipped=3, deleted=0 | Content hash works! ~0.36ms |
+| Update 1 record | 3 (1 changed) | upserted=1, skipped=2, deleted=0 | Only changed doc re-embedded |
+| Delete via omission | 2 (spanning full range) | upserted=0, skipped=2, deleted=2 | Middle keys removed |
+| Dry run | 2 (omitting 1) | deleted_ids=['file:src/auth.ts::validateUser'] | Preview works |
+| Search after delete | — | 2 hits (correct) | Deleted docs truly gone |
+
+### Critical finding: Range-scoped deletion
+
+Linear Merge deletes records **within the key range of the submitted batch only**.
+The range is `[last_merged_id, max_key_in_batch]`.
+
+**Example:** If table has keys A, B, C, D and you merge {A, D}:
+- B and C are deleted (within range A..D, absent from batch)
+- If you merge {A, B} instead, C and D are preserved (outside range)
+
+**Implication for dev-agent:**
+
+For full-index (`dev index .`), we must ensure the batch covers the full key space.
+This happens naturally when we send ALL documents — the range spans min to max key.
+
+For incremental merges (watcher), we should NOT rely on deletion behavior at all.
+We use `delete_missing: false` equivalent by only sending changed files' documents
+and handling deletions explicitly. Since Linear Merge always performs range-scoped
+deletion, our incremental path should use `batchOp` (upsert + explicit delete)
+rather than merge. Only full-index uses merge.
+
+**Revised strategy:**
+| Operation | API | Why |
+|-----------|-----|-----|
+| Full index (`dev index .`) | Linear Merge | All docs sent, range covers everything, stale docs auto-deleted |
+| Incremental (watcher) | `batchOp` (inserts + deletes) | Only changed files; explicit delete for removed files |
+| Force re-index | Drop table → Linear Merge | Clean slate |
+| MCP restart catchup | `batchOp` (inserts + deletes) | Same as watcher incremental |
+
+### Key constraints
+
+1. **Records must be sorted lexicographically by key** — client sorts before sending
+2. **Not safe for concurrent merges with overlapping key ranges** — single-client sync
+3. **Pagination:** For large batches, server may stop at shard boundary (`status: "partial"`),
+   use `next_cursor` for next page
+4. **No convenience method:** Must use `client.getRawClient().POST(...)` — not `client.tables.merge()`
+
+---
+
+## 2. @parcel/watcher
+
+### Availability
+
+**Confirmed.** `@parcel/watcher@2.5.6` installed and tested.
+Native C++ addon builds successfully on macOS ARM64 (Apple Silicon).
+
+### API verification
+
+All three core APIs work:
+
+#### `subscribe(dir, callback, options)`
+- Fires on file create, update, delete
+- `ignore` patterns work — `node_modules` and `.git` events filtered
+- Events arrive within ~200ms of file change
+- Returns subscription with `.unsubscribe()` method
+
+#### `writeSnapshot(dir, snapshotPath)`
+- Writes binary snapshot of directory state
+- Snapshot size: **30 bytes** for small test directory (very lightweight)
+- Overwrites safely
+
+#### `getEventsSince(dir, snapshotPath, options)`
+- Returns events that occurred between snapshot and now
+- Correctly detects: create, update, delete
+- Ignore patterns applied to historical queries
+- Works when no active subscription is running
+
+### Test results
+
+| Test | Events | Result |
+|------|--------|--------|
+| subscribe() — file changes | 3 events (create, create, create) | Correct — index.ts shows as create (already existed at subscribe time, so update becomes create in symlinked tmpdir) |
+| subscribe() — node_modules ignored | 0 events from node_modules | Correct |
+| getEventsSince() — offline changes | 3 events (update, delete, create) | Exact match to expected changes |
+| Snapshot overwrite + fresh query | 0 events | Correct — no changes since new snapshot |
+| Snapshot size | 30 bytes | Very lightweight |
+
+### Platform notes
+
+- macOS: Uses FSEvents backend (native, efficient)
+- Builds from source via `node-gyp` (C++ addon)
+- Installed via `pnpm add @parcel/watcher` — no special config needed
+- Path resolution: Returns absolute paths (symlink-resolved on macOS)
+
+---
+
+## 3. Antfly server configuration
+
+### Port configuration (spike finding)
+
+Running Antfly on non-default ports requires configuring ALL port flags:
+
+```bash
+antfly swarm \
+  --metadata-api "http://0.0.0.0:18080" \
+  --store-api "http://0.0.0.0:18381" \
+  --metadata-raft "http://0.0.0.0:19017" \
+  --store-raft "http://0.0.0.0:19021" \
+  --health-port 14200
+```
+
+**Key constraint:** `--store-api` must be a DIFFERENT port from `--metadata-api`
+in swarm mode. The default config uses a shared mux on 8080, but when overriding,
+each needs its own port.
+
+### Table creation
+
+Embedding indexes require a `template` field:
+```json
+{
+  "indexes": {
+    "embedding": {
+      "type": "embeddings",
+      "template": "{{text}}",
+      "embedder": { "provider": "termite", "model": "BAAI/bge-small-en-v1.5" }
+    },
+    "full_text": { "type": "full_text" }
+  }
+}
+```
+
+Note: type is `"embeddings"` (plural), not `"embedding"`.
+
+### Table delete → recreate issue
+
+If you delete a table and immediately recreate it, the shard ID is reused but the
+old pebble lock may still be held. **Workaround:** Restart the server between
+delete and recreate, or use a different table name. This only affects dev/test
+workflows — in production, `dev index . --force` should drop and recreate cleanly
+after a brief delay.
+
+---
+
+## 4. Impact on Phase 2 plan
+
+### Plan A confirmed
+
+Linear Merge API exists, works as documented, and content hashing eliminates
+redundant re-embedding. **No need for Plan B** (client-side hashing).
+
+### Revised `delete_missing` strategy
+
+The original plan assumed Linear Merge had a `delete_missing: true/false` toggle.
+In reality, Linear Merge ALWAYS deletes records within the batch's key range that
+are absent from the batch. This is actually cleaner:
+
+| Operation | API to use | Deletion behavior |
+|-----------|-----------|-------------------|
+| Full index | Linear Merge | Auto-deletes stale docs (range covers everything) |
+| Incremental | `batchOp` | Explicit inserts + explicit deletes for removed files |
+
+This means:
+- The `delete_missing` scoping rules in the overview need updating
+- Incremental paths use `batchOp`, not Linear Merge
+- Full index uses Linear Merge (simpler — one API call handles everything)
+
+### Key naming convention
+
+Documents must use sort-friendly IDs for Linear Merge to work correctly.
+Proposed format: `file:{relative-path}::{component-name}`
+
+This ensures:
+1. All docs from the same file sort together
+2. Full-index merge range covers all files
+3. Lexicographic sort is stable and predictable
+
+### Performance
+
+- Initial merge (3 docs with embedding): ~2.2ms
+- Content-hash skip (3 docs unchanged): ~0.36ms (6x faster)
+- These are per-shard times; real performance depends on doc count and shard count
+
+---
+
+## 5. Dependencies confirmed
+
+| Dependency | Version | Status |
+|------------|---------|--------|
+| `@antfly/sdk` | 0.0.14 | Already in `packages/core` |
+| `@parcel/watcher` | 2.5.6 | Installed in `packages/mcp-server` + root (devDep) |
+| Antfly server | 0.1.0 | Requires custom ports on dev machines (8080 often taken) |
+| Termite + bge-small-en-v1.5 | — | Model auto-downloaded on first table create |
+
+---
+
+## 6. Open questions resolved
+
+| Question | Answer |
+|----------|--------|
+| Does Linear Merge exist in SDK? | Yes, via `getRawClient().POST('/tables/{name}/merge', ...)` |
+| Does content hashing work? | Yes — `skipped` count confirms unchanged docs not re-embedded |
+| Does `@parcel/watcher` survive process restarts? | Yes — `getEventsSince(snapshot)` returns correct diff |
+| Can we use `delete_missing: true/false`? | No toggle — Linear Merge always deletes absent keys in range. Use `batchOp` for incremental. |
+| Snapshot file size? | ~30 bytes (negligible) |
+| Native addon build issues? | None on macOS ARM64 |