feat: Phase 2 — Indexing Rethink + CLI UX Overhaul#3
Merged
Conversation
Part 2.1 spike confirmed Antfly Linear Merge API and @parcel/watcher. Key finding: Linear Merge deletion is range-scoped (no delete_missing toggle), so incremental paths use batchOp instead. Added detailed implementation parts 2.2-2.8 with file-level changes, code signatures, test plans, and verification steps. Addressed all plan-reviewer findings: LocalGitExtractor preserved for dev_map, dev plan/update/explore removal, flush serialization, cache invalidation, ESM-compatible tests, dual-path state file migration. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…arts 2.2-2.3) Part 2.2: Add linearMerge() and batchUpsertAndDelete() to AntflyVectorStore and VectorStorage facade. linearMerge uses server-side content hashing for full-index dedup. batchUpsertAndDelete combines inserts + deletes for incremental updates. Part 2.3: Rewrite RepositoryIndexer to use linearMerge instead of batched addDocuments. Remove state file (indexer-state.json), update(), detectChangedFiles(), hash comparison, FileMetadata, IndexerState types. Add applyIncremental() for watcher path. Add legacy state file migration. Update getStorageFilePaths() with watcherSnapshot path. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…d services (Parts 2.6a-2.6b) Part 2.6a: Remove HistoryAdapter, GitHubAdapter, PlanAdapter from MCP server. Remove dev git, dev github, dev plan, dev update CLI commands. Clean up mcp.ts and dev-agent-mcp.ts wiring. MCP tools reduced from 9 to 6. Part 2.6b: Move LocalGitExtractor to packages/core/src/map/ (preserved for dev_map). Delete git indexer, GitHub service, GitHub agent, and all related types. Clean up PlannerAgent to remove fetchGitHubIssue dependency. Remove packages/types/src/github.ts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…p catchup (Parts 2.4-2.5) Part 2.4: Add file watcher to MCP server with 500ms debounce, serial flush queue, pending set dedup, and default ignore patterns. Incremental indexer connects watcher events to RepositoryIndexer.applyIncremental() with lazy path-to-docID cache for delete resolution. Part 2.5: On MCP server startup, query @parcel/watcher for changes since last snapshot. If snapshot missing, run full index. Otherwise run incremental update for only changed files. Write fresh snapshot after catchup completes. Catchup runs before server.start() so MCP clients get fresh data immediately. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…atus (Part 2.7) Rewrite StatusAdapter to query Antfly directly for document count and storage size. Add watcher snapshot age and auto-index status. Remove GitHub section and LanceDB references. Health check tests Antfly connectivity instead of GitHub CLI. Delete StatsService (no consumers). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Update MCP tools table from 9 to 6 adapters. Remove dev_history, dev_gh, dev_plan references. Update dev_status description to mention Antfly stats and watcher status. Update plan status. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ndex (Part 2.8) Three E2E test suites guarded by ANTFLY_INTEGRATION=true: - e2e-index-dev-agent: full index of this repo, keyword + semantic search, content hash dedup, performance targets (120s index, 500ms search) - e2e-incremental: applyIncremental upsert + delete, verify searchable - e2e-force-reindex: force clear + rebuild, verify search works after Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The spike found that `antfly swarm` has no `--port` flag. Native startup must use --metadata-api, --store-api, --metadata-raft, --store-raft, and --health-port to avoid binding to 8080 (commonly taken by Docker). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Native C++ addon cannot be bundled by esbuild. Add to externals list and as runtime dependency of the published package. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Setup: - Replace logger.info with ora spinners for consistent output - Fix spinner animation during docker pull (use spawn) - Add Docker model pull step (was missing, caused index failure) - Add Docker memory warning (<4GB) and 8GB container limit - Pass through Antfly's native progress during model download Index: - Replace ProgressRenderer with ora spinners - Auto-start Antfly if not running (no more "fetch failed") - Remove git history line from indexing plan (dead feature) - Remove buildCodeMetadata (N+1 git calls, 32s → 0s) - Add linearMerge onProgress callback for embedding progress - Better error messages with Docker/Podman memory guidance - Improved next steps with MCP install and try-it-out commands - Upgrade ora to 9.x Reset: - Add `dev reset` command to tear down Antfly and clean data Docs: - Normalize `dev index .` to `dev index` across all source and docs - Add 8GB Docker memory requirement to README, quickstart, install - Add OOM troubleshooting (exit code 137) with Podman guidance Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Setup: - Native-first, Docker via `dev setup --docker` flag only - Remove Docker from user-facing docs, keep as silent fallback in code - Update all docs to recommend Antfly native install Index: - Better OOM error: "data is safe, re-run dev index" messaging - Pre-flight model check: auto-pull if missing - Handle "model not found" errors with clear guidance MCP server: - Auto-start Antfly on MCP server startup (native first, Docker fallback) - Retry wrapper in AdapterRegistry: on Antfly connection error, auto-restart and retry once — covers server crash mid-session transparently - Human-readable error messages when recovery fails Antfly: - Detect port conflicts (lsof) with actionable guidance - Native-first priority in ensureAntfly Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
prosdev
force-pushed
the
feat/phase-2-indexing-rethink
branch
4 times, most recently
from
cf1bd89 to
a4e010d
Compare
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
prosdev
force-pushed
the
feat/phase-2-indexing-rethink
branch
from
a4e010d to
89f7b42
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 2 of the Antfly migration: rethinks the indexing pipeline and overhauls CLI UX.
Indexing Rethink (Parts 2.2–2.8)
dev_status— shows Antfly stats + watcher statusCLI UX Overhaul
buildCodeMetadata(32s N+1 git calls → 0s)dev setupuses native binary by default,--dockerflag for Dockerdev resetcommand — tear down Antfly and clean all datadev indexand MCP server auto-start if not runningdev index .→dev index— normalized across all source and docsDocs
Test plan
pnpm buildpassespnpm test— 1,630 tests pass, 0 failuresdev reset && dev setup && dev index— full flow with native Antflydev setup --docker— Docker path still worksdev indexauto-starts Antfly if not runningdev setupafter rebootdev indexwith missing model → auto-pulls before indexing