diff --git a/.changeset/antfly-migration.md b/.changeset/antfly-migration.md new file mode 100644 index 0000000..c4bb085 --- /dev/null +++ b/.changeset/antfly-migration.md @@ -0,0 +1,11 @@ +--- +"@prosdevlab/dev-agent": minor +--- + +Replace LanceDB + @xenova/transformers with Antfly for hybrid search + +- **Hybrid search**: `dev_search` now uses BM25 + vector + RRF fusion — exact keyword matches AND semantic understanding in one query +- **New command**: `dev setup` handles search backend installation (Docker-first, native fallback) +- **Auto-embedding**: Antfly generates embeddings locally via Termite — no separate embedding pipeline +- **Direct key lookup**: Replaces O(n) zero-vector scan with instant key fetch +- **Breaking**: Requires Antfly server running (`dev setup` handles this). Existing LanceDB indexes are not migrated — run `dev index . --force` to rebuild. diff --git a/.claude/da-plans/README.md b/.claude/da-plans/README.md index f436ee8..3f7e9c4 100644 --- a/.claude/da-plans/README.md +++ b/.claude/da-plans/README.md @@ -9,9 +9,9 @@ Implementation deviations are logged at the bottom of each plan file. | Track | Description | Status | |-------|-------------|--------| -| [Core](core/) | Scanner, vector storage, services, indexer | Not started | +| [Core](core/) | Scanner, vector storage, services, indexer | Phase 1: Draft | | [CLI](cli/) | Command-line interface | Not started | -| [MCP Server](mcp-server/) | Model Context Protocol server + adapters | Not started | +| [MCP Server](mcp-server/) | Model Context Protocol server + adapters | Phase 1: Draft (blocked on core/phase-1) | | [Subagents](subagents/) | Coordinator, explorer, planner, GitHub agents | Not started | | [Integrations](integrations/) | Claude Code, VS Code, Cursor | Not started | | [Logger](logger/) | @prosdevlab/kero centralized logging | Not started | diff --git a/.claude/da-plans/core/phase-1-antfly-migration/1.1-spike-findings.md b/.claude/da-plans/core/phase-1-antfly-migration/1.1-spike-findings.md new file mode 100644 index 0000000..5e2d7cd --- /dev/null +++ b/.claude/da-plans/core/phase-1-antfly-migration/1.1-spike-findings.md @@ -0,0 +1,122 @@ +# Part 1.1 — Spike Findings + +**Date:** 2026-03-29 +**Antfly version:** 0.1.0 (native binary, macOS ARM64) +**SDK version:** @antfly/sdk 0.0.14 + +## Results + +| # | Question | Answer | +|---|----------|--------| +| 1 | Does batch insert overwrite existing keys (upsert)? | **Yes.** Re-inserting same key overwrites the document. Confirmed via lookup after upsert. | +| 2 | How long does background embedding take? | **~2 seconds** for a single document to become searchable. First batch (10 docs) searchable within 5-8s. | +| 3 | Can we query immediately after insert? | **No — ~2s delay.** Embeddings are generated asynchronously. `dev index` should wait or poll for completion. | +| 4 | What does `client.tables.get()` return? | Returns table info including `storage_status.disk_usage` (bytes), index configs, and shard info. **No direct doc count** — need to use a query with limit to count. | +| 5 | Latency of lookup vs vector search? | Lookup is near-instant. Semantic search ~1-2ms for 10 docs. Both fast at this scale. | +| 6 | Can we full-scan without a query vector? | **Yes** — use the global `/api/v1/query` endpoint with just `table` and `limit`, no `semantic_search`. Returns all docs. | +| 7 | Does the SDK handle connection errors gracefully? | **SDK works fine via ESM** (our default). CJS build has a bug with `openapi-fetch` default export — only affects CJS consumers. See SDK notes below. | +| 8 | What happens when antfly server is not running? | curl gets `ECONNREFUSED`. Clear and fast failure. | +| 9 | Does `getAll()` paginate beyond 10000 docs? | Not tested at scale in this spike. The query endpoint accepts `limit` — likely works up to a reasonable size. Need to test with a real repo index. | +| 10 | Does `dev index` need to wait for embedding completion? | **Yes.** There's a ~2s delay between insert and searchability. For a full index run, we should wait for all embeddings to complete before declaring success. Poll embedding status or add a brief wait. | + +## API Endpoint Reference (verified) + +| Operation | Method | Endpoint | +|-----------|--------|----------| +| Create table | POST | `/api/v1/tables/{name}` | +| Get table info | GET | `/api/v1/tables/{name}` | +| Drop table | DELETE | `/api/v1/tables/{name}` | +| List tables | GET | `/api/v1/tables` | +| Batch insert/delete | POST | `/api/v1/tables/{name}/batch` | +| Lookup by key | GET | `/api/v1/tables/{name}/lookup/{key}` | +| Query (table-specific) | POST | `/api/v1/tables/{name}/query` | +| Query (global) | POST | `/api/v1/query` | + +**Important:** The global query endpoint (`/api/v1/query`) returns results in `responses[0].hits.hits[]` format. Table-specific query (`/api/v1/tables/{name}/query`) returns in `hits.hits[]` format. + +## Key Findings + +### 1. Table creation auto-creates full-text index + +When creating a table with an embeddings index, antfly automatically adds a +`full_text_index_v0` full-text index. This means **every table gets hybrid search +for free** — no extra configuration needed. + +### 2. Hybrid search with RRF works beautifully + +Tested: `semantic_search: "error handling and retry"` + `full_text_search: "retryWithBackoff"` + +Result: `func-retryBackoff` ranked #1 with scores from BOTH BM25 and vector similarity. +The `_index_scores` object shows which indexes contributed. RRF doubled its score vs +semantic-only results. This is exactly the upgrade we wanted for `dev_search`. + +### 3. Document structure is flexible (schemaless) + +Documents are JSON objects. No predefined schema required. We can store `text`, `metadata`, +`type`, `file`, `line` — whatever we want. The embedding index uses the `template` field +(Handlebars) to know which field(s) to embed. + +### 4. Embedding model confirmed: bge-small-en-v1.5, dimension 384 + +Table info shows `dimension: 384` and `model: BAAI/bge-small-en-v1.5`. Same dimension +as our current all-MiniLM-L6-v2 (384), so result structures don't change. + +Note: i8 variant 404'd during model pull. f32 variant (127.8MB) works. The plan should +use default variant (no `--variants i8` flag) until i8 is fixed. + +### 5. Lookup by key replaces O(n) zero-vector hack + +`GET /api/v1/tables/{name}/lookup/{key}` returns the document directly. Returns 404 if +not found. This is a massive improvement over the current `get()` implementation in +`LanceDBVectorStore` which does a full vector scan with a zero vector. + +### 6. Storage info available + +`client.tables.get()` returns `storage_status.disk_usage` in bytes. This can replace +the `storageSize` field in `VectorStats` (currently reads local LanceDB directory). + +## SDK Notes + +### CJS build has a bug (doesn't affect us) + +The SDK's CJS bundle (`dist/index.cjs`) fails because `openapi-fetch` is ESM-only. +`tsup` wraps it with `__toESM(require("openapi-fetch"))` and accesses `.default`, +which is `undefined` in CJS context. + +**This doesn't affect dev-agent.** All our packages use `"type": "module"` (ESM). +The ESM import path (`dist/index.js`) works correctly. + +The spike error was from `npx tsx` which loaded the CJS path — not representative +of our actual runtime. + +**Recommendation:** Use `@antfly/sdk` directly. It's type-safe, auto-generated from +OpenAPI spec, and works fine via ESM. Worth mentioning the CJS bug to the antfly +team (fix: `noExternal: ['openapi-fetch']` in tsup config) for other consumers. + +## Docker Findings + +### `ghcr.io/antflydb/antfly:omni` +- No ARM64 image available. Runs under Rosetta with `--platform linux/amd64`. +- Pull succeeded but entrypoint errored: `Error: unknown flag: --api-url` + +### `ghcr.io/antflydb/antfly:latest` +- Pulls successfully on ARM64 (via amd64 emulation) +- Does NOT auto-start — just shows help. Needs explicit `swarm` command. +- Would need: `docker run -d ... ghcr.io/antflydb/antfly:latest swarm` + +### Port conflict on native +- `antfly swarm` binds to ports 8080, 9017, 9021, 12380, 11433 +- If any are occupied (e.g., old Docker container), it crashes with `bind: address already in use` +- Docker is preferred because it isolates ports inside the container + +**Recommendation:** Docker-first with `antfly swarm` as the command, native fallback. +Need to verify Docker image + `swarm` command works end-to-end. + +## Impact on Plan + +1. **Use `@antfly/sdk` directly** — ESM works fine, type-safe, auto-generated from OpenAPI +2. **Model pull: use default variant** (not `--variants i8`) until i8 is fixed +3. **`dev index` must wait for embeddings** — poll or add brief sleep after batch insert +4. **Table info provides disk_usage** — can populate `VectorStats.storageSize` +5. **Auto full-text index** — every table gets BM25 for free, simplifies table creation +6. **Docker needs `swarm` command** — `docker run ... antfly swarm` not just `docker run ... antfly` diff --git a/.claude/da-plans/core/phase-1-antfly-migration/1.1-spike-validate-api.md b/.claude/da-plans/core/phase-1-antfly-migration/1.1-spike-validate-api.md new file mode 100644 index 0000000..bb888b7 --- /dev/null +++ b/.claude/da-plans/core/phase-1-antfly-migration/1.1-spike-validate-api.md @@ -0,0 +1,154 @@ +# Part 1.1 — Spike: Validate Antfly API + +## Goal + +Install antfly locally and confirm it can satisfy every operation our `VectorStore` interface +needs. This is a throwaway spike — no code is committed. + +## Prerequisites + +```bash +brew install --cask antflydb/antfly/antfly +antfly termite pull --variants i8 BAAI/bge-small-en-v1.5 +antfly swarm +``` + +## Tasks + +### 1. Create a table with embedding index + +```typescript +import { AntflyClient } from '@antfly/sdk'; + +const client = new AntflyClient({ baseUrl: 'http://localhost:8080' }); + +await client.tables.create('spike-code', { + indexes: { + content: { + type: 'embeddings', + template: '{{text}}', + embedder: { + provider: 'termite', + model: 'BAAI/bge-small-en-v1.5', + }, + }, + }, +}); +``` + +**Confirm:** Table created, index active. + +### 2. Batch insert documents + +Insert 100 code documents matching our `EmbeddingDocument` shape: + +```typescript +const inserts: Record = {}; +for (const doc of documents) { + inserts[doc.id] = { + text: doc.text, + metadata: JSON.stringify(doc.metadata), + }; +} +await client.tables.batch('spike-code', { inserts }); +``` + +**Confirm:** Documents inserted. Check background embedding progress: +```bash +antfly index list --table spike-code +``` + +### 3. Test upsert behavior + +Re-insert a document with the same key but different text. + +**Confirm:** Does it overwrite? Or error? We need overwrite (upsert) semantics. + +### 4. Run hybrid search + +```typescript +const results = await client.query({ + table: 'spike-code', + semantic_search: 'authentication middleware', + full_text_search: { query: 'validateUser' }, + indexes: ['content'], + fields: ['text', 'metadata'], + limit: 10, +}); +``` + +**Confirm:** Results come back. Both semantic and keyword matches appear. + +### 5. Test semantic-only search + +```typescript +const results = await client.query({ + table: 'spike-code', + semantic_search: 'error handling patterns', + indexes: ['content'], + limit: 10, +}); +``` + +**Confirm:** Pure semantic search works (our current default path). + +### 6. Test key lookup + +```typescript +const doc = await client.tables.lookup('spike-code', 'some-doc-id'); +``` + +**Confirm:** Returns the document by key. Fast (not a vector scan). + +### 7. Test batch delete + +```typescript +await client.tables.batch('spike-code', { deletes: ['doc-1', 'doc-2'] }); +``` + +**Confirm:** Documents removed. Search no longer returns them. + +### 8. Test count / table stats + +```typescript +const info = await client.tables.get('spike-code'); +``` + +**Confirm:** Can we get document count from table info? + +### 9. Test full scan (no query vector) + +```typescript +const all = await client.tables.query('spike-code', { limit: 1000 }); +// or +const all = await client.query({ table: 'spike-code', limit: 1000 }); +``` + +**Confirm:** Can we retrieve all documents without a search query? This maps to `getAll()`. + +### 10. Test embedding availability timing + +Insert a batch, then immediately search for it. + +**Confirm:** How long until newly-inserted docs appear in search results? +If there's a delay, we need to handle this in the index command (wait for embedding completion). + +## Questions to answer + +| # | Question | Answer | +|---|----------|--------| +| 1 | Does batch insert overwrite existing keys (upsert)? | | +| 2 | How long does background embedding take for 100/1000/10000 docs? | | +| 3 | Can we query immediately after insert? | | +| 4 | What does `client.tables.get()` return? (need count) | | +| 5 | Latency of `client.tables.lookup()` vs vector search? | | +| 6 | Can we full-scan without a query vector? | | +| 7 | Does the SDK handle connection errors gracefully? | | +| 8 | What happens when antfly server is not running? | | +| 9 | Does `getAll()` paginate beyond 10000 docs? How? | | +| 10 | Does `dev index` need to wait for embedding completion before returning? | | + +## Exit criteria + +All 8 questions answered. If any answer blocks the migration, document it and reassess. +If all answers are compatible, proceed to Part 1.2. diff --git a/.claude/da-plans/core/phase-1-antfly-migration/1.2-antfly-vector-store.md b/.claude/da-plans/core/phase-1-antfly-migration/1.2-antfly-vector-store.md new file mode 100644 index 0000000..7ac36e1 --- /dev/null +++ b/.claude/da-plans/core/phase-1-antfly-migration/1.2-antfly-vector-store.md @@ -0,0 +1,198 @@ +# Part 1.2 — Implement AntflyVectorStore + +## Goal + +Create `AntflyVectorStore` class that implements the `VectorStore` interface using `@antfly/sdk`. +This is the core swap — everything else builds on it. + +## New file + +`packages/core/src/vector/antfly-store.ts` + +## Interface to implement + +From `types.ts`: + +```typescript +interface VectorStore { + readonly path: string; + initialize(): Promise; + add(documents: EmbeddingDocument[], embeddings: number[][]): Promise; + search(queryEmbedding: number[], options?: SearchOptions): Promise; + get(id: string): Promise; + delete(ids: string[]): Promise; + count(): Promise; + optimize(): Promise; + close(): Promise; +} +``` + +Plus the concrete-only methods on the current `LanceDBVectorStore`: +- `getAll(): Promise` +- `searchByDocumentId(id: string, options?): Promise` +- `clear(): Promise` + +## Design + +### Constructor + +```typescript +interface AntflyConfig { + baseUrl: string; // default: 'http://localhost:8080' + table: string; // e.g., 'dev-agent-code', 'dev-agent-git', 'dev-agent-github' + indexName: string; // e.g., 'content' + template?: string; // Handlebars template for embedding, default: '{{text}}' + model?: string; // Termite model — read from config, default: 'BAAI/bge-small-en-v1.5' +} +``` + +The `model` field comes from `~/.dev-agent/config.json`, set by `dev setup --model`. +This flows into the antfly table creation (embedding index config). + +### Search interface design (BLOCKER resolution) + +The `VectorStore.search()` interface takes `queryEmbedding: number[]`, but antfly needs +query text, not a pre-computed vector. + +**Decision:** Add `searchText()` to `AntflyVectorStore` as a concrete method (not on the +`VectorStore` interface). The `VectorStorage` facade calls `searchText()` directly since +it already receives the query as a string. + +The old `search(queryEmbedding: number[])` remains on the interface for type compatibility +but throws `Error('Use searchText() — antfly handles embeddings')` if called directly. +In practice it's never called directly — only the facade calls it, and the facade is +updated in Part 1.3 to call `searchText()` instead. + +```typescript +class AntflyVectorStore implements VectorStore { + // Interface method — kept for compatibility, not called in practice + async search(queryEmbedding: number[], options?: SearchOptions): Promise { + throw new Error('Use searchText() — antfly handles embeddings internally'); + } + + // The real search method — called by VectorStorage facade + async searchText(query: string, options?: SearchOptions): Promise { + const results = await this.client.query({ + table: this.config.table, + semantic_search: query, + indexes: [this.config.indexName], + limit: options?.limit ?? 10, + }); + return this.mapHitsToSearchResults(results.hits); + } +} +``` + +### searchByDocumentId behavioral change + +**Acknowledged tradeoff:** Currently, `searchByDocumentId` fetches the stored embedding +vector and does a vector-space nearest-neighbor search. After migration, it becomes +"lookup doc → search with its text." This may produce slightly different results because +text-based search goes through antfly's tokenization + embedding pipeline rather than using +the exact stored vector. + +In practice this should be **equivalent or better** — the text goes through the same +embedding model, and hybrid search (BM25 + vector) adds keyword matching that pure +vector search lacked. The `dev_inspect` tool (primary consumer) finds similar code files, +where text-based similarity is a natural fit. + +### Method implementations + +**`initialize()`** +- Create table with embedding index if not exists +- Handle "already exists" gracefully (idempotent) + +**`add(documents, embeddings)`** +- Ignore `embeddings` parameter — antfly auto-embeds +- Convert `EmbeddingDocument[]` to antfly batch format: `{ [id]: { text, metadata } }` +- Batch in chunks of 500 (antfly may have payload limits) + +**`searchText(query, options)`** +- Use `client.query()` with `semantic_search` (and optionally `full_text_search`) +- Map antfly `hits` to `SearchResult[]` + +**`get(id)`** +- Use `client.tables.lookup(table, id)` +- Map to `EmbeddingDocument | null` + +**`delete(ids)`** +- Use `client.tables.batch(table, { deletes: ids })` + +**`count()`** +- Use `client.tables.get(table)` and extract doc count from stats + +**`getAll()`** +- Use `client.tables.query(table, { limit: 10000 })` or paginate +- If more than 10000 docs, paginate with offset (test this in spike) + +**`searchByDocumentId(id)`** +- Lookup document by key → get its text → run `searchText()` with that text +- Note: behavioral change from vector-based to text-based similarity (see above) + +**`clear()`** +- Drop and recreate the table + +**`optimize()`** — No-op (antfly manages compaction) +**`close()`** — No-op (SDK is stateless HTTP) + +**`path` (readonly property)** +- Return the antfly base URL + table name as identifier (e.g., `http://localhost:8080/dev-agent-code`) +- Used for logging and stats, not for file I/O + +### Stats support + +`VectorStorage.getStats()` currently reads `dimension` and `modelName` from the embedder, +and `storageSize` from the local LanceDB directory. After migration: + +- `dimension` — read from antfly config (known at table creation time from model) +- `modelName` — read from antfly config (stored in `AntflyConfig.model`) +- `storageSize` — antfly manages storage; report 0 or get from `client.tables.get()` if + it exposes size stats. Spike will confirm. +- `totalDocuments` — from `count()` + +Add a `getModelInfo()` method to `AntflyVectorStore`: + +```typescript +getModelInfo(): { dimension: number; modelName: string } { + return { + dimension: MODEL_DIMENSIONS[this.config.model] ?? 384, + modelName: this.config.model ?? 'BAAI/bge-small-en-v1.5', + }; +} +``` + +## Tests + +New file: `packages/core/src/vector/__tests__/antfly-store.test.ts` + +Tests require running antfly server. Tagged with `describe.runIf(process.env.ANTFLY_URL)` +so CI runs them in the docker-based job and local devs can skip them. + +Use a dedicated test table (`test-antfly-{random}`), clean up after each test. + +| Test | Description | +|------|-------------| +| creates table on initialize | Idempotent table creation | +| inserts and retrieves documents | batch insert → lookup by key | +| upserts on duplicate key | insert key X, re-insert with different text → second version stored | +| searches by semantic query | insert → searchText → verify relevance | +| handles hybrid search | BM25 + vector returns combined results | +| deletes documents | insert → delete → lookup returns null | +| counts documents | insert N → count returns N | +| gets all documents | insert → getAll → verify all returned | +| paginates getAll for large sets | insert 100+ → getAll returns all | +| searches by document ID | insert A,B → searchByDocumentId(A) → B appears if similar | +| clears all data | insert → clear → count returns 0 | +| returns model info | getModelInfo() returns dimension + model name | +| handles empty table search | search on empty table → returns [] | +| handles missing server gracefully | Connection refused → meaningful error | +| search(embedding) throws | Direct call to search() with vector → throws with guidance | +| detects model mismatch | Table has model A, config says model B → warns user to re-index | + +## Exit criteria + +- `AntflyVectorStore` passes all tests +- `searchText()` is the primary search method, `search()` throws +- `searchByDocumentId` uses text-based similarity (behavioral change documented) +- `getModelInfo()` provides dimension + model for stats +- No antfly-specific concepts leak above this layer diff --git a/.claude/da-plans/core/phase-1-antfly-migration/1.3-update-facade.md b/.claude/da-plans/core/phase-1-antfly-migration/1.3-update-facade.md new file mode 100644 index 0000000..f9eabec --- /dev/null +++ b/.claude/da-plans/core/phase-1-antfly-migration/1.3-update-facade.md @@ -0,0 +1,151 @@ +# Part 1.3 — Update VectorStorage Facade + +## Goal + +Simplify the `VectorStorage` class to use `AntflyVectorStore` instead of +`LanceDBVectorStore` + `TransformersEmbedder`. The facade interface stays identical +so consumers don't change. + +## Files to modify + +- `packages/core/src/vector/index.ts` — VectorStorage class +- `packages/core/src/vector/types.ts` — VectorStorageConfig, VectorStats +- `packages/core/src/indexer/index.ts` — RepositoryIndexer.initialize() passes skipEmbedder + +## Current flow (VectorStorage.add) + +``` +1. Consumer calls vectorStorage.add(documents) +2. VectorStorage calls embedder.embedBatch(texts) ← REMOVE +3. VectorStorage calls store.add(documents, vectors) ← vectors ignored by antfly +``` + +## New flow (VectorStorage.add) + +``` +1. Consumer calls vectorStorage.add(documents) +2. VectorStorage calls store.add(documents) ← antfly auto-embeds +``` + +## Current flow (VectorStorage.search) + +``` +1. Consumer calls vectorStorage.search(query, options) +2. VectorStorage calls embedder.embed(query) ← REMOVE +3. VectorStorage calls store.search(vector, options) +``` + +## New flow (VectorStorage.search) + +``` +1. Consumer calls vectorStorage.search(query, options) +2. VectorStorage calls store.searchText(query, options) ← pass text directly to antfly +``` + +This is the key interface bridge: `VectorStorage.search()` still takes a `string` query +from consumers (unchanged), but now calls `AntflyVectorStore.searchText()` instead of +embedding first then calling `store.search()` with a vector. + +## Config changes + +Current `VectorStorageConfig`: +```typescript +interface VectorStorageConfig { + storePath: string; + embeddingModel?: string; + dimension?: number; +} +``` + +New: +```typescript +interface VectorStorageConfig { + antflyBaseUrl?: string; // default: 'http://localhost:8080' + antflyTable: string; // e.g., 'dev-agent-code' + embeddingModel?: string; // Termite model, default: 'BAAI/bge-small-en-v1.5' +} +``` + +## Remove: skipEmbedder + +The `initialize({ skipEmbedder?: boolean })` option exists because initializing the +@xenova/transformers embedder is slow (~2-3s model load). With antfly, there's no local +model to load — the SDK is a stateless HTTP client. So: + +- Remove `skipEmbedder` option from `initialize()` +- Update `packages/cli/src/commands/map.ts` (line 94) which passes `skipEmbedder: true` +- `initialize()` becomes just `await this.store.initialize()` + +## Update: getStats() + +Current implementation reads from embedder and local filesystem: + +```typescript +return { + totalDocuments, + storageSize, // fs.stat on LanceDB directory + dimension, // this.embedder.dimension + modelName, // this.embedder.modelName +}; +``` + +After migration: +```typescript +const modelInfo = this.store.getModelInfo(); // from AntflyVectorStore +return { + totalDocuments: await this.store.count(), + storageSize: 0, // antfly manages storage; no local dir to measure + dimension: modelInfo.dimension, + modelName: modelInfo.modelName, +}; +``` + +Note: `storageSize: 0` is acceptable — antfly manages its own storage. If antfly's +`client.tables.get()` exposes size stats (spike will confirm), we can populate it. +The field remains on `VectorStats` for backward compatibility with stats-service and +status-adapter consumers. + +## Update: barrel exports + +Current `index.ts` exports: +```typescript +export { TransformersEmbedder } from './embedder'; +export { LanceDBVectorStore } from './store'; +``` + +After migration: +```typescript +export { AntflyVectorStore } from './antfly-store'; +// TransformersEmbedder and LanceDBVectorStore removed +``` + +Consumers importing these directly (if any) need updating — but research confirmed +no external consumers import the concrete classes. + +## What to remove + +- `TransformersEmbedder` instantiation in VectorStorage constructor +- `embedBatch()` calls in `add()` pipeline +- `embed()` call in `search()` pipeline +- `skipEmbedder` option on `initialize()` +- `EmbeddingProvider` interface usage (can keep in types.ts but unused) + +## What to keep + +- VectorStorage public API (identical to consumers) +- `initialize()`, `close()`, `count()`, `getAll()`, `search()` method signatures +- Error handling and logging patterns + +## Tests + +Existing mock-based tests for consumers should pass unchanged since the VectorStorage +interface doesn't change. Update `vector.test.ts` integration test to use antfly. + +## Exit criteria + +- VectorStorage facade works with AntflyVectorStore +- No TransformersEmbedder usage remains +- `skipEmbedder` option removed, CLI map command updated +- `getStats()` returns correct dimension + modelName from AntflyVectorStore +- All consumer-level tests pass (they mock VectorStorage) +- Integration test passes against running antfly diff --git a/.claude/da-plans/core/phase-1-antfly-migration/1.4-swap-deps-tests.md b/.claude/da-plans/core/phase-1-antfly-migration/1.4-swap-deps-tests.md new file mode 100644 index 0000000..194da5e --- /dev/null +++ b/.claude/da-plans/core/phase-1-antfly-migration/1.4-swap-deps-tests.md @@ -0,0 +1,215 @@ +# Part 1.4 — Swap Dependencies, CI Infrastructure, and Tests + +## Goal + +Remove LanceDB and @xenova/transformers dependencies. Add @antfly/sdk. +Set up Docker-based CI for integration tests. Use graphweave as the test corpus. +Verify all 1900+ tests still pass. + +## Dependency changes + +### packages/core/package.json + +Remove: +```json +"@lancedb/lancedb": "^0.22.3", +"@xenova/transformers": "^2.17.2" +``` + +Add: +```json +"@antfly/sdk": "0.0.14" +``` + +### packages/dev-agent/package.json + +Remove: +```json +"@lancedb/lancedb": "^0.22.3", +"@xenova/transformers": "^2.17.2" +``` + +Add: +```json +"@antfly/sdk": "0.0.14" +``` + +### packages/dev-agent/tsup.config.ts + +Update the external array (lines 11-22): + +Remove: +```typescript +'@lancedb/lancedb', +'@xenova/transformers', +``` + +Add: +```typescript +'@antfly/sdk', +``` + +Or remove the entry entirely if @antfly/sdk can be bundled (it's a lightweight +openapi-fetch wrapper). Test both approaches — bundling is simpler for end users. + +## File cleanup + +### Remove + +- `packages/core/src/vector/store.ts` (LanceDBVectorStore) +- `packages/core/src/vector/embedder.ts` (TransformersEmbedder) +- `packages/core/src/vector/__tests__/store.test.ts` (LanceDB-specific) +- `packages/core/src/vector/__tests__/embedder.test.ts` (transformers-specific) + +### Keep (updated) + +- `packages/core/src/vector/__tests__/vector.test.ts` — repoint to antfly +- `packages/core/src/vector/__tests__/antfly-store.test.ts` — new (from Part 1.2) + +## CI infrastructure (BLOCKER resolution) + +### Docker-based integration tests (CI and local) + +Both CI and local development use the same Docker setup for consistency. + +New file: `docker-compose.test.yml` + +```yaml +services: + antfly: + image: ghcr.io/antflydb/antfly:omni + ports: + - "8080:8080" + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:8080/api/v1/status"] + interval: 5s + timeout: 3s + retries: 10 +``` + +The `omni` image includes Termite with bundled models — no separate model pull needed. + +**Local usage:** +```bash +docker compose -f docker-compose.test.yml up -d # start antfly +pnpm test:integration # run integration tests +docker compose -f docker-compose.test.yml down # cleanup +``` + +### CI workflow update + +Update `.github/workflows/ci.yml` to add an integration test job: + +```yaml +integration-tests: + runs-on: ubuntu-latest + services: + antfly: + image: ghcr.io/antflydb/antfly:omni + ports: + - 8080:8080 + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: { node-version: 22 } + - run: pnpm install + - run: pnpm build + - run: pnpm test:integration + env: + ANTFLY_URL: http://localhost:8080 +``` + +### Test tagging + +Integration tests use `describe.runIf(process.env.ANTFLY_URL)`: + +```typescript +const ANTFLY_URL = process.env.ANTFLY_URL; + +describe.runIf(ANTFLY_URL)('AntflyVectorStore', () => { + // ... tests that need a running server +}); +``` + +Add a script to `package.json`: + +```json +"test:integration": "ANTFLY_URL=http://localhost:8080 vitest run --reporter=verbose" +``` + +### Test corpus: graphweave + +For end-to-end integration tests, use [graphweave](https://github.com/prosdevlab/graphweave) +as the test codebase. It's: +- Owned by prosdevlab (no permission issues) +- A real TypeScript + Python monorepo (realistic workload) +- Open source +- Dogfooding dev-agent on a sibling project + +Clone graphweave as a git submodule or download a snapshot in CI: + +```yaml +- name: Download test corpus + run: gh repo clone prosdevlab/graphweave /tmp/graphweave -- --depth 1 +``` + +End-to-end test: index graphweave → search for known code → verify results. + +## Test audit + +### Tests that pass unchanged (mock VectorStorage) + +These inject mocked `VectorStorage` and never touch LanceDB directly: + +- `core/src/git/__tests__/indexer.test.ts` +- `core/src/services/__tests__/health-service.test.ts` +- `core/src/services/__tests__/git-history-service.test.ts` +- `subagents/src/github/__tests__/indexer.test.ts` +- `subagents/src/coordinator/__tests__/github-coordinator.integration.test.ts` +- All MCP adapter tests (mock services) + +### Tests that need updating + +- `core/src/vector/__tests__/vector.test.ts` — currently uses real LanceDB + real embedder +- `core/src/indexer/__tests__/indexer.test.ts` — if it uses real VectorStorage + +### New integration tests (require antfly) + +- `antfly-store.test.ts` — unit-level (Part 1.2) +- `e2e-graphweave.test.ts` — end-to-end: clone graphweave, index, search, verify + +## Rollback path + +Since this is a clean fork with no existing users, rollback is straightforward: +revert the commits. No data migration concerns. Old LanceDB code is in git history +if ever needed. + +## Verification + +```bash +pnpm install +pnpm build +pnpm typecheck +pnpm test # unit tests (no antfly needed) +pnpm test:integration # integration tests (antfly running) +``` + +All must pass. No `@lancedb` or `@xenova/transformers` imports remaining anywhere: + +```bash +grep -r "@lancedb" packages/ --include="*.ts" --include="*.json" +grep -r "@xenova/transformers" packages/ --include="*.ts" --include="*.json" +``` + +Both should return zero results. + +## Exit criteria + +- Old deps removed, @antfly/sdk added +- `tsup.config.ts` updated (LanceDB/@xenova externals removed) +- Old store/embedder files deleted +- Docker-based CI job runs integration tests with antfly +- graphweave used as test corpus for end-to-end tests +- All mock-based unit tests pass unchanged +- All integration tests pass against running antfly +- `pnpm build && pnpm typecheck && pnpm test` green diff --git a/.claude/da-plans/core/phase-1-antfly-migration/1.5-dev-setup-command.md b/.claude/da-plans/core/phase-1-antfly-migration/1.5-dev-setup-command.md new file mode 100644 index 0000000..a1ffc05 --- /dev/null +++ b/.claude/da-plans/core/phase-1-antfly-migration/1.5-dev-setup-command.md @@ -0,0 +1,300 @@ +# Part 1.5 — `dev setup` Command + Auto-Start + +## Goal + +The user never runs `antfly` directly. `dev setup` handles one-time installation, +and any command that needs antfly auto-starts it as a background process. + +**Docker-first, native fallback.** Prefer Docker (isolated, no port conflicts). +Fall back to native binary if Docker isn't available. + +## UX + +### First time (Docker available) + +```bash +$ dev setup + +✓ Docker found +✓ Pulling antfly image... + ghcr.io/antflydb/antfly:omni ready +✓ Starting container "dev-agent-antfly"... + Running on http://localhost:8080 + +✓ Setup complete! + + Next steps: + dev index . # Index your repository + dev mcp install --cursor # Connect to Cursor +``` + +### First time (no Docker, native fallback) + +```bash +$ dev setup + +Docker not found. Falling back to native binary. + +Antfly is not installed. Install it now? (Y/n) y + +Installing via Homebrew... +✓ Antfly v0.4.2 installed +✓ Pulling embedding model... + BAAI/bge-small-en-v1.5 ready +✓ Starting Antfly server... + Running on http://localhost:8080 + +✓ Setup complete! +``` + +### Already set up + +```bash +$ dev setup + +✓ Container "dev-agent-antfly" already running +✓ Server healthy on http://localhost:8080 + + Nothing to do — you're all set! +``` + +### Neither Docker nor native + +```bash +$ dev setup + +✗ No runtime found. + + Install one of: + Docker Desktop → https://docker.com/get-started + Antfly native → brew install --cask antflydb/antfly/antfly + + Then run `dev setup` again. +``` + +### Any command when antfly is down + +```bash +$ dev index . + +Starting Antfly server... +✓ Running on http://localhost:8080 + +✓ Scanning Repository (3.2s) + 433 files → 2,525 components +... +``` + +No error — it just starts it. Transparent. + +## Implementation + +### Auto-start utility + +New file: `packages/cli/src/utils/antfly.ts` + +Docker-first, native fallback: + +```typescript +export async function ensureAntfly(options?: { quiet?: boolean }): Promise { + const client = new AntflyClient({ baseUrl: getAntflyUrl() }); + + // 1. Check if already running (Docker or native) + try { + await client.tables.list(); + return client; + } catch { + // Not running — try to start + } + + // 2. Try Docker first (preferred — isolated, no port conflicts) + if (hasDocker()) { + if (!options?.quiet) log.info('Starting Antfly via Docker...'); + execSync( + 'docker run -d --name dev-agent-antfly -p 8080:8080 ghcr.io/antflydb/antfly:omni', + { stdio: 'pipe' } + ); + await waitForServer(getAntflyUrl(), { timeout: 30000 }); + if (!options?.quiet) log.success('Running on ' + getAntflyUrl()); + return client; + } + + // 3. Fall back to native binary + try { + execSync('antfly --version', { stdio: 'pipe' }); + } catch { + throw new Error( + 'No runtime found. Run `dev setup` or install:\n' + + ' Docker Desktop → https://docker.com/get-started\n' + + ' Native binary → brew install --cask antflydb/antfly/antfly' + ); + } + + if (!options?.quiet) log.info('Starting Antfly server...'); + const child = spawn('antfly', ['swarm'], { detached: true, stdio: 'ignore' }); + child.unref(); + await waitForServer(getAntflyUrl(), { timeout: 15000 }); + if (!options?.quiet) log.success('Running on ' + getAntflyUrl()); + + return client; +} + +function hasDocker(): boolean { + try { + execSync('docker info', { stdio: 'pipe' }); + return true; + } catch { + return false; + } +} + +function getAntflyUrl(): string { + return process.env.ANTFLY_URL || 'http://localhost:8080'; +} + +async function waitForServer(url: string, opts: { timeout: number }): Promise { + const start = Date.now(); + while (Date.now() - start < opts.timeout) { + try { + await fetch(`${url}/api/v1/status`); + return; + } catch { + await new Promise(r => setTimeout(r, 500)); + } + } + throw new Error(`Antfly server did not start within ${opts.timeout / 1000}s`); +} +``` + +### `dev setup` command + +New file: `packages/cli/src/commands/setup.ts` + +```typescript +commander + .command('setup') + .description('One-time setup: install search backend and embedding model') + .option('--model ', 'Termite embedding model', 'BAAI/bge-small-en-v1.5') + .action(async (opts) => { + // 1. Docker path (preferred) + if (hasDocker()) { + log.success('Docker found'); + // Pull image if needed + execSync('docker pull ghcr.io/antflydb/antfly:omni', { stdio: 'inherit' }); + // Start container (ensureAntfly handles this) + await ensureAntfly(); + // Save runtime preference + saveConfig({ antflyRuntime: 'docker', embeddingModel: opts.model }); + log.success('Setup complete!'); + return; + } + + // 2. Native fallback + log.info('Docker not found. Falling back to native binary.'); + + try { + const version = execSync('antfly --version', { stdio: 'pipe' }).toString().trim(); + log.success(`Antfly ${version} found`); + } catch { + const installCmd = process.platform === 'darwin' + ? 'brew install --cask antflydb/antfly/antfly' + : 'curl -fsSL https://releases.antfly.io/antfly/latest/install.sh | sh -s -- --omni'; + + const answer = await confirm('Antfly is not installed. Install it now?'); + if (answer) { + execSync(installCmd, { stdio: 'inherit' }); + log.success('Antfly installed'); + } else { + log.info(`Install manually: ${installCmd}`); + return; + } + } + + // 3. Pull embedding model (native only — Docker image bundles models) + // 4. ensureAntfly() — starts server + // 5. Save config + saveConfig({ antflyRuntime: 'native', embeddingModel: opts.model }); + }); +``` + +Step 3 (model pull — native path only) uses the selected model: + +```typescript +const model = opts.model; // default: 'BAAI/bge-small-en-v1.5' +const models = execSync('antfly termite list', { stdio: 'pipe' }).toString(); +if (!models.includes(model.split('/').pop())) { + log.info(`Pulling embedding model: ${model}...`); + execSync(`antfly termite pull --variants i8 ${model}`, { stdio: 'inherit' }); +} +// Save model choice to config for table creation +saveConfig({ embeddingModel: model }); +log.success(`Embedding model ready: ${model}`); +``` + +### Model selection + +Users can pick a model at setup time or change it later: + +```bash +# At setup (default: bge-small-en-v1.5) +dev setup --model mxbai-embed-large-v1 + +# Change model later (requires re-index) +dev setup --model BAAI/bge-small-en-v1.5 +dev index . --force +``` + +Available Termite models (local ONNX): +- `BAAI/bge-small-en-v1.5` — fast, good quality (default) +- `mxbai-embed-large-v1` — slower, better quality +- `openai/clip-vit-base-patch32` — multimodal (images) + +The chosen model is stored in `~/.dev-agent/config.json` and used when creating +antfly tables. Changing models requires a full re-index since embeddings are incompatible. + +### Wire into existing commands + +Every command that needs antfly calls `ensureAntfly()` before proceeding: + +```typescript +// packages/cli/src/commands/index.ts +const client = await ensureAntfly(); +// ... proceed with indexing + +// packages/mcp-server/bin/dev-agent-mcp.ts +const client = await ensureAntfly({ quiet: true }); +// ... start MCP server +``` + +If antfly binary is missing, the error tells them to run `dev setup`. +If antfly is installed but not running, it auto-starts silently. + +### Server lifecycle + +- `dev setup` / `dev index` / `dev mcp start` all auto-start antfly if needed +- antfly runs as a detached background process — survives terminal close +- No explicit stop command needed (antfly is lightweight, ~50MB RSS) +- If user wants to stop: `pkill antfly` or system restart +- `ANTFLY_URL` env var overrides the default localhost:8080 + +## Tests + +| Test | Description | +|------|-------------| +| ensureAntfly — server already running | Mock fetch success → returns client, no spawn | +| ensureAntfly — server not running, binary exists | Mock fetch fail → spawn called → poll succeeds | +| ensureAntfly — binary missing | Mock execSync throw → error with install instructions | +| ensureAntfly — server fails to start in time | Mock poll timeout → error message | +| ensureAntfly — concurrent calls | Two simultaneous calls → only one spawns, second detects running server | +| dev setup — model missing | Mock termite list → triggers pull | +| dev setup — everything ready | All checks pass → "Nothing to do" | + +## Exit criteria + +- `dev setup` works end-to-end on macOS +- `dev index .` auto-starts antfly if not running (no user intervention) +- `dev mcp start` auto-starts antfly if not running +- Antfly binary missing → clear error pointing to install instructions +- No user-facing command requires typing `antfly` directly +- `ANTFLY_URL` env var works for custom setups +- Concurrent auto-start is safe (check port before spawning, brief retry delay) diff --git a/.claude/da-plans/core/phase-1-antfly-migration/1.6-documentation.md b/.claude/da-plans/core/phase-1-antfly-migration/1.6-documentation.md new file mode 100644 index 0000000..ac8869e --- /dev/null +++ b/.claude/da-plans/core/phase-1-antfly-migration/1.6-documentation.md @@ -0,0 +1,69 @@ +# Part 1.6 — Documentation Updates + +## Goal + +Update all documentation to reflect antfly as a prerequisite and the new +hybrid search capabilities. + +## Files to update + +### README.md + +- Add antfly to Prerequisites section +- Update Technology Stack (LanceDB → Antfly) +- Update install instructions to use `dev setup`: + ```bash + npm install -g dev-agent + dev setup # One-time: installs model, starts server + dev index . # Index your repository + dev mcp install --cursor # Connect to Cursor + ``` +- Document what `dev setup` does (so users aren't surprised): + - Checks antfly binary is installed (user must install via brew/curl) + - Pulls the embedding model (bge-small-en-v1.5 INT8, ~50MB) + - Starts antfly server in the background +- Note: all commands auto-start the server if it's not running — `dev setup` is + only needed once for the model download +- Update troubleshooting: antfly connection issues + +### CLAUDE.md + +- Update monorepo structure description (vector storage → antfly) +- Note antfly prerequisite in commands section +- Update MCP tools table: `dev_search` now uses hybrid search + +### website/content/docs/install.mdx + +- Add antfly installation step +- Add model download step +- Note: antfly must be running for dev-agent to work + +### website/content/docs/quickstart.mdx + +- Update quick start flow to include antfly setup + +### website/content/docs/architecture.mdx + +- Update architecture diagram: LanceDB → Antfly +- Explain hybrid search (BM25 + vector + RRF) + +### website/content/docs/troubleshooting.mdx + +- Add section: "Antfly server not running" +- Add section: "Old LanceDB data detected" +- Update "Vector storage initialization failed" section + +### website/content/index.mdx + +- Update "How it works" diagram +- Update "Features" to highlight hybrid search + +### website/content/updates/index.mdx + +- Add release notes for the antfly migration version + +## Exit criteria + +- All docs reflect antfly as prerequisite +- No mentions of LanceDB or @xenova/transformers in current docs (changelogs exempt) +- New user can follow install → index → search flow successfully diff --git a/.claude/da-plans/core/phase-1-antfly-migration/overview.md b/.claude/da-plans/core/phase-1-antfly-migration/overview.md new file mode 100644 index 0000000..b265691 --- /dev/null +++ b/.claude/da-plans/core/phase-1-antfly-migration/overview.md @@ -0,0 +1,348 @@ +# Phase 1: Migrate from LanceDB + @xenova/transformers to Antfly + +## Context + +dev-agent currently uses **LanceDB** for vector storage and **@xenova/transformers** for local +embeddings (all-MiniLM-L6-v2, 384-dim). These are wired together via a `VectorStorage` facade +in `packages/core/src/vector/`. + +[Antfly](https://antfly.io) is an AI database that combines vector search, BM25 full-text search, +local embedding generation (Termite), chunking, and reranking into a single service. It runs +locally, has a TypeScript SDK (`@antfly/sdk`), and is open source. + +### Why migrate + +| Current (LanceDB + transformers) | Antfly | +|----------------------------------|--------| +| Vector-only similarity search | Hybrid search: BM25 + vector + RRF/RSF fusion | +| Manual embedding pipeline (@xenova/transformers) | Auto-embedding on insert via Termite (ONNX, local) | +| Custom upsert logic (mergeInsert) | Batch insert/delete by key | +| No full-text fallback | BM25 for exact keyword matches (error codes, function names) | +| No reranking | Cross-encoder reranking built in | +| No result pruning | Score-based pruning (min_score_ratio, gap detection) | +| ~1200 lines of vector plumbing | SDK handles storage + embeddings + search | + +**Biggest win:** `dev_search` goes from pure vector similarity to hybrid search (BM25 + vector + RRF). +For code search, this is a massive upgrade — exact matches on function names AND semantic understanding +of what code does, fused into one ranked result set. + +### What exists today + +**Three separate vector stores** at runtime: + +| Store | Path | Content | Created by | +|-------|------|---------|------------| +| vectors/ | `~/.dev-agent/indexes/{repo}/vectors` | Code components (functions, classes, types) | `RepositoryIndexer` | +| vectors-git/ | `~/.dev-agent/indexes/{repo}/vectors-git` | Git commits | `GitIndexer` | +| vectors-github/ | `~/.dev-agent/indexes/{repo}/vectors-github` | GitHub issues/PRs | `GitHubIndexer` | + +Each goes through the same pipeline: +``` +Scanner/Extractor → EmbeddingDocument → TransformersEmbedder.embedBatch() → LanceDBVectorStore.add() +``` + +**After migration**, these become three antfly tables in one server: +``` +Scanner/Extractor → JSON document → AntflyClient.tables.batch() → auto-embedded by Termite +``` + +--- + +## Architecture + +### Current layers + +``` +┌─────────────────────────────────────────────┐ +│ Consumers (CLI, MCP, Services, Subagents) │ +├─────────────────────────────────────────────┤ +│ VectorStorage (facade) │ ← packages/core/src/vector/index.ts +│ ├── TransformersEmbedder │ ← packages/core/src/vector/embedder.ts +│ └── LanceDBVectorStore │ ← packages/core/src/vector/store.ts +├─────────────────────────────────────────────┤ +│ Interfaces: VectorStore, EmbeddingProvider │ ← packages/core/src/vector/types.ts +└─────────────────────────────────────────────┘ +``` + +### After migration + +``` +┌─────────────────────────────────────────────┐ +│ Consumers (CLI, MCP, Services, Subagents) │ ← NO CHANGES +├─────────────────────────────────────────────┤ +│ VectorStorage (facade — simplified) │ ← delegates to AntflyVectorStore +│ └── AntflyVectorStore │ ← replaces both embedder + store +│ └── @antfly/sdk (AntflyClient) │ +├─────────────────────────────────────────────┤ +│ Interfaces: VectorStore │ ← EmbeddingProvider removed or no-op'd +└─────────────────────────────────────────────┘ +│ │ +│ antfly server (local, runs Termite) │ ← `antfly swarm` — manages embeddings, +│ │ search, storage, reranking +└─────────────────────────────────────────────┘ +``` + +**Key insight:** nothing above `VectorStorage` changes. The facade preserves the interface contract. +Indexers, services, CLI, and MCP all consume `VectorStorage` — they don't know what's underneath. + +--- + +## Surface area (from research) + +### Files that change + +| File | Change | Reason | +|------|--------|--------| +| `core/src/vector/store.ts` | **Rewrite** | LanceDBVectorStore → AntflyVectorStore | +| `core/src/vector/embedder.ts` | **Remove or no-op** | Antfly handles embeddings via Termite | +| `core/src/vector/index.ts` | **Simplify** | VectorStorage facade drops embedder orchestration | +| `core/src/vector/types.ts` | **Update** | Config changes (storePath → antfly connection), EmbeddingProvider optional | +| `core/package.json` | **Swap deps** | Remove @lancedb/lancedb, @xenova/transformers; add @antfly/sdk | +| `dev-agent/package.json` | **Swap deps** | Same for bundled binary | +| `core/src/vector/__tests__/*` | **Rewrite** | New integration tests against antfly | +| `dev-agent/tsup.config.ts` | **Update** | Remove @lancedb/@xenova externals, add @antfly/sdk | +| `.github/workflows/ci.yml` | **Update** | Add docker-based integration test job with antfly | +| `cli/src/commands/map.ts` | **Update** | Remove `skipEmbedder: true` (no longer needed) | +| `cli/src/commands/setup.ts` | **New** | `dev setup` command (Part 1.5) | + +### Files that DON'T change (32+ consumers) + +All indexers, services, CLI commands, MCP adapters, and subagents consume `VectorStorage` +via the interface contract. None of them import LanceDB or transformers directly. + +Verified consumers (no changes needed): +- `core/src/indexer/index.ts` — RepositoryIndexer +- `core/src/git/indexer.ts` — GitIndexer +- `subagents/src/github/indexer.ts` — GitHubIndexer +- `core/src/services/search-service.ts` — SearchService +- `core/src/services/health-service.ts` — HealthService +- `core/src/services/git-history-service.ts` — GitHistoryService +- `cli/src/commands/index.ts` — CLI index command +- `mcp-server/bin/dev-agent-mcp.ts` — MCP server entry + +--- + +## Antfly API mapping + +### Table creation (one per store type) + +```typescript +// Code components table +await client.tables.create('dev-agent-code', { + indexes: { + content: { + type: 'embeddings', + template: '{{text}}', + embedder: { + provider: 'termite', + model: 'BAAI/bge-small-en-v1.5', + }, + }, + }, +}); +``` + +Three tables: `dev-agent-code`, `dev-agent-git`, `dev-agent-github` — replacing three +separate LanceDB directories. + +### Method mapping + +| Our VectorStore method | Antfly SDK | Notes | +|------------------------|-----------|-------| +| `initialize()` | `client.tables.create(table, { indexes })` | Creates table + embedding index if not exists | +| `add(docs, embeddings)` | `client.tables.batch(table, { inserts: { [id]: fields } })` | **Embeddings auto-generated** — we pass text, antfly embeds via Termite | +| `search(embedding, opts)` | `client.query({ table, semantic_search: text, full_text_search?, limit })` | Hybrid search! We pass the query TEXT, not a vector. Antfly embeds + searches | +| `get(id)` | `client.tables.lookup(table, id)` | Direct key lookup — replaces O(n) zero-vector hack | +| `delete(ids)` | `client.tables.batch(table, { deletes: ids })` | Batch delete by key | +| `count()` | `client.tables.get(table)` → stats | Table info likely includes doc count | +| `optimize()` | No-op | Antfly manages compaction internally | +| `close()` | No-op | SDK is stateless HTTP client | +| `getAll()` | `client.tables.query(table, { limit: large })` or `client.query(...)` | Full scan without vector query | +| `searchByDocumentId(id)` | Lookup doc → query with its text | Two-step: fetch doc, then `semantic_search` with its text | +| `EmbeddingProvider.embed(text)` | **Not needed** | Antfly embeds at query time | +| `EmbeddingProvider.embedBatch(texts)` | **Not needed** | Antfly embeds at insert time | + +### The big decision: let Antfly own embeddings + +**Yes.** This is the right call. Benefits: + +1. **No separate embedding pipeline to maintain** — drop @xenova/transformers entirely +2. **Auto re-embedding when you swap models** — change the index config, antfly re-embeds +3. **Consistent embedding** — same model for indexing and querying, guaranteed +4. **Background processing** — antfly embeds asynchronously on insert, no blocking +5. **Model flexibility** — swap bge-small-en-v1.5 for a larger model later without code changes + +Tradeoff: we lose control over embedding timing. But since antfly embeds in the background +and we can monitor progress via `antfly index list`, this is manageable. + +### Hybrid search upgrade for dev_search + +Current `dev_search`: +``` +query → embed(query) → vector similarity → results +``` + +After migration: +``` +query → antfly hybrid search (BM25 + vector + RRF) → rerank → prune → results +``` + +This means: +- Searching "validateUser" finds the exact function name (BM25) AND semantically related auth code (vector) +- Searching "authentication middleware" finds related concepts even without exact keyword matches +- RRF fusion combines both signals without tuning +- Optional: add reranking + pruning for even better precision + +--- + +## Parts + +| Part | Description | Risk | Commits | +|------|-------------|------|---------| +| 1.1 | Spike: install antfly, create table, insert, search, validate API | Low | 0 (throwaway) | +| 1.2 | Implement AntflyVectorStore class | Medium | 1-2 | +| 1.3 | Update VectorStorage facade, drop embedder dependency | Low | 1 | +| 1.4 | Swap dependencies, update integration tests | Medium | 1 | +| 1.5 | `dev setup` command + antfly health check | Low | 1 | +| 1.6 | Update docs, README, CLAUDE.md with new prerequisites | Low | 1 | + +### Part 1.1 — Spike (no code committed) + +Install antfly locally, pull bge-small-en-v1.5 model, and manually test: + +```bash +brew install --cask antflydb/antfly/antfly +antfly termite pull --variants i8 BAAI/bge-small-en-v1.5 +antfly swarm +``` + +Then write a throwaway script that: +1. Creates a table with an embedding index +2. Batch-inserts 100 code documents +3. Runs a hybrid search query +4. Runs a key lookup +5. Runs a delete + re-insert (upsert simulation) +6. Confirms count/stats + +**Resolve these open questions:** +- Does batch insert with an existing key overwrite (upsert) or error? +- How long does background embedding take for 100 docs? 1000? 10000? +- Can we query immediately after insert or do we need to wait for embedding? +- What does `client.tables.get()` return? (need count) +- What's the latency of `client.tables.lookup()` vs current O(n) get? + +### Part 1.2 — AntflyVectorStore + +New file: `packages/core/src/vector/antfly-store.ts` + +Implements `VectorStore` interface. Key design: +- Constructor takes `{ baseUrl, table }` config +- `initialize()` creates table with embedding index if not exists +- `add()` converts `EmbeddingDocument[]` to antfly batch insert (ignores embeddings param — antfly generates them) +- `search()` takes query TEXT (not vector), uses `client.query()` with hybrid search +- `get()` uses `client.tables.lookup()` +- `delete()` uses `client.tables.batch({ deletes })` + +**Interface change:** `search()` currently takes `queryEmbedding: number[]`. After migration +it should take `queryText: string` since antfly embeds the query. This is an interface-level +change — but since VectorStorage facade controls the call, we can handle the translation there. + +### Part 1.3 — Update VectorStorage facade + +Simplify `VectorStorage` (currently in `index.ts`): +- Remove `TransformersEmbedder` instantiation +- Remove `embedBatch()` calls in `add()` pipeline +- Change `search()` to pass query text instead of embedding +- Keep the facade interface identical to consumers + +### Part 1.4 — Swap deps + tests + +- Remove `@lancedb/lancedb` and `@xenova/transformers` from both package.json files +- Add `@antfly/sdk` +- Update vector integration tests to use real antfly (requires running server) +- Existing mock-based tests (indexer, services) should pass unchanged + +### Part 1.5 — `dev setup` command + auto-start + +One-time `dev setup`: checks antfly binary, pulls embedding model, starts server. +All commands auto-start antfly if not running — user never types `antfly` directly. +If antfly binary is missing, clear error with install instructions. + +### Part 1.6 — Documentation + +- Update README: add antfly as prerequisite +- Update CLAUDE.md: mention antfly +- Update doc site install page +- Update troubleshooting guide + +--- + +## Decisions + +| Decision | Rationale | Alternatives | +|----------|-----------|-------------| +| Let Antfly own embeddings (drop @xenova/transformers) | Eliminates embedding pipeline, auto re-embed on model swap, consistent index/query | Keep transformers and pass pre-computed vectors (more control, more plumbing) | +| Default to bge-small-en-v1.5 via Termite, allow user to choose | Good default for speed/quality balance. Advanced users can pick a larger model (mxbai-embed-large-v1) or use external providers via Termite. Model stored in config, used at table creation. | Hardcode model (inflexible), expose full Termite config (too complex for most users) | +| Three antfly tables (not one with type field) | Clean separation, independent index configs, matches current architecture | One table with metadata filtering (complicates queries, mixes schemas) | +| Use hybrid search (BM25 + vector + RRF) by default | Strictly better for code search — exact matches + semantic, no tuning needed | Pure vector (current, inferior), pure BM25 (misses semantics) | +| CLI fully owns antfly lifecycle | `dev setup` handles one-time install check + model pull. All commands auto-start the server if not running. User never types `antfly` directly. | Require user to manage antfly manually (leaky abstraction), setup-only start without auto-start (fragile) | +| No feature flag | Clean cut — antfly is strictly better; dual-backend adds complexity for no benefit | Ship behind --backend flag (delays migration, doubles test surface) | +| Require re-index on upgrade | Different embedding model (bge-small-en vs MiniLM) means vectors are incompatible | Dual-read from old + new (massive complexity for temporary benefit) | + +--- + +## Risk register + +| Risk | Likelihood | Impact | Mitigation | +|------|-----------|--------|------------| +| Antfly server must be running for dev-agent to work | Certain | Medium | CLI auto-starts via `ensureAntfly()`; `dev setup` handles first-time install | +| Antfly SDK is early (v0.0.14) — API may change | Medium | Medium | Pin SDK version; wrapped behind our VectorStore interface anyway | +| Background embedding means queries may miss recently-inserted docs | Medium | Low | Monitor embedding progress; optionally wait for completion in index command | +| Search result ranking changes (hybrid vs pure vector) | Certain | Low | Expected and desired — hybrid should be better; benchmark to confirm | +| `searchByDocumentId` behavior changes (vector → text-based) | Certain | Low | Text-based similarity + hybrid search should be equivalent or better for code | +| antfly swarm startup time adds latency to first command | Low | Low | Auto-start polls for readiness; MCP health check already exists | +| Need to rollback | Low | Low | Revert commits — clean fork, no existing users, no data migration | + +--- + +## Test strategy + +| Test | Priority | Description | +|------|----------|-------------| +| `antfly-store.test.ts` | P0 | AntflyVectorStore: create, insert, search, lookup, delete against running antfly | +| `vector.test.ts` (updated) | P0 | VectorStorage facade: end-to-end insert → search via antfly | +| Existing mock-based tests | P0 | All indexer/service tests use mocked VectorStorage — should pass unchanged | +| Hybrid search quality | P1 | Compare dev_search results before/after for 10 known queries | +| `dev index .` end-to-end | P1 | Full indexing pipeline with antfly backend | +| Performance benchmark | P2 | Index time + search latency: LanceDB vs antfly on this repo | +| Background embedding timing | P2 | Measure delay between insert and searchability | + +--- + +## Verification checklist + +- [ ] `pnpm build` passes +- [ ] `pnpm typecheck` passes +- [ ] `pnpm test` — all existing tests pass (mock-based tests unchanged) +- [ ] `dev index .` works end-to-end with antfly +- [ ] `dev_search` returns relevant results via MCP (hybrid search) +- [ ] `dev_search "validateUser"` finds exact function (BM25 signal) +- [ ] `dev_search "authentication middleware"` finds related code (vector signal) +- [ ] `dev_history` returns relevant git commits +- [ ] `dev_gh` returns relevant GitHub issues +- [ ] `client.tables.lookup()` works for `dev_refs` and `dev_inspect` +- [ ] No `@lancedb/lancedb` or `@xenova/transformers` in any package.json +- [ ] `dev setup` works end-to-end (checks binary, pulls model, starts server) +- [ ] `dev index .` auto-starts antfly if not running (no user intervention) +- [ ] README and doc site document `dev setup` and what it does (no surprises) +- [ ] Antfly binary missing → clear error with install instructions + +--- + +## Dependencies + +- **antfly binary** — `dev setup` prompts to install automatically (brew on macOS, curl on Linux) with user confirmation +- **Termite model** — `dev setup` handles this automatically +- **antfly server** — auto-started by CLI, user never runs it directly +- **@antfly/sdk** — `pnpm add @antfly/sdk` in core + dev-agent packages +- Part 1.1 spike must be completed before Part 1.2 begins diff --git a/.claude/da-plans/mcp-server/phase-1-plugin-architecture/overview.md b/.claude/da-plans/mcp-server/phase-1-plugin-architecture/overview.md new file mode 100644 index 0000000..293c782 --- /dev/null +++ b/.claude/da-plans/mcp-server/phase-1-plugin-architecture/overview.md @@ -0,0 +1,89 @@ +# Phase 1: Plugin Architecture for MCP Adapters + +**Status:** Not started — depends on antfly migration (core/phase-1) completing first. + +## Context + +dev-agent's MCP server currently has 9 adapters that follow a class-based pattern: +extend `ToolAdapter`, implement `getToolDefinition()` + `execute()`, register manually +in `bin/dev-agent-mcp.ts`. This works but is rigid — adding adapters requires touching +multiple files, config is scattered, and there's no plugin lifecycle. + +[sdk-kit](https://github.com/lytics/sdk-kit) (a Lytics open-source project) provides a +proven plugin architecture: `use()`, `ns()`, `expose()`, `defaults()`, event-driven +coordination, and capability injection. This pattern maps naturally to MCP adapters. + +## Vision + +Each MCP adapter becomes a plugin that: +- **Registers itself** via `use()` — no manual wiring in entry point +- **Declares its config defaults** via `defaults()` — no scattered config +- **Exposes its tools** via `expose()` — type-safe tool registration +- **Emits events** — `tool:search:start`, `tool:search:complete` for observability +- **Declares dependencies** — e.g., search adapter requires vector storage + +Third-party adapters become possible: `dev mcp add @someone/dev-agent-jira-adapter`. + +## What we'd lift from sdk-kit + +| sdk-kit concept | MCP adapter equivalent | +|-----------------|----------------------| +| `plugin.ns('transport')` | `adapter.ns('dev_search')` | +| `plugin.defaults({...})` | `adapter.defaults({ limit: 10, threshold: 0.3 })` | +| `plugin.expose({ send })` | `adapter.expose({ toolDefinition, execute })` | +| `plugin.emit('transport:send')` | `adapter.emit('tool:search:start', query)` | +| `plugin.hold({ log })` | `adapter.hold({ vectorStorage })` — shared capabilities | +| `sdk.use(transportPlugin)` | `server.use(searchAdapter)` | + +## Also relevant from sdk-kit + +- **Transport plugin** — `sendWithRetry` pattern with exponential backoff, skip 4xx +- **Poll plugin** — `waitFor` pattern for async conditions (server readiness, etc.) +- **Lifecycle** — `sdk:init`, `sdk:ready`, `sdk:destroy` events for clean startup/shutdown + +## Why this matters + +1. **Community adapters** — third parties can build MCP tools as plugins +2. **Config in one place** — each adapter declares defaults, user overrides via config file +3. **Observability** — event emission gives free logging/metrics hooks +4. **Testability** — plugins are pure functions, easy to test in isolation +5. **Reduced boilerplate** — no manual registration, no entry point changes per adapter + +## Scope + +This is a significant architectural refactor: +- Core plugin system (from sdk-kit patterns) +- Rewrite 9 adapters as plugins +- New adapter registration/discovery +- Config system changes +- Event bus integration +- Third-party adapter loading + +## Parts (to be detailed) + +| Part | Description | +|------|-------------| +| 1.1 | Design: adapt sdk-kit core for MCP context (spike) | +| 1.2 | Implement plugin core (use, ns, expose, defaults, emit) | +| 1.3 | Convert SearchAdapter as first plugin (proof of concept) | +| 1.4 | Convert remaining 8 adapters | +| 1.5 | Third-party adapter loading (`dev mcp add`) | +| 1.6 | Documentation and migration guide | + +## Dependencies + +- **core/phase-1 (antfly migration)** must complete first — the vector storage layer + is a shared capability that adapters depend on, and it's changing +- **sdk-kit source** — reference implementation at github.com/lytics/sdk-kit + +## Open questions + +1. Do we vendor sdk-kit core, fork it, or depend on it as a package? +2. How do third-party adapters discover the MCP server? npm package convention? +3. Does the event bus replace or complement the existing `AsyncEventBus` in core? +4. How does this interact with the antfly client (shared capability via `hold()`)? + +--- + +*This plan will be fleshed out after the antfly migration lands. The overview here +captures the vision and key decisions so the idea isn't lost.* diff --git a/.env.example b/.env.example new file mode 100644 index 0000000..abbc660 --- /dev/null +++ b/.env.example @@ -0,0 +1,3 @@ +# Antfly server URL (used by dev-agent and tests) +# Port 18080 avoids conflicts with common dev servers on 8080 +ANTFLY_URL=http://localhost:18080/api/v1 diff --git a/CLAUDE.md b/CLAUDE.md index a13eca9..0ef6be3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,7 +4,8 @@ **Local-first repository context for AI tools — no hallucinations.** -Semantic search, code analysis, and GitHub integration through MCP. +Hybrid search (BM25 + vector), code analysis, and GitHub integration through MCP. +Powered by [Antfly](https://antfly.io) for search and embeddings. Everything runs on your machine. No data leaves. --- @@ -20,7 +21,7 @@ Everything runs on your machine. No data leaves. ``` packages/ - core/ # Scanner (ts-morph, tree-sitter), vector storage (LanceDB), services + core/ # Scanner (ts-morph, tree-sitter), vector storage (Antfly), services cli/ # Commander.js CLI — dev index, dev mcp install, etc. mcp-server/ # MCP server with 9 built-in adapters subagents/ # Coordinator, explorer, planner, PR agents @@ -59,6 +60,7 @@ pnpm format # Biome format pnpm dev # Watch mode pnpm clean # Clean build outputs pnpm changeset # Document changes for release +dev setup # One-time: start Antfly search backend ``` --- @@ -70,13 +72,16 @@ pnpm changeset # Document changes for release - **Workspace protocol** for internal deps: `"@prosdevlab/dev-agent-core": "workspace:*"` - **Tests run from root only:** `pnpm test` — centralized Vitest config. - **Logger:** Use `@prosdevlab/kero` — never `console.log` in packages. -- **Local-first:** No data sent externally. Embeddings via @xenova/transformers. +- **Local-first:** No data sent externally. Embeddings via Antfly/Termite (local ONNX). - **Code review before PR.** Always run the `code-reviewer` agent (which launches security-reviewer, logic-reviewer, and quality-reviewer in parallel) on the branch diff before creating a pull request. Address any CRITICAL or WARNING findings before merging. - **Plan before building.** For non-trivial features, write a plan in `.claude/da-plans/` and run the `plan-reviewer` agent before implementation. +- **Changesets target published packages only.** Only `@prosdevlab/dev-agent` + and `@prosdevlab/kero` are published to npm. All other packages are private + and bundled into dev-agent via tsup. Never add private packages to changesets. --- @@ -113,7 +118,7 @@ See `.claude/da-plans/README.md` for status and format details. | Tool | Purpose | |------|---------| -| `dev_search` | Semantic code search (use FIRST for conceptual queries) | +| `dev_search` | Hybrid code search — BM25 + vector + RRF (use FIRST for conceptual queries) | | `dev_refs` | Find callers/callees of functions | | `dev_map` | Codebase structure with change frequency | | `dev_history` | Semantic search over git commits | @@ -142,6 +147,7 @@ See `.claude/da-plans/README.md` for status and format details. ```bash # Common workflows pnpm install && pnpm build && pnpm test # Full setup +dev setup # One-time: start Antfly dev index . # Index repository dev mcp install # Install for Claude Code dev mcp install --cursor # Install for Cursor diff --git a/README.md b/README.md index 18463ef..a45867e 100644 --- a/README.md +++ b/README.md @@ -52,7 +52,10 @@ We benchmarked dev-agent against baseline Claude Code across 5 task types: # Install globally npm install -g dev-agent -# Index your repository (initial indexing can take 5-10 minutes for large codebases) +# One-time setup (starts search backend via Docker or native) +dev setup + +# Index your repository cd /path/to/your/repo dev index . @@ -60,9 +63,6 @@ dev index . dev mcp install --cursor # For Cursor IDE dev mcp install # For Claude Code -# Keep index up to date (fast incremental updates) -dev update - # That's it! AI tools now have access to dev-agent capabilities. ``` @@ -213,15 +213,18 @@ Check MCP server and component health. ### Prerequisites - [Node.js](https://nodejs.org/) v22 LTS or higher -- [pnpm](https://pnpm.io/) v8.15.4 or higher +- [Docker Desktop](https://docker.com/get-started) (recommended) or [Antfly](https://antfly.io) native binary - [GitHub CLI](https://cli.github.com/) (for GitHub features) ### Global Install (Recommended) ```bash npm install -g dev-agent +dev setup # One-time: starts search backend (Docker or native) ``` +`dev setup` handles everything — pulls the Docker image, starts the server, and verifies the connection. If Docker isn't available, it falls back to the native Antfly binary and offers to install it. + ### From Source ```bash @@ -363,11 +366,10 @@ dev-agent/ - **TypeScript** (strict mode) - **ts-morph** / TypeScript Compiler API (TypeScript/JS analysis) - **tree-sitter** WASM (Go analysis, extensible to Python/Rust) -- **LanceDB** (embedded vector storage) -- **@xenova/transformers** (local embeddings) +- **[Antfly](https://antfly.io)** (hybrid search: BM25 + vector + RRF, local embeddings via Termite) - **MCP** (Model Context Protocol) - **Turborepo** (monorepo builds) -- **Vitest** (1500+ tests) +- **Vitest** (1900+ tests) ## Development diff --git a/biome.json b/biome.json index 0964c24..754173a 100644 --- a/biome.json +++ b/biome.json @@ -32,6 +32,16 @@ "linter": { "enabled": false } + }, + { + "includes": ["**/vector/antfly-store.ts"], + "linter": { + "rules": { + "complexity": { + "noBannedTypes": "off" + } + } + } } ], "formatter": { diff --git a/packages/cli/src/cli.ts b/packages/cli/src/cli.ts index fa52381..a5b8a3d 100644 --- a/packages/cli/src/cli.ts +++ b/packages/cli/src/cli.ts @@ -14,6 +14,7 @@ import { mapCommand } from './commands/map.js'; import { mcpCommand } from './commands/mcp.js'; import { planCommand } from './commands/plan.js'; import { searchCommand } from './commands/search.js'; +import { setupCommand } from './commands/setup.js'; import { statsCommand } from './commands/stats.js'; import { storageCommand } from './commands/storage.js'; import { updateCommand } from './commands/update.js'; @@ -45,6 +46,7 @@ program.addCommand(compactCommand); program.addCommand(cleanCommand); program.addCommand(storageCommand); program.addCommand(mcpCommand); +program.addCommand(setupCommand); // Show help if no command provided if (process.argv.length === 2) { diff --git a/packages/cli/src/commands/commands.test.ts b/packages/cli/src/commands/commands.test.ts index 362695d..d0b1f84 100644 --- a/packages/cli/src/commands/commands.test.ts +++ b/packages/cli/src/commands/commands.test.ts @@ -2,6 +2,38 @@ import * as fs from 'node:fs/promises'; import * as os from 'node:os'; import * as path from 'node:path'; import { Command } from 'commander'; + +// Mock VectorStorage to avoid needing antfly server +vi.mock('../../../core/src/vector/index', async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + VectorStorage: class MockVectorStorage { + async initialize() {} + async addDocuments() {} + async search() { + return []; + } + async searchByDocumentId() { + return []; + } + async getAll() { + return []; + } + async getDocument() { + return null; + } + async deleteDocuments() {} + async clear() {} + async getStats() { + return { totalDocuments: 0, storageSize: 0, dimension: 384, modelName: 'mock' }; + } + async optimize() {} + async close() {} + }, + }; +}); + import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest'; import { cleanCommand } from './clean'; import { indexCommand } from './index'; diff --git a/packages/cli/src/commands/setup.ts b/packages/cli/src/commands/setup.ts new file mode 100644 index 0000000..a120caa --- /dev/null +++ b/packages/cli/src/commands/setup.ts @@ -0,0 +1,147 @@ +/** + * dev setup — One-time setup for dev-agent's search backend + * + * Docker-first, native fallback. Handles installation, model download, + * and server startup so users never need to run `antfly` directly. + */ + +import { execSync } from 'node:child_process'; +import * as readline from 'node:readline'; +import { Command } from 'commander'; +import ora from 'ora'; +import { + ensureAntfly, + getAntflyUrl, + getNativeVersion, + hasDocker, + hasModel, + hasNativeBinary, + isServerReady, + pullModel, +} from '../utils/antfly.js'; +import { logger } from '../utils/logger.js'; + +const DEFAULT_MODEL = 'BAAI/bge-small-en-v1.5'; + +async function confirm(question: string): Promise { + const rl = readline.createInterface({ input: process.stdin, output: process.stdout }); + return new Promise((resolve) => { + rl.question(`${question} (Y/n) `, (answer) => { + rl.close(); + resolve(answer.toLowerCase() !== 'n'); + }); + }); +} + +export const setupCommand = new Command('setup') + .description('One-time setup: install search backend and embedding model') + .option('--model ', 'Termite embedding model', DEFAULT_MODEL) + .action(async (options) => { + const model = options.model as string; + const spinner = ora(); + + try { + // ── Step 1: Check runtime ── + if (hasDocker()) { + logger.info('Docker found'); + + // Check if server is already running + if (await isServerReady()) { + logger.info('Antfly server already running'); + logger.log("\n Nothing to do — you're all set!\n"); + logger.log(' Next steps:'); + logger.log(' dev index . Index your repository'); + logger.log(' dev mcp install --cursor Connect to Cursor\n'); + return; + } + + // Pull image and start + spinner.start('Pulling Antfly image...'); + try { + execSync(`docker pull --platform linux/amd64 ${getDockerImage()}`, { stdio: 'pipe' }); + spinner.succeed('Antfly image ready'); + } catch { + spinner.succeed('Antfly image available'); + } + + spinner.start('Starting Antfly server...'); + await ensureAntfly({ quiet: true }); + spinner.succeed(`Antfly running on ${getAntflyUrl()}`); + } else if (hasNativeBinary()) { + // ── Native fallback ── + const version = getNativeVersion(); + logger.info(`Antfly ${version} found (native)`); + logger.info('Docker not found — using native binary'); + + // Check if server is already running + if (await isServerReady()) { + logger.info('Antfly server already running'); + } else { + // Pull embedding model (Docker image bundles models, native needs manual pull) + if (!hasModel(model)) { + spinner.start(`Pulling embedding model: ${model}...`); + pullModel(model); + spinner.succeed(`Embedding model ready: ${model}`); + } else { + logger.info(`Embedding model ready: ${model}`); + } + + spinner.start('Starting Antfly server...'); + await ensureAntfly({ quiet: true }); + spinner.succeed(`Antfly running on ${getAntflyUrl()}`); + } + } else { + // ── Nothing installed ── + const platform = process.platform; + const installCmd = + platform === 'darwin' + ? 'brew install --cask antflydb/antfly/antfly' + : 'curl -fsSL https://releases.antfly.io/antfly/latest/install.sh | sh -s -- --omni'; + + if (hasDocker === undefined) { + // This shouldn't happen but just in case + logger.error('No runtime found.'); + } + + const shouldInstall = await confirm('\nAntfly is not installed. Install it now?'); + + if (shouldInstall) { + spinner.start( + `Installing via ${platform === 'darwin' ? 'Homebrew' : 'install script'}...` + ); + execSync(installCmd, { stdio: 'inherit' }); + spinner.succeed('Antfly installed'); + + // Pull model and start + if (!hasModel(model)) { + spinner.start(`Pulling embedding model: ${model}...`); + pullModel(model); + spinner.succeed(`Embedding model ready: ${model}`); + } + + spinner.start('Starting Antfly server...'); + await ensureAntfly({ quiet: true }); + spinner.succeed(`Antfly running on ${getAntflyUrl()}`); + } else { + logger.log('\nInstall manually, then run `dev setup` again:'); + logger.log(` Docker: https://docker.com/get-started`); + logger.log(` Native: ${installCmd}\n`); + return; + } + } + + // ── Success ── + logger.log('\n Setup complete!\n'); + logger.log(' Next steps:'); + logger.log(' dev index . Index your repository'); + logger.log(' dev mcp install --cursor Connect to Cursor\n'); + } catch (error) { + spinner.fail('Setup failed'); + logger.error(error instanceof Error ? error.message : String(error)); + process.exit(1); + } + }); + +function getDockerImage(): string { + return 'ghcr.io/antflydb/antfly:latest'; +} diff --git a/packages/cli/src/utils/antfly.ts b/packages/cli/src/utils/antfly.ts new file mode 100644 index 0000000..45a32f6 --- /dev/null +++ b/packages/cli/src/utils/antfly.ts @@ -0,0 +1,160 @@ +/** + * Antfly server lifecycle management + * + * Docker-first, native fallback. The user never needs to run `antfly` directly. + */ + +import { execSync, spawn } from 'node:child_process'; +import { logger } from './logger.js'; + +const DEFAULT_ANTFLY_URL = process.env.ANTFLY_URL ?? 'http://localhost:18080/api/v1'; +const CONTAINER_NAME = 'dev-agent-antfly'; +const DOCKER_IMAGE = 'ghcr.io/antflydb/antfly:latest'; +const DOCKER_PORT = 18080; +const STARTUP_TIMEOUT_MS = 30_000; +const POLL_INTERVAL_MS = 500; + +/** + * Ensure antfly is running. Auto-starts if needed. + * + * Priority: Docker container → native binary → error with guidance. + */ +export async function ensureAntfly(options?: { quiet?: boolean }): Promise { + const url = getAntflyUrl(); + + // 1. Already running? + if (await isServerReady(url)) { + return url; + } + + // 2. Try Docker first + if (hasDocker()) { + if (isContainerExists()) { + if (!options?.quiet) logger.info('Starting Antfly container...'); + execSync(`docker start ${CONTAINER_NAME}`, { stdio: 'pipe' }); + } else { + if (!options?.quiet) logger.info('Starting Antfly via Docker...'); + execSync( + `docker run -d --name ${CONTAINER_NAME} -p ${DOCKER_PORT}:8080 --platform linux/amd64 ${DOCKER_IMAGE} swarm`, + { stdio: 'pipe' } + ); + } + + await waitForServer(url); + if (!options?.quiet) logger.info(`Antfly running on ${url}`); + return url; + } + + // 3. Native fallback + if (hasNativeBinary()) { + if (!options?.quiet) logger.info('Starting Antfly server...'); + const child = spawn('antfly', ['swarm'], { + detached: true, + stdio: 'ignore', + }); + child.unref(); + + await waitForServer(url); + if (!options?.quiet) logger.info(`Antfly running on ${url}`); + return url; + } + + // 4. Nothing available + throw new Error( + 'Antfly is not installed. Run `dev setup` to install, or:\n' + + ' Docker: docker pull ghcr.io/antflydb/antfly:latest\n' + + ' Native: brew install --cask antflydb/antfly/antfly' + ); +} + +export function getAntflyUrl(): string { + return process.env.ANTFLY_URL ?? DEFAULT_ANTFLY_URL; +} + +export function hasDocker(): boolean { + try { + execSync('docker info', { stdio: 'pipe', timeout: 5000 }); + return true; + } catch { + return false; + } +} + +export function hasNativeBinary(): boolean { + try { + execSync('antfly --version', { stdio: 'pipe', timeout: 5000 }); + return true; + } catch { + return false; + } +} + +export function isContainerExists(): boolean { + try { + const result = execSync(`docker ps -a --filter name=${CONTAINER_NAME} --format "{{.Names}}"`, { + encoding: 'utf-8', + stdio: ['pipe', 'pipe', 'pipe'], + }); + return result.trim() === CONTAINER_NAME; + } catch { + return false; + } +} + +export async function isServerReady(url?: string): Promise { + const baseUrl = (url ?? getAntflyUrl()).replace('/api/v1', ''); + try { + const resp = await fetch(`${baseUrl}/api/v1/tables`, { signal: AbortSignal.timeout(3000) }); + return resp.ok; + } catch { + return false; + } +} + +async function waitForServer(url: string): Promise { + const start = Date.now(); + while (Date.now() - start < STARTUP_TIMEOUT_MS) { + if (await isServerReady(url)) return; + await new Promise((r) => setTimeout(r, POLL_INTERVAL_MS)); + } + throw new Error( + `Antfly server did not start within ${STARTUP_TIMEOUT_MS / 1000}s. Check: docker logs ${CONTAINER_NAME}` + ); +} + +/** + * Get the antfly version (native binary). + */ +export function getNativeVersion(): string | null { + try { + return execSync('antfly --version', { + encoding: 'utf-8', + stdio: ['pipe', 'pipe', 'pipe'], + }).trim(); + } catch { + return null; + } +} + +/** + * Pull a Termite embedding model (native binary only). + */ +export function pullModel(model: string): void { + execSync(`antfly termite pull ${model}`, { stdio: 'inherit' }); +} + +/** + * Check if a Termite model is available locally (native binary only). + */ +export function hasModel(model: string): boolean { + try { + const output = execSync('antfly termite list', { + encoding: 'utf-8', + stdio: ['pipe', 'pipe', 'pipe'], + }); + const shortName = model.split('/').pop() ?? model; + return output.includes(shortName); + } catch { + return false; + } +} diff --git a/packages/core/package.json b/packages/core/package.json index 13a29d2..c9134cd 100644 --- a/packages/core/package.json +++ b/packages/core/package.json @@ -37,10 +37,9 @@ "typescript": "^5.3.3" }, "dependencies": { - "@lancedb/lancedb": "^0.22.3", + "@antfly/sdk": "0.0.14", "@prosdevlab/dev-agent-types": "workspace:*", "@prosdevlab/kero": "workspace:*", - "@xenova/transformers": "^2.17.2", "better-sqlite3": "^12.5.0", "globby": "^16.0.0", "remark": "^15.0.1", diff --git a/packages/core/src/indexer/__tests__/detailed-stats.integration.test.ts b/packages/core/src/indexer/__tests__/detailed-stats.integration.test.ts index 2226a28..6839bdb 100644 --- a/packages/core/src/indexer/__tests__/detailed-stats.integration.test.ts +++ b/packages/core/src/indexer/__tests__/detailed-stats.integration.test.ts @@ -1,7 +1,11 @@ import * as fs from 'node:fs/promises'; import * as os from 'node:os'; import * as path from 'node:path'; -import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +// Use mock VectorStorage (no antfly server needed) +vi.mock('../../vector/index'); + import { RepositoryIndexer } from '../index'; import type { DetailedIndexStats } from '../types'; diff --git a/packages/core/src/indexer/__tests__/indexer-edge.test.ts b/packages/core/src/indexer/__tests__/indexer-edge.test.ts index 8c06da9..b07c8c5 100644 --- a/packages/core/src/indexer/__tests__/indexer-edge.test.ts +++ b/packages/core/src/indexer/__tests__/indexer-edge.test.ts @@ -2,7 +2,11 @@ import * as fs from 'node:fs/promises'; import * as os from 'node:os'; import * as path from 'node:path'; -import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest'; + +// Use mock VectorStorage (no antfly server needed) +vi.mock('../../vector/index'); + import { RepositoryIndexer } from '../index'; /** diff --git a/packages/core/src/indexer/__tests__/indexer.test.ts b/packages/core/src/indexer/__tests__/indexer.test.ts index 2625a31..1b2bd7d 100644 --- a/packages/core/src/indexer/__tests__/indexer.test.ts +++ b/packages/core/src/indexer/__tests__/indexer.test.ts @@ -1,7 +1,11 @@ import * as fs from 'node:fs/promises'; import * as os from 'node:os'; import * as path from 'node:path'; -import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest'; + +// Use mock VectorStorage (no antfly server needed) +vi.mock('../../vector/index'); + import { RepositoryIndexer } from '../index'; import type { IndexProgress } from '../types'; diff --git a/packages/core/src/vector/__mocks__/index.ts b/packages/core/src/vector/__mocks__/index.ts new file mode 100644 index 0000000..96f161e --- /dev/null +++ b/packages/core/src/vector/__mocks__/index.ts @@ -0,0 +1,110 @@ +/** + * Mock VectorStorage for tests that don't need a real antfly server. + * + * Used by indexer, subagent, and CLI tests that test their own logic, + * not vector storage behavior. The real antfly tests are in antfly-store.test.ts. + */ + +import { vi } from 'vitest'; + +export { type AntflyStoreConfig, AntflyVectorStore } from '../antfly-store.js'; +// Re-export real types +export * from '../types.js'; + +// In-memory document store for mock +const docs = new Map }>(); + +export class VectorStorage { + private initialized = false; + + constructor(_config: { storePath: string; embeddingModel?: string; dimension?: number }) { + // No-op — mock doesn't connect to anything + } + + async initialize(_options?: { skipEmbedder?: boolean }): Promise { + this.initialized = true; + } + + async addDocuments( + documents: Array<{ id: string; text: string; metadata: Record }> + ): Promise { + for (const doc of documents) { + docs.set(doc.id, { text: doc.text, metadata: doc.metadata }); + } + } + + async search( + _query: string, + options?: { limit?: number; scoreThreshold?: number } + ): Promise }>> { + const limit = options?.limit ?? 10; + const results = Array.from(docs.entries()) + .slice(0, limit) + .map(([id, doc]) => ({ + id, + score: 0.85, + metadata: doc.metadata, + })); + return results; + } + + async searchByDocumentId( + documentId: string, + options?: { limit?: number } + ): Promise }>> { + return this.search(documentId, options); + } + + async getAll(options?: { + limit?: number; + }): Promise }>> { + const limit = options?.limit ?? 10000; + return Array.from(docs.entries()) + .slice(0, limit) + .map(([id, doc]) => ({ + id, + score: 1, + metadata: doc.metadata, + })); + } + + async getDocument( + id: string + ): Promise<{ id: string; text: string; metadata: Record } | null> { + const doc = docs.get(id); + if (!doc) return null; + return { id, ...doc }; + } + + async deleteDocuments(ids: string[]): Promise { + for (const id of ids) { + docs.delete(id); + } + } + + async clear(): Promise { + docs.clear(); + } + + async getStats(): Promise<{ + totalDocuments: number; + storageSize: number; + dimension: number; + modelName: string; + }> { + return { + totalDocuments: docs.size, + storageSize: 0, + dimension: 384, + modelName: 'BAAI/bge-small-en-v1.5', + }; + } + + async optimize(): Promise { + // no-op + } + + async close(): Promise { + this.initialized = false; + } +} diff --git a/packages/core/src/vector/__tests__/antfly-store.test.ts b/packages/core/src/vector/__tests__/antfly-store.test.ts new file mode 100644 index 0000000..a12ae52 --- /dev/null +++ b/packages/core/src/vector/__tests__/antfly-store.test.ts @@ -0,0 +1,253 @@ +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import { AntflyVectorStore } from '../antfly-store.js'; +import type { EmbeddingDocument } from '../types.js'; + +const ANTFLY_URL = process.env.ANTFLY_URL ?? 'http://localhost:8080/api/v1'; +const TABLE = `test-antfly-${Date.now()}`; + +// Skip entire suite if antfly is not available +const isAntflyAvailable = async (): Promise => { + try { + const resp = await fetch(`${ANTFLY_URL}/tables`); + return resp.ok; + } catch { + return false; + } +}; + +function makeDocs(count: number, prefix = 'doc'): EmbeddingDocument[] { + const snippets = [ + 'export function authenticate(token: string): boolean { return jwt.verify(token, SECRET); }', + 'export function validateUser(userId: string): Promise { return db.users.findOne({ id: userId }); }', + 'export function handleError(err: Error): Response { logger.error(err); return new Response(err.message, { status: 500 }); }', + 'export async function retryWithBackoff(fn: () => Promise, maxRetries = 3): Promise { /* backoff */ }', + 'export function searchDocuments(query: string, limit = 10): Promise { return vectorStore.search(query); }', + 'export class RateLimiter { private tokens: number; consume(): boolean { return this.tokens-- > 0; } }', + 'export class EventBus { private listeners = new Map(); emit(event: string, data: unknown) { /* emit */ } }', + 'export async function healthCheck(): Promise { return Promise.all([checkDB(), checkVector()]); }', + 'export function indexRepository(path: string): Promise { return vectorStore.add(scanner.scan(path)); }', + 'export function parseConfig(path: string): Config { return JSON.parse(fs.readFileSync(path, "utf-8")); }', + ]; + + return Array.from({ length: count }, (_, i) => ({ + id: `${prefix}-${i}`, + text: snippets[i % snippets.length], + metadata: { type: 'function', file: `src/${prefix}-${i}.ts`, line: i * 10 }, + })); +} + +function sleep(ms: number): Promise { + return new Promise((r) => setTimeout(r, ms)); +} + +describe.runIf(await isAntflyAvailable())('AntflyVectorStore', () => { + let store: AntflyVectorStore; + + beforeAll(async () => { + store = new AntflyVectorStore({ baseUrl: ANTFLY_URL, table: TABLE }); + await store.initialize(); + }, 30_000); + + afterAll(async () => { + try { + await store.clear(); + await store.close(); + } catch { + // Best-effort cleanup + } + }); + + it('creates table on initialize (idempotent)', async () => { + // Second initialize should not throw + const store2 = new AntflyVectorStore({ baseUrl: ANTFLY_URL, table: TABLE }); + await expect(store2.initialize()).resolves.not.toThrow(); + await store2.close(); + }); + + it('inserts and retrieves documents by key', async () => { + const docs = makeDocs(3); + await store.add(docs); + + // Wait for antfly to process + await sleep(3000); + + const result = await store.get('doc-0'); + expect(result).not.toBeNull(); + expect(result!.id).toBe('doc-0'); + expect(result!.text).toContain('authenticate'); + expect(result!.metadata).toHaveProperty('type', 'function'); + }, 15_000); + + it('upserts on duplicate key', async () => { + const original = await store.get('doc-0'); + expect(original).not.toBeNull(); + + // Re-insert same key with different text + await store.add([ + { + id: 'doc-0', + text: 'UPDATED: export function authenticate(token: string, opts?: AuthOptions): boolean { /* v2 */ }', + metadata: { type: 'function', file: 'src/auth-v2.ts', line: 1 }, + }, + ]); + + await sleep(2000); + + const updated = await store.get('doc-0'); + expect(updated).not.toBeNull(); + expect(updated!.text).toContain('UPDATED'); + expect(updated!.metadata).toHaveProperty('file', 'src/auth-v2.ts'); + }, 15_000); + + it('searches by semantic query', async () => { + // Insert more docs for better search results + await store.add(makeDocs(10, 'search')); + await sleep(5000); + + const results = await store.searchText('authentication and user validation'); + expect(results.length).toBeGreaterThan(0); + expect(results[0]).toHaveProperty('id'); + expect(results[0]).toHaveProperty('score'); + expect(results[0]).toHaveProperty('metadata'); + }, 20_000); + + it('respects search limit', async () => { + const results = await store.searchText('authentication function', { limit: 3 }); + expect(results.length).toBeLessThanOrEqual(3); + }, 10_000); + + it('respects scoreThreshold', async () => { + const allResults = await store.searchText('authentication'); + const filtered = await store.searchText('authentication', { scoreThreshold: 999 }); + expect(filtered.length).toBe(0); + expect(allResults.length).toBeGreaterThan(0); + }, 10_000); + + it('deletes documents', async () => { + await store.add([ + { id: 'to-delete-1', text: 'temporary document one', metadata: {} }, + { id: 'to-delete-2', text: 'temporary document two', metadata: {} }, + ]); + await sleep(2000); + + await store.delete(['to-delete-1', 'to-delete-2']); + + const result = await store.get('to-delete-1'); + expect(result).toBeNull(); + }, 15_000); + + it('counts documents', async () => { + // Ensure data exists + await store.add(makeDocs(2, 'cnt')); + await sleep(3000); + + const count = await store.count(); + expect(count).toBeGreaterThan(0); + }, 15_000); + + it('gets all documents', async () => { + // Ensure we have data + const count = await store.count(); + if (count === 0) { + await store.add(makeDocs(3, 'getall')); + await sleep(3000); + } + + const all = await store.getAll(); + expect(all.length).toBeGreaterThan(0); + expect(all[0]).toHaveProperty('id'); + expect(all[0]).toHaveProperty('score', 1); // Full scan = score 1 + expect(all[0]).toHaveProperty('metadata'); + }, 15_000); + + it('searches by document ID', async () => { + // Ensure we have data + const count = await store.count(); + if (count === 0) { + await store.add(makeDocs(5, 'sbd')); + await sleep(5000); + } + + // Find a doc that exists + const all = await store.getAll({ limit: 1 }); + if (all.length === 0) return; // Skip if still empty + + const results = await store.searchByDocumentId(all[0].id); + expect(results.length).toBeGreaterThan(0); + }, 20_000); + + it('returns empty for searchByDocumentId with missing ID', async () => { + const results = await store.searchByDocumentId('nonexistent-id'); + expect(results).toEqual([]); + }, 10_000); + + it('returns model info', () => { + const info = store.getModelInfo(); + expect(info.dimension).toBe(384); + expect(info.modelName).toBe('BAAI/bge-small-en-v1.5'); + }); + + it('returns storage size', async () => { + const size = await store.getStorageSize(); + expect(size).toBeGreaterThanOrEqual(0); + }, 10_000); + + it('handles empty table search', async () => { + // Create a fresh empty table + const emptyTable = `test-empty-${Date.now()}`; + const emptyStore = new AntflyVectorStore({ baseUrl: ANTFLY_URL, table: emptyTable }); + await emptyStore.initialize(); + + const results = await emptyStore.searchText('anything'); + expect(results).toEqual([]); + + // Cleanup + await emptyStore.clear(); + await emptyStore.close(); + }, 15_000); + + it('search() with embedding vector throws', async () => { + await expect(store.search([0.1, 0.2, 0.3])).rejects.toThrow( + 'does not accept pre-computed embeddings' + ); + }); + + it('throws when not initialized', async () => { + const uninitialized = new AntflyVectorStore({ baseUrl: ANTFLY_URL, table: 'nope' }); + await expect(uninitialized.searchText('test')).rejects.toThrow('not initialized'); + // Empty add is a no-op (early return before assertReady) + await expect(uninitialized.add([])).resolves.not.toThrow(); + // Non-empty add should throw + await expect(uninitialized.add(makeDocs(1))).rejects.toThrow('not initialized'); + }); + + it('detects model mismatch', async () => { + const mismatchStore = new AntflyVectorStore({ + baseUrl: ANTFLY_URL, + table: TABLE, + model: 'nomic-ai/nomic-embed-text-v1.5', // Different model than the table was created with + }); + + await expect(mismatchStore.initialize()).rejects.toThrow('Model mismatch'); + }, 10_000); + + it('clears all data', async () => { + await store.clear(); + + const count = await store.count(); + expect(count).toBe(0); + + const all = await store.getAll(); + expect(all).toEqual([]); + }, 15_000); + + it('handles delete with empty array', async () => { + // Should not throw + await expect(store.delete([])).resolves.not.toThrow(); + }); + + it('handles add with empty array', async () => { + // Should not throw + await expect(store.add([])).resolves.not.toThrow(); + }); +}); diff --git a/packages/core/src/vector/__tests__/embedder.test.ts b/packages/core/src/vector/__tests__/embedder.test.ts deleted file mode 100644 index f8e0946..0000000 --- a/packages/core/src/vector/__tests__/embedder.test.ts +++ /dev/null @@ -1,145 +0,0 @@ -import { beforeAll, describe, expect, it } from 'vitest'; -import { TransformersEmbedder } from '../embedder'; - -describe('TransformersEmbedder', () => { - let embedder: TransformersEmbedder; - - beforeAll(async () => { - embedder = new TransformersEmbedder(); - await embedder.initialize(); - }, 60000); // Longer timeout for model download - - it('should have correct dimension', () => { - expect(embedder.dimension).toBe(384); - expect(embedder.modelName).toBe('Xenova/all-MiniLM-L6-v2'); - }); - - it('should initialize with custom model', () => { - const customEmbedder = new TransformersEmbedder('Xenova/all-MiniLM-L6-v2', 384); - expect(customEmbedder.dimension).toBe(384); - expect(customEmbedder.modelName).toBe('Xenova/all-MiniLM-L6-v2'); - }); - - it('should generate embeddings for single text', async () => { - const text = 'This is a test sentence'; - const embedding = await embedder.embed(text); - - expect(Array.isArray(embedding)).toBe(true); - expect(embedding.length).toBe(384); - expect(typeof embedding[0]).toBe('number'); - }); - - it('should generate consistent embeddings', async () => { - const text = 'Consistent test'; - const embedding1 = await embedder.embed(text); - const embedding2 = await embedder.embed(text); - - // Should be identical for same input - expect(embedding1.length).toBe(embedding2.length); - expect(embedding1[0]).toBeCloseTo(embedding2[0], 5); - }); - - it('should generate different embeddings for different text', async () => { - const text1 = 'First sentence'; - const text2 = 'Completely different meaning about programming'; - - const embedding1 = await embedder.embed(text1); - const embedding2 = await embedder.embed(text2); - - // Should be different - check multiple dimensions - const totalDiff = embedding1.reduce((sum, val, i) => sum + Math.abs(val - embedding2[i]), 0); - expect(totalDiff).toBeGreaterThan(1); // Significant difference across all dimensions - }); - - it('should handle empty string', async () => { - const embedding = await embedder.embed(''); - expect(Array.isArray(embedding)).toBe(true); - expect(embedding.length).toBe(384); - }); - - it('should handle long text', async () => { - const longText = 'word '.repeat(1000); - const embedding = await embedder.embed(longText); - - expect(Array.isArray(embedding)).toBe(true); - expect(embedding.length).toBe(384); - }); - - it('should generate batch embeddings', async () => { - const texts = ['First text', 'Second text', 'Third text']; - const embeddings = await embedder.embedBatch(texts); - - expect(Array.isArray(embeddings)).toBe(true); - expect(embeddings.length).toBe(3); - for (const embedding of embeddings) { - expect(embedding.length).toBe(384); - } - }); - - it('should handle empty batch', async () => { - const embeddings = await embedder.embedBatch([]); - expect(Array.isArray(embeddings)).toBe(true); - expect(embeddings.length).toBe(0); - }); - - it('should handle single item batch', async () => { - const embeddings = await embedder.embedBatch(['Single item']); - expect(embeddings.length).toBe(1); - expect(embeddings[0].length).toBe(384); - }); - - it('should handle multiple initializations', async () => { - await embedder.initialize(); - await embedder.initialize(); - // Should not throw - }); - - it('should throw error when embedding without initialization', async () => { - const uninitEmbedder = new TransformersEmbedder(); - await expect(uninitEmbedder.embed('test')).rejects.toThrow(); - }); - - it('should throw error when batch embedding without initialization', async () => { - const uninitEmbedder = new TransformersEmbedder(); - await expect(uninitEmbedder.embedBatch(['test'])).rejects.toThrow(); - }); - - it('should handle batch size configuration', () => { - embedder.setBatchSize(16); - expect(embedder.getBatchSize()).toBe(16); - - embedder.setBatchSize(32); - expect(embedder.getBatchSize()).toBe(32); - }); - - it('should reject invalid batch size', () => { - expect(() => embedder.setBatchSize(0)).toThrow(); - expect(() => embedder.setBatchSize(-1)).toThrow(); - }); - - it('should handle large batch efficiently', async () => { - const texts = Array.from({ length: 100 }, (_, i) => `Text number ${i}`); - const embeddings = await embedder.embedBatch(texts); - - expect(embeddings.length).toBe(100); - for (const embedding of embeddings) { - expect(embedding.length).toBe(384); - } - }); - - it('should handle special characters in text', async () => { - const specialText = 'Test with émojis 🚀 and spëcial çharacters @#$%'; - const embedding = await embedder.embed(specialText); - - expect(Array.isArray(embedding)).toBe(true); - expect(embedding.length).toBe(384); - }); - - it('should handle very long single token', async () => { - const longWord = 'a'.repeat(1000); - const embedding = await embedder.embed(longWord); - - expect(Array.isArray(embedding)).toBe(true); - expect(embedding.length).toBe(384); - }); -}); diff --git a/packages/core/src/vector/__tests__/store.test.ts b/packages/core/src/vector/__tests__/store.test.ts deleted file mode 100644 index b8c55af..0000000 --- a/packages/core/src/vector/__tests__/store.test.ts +++ /dev/null @@ -1,178 +0,0 @@ -/** - * Unit tests for LanceDBVectorStore - * Focus on testing the distance-to-similarity conversion bug fix - */ - -import { describe, expect, it } from 'vitest'; - -describe('LanceDB Distance to Similarity Conversion', () => { - describe('Score Calculation', () => { - /** - * This is the core bug fix being tested: - * LanceDB returns L2 distance, we need to convert to similarity score (0-1) - */ - - it('should convert L2 distance to valid similarity score', () => { - // Simulate the conversion we use: score = e^(-distance²) - const calculateScore = (distance: number): number => { - return Math.exp(-(distance * distance)); - }; - - // Test cases from the bug: - // Before fix: score = 1 - distance = 1 - 1.0 = 0 ❌ - // After fix: score = e^(-distance²) ✅ - - // Distance ~1.0 (similar vectors in normalized space) - const distance1 = 0.9999990463256836; // Real value from our testing - const score1 = calculateScore(distance1); - - expect(score1).toBeGreaterThan(0); // Should NOT be 0 (the bug!) - expect(score1).toBeLessThan(1); - expect(score1).toBeCloseTo(0.37, 1); // e^(-1²) ≈ 0.37 - }); - - it('should give high scores for low distances', () => { - const calculateScore = (distance: number): number => { - return Math.exp(-(distance * distance)); - }; - - // Very similar vectors (distance ≈ 0) - const veryClose = calculateScore(0.1); - expect(veryClose).toBeGreaterThan(0.99); // e^(-0.01) ≈ 0.99 - - // Moderately similar (distance ≈ 0.5) - const moderate = calculateScore(0.5); - expect(moderate).toBeGreaterThan(0.7); // e^(-0.25) ≈ 0.78 - - // Less similar (distance ≈ 1.0) - const less = calculateScore(1.0); - expect(less).toBeGreaterThan(0.3); // e^(-1) ≈ 0.37 - expect(less).toBeLessThan(0.4); - }); - - it('should give low scores for high distances', () => { - const calculateScore = (distance: number): number => { - return Math.exp(-(distance * distance)); - }; - - // Dissimilar vectors (distance ≈ 2.0) - const dissimilar = calculateScore(2.0); - expect(dissimilar).toBeLessThan(0.02); // e^(-4) ≈ 0.018 - - // Very dissimilar (distance ≈ 3.0) - const veryDissimilar = calculateScore(3.0); - expect(veryDissimilar).toBeLessThan(0.001); // e^(-9) ≈ 0.00012 - }); - - it('should return scores in valid range (0-1)', () => { - const calculateScore = (distance: number): number => { - return Math.exp(-(distance * distance)); - }; - - // Test a range of distances - const distances = [0, 0.1, 0.5, 1.0, 1.5, 2.0, 3.0, 5.0]; - - for (const distance of distances) { - const score = calculateScore(distance); - expect(score).toBeGreaterThanOrEqual(0); - expect(score).toBeLessThanOrEqual(1); - } - }); - - it('should be monotonically decreasing', () => { - const calculateScore = (distance: number): number => { - return Math.exp(-(distance * distance)); - }; - - // Scores should decrease as distance increases - const distances = [0, 0.5, 1.0, 1.5, 2.0]; - const scores = distances.map(calculateScore); - - for (let i = 0; i < scores.length - 1; i++) { - expect(scores[i]).toBeGreaterThan(scores[i + 1]); - } - }); - - it('should handle edge cases', () => { - const calculateScore = (distance: number): number => { - return Math.exp(-(distance * distance)); - }; - - // Distance = 0 (identical vectors) - expect(calculateScore(0)).toBe(1); - - // Very large distance - const huge = calculateScore(100); - expect(huge).toBeCloseTo(0, 10); - - // Undefined/infinity handling - const inf = calculateScore(Number.POSITIVE_INFINITY); - expect(inf).toBe(0); - }); - }); - - describe('Threshold Filtering', () => { - it('should filter out results below threshold', () => { - const results = [ - { distance: 0.5, id: 'a' }, // score ≈ 0.78 - { distance: 1.0, id: 'b' }, // score ≈ 0.37 - { distance: 1.5, id: 'c' }, // score ≈ 0.11 - { distance: 2.0, id: 'd' }, // score ≈ 0.02 - ]; - - const calculateScore = (distance: number): number => { - return Math.exp(-(distance * distance)); - }; - - const threshold = 0.3; - const filtered = results - .map((r) => ({ ...r, score: calculateScore(r.distance) })) - .filter((r) => r.score >= threshold); - - // Should keep scores >= 0.3 (distances <= ~1.0) - expect(filtered.length).toBe(2); - expect(filtered[0].id).toBe('a'); - expect(filtered[1].id).toBe('b'); - }); - - it('should keep all results with threshold 0', () => { - const results = [ - { distance: 1.0, id: 'a' }, - { distance: 2.0, id: 'b' }, - { distance: 3.0, id: 'c' }, - ]; - - const calculateScore = (distance: number): number => { - return Math.exp(-(distance * distance)); - }; - - const threshold = 0.0; - const filtered = results - .map((r) => ({ ...r, score: calculateScore(r.distance) })) - .filter((r) => r.score >= threshold); - - expect(filtered.length).toBe(3); - }); - - it('should filter out low scores with high threshold', () => { - const results = [ - { distance: 0.5, id: 'a' }, // score ≈ 0.78 - { distance: 1.0, id: 'b' }, // score ≈ 0.37 - { distance: 1.5, id: 'c' }, // score ≈ 0.11 - ]; - - const calculateScore = (distance: number): number => { - return Math.exp(-(distance * distance)); - }; - - const threshold = 0.7; - const filtered = results - .map((r) => ({ ...r, score: calculateScore(r.distance) })) - .filter((r) => r.score >= threshold); - - // Only the first result should pass - expect(filtered.length).toBe(1); - expect(filtered[0].id).toBe('a'); - }); - }); -}); diff --git a/packages/core/src/vector/__tests__/vector.test.ts b/packages/core/src/vector/__tests__/vector.test.ts deleted file mode 100644 index 4abcdf3..0000000 --- a/packages/core/src/vector/__tests__/vector.test.ts +++ /dev/null @@ -1,330 +0,0 @@ -import * as fs from 'node:fs/promises'; -import * as os from 'node:os'; -import * as path from 'node:path'; -import { afterAll, beforeAll, describe, expect, it } from 'vitest'; -import { VectorStorage } from '../index'; -import type { EmbeddingDocument } from '../types'; - -describe('Vector Storage', () => { - let vectorStorage: VectorStorage; - let testDir: string; - - beforeAll(async () => { - // Create temporary directory for tests - testDir = path.join(os.tmpdir(), `vector-test-${Date.now()}`); - await fs.mkdir(testDir, { recursive: true }); - - vectorStorage = new VectorStorage({ - storePath: path.join(testDir, 'test.lance'), - embeddingModel: 'Xenova/all-MiniLM-L6-v2', - dimension: 384, - }); - - // Initialize (downloads model on first run) - await vectorStorage.initialize(); - }, 60000); // Longer timeout for model download - - afterAll(async () => { - await vectorStorage.close(); - // Clean up test directory - await fs.rm(testDir, { recursive: true, force: true }); - }); - - it('should initialize successfully', async () => { - const stats = await vectorStorage.getStats(); - expect(stats.modelName).toBe('Xenova/all-MiniLM-L6-v2'); - expect(stats.dimension).toBe(384); - expect(stats.totalDocuments).toBe(0); - }); - - it('should add documents and generate embeddings', async () => { - const documents: EmbeddingDocument[] = [ - { - id: 'doc1', - text: 'TypeScript is a typed superset of JavaScript', - metadata: { type: 'definition', language: 'typescript' }, - }, - { - id: 'doc2', - text: 'JavaScript is a dynamic programming language', - metadata: { type: 'definition', language: 'javascript' }, - }, - { - id: 'doc3', - text: 'Python is a high-level programming language', - metadata: { type: 'definition', language: 'python' }, - }, - ]; - - await vectorStorage.addDocuments(documents); - - const stats = await vectorStorage.getStats(); - expect(stats.totalDocuments).toBe(3); - }); - - it('should search for similar documents', async () => { - const query = 'What is TypeScript?'; - const results = await vectorStorage.search(query, { limit: 2 }); - - expect(results.length).toBeLessThanOrEqual(2); - expect(results[0]).toHaveProperty('id'); - expect(results[0]).toHaveProperty('score'); - expect(results[0]).toHaveProperty('metadata'); - - // TypeScript document should be most similar - expect(results[0].id).toBe('doc1'); - expect(results[0].score).toBeGreaterThan(0); - }); - - it('should respect score threshold', async () => { - const query = 'What is Rust?'; // Not in our documents - const results = await vectorStorage.search(query, { - limit: 10, - scoreThreshold: 0.8, // High threshold - }); - - // Should return fewer or no results due to high threshold - expect(results.length).toBeLessThanOrEqual(3); - for (const result of results) { - expect(result.score).toBeGreaterThanOrEqual(0.8); - } - }); - - it('should retrieve document by ID', async () => { - const doc = await vectorStorage.getDocument('doc2'); - - expect(doc).toBeDefined(); - expect(doc?.id).toBe('doc2'); - expect(doc?.text).toContain('JavaScript'); - expect(doc?.metadata.language).toBe('javascript'); - }); - - it('should return null for non-existent document', async () => { - const doc = await vectorStorage.getDocument('nonexistent'); - expect(doc).toBeNull(); - }); - - it('should handle empty search results', async () => { - // Create a new empty store - const emptyDir = path.join(testDir, 'empty'); - await fs.mkdir(emptyDir, { recursive: true }); - - const emptyStorage = new VectorStorage({ - storePath: path.join(emptyDir, 'empty.lance'), - }); - - await emptyStorage.initialize(); - - const results = await emptyStorage.search('test query'); - expect(results).toEqual([]); - - await emptyStorage.close(); - }); - - it('should handle batch embedding efficiently', async () => { - const largeBatch: EmbeddingDocument[] = Array.from({ length: 50 }, (_, i) => ({ - id: `batch-${i}`, - text: `This is document number ${i} about programming languages`, - metadata: { index: i }, - })); - - const startTime = Date.now(); - await vectorStorage.addDocuments(largeBatch); - const duration = Date.now() - startTime; - - // Should complete in reasonable time (batching should help) - expect(duration).toBeLessThan(30000); // 30 seconds - - const stats = await vectorStorage.getStats(); - expect(stats.totalDocuments).toBeGreaterThanOrEqual(50); - }); - - it('should delete documents by ID', async () => { - // Add a document to delete - await vectorStorage.addDocuments([ - { id: 'to-delete', text: 'This document will be deleted', metadata: { temp: true } }, - ]); - - // Verify it exists - const beforeDelete = await vectorStorage.getDocument('to-delete'); - expect(beforeDelete).toBeDefined(); - - // Delete it - await vectorStorage.deleteDocuments(['to-delete']); - - // Verify it's gone - const afterDelete = await vectorStorage.getDocument('to-delete'); - expect(afterDelete).toBeNull(); - }); - - it('should handle empty document array', async () => { - // Should not throw - await vectorStorage.addDocuments([]); - - const stats = await vectorStorage.getStats(); - expect(stats.totalDocuments).toBeGreaterThanOrEqual(0); - }); - - it('should handle empty delete array', async () => { - // Should not throw - await vectorStorage.deleteDocuments([]); - }); - - it('should get embedder stats', async () => { - const stats = await vectorStorage.getStats(); - expect(stats.modelName).toBe('Xenova/all-MiniLM-L6-v2'); - expect(stats.dimension).toBe(384); - expect(stats.totalDocuments).toBeGreaterThan(0); - }); - - it('should handle search with default options', async () => { - const results = await vectorStorage.search('programming language'); - expect(Array.isArray(results)).toBe(true); - expect(results.length).toBeGreaterThan(0); - }); - - it('should handle search with empty query', async () => { - const results = await vectorStorage.search(''); - expect(Array.isArray(results)).toBe(true); - }); - - it('should handle multiple initializations gracefully', async () => { - // Should not throw or re-initialize - await vectorStorage.initialize(); - await vectorStorage.initialize(); - - const stats = await vectorStorage.getStats(); - expect(stats.modelName).toBe('Xenova/all-MiniLM-L6-v2'); - }); -}); - -describe('Vector Storage - Low-level Components', () => { - let testDir: string; - - beforeAll(async () => { - testDir = path.join(os.tmpdir(), `vector-component-test-${Date.now()}`); - await fs.mkdir(testDir, { recursive: true }); - }); - - afterAll(async () => { - await fs.rm(testDir, { recursive: true, force: true }); - }); - - it('should use default model and dimension', async () => { - const storage = new VectorStorage({ - storePath: path.join(testDir, 'defaults.lance'), - }); - - await storage.initialize(); - - const stats = await storage.getStats(); - expect(stats.modelName).toBe('Xenova/all-MiniLM-L6-v2'); - expect(stats.dimension).toBe(384); - - await storage.close(); - }); - - it('should throw error for operations before initialization', async () => { - const storage = new VectorStorage({ - storePath: path.join(testDir, 'uninit.lance'), - }); - - // Operations on uninitialized store should throw - await expect(storage.search('test')).rejects.toThrow('not initialized'); - await expect(storage.getDocument('test')).rejects.toThrow('not initialized'); - - await storage.close(); - }); - - it('should get store document count', async () => { - const storage = new VectorStorage({ - storePath: path.join(testDir, 'count-test.lance'), - }); - - await storage.initialize(); - - await storage.addDocuments([ - { id: 'count1', text: 'Test document 1', metadata: {} }, - { id: 'count2', text: 'Test document 2', metadata: {} }, - ]); - - const stats = await storage.getStats(); - expect(stats.totalDocuments).toBe(2); - - await storage.close(); - }); - - it('should handle close without initialization', async () => { - const storage = new VectorStorage({ - storePath: path.join(testDir, 'never-init.lance'), - }); - - // Should not throw - await storage.close(); - }); - - it('should filter search results by score threshold', async () => { - const storage = new VectorStorage({ - storePath: path.join(testDir, 'threshold-test.lance'), - }); - - await storage.initialize(); - - await storage.addDocuments([ - { id: 'exact', text: 'Machine learning algorithms', metadata: {} }, - { id: 'related', text: 'Artificial intelligence methods', metadata: {} }, - { id: 'unrelated', text: 'Cooking recipes for dinner', metadata: {} }, - ]); - - // High threshold should return fewer results - const strictResults = await storage.search('machine learning', { scoreThreshold: 0.9 }); - const lenientResults = await storage.search('machine learning', { scoreThreshold: 0.1 }); - - expect(strictResults.length).toBeLessThanOrEqual(lenientResults.length); - - await storage.close(); - }); - - it('should handle very long documents', async () => { - const storage = new VectorStorage({ - storePath: path.join(testDir, 'long-doc.lance'), - }); - - await storage.initialize(); - - const longText = 'This is a very detailed explanation about machine learning. '.repeat(100); // Very long document - await storage.addDocuments([{ id: 'long1', text: longText, metadata: { type: 'long' } }]); - - const stats = await storage.getStats(); - expect(stats.totalDocuments).toBe(1); - - // Verify we can retrieve it by ID - const doc = await storage.getDocument('long1'); - expect(doc).toBeDefined(); - expect(doc?.id).toBe('long1'); - - await storage.close(); - }); - - it('should handle sequential batch operations', async () => { - const storage = new VectorStorage({ - storePath: path.join(testDir, 'sequential.lance'), - }); - - await storage.initialize(); - - // Add documents sequentially - await storage.addDocuments([{ id: 'seq1', text: 'First doc', metadata: {} }]); - await storage.addDocuments([{ id: 'seq2', text: 'Second doc', metadata: {} }]); - await storage.addDocuments([{ id: 'seq3', text: 'Third doc', metadata: {} }]); - - const stats = await storage.getStats(); - expect(stats.totalDocuments).toBeGreaterThanOrEqual(3); - - // Verify all docs are searchable - const results = await storage.search('doc', { limit: 10 }); - expect(results.length).toBeGreaterThanOrEqual(3); - - await storage.close(); - }); -}); diff --git a/packages/core/src/vector/antfly-store.ts b/packages/core/src/vector/antfly-store.ts new file mode 100644 index 0000000..4497676 --- /dev/null +++ b/packages/core/src/vector/antfly-store.ts @@ -0,0 +1,392 @@ +/** + * Vector store implementation using Antfly + * + * Replaces LanceDBVectorStore + TransformersEmbedder with a single class. + * Antfly handles embedding generation (via Termite), storage, and hybrid search + * (BM25 + vector + RRF) internally. + */ + +import { AntflyClient } from '@antfly/sdk'; +import type { + EmbeddingDocument, + SearchOptions, + SearchResult, + SearchResultMetadata, + VectorStore, +} from './types.js'; + +// ── Antfly response types ── +// Local types for the SDK boundary. The SDK is auto-generated from OpenAPI; +// these provide type safety without coupling to internal SDK types. + +interface AntflyHit { + _id: string; + _score: number; + _index_scores?: Record; + _source?: Record; +} + +/** SDK query() returns this shape (already unwrapped from the REST responses[] wrapper) */ +interface AntflyQueryResponse { + hits: { hits: AntflyHit[] | null; total: number }; + status: number; + took?: number; +} + +interface AntflyTableInfo { + name: string; + indexes: Record< + string, + { + type: string; + dimension?: number; + embedder?: { provider: string; model: string }; + } + >; + storage_status?: { disk_usage: number }; +} + +/** Known embedding model dimensions */ +const MODEL_DIMENSIONS: Record = { + 'BAAI/bge-small-en-v1.5': 384, + 'mxbai-embed-large-v1': 1024, + 'nomic-ai/nomic-embed-text-v1.5': 768, + 'openai/clip-vit-base-patch32': 512, +}; + +const DEFAULT_MODEL = 'BAAI/bge-small-en-v1.5'; +const DEFAULT_BASE_URL = process.env.ANTFLY_URL ?? 'http://localhost:18080/api/v1'; +const BATCH_SIZE = 500; + +/** + * Configuration for AntflyVectorStore + */ +export interface AntflyStoreConfig { + baseUrl?: string; + table: string; + indexName?: string; + template?: string; + model?: string; +} + +/** + * Vector store backed by Antfly. + * + * Antfly handles embedding generation (Termite), vector storage, + * BM25 full-text indexing, and hybrid search with RRF fusion. + */ +export class AntflyVectorStore implements VectorStore { + readonly path: string; + private readonly cfg: Required; + private readonly client: AntflyClient; + private initialized = false; + + constructor(config: AntflyStoreConfig) { + this.cfg = { + baseUrl: config.baseUrl ?? DEFAULT_BASE_URL, + table: config.table, + indexName: config.indexName ?? 'content', + template: config.template ?? '{{text}}', + model: config.model ?? DEFAULT_MODEL, + }; + this.path = `${this.cfg.baseUrl}/${this.cfg.table}`; + this.client = new AntflyClient({ baseUrl: this.cfg.baseUrl }); + } + + async initialize(): Promise { + if (this.initialized) return; + + try { + const tableNames = await this.listTableNames(); + + if (tableNames.includes(this.cfg.table)) { + await this.checkModelMismatch(); + } else { + await this.createTableWithIndex(); + } + + this.initialized = true; + } catch (error) { + if (error instanceof Error && error.message.includes('Model mismatch')) throw error; + throw new Error( + `Failed to initialize Antfly store (${this.cfg.table}): ${error instanceof Error ? error.message : String(error)}` + ); + } + } + + /** + * Add documents. Embeddings param is ignored — Antfly auto-embeds via Termite. + */ + async add(documents: EmbeddingDocument[], _embeddings?: number[][]): Promise { + if (documents.length === 0) return; + this.assertReady(); + + for (let i = 0; i < documents.length; i += BATCH_SIZE) { + const batch = documents.slice(i, i + BATCH_SIZE); + const inserts: Record> = {}; + for (const doc of batch) { + inserts[doc.id] = { text: doc.text, metadata: JSON.stringify(doc.metadata) }; + } + + try { + await this.batchOp({ inserts }); + } catch (error) { + throw new Error( + `Failed to add documents (batch ${Math.floor(i / BATCH_SIZE) + 1}): ${error instanceof Error ? error.message : String(error)}` + ); + } + } + } + + /** + * VectorStore interface method — throws. Use searchText() instead. + */ + async search(_queryEmbedding: number[], _options?: SearchOptions): Promise { + throw new Error( + 'AntflyVectorStore.search() does not accept pre-computed embeddings. ' + + 'Use searchText(query, options) instead — Antfly handles embedding internally.' + ); + } + + /** + * Search by text using Antfly's hybrid search (BM25 + vector + RRF). + */ + async searchText(query: string, options: SearchOptions = {}): Promise { + this.assertReady(); + const { limit = 10, scoreThreshold = 0 } = options; + + try { + const resp = await this.queryTable({ + semantic_search: query, + indexes: [this.cfg.indexName], + limit, + }); + + return this.extractHits(resp) + .map((hit) => ({ + id: hit._id, + score: hit._score, + metadata: this.parseMetadata(hit._source), + })) + .filter((r) => r.score >= scoreThreshold); + } catch (error) { + throw new Error( + `Failed to search: ${error instanceof Error ? error.message : String(error)}` + ); + } + } + + async get(id: string): Promise { + this.assertReady(); + + try { + const result = (await this.client.tables.lookup(this.cfg.table, id)) as + | Record + | undefined; + if (!result) return null; + + return { + id, + text: (result.text as string) ?? '', + metadata: this.parseRawMetadata(result.metadata), + }; + } catch (error) { + if (String(error).includes('404') || String(error).includes('not found')) return null; + throw new Error( + `Failed to get document: ${error instanceof Error ? error.message : String(error)}` + ); + } + } + + async delete(ids: string[]): Promise { + this.assertReady(); + if (ids.length === 0) return; + + try { + await this.batchOp({ deletes: ids }); + } catch (error) { + throw new Error( + `Failed to delete documents: ${error instanceof Error ? error.message : String(error)}` + ); + } + } + + async count(): Promise { + this.assertReady(); + + try { + const resp = await this.queryTable({ limit: 1 }); + return resp?.hits?.total ?? 0; + } catch { + // Empty or newly-created table may return unexpected shapes + return 0; + } + } + + async getAll(options: { limit?: number } = {}): Promise { + this.assertReady(); + const { limit = 10000 } = options; + + try { + const resp = await this.queryTable({ limit }); + return this.extractHits(resp).map((hit) => ({ + id: hit._id, + score: 1, + metadata: this.parseMetadata(hit._source), + })); + } catch (error) { + throw new Error( + `Failed to get all documents: ${error instanceof Error ? error.message : String(error)}` + ); + } + } + + async searchByDocumentId( + documentId: string, + options: SearchOptions = {} + ): Promise { + this.assertReady(); + const doc = await this.get(documentId); + if (!doc) return []; + return this.searchText(doc.text, options); + } + + async clear(): Promise { + this.assertReady(); + + try { + await this.client.tables.drop(this.cfg.table); + await this.createTableWithIndex(); + } catch (error) { + throw new Error( + `Failed to clear store: ${error instanceof Error ? error.message : String(error)}` + ); + } + } + + async optimize(): Promise { + // Antfly manages compaction internally + } + + async close(): Promise { + this.initialized = false; + } + + getModelInfo(): { dimension: number; modelName: string } { + return { + dimension: MODEL_DIMENSIONS[this.cfg.model] ?? 384, + modelName: this.cfg.model, + }; + } + + async getStorageSize(): Promise { + try { + const info = await this.getTableInfo(); + return info?.storage_status?.disk_usage ?? 0; + } catch { + return 0; + } + } + + // ── SDK boundary layer ── + // These methods isolate the SDK's loosely-typed API behind our own types. + + private async listTableNames(): Promise { + const raw = await this.client.tables.list(); + const tables = raw as unknown as AntflyTableInfo[] | string[]; + if (!Array.isArray(tables)) return []; + return tables.map((t) => (typeof t === 'string' ? t : t.name)); + } + + private async createTableWithIndex(): Promise { + const body = { + indexes: { + [this.cfg.indexName]: { + type: 'embeddings', + template: this.cfg.template, + embedder: { provider: 'termite', model: this.cfg.model }, + }, + }, + }; + await (this.client.tables.create as Function)(this.cfg.table, body); + } + + private async getTableInfo(): Promise { + const raw = await this.client.tables.get(this.cfg.table); + return (raw ?? null) as AntflyTableInfo | null; + } + + private async queryTable(params: Record): Promise { + const raw = await (this.client.query as Function)({ + table: this.cfg.table, + ...params, + }); + return raw as AntflyQueryResponse; + } + + private async batchOp(body: Record): Promise { + await (this.client.tables.batch as Function)(this.cfg.table, body); + } + + // ── Helpers ── + + private assertReady(): void { + if (!this.initialized) { + throw new Error('Store not initialized. Call initialize() first.'); + } + } + + private async checkModelMismatch(): Promise { + try { + const info = await this.getTableInfo(); + const embeddingIndex = info?.indexes?.[this.cfg.indexName]; + + if (embeddingIndex?.embedder?.model && embeddingIndex.embedder.model !== this.cfg.model) { + throw new Error( + `Model mismatch: table "${this.cfg.table}" uses "${embeddingIndex.embedder.model}" ` + + `but config specifies "${this.cfg.model}". ` + + 'Run `dev index . --force` to re-index with the new model.' + ); + } + } catch (error) { + if (error instanceof Error && error.message.includes('Model mismatch')) throw error; + } + } + + private extractHits(resp: AntflyQueryResponse): AntflyHit[] { + return resp?.hits?.hits ?? []; + } + + private parseMetadata(source: Record | undefined): SearchResultMetadata { + if (!source) return {}; + + const metadataField = source.metadata; + if (typeof metadataField === 'string') { + try { + return JSON.parse(metadataField) as SearchResultMetadata; + } catch { + return {}; + } + } + if (metadataField && typeof metadataField === 'object') { + return metadataField as SearchResultMetadata; + } + + const { text: _, metadata: __, _timestamp: ___, ...rest } = source; + return rest as SearchResultMetadata; + } + + private parseRawMetadata(metadata: unknown): Record { + if (typeof metadata === 'string') { + try { + return JSON.parse(metadata); + } catch { + return {}; + } + } + if (metadata && typeof metadata === 'object') { + return metadata as Record; + } + return {}; + } +} diff --git a/packages/core/src/vector/embedder.ts b/packages/core/src/vector/embedder.ts deleted file mode 100644 index 6627485..0000000 --- a/packages/core/src/vector/embedder.ts +++ /dev/null @@ -1,134 +0,0 @@ -import { type FeatureExtractionPipeline, pipeline } from '@xenova/transformers'; -import type { EmbeddingProvider } from './types'; - -/** - * Options for feature extraction from transformers.js - */ -interface FeatureExtractionOptions { - pooling?: 'none' | 'mean' | 'cls'; - normalize?: boolean; - quantize?: boolean; - precision?: 'binary' | 'ubinary'; -} - -/** - * Embedding provider using Transformers.js - * Uses all-MiniLM-L6-v2 model for generating embeddings - */ -export class TransformersEmbedder implements EmbeddingProvider { - readonly modelName: string; - readonly dimension: number; - private pipeline: FeatureExtractionPipeline | null = null; - private batchSize = 32; - - constructor(modelName = 'Xenova/all-MiniLM-L6-v2', dimension = 384) { - this.modelName = modelName; - this.dimension = dimension; - } - - /** - * Initialize the embedding model - * Downloads and caches the model on first run - */ - async initialize(): Promise { - if (this.pipeline) { - return; // Already initialized - } - - try { - // Create pipeline with the feature-extraction task - this.pipeline = (await pipeline( - 'feature-extraction', - this.modelName - )) as FeatureExtractionPipeline; - } catch (error) { - throw new Error( - `Failed to initialize embedding model ${this.modelName}: ${error instanceof Error ? error.message : String(error)}` - ); - } - } - - /** - * Generate embedding for a single text - */ - async embed(text: string): Promise { - if (!this.pipeline) { - throw new Error('Embedder not initialized. Call initialize() first.'); - } - - try { - // Call pipeline with proper options - const options: FeatureExtractionOptions = { - pooling: 'mean', // Mean pooling - normalize: true, // L2 normalization - }; - - const output = await this.pipeline(text, options); - - // Extract data from tensor output - // Note: Using any because transformers.js doesn't export specific Tensor types - // biome-ignore lint/suspicious/noExplicitAny: Tensor type not exported - const tensorOutput = output as any; - - if (tensorOutput?.data) { - return Array.from(tensorOutput.data as Float32Array); - } - - throw new Error('Unexpected output format from embedding model'); - } catch (error) { - throw new Error( - `Failed to generate embedding: ${error instanceof Error ? error.message : String(error)}` - ); - } - } - - /** - * Generate embeddings for multiple texts (batched for efficiency) - */ - async embedBatch(texts: string[]): Promise { - if (!this.pipeline) { - throw new Error('Embedder not initialized. Call initialize() first.'); - } - - if (texts.length === 0) { - return []; - } - - try { - // Process in batches to avoid memory issues - const embeddings: number[][] = []; - - for (let i = 0; i < texts.length; i += this.batchSize) { - const batch = texts.slice(i, i + this.batchSize); - - // Process batch in parallel - const batchEmbeddings = await Promise.all(batch.map((text) => this.embed(text))); - - embeddings.push(...batchEmbeddings); - } - - return embeddings; - } catch (error) { - throw new Error( - `Failed to generate batch embeddings: ${error instanceof Error ? error.message : String(error)}` - ); - } - } - - /** - * Set batch size for batch processing - */ - setBatchSize(size: number): void { - if (size < 1) { - throw new Error('Batch size must be at least 1'); - } - this.batchSize = size; - } - - /** - * Get current batch size - */ - getBatchSize(): number { - return this.batchSize; - } -} diff --git a/packages/core/src/vector/index.ts b/packages/core/src/vector/index.ts index 09e88ae..70367d0 100644 --- a/packages/core/src/vector/index.ts +++ b/packages/core/src/vector/index.ts @@ -1,132 +1,108 @@ /** - * Vector storage and embedding system + * Vector storage system + * + * Backed by Antfly — handles embedding generation, vector storage, + * and hybrid search (BM25 + vector + RRF) internally. */ -export * from './embedder'; -export * from './store'; -export * from './types'; +export * from './antfly-store.js'; +export * from './types.js'; -import * as fs from 'node:fs/promises'; -import { TransformersEmbedder } from './embedder'; -import { LanceDBVectorStore } from './store'; +import { type AntflyStoreConfig, AntflyVectorStore } from './antfly-store.js'; import type { EmbeddingDocument, SearchOptions, SearchResult, VectorStats, VectorStorageConfig, -} from './types'; +} from './types.js'; /** - * Convenience class that combines embedder and vector store - * Provides a simple API for storing and searching documents + * Derives an antfly table name from a storePath. + * + * storePath examples: + * ~/.dev-agent/indexes/my-project/vectors → dev-agent-my-project-code + * ~/.dev-agent/indexes/my-project/vectors-git → dev-agent-my-project-git + * ~/.dev-agent/indexes/my-project/vectors-github → dev-agent-my-project-github + */ +function deriveTableName(storePath: string): string { + const parts = storePath.replace(/\/$/, '').split('/'); + const last = parts.at(-1) ?? 'code'; + const projectDir = parts.at(-2) ?? 'default'; + + // Sanitize for antfly table names (alphanumeric + hyphens) + const project = projectDir.replace(/[^a-zA-Z0-9-]/g, '-').toLowerCase(); + + if (last === 'vectors') return `dev-agent-${project}-code`; + if (last === 'vectors-git') return `dev-agent-${project}-git`; + if (last === 'vectors-github') return `dev-agent-${project}-github`; + return `dev-agent-${project}-${last.replace(/[^a-zA-Z0-9-]/g, '-')}`; +} + +/** + * High-level vector storage API. + * + * Wraps AntflyVectorStore and preserves the same public interface that + * all consumers (indexers, services, CLI, MCP) depend on. + * + * With Antfly, there is no separate embedding step — documents are + * embedded automatically on insert and queries use hybrid search. */ export class VectorStorage { - private readonly embedder: TransformersEmbedder; - private readonly store: LanceDBVectorStore; + private readonly store: AntflyVectorStore; private initialized = false; constructor(config: VectorStorageConfig) { - const { storePath, embeddingModel = 'Xenova/all-MiniLM-L6-v2', dimension = 384 } = config; + const antflyConfig: AntflyStoreConfig = { + table: deriveTableName(config.storePath), + model: config.embeddingModel, + }; - this.embedder = new TransformersEmbedder(embeddingModel, dimension); - this.store = new LanceDBVectorStore(storePath, dimension); + this.store = new AntflyVectorStore(antflyConfig); } /** - * Initialize both embedder and store - * @param options Optional initialization options - * @param options.skipEmbedder Skip embedder initialization (useful for read-only operations) + * Initialize the storage. + * + * The skipEmbedder option is accepted for backward compatibility but + * has no effect — Antfly handles embeddings internally. */ - async initialize(options?: { skipEmbedder?: boolean }): Promise { - if (this.initialized) { - return; - } - - const { skipEmbedder = false } = options || {}; - - if (skipEmbedder) { - // Only initialize store, skip embedder (much faster for read-only operations) - await this.store.initialize(); - } else { - // Initialize both embedder and store - await Promise.all([this.embedder.initialize(), this.store.initialize()]); - } - + async initialize(_options?: { skipEmbedder?: boolean }): Promise { + if (this.initialized) return; + await this.store.initialize(); this.initialized = true; } /** - * Ensure embedder is initialized (lazy initialization for search operations) - */ - private async ensureEmbedder(): Promise { - if (!this.embedder) { - throw new Error('Embedder not available'); - } - // Initialize embedder if not already done - await this.embedder.initialize(); - } - - /** - * Add documents to the store (automatically generates embeddings) + * Add documents (Antfly generates embeddings automatically via Termite) */ async addDocuments(documents: EmbeddingDocument[]): Promise { - if (!this.initialized) { - throw new Error('VectorStorage not initialized. Call initialize() first.'); - } - - if (documents.length === 0) { - return; - } - - // Generate embeddings - const texts = documents.map((doc) => doc.text); - const embeddings = await this.embedder.embedBatch(texts); - - // Store documents with embeddings - await this.store.add(documents, embeddings); + this.assertReady(); + if (documents.length === 0) return; + await this.store.add(documents); } /** - * Search for similar documents using natural language query + * Search using hybrid search (BM25 + vector + RRF) */ async search(query: string, options?: SearchOptions): Promise { - if (!this.initialized) { - throw new Error('VectorStorage not initialized. Call initialize() first.'); - } - - // Ensure embedder is initialized (lazy load if needed) - await this.ensureEmbedder(); - - // Generate query embedding - const queryEmbedding = await this.embedder.embed(query); - - // Search vector store - return this.store.search(queryEmbedding, options); + this.assertReady(); + return this.store.searchText(query, options); } /** - * Find similar documents to a given document by ID - * More efficient than search() as it reuses the document's existing embedding + * Find documents similar to a given document by ID */ async searchByDocumentId(documentId: string, options?: SearchOptions): Promise { - if (!this.initialized) { - throw new Error('VectorStorage not initialized. Call initialize() first.'); - } - + this.assertReady(); return this.store.searchByDocumentId(documentId, options); } /** - * Get all documents without semantic search (fast scan) - * Use this when you need all documents and don't need relevance ranking - * This is 10-20x faster than search() as it skips embedding generation + * Get all documents without semantic search (full scan) */ async getAll(options?: { limit?: number }): Promise { - if (!this.initialized) { - throw new Error('VectorStorage not initialized. Call initialize() first.'); - } - + this.assertReady(); return this.store.getAll(options); } @@ -134,10 +110,7 @@ export class VectorStorage { * Get a document by ID */ async getDocument(id: string): Promise { - if (!this.initialized) { - throw new Error('VectorStorage not initialized. Call initialize() first.'); - } - + this.assertReady(); return this.store.get(id); } @@ -145,22 +118,15 @@ export class VectorStorage { * Delete documents by ID */ async deleteDocuments(ids: string[]): Promise { - if (!this.initialized) { - throw new Error('VectorStorage not initialized. Call initialize() first.'); - } - + this.assertReady(); await this.store.delete(ids); } /** - * Clear all documents from the store (destructive operation) - * Used for force re-indexing + * Clear all documents (destructive — used for force re-indexing) */ async clear(): Promise { - if (!this.initialized) { - throw new Error('VectorStorage not initialized. Call initialize() first.'); - } - + this.assertReady(); await this.store.clear(); } @@ -168,40 +134,24 @@ export class VectorStorage { * Get statistics about the vector store */ async getStats(): Promise { - if (!this.initialized) { - throw new Error('VectorStorage not initialized. Call initialize() first.'); - } - + this.assertReady(); + const modelInfo = this.store.getModelInfo(); const totalDocuments = await this.store.count(); - - // Get storage size - let storageSize = 0; - try { - const storePath = this.store.path; - const stats = await fs.stat(storePath); - storageSize = stats.size; - } catch { - // Directory might not exist yet - storageSize = 0; - } + const storageSize = await this.store.getStorageSize(); return { totalDocuments, storageSize, - dimension: this.embedder.dimension, - modelName: this.embedder.modelName, + dimension: modelInfo.dimension, + modelName: modelInfo.modelName, }; } /** - * Optimize the vector store (compact fragments, update indices) - * Call this after bulk indexing operations for better performance + * Optimize the store (no-op for Antfly — manages compaction internally) */ async optimize(): Promise { - if (!this.initialized) { - throw new Error('VectorStorage not initialized. Call initialize() first.'); - } - + this.assertReady(); await this.store.optimize(); } @@ -212,4 +162,10 @@ export class VectorStorage { await this.store.close(); this.initialized = false; } + + private assertReady(): void { + if (!this.initialized) { + throw new Error('VectorStorage not initialized. Call initialize() first.'); + } + } } diff --git a/packages/core/src/vector/store.ts b/packages/core/src/vector/store.ts deleted file mode 100644 index f1bcd67..0000000 --- a/packages/core/src/vector/store.ts +++ /dev/null @@ -1,361 +0,0 @@ -import type { Connection, Table } from '@lancedb/lancedb'; -import * as lancedb from '@lancedb/lancedb'; -import type { - EmbeddingDocument, - SearchOptions, - SearchResult, - SearchResultMetadata, - VectorStore, -} from './types'; - -/** - * Vector store implementation using LanceDB - */ -export class LanceDBVectorStore implements VectorStore { - readonly path: string; - private readonly tableName = 'documents'; - private connection: Connection | null = null; - private table: Table | null = null; - - constructor(path: string, _dimension = 384) { - this.path = path; - // Note: dimension is determined by the embeddings passed to add() - } - - /** - * Initialize the vector store - */ - async initialize(): Promise { - if (this.connection) { - return; // Already initialized - } - - try { - // Connect to LanceDB (creates directory if it doesn't exist) - this.connection = await lancedb.connect(this.path); - - // Try to open existing table - const tableNames = await this.connection.tableNames(); - - if (tableNames.includes(this.tableName)) { - this.table = await this.connection.openTable(this.tableName); - } - // Table will be created on first add() call - } catch (error) { - throw new Error( - `Failed to initialize LanceDB at ${this.path}: ${error instanceof Error ? error.message : String(error)}` - ); - } - } - - /** - * Add documents to the store using upsert (prevents duplicates) - */ - async add(documents: EmbeddingDocument[], embeddings: number[][]): Promise { - if (!this.connection) { - throw new Error('Store not initialized. Call initialize() first.'); - } - - if (documents.length !== embeddings.length) { - throw new Error('Number of documents must match number of embeddings'); - } - - if (documents.length === 0) { - return; - } - - try { - // Prepare data for LanceDB - const data = documents.map((doc, i) => ({ - id: doc.id, - text: doc.text, - vector: embeddings[i], - metadata: JSON.stringify(doc.metadata), - })); - - if (!this.table) { - // Create table on first add - try { - this.table = await this.connection.createTable(this.tableName, data); - // Create scalar index on 'id' column for fast upsert operations - await this.ensureIdIndex(); - } catch (createError) { - // Handle race condition: another process might have created the table - if (createError instanceof Error && createError.message.includes('already exists')) { - // Open the existing table - this.table = await this.connection.openTable(this.tableName); - // Now add the data using mergeInsert - await this.table - .mergeInsert('id') - .whenMatchedUpdateAll() - .whenNotMatchedInsertAll() - .execute(data); - } else { - throw createError; - } - } - } else { - // Use mergeInsert to prevent duplicates (upsert operation) - // This updates existing documents with the same ID or inserts new ones - await this.table - .mergeInsert('id') - .whenMatchedUpdateAll() - .whenNotMatchedInsertAll() - .execute(data); - } - } catch (error) { - throw new Error( - `Failed to add documents: ${error instanceof Error ? error.message : String(error)}` - ); - } - } - - /** - * Search for similar documents - */ - async search(queryEmbedding: number[], options: SearchOptions = {}): Promise { - if (!this.table) { - return []; // No documents yet - } - - const { limit = 10, scoreThreshold = 0 } = options; - - try { - // Perform vector search - // LanceDB uses L2 distance by default, returning lower values for more similar vectors - const results = await this.table.search(queryEmbedding).limit(limit).toArray(); - - // Transform results - // Convert L2 distance to a similarity score (0-1 range) - // For normalized embeddings, L2 distance ≈ sqrt(2 * (1 - cosine_similarity)) - // So cosine_similarity ≈ 1 - (L2_distance^2 / 2) - // We'll use an exponential decay to convert distance to similarity - return results - .map((result) => { - const distance = - result._distance !== undefined ? result._distance : Number.POSITIVE_INFINITY; - // Use exponential decay: score = e^(-distance^2) - // This gives scores close to 1 for distance≈0, and approaches 0 for large distances - const score = Math.exp(-(distance * distance)); - - return { - id: result.id as string, - score, - metadata: JSON.parse(result.metadata as string) as SearchResultMetadata, - }; - }) - .filter((result) => result.score >= scoreThreshold); - } catch (error) { - throw new Error( - `Failed to search: ${error instanceof Error ? error.message : String(error)}` - ); - } - } - - /** - * Get all documents without semantic search (fast scan) - * Use this when you need all documents and don't need relevance ranking - */ - async getAll(options: { limit?: number } = {}): Promise { - if (!this.table) { - return []; // No documents yet - } - - const { limit = 10000 } = options; - - try { - // Use query() instead of search() - no vector similarity calculation needed - // This is much faster as it skips embedding generation and distance computation - const results = await this.table - .query() - .select(['id', 'text', 'metadata']) - .limit(limit) - .toArray(); - - // Transform results (all have score of 1 since no ranking) - return results.map((result) => ({ - id: result.id as string, - score: 1, // No relevance score for full scan - metadata: JSON.parse(result.metadata as string) as SearchResultMetadata, - })); - } catch (error) { - throw new Error( - `Failed to get all documents: ${error instanceof Error ? error.message : String(error)}` - ); - } - } - - /** - * Find similar documents to a given document by ID - * Uses the document's existing embedding for efficient similarity search - */ - async searchByDocumentId( - documentId: string, - options: SearchOptions = {} - ): Promise { - if (!this.table) { - return []; - } - - try { - // Get the document and its embedding - const results = await this.table - .query() - .where(`id = '${documentId}'`) - .select(['id', 'vector']) - .limit(1) - .toArray(); - - if (results.length === 0) { - return []; // Document not found - } - - const documentEmbedding = results[0].vector as number[]; - - // Use the document's embedding to find similar documents - return this.search(documentEmbedding, options); - } catch (error) { - throw new Error( - `Failed to search by document ID: ${error instanceof Error ? error.message : String(error)}` - ); - } - } - - /** - * Get a document by ID - */ - async get(id: string): Promise { - if (!this.table) { - return null; - } - - try { - // Use a dummy vector for search since LanceDB requires it - // We'll search with a zero vector and filter results by ID - const dummyVector = new Array(384).fill(0); // Default dimension - const results = await this.table.search(dummyVector).limit(10000).toArray(); - - const result = results.find((r) => r.id === id); - - if (!result) { - return null; - } - - return { - id: result.id as string, - text: result.text as string, - metadata: JSON.parse(result.metadata as string) as Record, - }; - } catch (error) { - throw new Error( - `Failed to get document: ${error instanceof Error ? error.message : String(error)}` - ); - } - } - - /** - * Delete documents by ID - */ - async delete(ids: string[]): Promise { - if (!this.table || ids.length === 0) { - return; - } - - try { - // Delete using SQL IN predicate - // Escape single quotes in IDs to prevent SQL injection - const escapedIds = ids.map((id) => id.replace(/'/g, "''")); - const predicate = `id IN ('${escapedIds.join("', '")}')`; - await this.table.delete(predicate); - } catch (error) { - throw new Error( - `Failed to delete documents: ${error instanceof Error ? error.message : String(error)}` - ); - } - } - - /** - * Clear all documents from the store (destructive operation) - */ - async clear(): Promise { - if (!this.connection) { - return; - } - - try { - // Drop the table if it exists - if (this.table) { - await this.connection.dropTable('documents'); - this.table = null; - } - } catch (error) { - throw new Error( - `Failed to clear vector store: ${error instanceof Error ? error.message : String(error)}` - ); - } - } - - /** - * Count total documents - */ - async count(): Promise { - if (!this.table) { - return 0; - } - - try { - return await this.table.countRows(); - } catch (error) { - throw new Error( - `Failed to count documents: ${error instanceof Error ? error.message : String(error)}` - ); - } - } - - /** - * Optimize the vector store (compact fragments, update indices) - */ - async optimize(): Promise { - if (!this.table) { - return; - } - - try { - await this.table.optimize(); - } catch (error) { - throw new Error( - `Failed to optimize: ${error instanceof Error ? error.message : String(error)}` - ); - } - } - - /** - * Ensure scalar index exists on 'id' column for fast upsert operations - */ - private async ensureIdIndex(): Promise { - if (!this.table) { - return; - } - - try { - // Create a scalar index on the 'id' column to speed up mergeInsert operations - // LanceDB will use an appropriate index type automatically - await this.table.createIndex('id'); - } catch (error) { - // Index may already exist or not be supported - log but don't fail - // Some versions of LanceDB may not support this or it may already exist - console.warn( - `Could not create index on 'id' column: ${error instanceof Error ? error.message : String(error)}` - ); - } - } - - /** - * Close the store - */ - async close(): Promise { - // LanceDB doesn't require explicit closing - this.table = null; - this.connection = null; - } -} diff --git a/packages/dev-agent/package.json b/packages/dev-agent/package.json index a3aa95f..c693fce 100644 --- a/packages/dev-agent/package.json +++ b/packages/dev-agent/package.json @@ -43,17 +43,15 @@ "prepublishOnly": "pnpm run build" }, "dependencies": { - "@lancedb/lancedb": "^0.22.3", - "@xenova/transformers": "^2.17.2", "better-sqlite3": "^12.5.0", "ts-morph": "^27.0.2", "web-tree-sitter": "^0.25.10" }, "devDependencies": { - "tree-sitter-wasms": "^0.1.13", "@prosdevlab/dev-agent-cli": "workspace:*", "@prosdevlab/dev-agent-mcp": "workspace:*", "@types/node": "^22.0.0", + "tree-sitter-wasms": "^0.1.13", "tsup": "^8.3.0" }, "engines": { diff --git a/packages/dev-agent/tsup.config.ts b/packages/dev-agent/tsup.config.ts index c4b2dc2..48a9c9f 100644 --- a/packages/dev-agent/tsup.config.ts +++ b/packages/dev-agent/tsup.config.ts @@ -9,12 +9,6 @@ const version = packageJson.version; // - Native modules (have platform-specific binaries) // - Large libraries with their own loading mechanisms const external = [ - // Native modules with platform-specific binaries - '@lancedb/lancedb', - - // Large ML library - has its own model loading mechanism - '@xenova/transformers', - // These have native bindings or complex loading 'ts-morph', 'typescript', diff --git a/packages/subagents/src/coordinator/__tests__/coordinator.integration.test.ts b/packages/subagents/src/coordinator/__tests__/coordinator.integration.test.ts index f2201d4..039720f 100644 --- a/packages/subagents/src/coordinator/__tests__/coordinator.integration.test.ts +++ b/packages/subagents/src/coordinator/__tests__/coordinator.integration.test.ts @@ -6,6 +6,39 @@ import { mkdir, rm } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; + +// Mock VectorStorage to avoid needing antfly server +// Must mock the resolved path (vitest aliases @prosdevlab/dev-agent-core → packages/core/src) +vi.mock('../../../../core/src/vector/index', async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + VectorStorage: class MockVectorStorage { + async initialize() {} + async addDocuments() {} + async search() { + return []; + } + async searchByDocumentId() { + return []; + } + async getAll() { + return []; + } + async getDocument() { + return null; + } + async deleteDocuments() {} + async clear() {} + async getStats() { + return { totalDocuments: 0, storageSize: 0, dimension: 384, modelName: 'mock' }; + } + async optimize() {} + async close() {} + }, + }; +}); + import { RepositoryIndexer } from '@prosdevlab/dev-agent-core'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { ExplorerAgent } from '../../explorer'; diff --git a/packages/subagents/src/coordinator/__tests__/github-coordinator.integration.test.ts b/packages/subagents/src/coordinator/__tests__/github-coordinator.integration.test.ts index 3b10eb7..14ee638 100644 --- a/packages/subagents/src/coordinator/__tests__/github-coordinator.integration.test.ts +++ b/packages/subagents/src/coordinator/__tests__/github-coordinator.integration.test.ts @@ -6,6 +6,38 @@ import { mkdtemp, rm } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; + +// Mock VectorStorage to avoid needing antfly server +vi.mock('@prosdevlab/dev-agent-core', async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + VectorStorage: class MockVectorStorage { + async initialize() {} + async addDocuments() {} + async search() { + return []; + } + async searchByDocumentId() { + return []; + } + async getAll() { + return []; + } + async getDocument() { + return null; + } + async deleteDocuments() {} + async clear() {} + async getStats() { + return { totalDocuments: 0, storageSize: 0, dimension: 384, modelName: 'mock' }; + } + async optimize() {} + async close() {} + }, + }; +}); + import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import type { GitHubAgentConfig } from '../../github/agent'; import { GitHubAgent } from '../../github/agent'; diff --git a/packages/subagents/src/explorer/__tests__/index.test.ts b/packages/subagents/src/explorer/__tests__/index.test.ts index 68ba939..8118703 100644 --- a/packages/subagents/src/explorer/__tests__/index.test.ts +++ b/packages/subagents/src/explorer/__tests__/index.test.ts @@ -1,6 +1,38 @@ import { mkdtemp, rm, writeFile } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; + +// Mock VectorStorage to avoid needing antfly server +vi.mock('../../../../core/src/vector/index', async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + VectorStorage: class MockVectorStorage { + async initialize() {} + async addDocuments() {} + async search() { + return []; + } + async searchByDocumentId() { + return []; + } + async getAll() { + return []; + } + async getDocument() { + return null; + } + async deleteDocuments() {} + async clear() {} + async getStats() { + return { totalDocuments: 0, storageSize: 0, dimension: 384, modelName: 'mock' }; + } + async optimize() {} + async close() {} + }, + }; +}); + import { RepositoryIndexer } from '@prosdevlab/dev-agent-core'; import { beforeEach, describe, expect, it, vi } from 'vitest'; import { ContextManagerImpl } from '../../coordinator/context-manager'; @@ -632,8 +664,8 @@ describe('ExplorerAgent', () => { }); it('should handle errors during pattern search', async () => { - // Close the indexer to cause an error - await indexer.close(); + // Force search to throw by mocking the indexer's search method + vi.spyOn(indexer, 'search').mockRejectedValueOnce(new Error('Store not initialized')); // Mock the logger to suppress expected error output const errorSpy = vi.spyOn(context.logger, 'error').mockImplementation(() => {}); diff --git a/packages/subagents/src/github/utils/__tests__/fetcher.test.ts b/packages/subagents/src/github/utils/__tests__/fetcher.test.ts index b0071a6..38b30ac 100644 --- a/packages/subagents/src/github/utils/__tests__/fetcher.test.ts +++ b/packages/subagents/src/github/utils/__tests__/fetcher.test.ts @@ -63,7 +63,7 @@ describe('GitHub Fetcher - Configuration', () => { }); it('should return repository in owner/repo format', () => { - vi.mocked(execSync).mockReturnValueOnce('lytics/dev-agent\n' as any); + vi.mocked(execSync).mockReturnValueOnce('prosdevlab/dev-agent\n' as any); const repo = getCurrentRepository(); expect(repo).toBe('prosdevlab/dev-agent'); diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index c905543..d2e987a 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -81,18 +81,15 @@ importers: packages/core: dependencies: - '@lancedb/lancedb': - specifier: ^0.22.3 - version: 0.22.3(apache-arrow@18.1.0) + '@antfly/sdk': + specifier: 0.0.14 + version: 0.0.14 '@prosdevlab/dev-agent-types': specifier: workspace:* version: link:../types '@prosdevlab/kero': specifier: workspace:* version: link:../logger - '@xenova/transformers': - specifier: ^2.17.2 - version: 2.17.2 better-sqlite3: specifier: ^12.5.0 version: 12.5.0 @@ -139,12 +136,6 @@ importers: packages/dev-agent: dependencies: - '@lancedb/lancedb': - specifier: ^0.22.3 - version: 0.22.3(apache-arrow@18.1.0) - '@xenova/transformers': - specifier: ^2.17.2 - version: 2.17.2 better-sqlite3: specifier: ^12.5.0 version: 12.5.0 @@ -261,6 +252,13 @@ importers: packages: + /@antfly/sdk@0.0.14: + resolution: {integrity: sha512-2GduXnYovTji5FbLrIff3iOx+JVUJ/C/6joUHtbmR9bpCUfh34U7ckViHvHPu8UuDpDjadyRqvz0CNEFp6i3IQ==} + engines: {node: '>=18'} + dependencies: + openapi-fetch: 0.17.0 + dev: false + /@babel/code-frame@7.27.1: resolution: {integrity: sha512-cjQ7ZlQ0Mv3b47hABuTevyTuYN4i+loJKGeV9flcCgIK37cCXRh+L1bd3iBHlynerhQ7BhCkn2BPbQUL+rGqFg==} engines: {node: '>=6.9.0'} @@ -1418,11 +1416,6 @@ packages: dev: true optional: true - /@huggingface/jinja@0.2.2: - resolution: {integrity: sha512-/KPde26khDUIPkTGU82jdtTW9UAuvUTumCAbFs/7giR0SxsvZC4hru51PBvpijH6BVkHcROcvZM/lpy5h1jRRA==} - engines: {node: '>=18'} - dev: false - /@inquirer/external-editor@1.0.2(@types/node@24.10.1): resolution: {integrity: sha512-yy9cOoBnx58TlsPrIxauKIFQTiyH+0MK4e97y4sV9ERbI+zDxw7i2hxHLCIEGIE/8PPvDxGhgzIOTSOWcs6/MQ==} engines: {node: '>=18'} @@ -1479,98 +1472,6 @@ packages: '@jridgewell/sourcemap-codec': 1.5.5 dev: true - /@lancedb/lancedb-darwin-arm64@0.22.3: - resolution: {integrity: sha512-oP2Kic51nLqs27Xo+AzSVlcMgmmfZbU/PseQ3KBtU92rczO5DYU2St1Y7qDUWcjw+RF3H2v/DKzYed16h1wCBQ==} - engines: {node: '>= 18'} - cpu: [arm64] - os: [darwin] - requiresBuild: true - dev: false - optional: true - - /@lancedb/lancedb-darwin-x64@0.22.3: - resolution: {integrity: sha512-wOwgZkvBgQM8asjolz4NeyPa8W/AjZv4fwyQxJhTqKTGlB3ntrpdn1m84K5qncTmFFDcDfGgZ4DkNVkVK+ydoQ==} - engines: {node: '>= 18'} - cpu: [x64] - os: [darwin] - requiresBuild: true - dev: false - optional: true - - /@lancedb/lancedb-linux-arm64-gnu@0.22.3: - resolution: {integrity: sha512-YUbFuBKQniTZOR9h2/es1f7lDzdHNt8qXs5GaqFmLQv2GNWpnvKXVA/vVffhCNpFB5nV132o1VhXW3KoMubPsw==} - engines: {node: '>= 18'} - cpu: [arm64] - os: [linux] - requiresBuild: true - dev: false - optional: true - - /@lancedb/lancedb-linux-arm64-musl@0.22.3: - resolution: {integrity: sha512-jVRMtXxxYaDlZSaclCIHB2N+NJvQ1Fj9EaPeBx+HxG2VqUg0vXKef+yiaD2aGo9sAH6mMmkKJsrPhwABpUC4rQ==} - engines: {node: '>= 18'} - cpu: [arm64] - os: [linux] - requiresBuild: true - dev: false - optional: true - - /@lancedb/lancedb-linux-x64-gnu@0.22.3: - resolution: {integrity: sha512-co7idTwvNAtbFoFHojhHlTpKsydOm5sZfbtAsQRdoa7g6a61yIrqrMm8D7Ngh756JfzZLFQBMkDUZEW3X4vP/g==} - engines: {node: '>= 18'} - cpu: [x64] - os: [linux] - requiresBuild: true - dev: false - optional: true - - /@lancedb/lancedb-linux-x64-musl@0.22.3: - resolution: {integrity: sha512-+ipFsn5PCODK7mOMq1gZ5OAZWks5YlgmjAlnYMmU8XxvaK0b6lZdA3s1hTmBaBO9+wv+31ulO55oBN4/U8Yldg==} - engines: {node: '>= 18'} - cpu: [x64] - os: [linux] - requiresBuild: true - dev: false - optional: true - - /@lancedb/lancedb-win32-arm64-msvc@0.22.3: - resolution: {integrity: sha512-E0XywJYnelIe4pzOlvog+aMHKt5ChW27tgmT2V80Z6PXcX6eN9I69Fj0Q6DK6z1YCTPIPu6Na1Hd6d4GqUNKPw==} - engines: {node: '>= 18'} - cpu: [arm64] - os: [win32] - requiresBuild: true - dev: false - optional: true - - /@lancedb/lancedb-win32-x64-msvc@0.22.3: - resolution: {integrity: sha512-/1feFnjz5MIhzXOEU4+1OeGwpAFYczGfefuOGZRsmGWDdt4V6/fza7Hkkxyb2OnTzqpBfy6BdW2+iBguE1JMyQ==} - engines: {node: '>= 18'} - cpu: [x64] - os: [win32] - requiresBuild: true - dev: false - optional: true - - /@lancedb/lancedb@0.22.3(apache-arrow@18.1.0): - resolution: {integrity: sha512-nRC0fkg+d7dzCtudKHT+VH7znk6KUXRZyuS6HJYNnIrbvXBxaT6wAPjEbf70KTuqvP2znj48Zg+kiwRqkRnAJw==} - engines: {node: '>= 18'} - os: [darwin, linux, win32] - peerDependencies: - apache-arrow: '>=15.0.0 <=18.1.0' - dependencies: - apache-arrow: 18.1.0 - reflect-metadata: 0.2.2 - optionalDependencies: - '@lancedb/lancedb-darwin-arm64': 0.22.3 - '@lancedb/lancedb-darwin-x64': 0.22.3 - '@lancedb/lancedb-linux-arm64-gnu': 0.22.3 - '@lancedb/lancedb-linux-arm64-musl': 0.22.3 - '@lancedb/lancedb-linux-x64-gnu': 0.22.3 - '@lancedb/lancedb-linux-x64-musl': 0.22.3 - '@lancedb/lancedb-win32-arm64-msvc': 0.22.3 - '@lancedb/lancedb-win32-x64-msvc': 0.22.3 - dev: false - /@manypkg/find-root@1.1.0: resolution: {integrity: sha512-mki5uBvhHzO8kYYix/WRy2WX8S3B5wdVSc9D6KcU5lQNglP2yt58/VfLuAK49glRXChosY8ap2oJ1qgma3GUVA==} dependencies: @@ -1609,49 +1510,6 @@ packages: '@nodelib/fs.scandir': 2.1.5 fastq: 1.19.1 - /@protobufjs/aspromise@1.1.2: - resolution: {integrity: sha512-j+gKExEuLmKwvz3OgROXtrJ2UG2x8Ch2YZUxahh+s1F2HZ+wAceUNLkvy6zKCPVRkU++ZWQrdxsUeQXmcg4uoQ==} - dev: false - - /@protobufjs/base64@1.1.2: - resolution: {integrity: sha512-AZkcAA5vnN/v4PDqKyMR5lx7hZttPDgClv83E//FMNhR2TMcLUhfRUBHCmSl0oi9zMgDDqRUJkSxO3wm85+XLg==} - dev: false - - /@protobufjs/codegen@2.0.4: - resolution: {integrity: sha512-YyFaikqM5sH0ziFZCN3xDC7zeGaB/d0IUb9CATugHWbd1FRFwWwt4ld4OYMPWu5a3Xe01mGAULCdqhMlPl29Jg==} - dev: false - - /@protobufjs/eventemitter@1.1.0: - resolution: {integrity: sha512-j9ednRT81vYJ9OfVuXG6ERSTdEL1xVsNgqpkxMsbIabzSo3goCjDIveeGv5d03om39ML71RdmrGNjG5SReBP/Q==} - dev: false - - /@protobufjs/fetch@1.1.0: - resolution: {integrity: sha512-lljVXpqXebpsijW71PZaCYeIcE5on1w5DlQy5WH6GLbFryLUrBD4932W/E2BSpfRJWseIL4v/KPgBFxDOIdKpQ==} - dependencies: - '@protobufjs/aspromise': 1.1.2 - '@protobufjs/inquire': 1.1.0 - dev: false - - /@protobufjs/float@1.0.2: - resolution: {integrity: sha512-Ddb+kVXlXst9d+R9PfTIxh1EdNkgoRe5tOX6t01f1lYWOvJnSPDBlG241QLzcyPdoNTsblLUdujGSE4RzrTZGQ==} - dev: false - - /@protobufjs/inquire@1.1.0: - resolution: {integrity: sha512-kdSefcPdruJiFMVSbn801t4vFK7KB/5gd2fYvrxhuJYg8ILrmn9SKSX2tZdV6V+ksulWqS7aXjBcRXl3wHoD9Q==} - dev: false - - /@protobufjs/path@1.1.2: - resolution: {integrity: sha512-6JOcJ5Tm08dOHAbdR3GrvP+yUUfkjG5ePsHYczMFLq3ZmMkAD98cDgcT2iA1lJ9NVwFd4tH/iSSoe44YWkltEA==} - dev: false - - /@protobufjs/pool@1.1.0: - resolution: {integrity: sha512-0kELaGSIDBKvcgS4zkjz1PeddatrjYcmMWOlAuAPwAeccUrPHdUqo/J6LiymHHEiJT5NrF1UVwxY14f+fy4WQw==} - dev: false - - /@protobufjs/utf8@1.1.0: - resolution: {integrity: sha512-Vvn3zZrhQZkkBE8LSuW3em98c0FwgO4nxzv6OdSxPKJIEKY2bGbHn+mhGIPerzI4twdxaP8/0+06HBpwf345Lw==} - dev: false - /@rollup/rollup-android-arm-eabi@4.52.4: resolution: {integrity: sha512-BTm2qKNnWIQ5auf4deoetINJm2JzvihvGb9R6K/ETwKLql/Bb3Eg2H1FBp1gUb4YGbydMA3jcmQTR73q7J+GAA==} cpu: [arm] @@ -1841,12 +1699,6 @@ packages: resolution: {integrity: sha512-m2bOd0f2RT9k8QJx1JN85cZYyH1RqFBdlwtkSlf4tBDYLCiiZnv1fIIwacK6cqwXavOydf0NPToMQgpKq+dVlA==} dev: true - /@swc/helpers@0.5.17: - resolution: {integrity: sha512-5IKx/Y13RsYd+sauPb2x+U/xZikHjolzfuDgTAl/Tdf3Q8rslRvC19NKDLgAJQ6wsqADk10ntlv08nPFw/gO/A==} - dependencies: - tslib: 2.8.1 - dev: false - /@ts-morph/common@0.28.1: resolution: {integrity: sha512-W74iWf7ILp1ZKNYXY5qbddNaml7e9Sedv5lvU1V8lftlitkc9Pq1A+jlH23ltDgWYeZFFEqGCD1Ies9hqu3O+g==} dependencies: @@ -1872,14 +1724,6 @@ packages: assertion-error: 2.0.1 dev: true - /@types/command-line-args@5.2.3: - resolution: {integrity: sha512-uv0aG6R0Y8WHZLTamZwtfsDLVRnOa+n+n5rEvFWL5Na5gZ8V2Teab/duDPFzIIIhs9qizDpcavCusCLJZu62Kw==} - dev: false - - /@types/command-line-usage@5.0.4: - resolution: {integrity: sha512-BwR5KP3Es/CSht0xqBcUXS3qCAUVXwpRKsV2+arxeb65atasuXG9LykC9Ab10Cw3s2raH92ZqOeILaQbsB2ACg==} - dev: false - /@types/conventional-commits-parser@5.0.2: resolution: {integrity: sha512-BgT2szDXnVypgpNxOK8aL5SGjUdaQbC++WZNjF1Qge3Og2+zhHj+RWhmehLhYyvQwqAmvezruVfOf8+3m74W+g==} dependencies: @@ -1900,10 +1744,6 @@ packages: resolution: {integrity: sha512-dWHzHa2WqEXI/O1E9OjrocMTKJl2mSrEolh1Iomrv6U+JuNwaHXsXx9bLu5gG7BUWFIN0skIQJQ/L1rIex4X6w==} dev: true - /@types/long@4.0.2: - resolution: {integrity: sha512-MqTGEo5bj5t157U6fA/BiDynNkn0YknVdh48CMPkTSpFTVmvao5UQmm7uEF6xBEo7qIMAlY/JSleYaE6VOdpaA==} - dev: false - /@types/mdast@4.0.4: resolution: {integrity: sha512-kGaNbPh1k7AFzgpud/gMdvIm5xuECykRR+JnWKQno9TAXVa6WIVCGTPvYGekIDL4uwCZQSYbUxNBSb1aUo79oA==} dependencies: @@ -1917,16 +1757,11 @@ packages: resolution: {integrity: sha512-J8xLz7q2OFulZ2cyGTLE1TbbZcjpno7FaN6zdJNrgAdrJ+DZzh/uFR6YrTb4C+nXakvud8Q4+rbhoIWlYQbUFQ==} dev: true - /@types/node@20.19.25: - resolution: {integrity: sha512-ZsJzA5thDQMSQO788d7IocwwQbI8B5OPzmqNvpf3NY/+MHDAS759Wo0gd2WQeXYt5AAAQjzcrTVC6SKCuYgoCQ==} - dependencies: - undici-types: 6.21.0 - dev: false - /@types/node@22.19.1: resolution: {integrity: sha512-LCCV0HdSZZZb34qifBsyWlUmok6W7ouER+oQIGBScS8EsZsQbrtFTUrDX4hOl+CS6p7cnNC4td+qrSVGSCTUfQ==} dependencies: undici-types: 6.21.0 + dev: true /@types/node@24.10.1: resolution: {integrity: sha512-GNWcUTRBgIRJD5zj+Tq0fKOJ5XZajIiBroOF0yvj2bSU1WvNdYS/dn9UxwsujGW4JX06dnHyjV2y9rRaybH0iQ==} @@ -2122,20 +1957,6 @@ packages: tinyrainbow: 3.0.3 dev: true - /@xenova/transformers@2.17.2: - resolution: {integrity: sha512-lZmHqzrVIkSvZdKZEx7IYY51TK0WDrC8eR0c5IMnBsO8di8are1zzw8BlLhyO2TklZKLN5UffNGs1IJwT6oOqQ==} - dependencies: - '@huggingface/jinja': 0.2.2 - onnxruntime-web: 1.14.0 - sharp: 0.32.6 - optionalDependencies: - onnxruntime-node: 1.14.0 - transitivePeerDependencies: - - bare-abort-controller - - bare-buffer - - react-native-b4a - dev: false - /JSONStream@1.3.5: resolution: {integrity: sha512-E+iruNOY8VV9s4JEbe1aNEm6MiszPRr/UfcHMz0TQh1BXSxHK+ASV1R6W4HpjBhSeS+54PIsAMCBmwD06LLsqQ==} hasBin: true @@ -2192,6 +2013,7 @@ packages: engines: {node: '>=8'} dependencies: color-convert: 2.0.1 + dev: true /ansi-styles@5.2.0: resolution: {integrity: sha512-Cxwpt2SfTzTtXcfOlzGEee8O+c+MmUgGrNiBcXnuWxuFJHe6a5Hz7qwhwe5OgaSYI0IJvkLqWX1ASG+cJOkEiA==} @@ -2207,21 +2029,6 @@ packages: resolution: {integrity: sha512-7UvmKalWRt1wgjL1RrGxoSJW/0QZFIegpeGvZG9kjp8vrRu55XTHbwnqq2GpXm9uLbcuhxm3IqX9OB4MZR1b2A==} dev: true - /apache-arrow@18.1.0: - resolution: {integrity: sha512-v/ShMp57iBnBp4lDgV8Jx3d3Q5/Hac25FWmQ98eMahUiHPXcvwIMKJD0hBIgclm/FCG+LwPkAKtkRO1O/W0YGg==} - hasBin: true - dependencies: - '@swc/helpers': 0.5.17 - '@types/command-line-args': 5.2.3 - '@types/command-line-usage': 5.0.4 - '@types/node': 20.19.25 - command-line-args: 5.2.1 - command-line-usage: 7.0.3 - flatbuffers: 24.12.23 - json-bignum: 0.0.3 - tslib: 2.8.1 - dev: false - /argparse@1.0.10: resolution: {integrity: sha512-o5Roy6tNG4SL/FOkCAN6RzjiakZS25RLYFrcMttJqbdd8BWrnA+fGz57iN5Pb06pvBGvl5gQ0B48dJlslXvoTg==} dependencies: @@ -2232,16 +2039,6 @@ packages: resolution: {integrity: sha512-8+9WqebbFzpX9OR+Wa6O29asIogeRMzcGtAINdpMHHyAg10f05aSFVBbcEqGf/PXw1EjAZ+q2/bEBg3DvurK3Q==} dev: true - /array-back@3.1.0: - resolution: {integrity: sha512-TkuxA4UCOvxuDK6NZYXCalszEzj+TLszyASooky+i742l9TqsOdYCMJJupxRic61hwquNtppB3hgcuq9SVSH1Q==} - engines: {node: '>=6'} - dev: false - - /array-back@6.2.2: - resolution: {integrity: sha512-gUAZ7HPyb4SJczXAMUXMGAvI976JoK3qEx9v1FTmeYuJj0IBiaKttG1ydtGKdkfqWkIkouke7nG8ufGy77+Cvw==} - engines: {node: '>=12.17'} - dev: false - /array-ify@1.0.0: resolution: {integrity: sha512-c5AMf34bKdvPhQ7tBGhqkgKNUzMr4WUs+WDtC2ZUGOUncbxKMTvqxYctiseW3+L4bA8ec+GcZ6/A/FW4m8ukng==} dev: true @@ -2268,92 +2065,10 @@ packages: js-tokens: 9.0.1 dev: true - /b4a@1.7.3: - resolution: {integrity: sha512-5Q2mfq2WfGuFp3uS//0s6baOJLMoVduPYVeNmDYxu5OUA1/cBfvr2RIS7vi62LdNj/urk1hfmj867I3qt6uZ7Q==} - peerDependencies: - react-native-b4a: '*' - peerDependenciesMeta: - react-native-b4a: - optional: true - dev: false - /bail@2.0.2: resolution: {integrity: sha512-0xO6mYd7JB2YesxDKplafRpsiOzPt9V02ddPCLbY1xYGPOX24NTyN50qnUxgCPcSoYMhKpAuBTjQoRZCAkUDRw==} dev: false - /bare-events@2.8.2: - resolution: {integrity: sha512-riJjyv1/mHLIPX4RwiK+oW9/4c3TEUeORHKefKAKnZ5kyslbN+HXowtbaVEqt4IMUB7OXlfixcs6gsFeo/jhiQ==} - peerDependencies: - bare-abort-controller: '*' - peerDependenciesMeta: - bare-abort-controller: - optional: true - dev: false - - /bare-fs@4.5.1: - resolution: {integrity: sha512-zGUCsm3yv/ePt2PHNbVxjjn0nNB1MkIaR4wOCxJ2ig5pCf5cCVAYJXVhQg/3OhhJV6DB1ts7Hv0oUaElc2TPQg==} - engines: {bare: '>=1.16.0'} - requiresBuild: true - peerDependencies: - bare-buffer: '*' - peerDependenciesMeta: - bare-buffer: - optional: true - dependencies: - bare-events: 2.8.2 - bare-path: 3.0.0 - bare-stream: 2.7.0(bare-events@2.8.2) - bare-url: 2.3.2 - fast-fifo: 1.3.2 - transitivePeerDependencies: - - bare-abort-controller - - react-native-b4a - dev: false - optional: true - - /bare-os@3.6.2: - resolution: {integrity: sha512-T+V1+1srU2qYNBmJCXZkUY5vQ0B4FSlL3QDROnKQYOqeiQR8UbjNHlPa+TIbM4cuidiN9GaTaOZgSEgsvPbh5A==} - engines: {bare: '>=1.14.0'} - requiresBuild: true - dev: false - optional: true - - /bare-path@3.0.0: - resolution: {integrity: sha512-tyfW2cQcB5NN8Saijrhqn0Zh7AnFNsnczRcuWODH0eYAXBsJ5gVxAUuNr7tsHSC6IZ77cA0SitzT+s47kot8Mw==} - requiresBuild: true - dependencies: - bare-os: 3.6.2 - dev: false - optional: true - - /bare-stream@2.7.0(bare-events@2.8.2): - resolution: {integrity: sha512-oyXQNicV1y8nc2aKffH+BUHFRXmx6VrPzlnaEvMhram0nPBrKcEdcyBg5r08D0i8VxngHFAiVyn1QKXpSG0B8A==} - requiresBuild: true - peerDependencies: - bare-buffer: '*' - bare-events: '*' - peerDependenciesMeta: - bare-buffer: - optional: true - bare-events: - optional: true - dependencies: - bare-events: 2.8.2 - streamx: 2.23.0 - transitivePeerDependencies: - - bare-abort-controller - - react-native-b4a - dev: false - optional: true - - /bare-url@2.3.2: - resolution: {integrity: sha512-ZMq4gd9ngV5aTMa5p9+UfY0b3skwhHELaDkhEHetMdX0LRkW9kzaym4oo/Eh+Ghm0CCDuMTsRIGM/ytUc1ZYmw==} - requiresBuild: true - dependencies: - bare-path: 3.0.0 - dev: false - optional: true - /base64-js@1.5.1: resolution: {integrity: sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA==} dev: false @@ -2450,21 +2165,6 @@ packages: engines: {node: '>=18'} dev: true - /chalk-template@0.4.0: - resolution: {integrity: sha512-/ghrgmhfY8RaSdeo43hNXxpoHAtxdbskUHjPpfqUWGttFgycUhYPGx3YZBCnUCvOa7Doivn1IZec3DEGFoMgLg==} - engines: {node: '>=12'} - dependencies: - chalk: 4.1.2 - dev: false - - /chalk@4.1.2: - resolution: {integrity: sha512-oKnbhFyRIXpUuez8iBMmyEa4nbj4IOQyuhc/wy9kY7/WVPcwIO9VA668Pu8RkO7+0G76SLROeyw9CpQ061i4mA==} - engines: {node: '>=10'} - dependencies: - ansi-styles: 4.3.0 - supports-color: 7.2.0 - dev: false - /chalk@5.6.2: resolution: {integrity: sha512-7NzBL0rN6fMUW+f7A6Io4h40qQlG+xGmtMxfbnH/K7TAtt8JQWVQK+6g0UXKMeVJoyV5EkkNsErQ8pVD3bLHbA==} engines: {node: ^12.17.0 || ^14.13 || >=16.0.0} @@ -2543,44 +2243,11 @@ packages: engines: {node: '>=7.0.0'} dependencies: color-name: 1.1.4 + dev: true /color-name@1.1.4: resolution: {integrity: sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA==} - - /color-string@1.9.1: - resolution: {integrity: sha512-shrVawQFojnZv6xM40anx4CkoDP+fZsw/ZerEMsW/pyzsRbElpsL/DBVW7q3ExxwusdNXI3lXpuhEZkzs8p5Eg==} - dependencies: - color-name: 1.1.4 - simple-swizzle: 0.2.4 - dev: false - - /color@4.2.3: - resolution: {integrity: sha512-1rXeuUUiGGrykh+CeBdu5Ie7OJwinCgQY0bc7GCRxy5xVHy+moaqkpL/jqQq0MtQOeYcrqEz4abc5f0KtU7W4A==} - engines: {node: '>=12.5.0'} - dependencies: - color-convert: 2.0.1 - color-string: 1.9.1 - dev: false - - /command-line-args@5.2.1: - resolution: {integrity: sha512-H4UfQhZyakIjC74I9d34fGYDwk3XpSr17QhEd0Q3I9Xq1CETHo4Hcuo87WyWHpAF1aSLjLRf5lD9ZGX2qStUvg==} - engines: {node: '>=4.0.0'} - dependencies: - array-back: 3.1.0 - find-replace: 3.0.0 - lodash.camelcase: 4.3.0 - typical: 4.0.0 - dev: false - - /command-line-usage@7.0.3: - resolution: {integrity: sha512-PqMLy5+YGwhMh1wS04mVG44oqDsgyLRSKJBdOo1bnYhMKBW65gZF1dRp2OZRhiTjgUHljy99qkO7bsctLaw35Q==} - engines: {node: '>=12.20.0'} - dependencies: - array-back: 6.2.2 - chalk-template: 0.4.0 - table-layout: 4.1.1 - typical: 7.3.0 - dev: false + dev: true /commander@12.1.0: resolution: {integrity: sha512-Vw8qHK3bZM9y/P10u3Vib8o/DdkvA2OtPtZvD871QKjy74Wj1WSKFILMPRPSdUSx5RFK1arlJzEtA4PkFgnbuA==} @@ -2915,14 +2582,6 @@ packages: '@types/estree': 1.0.8 dev: true - /events-universal@1.0.1: - resolution: {integrity: sha512-LUd5euvbMLpwOF8m6ivPCbhQeSiYVNb8Vs0fQ8QjXo0JTkEHpz8pxdQf0gStltaPpw0Cca8b39KxvK9cfKRiAw==} - dependencies: - bare-events: 2.8.2 - transitivePeerDependencies: - - bare-abort-controller - dev: false - /execa@8.0.1: resolution: {integrity: sha512-VyhnebXciFV2DESc+p6B+y0LjSm0krU4OgJN44qFAhBY0TJ+1V61tYD2+wHusZ6F9n5K+vl8k0sTy7PEfV4qpg==} engines: {node: '>=16.17'} @@ -2960,10 +2619,6 @@ packages: resolution: {integrity: sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q==} dev: true - /fast-fifo@1.3.2: - resolution: {integrity: sha512-/d9sfos4yxzpwkDkuN7k2SqFKtYNmCTzgfEpz82x34IM9/zc8KGxQoXg1liNC/izpRM/MBdt44Nmx41ZWqk+FQ==} - dev: false - /fast-glob@3.3.3: resolution: {integrity: sha512-7MptL8U0cqcFdzIzwOTHoilX9x5BrNqye7Z/LuC7kCMRio1EMSyqRK3BEAUD7sXRq4iT4AzTVuZdhgQ2TCvYLg==} engines: {node: '>=8.6.0'} @@ -3004,13 +2659,6 @@ packages: dependencies: to-regex-range: 5.0.1 - /find-replace@3.0.0: - resolution: {integrity: sha512-6Tb2myMioCAgv5kfvP5/PkZZ/ntTpVK39fHY7WkWBgvbeE+VHd/tZuZ4mrC+bxh4cfOZeYKVPaJIZtZXV7GNCQ==} - engines: {node: '>=4.0.0'} - dependencies: - array-back: 3.1.0 - dev: false - /find-up@4.1.0: resolution: {integrity: sha512-PpOwAdQ/YlXQ2vj8a3h8IipDuYRi3wceVQQGYWxNINccq40Anw7BlsEXCMbt1Zt+OLA6Fq9suIpIWD0OsnISlw==} engines: {node: '>=8'} @@ -3036,14 +2684,6 @@ packages: rollup: 4.52.4 dev: true - /flatbuffers@1.12.0: - resolution: {integrity: sha512-c7CZADjRcl6j0PlvFy0ZqXQ67qSEZfrVPynmnL+2zPc+NtMvrF8Y0QceMo7QqnSPc7+uWjUIAbvCQ5WIKlMVdQ==} - dev: false - - /flatbuffers@24.12.23: - resolution: {integrity: sha512-dLVCAISd5mhls514keQzmEG6QHmUUsNuWsb4tFafIUwvvgDjXhtfAYSKOzt5SWOy+qByV5pbsDZ+Vb7HUOBEdA==} - dev: false - /fs-constants@1.0.0: resolution: {integrity: sha512-y6OAwoSIf7FyjMIv94u+b5rdheZEjzR63GTyZJm5qh4Bi+2YgwLCcI/fPFZkL5PSixOt6ZNKm+w+Hfp/Bciwow==} dev: false @@ -3148,13 +2788,10 @@ packages: resolution: {integrity: sha512-RbJ5/jmFcNNCcDV5o9eTnBLJ/HszWV0P73bc+Ff4nS/rJj+YaS6IGyiOL0VoBYX+l1Wrl3k63h/KrH+nhJ0XvQ==} dev: true - /guid-typescript@1.0.9: - resolution: {integrity: sha512-Y8T4vYhEfwJOTbouREvG+3XDsjr8E3kIr7uf+JZ0BYloFsttiHU0WfvANVsR7TxNUJa/WpCnw/Ino/p+DeBhBQ==} - dev: false - /has-flag@4.0.0: resolution: {integrity: sha512-EykJT/Q1KjTWctppgIAgfSO0tKVuZUjhgMr17kqTumMl6Afv3EISleU7qZUzoXDFTAHTDC4NOoG/ZxU3EvlMPQ==} engines: {node: '>=8'} + dev: true /html-escaper@2.0.2: resolution: {integrity: sha512-H2iMtd0I4Mt5eYiapRdIDjp+XzelXQ0tFE4JS7YFwFevXXMmOp9myNrUvCg0D6ws8iqkRPBfKHgbwig1SmlLfg==} @@ -3226,10 +2863,6 @@ packages: resolution: {integrity: sha512-zz06S8t0ozoDXMG+ube26zeCTNXcKIPJZJi8hBrF4idCLms4CG9QtK7qBl1boi5ODzFpjswb5JPmHCbMpjaYzg==} dev: true - /is-arrayish@0.3.4: - resolution: {integrity: sha512-m6UrgzFVUYawGBh1dUsWR5M2Clqic9RVXC/9f8ceNlv2IcO9j9J/z8UoCLPqtsPBFNzEpfR3xftohbfqDx8EQA==} - dev: false - /is-extglob@2.1.1: resolution: {integrity: sha512-SbKbANkN603Vi4jEZv49LeVJMn4yGwsbzZworEoyEiutsN3nJYdbO36zfhGJ6QEDpOZIFkDtnq5JRxmvl3jsoQ==} engines: {node: '>=0.10.0'} @@ -3379,11 +3012,6 @@ packages: argparse: 2.0.1 dev: true - /json-bignum@0.0.3: - resolution: {integrity: sha512-2WHyXj3OfHSgNyuzDbSxI1w2jgw5gkWSWhS7Qg4bWXx1nLk3jnbwfUeS0PSba3IzpTUWdHxBieELUzXRjQB2zg==} - engines: {node: '>=0.8'} - dev: false - /json-parse-even-better-errors@2.3.1: resolution: {integrity: sha512-xyFwyhro/JEof6Ghe2iz2NcXoj2sloNsWr/XsERDK/oiPCfaNhl5ONfp+jQdAZRQQ0IJWNzH9zIZF7li91kh2w==} dev: true @@ -3441,6 +3069,7 @@ packages: /lodash.camelcase@4.3.0: resolution: {integrity: sha512-TwuEnCnxbc3rAvhf/LbG7tJUDzhqXyFnv3dtzLOPgCG/hODL7WFnsbwktkD7yUV0RrreP/l1PALq/YSg6VvjlA==} + dev: true /lodash.isplainobject@4.0.6: resolution: {integrity: sha512-oSXzaWypCMHkPC3NvBEaPHf0KsA5mvPrOPgQWDsbg8n7orZ290M0BmC/jgRZ4vcJ6DTAhjrsSYgdsW/F+MFOBA==} @@ -3493,10 +3122,6 @@ packages: wrap-ansi: 9.0.2 dev: false - /long@4.0.0: - resolution: {integrity: sha512-XsP+KhQif4bjX1kbuSiySJFNAehNxgLb6hPRGJ9QsUr8ajHkuXGdrHmFUTUUXhDwVX2R5bY4JNZEwbUiMhV+MA==} - dev: false - /longest-streak@3.1.0: resolution: {integrity: sha512-9Ri+o0JYgehTaVBBDoMqIl8GXtbWg711O3srftcHhZ0dqnETqLaoIK0x17fUw9rFSlK/0NlsKe0Ahhyl5pXE2g==} dev: false @@ -3844,10 +3469,6 @@ packages: semver: 7.7.3 dev: false - /node-addon-api@6.1.0: - resolution: {integrity: sha512-+eawOlIgy680F0kBzPUNFhMZGtJ1YmqM6l4+Crf4IkImjYrO/mqPwRMh352g23uIaQKFItcQ64I7KMaJxHgAVA==} - dev: false - /npm-run-path@5.3.0: resolution: {integrity: sha512-ppwTtiJZq0O/ai0z7yfudtBpWIoxM8yE6nHi1X47eFR2EWORqfbu6CnPlNsjeN683eT0qG6H/Pyf9fCcvjnnnQ==} engines: {node: ^12.20.0 || ^14.13.1 || >=16.0.0} @@ -3880,34 +3501,14 @@ packages: mimic-function: 5.0.1 dev: false - /onnx-proto@4.0.4: - resolution: {integrity: sha512-aldMOB3HRoo6q/phyB6QRQxSt895HNNw82BNyZ2CMh4bjeKv7g/c+VpAFtJuEMVfYLMbRx61hbuqnKceLeDcDA==} - dependencies: - protobufjs: 6.11.4 - dev: false - - /onnxruntime-common@1.14.0: - resolution: {integrity: sha512-3LJpegM2iMNRX2wUmtYfeX/ytfOzNwAWKSq1HbRrKc9+uqG/FsEA0bbKZl1btQeZaXhC26l44NWpNUeXPII7Ew==} - dev: false - - /onnxruntime-node@1.14.0: - resolution: {integrity: sha512-5ba7TWomIV/9b6NH/1x/8QEeowsb+jBEvFzU6z0T4mNsFwdPqXeFUM7uxC6QeSRkEbWu3qEB0VMjrvzN/0S9+w==} - os: [win32, darwin, linux] - requiresBuild: true + /openapi-fetch@0.17.0: + resolution: {integrity: sha512-PsbZR1wAPcG91eEthKhN+Zn92FMHxv+/faECIwjXdxfTODGSGegYv0sc1Olz+HYPvKOuoXfp+0pA2XVt2cI0Ig==} dependencies: - onnxruntime-common: 1.14.0 + openapi-typescript-helpers: 0.1.0 dev: false - optional: true - /onnxruntime-web@1.14.0: - resolution: {integrity: sha512-Kcqf43UMfW8mCydVGcX9OMXI2VN17c0p6XvR7IPSZzBf/6lteBzXHvcEVWDPmCKuGombl997HgLqj91F11DzXw==} - dependencies: - flatbuffers: 1.12.0 - guid-typescript: 1.0.9 - long: 4.0.0 - onnx-proto: 4.0.4 - onnxruntime-common: 1.14.0 - platform: 1.3.6 + /openapi-typescript-helpers@0.1.0: + resolution: {integrity: sha512-OKTGPthhivLw/fHz6c3OPtg72vi86qaMlqbJuVJ23qOvQ+53uw1n7HdmkJFibloF7QEjDrDkzJiOJuockM/ljw==} dev: false /ora@8.2.0: @@ -4080,10 +3681,6 @@ packages: pathe: 2.0.3 dev: true - /platform@1.3.6: - resolution: {integrity: sha512-fnWVljUchTro6RiCFvCXBbNhJc2NijN7oIQxbwsyL0buWJPG85v81ehlHI9fXrJsMNgTofEoWIQeClKpgxFLrg==} - dev: false - /postcss-load-config@6.0.1: resolution: {integrity: sha512-oPtTM4oerL+UXmx+93ytZVN82RrlY/wPUV8IeDxFrzIjXOLF1pN+EmKPLbubvKHT2HC20xXsCAH2Z+CKV6Oz/g==} engines: {node: '>= 18'} @@ -4148,26 +3745,6 @@ packages: react-is: 18.3.1 dev: true - /protobufjs@6.11.4: - resolution: {integrity: sha512-5kQWPaJHi1WoCpjTGszzQ32PG2F4+wRY6BmAT4Vfw56Q2FZ4YZzK20xUYQH4YkfehY1e6QSICrJquM6xXZNcrw==} - hasBin: true - requiresBuild: true - dependencies: - '@protobufjs/aspromise': 1.1.2 - '@protobufjs/base64': 1.1.2 - '@protobufjs/codegen': 2.0.4 - '@protobufjs/eventemitter': 1.1.0 - '@protobufjs/fetch': 1.1.0 - '@protobufjs/float': 1.0.2 - '@protobufjs/inquire': 1.1.0 - '@protobufjs/path': 1.1.2 - '@protobufjs/pool': 1.1.0 - '@protobufjs/utf8': 1.1.0 - '@types/long': 4.0.2 - '@types/node': 22.19.1 - long: 4.0.0 - dev: false - /pump@3.0.3: resolution: {integrity: sha512-todwxLMY7/heScKmntwQG8CXVkWUOdYxIvY2s0VWAAMh/nd8SoYiRaKjlr7+iCs984f2P8zvrfWcDDYVb73NfA==} dependencies: @@ -4220,10 +3797,6 @@ packages: engines: {node: '>= 14.18.0'} dev: true - /reflect-metadata@0.2.2: - resolution: {integrity: sha512-urBwgfrvVP/eAyXx4hluJivBKzuEbSQs9rKWCrCkbSxNv8mxPcUZKeuoF3Uy4mJl3Lwprp6yy5/39VWigZ4K6Q==} - dev: false - /remark-parse@11.0.0: resolution: {integrity: sha512-FCxlKLNGknS5ba/1lmpYijMUzX2esxW5xQqjWxw2eHFfS2MSdaHVINFmhjo+qN1WhZhNimq0dZATN9pH0IDrpA==} dependencies: @@ -4336,25 +3909,6 @@ packages: engines: {node: '>=10'} hasBin: true - /sharp@0.32.6: - resolution: {integrity: sha512-KyLTWwgcR9Oe4d9HwCwNM2l7+J0dUQwn/yf7S0EnTtb0eVS4RxO0eUSvxPtzT4F3SY+C4K6fqdv/DO27sJ/v/w==} - engines: {node: '>=14.15.0'} - requiresBuild: true - dependencies: - color: 4.2.3 - detect-libc: 2.1.2 - node-addon-api: 6.1.0 - prebuild-install: 7.1.3 - semver: 7.7.3 - simple-get: 4.0.1 - tar-fs: 3.1.1 - tunnel-agent: 0.6.0 - transitivePeerDependencies: - - bare-abort-controller - - bare-buffer - - react-native-b4a - dev: false - /shebang-command@2.0.0: resolution: {integrity: sha512-kHxr2zZpYtdmrN1qDjrrX/Z1rR1kG8Dx+gkpK1G4eXmvXswmcE1hTWBWYUzlraYw1/yZp6YuDY77YtvbN0dmDA==} engines: {node: '>=8'} @@ -4387,12 +3941,6 @@ packages: simple-concat: 1.0.1 dev: false - /simple-swizzle@0.2.4: - resolution: {integrity: sha512-nAu1WFPQSMNr2Zn9PGSZK9AGn4t/y97lEm+MXTtUDwfP0ksAIX4nO+6ruD9Jwut4C49SB1Ws+fbXsm/yScWOHw==} - dependencies: - is-arrayish: 0.3.4 - dev: false - /slash@3.0.0: resolution: {integrity: sha512-g9Q1haeby36OSStwb4ntCGGGaKsaVSjQ68fBxoQcutl5fS1vuY18H3wSt3jFyFtrkx+Kz0V1G85A4MyAdDMi2Q==} engines: {node: '>=8'} @@ -4450,17 +3998,6 @@ packages: engines: {node: '>=18'} dev: false - /streamx@2.23.0: - resolution: {integrity: sha512-kn+e44esVfn2Fa/O0CPFcex27fjIL6MkVae0Mm6q+E6f0hWv578YCERbv+4m02cjxvDsPKLnmxral/rR6lBMAg==} - dependencies: - events-universal: 1.0.1 - fast-fifo: 1.3.2 - text-decoder: 1.2.3 - transitivePeerDependencies: - - bare-abort-controller - - react-native-b4a - dev: false - /string-width@4.2.3: resolution: {integrity: sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==} engines: {node: '>=8'} @@ -4537,14 +4074,7 @@ packages: engines: {node: '>=8'} dependencies: has-flag: 4.0.0 - - /table-layout@4.1.1: - resolution: {integrity: sha512-iK5/YhZxq5GO5z8wb0bY1317uDF3Zjpha0QFFLA8/trAoiLbQD0HUbMesEaxyzUgDxi2QlcbM8IvqOlEjgoXBA==} - engines: {node: '>=12.17'} - dependencies: - array-back: 6.2.2 - wordwrapjs: 5.1.1 - dev: false + dev: true /tar-fs@2.1.4: resolution: {integrity: sha512-mDAjwmZdh7LTT6pNleZ05Yt65HC3E+NiQzl672vQG38jIrehtJk/J3mNwIg+vShQPcLF/LV7CMnDW6vjj6sfYQ==} @@ -4555,20 +4085,6 @@ packages: tar-stream: 2.2.0 dev: false - /tar-fs@3.1.1: - resolution: {integrity: sha512-LZA0oaPOc2fVo82Txf3gw+AkEd38szODlptMYejQUhndHMLQ9M059uXR+AfS7DNo0NpINvSqDsvyaCrBVkptWg==} - dependencies: - pump: 3.0.3 - tar-stream: 3.1.7 - optionalDependencies: - bare-fs: 4.5.1 - bare-path: 3.0.0 - transitivePeerDependencies: - - bare-abort-controller - - bare-buffer - - react-native-b4a - dev: false - /tar-stream@2.2.0: resolution: {integrity: sha512-ujeqbceABgwMZxEJnk2HDY2DlnUZ+9oEcb1KzTVfYHio0UE6dG71n60d8D2I4qNvleWrrXpmjpt7vZeF1LnMZQ==} engines: {node: '>=6'} @@ -4580,17 +4096,6 @@ packages: readable-stream: 3.6.2 dev: false - /tar-stream@3.1.7: - resolution: {integrity: sha512-qJj60CXt7IU1Ffyc3NJMjh6EkuCFej46zUqJ4J7pqYlThyd9bO0XBTmcOIhSzZJVWfsLks0+nle/j538YAW9RQ==} - dependencies: - b4a: 1.7.3 - fast-fifo: 1.3.2 - streamx: 2.23.0 - transitivePeerDependencies: - - bare-abort-controller - - react-native-b4a - dev: false - /term-size@2.2.1: resolution: {integrity: sha512-wK0Ri4fOGjv/XPy8SBHZChl8CM7uMc5VML7SqiQ0zG7+J5Vr+RMQDoHa2CNT6KHUnTGIXH34UDMkPzAUyapBZg==} engines: {node: '>=8'} @@ -4601,14 +4106,6 @@ packages: engines: {node: '>=18'} dev: false - /text-decoder@1.2.3: - resolution: {integrity: sha512-3/o9z3X0X0fTupwsYvR03pJ/DjWuqqrfwBgTQzdWDiQSm9KitAyz/9WqsT2JQW7KV2m+bC2ol/zqpW37NHxLaA==} - dependencies: - b4a: 1.7.3 - transitivePeerDependencies: - - react-native-b4a - dev: false - /text-extensions@2.4.0: resolution: {integrity: sha512-te/NtwBwfiNRLf9Ijqx3T0nlqZiQ2XrrtBvu+cLL8ZRrGkO0NHTug8MYFKyoSrv/sHTaSKfilUkizV6XhxMJ3g==} engines: {node: '>=8'} @@ -4710,10 +4207,6 @@ packages: code-block-writer: 13.0.3 dev: false - /tslib@2.8.1: - resolution: {integrity: sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==} - dev: false - /tsup@8.5.1(typescript@5.9.3): resolution: {integrity: sha512-xtgkqwdhpKWr3tKPmCkvYmS9xnQK3m3XgxZHwSUjvfTjp7YfXe5tT3GgWi0F2N+ZSMsOeWeZFh7ZZFg5iPhing==} engines: {node: '>=18'} @@ -4835,22 +4328,13 @@ packages: hasBin: true dev: true - /typical@4.0.0: - resolution: {integrity: sha512-VAH4IvQ7BDFYglMd7BPRDfLgxZZX4O4TFcRDA6EN5X7erNJJq+McIEp8np9aVtxrCJ6qx4GTYVfOWNjcqwZgRw==} - engines: {node: '>=8'} - dev: false - - /typical@7.3.0: - resolution: {integrity: sha512-ya4mg/30vm+DOWfBg4YK3j2WD6TWtRkCbasOJr40CseYENzCUby/7rIvXA99JGsQHeNxLbnXdyLLxKSv3tauFw==} - engines: {node: '>=12.17'} - dev: false - /ufo@1.6.1: resolution: {integrity: sha512-9a4/uxlTWJ4+a5i0ooc1rU7C7YOw3wT+UGqdeNNHWnOF9qcMBgLRS+4IYUqbczewFx4mLEig6gawh7X6mFlEkA==} dev: true /undici-types@6.21.0: resolution: {integrity: sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==} + dev: true /undici-types@7.16.0: resolution: {integrity: sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw==} @@ -5312,11 +4796,6 @@ packages: stackback: 0.0.2 dev: true - /wordwrapjs@5.1.1: - resolution: {integrity: sha512-0yweIbkINJodk27gX9LBGMzyQdBDan3s/dEAiwBOj+Mf0PPyWL6/rikalkv8EeD0E8jm4o5RXEOrFTP3NXbhJg==} - engines: {node: '>=12.17'} - dev: false - /wrap-ansi@7.0.0: resolution: {integrity: sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q==} engines: {node: '>=10'} diff --git a/website/content/docs/architecture.mdx b/website/content/docs/architecture.mdx index 79b24bb..fa7831a 100644 --- a/website/content/docs/architecture.mdx +++ b/website/content/docs/architecture.mdx @@ -37,7 +37,7 @@ dev-agent is built as a TypeScript monorepo with specialized packages for differ │ │ │ │ │ │ ┌──────▼─────────────────▼─────────────────▼───────────────────┐ │ │ │ Vector Storage │ │ -│ │ (LanceDB) │ │ +│ │ (Antfly — hybrid search) │ │ │ └──────────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────┘ ``` @@ -52,8 +52,8 @@ The foundation layer providing: - TypeScript/JavaScript via ts-morph - Go via tree-sitter WASM - Markdown via remark -- **Indexer** — Converts code components to vector embeddings -- **Vector Storage** — LanceDB for fast similarity search +- **Indexer** — Sends code components to Antfly for embedding and storage +- **Vector Storage** — Antfly for hybrid search (BM25 + vector + RRF) - **GitHub Integration** — Fetches and indexes issues/PRs via GitHub CLI ### @prosdevlab/dev-agent-cli @@ -110,9 +110,8 @@ Centralized logging: | Build | Turborepo | Monorepo orchestration | | Testing | Vitest | 1500+ tests | | Linting | Biome | Fast linting/formatting | -| Vector DB | LanceDB | Embedded vector storage | -| Embeddings | @xenova/transformers | Local ML inference | -| Model | all-MiniLM-L6-v2 | Sentence embeddings | +| Search | [Antfly](https://antfly.io) | Hybrid search (BM25 + vector + RRF) | +| Embeddings | Termite (via Antfly) | Local ONNX inference (bge-small-en-v1.5) | | Protocol | MCP | AI tool integration | | TS/JS Parser | ts-morph | TypeScript Compiler API | | Go Parser | tree-sitter WASM | Fast, portable parsing | @@ -126,14 +125,10 @@ Source Code │ ▼ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ -│ Glob │ ──▶ │ Parse │ ──▶ │ Extract │ ──▶ │ Embed │ -│ Files │ │ AST │ │Components│ │ Vectors │ +│ Glob │ ──▶ │ Parse │ ──▶ │ Extract │ ──▶ │ Store │ +│ Files │ │ AST │ │Components│ │ (Antfly) │ └──────────┘ └──────────┘ └──────────┘ └──────────┘ - │ - ▼ - ┌──────────┐ - │ LanceDB │ - └──────────┘ + Antfly auto-embeds via Termite ``` ### Query Flow @@ -143,32 +138,24 @@ User Query: "Where is auth handled?" │ ▼ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ -│ Embed │ ──▶ │ Vector │ ──▶ │ Rank │ ──▶ │ Format │ -│ Query │ │ Search │ │ Results │ │ Output │ +│ Hybrid │ ──▶ │BM25+Vec │ ──▶ │ RRF │ ──▶ │ Format │ +│ Query │ │ Search │ │ Fusion │ │ Output │ └──────────┘ └──────────┘ └──────────┘ └──────────┘ ``` ## Storage -All data is stored locally in `~/.dev-agent/`: +Data is stored locally by Antfly (default `~/.antfly/`) and dev-agent config in `~/.dev-agent/`. -``` -~/.dev-agent/ -├── indexes/ -│ └── / -│ ├── code.lance/ # Code vector index -│ └── metadata.json # Index metadata -├── github/ -│ └── / -│ ├── issues.lance/ # Issues vector index -│ └── prs.lance/ # PRs vector index -└── config.json # Global config (optional) -``` +Antfly manages its own storage, indexes, and embeddings. dev-agent creates three tables: +- `dev-agent-{project}-code` — code components +- `dev-agent-{project}-git` — git commit history +- `dev-agent-{project}-github` — GitHub issues/PRs ## Security - **100% Local** — No data sent to external services - **No Cloud** — All processing happens on your machine -- **Embeddings** — Generated locally via @xenova/transformers +- **Embeddings** — Generated locally via Antfly's Termite engine (ONNX) - **GitHub Data** — Fetched via authenticated GitHub CLI diff --git a/website/content/docs/install.mdx b/website/content/docs/install.mdx index 198819e..0c546fc 100644 --- a/website/content/docs/install.mdx +++ b/website/content/docs/install.mdx @@ -3,7 +3,7 @@ ## Requirements - **Node.js 22+** (LTS recommended) -- **pnpm**, **npm**, or **yarn** +- **[Docker Desktop](https://docker.com/get-started)** (recommended) or [Antfly](https://antfly.io) native binary - **Cursor** or **Claude Code** (for MCP integration) ## Install dev-agent @@ -29,6 +29,23 @@ Verify the installation: dev --version ``` +## One-time setup + +Run `dev setup` to start the search backend: + +```bash +dev setup +``` + +This does three things: +1. **Checks for Docker** (preferred) or native Antfly binary +2. **Starts the Antfly server** — handles hybrid search and embeddings locally +3. **Verifies the connection** — confirms everything is working + +If Docker isn't available, `dev setup` falls back to the native binary and offers to install it. + +> **What is Antfly?** [Antfly](https://antfly.io) is the search engine that powers dev-agent. It runs locally on your machine — your code never leaves. It provides hybrid search (BM25 keyword matching + vector similarity) which is significantly better than pure vector search for code. + ## Setup for Cursor ### 1. Index your repository @@ -40,8 +57,6 @@ cd /path/to/your/project dev index . ``` -This creates a local vector database in `~/.dev-agent/indexes/`. - ### 2. Install MCP integration ```bash diff --git a/website/content/docs/quickstart.mdx b/website/content/docs/quickstart.mdx index 1e73e4e..7351208 100644 --- a/website/content/docs/quickstart.mdx +++ b/website/content/docs/quickstart.mdx @@ -5,15 +5,19 @@ Get from zero to semantic search in 5 minutes. ## Prerequisites - Node.js 22+ installed +- Docker Desktop (recommended) or Antfly native binary - Cursor IDE (or Claude Code) - A code repository to index -## Step 1: Install dev-agent +## Step 1: Install and setup ```bash npm install -g @prosdevlab/dev-agent +dev setup ``` +`dev setup` starts the search backend (via Docker or native). One-time step. + ## Step 2: Index your repository ```bash @@ -70,15 +74,14 @@ Try these prompts: When you run `dev index .`: -1. **Scanner** parses your code using the TypeScript Compiler API +1. **Scanner** parses your code using the TypeScript Compiler API (ts-morph) and tree-sitter (Go) 2. **Extractor** identifies functions, classes, interfaces, types, arrow functions, hooks, and exported constants -3. **Embedder** generates vector embeddings locally (MiniLM-L6-v2) -4. **Storage** saves everything to LanceDB (`~/.dev-agent/indexes/`) +3. **Storage** sends documents to Antfly, which generates embeddings locally via Termite (ONNX) When AI tools call `dev_search`: -1. **Query** gets embedded using the same model -2. **Vector search** finds semantically similar code +1. **Hybrid search** runs BM25 (keyword matching) + vector similarity in parallel +2. **RRF fusion** combines both signals into a single ranked result set 3. **Formatter** returns results optimized for LLM context windows ## Next Steps diff --git a/website/content/docs/troubleshooting.mdx b/website/content/docs/troubleshooting.mdx index a65fe1c..d199d8a 100644 --- a/website/content/docs/troubleshooting.mdx +++ b/website/content/docs/troubleshooting.mdx @@ -126,14 +126,51 @@ gh auth login dev github index # Re-index ``` +## Antfly Issues + +### Antfly server not running + +If you see `Failed to initialize Antfly store` or `fetch failed`: + +```bash +# Run setup (starts the server via Docker or native) +dev setup + +# Or start manually +antfly swarm +``` + +### Port conflict on startup + +Antfly uses port 18080 (Docker) or 8080 (native). If another service is using that port: + +```bash +# Check what's using the port +lsof -i :18080 + +# Use a custom port via environment variable +ANTFLY_URL=http://localhost:19090/api/v1 dev index . +``` + +### Old LanceDB data + +If you're upgrading from a previous version that used LanceDB: + +```bash +# Old indexes are not compatible — re-index with Antfly +dev index . --force + +# Optionally clean up old LanceDB data +rm -rf ~/.dev-agent/indexes/*/vectors* +``` + ## Quick Fixes ### Clear everything and start fresh ```bash -rm -rf ~/.dev-agent/indexes/* -dev index . -dev github index +dev setup # Ensure Antfly is running +dev index . --force # Re-index from scratch dev mcp install --cursor ``` diff --git a/website/content/index.mdx b/website/content/index.mdx index 6ce3407..f3fb7f2 100644 --- a/website/content/index.mdx +++ b/website/content/index.mdx @@ -134,24 +134,24 @@ flowchart LR subgraph IDE["Your AI Tool"] A["Cursor / Claude Code"] end - + subgraph Agent["dev-agent"] B["MCP Server"] C["9 Tools"] end - - subgraph Local["Local Storage"] - D["Vector DB"] - E["Embeddings"] + + subgraph Local["Antfly (local)"] + D["Hybrid Search"] + E["BM25 + Vector + RRF"] end - + A <-->|"MCP Protocol"| B B --> C C <--> D D <--> E ``` -**Key insight:** dev-agent returns **code snippets with context** — Claude doesn't read entire files. This is why input tokens drop by 99%. +**Key insight:** dev-agent returns **code snippets with context** using hybrid search (keyword matching + semantic understanding) — Claude doesn't read entire files. This is why input tokens drop by 99%. ## Quick Start diff --git a/website/content/latest-version.ts b/website/content/latest-version.ts index 704e688..30c9db2 100644 --- a/website/content/latest-version.ts +++ b/website/content/latest-version.ts @@ -4,10 +4,10 @@ */ export const latestVersion = { - version: '0.8.5', - title: 'Enhanced Pattern Analysis & Performance', - date: 'December 14, 2025', + version: '0.9.0', + title: 'Antfly Hybrid Search', + date: 'March 29, 2026', summary: - 'Refactored dev_inspect with 5-10x faster pattern analysis, comprehensive code comparison across 5 pattern categories, and improved semantic search accuracy.', - link: '/updates#v085--enhanced-pattern-analysis--performance', + 'Replaced LanceDB with Antfly — dev_search now uses hybrid search (BM25 + vector + RRF). New `dev setup` command handles backend installation.', + link: '/updates#v090--antfly-hybrid-search', } as const; diff --git a/website/content/updates/index.mdx b/website/content/updates/index.mdx index 40b126b..0cc2cc2 100644 --- a/website/content/updates/index.mdx +++ b/website/content/updates/index.mdx @@ -9,6 +9,64 @@ What's new in dev-agent. We ship improvements regularly to help AI assistants un --- +## v0.9.0 — Antfly Hybrid Search + +*March 29, 2026* + +**Replaced LanceDB + @xenova/transformers with [Antfly](https://antfly.io) — a major upgrade to search quality and architecture.** + +### What's Changed + +**🔍 Hybrid Search (BM25 + Vector + RRF)** + +`dev_search` now combines keyword matching and semantic understanding in every query. Searching for "validateUser" finds the exact function (BM25) AND semantically related authentication code (vector), fused into one ranked result set via Reciprocal Rank Fusion. + +**⚡ New: `dev setup` Command** + +One command handles the entire search backend: + +```bash +dev setup +``` + +- Docker-first: pulls the Antfly image, starts a container +- Native fallback: detects or installs the Antfly binary, pulls the embedding model +- Auto-start: all commands (`dev index`, `dev mcp start`) auto-start Antfly if needed + +**🧠 Auto-Embedding via Termite** + +No more managing an embedding pipeline. Antfly generates embeddings locally via Termite (ONNX-optimized models). Insert documents, search immediately — Antfly handles the rest. + +**🔑 Direct Key Lookup** + +Document retrieval by ID is now instant via `tables.lookup()`, replacing the previous O(n) zero-vector scan workaround. + +### Architecture + +``` +Before: Scanner → @xenova/transformers (embed) → LanceDB (store) → vector search +After: Scanner → Antfly (embed + store + hybrid search) +``` + +- Removed ~1,400 lines of vector plumbing code +- `@lancedb/lancedb` and `@xenova/transformers` fully removed +- `@antfly/sdk` added as the sole search dependency +- VectorStorage facade preserved — all consumers unchanged + +### Breaking Changes + +- **Requires Antfly server.** Run `dev setup` (one-time) to install and start. +- **Existing LanceDB indexes are not migrated.** Run `dev index . --force` to rebuild with Antfly. +- **Default port changed to 18080** to avoid conflicts with common dev servers on 8080. + +### Testing + +- 1,935 tests passing, 0 failures +- 20 new AntflyVectorStore integration tests (insert, upsert, hybrid search, delete, count, model mismatch detection) +- VectorStorage mock for tests that don't need a live server + +--- + ## v0.8.5 — Enhanced Pattern Analysis & Performance *December 14, 2025*