Merge pull request #366 from arabold/vector-dimension

feat(store): add embedding model change safety

Author: arabold

db/migrations/013-create-metadata-table.sql

@@ -0,0 +1,7 @@
+-- Migration: Create metadata table for tracking global configuration state.
+-- Used to persist the active embedding model and vector dimension so the system
+-- can detect configuration changes between startups and prevent silent data corruption.
+CREATE TABLE IF NOT EXISTS metadata (
+  key TEXT PRIMARY KEY,
+  value TEXT NOT NULL
+);

docs/concepts/data-storage.md

@@ -2,11 +2,11 @@
 
 ## Overview
 
-The storage system uses SQLite with a normalized, four-table schema design for efficient document storage, retrieval, and version management. The schema supports page-level metadata tracking, ETag-based change detection, and hierarchical document chunking.
+The storage system uses SQLite with a normalized schema design for efficient document storage, retrieval, and version management. The schema supports page-level metadata tracking, ETag-based change detection, hierarchical document chunking, and embedding model identity tracking.
 
 ## Database Schema
 
-The database consists of four core tables with normalized relationships:
+The database consists of four core tables with normalized relationships, plus a metadata table for system-level configuration tracking:
 
 ```mermaid
 erDiagram
@@ -58,6 +58,11 @@ erDiagram
         blob embedding
         datetime created_at
     }
+
+    metadata {
+        text key PK
+        text value
+    }
 ```
 
 ### Libraries Table
@@ -137,6 +142,18 @@ Document chunks with embeddings and hierarchical metadata.
 
 **Code Reference:** `src/store/types.ts` lines 39-48 (DbChunk interface)
 
+### Metadata Table
+
+Key-value store for system-level configuration tracking, independent of library/version data.
+
+**Schema:**
+- `key` (TEXT PRIMARY KEY): Configuration key name
+- `value` (TEXT NOT NULL): Configuration value
+
+**Purpose:** Tracks the active embedding model identity (`embedding_model` and `embedding_dimension` keys) to detect incompatible configuration changes between server restarts. When a model or dimension change is detected, the server prompts the user to confirm vector invalidation before proceeding.
+
+**Code Reference:** `db/migrations/013-create-metadata-table.sql`, `src/store/DocumentStore.ts` - `getEmbeddingMetadata()`, `setEmbeddingMetadata()`, `checkEmbeddingModelChange()`
+
 ## Schema Evolution
 
 ### Migration System
@@ -154,6 +171,9 @@ Sequential SQL migrations in `db/migrations/`:
 9. `008-case-insensitive-names.sql` - Case-insensitive library name handling
 10. `009-add-pages-table.sql` - Page-level metadata normalization
 11. `010-add-depth-to-pages.sql` - Crawl depth tracking for refresh operations
+12. `011-add-vector-triggers.sql` - FTS and vector table trigger maintenance
+13. `012-add-source-content-type.sql` - Source content type tracking on pages
+14. `013-create-metadata-table.sql` - Key-value metadata table for embedding model tracking
 
 **Code Reference:** All migration files in `db/migrations/` directory
 

docs/guides/embedding-models.md

@@ -113,3 +113,36 @@ AZURE_OPENAI_API_VERSION="2024-02-01" \
 DOCS_MCP_EMBEDDING_MODEL="microsoft:text-embedding-ada-002" \
 npx @arabold/docs-mcp-server@latest
 ```
+
+## Changing the Embedding Model
+
+When you change the embedding model or vector dimension after initial setup, existing embedding vectors become semantically incompatible with the new configuration. The server detects this automatically by tracking the active model identity in a metadata table.
+
+### What Happens on Model Change
+
+**Interactive mode (TTY connected):** The server displays a warning and prompts for confirmation before proceeding. Rejecting the prompt aborts startup with no changes made.
+
+```
+⚠️  Embedding model change detected:
+   Previous: openai:text-embedding-3-small (1536 dimensions)
+   Current:  openai:text-embedding-ada-002 (1536 dimensions)
+
+   All existing embedding vectors will be invalidated.
+   Libraries must be re-scraped to restore vector search.
+   Full-text search will continue working for all existing documents.
+
+   Proceed with model change? (y/N)
+```
+
+**Non-interactive mode (MCP/stdio, CI/CD):** The server fails startup entirely with a descriptive error message. To resolve the change, start the server interactively once to confirm the migration.
+
+### After Confirming a Model Change
+
+- All stored embedding vectors are set to NULL
+- The vector search index (`documents_vec`) is recreated empty with the new dimension
+- Full-text search continues working for all existing documents
+- Libraries must be re-scraped to regenerate embeddings with the new model
+
+### Vector Dimension Override
+
+The vector dimension defaults to the model's native dimension (e.g., 1536 for `text-embedding-3-small`). You can override it with `embeddings.vectorDimension` in the config file or `DOCS_MCP_EMBEDDINGS_VECTOR_DIMENSION` as an environment variable. The value must be a positive integer (minimum 1).

docs/setup/configuration.md

@@ -223,7 +223,7 @@ Settings for the vector embedding generation.
 | `batchChars` | `50000` | Maximum total characters per embedding batch. |
 | `requestTimeoutMs` | `30000` | Timeout for each embedding API request (ms). |
 | `initTimeoutMs` | `30000` | Timeout for the initial test embedding during model initialization (ms). |
-| `vectorDimension` | `1536` | Dimension of the vector space (must match model). |
+| `vectorDimension` | `1536` | Dimension of the vector space. Must be a positive integer (minimum 1). Override with `DOCS_MCP_EMBEDDINGS_VECTOR_DIMENSION`. Changing this value triggers a model change confirmation on next startup. |
 
 ### Search (`search`)
 

openspec/changes/add-embedding-model-change-safety/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-03-17

openspec/changes/add-embedding-model-change-safety/design.md

@@ -0,0 +1,128 @@
+## Context
+
+The system uses embedding vectors for semantic search via `sqlite-vec`. Vectors are produced by a configured embedding model (e.g., `openai:text-embedding-3-small`) and stored in `documents.embedding` (JSON blob) alongside an indexed `documents_vec` virtual table. The vector table is currently created by a SQL migration with a hard-coded dimension of 1536.
+
+PR 330 introduces runtime-configurable vector dimensions so different embedding providers (e.g., 1536d for OpenAI, 3584d for Gemini) can work without migration changes. However, it has critical gaps:
+
+1. **Silent backfill of incompatible vectors**: When the dimension changes, `ensureVectorTable()` drops and recreates the vec table, then backfills old vectors that were padded to the previous dimension. These old vectors are dimensionally and semantically incompatible with the new configuration.
+2. **No model change detection**: Switching between models with the same dimension (e.g., `text-embedding-3-small` to `text-embedding-ada-002`, both 1536d) is completely undetected. Old vectors remain, producing silently degraded search results.
+3. **No user feedback**: There is no warning, confirmation prompt, or guidance when the embedding configuration changes.
+
+The existing specs document this as a "known gap" in `embedding-generation/spec.md:154` but provide no solution.
+
+**Stakeholders**: All users who configure embedding models, especially those running the server in MCP/stdio mode where there is no TTY for interactive prompts.
+
+## Goals / Non-Goals
+
+**Goals:**
+- Detect embedding model and/or vector dimension changes at startup by comparing the current configuration against persisted metadata in the database.
+- Require explicit user confirmation (interactive TTY) before proceeding with a destructive model change.
+- Fail startup entirely in non-interactive mode (no TTY) when a model change is detected, with a clear error message explaining what happened and how to resolve it.
+- On confirmed change, invalidate all existing vectors (set to NULL) and recreate the vec table empty -- no backfill of incompatible data.
+- Silently initialize metadata on first startup or upgrade from a pre-metadata database.
+- Adopt PR 330's `ensureVectorTable()` approach for runtime-configurable dimensions, but fix the broken backfill.
+
+**Non-Goals:**
+- Per-library or per-version model tracking. A global metadata record is sufficient; per-vector model tracking would bloat table sizes with no practical benefit since you cannot mix models within a search anyway.
+- Automatic re-scraping after model change. The system invalidates vectors and continues in FTS-only mode; the user must manually re-scrape to regenerate vectors.
+- Supporting multiple embedding models simultaneously. The system uses one model globally.
+- Migrating or converting vectors between models. This is mathematically impossible for different embedding spaces.
+
+## Decisions
+
+### Decision 1: Global Metadata Table for Model Tracking
+
+**Choice**: Store embedding model identity and dimension in a new `metadata` key-value table (`CREATE TABLE metadata (key TEXT PRIMARY KEY, value TEXT NOT NULL)`).
+
+**Alternatives considered**:
+- *Per-vector model column in `documents`*: Rejected -- massive storage bloat (one string per row) for information that is always the same globally. The system enforces a single model across the entire database.
+- *Per-version model tracking*: Rejected -- adds complexity without benefit. Even if tracked per-version, you cannot search across versions with mixed models. A global record is sufficient to detect changes.
+- *File-based metadata (e.g., `.embedding-meta.json`)*: Rejected -- metadata should live alongside the data it describes. A separate file can get out of sync with the database.
+
+**Keys stored**:
+- `embedding_model`: The full model spec string (e.g., `openai:text-embedding-3-small`). Compared against `config.app.embeddingModel`.
+- `embedding_dimension`: The configured vector dimension as a string (e.g., `"1536"`). Compared against `config.embeddings.vectorDimension`.
+
+### Decision 2: Fail Non-Interactive Startup on Model Change
+
+**Choice**: When a model or dimension change is detected and `process.stdout.isTTY` is falsy, throw a fatal `EmbeddingModelChangedError` that prevents startup entirely.
+
+**Rationale**: In MCP/stdio mode, there is no user present to see warnings. Silently degrading search quality or silently dropping vectors could lead to hours of confusion. A hard failure with a descriptive error message is the safest option for automated environments.
+
+**Error message format**:
+```
+Embedding model change detected:
+  Previous: openai:text-embedding-3-small (1536 dimensions)
+  Current:  gemini:embedding-001 (768 dimensions)
+
+All existing vectors are incompatible and must be invalidated.
+To confirm this change, start the server interactively (with a TTY connected)
+and follow the prompts.
+```
+
+**Alternative considered**:
+- *Escape hatch env var (`DOCS_MCP_CONFIRM_MODEL_CHANGE=true`)*: Not adopted initially to keep the surface area small. Can be added later if automated deployments need it.
+
+### Decision 3: Interactive Confirmation Flow
+
+**Choice**: When a model change is detected and `process.stdout.isTTY` is truthy, display a warning and prompt the user for explicit confirmation before proceeding.
+
+**Flow**:
+1. Display warning with previous vs. current model/dimension
+2. Explain consequences: "All existing embedding vectors will be invalidated. Libraries must be re-scraped to restore vector search."
+3. Prompt: `Proceed with model change? (y/N)`
+4. Default is `N` (abort). Only `y` or `Y` confirms.
+5. On confirm: invalidate vectors, update metadata, continue startup.
+6. On abort: throw error, exit.
+
+**Implementation**: The confirmation prompt is handled at the CLI layer (not in `DocumentStore`). `DocumentStore.initialize()` detects the mismatch and throws a structured error (`EmbeddingModelChangedError`) containing old/new model info. The CLI command handler catches this error and either prompts (TTY) or re-throws (non-TTY).
+
+### Decision 4: Vector Invalidation Strategy
+
+**Choice**: On confirmed model change, execute two operations in a transaction:
+1. `UPDATE documents SET embedding = NULL` -- clears all stored embedding blobs
+2. Drop and recreate `documents_vec` as empty (no backfill)
+
+**Rationale**: Setting embeddings to NULL rather than deleting documents preserves all indexed content. FTS search continues working immediately. The user can re-scrape at their convenience to regenerate vectors.
+
+**Alternative considered**:
+- *Delete only from `documents_vec`, keep `documents.embedding` blobs*: Rejected -- stale blobs would be backfilled into the vec table on next dimension reconciliation, recreating the incompatibility problem.
+
+### Decision 5: First-Run / Upgrade Behavior
+
+**Choice**: If the `metadata` table does not contain `embedding_model` or `embedding_dimension` keys, silently store the current configuration values without prompting.
+
+**Rationale**: Existing deployments upgrading to this version have no stored metadata. Requiring confirmation on upgrade would break all existing automated deployments. The first run establishes the baseline; subsequent changes are detected against it.
+
+### Decision 6: Initialization Order in DocumentStore
+
+**Choice**: The startup sequence in `DocumentStore.initialize()` becomes:
+1. Load `sqlite-vec` extension
+2. Apply migrations (including metadata table creation)
+3. Check embedding model/dimension metadata (detect changes, throw if mismatch)
+4. If no mismatch (or after confirmed invalidation): `ensureVectorTable()` with current dimension
+5. Prepare SQL statements
+6. Initialize embeddings client
+
+This ensures the metadata check happens before any table manipulation, preventing the table from being left in an inconsistent state if the user aborts.
+
+### Decision 7: Adopt PR 330's ensureVectorTable() With Fixes
+
+**Choice**: Keep PR 330's approach of creating `documents_vec` at runtime with configurable dimensions, but modify the backfill behavior:
+- On **first creation** (table doesn't exist): Create empty. Vectors will be populated during scraping.
+- On **dimension match** (table exists, same dimension): No-op. Existing vectors are valid.
+- On **dimension mismatch**: This case is now handled by the model change detection in step 3 above. By the time `ensureVectorTable()` runs, vectors have already been invalidated if needed.
+
+The original PR 330 backfill (`INSERT OR REPLACE ... FROM documents WHERE embedding IS NOT NULL`) is removed entirely.
+
+## Risks / Trade-offs
+
+**[Risk] Non-interactive failure blocks automated deployments that intentionally change models** → Mitigation: The error message clearly explains how to resolve it (interactive startup). A future enhancement could add a `DOCS_MCP_CONFIRM_MODEL_CHANGE=true` env var escape hatch if demand warrants it.
+
+**[Risk] First-run silent initialization stores wrong baseline if config is misconfigured** → Mitigation: The baseline is only stored after successful embedding initialization. If the model fails to initialize (bad credentials, invalid model), no metadata is stored, so the next attempt with correct config becomes the new "first run."
+
+**[Risk] Setting all embeddings to NULL is expensive on large databases** → Mitigation: This is a single UPDATE statement that SQLite handles efficiently. The alternative (deleting and re-inserting documents) would be far more expensive and would break FTS indexes.
+
+**[Risk] Users may not realize they need to re-scrape after model change** → Mitigation: The confirmation prompt and post-invalidation log message explicitly state that re-scraping is required. The system continues in FTS-only mode as a safety net.
+
+**[Trade-off] Global model tracking vs per-version tracking** → We chose global for simplicity. This means the system cannot detect if individual versions were scraped with different models (e.g., if a user changed models between scraping two libraries). This is acceptable because: (a) search is always performed within a single version, and (b) the model change detection catches the transition point.

openspec/changes/add-embedding-model-change-safety/proposal.md

@@ -0,0 +1,32 @@
+## Why
+
+The system currently has no mechanism to detect when the configured embedding model or vector dimension changes between startups. Vectors produced by different embedding models are semantically incompatible -- cosine similarity between them is meaningless -- yet the system silently continues serving degraded search results. PR 330 introduces configurable vector dimensions but compounds this problem by silently backfilling old vectors into a potentially incompatible table. This change adds safety rails to prevent silent data corruption and enforce explicit user acknowledgment before invalidating existing vectors.
+
+## What Changes
+
+- **BREAKING**: Non-interactive startup (MCP/stdio) SHALL fail with a descriptive error when an embedding model or dimension change is detected, requiring manual interactive confirmation.
+- Store the active embedding model identifier and vector dimension as global metadata in the database to enable change detection across restarts.
+- On first startup (or upgrade from a database without metadata), silently initialize the metadata record with the current configuration -- no user action required.
+- When a model or dimension change is detected during interactive startup, display a warning explaining that all existing vectors will be invalidated, and require explicit user confirmation before proceeding.
+- On confirmed model/dimension change, set all `documents.embedding` values to `NULL` and recreate the `documents_vec` table empty (no backfill of incompatible vectors).
+- Adopt PR 330's runtime-configurable `documents_vec` creation via `ensureVectorTable()`, but remove the broken backfill that inserts old-dimension vectors into a new-dimension table.
+- Add Zod validation ensuring `vectorDimension >= 1`.
+
+## Capabilities
+
+### New Capabilities
+- `embedding-model-change-safety`: Defines startup-time detection of embedding model/dimension changes, interactive confirmation flow, non-interactive failure behavior, vector invalidation on confirmed change, and metadata persistence for tracking the active embedding configuration.
+
+### Modified Capabilities
+- `embedding-generation`: Close the documented "known gap" about model change tracking. Add requirements for vector invalidation when the model changes, and update dimension normalization to cover runtime-configurable vector table creation.
+- `embedding-resolution`: Add requirements for persisting resolved model identity to the database and detecting mismatches on subsequent startups.
+- `configuration`: Add `embeddings.vectorDimension` as a documented configurable parameter with Zod validation (`>= 1`).
+
+## Impact
+
+- **`src/store/DocumentStore.ts`**: Major changes to `initialize()`, new `ensureVectorTable()` method (from PR 330, modified), new metadata read/write methods, new vector invalidation method.
+- **`src/store/embeddings/`**: No structural changes, but `EmbeddingConfig` output is now persisted to the database.
+- **`src/utils/config.ts`**: Minor -- add `.min(1)` validation on `vectorDimension` (from PR 330).
+- **`db/migrations/`**: New migration to create `metadata` table. Migration 012 from PR 330 (drop `documents_vec`) is adopted.
+- **`src/cli/`**: Startup commands need to handle interactive confirmation prompts and non-interactive failure for model changes.
+- **User-facing docs**: `docs/guides/embedding-models.md` and `docs/setup/configuration.md` need updates explaining model change behavior and `vectorDimension` configuration.