feat: replace vec0 with viant/sqlite-vec for vector search#49
Merged
Conversation
- Replace asg017/sqlite-vec (vec0) with viant/sqlite-vec (vec module) - Pure Go, compatible with modernc.org/sqlite — no CGO or shared libraries - Fix stale vec0 vtab cleanup: use PRAGMA writable_schema to remove entries when vec0 module is unavailable (common in Go-only builds) - Fix connection pooling: vec.Register only applies to new connections, so close and reopen DB after registration to pick up the module - Fix deadlock: eagerly create shadow table, vector_storage, and vector_storage_locks to prevent lazy-creation deadlock with database/sql - Fix shadow table sync: Add/Update/Invalidate/Delete now maintain the _vec_memories_vec shadow table in Go code instead of SQL triggers - Fix searchByEmbeddingVec: use doc_id MATCH (not embedding MATCH) per viant/sqlite-vec schema (dataset_id, doc_id, match_score HIDDEN) - Fix Update sync: content-only updates preserve existing embedding instead of deleting the shadow table entry - Fix SemanticOnly error: return clear error instead of duplicate returns - Remove DisableVec field from MemoryStore, remove dropVecTriggers, remove reVecDimensions regex - Eagerly populate shadow table from memories.embedding on startup - Store test changes: update TestMigration_Vec0ObjectsDroppedBeforeMigration to test new dropLegacyVec0Objects behavior
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the broken vec0 (asg017/sqlite-vec C extension) with viant/sqlite-vec (pure Go, compatible with modernc.org/sqlite). This fixes the long-standing issue where vector search was completely non-functional in Go builds.
Key changes
USING vec(doc_id, embedding float[N])instead ofUSING vec0(...). No CGO, no shared libraries required.memories_vecvtab registered under the vec0 module, which can't be dropped ("no such module: vec0"). UsesPRAGMA writable_schemato remove the stale entry from sqlite_master.vec.Register(db)registers the module globally but only on new connections. The PRAGMA calls inopenDBcreate warm connections that lack the module. Close and reopen DB after registration._vec_memories_vec,vector_storage, andvector_storage_locksupfront.doc_id MATCH ?(notembedding MATCH ?) per viant's vtab schema(dataset_id, doc_id, match_score HIDDEN).Updatewith content changes but no embedding change now updates the shadow table content column, preserving the existing embedding.Known issues
TestIntrospectCmd_AutoFlags(removed flags) andTestSessionHookCoordinator_OnEndingWithIntrospect(source mismatch) — both unrelated to this PR.internal/introspectbuild failure (IntrospectAutoundefined) — pre-existing, same as above.Testing
llmem searchreturns semantically relevant resultsdropLegacyVec0Objectssuccessfully removes stale vec0 vtab entries viaPRAGMA writable_schema