feat(worlds): add worlds provider with neurosymbolic ingestion for MemoryBench#4
Open
EthanThatOneKid wants to merge 13 commits into
Open
feat(worlds): add worlds provider with neurosymbolic ingestion for MemoryBench#4EthanThatOneKid wants to merge 13 commits into
EthanThatOneKid wants to merge 13 commits into
Conversation
- Vendor memorybench from supermemoryai/memorybench (MIT) as baseline - Add @worlds/client (jsr:@jsr/worlds__client@^0.0.14) dependency, same version as worlds-client-evals - Add @comunica/query-sparql-rdfjs-lite as required by @worlds/client adapters - Implement WorldsProvider: in-memory LibSQL, RDF/Turtle ingest, rebuildSearchIndex, client.search() - Add worlds to ProviderName union and register in providers map - Add worlds case to getProviderConfig() (uses OPENAI_API_KEY for judge) - Add .env.example with required API keys - Add src/providers/worlds/README.md documenting phase mapping and smoke run - Fix formatting across upstream files via prettier
feat(worlds): neurosymbolic ingestion harness
… prompt - Accept questionDate parameter for temporal reasoning on LoCoMo questions - Strip RDF metadata (subject, predicate, graph URIs) in prompt builder, showing only text + relevance score to reduce token waste - Keep full search results in search() for show-failures debuggability - Add step-by-step reasoning format matching supermemory prompt pattern
FTS5 uses AND between tokens after stopword removal, so long questions like "When did Caroline go to the LGBTQ support group?" match nothing. When the full query returns empty, fall back to per-term OR-style search with best-score dedup. Also log rebuildSearchIndex quad/chunk counts for indexing diagnostics. LoCoMo -l 5 now completes: 20% accuracy, Hit@4=60%, MemScore 20%/35ms/2306tok.
Wire GeminiEmbeddingService into @worlds/client search index using @ai-sdk/google embedMany(). 768-dim vectors fused with FTS5 keywords via Reciprocal Rank Fusion. Graceful degradation to keyword-only mode when GOOGLE_GENERATIVE_AI_API_KEY is absent.
…ch chunks - config.ts: worlds provider now uses googleApiKey (was openaiApiKey) - gemini-embedding-service: switch from deprecated text-embedding-004 to gemini-embedding-2 with 768d output via outputDimensionality - gemini-embedding-service: chunk embed() calls at 100 items to stay within BatchEmbedContents API limit - index.ts: use searchIndexOnImport:false to defer all vectorization to rebuildSearchIndex() in the indexing phase - README: document two-step ingest-once/iterate workflow with -f search Smoke test: 0% -> 40% accuracy, Hit@10 60% -> 80%, MRR 0.18 -> 0.36
…eaker attribution Enrich search results with session metadata via a batched SPARQL query before passing context to the answer LLM. Stores speaker names (schema:creator) and session participants (worlds:speakerA/B) during ingestion, resolves them at search time, and surfaces them in the prompt with improved temporal reasoning and speaker attribution instructions. smoke-004: 80% accuracy (4/5), up from 40% in smoke-003.
…bility Extract structured claims at ingest (retry, disk cache) and query them at search time via entity-aware SPARQL. Interleave facts with hybrid search hits, rate-limit embeddings, and use 3-pass majority voting plus equivalence rubric for more consistent judge scores.
Add a 5-question agent runner with AI SDK tool calling, billing check, and JSONL trace logging for prompt/tool iteration via replay. Expose a WorldsProvider client getter for SPARQL tooling.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add
@worlds/clientas a MemoryBench provider (-p worlds) with a neurosymbolic typed-graph ingestion pipeline.Motivation
Enables
@worlds/clientto be evaluated on published memory/RAG benchmarks (LoCoMo, LongMemEval) using the MemoryBench framework, alongside Supermemory, Mem0, Zep, etc. — without duplicating harness code or blurring CI semantics inworlds-client-evals.What changed
Provider bootstrap
supermemoryai/memorybench(MIT) as the baseline harness@worlds/client(jsr:@jsr/worlds__client@^0.0.14), pinned to same version family asworlds-client-evals@comunica/query-sparql-rdfjs-lite(required by@worlds/clientadapters)WorldsProvider(src/providers/worlds/):client.import()client.rebuildSearchIndex()after ingest for FTS/vector discoverabilityclient.search({ query })for retrievalworldsprovider in CLI (-p worlds)worldstoProviderNameunion andgetProviderConfig()(usesOPENAI_API_KEYfor judge).env.exampleandsrc/providers/worlds/README.mddocumenting the phase mapping and smoke runNeurosymbolic ingestion harness (Phase 1)
worlds:custom namespace (https://worlds.wazoo.dev/). Exports a reusableTURTLE_PREFIXESblock.validateGraph()checker that runs during ingestion.schema:Conversation+prov:Activity, messages asschema:Message+prov:Entitywithschema:text,schema:position,schema:author,schema:hasPart, andprov:wasGeneratedByprovenance links. Replaces inline URI strings with ontology constants.Design decisions
prov:wasDerivedFrom).LoCoMo smoke test
bun install cp .env.example .env.local # add API keys bun run src/index.ts run -p worlds -b locomo -l 5 -j gpt-4o -m gemini-3.1-flash-liteRelationship to worlds-client-evals
sparql-handoff-valid,updates-blocked, step budgetworlds-client-evalsworlds-client-memorybenchCloses wazootech/worlds-client-evals#34