feat: add MemoryManager backend to LongMemEval runner#14
Open
rohram04 wants to merge 6 commits into
Open
Conversation
New --backend memorymanager path that benchmarks the custom MemoryManager system instead of mem0, holding chunks/answerer/judge identical (memory system is the only variable). Integrates directly at the runner's call sites — no Mem0Client replacement, no search() seam. - benchmarks/common/mm_bridge.py: per-question Agent (isolated in-memory LongTermStore), OpenRouter gpt-4o/4o-mini backend + shared embedder singletons; mm_ingest / mm_surface_and_format (PREP) / mm_persist (PERSIST). - run.py: --backend memorymanager + --mm-* flags; ingest_question_mm; process_question_answerer surfaces via the agent's PREP and persists via PERSIST, answer still generated by the SAME harness answerer; single managed-window cutoff; mem0 paths untouched. Validated end-to-end through the bridge with real gpt-4o (construction, ingest, PREP, PERSIST). Embedder parity (text-embedding-3-small) is a follow-up needing a small MemoryManager-side change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-small Embedder parity with mem0 OSS (text-embedding-3-small), routed through OpenRouter so the whole stack uses one OpenRouter key (no OpenAI key): route the bridge's embedder through MM's make_embedder (was hardcoding SentenceTransformer), default to "openrouter:openai/text-embedding-3-small" (1536-d, matches LongTermStore's default embedding_dim), and add --mm-embedding-model (accepts openrouter:/openai: specs or a local sentence-transformers name). Verified: OpenRouter's embeddings endpoint returns a 1536-d vector for openai/text-embedding-3-small. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ycles. Use PREP+PERSIST ingest and query in llm mode and dual receive() in algorithmic mode so LongMemEval exercises the same memory paths as Agent, with the harness answerer substituting for REPLY. Co-authored-by: Cursor <cursoragent@cursor.com>
How-to for the MM-vs-mem0 LongMemEval head-to-head: the mm_bridge integration, all-OpenRouter routing (3 points + env), reusing MM's venv, mem0 OpenRouter config, run commands, and fairness caveats. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Thread each haystack pair's real-world session date through ingest: run.py builds a per-pair date list (parse_longmemeval_date -> UTC datetime) parallel to the pairs and passes it to mm_ingest, which stamps each pair's blocks via the scoped ContextManager.using_source_date(). _format_blocks now emits the block's source_date as an ISO created_at, so the answerer prompt date-orders/-groups MM memories the same way it does mem0's per-memory timestamps — closing the temporal-reasoning gap. dates is optional (un-dated ingest still works). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-turn. Thread clock_seconds_per_turn through mm_bridge into MemoryConfig so batch LongMemEval runs can differentiate recency during fast ingest. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
memorymanagerbackend choice (--backend memorymanager) to the LongMemEval runner, enabling apples-to-apples comparison against Mem0's OSS and cloud backends with the same answerer prompt, judge, and metrics pipeline.benchmarks/common/mm_bridge.py— a bridge module that wires up a fresh, isolatedMemoryManager.Agentper question with an in-memoryLongTermStore, then exposes three phases: ingest (turn-by-turn memory lifecycle), PREP (surface relevant blocks before the answerer), and PERSIST (store the exchange after).benchmarks/longmemeval/run.pyto route MM questions throughingest_question_mm→process_question_answerer(agent=...), collapsing the cutoff loop to a single window since MM manages its own context budget.How it works
The MemoryManager system lives in a sibling repo (
~/MemoryManager, orMEMORYMANAGER_PATH). Each benchmark question gets its ownAgentinstance with an isolated SQLite in-memory store — no state leaks between questions.New CLI flags (ignored for Mem0 backends):
--mm-max-tokens— context-window token budget (default8000)--mm-model— main answerer model (defaultopenai/gpt-4o)--mm-util-model— compress/merge/novelty util model (defaultopenai/gpt-4o-mini)Test plan
--max-questions 1 --backend memorymanagerMade with Cursor