Add Audrey as an AMB memory provider / leaderboard candidate

[Evilander]

Audrey benchmark submission request

Project: https://github.com/Evilander/Audrey
Package: https://www.npmjs.com/package/audrey

Audrey is a local-first memory runtime and MCP server for AI agents with recall, memory reflexes, preflight checks, tool-trace learning, validation, contradiction handling, and SQLite/sqlite-vec storage.

We have a local benchmark report already generated with JSON/HTML/SVG artifacts, but we do not want to submit those as AMB leaderboard numbers without running the official AMB harness. We would like to add an Audrey memory provider adapter and run Audrey through AMB datasets using the official methodology.

Current local evidence available in the Audrey repo:

• benchmarks/output/summary.json
• benchmarks/output/report.html
• benchmarks/output/local-overall.svg
• benchmarks/output/retrieval-overall.svg
• benchmarks/output/operations-overall.svg
• benchmarks/output/published-locomo.svg
• benchmarks/snapshots/perf-0.22.2.json

Fresh local benchmark:

• Generated at: 2026-05-01T03:20:07.968Z
• Command: node benchmarks/run.js --provider mock --dimensions 64
• Audrey local regression suite: 100.0% score / 100.0% pass rate

Fresh perf snapshot:

• Generated at: 2026-05-01T03:39:02.431Z
• 5,000-memory hybrid recall p95: 3.6 ms with mock 64-dim embeddings

Question for maintainers:

What is the preferred submission route for a new memory provider: PR with provider adapter plus outputs, issue first, or maintainer upload after a reproducible run?

[Evilander]

Update with current public artifact links and a stricter claim boundary.

I corrected the earlier local-only evidence so this issue does not imply an official AMB score. Audrey has now been run through MemoryGym Core against the current local checkout via the Audrey HTTP sidecar. This is not an AMB leaderboard result, but it is current public evidence while we work on the official AMB provider adapter.

Current MemoryGym Core run:

• Generated: 2026-05-01T04:19:26.858Z
• Audrey transport: HTTP sidecar on loopback
• Providers: mock embeddings, mock LLM, isolated data directory
• typed-semantic baseline: 72.7% score, 100.0% hit rate, 0.3 ms p95 recall
• hybrid baseline: 72.7% score, 100.0% hit rate, 0.3 ms p95 recall
• audrey-http: 34.3% score, 42.9% hit rate, 3.7 ms p95 recall

Public artifacts:

• Live report Space: https://huggingface.co/spaces/Evilander/audrey-memory-benchmark-report
• Raw artifact dataset: https://huggingface.co/datasets/Evilander/audrey-memory-benchmark-artifacts
• Compact summary: https://huggingface.co/datasets/Evilander/audrey-memory-benchmark-artifacts/blob/main/memorygym-current-summary.json
• Raw run JSON: https://huggingface.co/datasets/Evilander/audrey-memory-benchmark-artifacts/blob/main/memorygym-current-run.json

What the current run found: Audrey is fast in this small run, but ranking/isolation underperformed. The main gaps are scenario/tag contamination, current-belief precedence, and context-aware routing. That is exactly why we want the official AMB path: we need a reproducible third-party harness result, not a polished self-claim.

Still looking for maintainer guidance on the preferred submission route: provider adapter PR plus outputs, issue-first review, or maintainer-run upload.

[Evilander]

Second update: I fixed the benchmark contamination issue the first MemoryGym run exposed and reran the same suite.

Audrey change tested: multiple tag filters now require all requested tags to match. The earlier run showed MemoryGym scenario leakage because Audrey treated tags as OR; MemoryGym passes memorygym, run id, and scenario id together.

MemoryGym Core before/after:

• Before fix: audrey-http 34.3% score, 42.9% hit rate, 3.7 ms p95 recall
• After fix: audrey-http 57.5% score, 85.7% hit rate, 3.8 ms p95 recall
• Delta: +23.2 score points, +42.9 hit-rate points, +0.1 ms p95 recall

Current public artifacts:

• Live before/after report: https://huggingface.co/spaces/Evilander/audrey-memory-benchmark-report
• Dataset: https://huggingface.co/datasets/Evilander/audrey-memory-benchmark-artifacts
• Current summary: https://huggingface.co/datasets/Evilander/audrey-memory-benchmark-artifacts/blob/main/memorygym-current-summary.json
• Before JSON: https://huggingface.co/datasets/Evilander/audrey-memory-benchmark-artifacts/blob/main/memorygym-before-tag-filter.json
• After JSON: https://huggingface.co/datasets/Evilander/audrey-memory-benchmark-artifacts/blob/main/memorygym-after-tag-filter.json

Still not claiming an AMB leaderboard score. The rerun makes the Audrey provider path stronger, but official AMB still needs a proper adapter and official harness output.