Skip to content

feat(bench): implement per-scenario memory isolation (Qdrant + SQLite) #2830

@bug-ops

Description

@bug-ops

Description

Before each benchmark scenario begins, reset the bench-session Qdrant collection and SQLite database so earlier scenarios cannot contaminate later ones.

Part of epic #2827. See spec: .local/specs/zeph-bench/spec.md NFR-001, FR-002.

Scope

  • BenchIsolation::reset async function that:
    • Deletes and recreates the Qdrant collection named bench_<dataset>_<run_id>
    • Clears or recreates the SQLite conversation-history tables for the bench session DB path (bench-<run_id>.db)
  • Bench session config overrides: memory.qdrant.collection and DB path are set to bench-namespaced values before agent construction
  • Verify production collection names (zeph_memory, zeph_skills, etc.) are never touched
  • Isolation overhead must be < 2 seconds (NFR-007)

Acceptance Criteria

  • After scenario N completes, no memories from N are visible to scenario N+1
  • Production Qdrant collections and SQLite DB are untouched after a bench run
  • Isolation reset completes in < 2 seconds on a local Qdrant instance
  • Integration test (ignored, requires Qdrant) verifies isolation

Metadata

Metadata

Assignees

Labels

P2High value, medium complexityenhancementNew feature or requestmemoryzeph-memory crate (SQLite)qdrantQdrant vector database integration

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions