Skip to content

feat(bench): implement FRAMES dataset loader #2837

@bug-ops

Description

@bug-ops

Description

Implement the FRAMES (Google 2024) dataset loader and accuracy evaluator.

Part of epic #2827. See spec: .local/specs/zeph-bench/spec.md section 6.

Scope

  • FramesLoader parsing FRAMES JSON schema into Vec<Scenario>
  • Accuracy evaluator (exact match against gold answer)
  • Download and cache to ~/.local/share/zeph/bench/frames/
  • Unit tests with a synthetic fixture

Acceptance Criteria

  • Loader parses official FRAMES schema
  • Accuracy evaluator correct on fixture
  • zeph bench download --dataset frames works

Metadata

Metadata

Assignees

Labels

P2High value, medium complexityenhancementNew feature or requestmemoryzeph-memory crate (SQLite)

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions