Skip to content

[Chore] agent-url dispatch precedence + housing-exchange eval datasets #1

Description

@Ref34t

Driver

While running the Housing Exchange analytics-agent POC eval, --agent-url was silently shadowed by the default --provider openai, so every run hit OpenAI instead of the target HTTP agent (surfaced as a phantom "credit balance too low" / 404 model_not_found). The HA eval corpus also needs a committed home.

Scope

  • harness: make explicit --agent-url take precedence over the default provider
  • add the 357-question housing-exchange corpus (analytics/maltese/refusal/security/ambiguous) under datasets/
  • add the Haiku POC baseline runs under runs/

Acceptance Criteria

  • --agent-url <url> routes to the HTTP agent regardless of --provider default
  • datasets/ holds the 5 labelled sets; python3 -m evalspec.harness --agent-url ... --all runs end to end
  • Haiku baseline reports committed

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions