Driver
While running the Housing Exchange analytics-agent POC eval, --agent-url was silently shadowed by the default --provider openai, so every run hit OpenAI instead of the target HTTP agent (surfaced as a phantom "credit balance too low" / 404 model_not_found). The HA eval corpus also needs a committed home.
Scope
- harness: make explicit
--agent-url take precedence over the default provider
- add the 357-question housing-exchange corpus (analytics/maltese/refusal/security/ambiguous) under
datasets/
- add the Haiku POC baseline runs under
runs/
Acceptance Criteria
Driver
While running the Housing Exchange analytics-agent POC eval,
--agent-urlwas silently shadowed by the default--provider openai, so every run hit OpenAI instead of the target HTTP agent (surfaced as a phantom "credit balance too low" / 404 model_not_found). The HA eval corpus also needs a committed home.Scope
--agent-urltake precedence over the default providerdatasets/runs/Acceptance Criteria
--agent-url <url>routes to the HTTP agent regardless of--providerdefaultdatasets/holds the 5 labelled sets;python3 -m evalspec.harness --agent-url ... --allruns end to end