single command experience

[benjibc]

Note

Simplifies RFT creation by auto-uploading evaluators, inferring/creating datasets from JSONL loaders, requiring explicit base model, and improving host mapping; adds a JSONL data loader and updates tests.

• CLI (eval_protocol/cli_commands/create_rft.py):
  • Auto-resolves evaluator to accounts/{account}/evaluators/{id} and programmatically uploads/ensures it.
  • Infers dataset from a single test’s JSONL data loader; creates and uploads dataset with a trimmed time-stamped id; resolves JSONL paths safely.
  • Requires --base-model; builds training/inference configs from args; prints concise job prep summary.
• Data Loading:
  • Add EvaluationRowJsonlDataLoader to load EvaluationRow from JSONL; export via __init__.py.
• Infrastructure:
  • Improve _map_api_host_to_app_host to handle dev/staging and generic api→app mapping robustly.
• Tests:
  • Update GSM8K example to use EvaluationRowJsonlDataLoader and make answer extraction more robust.

^{Written by Cursor Bugbot for commit 32c7916. This will update automatically on new commits. Configure here.}