Skip to content

refactor cli and validate before create rft#348

Merged
xzrderek merged 7 commits intomainfrom
derekx/refactor-cli
Nov 26, 2025
Merged

refactor cli and validate before create rft#348
xzrderek merged 7 commits intomainfrom
derekx/refactor-cli

Conversation

@xzrderek
Copy link
Contributor

@xzrderek xzrderek commented Nov 25, 2025

Note

Add dataset/evaluator pre-validation and Docker controls to RFT creation, extract shared CLI utils, and update local-test/upload plus tests accordingly.

  • CLI (create rft):
    • Add flags: --skip-validation, --ignore-docker, --docker-build-extra, --docker-run-extra.
    • Refactor into helpers to resolve evaluator/dataset, validate locally, upload dataset, ensure evaluator, and create job.
    • Enforce dataset source rules: reject data_loaders JSONL; prefer input_dataset; fallback to detected dataset builders.
    • Validate dataset rows against EvaluationRow schema (sampled) before upload; optionally run selected test via pytest/Docker for evaluator validation.
  • CLI (local-test):
    • Extract shared run path via run_evaluator_test; support Docker extras; improved error messages for multiple Dockerfiles.
    • Use shared utils for test discovery and entrypoint building.
  • CLI (upload):
    • Use shared utils for discovery, ID normalization, entrypoint, account resolution, and dashboard URL building.
  • Shared utilities:
    • New cli_commands/utils.py with test discovery/selection, evaluator ID normalization, account resolution, dashboard URL/link builders, dataset ID builder, entrypoint builder, and helpers.
  • Tests:
    • Add/expand tests for RFT validation paths (dataset schema, evaluator pytest/Docker), dataset inference precedence, Docker flag propagation, and local-test behavior; update fixtures and helpers.

Written by Cursor Bugbot for commit 4a25e45. This will update automatically on new commits. Configure here.

Copy link
Collaborator

@dphuang2 dphuang2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

big PR, i'm assuming its just reorganization

@xzrderek xzrderek merged commit 7aa064f into main Nov 26, 2025
9 checks passed
@xzrderek xzrderek deleted the derekx/refactor-cli branch November 26, 2025 00:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants