Skip to content

tools: add setup preflight checker#16

Open
Derekko-web wants to merge 1 commit into
InternLM:mainfrom
Derekko-web:add-setup-check
Open

tools: add setup preflight checker#16
Derekko-web wants to merge 1 commit into
InternLM:mainfrom
Derekko-web:add-setup-check

Conversation

@Derekko-web
Copy link
Copy Markdown

Summary

  • add eval/check_setup.py to catch common setup blockers before launching benchmark containers
  • check repo layout, .env/API keys, selected harness Docker image, workspace data, task category paths, and prepare-script commands
  • document the preflight command in the README

Validation

  • OPENROUTER_API_KEY=dummy BRAVE_API_KEY=dummy python3 eval/check_setup.py --agent-backend openclaw --category all --model openrouter/test/model --skip-docker --skip-data
  • OPENROUTER_API_KEY=dummy python3 eval/check_setup.py --agent-backend codex --category 01_Productivity_Flow --model openrouter/test/model --skip-docker --skip-data
  • python3 eval/check_setup.py --help
  • python3 -m py_compile eval/check_setup.py
  • python3 -m compileall -q eval src
  • git diff --check

@LennoxDai
Copy link
Copy Markdown
Collaborator

Thanks! Two small suggestions:

  1. hf, yt-dlp, ffmpeg, and modelscope are only needed by script/prepare.sh, not for running the benchmark itself.

  2. BRAVE_API_KEY is currently only checked for all / 04_Search_Retrieval, but other categories may also rely on web search. Suggest checking it unconditionally.

Happy to send a follow-up PR if helpful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants