Replace submit_eval_jobs.py with thin wrapper around submit_eval_jobs.sh

[finbarrtimbers]

Summary

• Replaces scripts/submit_eval_jobs.py's Beaker spec construction with a thin argparse translator that shells out to scripts/submit_eval_jobs.sh.
• New scripts/submit_eval_jobs.sh clones olmo-eval-internal and invokes olmo-eval beaker launch directly.

Split out from the icepop branch.

Test plan

• Dry-run uv run python scripts/submit_eval_jobs.py --model_name foo --location /weka/... --dry_run
• Submit a real eval job

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request refactors the evaluation job submission process, transforming scripts/submit_eval_jobs.py into a wrapper that delegates to a new shell script, scripts/submit_eval_jobs.sh, which uses olmo-eval beaker launch. The review feedback identifies a concurrency risk in the shell script's cloning logic, suggesting the use of mktemp and an exit trap for isolation. Additionally, it is recommended to use shlex.join in the Python script to ensure that logged commands are properly shell-escaped.

[gemini-code-assist]

Cloning into a fixed path within the repository ($REPO_ROOT/olmo-eval-internal) is problematic for concurrency and repository cleanliness. If multiple jobs are submitted simultaneously, they will conflict on the same directory. Using a temporary directory with an exit trap ensures isolation and automatic cleanup. Note that exec is removed to allow the trap to fire upon completion.

Suggested change

	REPO_ROOT=$(git -C "$(dirname "$0")" rev-parse --show-toplevel)
	CLONE="$REPO_ROOT/olmo-eval-internal"
	REF="${OLMO_EVAL_REF:-main}"
	if [ ! -d "$CLONE/.git" ]; then
	git clone --depth=1 https://github.com/allenai/olmo-eval-internal.git "$CLONE"
	fi
	git -C "$CLONE" fetch --depth=1 origin "$REF"
	git -C "$CLONE" checkout FETCH_HEAD
	cd "$CLONE"
	exec uv run olmo-eval beaker launch -y \
	--harness default \
	-o provider.kind=vllm_server \
	-o provider.trust_remote_code=true \
	-e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
	"$@"
	CLONE=$(mktemp -d)
	trap 'rm -rf "$CLONE"' EXIT
	REF="${OLMO_EVAL_REF:-main}"
	git clone --depth=1 https://github.com/allenai/olmo-eval-internal.git "$CLONE"
	git -C "$CLONE" fetch --depth=1 origin "$REF"
	git -C "$CLONE" checkout FETCH_HEAD
	cd "$CLONE"
	uv run olmo-eval beaker launch -y \
	--harness default \
	-o provider.kind=vllm_server \
	-o provider.trust_remote_code=true \
	-e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
	"$@"

[gemini-code-assist]

Re-add the shlex import to ensure that the command logging is properly escaped and readable.

Suggested change

	import os
	import subprocess
	import os
	import shlex
	import subprocess

[gemini-code-assist]

Using shlex.join(cmd) is preferred over " ".join(cmd) for logging. It produces a shell-escaped string that accurately represents the command, which is especially important if any arguments (like experiment names or paths) contain spaces.

Suggested change

	print("Running:", " ".join(cmd))
	print("Running:", shlex.join(cmd))

[chatgpt-codex-connector]

💡 Codex Review

open-instruct/scripts/submit_eval_jobs.py

Line 46 in cff3d8c

out += ["-m", args.location]

Restore Beaker dataset locations in the wrapper

When --location is a bare Beaker dataset id or beaker://..., this now forwards that string directly as -m instead of resolving it to /model and mounting the dataset as the previous implementation did. The parent version explicitly accepted those locations, and the repository docs still show Beaker dataset ids for internal evals, so those legacy invocations will submit jobs whose model path is just the dataset id rather than an accessible mounted checkpoint.

---

open-instruct/scripts/submit_eval_jobs.py

Line 79 in cff3d8c

sys.exit(subprocess.run(cmd, env=env).returncode)

Avoid running the launcher during dry runs

With --dry_run, the wrapper still invokes submit_eval_jobs.sh; that script clones/fetches olmo-eval-internal before forwarding -d to the CLI. This changes dry-run from a local spec/command preview into an operation that needs GitHub access and writes a checkout into the repo, so the documented/test-plan dry run can fail on a fresh or offline clone before it ever reaches the dry-run path.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".