AgentOpt · doxav · Jun 13, 2026 · May 8, 2026
diff --git a/README.md b/README.md
@@ -2,7 +2,7 @@
 
 A benchmarking framework for evaluating LLM-as-optimizer algorithms, built on [OpenTrace](https://github.com/AgentOpt/Trace).
 
-Trace-Bench provides a **CLI**, **Gradio UI**, and **notebook workflows** to run reproducible experiments across LLM4AD, VeriBench, KernelBench, and HuggingFace QA tasks with fair comparisons between trainers, optimizers, and LLM backends.
+Trace-Bench provides a **CLI**, **Gradio UI**, and **notebook workflows** to run reproducible experiments across LLM4AD, VeriBench, KernelBench, HuggingFace QA, and Terminal-Bench via Harbor with fair comparisons between trainers, optimizers, and LLM backends.
 
 **Full documentation:** [docs/](docs/README.md)
 
@@ -28,9 +28,12 @@ pip install -e .
 # Optional extras can be combined:
 pip install -e ".[hf]"          # HuggingFace QA benchmarks
 pip install -e ".[dspy]"        # DSPy trainer adapter
+pip install -e ".[terminal-bench]"  # Terminal-Bench 2.0 via Harbor
 pip install -e ".[all-external]"  # everything
 ```
 
+Note: the `terminal-bench` extra installs Harbor, which currently requires Python >=3.12.
+
 ## Quick Start
 
 ```bash
@@ -71,6 +74,70 @@ Then run with a real-mode config:
 trace-bench run --config configs/smoke_real.yaml --runs-dir runs
 ```
 
+
+## Terminal-Bench / Harbor Support
+
+Trace-Bench can optimize Harbor's Terminus-2 prompt template against Terminal-Bench 2.0 tasks. Task IDs use the `terminal_bench:<task_slug>` namespace, for example `terminal_bench:regex-log`.
+
+### Install and configure
+
+```bash
+# Install Trace-Bench plus Harbor support.
+pip install -e ".[terminal-bench]"
+
+# For local Docker execution:
+export HARBOR_MODEL="openai/gpt-5-nano"
+export OPENAI_API_KEY="sk-..."
+
+# For hosted Daytona execution, recommended in Colab or environments without Docker:
+export DAYTONA_API_KEY="dtn_..."
+export HARBOR_MODEL="openai/gpt-5-nano"
+export OPENAI_API_KEY="sk-..."
+```
+
+Do not put API keys in config files or commit them to the repository. Trace-Bench reads `HARBOR_MODEL`, `OPENAI_API_KEY`, and `DAYTONA_API_KEY` from the process environment at runtime. If you install Harbor manually instead of using the extra, use `pip install "harbor[daytona]>=0.6.0"` for Daytona-backed runs or `pip install "harbor>=0.6.0"` for Docker-only runs.
+
+### Offline smoke run
+
+The demo config defaults to `mode: stub`, so Harbor is not invoked and no API keys are required:
+
+```bash
+trace-bench validate --config configs/terminal_bench_demo.yaml --runs-dir runs --strict
+trace-bench run --config configs/terminal_bench_demo.yaml --runs-dir runs
+```
+
+### Real Terminal-Bench run
+
+Switch the config to `mode: real`, or create a small real-mode config with a Terminal-Bench task. In real mode, each guide evaluation shells out to `harbor run` and stores Harbor command/stdout/stderr/results/trajectory artifacts under the Trace-Bench job artifacts directory (`runs/<run_id>/jobs/<job_id>/artifacts/terminal_bench/`).
+
+```yaml
+mode: real
+tasks:
+  - id: terminal_bench:regex-log
+    eval_kwargs:
+      harbor_dataset: terminal-bench@2.0
+      harbor_env: daytona        # or docker
+      harbor_model: null         # read HARBOR_MODEL from the environment
+      harbor_cli_timeout_seconds: 1800
+```
+
+Then run:
+
+```bash
+trace-bench run --config path/to/terminal_bench_real.yaml --runs-dir runs
+```
+
+### Limits and operational notes
+
+- Terminal-Bench execution is external: real mode requires the Harbor CLI, a supported environment backend (`docker`, `daytona`, etc.), and model credentials accepted by Harbor/LiteLLM. The model must be compatible with Harbor/LiteLLM's chat-completion path; if a model reports a Responses API `messages`/`input` mismatch, use a compatible model alias or upgrade Harbor/LiteLLM.
+- The built-in task discovery intentionally lists only a curated smoke/demo subset. You can still run any Harbor Terminal-Bench 2.0 slug directly as `terminal_bench:<slug>`.
+- Real runs can be slow and cost-bearing because optimizers may evaluate the trainable prompt multiple times; keep `max_workers`, `ps_steps`, `ps_candidates`, and `ps_proposals` small while testing.
+- Daytona quotas/API limits are outside Trace-Bench's control. Use `harbor_cli_timeout_seconds` and low concurrency when keys are limited.
+- Stub mode verifies Trace-Bench wiring only; it does not prove Harbor, Daytona, Docker, credentials, or a specific Terminal-Bench task works.
+- Trace-Bench currently optimizes the Terminus-2 prompt template, not Harbor environment definitions or benchmark tests.
+
+See `configs/terminal_bench_demo.yaml` and `notebooks/05_terminal_bench_demo.ipynb` for an end-to-end walkthrough.
+
 ## Repository Layout
 
 ```
@@ -131,6 +198,7 @@ See [docs/running-experiments.md](docs/running-experiments.md) for full CLI usag
 |-------|----------|---------|
 | `hf` | `datasets>=2.0` | `hf:*` QA tasks |
 | `dspy` | `dspy-ai>=2.0` | `DSPyTrainer` in `trace_bench/trainers/` |
+| `terminal-bench` | `harbor[daytona]>=0.6.0` | `terminal_bench:*` tasks through Harbor, including Daytona env support |
 | `all-external` | all of the above | everything |
 
 ## Tests

diff --git a/configs/09_harbor_bench_colab.json b/configs/09_harbor_bench_colab.json
@@ -0,0 +1,42 @@
+{
+  "runs_dir": "/home/user/code/Trace-Bench/notebooks/runs/harbor_trace_bench",
+  "mode": "real",
+  "seeds": [
+    123
+  ],
+  "max_workers": 1,
+  "fail_fast": false,
+  "tasks": [
+    {
+      "id": "terminal_bench:regex-log",
+      "eval_kwargs": {
+        "harbor_dataset": "terminal-bench@2.0",
+        "harbor_env": "daytona",
+        "harbor_model": "openrouter/openai/gpt-5-nano",
+        "harbor_cli_timeout_seconds": 1800,
+        "harbor_agent_kwargs": {
+          "max_turns": 3,
+          "parser_name": "json"
+        }
+      }
+    }
+  ],
+  "trainers": [
+    {
+      "id": "PrioritySearch",
+      "params_variants": [
+        {
+          "threads": 1,
+          "ps_steps": 1,
+          "ps_batches": 1,
+          "ps_candidates": 1,
+          "ps_proposals": 1,
+          "ps_mem_update": 1
+        }
+      ],
+      "optimizer_kwargs": {
+        "memory_size": 5
+      }
+    }
+  ]
+}
diff --git a/configs/terminal_bench_demo.yaml b/configs/terminal_bench_demo.yaml
@@ -0,0 +1,27 @@
+runs_dir: runs
+mode: stub
+seeds: [123]
+max_workers: 1
+fail_fast: false
+
+# Terminal-Bench tasks require the Harbor CLI for real execution.
+# In stub mode (default here) Harbor is not invoked.
+
+tasks:
+  - id: terminal_bench:regex-log
+    eval_kwargs:
+      harbor_dataset: terminal-bench@2.0
+      # In Colab, prefer daytona (set DAYTONA_API_KEY). Locally, docker is fine.
+      harbor_env: daytona
+      # Real mode requires this (or env var HARBOR_MODEL), e.g. "openai/gpt-4o-mini".
+      harbor_model: null
+
+trainers:
+  - id: PrioritySearch
+    params_variants:
+      - threads: 1
+        ps_steps: 1
+        ps_batches: 1
+        ps_candidates: 2
+        ps_proposals: 2
+        ps_mem_update: 1
diff --git a/notebooks/05_terminal_bench_demo.ipynb b/notebooks/05_terminal_bench_demo.ipynb
@@ -0,0 +1,175 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Terminal\u2011Bench 2.0 demo (Trace\u2011Bench)\n",
+        "\n",
+        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/AgentOpt/Trace-Bench/blob/main/notebooks/05_terminal_bench_demo.ipynb)\n",
+        "\n",
+        "This notebook demonstrates how to run **Terminal\u2011Bench 2.0** tasks via **Harbor** from Trace\u2011Bench.\n",
+        "\n",
+        "Key points:\n",
+        "- Tasks are referenced as `terminal_bench:<task_slug>` (e.g. `terminal_bench:regex-log`).\n",
+        "- In Colab, Terminal\u2011Bench typically requires **Daytona** (`DAYTONA_API_KEY`) because Docker is not available.\n",
+        "- The optimization target is the **Terminus\u20112 prompt template** (a trainable Trace node).\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# If running from a cloned repo, install Trace-Bench in editable mode.\n",
+        "!pip -q install -e .\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "from pathlib import Path\n",
+        "\n",
+        "# Pick a persistent runs directory.\n",
+        "# In Colab you likely want to mount Drive first and set this under /content/drive/...\n",
+        "RUNS_DIR = Path(os.environ.get('TRACE_BENCH_RUNS_DIR', 'runs')).resolve()\n",
+        "RUNS_DIR.mkdir(parents=True, exist_ok=True)\n",
+        "print('RUNS_DIR =', RUNS_DIR)\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# List available trainers (from Trace/OpenTrace) and the curated TB task subset.\n",
+        "!trace-bench list-trainers | head -n 40\n",
+        "print('\\n--- terminal_bench sample tasks ---')\n",
+        "!trace-bench list-tasks --bench terminal_bench --root .\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## 1) Stub smoke (offline)\n",
+        "\n",
+        "The shipped config `configs/terminal_bench_demo.yaml` defaults to `mode: stub`, meaning Harbor is **not** invoked.\n",
+        "This is useful for CI and quick sanity checks.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!trace-bench validate --config configs/terminal_bench_demo.yaml --runs-dir \"$RUNS_DIR\" --strict\n",
+        "!trace-bench run --config configs/terminal_bench_demo.yaml --runs-dir \"$RUNS_DIR\"\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## 2) Real mode (Harbor + Daytona)\n",
+        "\n",
+        "Requirements:\n",
+        "- `harbor` CLI available (`pip install harbor` typically provides it)\n",
+        "- `DAYTONA_API_KEY` exported (Colab recommended)\n",
+        "- A Harbor model name provided via `HARBOR_MODEL` (e.g. `openai/gpt-4o-mini`)\n",
+        "\n",
+        "If you don't have these, keep using stub mode.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import shutil, textwrap\n",
+        "\n",
+        "if shutil.which('harbor') is None:\n",
+        "    print('harbor CLI not found. Install it first (e.g. pip install harbor).')\n",
+        "else:\n",
+        "    print('harbor CLI found:', shutil.which('harbor'))\n",
+        "\n",
+        "print('DAYTONA_API_KEY set:', bool(os.environ.get('DAYTONA_API_KEY')))\n",
+        "print('HARBOR_MODEL:', os.environ.get('HARBOR_MODEL'))\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Create a small real-mode config on the fly.\n",
+        "# NOTE: replace trainer settings / timeouts as needed.\n",
+        "real_cfg = f\"\"\"\n",
+        "runs_dir: {RUNS_DIR}\n",
+        "mode: real\n",
+        "seeds: [123]\n",
+        "max_workers: 1\n",
+        "fail_fast: false\n",
+        "\n",
+        "tasks:\n",
+        "  - id: terminal_bench:regex-log\n",
+        "    eval_kwargs:\n",
+        "      harbor_dataset: terminal-bench@2.0\n",
+        "      harbor_env: daytona\n",
+        "      # read from env in the guide if omitted\n",
+        "      harbor_model: null\n",
+        "      harbor_cli_timeout_seconds: 1800\n",
+        "\n",
+        "trainers:\n",
+        "  - id: PrioritySearch\n",
+        "    params_variants:\n",
+        "      - threads: 1\n",
+        "        ps_steps: 1\n",
+        "        ps_batches: 1\n",
+        "        ps_candidates: 2\n",
+        "        ps_proposals: 2\n",
+        "        ps_mem_update: 1\n",
+        "\"\"\"\n",
+        "\n",
+        "cfg_path = Path('configs/terminal_bench_real_tmp.yaml')\n",
+        "cfg_path.write_text(textwrap.dedent(real_cfg))\n",
+        "print('Wrote', cfg_path)\n",
+        "print(cfg_path.read_text()[:400])\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Run a single TB task in real mode.\n",
+        "# This will write Harbor artifacts under runs/<run_id>/jobs/<job_id>/artifacts/terminal_bench/.\n",
+        "!trace-bench validate --config configs/terminal_bench_real_tmp.yaml --runs-dir \"$RUNS_DIR\" --strict\n",
+        "!trace-bench run --config configs/terminal_bench_real_tmp.yaml --runs-dir \"$RUNS_DIR\"\n"
+      ]
+    }
+  ],
+  "metadata": {
+    "kernelspec": {
+      "display_name": "Python 3",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python",
+      "version": "3"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 5
+}