diff --git a/DOCS.md b/DOCS.md index cb54d79..dec061a 100644 --- a/DOCS.md +++ b/DOCS.md @@ -853,11 +853,11 @@ The full field-by-field data contract lives in [docs/hosted.md — Privacy model ### Plan limits -Local runs are always unlimited — plan limits apply only to what you push. Every new hosted account gets a 30-day Pro trial (no card) that covers every org it creates; after that each org needs its own Pro subscription. When a push exceeds your org's plan (monthly runs, seats, concurrent runs), the org's trial ended without a subscription, or the subscription has stopped paying, the server answers `402` and the CLI prints exactly what it said: +Local runs are always unlimited — plan limits apply only to what you push. Every new hosted account gets a 30-day Pro trial (no card) that covers every org it creates; after that each org needs its own Starter or Pro subscription. When a push exceeds your org's plan (monthly runs, seats, concurrent runs), the org's trial ended without a subscription, or the subscription has stopped paying, the server answers `402` and the CLI prints exactly what it said: ``` ✗ EvalShift: this run needs a paid plan. - The free trial for this organization ended on 2026-11-02. Subscribe to Pro to push runs and keep the CI gate — existing runs stay readable. + The free trial for this organization ended on 2026-11-02. Subscribe to Starter or Pro to push runs and keep the CI gate — existing runs stay readable. Upgrade: https://app.evalshift.dev/app/acme/settings/billing ``` @@ -870,7 +870,7 @@ Exit code is 1 and nothing is uploaded. The CLI never decides entitlements itsel `evalshift init --ci` scaffolds `.github/workflows/evalshift.yml` — a production-shaped, self-documenting workflow (the setup checklist lives in its header comment) with three jobs: - `discover` lists committed suites under `.evalshift/suites/*/golden.jsonl` — a suite added by `capture sync` is evaluated on the next run with no workflow edit, and a project with no suites yet skips green. Suites must be committed for CI to see them: keep `.evalshift/*` ignored but un-ignore `.evalshift/suites/` and `.evalshift/toolsets/`. -- `eval ` is a matrix job per suite (the action evaluates one suite per invocation) with `fail-on: policy` and `evalshift-version` pinned to the CLI that scaffolded the project. Each suite is selected with `suite-name:` — see [Selecting a suite: name, not path](#selecting-a-suite-name-not-path). `max-parallel` defaults to 1; raise it toward the hosted plan's concurrent-run limit (Pro: 5; Enterprise: by contract). Only the first matrix job posts the PR comment — the comment marker is a constant, so multiple suites would overwrite one another. +- `eval ` is a matrix job per suite (the action evaluates one suite per invocation) with `fail-on: policy` and `evalshift-version` pinned to the CLI that scaffolded the project. Each suite is selected with `suite-name:` — see [Selecting a suite: name, not path](#selecting-a-suite-name-not-path). `max-parallel` defaults to 1; raise it toward the hosted plan's concurrent-run limit (Starter: 2; Pro: 5; Enterprise: by contract). Only the first matrix job posts the PR comment — the comment marker is a constant, so multiple suites would overwrite one another. - `evalshift gate` is the single check to require in branch protection: it fails if any suite failed and passes when evaluation was skipped (fork PR, no suites, or `EVALSHIFT_TOKEN` not yet set). Don't require the per-suite jobs (dynamic names) or the `evalshift/regression` commit status (last writer wins across suites). Runs on pushes to main create the base-branch baselines PRs diff against, so the workflow cancels superseded runs on PRs only, never on main. diff --git a/docs/github-action.md b/docs/github-action.md index c987d62..e97833c 100644 --- a/docs/github-action.md +++ b/docs/github-action.md @@ -30,8 +30,8 @@ checklist) with three jobs: to the CLI that scaffolded the project: the CLI that *reads* the config in CI must be at least as new as the CLI that *wrote* it locally (`extra: forbid` rejects newer keys), and the CLI warns when the pin falls behind (or runs ahead of the local CLI) — see [Pin drift](#pin-drift). `max-parallel` defaults to 1 — - raise it toward your hosted plan's concurrent-run limit (Pro: 5; - Enterprise: by contract). The PR comment is posted by the first matrix job only: the + raise it toward your hosted plan's concurrent-run limit (Starter: 2; + Pro: 5; Enterprise: by contract). The PR comment is posted by the first matrix job only: the comment marker is a constant, so multiple suites would overwrite one another's summary. - **`evalshift gate`** — the join job to require in branch protection. It diff --git a/docs/hosted.md b/docs/hosted.md index 1509271..5cbc007 100644 --- a/docs/hosted.md +++ b/docs/hosted.md @@ -247,7 +247,7 @@ the wrong one fails exactly like a permissions problem. Running locally is always unlimited. Plan limits apply only to what you push. Every new hosted account gets a 30-day Pro trial (no card) that covers every -org it creates; after that each org needs its own Pro subscription. +org it creates; after that each org needs its own Starter or Pro subscription. When a push exceeds your organization's plan — monthly runs, seats, concurrent runs — the org's trial ended without a subscription, or the subscription has @@ -256,7 +256,7 @@ exactly what the server said: ``` ✗ EvalShift: this run needs a paid plan. - The free trial for this organization ended on 2026-11-02. Subscribe to Pro to push runs and keep the CI gate — existing runs stay readable. + The free trial for this organization ended on 2026-11-02. Subscribe to Starter or Pro to push runs and keep the CI gate — existing runs stay readable. Upgrade: https://app.evalshift.dev/app/acme/settings/billing ``` diff --git a/llms-full.txt b/llms-full.txt index 5102c26..bbc64cb 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -1906,7 +1906,7 @@ evalshift compare --suite-name --to gemini-3.1-pro-preview --gate critic # eval -> matrix job per suite (action evaluates ONE suite per invocation); # fail-on: policy; evalshift-version pinned to the scaffolding CLI (extra="forbid" config # from a newer CLI fails loudly on an older one -- reader >= writer; see CI pin drift); max-parallel: 1 by default — raise -# toward the hosted plan's concurrent-run limit (Pro: 5; Enterprise: by contract); PR comment only +# toward the hosted plan's concurrent-run limit (Starter: 2; Pro: 5; Enterprise: by contract); PR comment only # from the first matrix job (constant comment marker, multiple suites would overwrite); # provider key env matches init's --provider (anthropic/deepseek: plus the borrowed # embedding key as an optional second secret; unset -> semantic skipped with a warning); diff --git a/src/evalshift_cli/cli/commands/_scaffold.py b/src/evalshift_cli/cli/commands/_scaffold.py index 7e8fca9..55cfed2 100644 --- a/src/evalshift_cli/cli/commands/_scaffold.py +++ b/src/evalshift_cli/cli/commands/_scaffold.py @@ -242,8 +242,8 @@ # One suite regressing must not cancel the others' results. fail-fast: false # Hosted EvalShift refuses uploads past your org's concurrent-run - # limit (Pro: 5; Enterprise: by contract). Raise this toward it - # to evaluate suites in parallel. + # limit (Starter: 2; Pro: 5; Enterprise: by contract). Raise this + # toward it to evaluate suites in parallel. max-parallel: 1 matrix: suite: ${{ fromJSON(needs.discover.outputs.suites) }} diff --git a/tests/unit/test_docs_currency.py b/tests/unit/test_docs_currency.py index 08357d2..09931e5 100644 --- a/tests/unit/test_docs_currency.py +++ b/tests/unit/test_docs_currency.py @@ -205,7 +205,12 @@ def test_no_doc_shows_a_removed_top_level_key(name: str) -> None: #: Exact substrings that described the v5 hosted plans. v6 (2026-10) has no hosted Free or #: Team plan; the CLI itself is still free. Retiring another plan claim? Append it here. -RETIRED_PLAN_TERMS: tuple[str, ...] = ("Free 1, Pro 5, Team 10", "on the Free plan") +RETIRED_PLAN_TERMS: tuple[str, ...] = ( + "Free 1, Pro 5, Team 10", + "on the Free plan", + "each org needs its own Pro subscription", + "(Pro: 5; Enterprise: by contract)", +) _PLAN_COPY_FILES: tuple[str, ...] = tuple( dict.fromkeys( diff --git a/tests/unit/test_hosted_cli.py b/tests/unit/test_hosted_cli.py index 210cd72..c6cb698 100644 --- a/tests/unit/test_hosted_cli.py +++ b/tests/unit/test_hosted_cli.py @@ -1249,7 +1249,7 @@ def test_push_command_exits_non_zero_and_prints_the_upgrade_prompt_on_402( _TRIAL_ENDED = ( - "The free trial for this organization ended on 2026-11-02. Subscribe to Pro to push " + "The free trial for this organization ended on 2026-11-02. Subscribe to Starter or Pro to push " "runs and keep the CI gate — existing runs stay readable." ) _RUN_LIMIT = "Monthly run limit reached on the Pro plan (4,000 of 4,000 runs used)."