Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions DOCS.md
Original file line number Diff line number Diff line change
Expand Up @@ -853,11 +853,11 @@ The full field-by-field data contract lives in [docs/hosted.md — Privacy model

### Plan limits

Local runs are always unlimited — plan limits apply only to what you push. Every new hosted account gets a 30-day Pro trial (no card) that covers every org it creates; after that each org needs its own Pro subscription. When a push exceeds your org's plan (monthly runs, seats, concurrent runs), the org's trial ended without a subscription, or the subscription has stopped paying, the server answers `402` and the CLI prints exactly what it said:
Local runs are always unlimited — plan limits apply only to what you push. Every new hosted account gets a 30-day Pro trial (no card) that covers every org it creates; after that each org needs its own Starter or Pro subscription. When a push exceeds your org's plan (monthly runs, seats, concurrent runs), the org's trial ended without a subscription, or the subscription has stopped paying, the server answers `402` and the CLI prints exactly what it said:

```
✗ EvalShift: this run needs a paid plan.
The free trial for this organization ended on 2026-11-02. Subscribe to Pro to push runs and keep the CI gate — existing runs stay readable.
The free trial for this organization ended on 2026-11-02. Subscribe to Starter or Pro to push runs and keep the CI gate — existing runs stay readable.
Upgrade: https://app.evalshift.dev/app/acme/settings/billing
```

Expand All @@ -870,7 +870,7 @@ Exit code is 1 and nothing is uploaded. The CLI never decides entitlements itsel
`evalshift init --ci` scaffolds `.github/workflows/evalshift.yml` — a production-shaped, self-documenting workflow (the setup checklist lives in its header comment) with three jobs:

- `discover` lists committed suites under `.evalshift/suites/*/golden.jsonl` — a suite added by `capture sync` is evaluated on the next run with no workflow edit, and a project with no suites yet skips green. Suites must be committed for CI to see them: keep `.evalshift/*` ignored but un-ignore `.evalshift/suites/` and `.evalshift/toolsets/`.
- `eval <suite>` is a matrix job per suite (the action evaluates one suite per invocation) with `fail-on: policy` and `evalshift-version` pinned to the CLI that scaffolded the project. Each suite is selected with `suite-name:` — see [Selecting a suite: name, not path](#selecting-a-suite-name-not-path). `max-parallel` defaults to 1; raise it toward the hosted plan's concurrent-run limit (Pro: 5; Enterprise: by contract). Only the first matrix job posts the PR comment — the comment marker is a constant, so multiple suites would overwrite one another.
- `eval <suite>` is a matrix job per suite (the action evaluates one suite per invocation) with `fail-on: policy` and `evalshift-version` pinned to the CLI that scaffolded the project. Each suite is selected with `suite-name:` — see [Selecting a suite: name, not path](#selecting-a-suite-name-not-path). `max-parallel` defaults to 1; raise it toward the hosted plan's concurrent-run limit (Starter: 2; Pro: 5; Enterprise: by contract). Only the first matrix job posts the PR comment — the comment marker is a constant, so multiple suites would overwrite one another.
- `evalshift gate` is the single check to require in branch protection: it fails if any suite failed and passes when evaluation was skipped (fork PR, no suites, or `EVALSHIFT_TOKEN` not yet set). Don't require the per-suite jobs (dynamic names) or the `evalshift/regression` commit status (last writer wins across suites).

Runs on pushes to main create the base-branch baselines PRs diff against, so the workflow cancels superseded runs on PRs only, never on main.
Expand Down
4 changes: 2 additions & 2 deletions docs/github-action.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,8 @@ checklist) with three jobs:
to the CLI that scaffolded the project: the CLI that *reads* the config in CI must be at least as new
as the CLI that *wrote* it locally (`extra: forbid` rejects newer keys), and
the CLI warns when the pin falls behind (or runs ahead of the local CLI) — see [Pin drift](#pin-drift). `max-parallel` defaults to 1 —
raise it toward your hosted plan's concurrent-run limit (Pro: 5;
Enterprise: by contract). The PR comment is posted by the first matrix job only: the
raise it toward your hosted plan's concurrent-run limit (Starter: 2;
Pro: 5; Enterprise: by contract). The PR comment is posted by the first matrix job only: the
comment marker is a constant, so multiple suites would overwrite one
another's summary.
- **`evalshift gate`** — the join job to require in branch protection. It
Expand Down
4 changes: 2 additions & 2 deletions docs/hosted.md
Original file line number Diff line number Diff line change
Expand Up @@ -247,7 +247,7 @@ the wrong one fails exactly like a permissions problem.
Running locally is always unlimited. Plan limits apply only to what you push.

Every new hosted account gets a 30-day Pro trial (no card) that covers every
org it creates; after that each org needs its own Pro subscription.
org it creates; after that each org needs its own Starter or Pro subscription.

When a push exceeds your organization's plan — monthly runs, seats, concurrent
runs — the org's trial ended without a subscription, or the subscription has
Expand All @@ -256,7 +256,7 @@ exactly what the server said:

```
✗ EvalShift: this run needs a paid plan.
The free trial for this organization ended on 2026-11-02. Subscribe to Pro to push runs and keep the CI gate — existing runs stay readable.
The free trial for this organization ended on 2026-11-02. Subscribe to Starter or Pro to push runs and keep the CI gate — existing runs stay readable.
Upgrade: https://app.evalshift.dev/app/acme/settings/billing
```

Expand Down
2 changes: 1 addition & 1 deletion llms-full.txt
Original file line number Diff line number Diff line change
Expand Up @@ -1906,7 +1906,7 @@ evalshift compare --suite-name <suite> --to gemini-3.1-pro-preview --gate critic
# eval <suite> -> matrix job per suite (action evaluates ONE suite per invocation);
# fail-on: policy; evalshift-version pinned to the scaffolding CLI (extra="forbid" config
# from a newer CLI fails loudly on an older one -- reader >= writer; see CI pin drift); max-parallel: 1 by default — raise
# toward the hosted plan's concurrent-run limit (Pro: 5; Enterprise: by contract); PR comment only
# toward the hosted plan's concurrent-run limit (Starter: 2; Pro: 5; Enterprise: by contract); PR comment only
# from the first matrix job (constant comment marker, multiple suites would overwrite);
# provider key env matches init's --provider (anthropic/deepseek: plus the borrowed
# embedding key as an optional second secret; unset -> semantic skipped with a warning);
Expand Down
4 changes: 2 additions & 2 deletions src/evalshift_cli/cli/commands/_scaffold.py
Original file line number Diff line number Diff line change
Expand Up @@ -242,8 +242,8 @@
# One suite regressing must not cancel the others' results.
fail-fast: false
# Hosted EvalShift refuses uploads past your org's concurrent-run
# limit (Pro: 5; Enterprise: by contract). Raise this toward it
# to evaluate suites in parallel.
# limit (Starter: 2; Pro: 5; Enterprise: by contract). Raise this
# toward it to evaluate suites in parallel.
max-parallel: 1
matrix:
suite: ${{ fromJSON(needs.discover.outputs.suites) }}
Expand Down
7 changes: 6 additions & 1 deletion tests/unit/test_docs_currency.py
Original file line number Diff line number Diff line change
Expand Up @@ -205,7 +205,12 @@ def test_no_doc_shows_a_removed_top_level_key(name: str) -> None:

#: Exact substrings that described the v5 hosted plans. v6 (2026-10) has no hosted Free or
#: Team plan; the CLI itself is still free. Retiring another plan claim? Append it here.
RETIRED_PLAN_TERMS: tuple[str, ...] = ("Free 1, Pro 5, Team 10", "on the Free plan")
RETIRED_PLAN_TERMS: tuple[str, ...] = (
"Free 1, Pro 5, Team 10",
"on the Free plan",
"each org needs its own Pro subscription",
"(Pro: 5; Enterprise: by contract)",
)

_PLAN_COPY_FILES: tuple[str, ...] = tuple(
dict.fromkeys(
Expand Down
2 changes: 1 addition & 1 deletion tests/unit/test_hosted_cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -1249,7 +1249,7 @@ def test_push_command_exits_non_zero_and_prints_the_upgrade_prompt_on_402(


_TRIAL_ENDED = (
"The free trial for this organization ended on 2026-11-02. Subscribe to Pro to push "
"The free trial for this organization ended on 2026-11-02. Subscribe to Starter or Pro to push "
"runs and keep the CI gate — existing runs stay readable."
)
_RUN_LIMIT = "Monthly run limit reached on the Pro plan (4,000 of 4,000 runs used)."
Expand Down
Loading