Quickstart · What it does · Documentation · Report an issue
Nullius runs AI evaluations on your machines and opens every score into the evidence behind it: what failed, the agent or the benchmark.
Python 3.12+ and uv; a model key to run benchmarks; Docker for the docker backend and for Harbor.
Install the latest version with uv. The PyPI package is nullius-eval;
the application and its command remain nullius:
uv tool install nullius-eval
nullius serveTo select a specific version, use uv tool install nullius-eval==0.1.0rc2.
The Quickstart guide covers upgrades, wheel and
source installation, and migration from a checkout.
Open 127.0.0.1:5057 to go straight to Runs. A first open offers a tour and a First steps checklist; Settings, General, replays it. The Quickstart guide: open results, a first run, the app on another machine.
# open job folders from the command line
nullius serve /path/to/harbor/jobs /path/to/other/jobs
# another address or port; APP_HOST and APP_PORT also work as environment variables
nullius serve --host 127.0.0.1 --port 8080Settings, licences, user catalogue entries and runs live outside the installed
package. nullius paths shows their locations; NULLIUS_HOME selects a separate
profile. Provider keys can be saved in Settings without creating an .env file.
+ Add on Benchmarks, Agents or Capabilities writes a prompt for your coding assistant. Its import skill pins the upstream, wraps the upstream's own code, proves it and reports what fits; it asks before any code of ours. Additions go in your user catalogue and survive application upgrades.
The import skills: /import-benchmark (asks how it runs, Nullius or Harbor),
/import-standard, /import-agent, /import-capability.
A benchmark names the ports it needs; an agent or a capability offers a port once its kit passes. Only pairs that fit are offered; a refusal names the missing port.
With no agent, the nop control must score nothing and the oracle everything; the units come out the same twice; no answer sits in a prompt. GSM8K at openai/grade-school-math@3101c7d: nop 0/1319, oracle 1319/1319.
Several agents and datasets in one run. This machine, a host over ssh, or several machines, one shard each, merged into one job by cli merge. The docker backend gives each unit's agent or capability its own container. On the Nullius standard a relay per unit holds the key, which the agent never sees, and records every model call.
The Run builder: Runner → Benchmark → Dataset → Agent → Model → Limits → Where → Review. Review checks each machine before Run; follow each trial in the job's Live view.
A failure is the agent's (failed tests, no patch, an agent error), never reached it (rate limits, API and setup errors; the adjusted rate leaves them out), unknown (kept in the denominator), or the benchmark's, when a control finds the task broken. A trial opens into the verifier's report, the trace, the conversation and, with a judge key, a judge model's answers.
The same benchmark before and after a change: the tasks both runs share, the ones that disagree first.
Illustrated with one recorded task; — means not recorded.
| Way in | Runs | Reads |
|---|---|---|
| Nullius standard | Any agent (over ACP) or capability that proved the ports a benchmark needs; or the benchmark's own scripts as published, its Original setup. | Harbor-shaped job directories. |
| Harbor | Harbor's task folders in containers, with Harbor's agents. | Job directories from any Harbor agent. |
| ATIF trajectories | — | ATIF documents, converted into one job. |
| OpenTelemetry spans | — | OTLP/JSON span exports, converted into one job. |
| Measure | Question it answers |
|---|---|
| Raw solve rate | How many of all trials resolved? |
| Adjusted solve rate | How many resolved, leaving out infrastructure failures? |
| Mean score | For a scored benchmark (a token F1): how high? |
A run with no verdict shows —, not 0%.
| You see | It can mean | Where |
|---|---|---|
| Adjusted well above raw | Trials failed before the agent acted | Job page, outcome bar |
| Many unknown outcomes | The run left no verdict | Outcome mix |
| Low trace coverage | Little of the run was captured | Trace, coverage strip |
| A no-op passes, or the oracle fails | The task is broken | Runs page, task checks; Feasibility |
| The judge disagrees | The verifier needs a look | Judgement |
Every metric is in the field guide.
Loopback by default; nothing in a source is overwritten. The network serves a run (model calls, installs, a host), registry lookups and judge calls, nothing else.
docs/ also builds into a searchable website: building and publishing the
site.
| Page | Holds |
|---|---|
| Introduction | What Nullius does, where to start |
| Quickstart | Launch, open results, a first run |
| Core concepts | Benchmarks, agents, recipes, trials, evidence |
| Run builder | The stages, the run, Harbor's pin |
| Read results | Failures, controls, comparisons |
| Judgement | What a judge model is asked |
| The Nullius standard | Ports, kits, setups, controls, proofs; importing |
| Architecture | Execution paths, a run on disk, the pages |
| Settings | Providers, System One, LLM judge, hosts, keys |
| Benchmarks, Agents, Capabilities | What is supported; adding one |
| CLI | Every command, by topic |
| Metric field guide | Every metric and outcome |
| Repository map | Layout, conventions, test commands |
python3 app/tests/parallel.py
uv run --no-project --with-requirements app/requirements.txt \
python app/tests/parallel.py app/ui_testsThe tests workflow
runs both suites on pull requests and pushes to main; they can also be run locally.
Copyright 2026 Nullius contributors.
Nullius is licensed under the Apache License 2.0, except where otherwise noted. Bundled fonts retain their SIL Open Font License 1.1. Third-party benchmarks, agents, capabilities and datasets retain their own licenses.
Built by the Nullius contributors. To cite it:
@software{Nullius,
author = {{Nullius contributors}},
title = {{Nullius: the run behind a benchmark score}},
year = {2026},
url = {https://github.com/thinkflowlab/nullius}
}Nullius in verba.
Take nobody’s word for it.