Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .agents/skills/agents-shipgate/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,15 @@ Do not use it for general linting, runtime monitoring, evals, model-output quali
2. For reading `report.json`, summarizing release decisions, or deciding what may be auto-applied, read `references/report-reading.md`.
3. Set `AGENTS_SHIPGATE_AGENT_MODE=1` before running Shipgate commands so errors include structured `next_action` JSON.
4. Default first-time CI to advisory mode. Do not enable release-blocking CI or save a baseline until a human has reviewed current findings.
5. Always parse `agents-shipgate-reports/report.json`, not Markdown. Use `release_decision.decision` as the release signal.
5. For verify runs, read `agents-shipgate-reports/verifier.json` first and lead with `merge_verdict`; then parse `report.json` and use `release_decision.decision` as the release gate.
6. Auto-apply only high-confidence safe patches. Do not auto-assert approval, confirmation, idempotency, broad-scope, prohibited-action, or runtime-trace evidence.
7. Ensure `.gitignore` covers `agents-shipgate-reports/` before committing.

## Fast Paths

- First adoption: run `agents-shipgate detect --workspace . --json`, then follow `references/recipes.md`.
- Local agent-related diff: run `agents-shipgate verify --workspace . --config shipgate.yaml --ci-mode advisory --format json`. Add `--base origin/main --head HEAD` only for committed PR/CI verification after making the base ref available.
- Existing manifest: run `agents-shipgate scan -c shipgate.yaml --suggest-patches --format json`.
- Existing manifest / ongoing PR: run `agents-shipgate verify --workspace . --config shipgate.yaml --ci-mode advisory --format json`.
- First GitHub CI: copy `assets/advisory-pr-comment.yml` to `.github/workflows/agents-shipgate.yml`.
- Explain one finding: run `agents-shipgate explain-finding <fingerprint> --from agents-shipgate-reports/report.json --json`.
- Triage heuristic findings: run `agents-shipgate findings --from agents-shipgate-reports/report.json --provenance-kind keyword_heuristic,regex_heuristic --json`.
Expand Down
12 changes: 7 additions & 5 deletions .agents/skills/agents-shipgate/references/recipes.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,11 +56,13 @@ uncommitted edits. In committed PR or CI contexts, add
`--base origin/main --head HEAD` after making the base ref available. If you
pass a missing `--base`, `verify` exits 2 with an unknown merge verdict.

Read `agents-shipgate-reports/report.json` first. Use
`release_decision.decision` as the gate. Use `verifier_summary` only as a
composition summary: its `verdict` mirrors `release_decision.decision` and it
adds counts for protected-surface touches, policy weakening, human
acknowledgement, and top reason codes.
Read `agents-shipgate-reports/verifier.json` first. Lead with
`merge_verdict`, then inspect `capability_review.top_changes[]`,
`first_next_action.actor`, and `fix_task.safe_to_attempt`. Then read
`agents-shipgate-reports/report.json`; `release_decision.decision` remains the
gate. Use `verifier_summary` only as a composition summary: its `verdict`
mirrors `release_decision.decision` and it adds counts for protected-surface
touches, policy weakening, human acknowledgement, and top reason codes.

Do not bypass the verifier. Do not suppress findings, lower severity, expand
baselines or waivers, remove Shipgate CI, or weaken agent instructions to make
Expand Down
15 changes: 9 additions & 6 deletions .agents/skills/agents-shipgate/references/report-reading.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,15 @@
# Reading Agents Shipgate Reports

Always read `agents-shipgate-reports/report.json`. Do not scrape Markdown.
For verify runs, read `agents-shipgate-reports/verifier.json` first. Then read
`agents-shipgate-reports/report.json`. Do not scrape Markdown.

## Order

1. `release_decision.decision`: `blocked`, `review_required`, `insufficient_evidence`, or `passed`.
2. `release_decision.blockers[]`: items blocking release.
3. `release_decision.review_items[]`: accepted debt or human-review items.
4. `agent_summary`: one-fetch summary with `headline`, counts, safe patches, human-review needs, and `first_recommended_action`.
5. `verifier_summary`: one-fetch verifier composition for PR controllers. Its `verdict` mirrors `release_decision.decision`; use it for protected-surface, policy-weakening, human-ack, and reason-code rollups, never as a second gate.
1. `verifier.json.merge_verdict`: `mergeable`, `human_review_required`, `insufficient_evidence`, `blocked`, or `unknown`.
2. `verifier.json.capability_review.top_changes[]`: the highest-signal tool/action or trust-root changes.
3. `verifier.json.first_next_action` / `fix_task`: who acts next and whether a coding agent may safely attempt the fix.
4. `report.json.release_decision.decision`: `blocked`, `review_required`, `insufficient_evidence`, or `passed`; this is the release gate.
5. `release_decision.blockers[]` and `release_decision.review_items[]`.
6. `findings[]`: detailed evidence, source, severity, and remediation.

## Verifier Summary
Expand Down Expand Up @@ -58,7 +59,9 @@ For those, summarize the risk and the exact decision a human needs to make.
Report back with:

```text
Merge verdict: <verifier.json.merge_verdict>
Decision: <release_decision.decision>
Capability changes: <top verifier capability_review.top_changes entries>
Blockers: <count>
Review items: <count>
Safe patches applied: <count or none>
Expand Down
15 changes: 10 additions & 5 deletions .cursor/rules/agents-shipgate.mdc
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ capability changes — a local-first, static Tool-Use Readiness review.

When a change affects agent tools, MCP exports, OpenAPI specs, prompts,
permissions, approval policies, or release gates, run Agents Shipgate.
Default to advisory scans while adopting the gate.
Default to advisory verification while adopting the gate.

For an existing `shipgate.yaml`, prefer the ongoing-PR verifier before
finishing:
Expand All @@ -45,10 +45,15 @@ Omit `--base` and `--head` for local pre-commit work so uncommitted edits are
scanned; add `--base origin/main --head HEAD` only for committed PR/CI
verification after making the base ref available.

Use `agents-shipgate-reports/report.json` as the source of truth. Prefer
`release_decision.decision` over legacy severity/status summaries.
Use `agents-shipgate-reports/verifier.json` only for trigger/base orchestration
status, not as a second verdict.
Read `agents-shipgate-reports/verifier.json` first. Lead with
`merge_verdict`, then inspect `capability_review.top_changes[]`,
`first_next_action.actor`, and `fix_task.safe_to_attempt`. Use
`agents-shipgate-reports/report.json` as the source of truth for
`release_decision.decision`.

Do not claim completion when `merge_verdict` is `blocked`,
`insufficient_evidence`, or `human_review_required` unless the user explicitly
accepts human review.

Apply only high-confidence safe patches. Do not invent approval, confirmation,
or idempotency evidence.
Expand Down
18 changes: 10 additions & 8 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,13 +81,15 @@ AGENTS_SHIPGATE_AGENT_MODE=1 agents-shipgate verify \

Omit `--base`/`--head` for local pre-commit work so uncommitted edits are
scanned; add `--base origin/main --head HEAD` only for a committed PR/CI ref
after making the base ref available. The release gate is
`agents-shipgate-reports/report.json.release_decision.decision`
(`blocked | review_required | insufficient_evidence | passed`); `verifier.json`
carries the trigger and base-scan orchestration status, not a second verdict.
Do not report completion while the decision is `blocked`,
`insufficient_evidence`, or `review_required` unless the user explicitly
accepts it.
after making the base ref available. Read
`agents-shipgate-reports/verifier.json` first and lead with `merge_verdict`
(`mergeable | human_review_required | insufficient_evidence | blocked |
unknown`), `capability_review.top_changes[]`, and `first_next_action`.
Then read `agents-shipgate-reports/report.json.release_decision.decision`
(`blocked | review_required | insufficient_evidence | passed`), which remains
the release gate. Do not report completion while `merge_verdict` is `blocked`,
`insufficient_evidence`, or `human_review_required` unless the user explicitly
accepts human review.

Do not bypass the verifier by suppressing findings, lowering severity,
expanding baselines or waivers, removing Shipgate CI, or weakening agent
Expand All @@ -102,7 +104,7 @@ agents-shipgate fixture run support_refund_agent

---

## Single-turn agent flow (v0.6+)
## First-adoption helper flow (v0.6+)

For coding agents adopting Shipgate end-to-end in one turn:

Expand Down
109 changes: 75 additions & 34 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,13 +42,27 @@ no scanner network calls, no scanner telemetry. Audited exceptions are pinned
in [`tests/test_adapter_static_only.py::ALLOWED_EXCEPTIONS`](tests/test_adapter_static_only.py).
Apache-2.0.

## One-command quickstart
## Verify-first quickstart

The core loop is verify-first: when a PR changes what your agent can do, run the
deterministic verifier on the diff and read its merge verdict before you merge.
On a committed PR/CI ref, pass the base and head so the diff — the capability
delta and trust-root signals — is in scope (make the base ref available first,
e.g. `git fetch origin main`):

First ask whether Shipgate applies to the current repo or diff:

```bash
agents-shipgate verify --preview --json
```

If the repo is not configured yet, install the manifest, advisory CI, and
agent-facing instructions:

```bash
agents-shipgate init --workspace . --write --ci --agent-instructions=all
```

Then verify the committed PR/CI ref. Pass the base and head so the diff — the
capability delta and trust-root signals — is in scope (make the base ref
available first, e.g. `git fetch origin main`):

```bash
agents-shipgate verify --workspace . --config shipgate.yaml \
Expand All @@ -65,11 +79,13 @@ agents-shipgate verify --workspace . --config shipgate.yaml \

The release gate is `agents-shipgate-reports/report.json` →
`release_decision.decision` (`blocked | review_required | insufficient_evidence | passed`).
No `shipgate.yaml` yet? Run `agents-shipgate init --workspace . --write` first.
The PR/controller surface is `agents-shipgate-reports/verifier.json` →
`merge_verdict` (`mergeable | human_review_required | insufficient_evidence |
blocked | unknown`), a deterministic projection of the release decision.

Want a 5-minute first run with zero setup? Scan the bundled fixture. If you
already have [`uv`](https://docs.astral.sh/uv/) installed, the fixture path is a
one-command check with no persistent install:
Want a 5-minute demo with zero setup? Scan the bundled fixture. If you already
have [`uv`](https://docs.astral.sh/uv/) installed, the fixture path is a
one-command install check with no persistent install:

```bash
uvx agents-shipgate fixture run support_refund_agent
Expand All @@ -93,13 +109,23 @@ Reports: <tempdir>/reports
Fixture copy at <tempdir>; pass --keep to retain after the run.
```

Both blockers are on `stripe.create_refund`: missing approval policy and missing idempotency evidence. The fixture writes `report.{md,json}` and `packet.{md,json,html}` into the temp `reports/` directory. To scan your own repo and write the standard `agents-shipgate-reports/` directory, see [Scan your repo](#scan-your-repo) below.
Both blockers are on `stripe.create_refund`: missing approval policy and missing idempotency evidence. The fixture writes `report.{md,json}` and `packet.{md,json,html}` into the temp `reports/` directory. To verify your own repo and write the standard `agents-shipgate-reports/` directory, see [Verify your repo](#verify-your-repo) below.

![Sample Tool-Use Readiness Report showing 2 critical, 14 high, and 2 medium findings on the support_refund_agent fixture, including a missing approval policy on stripe.create_refund.](assets/sample-report.png)

## How to read your first result

Read `release_decision.decision` first:
For PR verification, read `verifier.json.merge_verdict` first:

| Merge verdict | Meaning | Next step |
|---|---|---|
| `blocked` | Active, unaccepted blockers exist. | Fix blockers or remove the risky capability. |
| `insufficient_evidence` | Static evidence is too weak to gate release confidently. | Add better sources and rerun; do not auto-merge. |
| `human_review_required` | A person must review accepted debt, trust-root changes, or authority-bearing gaps. | Surface the required review; a coding agent must not self-approve it. |
| `mergeable` | No active blocker or review signal was found. | Keep verifier/report artifacts with the PR record. |
| `unknown` | Verify could not produce a reliable head scan or diff context. | Fix setup, fetch the base ref, or rerun with usable inputs. |

Then read `report.json.release_decision.decision`, the source-of-truth gate:

| Decision | Meaning | Next step |
|---|---|---|
Expand All @@ -108,6 +134,10 @@ Read `release_decision.decision` first:
| `review_required` | Human review is needed, often for accepted debt or evidence gaps below the blocked threshold. | Review the listed items before promotion. |
| `passed` | No active blocker or review signal was found. | Keep the report artifact with the PR/release record. |

Common review signals include missing confirmation, missing idempotency
evidence, broad-scope permissions, prohibited-action policy gaps, and
trust-root changes such as weakened CI or manifest policy.

## GitHub Action Marketplace

The public Action is listed on the
Expand All @@ -125,7 +155,7 @@ curl -sSL https://raw.githubusercontent.com/ThreeMoonsLab/agents-shipgate/main/t
| python3 - --workspace . --json
```

Continue to [Scan your repo](#scan-your-repo) when the output has
Continue to [Verify your repo](#verify-your-repo) when the output has
`is_agent_project: true`, non-empty `suggested_sources`, non-empty
`codex_plugin_candidates`, or the workspace already has `shipgate.yaml`.

Expand All @@ -149,18 +179,18 @@ Evidence Packet in [`packet.md`](samples/support_refund_agent/expected/packet.md
```text
Add a Tool-Use Readiness release gate for this tool-using AI agent with Agents Shipgate.
Run:
agents-shipgate detect --workspace . --json
agents-shipgate verify --preview --json
If Shipgate is relevant, run:
agents-shipgate contract --json
agents-shipgate init --workspace . --write --ci --json
agents-shipgate scan -c shipgate.yaml --suggest-patches --format json
agents-shipgate apply-patches --from agents-shipgate-reports/report.json --confidence high --apply
Read `agents-shipgate-reports/report.json`, not Markdown. Summarize
`release_decision.decision`, blocker count, review item count, top critical/high
findings, and any safe patches applied. Auto-apply only high-confidence safe
patches. Do not auto-assert approval, confirmation, idempotency, broad-scope,
or prohibited-action policy decisions. Ensure `.gitignore` contains
`agents-shipgate-reports/` before committing.
agents-shipgate init --workspace . --write --ci --agent-instructions=all
agents-shipgate verify --workspace . --config shipgate.yaml --ci-mode advisory --format json
For committed PR/CI refs, add `--base origin/main --head HEAD` after making the
base ref available. Read `agents-shipgate-reports/verifier.json` first and lead
with `merge_verdict`, then read `agents-shipgate-reports/report.json` for
`release_decision.decision`. Do not claim completion when `merge_verdict` is
`blocked`, `insufficient_evidence`, or `human_review_required` unless the user
explicitly accepts human review. Do not auto-assert approval. Never suppress
findings, lower severity, expand baselines or waivers, remove Shipgate CI, or
weaken agent instructions just to make the verifier pass.
```

## Add the Codex adoption kit
Expand All @@ -173,7 +203,7 @@ agents-shipgate init --workspace . --write --agent-instructions=agents-md,codex-
```

The skill lives at `.agents/skills/agents-shipgate/`, can be invoked with
`$agents-shipgate`, and teaches Codex the bootstrap, scan, report-reading,
`$agents-shipgate`, and teaches Codex the verify, bootstrap, report-reading,
advisory CI, and finding-triage workflows.

To customize generated skill content in a downstream repo without rebuilding
Expand Down Expand Up @@ -204,15 +234,20 @@ evidence around them:
prohibited actions, or `shipgate.yaml`.
- GitHub Actions or CI release gates for a tool-using AI agent.

## Scan your repo
## Verify your repo

```bash
agents-shipgate init --workspace . --write --ci --json
agents-shipgate verify --preview --json
agents-shipgate init --workspace . --write --ci --agent-instructions=all
# Replace any CHANGE_ME placeholders reported by init.
agents-shipgate scan -c shipgate.yaml
agents-shipgate verify --workspace . --config shipgate.yaml \
--ci-mode advisory --format json
```

Reports land at `agents-shipgate-reports/report.{md,json,sarif}`; the Release Evidence Packet lands at `agents-shipgate-reports/packet.{md,json,html}`.
For committed PR/CI refs, add `--base origin/main --head HEAD` after making the
base ref available. Verify writes `agents-shipgate-reports/verifier.json`,
`pr-comment.md`, and the normal `report.{md,json,sarif}` / packet artifacts
when a scan is required.

Install alternatives (your agent project does **not** need Python 3.12 — install the CLI separately):

Expand All @@ -221,11 +256,11 @@ python -m pip install agents-shipgate # global pip
uv tool install agents-shipgate # via uv
```

## Adopt in one turn (for AI coding agents)
## Adopt in one turn (helper flow)

The v0.6 single-turn flow takes a workspace from "looks like an agent
project" to "Shipgate integrated, scan green or with safe patches
applied, CI workflow drafted":
The single-turn bootstrap flow remains useful for first adoption. It takes a
workspace from "looks like an agent project" to "Shipgate integrated, scan green
or with safe patches applied, CI workflow drafted":

```bash
agents-shipgate detect --json # 1. classify
Expand All @@ -246,15 +281,21 @@ minimal manifests, see [`docs/minimal-real-configs.md`](docs/minimal-real-config
## Use in CI

```yaml
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: ThreeMoonsLab/agents-shipgate@v0.11.0
with:
config: shipgate.yaml
ci_mode: advisory
diff_base: target
pr_comment: "true"
```

Set `pr_comment: "true"` to post a compact PR summary:
The PR comment leads with `merge_verdict`, capability changes, required next
action, and artifact links:

![Preview of the optional Agents Shipgate PR comment showing release blockers, severity counts, top findings, and report artifacts.](assets/pr-comment-preview.png)
![Preview of the optional Agents Shipgate PR comment showing merge verdict, capability changes, required next action, and report artifacts.](assets/pr-comment-preview.png)

## What it scans

Expand Down Expand Up @@ -355,7 +396,7 @@ Top findings:
The fastest way to understand what changes for a reviewer: walk through a Golden PR. Each one ships a sample manifest, the resulting report, the release decision, and the recommended PR-comment summary an agent should post.

- [`openai-agents-sdk-refund-agent`](examples/golden-prs/openai-agents-sdk-refund-agent/README.md) — refund agent adds `stripe.create_refund`. Shipgate decides `blocked` because approval policy and idempotency evidence are missing. Includes the recommended Markdown PR-comment template.
- [`golden-pr-from-coding-agent.md`](examples/golden-prs/golden-pr-from-coding-agent.md) — the *artifact* a coding agent should produce after running the canonical 4-call flow: PR comment, structured `agent_summary`, applied diff, review-item table.
- [`golden-pr-from-coding-agent.md`](examples/golden-prs/golden-pr-from-coding-agent.md) — the *artifact* a coding agent should produce after running the verify-first flow: PR comment, `merge_verdict`, `capability_review`, and human/coding-agent next action.
- [`mcp-only-tool-server`](examples/golden-prs/mcp-only-tool-server/README.md) — MCP server with no Python framework imports; demonstrates the MCP-only adoption path.
- [`openapi-support-agent`](examples/golden-prs/openapi-support-agent/README.md) — OpenAPI-described tool surface; shows scope-coverage findings.

Expand Down
Loading