diff --git a/AGENTS.md b/AGENTS.md index 7a9add2..bb7ad00 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -50,9 +50,10 @@ the success-path real Forge `open-pr` smoke has been rerun on the current checko held-merge and commit-status/comment block-surfacing legs remain unit-proven with real-effect evidence still owed. Strong no-phone-home evidence remains future work. EVRUN-partial is recorded; P11 also records a -blocked EVRUN-full capture attempt where the narrow Codex app-server and real-host smokes passed -outside the Codex sandbox, but the combined real GitHub path could not run without sandbox -credentials and an integrity key. EVRUN-full remains future work. +blocked EVRUN-full capture attempt plus a later successful combined EVRUN-full smoke: real GitHub +Issues, real Codex app-server, real host attestation, real GitHub Forge `open-pr`, and records +integrity on the disposable target repo. Strong no-phone-home, held-merge/idempotency, hosted, and +Windows evidence remain future work. ## Commands diff --git a/docs/delivery/target-state-implementation/README.md b/docs/delivery/target-state-implementation/README.md index 6080e81..b83b026 100644 --- a/docs/delivery/target-state-implementation/README.md +++ b/docs/delivery/target-state-implementation/README.md @@ -5,11 +5,11 @@ status: in progress # Target-state implementation — delivery track -**Status: in progress.** P01–P12 are merged (see the [phase table](#phase-table)): P11 merged with a -blocked EVRUN-full evidence record, so the full-capture demonstration is still owed. P13 is -currently blocked pending the contract-owner freeze decision and the remaining evidence/remediation -closeout; P14 is planned. Phase statuses live in the phase table below and in each phase doc's -frontmatter. +**Status: in progress.** P01–P12 are merged (see the [phase table](#phase-table)): P11 first +merged with a blocked EVRUN-full evidence record, and a later follow-up captured the combined +real Codex / real GitHub `open-pr` smoke. P13 is currently blocked pending the contract-owner +freeze decision and the remaining no-phone-home/idempotency evidence or explicit deferral; P14 is +planned. Phase statuses live in the phase table below and in each phase doc's frontmatter. ## Overview @@ -59,11 +59,11 @@ Verified against the repo after Phase 02 workspace split: driver) and Work source (GitHub Issues importer) are **real and usable**. The Agent seam now includes the private production Codex app-server transport: `bootstrap.ts` composes `createProductionCodexAgentSession`, and driver selection supports `agent: 'codex'`. The - remaining limitation is evidence, not absence of implementation: P11 still records only a - blocked EVRUN-full attempt, so the full real-effect proof for the combined Codex/GitHub path is - still owed. The real Execution host is now selectable on macOS and exercises a local - `process-group` confinement probe that reports an honest proven `weak` posture at compose time; - stronger no-phone-home evidence remains open for P11. + remaining limitation is evidence, not absence of implementation: P11 now records the combined + real Codex / real GitHub `open-pr` smoke, while stronger no-phone-home and repeated-effect + idempotency evidence remain open. The real Execution host is now selectable on macOS and + exercises a local `process-group` confinement probe that reports an honest proven `weak` posture + at compose time. - **Conformance and controlled doubles** now live in `packages/jig-testkit` (`packages/jig-testkit/src/provider-conformance.ts`, `packages/jig-testkit/tests/conformance/`), outside the production SDK dependency graph. @@ -77,10 +77,10 @@ Verified against the repo after Phase 02 workspace split: MCP adapter. - **Evidence**: EVRUN-partial is committed ([evidence index](../../design/evidence/README.md)) — one real - work-source → forge → records-integrity run with a **scripted** agent leg. P11 captured a - blocked EVRUN-full attempt: narrow Codex app-server and real-host smokes pass outside the - sandbox, but the combined real GitHub path could not run without sandbox credentials and an - integrity key. EVRUN-full remains open. + work-source → forge → records-integrity run with a **scripted** agent leg. P11 also records the + earlier blocked EVRUN-full attempt and the later combined real Codex / real-host / real GitHub + `open-pr` smoke. Strong no-phone-home, held-merge/idempotency, hosted, and Windows evidence + remain open. ## Desired target state @@ -103,8 +103,9 @@ Verified against the repo after Phase 02 workspace split: - **Observability** ([guarantee 5](../../product/guarantees.md#5-full-observability)): watch, notices with acknowledge/snooze, ask-why, and write-once redacted export, all answered from the run's own records. -- **Evidence**: EVRUN-full committed; contract-freeze readiness prepared for the contract - owner's T14 decision. +- **Evidence**: EVRUN-full combined smoke committed; remaining no-phone-home/idempotency evidence + captured or explicitly deferred; contract-freeze readiness prepared for the contract owner's T14 + decision. - **Docs**: README, AGENTS.md, and docs indexes state the shipped surface truthfully at every phase boundary. @@ -285,22 +286,22 @@ release posture`") ## Phase table -| ID | Phase | Status | Hard dependencies | Parallelization | -| --- | ------------------------------------------------------------------------------------------ | ------------------------------ | ----------------- | ---------------------------------------------------------------------------- | -| P01 | [SDK boundary and operator-control surface](./phases/01-sdk-boundary-and-operator-port.md) | merged (#55) | — | First; everything else keys off this surface. | -| P02 | [Package split: jig-sdk, jig-cli, jig-testkit](./phases/02-package-split-workspace.md) | merged (#56, #58) | P01 | Sole occupant of its slot — it moves every source file. | -| P03 | [Codex app-server transport](./phases/03-codex-app-server-transport.md) | merged (#57, #58, #59, #60) | P01 | Parallel with P04–P10 after P02 (soft). | -| P04 | [Execution-host containment and substrate](./phases/04-execution-host-containment.md) | merged (#63) | — | Parallel with P03, P05–P10 after P02 (soft). | -| P05 | [Forge and work-source completion](./phases/05-forge-and-work-source-completion.md) | merged (#64) | — | Parallel with P03, P04, P06–P10 after P02 (soft). | -| P06 | [Owner configuration model](./phases/06-owner-configuration-model.md) | merged (#65) | — | Parallel with P03–P05, P08–P10 after P02 (soft). | -| P07 | [Guided setup](./phases/07-guided-setup.md) | merged (#66) | P06 | Parallel with anything not touching config templates. | -| P08 | [Watch, notices, ask-why](./phases/08-observation-surfaces.md) | merged (#67) | P01 | Parallel with P03–P06, P09, P10; coordinate record vocabulary with P09. | -| P09 | [Decide and stop](./phases/09-owner-decision-and-run-control.md) | merged (#68) | P01 | Parallel with P03–P06, P08, P10; coordinate record vocabulary with P08. | -| P10 | [Export: write-once audit record](./phases/10-export-audit-record.md) | merged (#69) | P01 | Parallel with P03–P09. | -| P11 | [EVRUN-full evidence](./phases/11-evrun-full-evidence.md) | merged (#70; blocked evidence) | P03, P04 | Sequential after both provider phases; benefits from P05. | -| P12 | [MCP driving adapter](./phases/12-mcp-adapter.md) | merged (#71) | P02 | Parallel with P11; soft dependency on P08/P09 for verb coverage. | -| P13 | [Contract v0 freeze readiness](./phases/13-contract-freeze-readiness.md) | blocked | P05, P07–P12 | Requires evidence/remediation closeout and a contract-owner freeze decision. | -| P14 | [Target-state audit, docs, release posture](./phases/14-docs-and-release-readiness.md) | planned | All other phases | Last; closes the track. | +| ID | Phase | Status | Hard dependencies | Parallelization | +| --- | ------------------------------------------------------------------------------------------ | ----------------------------------------------------------- | ----------------- | ---------------------------------------------------------------------------- | +| P01 | [SDK boundary and operator-control surface](./phases/01-sdk-boundary-and-operator-port.md) | merged (#55) | — | First; everything else keys off this surface. | +| P02 | [Package split: jig-sdk, jig-cli, jig-testkit](./phases/02-package-split-workspace.md) | merged (#56, #58) | P01 | Sole occupant of its slot — it moves every source file. | +| P03 | [Codex app-server transport](./phases/03-codex-app-server-transport.md) | merged (#57, #58, #59, #60) | P01 | Parallel with P04–P10 after P02 (soft). | +| P04 | [Execution-host containment and substrate](./phases/04-execution-host-containment.md) | merged (#63) | — | Parallel with P03, P05–P10 after P02 (soft). | +| P05 | [Forge and work-source completion](./phases/05-forge-and-work-source-completion.md) | merged (#64) | — | Parallel with P03, P04, P06–P10 after P02 (soft). | +| P06 | [Owner configuration model](./phases/06-owner-configuration-model.md) | merged (#65) | — | Parallel with P03–P05, P08–P10 after P02 (soft). | +| P07 | [Guided setup](./phases/07-guided-setup.md) | merged (#66) | P06 | Parallel with anything not touching config templates. | +| P08 | [Watch, notices, ask-why](./phases/08-observation-surfaces.md) | merged (#67) | P01 | Parallel with P03–P06, P09, P10; coordinate record vocabulary with P09. | +| P09 | [Decide and stop](./phases/09-owner-decision-and-run-control.md) | merged (#68) | P01 | Parallel with P03–P06, P08, P10; coordinate record vocabulary with P08. | +| P10 | [Export: write-once audit record](./phases/10-export-audit-record.md) | merged (#69) | P01 | Parallel with P03–P09. | +| P11 | [EVRUN-full evidence](./phases/11-evrun-full-evidence.md) | merged (#70; combined smoke captured, stronger probes open) | P03, P04 | Sequential after both provider phases; benefits from P05. | +| P12 | [MCP driving adapter](./phases/12-mcp-adapter.md) | merged (#71) | P02 | Parallel with P11; soft dependency on P08/P09 for verb coverage. | +| P13 | [Contract v0 freeze readiness](./phases/13-contract-freeze-readiness.md) | blocked | P05, P07–P12 | Requires evidence/remediation closeout and a contract-owner freeze decision. | +| P14 | [Target-state audit, docs, release posture](./phases/14-docs-and-release-readiness.md) | planned | All other phases | Last; closes the track. | ## What can run in parallel diff --git a/docs/delivery/target-state-implementation/phases/11-evrun-full-evidence.md b/docs/delivery/target-state-implementation/phases/11-evrun-full-evidence.md index 47868dd..1954f09 100644 --- a/docs/delivery/target-state-implementation/phases/11-evrun-full-evidence.md +++ b/docs/delivery/target-state-implementation/phases/11-evrun-full-evidence.md @@ -1,6 +1,6 @@ --- title: "Phase 11 - EVRUN-full evidence" -status: "merged (#70; blocked evidence)" +status: "merged (#70; combined smoke captured, stronger probes open)" --- # Phase 11 - EVRUN-full evidence @@ -10,10 +10,10 @@ status: "merged (#70; blocked evidence)" Run and commit the EVRUN-full evidence: a real, end-to-end Codex-driven delivery — GitHub Issues work source, real Codex agent over the app-server transport, real execution-host containment, real GitHub Forge landing, integrity and redaction active — plus the adversarial -probes EVRUN-partial could not claim (no-phone-home, multi-run idempotency). The implemented P11 -record is an honest blocked capture attempt: the narrow Codex app-server and real-host smoke probes -were refreshed, but the combined EVRUN-full path could not run because the sandbox GitHub and -integrity prerequisites were absent. Product code changes are not this phase's purpose. +probes EVRUN-partial could not claim (no-phone-home, multi-run idempotency). P11 first landed an +honest blocked capture attempt; a later follow-up captured the combined real GitHub / real Codex / +real-host `open-pr` smoke. Strong no-phone-home, held-merge replay, and multi-run idempotency +remain open. Product code changes are not this phase's purpose. ## Background @@ -54,8 +54,9 @@ gated on this evidence path. This phase is the proof step: it converts "implemen in [`product/jig.md`](../../../product/jig.md#what-jig-isnt-yet). - A successful capture would prove `SEC-2` (no-phone-home proven, not asserted), `RESUME-3` (no double effect against real systems), and `MERGE-2`/`FENCE-3` observations on a real path. -- The implemented blocked capture attempt does not unlock P13 or authorize P14 status claims; - those remain gated on either EVRUN-full evidence or an explicit owner decision to defer the gate. +- The later combined smoke removes the original "real Codex plus real GitHub path unproven" + blocker. P13/P14 status claims still need either stronger no-phone-home/idempotency evidence or + an explicit owner decision to defer those gates. ## Technical Requirements @@ -86,28 +87,31 @@ gated on this evidence path. This phase is the proof step: it converts "implemen - **Requires:** P03 and P04 (hard). - **Benefits from:** P05 (block-surfacing and held-merge paths worth capturing in the same sweep). -- **Would unlock:** P13 (hard) and P14 status claims only after EVRUN-full evidence or an explicit - owner deferral decision. +- **Would unlock:** P13 (hard) and P14 status claims only after the remaining no-phone-home / + idempotency evidence is captured or explicitly deferred by the owner. - **Parallel:** P07–P10, P12 may proceed concurrently. ## Acceptance Criteria These criteria remain the bar for closing this phase. The per-AC disposition markers below -record honest current state as of the #70 merge; they do not lower the bar. +record honest current state after the #70 merge and the later 2026-07-06 follow-up capture; they +do not lower the bar for the stronger probes. 1. A committed, dated EVRUN-full evidence record (or record set) exists under `docs/design/evidence/`, indexed, convention-complete (versions, hashes, Limitations, redaction statement, ID citations). - **Disposition at #70 merge: not met.** What was delivered instead is the blocked-attempt - record `docs/design/evidence/2026-07-06-evrun-full-capture-attempt.md`. A full capture - re-attempt is still owed before this criterion is met. + **Current disposition: met for the combined smoke record.** + `docs/design/evidence/2026-07-06-evrun-full-smoke.md` records the successful combined path. + `docs/design/evidence/2026-07-06-evrun-full-capture-attempt.md` remains as provenance for the + earlier blocked attempt. 2. The record demonstrates: real Codex editing through the owned app-server transport; real, exercised confinement with honest strength; an adversarial no-phone-home observation; and multi-run idempotency against a real landed effect. - **Disposition at #70 merge: not met.** The combined EVRUN-full path could not run because - the sandbox GitHub and integrity prerequisites were absent; the blocked-attempt record - `docs/design/evidence/2026-07-06-evrun-full-capture-attempt.md` documents what was observed - instead. A full capture re-attempt remains required before this criterion is met. + **Current disposition: partially met.** The combined smoke demonstrates real Codex editing + through the owned app-server transport, real macOS execution-host attestation with honest + `weak` strength, real GitHub Issues / Forge effects, records integrity, and redaction. The + adversarial no-phone-home observation remains `ambiguous`, and multi-run idempotency against a + repeated real landed effect remains unproven. 3. Hosted/remote operation and Windows behavior are explicitly restated as out of scope in the Limitations section. 4. README/AGENTS/product status lines about EVRUN reflect the new boundary — no claim exceeds diff --git a/docs/delivery/target-state-implementation/verification.md b/docs/delivery/target-state-implementation/verification.md index 057284b..d1faaf2 100644 --- a/docs/delivery/target-state-implementation/verification.md +++ b/docs/delivery/target-state-implementation/verification.md @@ -106,13 +106,13 @@ byte-level compatibility surface for the observability-records v0 contract. dated filename and header, exact external tool versions, content hashes for captured transcripts, a required `Limitations` section, an explicit redaction statement, and citations to the guarantee/invariant/ADR IDs the evidence supports. -- **EVRUN-partial** is the committed baseline. **EVRUN-full** (P11) must prove at minimum: real - Codex editing through the app-server transport, real execution-host confinement, adversarial - no-phone-home behavior, and multi-run idempotency — four of the six gaps the - [evidence boundary](../../design/evidence/README.md#evrun-evidence-boundary) names. The - remaining two — hosted or remote operation and Windows behavior — stay explicitly out of this - track (remote is a product deferral; Windows gates on `N1A-P14`); that grouping is this - track's scoping decision, not the evidence doc's. +- **EVRUN-partial** is the committed baseline. **EVRUN-full** (P11) now includes a combined smoke + proving real Codex editing through the app-server transport, real macOS execution-host + attestation, real GitHub effects, records integrity, and redaction. The remaining P11 evidence + gates are adversarial no-phone-home behavior and multi-run idempotency against a repeated real + effect. Hosted or remote operation and Windows behavior stay explicitly out of this track + (remote is a product deferral; Windows gates on `N1A-P14`); that grouping is this track's + scoping decision, not the evidence doc's. - Evidence is host- and version-pinned, and is an input to decisions, not authority. P13's freeze package cites evidence records; the contract owner decides. - For any phase whose acceptance criteria claim real-provider behavior (P03, P04, P05, P11), the @@ -222,8 +222,9 @@ The track is delivered when all of the following hold, with evidence: 1. Every phase P01–P14 is merged with its acceptance criteria checked off in review. 2. `pnpm check` is green on `main` with the P02 package layout and boundary enforcement in the gate. -3. EVRUN-full is committed under `docs/design/evidence/` and indexed, with its limitations - honestly stated. +3. EVRUN-full combined-smoke evidence is committed under `docs/design/evidence/` and indexed, with + limitations honestly stated and remaining no-phone-home/idempotency gates either evidenced or + explicitly deferred. 4. The P14 guarantee-coverage audit maps every product ID in [`guarantees.md`](../../product/guarantees.md) to shipped behavior plus tests, or to an explicitly recorded deferral (remote hosts, triggers, Windows, ecosystem distribution). diff --git a/docs/design/README.md b/docs/design/README.md index 49feee8..28f7740 100644 --- a/docs/design/README.md +++ b/docs/design/README.md @@ -34,7 +34,7 @@ decision records and reference material. | Core data ports | [`core/plan-intake.md`](./core/plan-intake.md), [`core/records.md`](./core/records.md) | **draft** | implementation planning / later core-parts pass | | Contracts overview | `contracts/README.md` | overview | — | | Data contracts | `contracts/{execution-plan, observability-records}-contract-v0.md` | contract v0 | field-level schema (intentionally not frozen) | -| Driving / providers | [`contracts/driving.md`](./contracts/driving.md), [`contracts/providers.md`](./contracts/providers.md) | **draft** | package implementation / EVRUN-full evidence gates | +| Driving / providers | [`contracts/driving.md`](./contracts/driving.md), [`contracts/providers.md`](./contracts/providers.md) | **draft** | package implementation / remaining EVRUN evidence gates | | Provider realization roadmap | [`contracts/provider-realization-roadmap.md`](./contracts/provider-realization-roadmap.md) | roadmap | grows as later provider-realization phases land | | Security model | [`security-model.md`](./security-model.md) | **draft** | grows as new controls are designed | | Decisions | `decisions/*` | log | grows as decisions are made | @@ -157,6 +157,6 @@ already complete. - implementation of the ADR 0027 package split, including package files, export maps, project references, source moves, dependency rules, and publishing posture; - provider package publication or a third-party provider ecosystem commitment; -- EVRUN-full, prompt-size / bounded-context behavior, Windows / Git Bash support, and other evidence - gates that ADR 0028 leaves open; +- remaining EVRUN no-phone-home/idempotency, prompt-size / bounded-context behavior, Windows / + Git Bash support, and other evidence gates that ADR 0028 leaves open; - the implementation code itself. diff --git a/docs/design/evidence/2026-07-06-evrun-full-smoke.md b/docs/design/evidence/2026-07-06-evrun-full-smoke.md new file mode 100644 index 0000000..147d450 --- /dev/null +++ b/docs/design/evidence/2026-07-06-evrun-full-smoke.md @@ -0,0 +1,133 @@ +--- +title: "2026-07-06 - EVRUN-full combined smoke" +status: recorded - combined real-provider path passed; stronger probes remain open +date: 2026-07-06 +--- + +# 2026-07-06 - EVRUN-full combined smoke + +## Probe + +This record captures the first successful combined EVRUN-full smoke path in the disposable +repository `agentic-workflow-kit/jig-smoke-target`: GitHub Issues work source, real Codex +app-server agent leg, macOS real execution-host attestation, real GitHub Forge `open-pr` +landing, records integrity, and redaction checks in one operator-initiated run. + +The run used the smoke scenario added on branch `evidence/p11-evrun-full-capture` at +`packages/jig-sdk/tests/smoke/evrun-full.smoke.test.ts`. The base checkout before this branch's +changes was `dcffa56d5059414f50ba32c1d23f225b96b8698c`; the smoke harness itself is part of this +evidence PR. The disposable seed issue was +`https://github.com/agentic-workflow-kit/jig-smoke-target/issues/13`. + +## Tool Version Pins + +| Tool | Version | +| ----------------- | ---------------------------------------------- | +| Node.js | `v26.4.0` | +| pnpm | `11.9.0` | +| GitHub CLI (`gh`) | `2.87.2 (2026-02-20)` | +| Git | `2.53.0` | +| Codex CLI | `codex-cli 0.142.5` | +| Host | `Darwin 25.5.0 RELEASE_ARM64_T6000 arm64` | +| Jig base checkout | `dcffa56d5059414f50ba32c1d23f225b96b8698c` | +| Jig smoke harness | branch `evidence/p11-evrun-full-capture` | +| Git transport | HTTPS with token-backed environment Git config | + +## Invocation + +The operator supplied GitHub credentials and the records-integrity key via environment variables. +Secret values were not printed. + +```bash +GITHUB_TOKEN="$(gh auth token)" \ +GH_TOKEN="$(gh auth token)" \ +EVRUN_FULL_SMOKE=1 \ +EVRUN_FULL_KEEP_ARTIFACTS=1 \ +JIG_GITHUB_ISSUES_REPOSITORY=agentic-workflow-kit/jig-smoke-target \ +JIG_GITHUB_ISSUES_LABEL=jig-candidate \ +JIG_RECORDS_INTEGRITY_KEY="" \ +JIG_RECORDS_INTEGRITY_KEY_ID="p11-evrun-full-capture-" \ +EVRUN_FULL_EVIDENCE_OUT=/private/tmp/jig-p11-evrun-full-evidence.json \ +corepack pnpm exec vitest run packages/jig-sdk/tests/smoke/evrun-full.smoke.test.ts +``` + +Result: + +```text +Test Files 1 passed (1) +Tests 1 passed (1) +Duration 66.62s +``` + +## Result Summary + +| Field | Value | +| ---------------- | ------------------------------------------------------------------------------------------ | +| Run ID | `run-evrun-full-smoke-1783351643050-94edb35a-5f62-4315-a4b9-79320e60beb8` | +| Run status | `success` | +| Drivers | `agent: codex`, `executionHost: real`, `forge: github`, `workSource: github-issues` | +| Smoke PR | `https://github.com/agentic-workflow-kit/jig-smoke-target/pull/17` (closed during cleanup) | +| Landing outcome | `opened-pr` | +| Landing family | `runner-action.opened-pr` | +| Integrity status | `verified` | +| `run.json` hash | `30d8ada63f99ef1298e2bed78789f89a4f5eca658f5bd799b3f642783f280a88` | +| `integrity` hash | `62a1e7795d880e2057e80b80f6d20a9fa54abfcdc226f46a217626f81a7f7898` | + +## Observed Codex Leg + +The Codex app-server observation reported `driver: codex-app-server`, `cliVersion: 0.142.5`, +`threadResumed: false`, and `finalTurnStatus: completed`. + +Codex completed these story commands, including the real edit, commit, and token-backed HTTPS push: + +| Command family | Outcome | +| ------------------------------ | ----------- | +| `git rev-parse --abbrev-ref` | `completed` | +| `git add evrun-full-smoke.txt` | `completed` | +| `git commit -m ...` | `completed` | +| `git push -u ...` | `completed` | + +The adapter relayed and recorded owner approvals for `git add`, `git commit`, and `git push`. +The smoke then verified that the remote branch head matched the local Codex-authored commit before +accepting the GitHub `open-pr` landing evidence. + +## Host Attestation + +| Field | Value | +| -------------------------------- | ----------------- | +| Driver ID | `real-host` | +| Run context | `local-real-host` | +| Positive | `true` | +| Containment mechanism | `process-group` | +| Reported isolation strength | `weak` | +| Proven isolation strength | `weak` | +| Negative egress observed outcome | `ambiguous` | + +## Redaction + +The emitted evidence facts and generated run artifacts were checked for GitHub tokens, `GH_TOKEN`, +`GITHUB_TOKEN`, Git askpass paths, credential URLs, and the records-integrity key. No secret values +are committed in this record. The raw local run directory remains outside the repository under the +operator temp directory and is not a durable source of truth. + +## Supported IDs + +- P11-AC-1 is now satisfied for the committed evidence-record requirement: this dated record is + indexed, includes version pins, hashes, redaction posture, and limitations. +- P11-AC-2 is partially satisfied: this run demonstrates real Codex editing through the owned + app-server transport, real macOS execution-host attestation with honest `weak` strength, real + GitHub Issues / Forge effects, records integrity, and redaction on the combined path. +- `MERGE-2` and `FENCE-3` gain a real combined-path `open-pr` observation, but held-merge replay + and block-surfacing legs remain covered by unit or narrower smoke evidence. +- `SEC-2` is not closed by this record because the no-phone-home observation is `ambiguous`, not a + strong adversarial proof. +- `RESUME-3` is not closed by this record because multi-run idempotency against the same real + landed effect was not exercised. + +## Limitations + +This is a successful combined EVRUN-full smoke, not a complete closure of every P11 hard probe. It +does not prove strong no-phone-home behavior, multi-run idempotency against a repeated real effect, +held-merge replay, hosted/remote execution, Windows behavior, or a managed daemon posture. The +landing was intentionally limited to opening and then closing a disposable PR in +`agentic-workflow-kit/jig-smoke-target`; no production repository was touched. diff --git a/docs/design/evidence/README.md b/docs/design/evidence/README.md index acb814e..d026004 100644 --- a/docs/design/evidence/README.md +++ b/docs/design/evidence/README.md @@ -17,17 +17,18 @@ guarantee, invariant, or decision IDs the evidence supports. ## Records -| Date | Record | Purpose | -| ---------- | --------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| 2026-07-04 | [N1A-P01/P02/P03 app-server surface, readiness, and parentage](./2026-07-04-n1a-p01-p02-p03-surface-readiness-parentage.md) | Codex app-server schema, stdio readiness, managed-daemon availability, and owned-process parentage evidence for N1b. | -| 2026-07-04 | [N1A-P04/P05 turn lifecycle and tool events](./2026-07-04-n1a-p04-p05-turn-lifecycle-tool-events.md) | Basic harmless turn lifecycle and structured command execution event evidence for N1b. | -| 2026-07-04 | [N1A-P06/P07 approval relay and denial](./2026-07-04-n1a-p06-p07-approval-denial.md) | Structured approval request, accept, decline, and denial-state evidence for N1b. | -| 2026-07-04 | [N1A-P08 interrupt delivery](./2026-07-04-n1a-p08-interrupt.md) | Active-turn interrupt request and terminal interrupted-state evidence for N1b. | -| 2026-07-04 | [N1A-P09/P10 busy behavior and cleanup](./2026-07-04-n1a-p09-p10-busy-cleanup.md) | Overlapping turn-start behavior, malformed request handling, and restart cleanup evidence for N1b. | -| 2026-07-04 | [N1A-P11/P12 resume and durability](./2026-07-04-n1a-p11-p12-resume-durability.md) | Persistent thread resume and durable app-server session-state evidence for N1b. | -| 2026-07-04 | [EVRUN partial real-provider smoke evidence](./2026-07-04-evrun-partial-smoke.md) | Applied evidence for one `work-source -> forge -> records-integrity` smoke run against the disposable sandbox, with a scripted/injected agent leg and explicit EVRUN-full limitations. | -| 2026-07-06 | [EVRUN-full capture attempt](./2026-07-06-evrun-full-capture-attempt.md) | P11 capture attempt showing successful narrow Codex app-server and real-host smokes, plus the exact missing-prerequisite blocker that keeps EVRUN-full unproven. | -| 2026-07-06 | [P05 real Forge smoke rerun](./2026-07-06-p05-real-forge-smoke-rerun.md) | Current-checkout rerun of the real GitHub Issues -> GitHub Forge `open-pr` smoke against the disposable sandbox, with records integrity verified and explicit MERGE-5 limitations. | +| Date | Record | Purpose | +| ---------- | --------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 2026-07-04 | [N1A-P01/P02/P03 app-server surface, readiness, and parentage](./2026-07-04-n1a-p01-p02-p03-surface-readiness-parentage.md) | Codex app-server schema, stdio readiness, managed-daemon availability, and owned-process parentage evidence for N1b. | +| 2026-07-04 | [N1A-P04/P05 turn lifecycle and tool events](./2026-07-04-n1a-p04-p05-turn-lifecycle-tool-events.md) | Basic harmless turn lifecycle and structured command execution event evidence for N1b. | +| 2026-07-04 | [N1A-P06/P07 approval relay and denial](./2026-07-04-n1a-p06-p07-approval-denial.md) | Structured approval request, accept, decline, and denial-state evidence for N1b. | +| 2026-07-04 | [N1A-P08 interrupt delivery](./2026-07-04-n1a-p08-interrupt.md) | Active-turn interrupt request and terminal interrupted-state evidence for N1b. | +| 2026-07-04 | [N1A-P09/P10 busy behavior and cleanup](./2026-07-04-n1a-p09-p10-busy-cleanup.md) | Overlapping turn-start behavior, malformed request handling, and restart cleanup evidence for N1b. | +| 2026-07-04 | [N1A-P11/P12 resume and durability](./2026-07-04-n1a-p11-p12-resume-durability.md) | Persistent thread resume and durable app-server session-state evidence for N1b. | +| 2026-07-04 | [EVRUN partial real-provider smoke evidence](./2026-07-04-evrun-partial-smoke.md) | Applied evidence for one `work-source -> forge -> records-integrity` smoke run against the disposable sandbox, with a scripted/injected agent leg and explicit EVRUN-full limitations. | +| 2026-07-06 | [EVRUN-full capture attempt](./2026-07-06-evrun-full-capture-attempt.md) | P11 capture attempt showing successful narrow Codex app-server and real-host smokes, plus the exact missing-prerequisite blocker that kept the combined smoke unproven at that point. | +| 2026-07-06 | [EVRUN-full combined smoke](./2026-07-06-evrun-full-smoke.md) | Successful combined GitHub Issues -> real Codex app-server -> real host -> GitHub Forge `open-pr` -> records-integrity smoke, with stronger no-phone-home/idempotency probes still open. | +| 2026-07-06 | [P05 real Forge smoke rerun](./2026-07-06-p05-real-forge-smoke-rerun.md) | Current-checkout rerun of the real GitHub Issues -> GitHub Forge `open-pr` smoke against the disposable sandbox, with records integrity verified and explicit MERGE-5 limitations. | ## N1a deferred probes @@ -43,8 +44,8 @@ guarantee, invariant, or decision IDs the evidence supports. ## EVRUN evidence boundary The EVRUN-partial record was enough for the org M7 exit decision, but it is not EVRUN-full. The -2026-07-06 P11 capture attempt refreshes the narrow Codex app-server and macOS real-host smoke -evidence, but it still does not prove the combined real Codex, real GitHub, real execution-host, -integrity, adversarial no-phone-home, and multi-run idempotency path. EVRUN-full remains unproven -until the missing sandbox credentials/configuration are supplied and the combined scenario runs. -Hosted/remote operation and Windows behavior also remain unproven. +2026-07-06 P11 capture attempt records the original missing-prerequisite blocker. The later +2026-07-06 EVRUN-full combined smoke proves the combined real Codex, real GitHub, real +execution-host, records-integrity, and redaction path for an operator-initiated disposable +`open-pr` run. Strong adversarial no-phone-home evidence, multi-run idempotency against a repeated +real effect, hosted/remote operation, and Windows behavior remain unproven. diff --git a/docs/design/security-model.md b/docs/design/security-model.md index d0f527e..5b3ff43 100644 --- a/docs/design/security-model.md +++ b/docs/design/security-model.md @@ -324,10 +324,11 @@ owner assembles is automatically safe. These are existing evidence gaps named in the source docs, not new questions invented for this consolidation: -- **EVRUN-full** is not yet closed. The current evidence is EVRUN-partial: a scripted/injected agent - leg against a disposable sandbox, which does not prove real Codex editing, real execution-host - confinement, or adversarial no-phone-home behavior - ([`evidence/README.md`](./evidence/README.md); [`evidence/2026-07-04-evrun-partial-smoke.md`](./evidence/2026-07-04-evrun-partial-smoke.md)). +- **EVRUN-full is only partially closed.** EVRUN-partial remains the M7 baseline, and the later + EVRUN-full combined smoke proves the real Codex / real-host / real GitHub `open-pr` path against + a disposable sandbox. It still does not prove adversarial no-phone-home behavior or multi-run + idempotency against a repeated real effect ([`evidence/README.md`](./evidence/README.md); + [`evidence/2026-07-06-evrun-full-smoke.md`](./evidence/2026-07-06-evrun-full-smoke.md)). - **No-phone-home is not yet adversarially probed.** The design requires proven, not asserted, confinement, but red-team adversarial probing of the no-phone-home boundary remains later integration work, not yet exercised ([`contracts/providers.md`](./contracts/providers.md) diff --git a/docs/product/jig.md b/docs/product/jig.md index 7e0ad15..734cc7a 100644 --- a/docs/product/jig.md +++ b/docs/product/jig.md @@ -196,12 +196,12 @@ Jig is honest about its edges. These are deliberate non-goals or deferrals, not - **Operator-initiated.** A run starts because you start it; webhook and scheduler triggers come later. - **A tool you run, not a service you buy.** Jig is not a hosted, multi-tenant service in v1. -- **The full agent-driven real run is still debt.** Current evidence proves a real work-source, - Forge, and records-integrity path with a scripted agent leg. That scoped proof is - EVRUN-partial. A later P11 capture attempt refreshed narrow Codex app-server and real-host smoke - evidence but could not run the combined GitHub path without sandbox credentials and an integrity - key. EVRUN-full — the full Codex-driven agent leg — remains unproven, as do remote execution, - no-phone-home behavior, and every transport edge. +- **The full agent-driven real run is still bounded evidence.** EVRUN-partial proved a real + work-source, Forge, and records-integrity path with a scripted agent leg. The later EVRUN-full + combined smoke proves a real GitHub Issues, Codex app-server, real-host, GitHub Forge `open-pr`, + and records-integrity path against the disposable target repo. Remote execution, strong + no-phone-home behavior, held-merge/idempotency, Windows behavior, and every transport edge remain + unproven. - **No silent legacy coping.** Jig refuses configuration it doesn't understand, with guidance, rather than guessing at an outdated format. diff --git a/packages/jig-sdk/tests/evrun-smoke.unit.test.ts b/packages/jig-sdk/tests/evrun-smoke.unit.test.ts new file mode 100644 index 0000000..3b4bdee --- /dev/null +++ b/packages/jig-sdk/tests/evrun-smoke.unit.test.ts @@ -0,0 +1,139 @@ +import assert from 'node:assert'; +import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { afterEach, test } from 'vitest'; +import type { RunRecord } from '../src/types.js'; +import { buildEvidenceFacts, emitEvidenceFacts } from './smoke/evrun-smoke.js'; + +const cleanupDirs: string[] = []; + +afterEach(() => { + for (const dir of cleanupDirs.splice(0)) { + rmSync(dir, { recursive: true, force: true }); + } +}); + +test('P11-AC: EVRUN smoke evidence facts include real-host and Codex observations without secrets', () => { + const runDir = mkdtempSync(join(tmpdir(), 'jig-evrun-evidence-facts-')); + cleanupDirs.push(runDir); + + writeFileSync( + join(runDir, 'attestation.snapshot.json'), + JSON.stringify( + { + driverId: 'real-host', + runContext: 'local-real-host', + positive: true, + containmentMechanism: 'process-group', + reportedIsolationStrength: 'weak', + provenIsolationStrength: 'weak', + negativeEgressObservedOutcome: 'open', + }, + null, + 2, + ), + ); + + const record: RunRecord = { + run: { + id: 'run-evrun-full-smoke', + attempt: 1, + status: 'success', + planId: 'evrun-full-smoke', + binding: { + policyRef: 'smoke-policy', + configRef: 'mode=local-real-provider-smoke;recordDir=runs', + workspace: { + kind: 'unavailable', + reason: 'not-a-git-worktree', + detail: 'test fixture', + }, + drivers: { + agent: 'codex', + executionHost: 'real', + forge: 'github', + workSource: 'github-issues', + }, + }, + attestationSnapshot: { + ref: 'record-artifact:run-evrun-full-smoke/attestation.snapshot.json', + path: join(runDir, 'attestation.snapshot.json'), + }, + }, + events: [ + { + family: 'evidence.modeled', + storyId: 'EVRUN-FULL-SMOKE', + result: 'passed', + observed: { + driver: 'codex-app-server', + cliVersion: 'codex-cli 0.142.5', + threadResumed: false, + finalTurnStatus: 'completed', + approvals: [ + { + decision: 'approved', + command: 'git rev-parse --abbrev-ref HEAD', + }, + ], + commands: [ + { + command: 'cat evrun-full-smoke.txt', + status: 'completed', + exitCode: 0, + }, + ], + }, + }, + { + family: 'runner-action.pushed', + storyId: 'EVRUN-FULL-SMOKE', + targetRef: 'refs/heads/example', + targetHead: 'abc123', + }, + { + family: 'runner-action.opened-pr', + storyId: 'EVRUN-FULL-SMOKE', + targetRef: 'refs/heads/example', + targetHead: 'abc123', + prNumber: 42, + prUrl: 'https://github.com/agentic-workflow-kit/jig-smoke-target/pull/42', + }, + ], + }; + + const runJsonBytes = Buffer.from(JSON.stringify(record)); + const integrityJsonBytes = Buffer.from(JSON.stringify({ hmac: 'abc123' })); + const facts = buildEvidenceFacts({ + runDir, + record, + integrityStatus: 'verified', + runJsonBytes, + integrityJsonBytes, + }); + + assert.deepStrictEqual(facts.driverSelection, record.run.binding.drivers); + assert.strictEqual(facts.prNumber, 42); + assert.strictEqual(facts.prUrl, 'https://github.com/agentic-workflow-kit/jig-smoke-target/pull/42'); + assert.strictEqual(facts.landingOutcome, 'opened-pr'); + assert.deepStrictEqual(facts.landingFamilies, ['runner-action.pushed', 'runner-action.opened-pr']); + assert.strictEqual(facts.hostAttestation?.driverId, 'real-host'); + assert.strictEqual(facts.hostAttestation?.negativeEgressObservedOutcome, 'open'); + assert.strictEqual(facts.codexObserved?.driver, 'codex-app-server'); + assert.strictEqual(facts.codexObserved?.cliVersion, 'codex-cli 0.142.5'); + assert.strictEqual(facts.codexObserved?.finalTurnStatus, 'completed'); + + const outPath = join(runDir, 'evidence-facts.json'); + emitEvidenceFacts(outPath, { + runDir, + record, + integrityStatus: 'verified', + runJsonBytes, + integrityJsonBytes, + }); + const serialized = readFileSync(outPath, 'utf8'); + assert.ok(serialized.includes('"runId": "run-evrun-full-smoke"')); + assert.ok(serialized.includes('"driver": "codex-app-server"')); + assert.strictEqual(serialized.includes('x-access-token'), false); +}); diff --git a/packages/jig-sdk/tests/smoke/evrun-full.smoke.test.ts b/packages/jig-sdk/tests/smoke/evrun-full.smoke.test.ts new file mode 100644 index 0000000..8ce1b1b --- /dev/null +++ b/packages/jig-sdk/tests/smoke/evrun-full.smoke.test.ts @@ -0,0 +1,278 @@ +import assert from 'node:assert'; +import { execFileSync } from 'node:child_process'; +import { randomUUID } from 'node:crypto'; +import { mkdtempSync, readFileSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { afterEach, describe, test } from 'vitest'; +import { composeReferenceRun } from '../../src/bootstrap.js'; +import { LocalHarness } from '../../src/harness.js'; +import { validateCandidate } from '../../src/intake.js'; +import { INTEGRITY_FILE, verifyIntegritySidecar } from '../../src/integrity.js'; +import { RecordManager } from '../../src/records.js'; +import type { ConfigDoc, PlanInstance, PolicyDoc } from '../../src/types.js'; +import { + emitEvidenceFacts, + REAL_LANDING_FAMILIES, + readSingleRunRecord, + SMOKE_REPOSITORY, + SMOKE_SEED_ISSUE_LABEL, +} from './evrun-smoke.js'; + +export const SMOKE_SEED_ISSUE_TITLE = 'EVRUN full smoke seed: codex real-provider run'; +export const SMOKE_CHANGED_FILE = 'evrun-full-smoke.txt'; +export const SMOKE_PLAN_INSTANCE = { + plan: { + id: 'evrun-full-smoke', + version: 'execution-plan-shape-v0', + stories: [ + { + id: 'EVRUN-FULL-SMOKE', + title: + 'In this workspace, run `git rev-parse --abbrev-ref HEAD`, create evrun-full-smoke.txt with lines "EVRUN full smoke Codex-authored landing edit", "branch=", "landing-leg=codex-authored-commit", and "agent-leg=codex-app-server", commit it with message "test: evrun full smoke codex landing edit", push the branch with upstream tracking, then reply with one short confirmation sentence mentioning the branch and file.', + scope: [SMOKE_CHANGED_FILE], + authority: { + requests: ['edit-files', 'run-checks'], + }, + }, + ], + }, +} satisfies PlanInstance; +export const SMOKE_SEED_ISSUE_BODY = `\`\`\`json\n${JSON.stringify(SMOKE_PLAN_INSTANCE, null, 2)}\n\`\`\``; +export const SMOKE_REQUIRED_ENV = { + GITHUB_TOKEN: '', + JIG_GITHUB_ISSUES_REPOSITORY: SMOKE_REPOSITORY, + JIG_GITHUB_ISSUES_LABEL: SMOKE_SEED_ISSUE_LABEL, + JIG_RECORDS_INTEGRITY_KEY: '', + JIG_RECORDS_INTEGRITY_KEY_ID: '', + EVRUN_FULL_EVIDENCE_OUT: '', +} as const; + +const cleanupDirs: string[] = []; +const GIT_CONFIG_ENV_KEYS = [ + 'GIT_CONFIG_COUNT', + 'GIT_CONFIG_KEY_0', + 'GIT_CONFIG_VALUE_0', + 'GIT_CONFIG_KEY_1', + 'GIT_CONFIG_VALUE_1', +] as const; + +afterEach(() => { + if (process.env.EVRUN_FULL_KEEP_ARTIFACTS) { + return; + } + for (const dir of cleanupDirs.splice(0)) { + rmSync(dir, { recursive: true, force: true }); + } +}); + +function execGit(args: string[], cwd: string): string { + return execFileSync('git', args, { + cwd, + encoding: 'utf8', + env: process.env, + stdio: ['ignore', 'pipe', 'pipe'], + }); +} + +function restoreEnv(snapshot: Record): void { + for (const [key, value] of Object.entries(snapshot)) { + if (value === undefined) { + delete process.env[key]; + } else { + process.env[key] = value; + } + } +} + +function gitBasicAuthHeader(token: string): string { + const encoded = Buffer.from(`x-access-token:${token}`, 'utf8').toString('base64'); + return `AUTHORIZATION: basic ${encoded}`; +} + +function configureGitHttpsToken(token: string): void { + process.env.GIT_CONFIG_COUNT = '2'; + process.env.GIT_CONFIG_KEY_0 = 'credential.helper'; + process.env.GIT_CONFIG_VALUE_0 = ''; + process.env.GIT_CONFIG_KEY_1 = 'http.https://github.com/.extraHeader'; + process.env.GIT_CONFIG_VALUE_1 = gitBasicAuthHeader(token); + process.env.GIT_TERMINAL_PROMPT = '0'; +} + +function provisionCheckout(): { checkoutDir: string; branchName: string } { + assert.ok(process.env.GITHUB_TOKEN, 'set GITHUB_TOKEN before provisioning the full smoke checkout'); + const tempRoot = mkdtempSync(join(tmpdir(), 'jig-evrun-full-smoke-')); + cleanupDirs.push(tempRoot); + const checkoutDir = join(tempRoot, 'jig-smoke-target'); + const branchName = `evrun-full-smoke-${Date.now()}-${randomUUID().slice(0, 8)}`; + + process.env.GH_TOKEN = process.env.GITHUB_TOKEN; + configureGitHttpsToken(process.env.GITHUB_TOKEN); + + execFileSync('git', ['clone', `https://github.com/${SMOKE_REPOSITORY}.git`, checkoutDir], { + encoding: 'utf8', + env: process.env, + stdio: ['ignore', 'pipe', 'pipe'], + }); + execGit(['checkout', '-b', branchName], checkoutDir); + execGit(['config', 'user.name', 'jig smoke runner'], checkoutDir); + execGit(['config', 'user.email', 'jig-smoke-runner@example.invalid'], checkoutDir); + + return { checkoutDir, branchName }; +} + +function smokeConfig(recordDir: string): ConfigDoc { + return { + runner: { + mode: 'local-real-provider-smoke', + recordDir, + }, + drivers: { + agent: 'codex', + executionHost: 'real', + forge: 'github', + workSource: 'github-issues', + }, + }; +} + +function smokePolicy(): PolicyDoc { + return { + policy: { + id: 'evrun-full-smoke-policy', + rules: { + allowLocalDryRun: true, + mergeSpectrum: 'open-pr', + }, + }, + }; +} + +describe.skipIf(!process.env.EVRUN_FULL_SMOKE)('EVRUN full real-provider smoke', () => { + test('ties codex app-server evidence to real-host attestation and a real GitHub open-pr landing', async () => { + assert.ok(process.env.GITHUB_TOKEN, 'set GITHUB_TOKEN for the sandbox GitHub smoke run'); + assert.strictEqual( + process.env.JIG_GITHUB_ISSUES_REPOSITORY, + SMOKE_REPOSITORY, + 'set JIG_GITHUB_ISSUES_REPOSITORY to the sandbox repo', + ); + assert.strictEqual( + process.env.JIG_GITHUB_ISSUES_LABEL ?? 'jig-candidate', + SMOKE_SEED_ISSUE_LABEL, + 'set JIG_GITHUB_ISSUES_LABEL to the seed issue label', + ); + assert.ok(process.env.JIG_RECORDS_INTEGRITY_KEY, 'set JIG_RECORDS_INTEGRITY_KEY for the sidecar'); + + const previousCwd = process.cwd(); + const previousGitAskPass = process.env.GIT_ASKPASS; + const previousGitPrompt = process.env.GIT_TERMINAL_PROMPT; + const previousGhToken = process.env.GH_TOKEN; + const previousGitConfig = Object.fromEntries(GIT_CONFIG_ENV_KEYS.map((key) => [key, process.env[key]])); + const recordDir = mkdtempSync(join(tmpdir(), 'jig-evrun-full-records-')); + cleanupDirs.push(recordDir); + + try { + const { checkoutDir, branchName } = provisionCheckout(); + process.chdir(checkoutDir); + const config = smokeConfig(recordDir); + const policy = smokePolicy(); + const composed = await composeReferenceRun({ + config, + planInstance: SMOKE_PLAN_INSTANCE, + scriptedOutput: {}, + ownerDecisionSource: { + decide: async () => 'approve', + }, + }); + const candidates = await composed.workSource.candidates(); + const candidate = candidates.find((item) => item.planInstance.plan.id === SMOKE_PLAN_INSTANCE.plan.id); + assert.ok(candidate, `expected a ${SMOKE_SEED_ISSUE_LABEL} issue carrying plan ${SMOKE_PLAN_INSTANCE.plan.id}`); + + const harness = new LocalHarness( + composed.agent, + new RecordManager({ + launchAttestation: composed.substrateManifest ? composed.capabilityAttestation : undefined, + substrateManifest: composed.substrateManifest, + redaction: composed.redaction, + }), + null, + { + capabilityAttestation: composed.capabilityAttestation, + forge: composed.forge, + landingAction: 'open-pr', + }, + ); + + const status = await harness.run(validateCandidate(candidate), config, policy); + assert.strictEqual(status, 'success'); + + const { runDir, record, runJsonBytes, integrityJsonBytes } = readSingleRunRecord(recordDir); + assert.strictEqual(record.run.status, 'success'); + assert.deepStrictEqual(record.run.binding.drivers, { + agent: 'codex', + executionHost: 'real', + forge: 'github', + workSource: 'github-issues', + }); + assert.ok(record.events.some((event) => REAL_LANDING_FAMILIES.has(event.family))); + const openedPr = record.events.find((event) => event.family === 'runner-action.opened-pr' && event.prUrl); + assert.ok(openedPr, 'expected an opened PR event with a URL'); + const evidenceModeled = record.events.find((event) => event.family === 'evidence.modeled'); + assert.ok(evidenceModeled, 'expected recorded Codex evidence'); + assert.strictEqual(evidenceModeled?.result, 'passed'); + assert.deepStrictEqual(evidenceModeled?.changedFiles, undefined); + const observed = + typeof evidenceModeled?.observed === 'object' && evidenceModeled.observed !== null + ? (evidenceModeled.observed as Record) + : undefined; + assert.strictEqual(typeof observed?.driver, 'string'); + assert.strictEqual(observed?.driver, 'codex-app-server'); + const integrityStatus = verifyIntegritySidecar(runDir, { + expected: true, + }).status; + assert.strictEqual(integrityStatus, 'verified'); + assert.ok(readFileSync(join(runDir, INTEGRITY_FILE), 'utf8').includes('"hmac"')); + const smokeFile = readFileSync(join(checkoutDir, SMOKE_CHANGED_FILE), 'utf8'); + assert.ok(smokeFile.includes(`branch=${branchName}`), 'Codex-authored landing file should record its branch'); + assert.ok( + smokeFile.includes('agent-leg=codex-app-server'), + 'Codex-authored landing file should identify the agent leg', + ); + assert.match( + execGit(['log', '-1', '--pretty=%s'], checkoutDir), + /^test: evrun full smoke codex landing edit/, + 'Codex-authored landing commit should be HEAD', + ); + const localHead = execGit(['rev-parse', 'HEAD'], checkoutDir).trim(); + const remoteHead = execGit(['ls-remote', '--heads', 'origin', branchName], checkoutDir).trim().split(/\s+/)[0]; + assert.strictEqual(remoteHead, localHead, 'Codex-authored landing commit should be pushed before PR creation'); + if (process.env.EVRUN_FULL_EVIDENCE_OUT) { + emitEvidenceFacts(process.env.EVRUN_FULL_EVIDENCE_OUT, { + runDir, + record, + integrityStatus, + runJsonBytes, + integrityJsonBytes, + }); + } + } finally { + process.chdir(previousCwd); + if (previousGitAskPass === undefined) { + delete process.env.GIT_ASKPASS; + } else { + process.env.GIT_ASKPASS = previousGitAskPass; + } + if (previousGitPrompt === undefined) { + delete process.env.GIT_TERMINAL_PROMPT; + } else { + process.env.GIT_TERMINAL_PROMPT = previousGitPrompt; + } + if (previousGhToken === undefined) { + delete process.env.GH_TOKEN; + } else { + process.env.GH_TOKEN = previousGhToken; + } + restoreEnv(previousGitConfig); + } + }, 300_000); +}); diff --git a/packages/jig-sdk/tests/smoke/evrun-partial.smoke.test.ts b/packages/jig-sdk/tests/smoke/evrun-partial.smoke.test.ts index 382efd5..5ee5a71 100644 --- a/packages/jig-sdk/tests/smoke/evrun-partial.smoke.test.ts +++ b/packages/jig-sdk/tests/smoke/evrun-partial.smoke.test.ts @@ -1,7 +1,7 @@ import assert from 'node:assert'; import { execFileSync } from 'node:child_process'; -import { createHash, randomUUID } from 'node:crypto'; -import { chmodSync, existsSync, mkdtempSync, readdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; +import { randomUUID } from 'node:crypto'; +import { chmodSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { afterEach, describe, test } from 'vitest'; @@ -10,11 +10,16 @@ import { LocalHarness } from '../../src/harness.js'; import { validateCandidate } from '../../src/intake.js'; import { INTEGRITY_FILE, verifyIntegritySidecar } from '../../src/integrity.js'; import { RecordManager } from '../../src/records.js'; -import type { ConfigDoc, PlanInstance, PolicyDoc, RunRecord } from '../../src/types.js'; +import type { ConfigDoc, PlanInstance, PolicyDoc } from '../../src/types.js'; +import { + emitEvidenceFacts, + REAL_LANDING_FAMILIES, + readSingleRunRecord, + SMOKE_REPOSITORY, + SMOKE_SEED_ISSUE_LABEL, +} from './evrun-smoke.js'; -export const SMOKE_REPOSITORY = 'agentic-workflow-kit/jig-smoke-target'; export const SMOKE_SEED_ISSUE_TITLE = 'EVRUN partial smoke seed: scripted real-provider run'; -export const SMOKE_SEED_ISSUE_LABEL = 'jig-candidate'; export const SMOKE_CHANGED_FILE = 'evrun-partial-smoke.txt'; export const SMOKE_PLAN_INSTANCE = { plan: { @@ -50,8 +55,6 @@ export const SMOKE_REQUIRED_ENV = { JIG_RECORDS_INTEGRITY_KEY: '', JIG_RECORDS_INTEGRITY_KEY_ID: '', } as const; - -const REAL_LANDING_FAMILIES = new Set(['runner-action.pushed', 'runner-action.opened-pr', 'runner-action.merged']); const cleanupDirs: string[] = []; afterEach(() => { @@ -147,90 +150,6 @@ function smokePolicy(): PolicyDoc { }; } -function sha256(bytes: Buffer): string { - return createHash('sha256').update(bytes).digest('hex'); -} - -function assertSerializedTokenClean(serialized: string): void { - const forbiddenValues = [ - process.env.GITHUB_TOKEN, - process.env.GH_TOKEN, - 'x-access-token', - process.env.GIT_ASKPASS, - ].filter((value): value is string => typeof value === 'string' && value.length > 0); - - for (const forbidden of forbiddenValues) { - assert.strictEqual(serialized.includes(forbidden), false, 'serialized evidence must not contain credentials'); - } - assert.doesNotMatch(serialized, /https:\/\/[^:\s/]+:[^@\s/]+@/i); -} - -function assertArtifactTokenClean(runDir: string): { - runJson: string; - integrityJson: string; - runJsonBytes: Buffer; - integrityJsonBytes: Buffer; -} { - const runJsonBytes = readFileSync(join(runDir, 'run.json')); - const runJson = runJsonBytes.toString('utf8'); - const integrityPath = join(runDir, INTEGRITY_FILE); - assert.ok(existsSync(integrityPath), `${INTEGRITY_FILE} must exist before evidence assertions`); - const integrityJsonBytes = readFileSync(integrityPath); - const integrityJson = integrityJsonBytes.toString('utf8'); - const serializedArtifacts = `${runJson}\n${integrityJson}`; - assertSerializedTokenClean(serializedArtifacts); - - return { runJson, integrityJson, runJsonBytes, integrityJsonBytes }; -} - -function readSingleRunRecord(recordDir: string): { - runDir: string; - record: RunRecord; - runJsonBytes: Buffer; - integrityJsonBytes: Buffer; -} { - const [runName] = readdirSync(recordDir); - assert.ok(runName, 'expected one generated run directory'); - const runDir = join(recordDir, runName); - const { runJson, runJsonBytes, integrityJsonBytes } = assertArtifactTokenClean(runDir); - return { - runDir, - record: JSON.parse(runJson) as RunRecord, - runJsonBytes, - integrityJsonBytes, - }; -} - -function emitEvidenceFacts( - outPath: string, - input: { - record: RunRecord; - integrityStatus: string; - runJsonBytes: Buffer; - integrityJsonBytes: Buffer; - }, -): void { - const openedPr = input.record.events.find( - (event) => event.family === 'runner-action.opened-pr' && typeof event.prUrl === 'string', - ); - assert.ok(openedPr, 'expected opened PR event before emitting evidence facts'); - const landingFamilies = [ - ...new Set(input.record.events.map((event) => event.family).filter((family) => REAL_LANDING_FAMILIES.has(family))), - ]; - const evidenceFacts = { - runId: input.record.run.id, - prUrl: openedPr.prUrl as string, - landingFamilies, - runStatus: input.record.run.status, - integrityStatus: input.integrityStatus, - runJsonSha256: sha256(input.runJsonBytes), - integrityJsonSha256: sha256(input.integrityJsonBytes), - }; - const serialized = `${JSON.stringify(evidenceFacts, null, 2)}\n`; - assertSerializedTokenClean(serialized); - writeFileSync(outPath, serialized); -} - describe.skipIf(!process.env.EVRUN_SMOKE)('EVRUN partial real-provider smoke', () => { test('work-source to real forge landing records and verifies integrity with a scripted agent leg', async () => { assert.ok(process.env.GITHUB_TOKEN, 'set GITHUB_TOKEN for the sandbox GitHub smoke run'); @@ -289,11 +208,14 @@ describe.skipIf(!process.env.EVRUN_SMOKE)('EVRUN partial real-provider smoke', ( assert.strictEqual(record.run.status, 'success'); assert.ok(record.events.some((event) => REAL_LANDING_FAMILIES.has(event.family))); assert.ok(record.events.some((event) => event.family === 'runner-action.opened-pr' && event.prUrl)); - const integrityStatus = verifyIntegritySidecar(runDir, { expected: true }).status; + const integrityStatus = verifyIntegritySidecar(runDir, { + expected: true, + }).status; assert.strictEqual(integrityStatus, 'verified'); assert.ok(readFileSync(join(runDir, INTEGRITY_FILE), 'utf8').includes('"hmac"')); if (process.env.EVRUN_EVIDENCE_OUT) { emitEvidenceFacts(process.env.EVRUN_EVIDENCE_OUT, { + runDir, record, integrityStatus, runJsonBytes, diff --git a/packages/jig-sdk/tests/smoke/evrun-smoke.ts b/packages/jig-sdk/tests/smoke/evrun-smoke.ts new file mode 100644 index 0000000..284bd3b --- /dev/null +++ b/packages/jig-sdk/tests/smoke/evrun-smoke.ts @@ -0,0 +1,224 @@ +import assert from 'node:assert'; +import { createHash } from 'node:crypto'; +import { existsSync, readdirSync, readFileSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { INTEGRITY_FILE } from '../../src/integrity.js'; +import type { CapabilityAttestation } from '../../src/ports.js'; +import type { RunEvent, RunRecord } from '../../src/types.js'; + +export const SMOKE_REPOSITORY = 'agentic-workflow-kit/jig-smoke-target'; +export const SMOKE_SEED_ISSUE_LABEL = 'jig-candidate'; +export const REAL_LANDING_FAMILIES = new Set([ + 'runner-action.pushed', + 'runner-action.opened-pr', + 'runner-action.merged', +]); + +export interface RunArtifacts { + runDir: string; + record: RunRecord; + runJsonBytes: Buffer; + integrityJsonBytes: Buffer; +} + +export interface EvRunEvidenceFacts { + runId: string; + runStatus: string; + driverSelection?: RunRecord['run']['binding']['drivers']; + prUrl: string; + prNumber?: number; + landingOutcome: 'opened-pr' | 'merged' | 'held'; + landingFamilies: string[]; + integrityStatus: string; + runJsonSha256: string; + integrityJsonSha256: string; + hostAttestation?: { + driverId: string; + runContext: string; + positive: boolean; + containmentMechanism?: string; + reportedIsolationStrength?: string; + provenIsolationStrength?: string; + negativeEgressObservedOutcome?: string; + failureToken?: string; + }; + codexObserved?: { + driver?: string; + cliVersion?: string; + threadResumed?: boolean; + finalTurnStatus?: string; + approvals?: unknown[]; + commands?: unknown[]; + }; +} + +function sha256(bytes: Buffer): string { + return createHash('sha256').update(bytes).digest('hex'); +} + +function gitConfigValues(env: NodeJS.ProcessEnv = process.env): string[] { + const count = Number(env.GIT_CONFIG_COUNT ?? 0); + if (!Number.isInteger(count) || count <= 0) { + return []; + } + + return Array.from({ length: count }, (_, index) => env[`GIT_CONFIG_VALUE_${index}`]).filter( + (value): value is string => typeof value === 'string' && value.length > 0, + ); +} + +export function assertSerializedTokenClean(serialized: string): void { + const forbiddenValues = [ + process.env.GITHUB_TOKEN, + process.env.GH_TOKEN, + process.env.JIG_RECORDS_INTEGRITY_KEY, + ...gitConfigValues(), + 'x-access-token', + process.env.GIT_ASKPASS, + ].filter((value): value is string => typeof value === 'string' && value.length > 0); + + for (const forbidden of forbiddenValues) { + assert.strictEqual(serialized.includes(forbidden), false, 'serialized evidence must not contain credentials'); + } + assert.doesNotMatch(serialized, /https:\/\/[^:\s/]+:[^@\s/]+@/i); +} + +function assertArtifactTokenClean(runDir: string): { + runJson: string; + runJsonBytes: Buffer; + integrityJsonBytes: Buffer; +} { + const runJsonBytes = readFileSync(join(runDir, 'run.json')); + const runJson = runJsonBytes.toString('utf8'); + const integrityPath = join(runDir, INTEGRITY_FILE); + assert.ok(existsSync(integrityPath), `${INTEGRITY_FILE} must exist before evidence assertions`); + const integrityJsonBytes = readFileSync(integrityPath); + assertSerializedTokenClean(`${runJson}\n${integrityJsonBytes.toString('utf8')}`); + return { runJson, runJsonBytes, integrityJsonBytes }; +} + +export function readSingleRunRecord(recordDir: string): RunArtifacts { + const [runName] = readdirSync(recordDir); + assert.ok(runName, 'expected one generated run directory'); + const runDir = join(recordDir, runName); + const { runJson, runJsonBytes, integrityJsonBytes } = assertArtifactTokenClean(runDir); + return { + runDir, + record: JSON.parse(runJson) as RunRecord, + runJsonBytes, + integrityJsonBytes, + }; +} + +function readHostAttestation(runDir: string, record: RunRecord): CapabilityAttestation | undefined { + const attestationPath = record.run.attestationSnapshot?.path; + if (!attestationPath) { + return undefined; + } + const resolved = attestationPath.startsWith(runDir) ? attestationPath : join(runDir, 'attestation.snapshot.json'); + if (!existsSync(resolved)) { + return undefined; + } + return JSON.parse(readFileSync(resolved, 'utf8')) as CapabilityAttestation; +} + +function latestEvidenceModeled(record: RunRecord): RunEvent | undefined { + return [...record.events].reverse().find((event) => event.family === 'evidence.modeled'); +} + +function asObservedRecord(value: unknown): Record | undefined { + return typeof value === 'object' && value !== null && !Array.isArray(value) + ? (value as Record) + : undefined; +} + +export function buildEvidenceFacts(input: { + runDir: string; + record: RunRecord; + integrityStatus: string; + runJsonBytes: Buffer; + integrityJsonBytes: Buffer; +}): EvRunEvidenceFacts { + const openedPr = input.record.events.find( + (event) => event.family === 'runner-action.opened-pr' && typeof event.prUrl === 'string', + ); + assert.ok(openedPr, 'expected opened PR event before emitting evidence facts'); + + const landingFamilies = [ + ...new Set(input.record.events.map((event) => event.family).filter((family) => REAL_LANDING_FAMILIES.has(family))), + ]; + const heldOutcome = input.record.events.find( + (event) => + event.family === 'story.done' && typeof event.mergeability === 'string' && typeof event.targetHead === 'string', + ); + const hostAttestation = readHostAttestation(input.runDir, input.record); + const modeledEvidence = latestEvidenceModeled(input.record); + const observed = asObservedRecord(modeledEvidence?.observed); + const codexObserved = observed + ? { + driver: typeof observed.driver === 'string' ? observed.driver : undefined, + cliVersion: typeof observed.cliVersion === 'string' ? observed.cliVersion : undefined, + threadResumed: typeof observed.threadResumed === 'boolean' ? observed.threadResumed : undefined, + finalTurnStatus: typeof observed.finalTurnStatus === 'string' ? observed.finalTurnStatus : undefined, + approvals: Array.isArray(observed.approvals) ? observed.approvals : undefined, + commands: Array.isArray(observed.commands) ? observed.commands : undefined, + } + : undefined; + + return { + runId: input.record.run.id, + runStatus: input.record.run.status, + driverSelection: input.record.run.binding.drivers, + prUrl: openedPr.prUrl as string, + ...(typeof openedPr.prNumber === 'number' ? { prNumber: openedPr.prNumber } : {}), + landingOutcome: heldOutcome ? 'held' : landingFamilies.includes('runner-action.merged') ? 'merged' : 'opened-pr', + landingFamilies, + integrityStatus: input.integrityStatus, + runJsonSha256: sha256(input.runJsonBytes), + integrityJsonSha256: sha256(input.integrityJsonBytes), + ...(hostAttestation + ? { + hostAttestation: { + driverId: hostAttestation.driverId, + runContext: hostAttestation.runContext, + positive: hostAttestation.positive, + ...(hostAttestation.containmentMechanism + ? { containmentMechanism: hostAttestation.containmentMechanism } + : {}), + ...(hostAttestation.reportedIsolationStrength + ? { + reportedIsolationStrength: hostAttestation.reportedIsolationStrength, + } + : {}), + ...(hostAttestation.provenIsolationStrength + ? { + provenIsolationStrength: hostAttestation.provenIsolationStrength, + } + : {}), + ...(hostAttestation.negativeEgressObservedOutcome + ? { + negativeEgressObservedOutcome: hostAttestation.negativeEgressObservedOutcome, + } + : {}), + ...(hostAttestation.failureToken ? { failureToken: hostAttestation.failureToken } : {}), + }, + } + : {}), + ...(codexObserved ? { codexObserved } : {}), + }; +} + +export function emitEvidenceFacts( + outPath: string, + input: { + runDir: string; + record: RunRecord; + integrityStatus: string; + runJsonBytes: Buffer; + integrityJsonBytes: Buffer; + }, +): void { + const serialized = `${JSON.stringify(buildEvidenceFacts(input), null, 2)}\n`; + assertSerializedTokenClean(serialized); + writeFileSync(outPath, serialized); +}