diff --git a/docs/plans/2026-05-24-verify-capabilities-design.md b/docs/plans/2026-05-24-verify-capabilities-design.md new file mode 100644 index 00000000..42da43a8 --- /dev/null +++ b/docs/plans/2026-05-24-verify-capabilities-design.md @@ -0,0 +1,232 @@ +# `wfctl plugin verify-capabilities` Design + +**Issue:** [workflow#765](https://github.com/GoCodeAlone/workflow/issues/765) +**Status:** Revised 2026-05-24 (cycle 3 — scope-down per cycle-2 review) — awaiting re-review +**Author:** Jon Langevin +**Parent contract:** workflow#762 (plugin-version contract); workflow#764 (Layer 3b 71-PR sweep) + +## Revision history + +- **Cycle 1**: initial design. FAILED — 3 Critical (diff fields not on wire; IaC bridge Unimplemented; handshake path wrong). +- **Cycle 2**: pivot to Manifest scalars + ContractRegistry. FAILED — 2 Critical (plugin.json has no iacResources key; BuildContractRegistry returns ALL services including infra-internal noise). +- **Cycle 3**: scope-down per reviewer Option 2. Drop contract-diff entirely. FAILED — 2 Critical (isSentinel() missed `"dev"` form per SDK sentinel set; cited wrong fixture precedent — `validate-contract` is pure-static, never compiles fixtures). +- **Cycle 4**: fix isSentinel() superset; cite correct precedent; preflight + security note; honest Surface rationale; fixture Validate() prereqs. FAILED — 3 Important (spawn-helper cut-point ambiguous; fixture go.mod relative-replace conflicts with copy-to-TempDir precedent; jq picks first binary non-deterministic on multiarch). +- **Cycle 5**: cut-point at line 504; in-place fixture build; jq filter pins goos+goarch. FAILED — 3 Important (jq filter fix not propagated to §CI integration block; in-place build omits go.sum handling; cut-point missed `defer client.Kill()` transform). +- **Cycle 6** (this version): propagate jq filter to §CI integration; pin `-mod=readonly` + checked-in go.sum (sidesteps both pollution and copy-to-TempDir overhead); explicit defer-transform note in cut-point; reconcile line-range citations (462-504 throughout). + +## Problem + +`wfctl plugin validate-contract` is a SOURCE-tree static analysis pass. It verifies the source tree but cannot verify the BINARY actually surfaces what plugin.json declares at runtime. Workflow#764's Layer 3b sweep wired all 64 plugin repos with `sdk.WithBuildVersion + sdk.ResolveBuildVersion + ldflag`. But nothing yet verifies that the SHIPPED binary's runtime `Manifest.Version` matches the build tag. + +The single load-bearing truth-loop the user named: **"binary's BuildVersion is populated and matches the release tag — proving ldflag fired during goreleaser build."** This is the bug class verify-capabilities exists to catch. + +## Solution + +New subcommand `wfctl plugin verify-capabilities` that spawns the plugin binary as a go-plugin subprocess, calls `PluginService.GetManifest`, and verifies the returned `Manifest.Version` is non-sentinel + matches plugin.json + git tag context. Scope strictly limited to fields the SDK reliably surfaces; broader contract-diff deferred to follow-up. + +### Synopsis + +``` +wfctl plugin verify-capabilities --binary +``` + +`--binary` REQUIRED (cycle-1 build-from-source dropped per reviewer Option 2; produced false-PASS in dev when ldflag paths varied per repo). + +**⚠ Security note**: verify-capabilities EXECUTES `--binary ` as a subprocess. The plugin's `main()` runs (including any code before `sdk.ServeIaCPlugin`). Only run against build artifacts you trust. Matches `plugin conformance`'s posture. + +Preflight checks performed by the subcommand before exec: +- `os.Stat(path)` — file exists, is regular (not directory/symlink to one) +- `mode & 0o111 != 0` — file is executable +- `path != "" && path != "null"` — guards against CI lookups (jq) returning empty / null + +Documented invocation: + +```bash +# Local development: +go build -ldflags="-X github.com/GoCodeAlone/workflow-plugin-/internal.Version=v1.2.3" \ + -o /tmp/p ./cmd/ +wfctl plugin verify-capabilities --binary /tmp/p . + +# CI (post-goreleaser, in release.yml): +RUNNER_ARCH=$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/') +BIN=$(jq -r --arg arch "$RUNNER_ARCH" \ + '[.[] | select(.type=="Binary" and .goos=="linux" and .goarch==$arch)] | .[0].path // ""' \ + dist/artifacts.json) +wfctl plugin verify-capabilities --binary "$BIN" . +# jq filter: +# - select(.type=="Binary" and .goos=="linux" and .goarch==$arch) +# — pick THIS RUNNER's binary; multiarch builds (amd64+arm64+darwin+...) emit +# multiple Binary artifacts; without the goos/goarch filter, `.[0]` selects +# in arbitrary goreleaser-version-dependent order — may pick darwin-arm64 +# and produce "exec format error" on a linux-amd64 runner. +# - // "" fallback to empty string on null/missing → caught by --binary preflight +``` + +### Behavior + +1. Load `/plugin.json`. Parse + run `PluginManifest.Validate()` (reuse from validate-contract). +2. Spawn `` via shared `spawnAndDial(ctx, binaryPath) (*external.PluginAdapter, func())` helper extracted from `cmd/wfctl/plugin_conformance.go:462-504` (cut-point details in §Files). Uses `external.Handshake` from `workflow/plugin/external/handshake.go:23`. +3. Call `PluginService.GetManifest(Empty) → pb.Manifest` (6 scalar fields per `plugin/external/proto/plugin.proto:96-104`). +4. Diff strict: + +| Field | Source A (plugin.json) | Source B (binary `Manifest`) | Rule | +|---|---|---|---| +| `Name` | `name` | `Name` | exact string equal; FAIL on drift | +| `Version` | `version` | `Version` | matrix below | + +**Version rule** (cycle-4, addresses F-CYCLE3-1 sentinel-set drift): + +The dev-sentinel set MUST be a SUPERSET of SDK's `ResolveBuildVersion` sentinel set `{"", "dev", "(devel)"}` (source: `plugin/external/sdk/buildversion.go:36-42`) PLUS the on-disk plugin.json sentinel `"0.0.0"` (workflow#762 convention) PLUS any string starting with `"(devel)"` (since `buildInfoVersion()` returns `"(devel) [@ [.dirty]]"`). + +``` +isSentinel(v) := v == "" || v == "dev" || v == "0.0.0" || strings.HasPrefix(v, "(devel)") +``` + +Note: `"dev"` is in the predicate because a plugin author may set `-X ...Version=dev` (or pipeline accident) and the binary's `Manifest.Version` then surfaces literal `"dev"` (since SDK only consults build-info fallback when the *declared* string matches; the ldflag-set value flows through unchanged). Without this entry, the matrix's row "0.0.0 + non-sentinel → PASS" would green-light a broken build. + +| plugin.json `version` | binary `Manifest.Version` | Outcome | Rationale | +|---|---|---|---| +| `"0.0.0"` (dev sentinel) | non-sentinel (e.g. `"v1.2.3"`) | **PASS** | binary built via ldflag-injecting CI; verify-capabilities running on a real artifact | +| `"0.0.0"` | sentinel (`""` / `"(devel)..."` / `"0.0.0"`) | **FAIL** | ldflag injection missing — the truth-loop bug verify-capabilities exists to catch | +| `"X.Y.Z"` (release) | `"vX.Y.Z"` or `"X.Y.Z"` | **PASS** | normalize: strip leading `v` from binary, then exact compare to plugin.json | +| `"X.Y.Z"` | sentinel | **FAIL** | plugin.json declares release tag but binary lacks ldflag injection | +| `"X.Y.Z"` | non-sentinel and not `X.Y.Z` | **FAIL** | drift between declared release version and shipped binary | + +5. Exit 0 on clean. Exit 1 with report: + ``` + FAIL workflow-plugin-foo (plugin.json) + error: 1 mismatch + - version: plugin.json="1.2.3"; binary Manifest.Version="(devel) [@ a1b2c3d]" + (sdk.ResolveBuildVersion returned the build-info fallback; ldflag injection missing) + ``` + +### CI integration + +Append to scaffold-template `release.yml` post-goreleaser, pre-publish: + +```yaml +- name: Verify capabilities (post-build runtime check) + run: | + RUNNER_ARCH=$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/') + BIN=$(jq -r --arg arch "$RUNNER_ARCH" \ + '[.[] | select(.type=="Binary" and .goos=="linux" and .goarch==$arch)] | .[0].path // ""' \ + dist/artifacts.json) + "${RUNNER_TEMP}/wfctl-bin/wfctl" plugin verify-capabilities --binary "$BIN" . +``` + +`dist/artifacts.json` is goreleaser's manifest of all built artifacts. The jq filter pins to the CURRENT runner's goos/goarch — multiarch builds emit multiple Binary artifacts (linux+darwin × amd64+arm64); picking arbitrarily by `.[0]` would yield e.g. darwin-arm64 on a linux-amd64 runner → `exec format error`. The `uname -m` sed map converts kernel arch names to Go arch names (`x86_64`→`amd64`, `aarch64`→`arm64`). + +Scaffold-side wiring is a follow-up commit on `scaffold-workflow-plugin` after this workflow PR lands — not part of this design's scope. + +## Files (workflow repo) + +- `cmd/wfctl/plugin_spawn.go` — NEW; extracts `spawnAndDial(ctx, binaryPath) (*external.PluginAdapter, func())` from `plugin_conformance.go:462-504` ONLY (spawn + dial + Dispense + `NewExternalPluginAdapter`). **Cut-point: line 504 (right after adapter construction succeeds).** Lines 505-513 of conformance contain IaC-specific post-dial validation (`ContractRegistryError`, `AssertIaCPluginAdvertisesRequiredService`, typed-IaC adapter setup) — these REMAIN inline in `checkTypedIaCPlugin` after calling the helper. The helper is plugin-type-agnostic; IaC-specific assertions stay where they belong. **Defer transform**: existing line 484's `defer client.Kill()` does NOT move into the helper body — instead, the helper returns the `client.Kill` (or a closure wrapping it) as the `func()` cleanup; the CALLER defers the returned cleanup. Without this transformation, a literal extraction either (a) kills the plugin when the helper returns, before the caller can RPC against it, or (b) drops the defer and leaks processes. +- `cmd/wfctl/plugin_conformance.go` — refactored to call new shared helper; behavior unchanged. +- `cmd/wfctl/plugin_verify_capabilities.go` — NEW; subcommand entry + diff impl. +- `cmd/wfctl/plugin_verify_capabilities_test.go` — table-driven tests against `testdata/verify_capabilities//`. +- `cmd/wfctl/testdata/verify_capabilities/` — NEW fixture tree. **Precedent: `plugin_conformance_test.go:buildFixtureBinary` + `testdata/conformance/iac-pass/` layout** (NOT `validate-contract`'s pattern — that one is pure-static and never compiles fixtures). Each fixture is a self-contained compilable Go module. + + Scenarios: + - `good/` — plugin.json `version="0.0.0"`, ldflag-injected binary tag `v0.1.0`. Expect PASS. + - `release-good/` — plugin.json `version="1.2.3"`, ldflag tag `v1.2.3`. Expect PASS. + - `missing-ldflag/` — plugin.json `version="0.0.0"`, no ldflag (binary surfaces sentinel `(devel) [@ sha]`). Expect FAIL. + - `version-drift/` — plugin.json `version="1.2.3"`, ldflag tag `v0.9.0`. Expect FAIL. + - `name-drift/` — plugin.json `name="foo"`, binary advertises `Name="bar"`. Expect FAIL. + + Per-fixture layout (each scenario directory): + ``` + testdata/verify_capabilities// + plugin.json # MUST satisfy PluginManifest.Validate(): + # name, version, author, description ALL required + # (per plugin/manifest.go:194-225) + main.go # minimal `sdk.Serve(stub{}, sdk.WithBuildVersion(...))` at fixture root + # (matches plugin_conformance_test.go fixture layout; NOT cmd/plugin/main.go) + go.mod # module github.com/test/ + # replace github.com/GoCodeAlone/workflow => ../../../../.. + # (5 ups: / → verify_capabilities/ → testdata/ → wfctl/ → cmd/ → REPO_ROOT) + ``` + + **Build pattern: in-place + `-mod=readonly` + CHECKED-IN go.sum**. Each fixture ships a complete checked-in `go.mod` AND `go.sum` so `go build -mod=readonly` succeeds without writing into the source tree. (Cycle-5 F5-2: `go build -mod=mod` writes `go.sum` to the fixture source tree on first build — pollutes the repo and dirties `vcs.modified=true`. The conformance precedent dodges this via copy-to-TempDir; we sidestep both via readonly + checked-in sums.) + + ```go + cmd := exec.Command("go", "build", "-mod=readonly", + "-ldflags=-X github.com/test/.Version=", + "-o", filepath.Join(t.TempDir(), "p"), ".") + cmd.Dir = "testdata/verify_capabilities/" + cmd.Env = append(os.Environ(), "GOWORK=off") + ``` + + `GOWORK=off` is mandatory — without it, the workflow repo's workspace go.work overrides the per-fixture `replace` directive and the build resolves the wrong workflow version. The `-ldflags -X` package path is `github.com/test/` (the fixture's module path; Version var at fixture root), NOT a `/internal.Version` subpath (different from production plugins; simpler test fixture). + + **Fixture-maintenance note**: when workflow SDK adds a new transitive dep that the fixtures pick up, regenerate fixture `go.sum` files via a one-shot `for d in testdata/verify_capabilities/*/; do (cd "$d" && GOWORK=off go mod tidy); done` and commit. Documented in `cmd/wfctl/testdata/verify_capabilities/README.md` (NEW; one-page maintenance note). + + Optional: factor `buildFixtureBinary` from `plugin_conformance_test.go:509-519` into a shared `cmd/wfctl/fixture_build_test.go` helper if both test files use it. Defer if duplication is minimal. +- `cmd/wfctl/plugin.go` — register `case "verify-capabilities"`. +- `docs/PLUGIN_RELEASE_GATES.md` — append `Verify-Capabilities` section. + +## Architecture choices (cycle-3) + +| Choice | Picked | Rejected (reason) | +|---|---|---| +| Surface | new subcommand | flag on validate-contract (mixes static + runtime); flag on `plugin conformance --mode manifest-verify` (TECHNICALLY VIABLE — conformance's `checkTypedIaCPlugin` uses `external.NewExternalPluginAdapter` which handles ANY plugin type, not just IaC; chose new subcommand on separation-of-concerns basis: verify-capabilities is contract truth-check, conformance is typed-IaC interface scan — distinct mental models. F-CYCLE3-4 rejection rewritten per cycle-3 review) | +| Binary source | REQUIRE `--binary` | build-from-source default — rejected cycle 2: false-PASS in dev with per-repo ldflag-path variance | +| Diff scope | Manifest.Name + Manifest.Version ONLY | + per-type RPCs (rejected cycle 2: Unimplemented in IaC bridge); + ContractRegistry (rejected cycle 3: plugin.json has no iacResources LHS + BuildContractRegistry returns infra-internal noise; defer to follow-up #766) | +| Version diff rule | sentinel-pattern matrix (`{"", "(devel)...", "0.0.0"}`) | cycle-1 "non-empty" (broke truth-loop); cycle-2 literal "0.0.0" (didn't match SDK's `(devel)` output) | +| Spawn-and-dial | extract shared helper, refactor conformance | re-implement from scratch (cycle 1 F3); leave conformance unchanged (duplicates ~200 LOC) | +| CI binary path | `jq -r '...' dist/artifacts.json` lookup | hard-coded `dist/_linux_amd64/` (cycle 2 F-NEW-6: goreleaser layout varies by arch + goamd64 level) | + +## Assumptions + +1. **`PluginService.GetManifest` exists + uniformly returns 6 scalars across all plugin types.** Verified: `/tmp/wfprobe/plugin/external/proto/plugin.proto:96-104` defines `Manifest{name, version, author, description, config_mutable, sample_category}`. Non-IaC impl at `plugin/external/sdk/grpc_server.go:148-174`. IaC bridge impl at `plugin/external/sdk/iacserver.go:301`. All plugins built with workflow v0.62.0+ serve this RPC. (Pre-v0.20 plugins predate the RPC; not in our 64-repo target set per #764 audit — all pinned to v0.62.0.) + +2. **`external.Handshake` is exported at `workflow/plugin/external/handshake.go:23`.** Verified. wfctl imports it in `plugin_conformance.go`. + +3. **`sdk.ResolveBuildVersion` sentinel set is `{"", "dev", "(devel)"}` plus the function returns `"(devel) [@ sha[.dirty]]"` from build-info fallback.** Verified at `/tmp/wfprobe/plugin/external/sdk/buildversion.go:36-42`. Diff matrix's `isSentinel()` predicate covers all SDK-emitted sentinel forms. + +4. **plugin.json `version` field is canonical authority for declared version.** Verified at `plugin/manifest.go`. Set by goreleaser before-hook at release time per workflow#762 contract. + +5. **CI runner has `jq` available.** `jq` is preinstalled on `ubuntu-latest` GitHub runners (verified standard image). Custom runners must install it. + +6. **`--binary` path points to the exact post-goreleaser binary that will publish.** Operator responsibility. Documented in §Synopsis with `jq dist/artifacts.json` pattern for CI. + +## Failure modes addressed + +- **Spawn fails**: hard exit 1 with goplugin error (handled by shared spawnAndDial). +- **gRPC-dial fails**: hard exit 1. +- **GetManifest returns Unimplemented**: hard exit 1 with "plugin SDK appears stale; expected GetManifest available since workflow v0.20". +- **Plugin process leaks**: explicit `client.Kill()` in defer + cleanup via spawnAndDial helper. +- **Malformed plugin.json**: reuse `PluginManifest.Validate()`. +- **Mid-RPC plugin crash**: gRPC error surfaces; exit 1 with the error message. +- **plugin.json declares version "1.2.3" but binary ldflag never fired**: matrix row "X.Y.Z + sentinel → FAIL" catches this — the primary truth-loop bug class. +- **plugin.json declares "0.0.0" sentinel but binary somehow has non-sentinel Version**: matrix row "0.0.0 + non-sentinel → PASS" — acceptable, indicates CI artifact under verification. + +## Rollback + +Runtime-affecting change class (CLI subcommand + CI step). Rollback path: + +- **Subcommand**: revert workflow PR; subcommand stops being registered. Existing pipelines unaffected — nothing depends on it yet. +- **Shared spawnAndDial helper refactor**: revert is part of same PR; conformance returns to inline pattern; no behavior change in conformance. +- **CI step** (scaffold follow-up): revert scaffold-template PR; release.yml stops invoking the subcommand; existing release pipelines still pass. + +Backwards-compat: subcommand is purely additive. Older wfctl callers continue to work. + +## Decisions to record (ADRs) + +1. **--binary required (no build-from-source)** — chose explicit-binary requirement over dev-mode convenience to avoid per-repo ldflag-path divergence. ADR target: `decisions/NNNN-verify-capabilities-binary-required.md`. + +2. **Scope limited to Name + Version** — chose NOT to verify ContractRegistry (no plugin.json LHS exists today + BuildContractRegistry returns infra-internal services). Follow-up issue (to be filed: workflow#766) introduces `capabilities.iacServices` schema on PluginManifest + a server-side `BuildContractRegistryForPlugin()` filter; cycle-4 of verify-capabilities can then add the contract-diff against a clean wire surface. ADR target: `decisions/NNNN-verify-capabilities-scope-name-version-only.md`. + +## What this design does NOT do (explicit non-goals) + +- **Does NOT verify ModuleTypes/StepTypes/TriggerTypes** at runtime (per-type RPCs Unimplemented in IaC bridge; per-cycle-1 F2). Static check via `validate-contract` is authoritative. +- **Does NOT verify typed-contract surface** via `GetContractRegistry` (no plugin.json LHS + binary side emits infra-internal services as noise; cycle-2 F-NEW-1 + F-NEW-2). Deferred to follow-up issue. +- **Does NOT build the binary** — operator must produce one (local: `go build` with explicit ldflag; CI: goreleaser). +- **Does NOT verify `minEngineVersion`** at runtime (not on `pb.Manifest`). Static-check responsibility. +- **Does NOT run inside `plugin conformance`** (separate subcommand; shared helper is the only overlap). +- **Does NOT support `--json` output mode** (defer YAGNI; follow-up). +- **Does NOT support multi-binary repos** (runs against the binary passed via `--binary`; multi-binary repos invoke multiple times). +- **Does NOT verify Author/Description/ConfigMutable/SampleCategory** Manifest fields (Author/Description are display strings, drift not load-bearing; ConfigMutable/SampleCategory are runtime configuration not contract surface). Scope limited to fields that catch real bugs. + +## Open questions + +None blocking. Implementation can proceed. diff --git a/docs/plans/2026-05-24-verify-capabilities.md b/docs/plans/2026-05-24-verify-capabilities.md new file mode 100644 index 00000000..f8cd632d --- /dev/null +++ b/docs/plans/2026-05-24-verify-capabilities.md @@ -0,0 +1,1079 @@ +# wfctl plugin verify-capabilities Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Implement `wfctl plugin verify-capabilities --binary ` subcommand that spawns a plugin binary, calls `PluginService.GetManifest` directly via gRPC, and diffs `Name` + `Version` against plugin.json with sentinel-pattern matching to catch the ldflag-missing truth-loop bug from workflow#762/#764. + +**Architecture:** New subcommand registered in `cmd/wfctl/plugin.go`. Spawn-and-dial wired INLINE in the subcommand (~40 LOC) — `plugin_conformance.go`'s spawn pattern is the reference; no shared-helper extraction in this PR (defer until a 3rd caller appears). GetManifest called DIRECTLY via `pb.NewPluginServiceClient(pluginClient.Conn())` to bypass `ExternalPluginAdapter.EngineManifest()`'s precedence rules (which would defeat the truth-loop check). Diff logic: exact Name + sentinel-aware Version matrix. + +**Tech Stack:** Go (workflow CLI), `goplugin` (go-plugin v1.7), `pb.PluginService` (gRPC, raw client), `external.Handshake`, `external.PluginClient.Conn()`. + +**Base branch:** `main` + +**Design doc:** `docs/plans/2026-05-24-verify-capabilities-design.md` (cycle-6 PASS adversarial). + +**Issue:** workflow#765 + +**Revision history:** +- Cycle 1: 9-task plan with shared `spawnAndDial` helper extraction. FAILED — 4 Critical (fictional `EngineManifest()` signature; fixture template wrong PluginManifest type; missing-ldflag mechanics misstated; fixture plugin.json shape diverges from PluginManifest). +- Cycle 2: drop helper extraction; direct GetManifest RPC; fix fixture types; fix sentinel mechanics. FAILED — 3 Critical (anchor `case "validate-contract":` didn't exist on stale worktree base; duplicate `import (...)` blocks in test+production files would fail compile; name-drift test assertion too lenient). +- Cycle 3: rebased onto main; tightened import-block instructions Tasks 2-4; bumped fixture go directive; isolated name-drift via ldflag. FAILED — 1 Critical (Task 7 reintroduced duplicate-import-block defect — missed by cycle-3 fix that covered only Tasks 2-4) + 2 Important (name-drift comment misstated matrix row; generator shell-quoting hazard via REPO_ROOT in worktrees with spaces). +- Cycle 4: fix Task 7 import-block instruction; clarify name-drift matrix-row comment; rewrite generators. FAILED — 2 Critical (Task 1 pre-staged unused `os`/`path/filepath` imports → "imported and not used" compile error; Task 7 helper uses `fmt.Sprintf` but `"fmt"` not in import list → `undefined: fmt`). +- Cycle 5 (this version): Task 1 ships test file with only `strings`/`testing` imports; Task 2 explicitly adds `os` + `path/filepath` when first used; Task 7 explicitly adds `fmt` + `os/exec` (was only `os/exec`). Each task's imports match its actual usage. + +--- + +## Scope Manifest + +**PR Count:** 1 +**Tasks:** 8 +**Estimated Lines of Change:** ~500 (subcommand + tests + 5 fixtures + docs) + +**Out of scope:** +- Contract-diff (`GetContractRegistry` walk) — deferred to follow-up issue #766; needs `capabilities.iacServices` schema on `PluginManifest` first +- Per-type RPC walk (`GetModuleTypes`/`GetStepTypes`/`GetTriggerTypes`) — IaC bridge returns Unimplemented for these +- Build-from-source mode — `--binary` REQUIRED; dev convenience builds documented in §Synopsis +- `--json` output mode — defer YAGNI +- Multi-binary repos — runs against the binary passed; multi-binary plugins invoke multiple times +- Author/Description/ConfigMutable/SampleCategory diff — display fields, not contract surface +- `minEngineVersion` runtime check — not on `pb.Manifest`; static-check responsibility +- Scaffold-template release.yml wiring — separate follow-up PR on scaffold-workflow-plugin after this lands +- Shared `spawnAndDial` helper extraction — defer until a 3rd caller exists (cycle-2 reviewer Option 3) + +**PR Grouping:** + +| PR # | Title | Tasks | Branch | +|------|-------|-------|--------| +| 1 | feat(wfctl): plugin verify-capabilities subcommand (workflow#765) | Task 1, Task 2, Task 3, Task 4, Task 5, Task 6, Task 7, Task 8 | feat/765-verify-capabilities | + +**Status:** Locked 2026-05-24T05:01:50Z + +--- + +### Task 1: Subcommand registration + flag parsing skeleton + +**Change class:** CLI command. + +**Files:** +- Create: `cmd/wfctl/plugin_verify_capabilities.go` +- Modify: `cmd/wfctl/plugin.go` (add `case "verify-capabilities"` + help line) +- Test: `cmd/wfctl/plugin_verify_capabilities_test.go` + +**Step 1: Write the failing tests** + +Create `cmd/wfctl/plugin_verify_capabilities_test.go`: + +**Note: every "append" instruction in this plan EDITS the existing file's import block (Go disallows redundant imports across multiple `import` declarations in the same file). Adding new imports = Edit the existing block; never append a second `import (...)` block.** + +```go +package main + +import ( + "strings" + "testing" +) + +func TestVerifyCapabilitiesUsage(t *testing.T) { + err := runPluginVerifyCapabilities([]string{}) + if err == nil { + t.Fatal("want error for missing args") + } + if !strings.Contains(err.Error(), "--binary") { + t.Errorf("error %q should mention --binary", err.Error()) + } +} + +func TestVerifyCapabilitiesRequiresBinary(t *testing.T) { + err := runPluginVerifyCapabilities([]string{"."}) + if err == nil { + t.Fatal("want error when --binary missing") + } + if !strings.Contains(err.Error(), "--binary") { + t.Errorf("error %q should mention --binary", err.Error()) + } +} +``` + +**Step 2: Run test — verify FAIL** + +Run: `cd cmd/wfctl && go test -run TestVerifyCapabilities -count=1 ./...` +Expected: FAIL with `undefined: runPluginVerifyCapabilities`. + +**Step 3: Create the subcommand skeleton** + +Create `cmd/wfctl/plugin_verify_capabilities.go`: + +```go +// Package main — `wfctl plugin verify-capabilities` subcommand. +// Spawns a plugin binary, calls PluginService.GetManifest directly via gRPC, +// diffs returned Manifest against plugin.json. Catches ldflag-missing +// truth-loop bug from workflow#762/#764. +// +// Design: docs/plans/2026-05-24-verify-capabilities-design.md +// Issue: https://github.com/GoCodeAlone/workflow/issues/765 +package main + +import ( + "flag" + "fmt" +) + +func runPluginVerifyCapabilities(args []string) error { + fs := flag.NewFlagSet("plugin verify-capabilities", flag.ContinueOnError) + binary := fs.String("binary", "", "Path to plugin binary (REQUIRED)") + fs.Usage = func() { + fmt.Fprintf(fs.Output(), `Usage: wfctl plugin verify-capabilities --binary + +Spawn the plugin binary and verify its runtime PluginService.GetManifest +matches the declared plugin.json. Catches ldflag-missing / version-drift +bugs at release time (workflow#762 truth-loop closure). + +REQUIRED: --binary (no build-from-source; operator builds the binary) + +WARNING: this command EXECUTES as a subprocess. Only run against +build artifacts you trust. + +Examples: + # Local dev: + go build -ldflags="-X github.com/.../internal.Version=v1.2.3" -o /tmp/p ./cmd/ + wfctl plugin verify-capabilities --binary /tmp/p . + + # CI (post-goreleaser, in release.yml): + RUNNER_ARCH=$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/') + BIN=$(jq -r --arg arch "$RUNNER_ARCH" \ + '[.[] | select(.type=="Binary" and .goos=="linux" and .goarch==$arch)] | .[0].path // ""' \ + dist/artifacts.json) + wfctl plugin verify-capabilities --binary "$BIN" . + +Options: +`) + fs.PrintDefaults() + } + if err := fs.Parse(args); err != nil { + return err + } + if *binary == "" { + fs.Usage() + return fmt.Errorf("--binary is required") + } + if fs.NArg() != 1 { + fs.Usage() + return fmt.Errorf("exactly one argument required") + } + pluginDir := fs.Arg(0) + _ = pluginDir + return fmt.Errorf("not yet implemented") +} +``` + +Edit `cmd/wfctl/plugin.go`: +- Find the `case "validate-contract":` dispatcher block (~line 38) and add right after: + ```go + case "verify-capabilities": + return runPluginVerifyCapabilities(args[1:]) + ``` +- In the usage() help-text block, add (alphabetical): + ```go + fmt.Fprintln(out, " verify-capabilities Spawn plugin binary, verify runtime GetManifest matches plugin.json") + ``` + +**Step 4: Run test — verify PASS** + +Run: `cd cmd/wfctl && go build ./... && go test -run TestVerifyCapabilities -count=1 ./...` +Expected: build exit 0; both tests PASS. + +**Step 5: Commit** + +```bash +git add cmd/wfctl/plugin_verify_capabilities.go cmd/wfctl/plugin_verify_capabilities_test.go cmd/wfctl/plugin.go +git commit -m "feat(wfctl): plugin verify-capabilities subcommand skeleton (workflow#765)" +``` + +--- + +### Task 2: Preflight binary path validation + +**Change class:** Internal logic refactor (input validation). + +**Files:** +- Modify: `cmd/wfctl/plugin_verify_capabilities.go` +- Modify: `cmd/wfctl/plugin_verify_capabilities_test.go` + +**Step 1: Write the failing tests** + +In `cmd/wfctl/plugin_verify_capabilities_test.go`: **Edit the existing SINGLE import block** to add `"os"` and `"path/filepath"` (alongside existing `"strings"`, `"testing"`). DO NOT add a second `import (...)` declaration. Then append the test functions below. + +```go +func TestPreflightBinaryEmpty(t *testing.T) { + if err := preflightBinary(""); err == nil || !strings.Contains(err.Error(), "binary path") { + t.Errorf("want empty-path error, got %v", err) + } +} + +func TestPreflightBinaryNull(t *testing.T) { + if err := preflightBinary("null"); err == nil || !strings.Contains(err.Error(), "binary path") { + t.Errorf("want null-path error (jq fallback), got %v", err) + } +} + +func TestPreflightBinaryMissing(t *testing.T) { + if err := preflightBinary("/nonexistent/missing-xyz"); err == nil || !strings.Contains(err.Error(), "stat") { + t.Errorf("want stat error, got %v", err) + } +} + +func TestPreflightBinaryDirectory(t *testing.T) { + if err := preflightBinary(t.TempDir()); err == nil || !strings.Contains(err.Error(), "directory") { + t.Errorf("want directory error, got %v", err) + } +} + +func TestPreflightBinaryNonExecutable(t *testing.T) { + d := t.TempDir() + f := filepath.Join(d, "p") + if err := os.WriteFile(f, []byte("not-exec"), 0o644); err != nil { + t.Fatal(err) + } + if err := preflightBinary(f); err == nil || !strings.Contains(err.Error(), "executable") { + t.Errorf("want non-executable error, got %v", err) + } +} + +func TestPreflightBinaryOK(t *testing.T) { + d := t.TempDir() + f := filepath.Join(d, "p") + if err := os.WriteFile(f, []byte("#!/bin/sh\necho ok"), 0o755); err != nil { + t.Fatal(err) + } + if err := preflightBinary(f); err != nil { + t.Errorf("want PASS, got %v", err) + } +} +``` + +**Step 2: Run test — verify FAIL** + +Run: `cd cmd/wfctl && go test -run TestPreflightBinary -count=1 ./...` +Expected: FAIL `undefined: preflightBinary`. + +**Step 3: Implement** + +In `cmd/wfctl/plugin_verify_capabilities.go`: **Edit the existing single import block** to add `"os"` (alongside existing `"flag"`, `"fmt"`). DO NOT add a second `import (...)` block. Then append the `preflightBinary` function below. + +```go +// preflightBinary validates the --binary path before exec: +// - non-empty + not literal "null" (guards against jq fallback returning empty) +// - file exists and is a regular file (not directory) +// - has at least one executable bit set +func preflightBinary(path string) error { + if path == "" || path == "null" { + return fmt.Errorf("--binary path empty (jq filter may have returned no match)") + } + fi, err := os.Stat(path) + if err != nil { + return fmt.Errorf("stat %q: %w", path, err) + } + if fi.IsDir() { + return fmt.Errorf("--binary %q is a directory", path) + } + if !fi.Mode().IsRegular() { + return fmt.Errorf("--binary %q is not a regular file (mode=%s)", path, fi.Mode()) + } + if fi.Mode()&0o111 == 0 { + return fmt.Errorf("--binary %q is not executable (mode=%s)", path, fi.Mode()) + } + return nil +} +``` + +In `runPluginVerifyCapabilities`, before the `return fmt.Errorf("not yet implemented")` line: + +```go +if err := preflightBinary(*binary); err != nil { + return err +} +``` + +**Step 4: Run test — verify PASS** + +Run: `cd cmd/wfctl && go test -run TestPreflightBinary -count=1 ./...` +Expected: all 6 preflight tests PASS. + +**Step 5: Commit** + +```bash +git add cmd/wfctl/plugin_verify_capabilities.go cmd/wfctl/plugin_verify_capabilities_test.go +git commit -m "feat(wfctl): verify-capabilities preflight binary-path validation (workflow#765)" +``` + +--- + +### Task 3: Sentinel-pattern Version diff matrix + +**Change class:** Internal logic refactor (pure-logic diff function). + +**Files:** +- Modify: `cmd/wfctl/plugin_verify_capabilities.go` +- Modify: `cmd/wfctl/plugin_verify_capabilities_test.go` + +**Step 1: Write the failing tests** (table-driven) + +Append to `cmd/wfctl/plugin_verify_capabilities_test.go`: + +```go +func TestIsSentinel(t *testing.T) { + cases := map[string]bool{ + "": true, + "dev": true, + "0.0.0": true, + "(devel)": true, + "(devel) [@ a1b2c3d]": true, + "(devel) [@ a1b2c3d.dirty]": true, + "v1.2.3": false, + "1.2.3": false, + "v0.0.1": false, + } + for v, want := range cases { + if got := isSentinel(v); got != want { + t.Errorf("isSentinel(%q) = %v, want %v", v, got, want) + } + } +} + +func TestDiffVersion(t *testing.T) { + cases := []struct { + declared, runtime string + wantPass bool + wantReason string + }{ + // 0.0.0 + non-sentinel -> PASS (CI artifact) + {"0.0.0", "v1.2.3", true, ""}, + {"0.0.0", "0.1.0", true, ""}, + // 0.0.0 + sentinel -> FAIL (ldflag missing) + {"0.0.0", "", false, "ldflag"}, + {"0.0.0", "(devel)", false, "ldflag"}, + {"0.0.0", "(devel) [@ abc1234]", false, "ldflag"}, + {"0.0.0", "dev", false, "ldflag"}, + {"0.0.0", "0.0.0", false, "ldflag"}, + // X.Y.Z + vX.Y.Z or X.Y.Z -> PASS (normalize leading v) + {"1.2.3", "v1.2.3", true, ""}, + {"1.2.3", "1.2.3", true, ""}, + // X.Y.Z + sentinel -> FAIL + {"1.2.3", "", false, "ldflag"}, + {"1.2.3", "(devel)", false, "ldflag"}, + {"1.2.3", "(devel) [@ deadbee]", false, "ldflag"}, + // X.Y.Z + drift -> FAIL + {"1.2.3", "v0.9.0", false, "drift"}, + {"1.2.3", "v2.0.0", false, "drift"}, + } + for _, c := range cases { + pass, reason := diffVersion(c.declared, c.runtime) + if pass != c.wantPass { + t.Errorf("diffVersion(%q, %q) pass=%v want=%v reason=%q", + c.declared, c.runtime, pass, c.wantPass, reason) + continue + } + if !pass && !strings.Contains(reason, c.wantReason) { + t.Errorf("diffVersion(%q, %q) reason=%q want substring %q", + c.declared, c.runtime, reason, c.wantReason) + } + } +} +``` + +**Step 2: Run tests — verify FAIL** + +Run: `cd cmd/wfctl && go test -run "TestIsSentinel|TestDiffVersion" -count=1 ./...` +Expected: FAIL `undefined: isSentinel`, `undefined: diffVersion`. + +**Step 3: Implement** + +In `cmd/wfctl/plugin_verify_capabilities.go`: **Edit the existing single import block** to add `"strings"`. Then append `isSentinel` + `diffVersion` below. + +```go +// isSentinel returns true when v is one of the SDK's dev-sentinel forms +// OR the on-disk plugin.json sentinel "0.0.0". +// +// SDK sentinel set (per plugin/external/sdk/buildversion.go:36-42): +// "", "dev", "(devel)" — ResolveBuildVersion replaces these with build-info +// Plus build-info fallback produces "(devel) [@ [.dirty]]" — HasPrefix catches all forms. +// Plus on-disk plugin.json "0.0.0" sentinel (workflow#762 convention). +// +// The predicate MUST be a SUPERSET of the SDK's set; "dev" is defensive +// (canonical wiring through sdk.ResolveBuildVersion prevents literal "dev" +// from reaching the wire — included to catch non-canonical wiring accidents). +func isSentinel(v string) bool { + switch v { + case "", "dev", "0.0.0", "(devel)": + return true + } + return strings.HasPrefix(v, "(devel)") +} + +// diffVersion implements the Version-rule matrix from the design doc: +// +// plugin.json binary Manifest.Version outcome +// ------------ ------------------------ ------- +// "0.0.0" non-sentinel PASS (CI artifact under verification) +// "0.0.0" sentinel FAIL (ldflag injection missing) +// "X.Y.Z" "vX.Y.Z" or "X.Y.Z" PASS (normalize leading v) +// "X.Y.Z" sentinel FAIL (ldflag missing) +// "X.Y.Z" anything else FAIL (version drift) +// +// Returns (pass bool, reason string). reason is non-empty only when pass=false. +func diffVersion(declared, runtime string) (bool, string) { + runtimeSentinel := isSentinel(runtime) + if declared == "0.0.0" { + if runtimeSentinel { + return false, fmt.Sprintf("ldflag injection missing: plugin.json=%q; binary Manifest.Version=%q (sentinel)", declared, runtime) + } + return true, "" + } + if runtimeSentinel { + return false, fmt.Sprintf("ldflag injection missing: plugin.json=%q (release); binary Manifest.Version=%q (sentinel)", declared, runtime) + } + rNorm := strings.TrimPrefix(runtime, "v") + if rNorm == declared { + return true, "" + } + return false, fmt.Sprintf("version drift: plugin.json=%q; binary Manifest.Version=%q", declared, runtime) +} +``` + +**Step 4: Run tests — verify PASS** + +Run: `cd cmd/wfctl && go test -run "TestIsSentinel|TestDiffVersion" -count=1 ./...` +Expected: all matrix cases PASS. + +**Step 5: Commit** + +```bash +git add cmd/wfctl/plugin_verify_capabilities.go cmd/wfctl/plugin_verify_capabilities_test.go +git commit -m "feat(wfctl): verify-capabilities sentinel-pattern Version diff matrix (workflow#765)" +``` + +--- + +### Task 4: Inline spawn-and-dial + direct GetManifest RPC + +**Change class:** Plugin / extension (CLI subcommand that calls a plugin via raw gRPC). + +**Files:** +- Modify: `cmd/wfctl/plugin_verify_capabilities.go` + +This task wires the actual spawn-and-dial INLINE (no shared helper extraction — cycle-2 reviewer Option 3 + I2 elimination). GetManifest is called DIRECTLY via `pb.NewPluginServiceClient(pluginClient.Conn())` to bypass `ExternalPluginAdapter`'s precedence rules. + +**Step 1: Edit the existing import block** + +Add these to the SINGLE existing import block at the top of `cmd/wfctl/plugin_verify_capabilities.go` (do NOT add a second `import (...)` declaration): + +- `"context"` +- `"encoding/json"` +- `"os/exec"` +- `"path/filepath"` +- `"time"` +- `external "github.com/GoCodeAlone/workflow/plugin/external"` +- `pb "github.com/GoCodeAlone/workflow/plugin/external/proto"` +- `"github.com/GoCodeAlone/workflow/plugin"` +- `goplugin "github.com/GoCodeAlone/go-plugin"` +- `hclog "github.com/hashicorp/go-hclog"` +- `"google.golang.org/protobuf/types/known/emptypb"` + +Final import block should contain (alphabetical, stdlib then 3rd-party): + +```go +import ( + "context" + "encoding/json" + "flag" + "fmt" + "os" + "os/exec" + "path/filepath" + "strings" + "time" + + goplugin "github.com/GoCodeAlone/go-plugin" + "github.com/GoCodeAlone/workflow/plugin" + external "github.com/GoCodeAlone/workflow/plugin/external" + pb "github.com/GoCodeAlone/workflow/plugin/external/proto" + hclog "github.com/hashicorp/go-hclog" + "google.golang.org/protobuf/types/known/emptypb" +) +``` + +**Step 2: Load + validate plugin.json** + +In `runPluginVerifyCapabilities`, replace the `return fmt.Errorf("not yet implemented")` line with: + +```go +abs, err := filepath.Abs(pluginDir) +if err != nil { + return fmt.Errorf("resolve %q: %w", pluginDir, err) +} +manifestPath := filepath.Join(abs, "plugin.json") +manifestBytes, err := os.ReadFile(manifestPath) //nolint:gosec // operator-supplied path. +if err != nil { + return fmt.Errorf("plugin.json: %w", err) +} +var declared plugin.PluginManifest +if err := json.Unmarshal(manifestBytes, &declared); err != nil { + return fmt.Errorf("plugin.json parse: %w", err) +} +if err := declared.Validate(); err != nil { + return fmt.Errorf("plugin.json validate: %w", err) +} +``` + +**Step 3: Spawn + dial INLINE (no shared helper)** + +Continue in `runPluginVerifyCapabilities`: + +```go +ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) +defer cancel() + +binAbs, err := filepath.Abs(*binary) +if err != nil { + return fmt.Errorf("resolve --binary %q: %w", *binary, err) +} + +var stdout, stderr tailBuffer +cmd := exec.CommandContext(ctx, binAbs) //nolint:gosec // operator-supplied binary path. +client := goplugin.NewClient(&goplugin.ClientConfig{ + HandshakeConfig: external.Handshake, + Plugins: goplugin.PluginSet{"plugin": &external.GRPCPlugin{}}, + Cmd: cmd, + AllowedProtocols: []goplugin.Protocol{goplugin.ProtocolGRPC}, + Stderr: &stderr, + SyncStdout: &stdout, + SyncStderr: &stderr, + Logger: hclog.NewNullLogger(), +}) +defer client.Kill() + +rpcClient, err := client.Client() +if err != nil { + if ctx.Err() != nil { + return fmt.Errorf("timeout waiting for plugin handshake (stderr: %s)", stderr.String()) + } + return fmt.Errorf("plugin dial: %w (stderr: %s)", err, stderr.String()) +} +raw, err := rpcClient.Dispense("plugin") +if err != nil { + return fmt.Errorf("dispense plugin: %w (stderr: %s)", err, stderr.String()) +} +pluginClient, ok := raw.(*external.PluginClient) +if !ok { + return fmt.Errorf("dispensed object is %T, want *external.PluginClient", raw) +} +``` + +Note: `tailBuffer` is defined in `cmd/wfctl/plugin_conformance.go` (same package). All required imports already added in Step 1. + +**Step 4: Call GetManifest DIRECTLY via raw gRPC client** (bypasses adapter precedence) + +```go +pbClient := pb.NewPluginServiceClient(pluginClient.Conn()) +runtime, err := pbClient.GetManifest(ctx, &emptypb.Empty{}) +if err != nil { + return fmt.Errorf("GetManifest RPC: %w (stderr: %s)", err, stderr.String()) +} +``` + +**Step 5: Diff Name + Version and report** + +```go +var failures []string +if runtime.GetName() != declared.Name { + failures = append(failures, fmt.Sprintf("name: plugin.json=%q; binary Manifest.Name=%q", declared.Name, runtime.GetName())) +} +if pass, reason := diffVersion(declared.Version, runtime.GetVersion()); !pass { + failures = append(failures, "version: "+reason) +} +if len(failures) > 0 { + fmt.Fprintf(os.Stderr, "FAIL %s (plugin.json)\nerror: %d mismatch(es)\n", declared.Name, len(failures)) + for _, f := range failures { + fmt.Fprintf(os.Stderr, " - %s\n", f) + } + // Embed the joined failure list in the returned error so tests can assert + // on specific field names (e.g. "name:" prefix) without capturing stderr. + return fmt.Errorf("verify-capabilities: %d mismatch(es): %s", len(failures), strings.Join(failures, "; ")) +} +fmt.Printf("OK %s %s (plugin.json: %s)\n", declared.Name, runtime.GetVersion(), declared.Version) +return nil +``` + +**Step 6: Build + help-output sanity check** + +Run: +```bash +cd cmd/wfctl && go build -o /tmp/wfctl ./... && /tmp/wfctl plugin verify-capabilities --help +``` +Expected: help text printed; exit 0. Help contains "REQUIRED: --binary", "WARNING: this command EXECUTES", `jq` CI example. + +**Step 7: Commit** + +```bash +git add cmd/wfctl/plugin_verify_capabilities.go +git commit -m "feat(wfctl): wire inline spawn + direct GetManifest + Name/Version diff (workflow#765)" +``` + +--- + +### Task 5: Create 4 build-PASS fixture scenarios + +**Change class:** Test fixture (no runtime impact). + +**Files:** +- Create: `cmd/wfctl/testdata/verify_capabilities/good/{plugin.json,main.go,go.mod,go.sum}` +- Create: `cmd/wfctl/testdata/verify_capabilities/release-good/{plugin.json,main.go,go.mod,go.sum}` +- Create: `cmd/wfctl/testdata/verify_capabilities/missing-ldflag/{plugin.json,main.go,go.mod,go.sum}` +- Create: `cmd/wfctl/testdata/verify_capabilities/version-drift/{plugin.json,main.go,go.mod,go.sum}` +- Create: `cmd/wfctl/testdata/verify_capabilities/README.md` + +Cycle-2 fix C2: fixture template uses `sdk.PluginManifest` (sdk-package value, NOT `plugin.PluginManifest`), no error return on `Manifest()`. Cycle-2 fix C3: initial `Version = "dev"` so `ResolveBuildVersion("dev")` falls back to `(devel)` when no ldflag is applied (true exercise of the missing-ldflag scenario). Cycle-2 fix C4: minimal plugin.json with only fields PluginManifest actually models. + +**Step 1: Write the generator script** (one-off; not committed) + +Save as `/tmp/gen-verify-fixtures.sh`: + +```bash +#!/bin/bash +set -euo pipefail +BASE=cmd/wfctl/testdata/verify_capabilities +declare -A NAMES=( [good]=verify-good [release-good]=verify-release-good [missing-ldflag]=verify-missing-ldflag [version-drift]=verify-version-drift ) +declare -A VERS=( [good]=0.0.0 [release-good]=1.2.3 [missing-ldflag]=0.0.0 [version-drift]=1.2.3 ) +for s in good release-good missing-ldflag version-drift; do + d="$BASE/$s" + mkdir -p "$d" + name="${NAMES[$s]}" + version="${VERS[$s]}" + cat > "$d/plugin.json" < "$d/main.go" <]" when no +// ldflag fires (exercises the missing-ldflag scenario faithfully). +var Version = "dev" + +type stubProvider struct{} + +func (stubProvider) Manifest() sdk.PluginManifest { + return sdk.PluginManifest{ + Name: "$name", + Version: "$version", + Author: "test fixture", + Description: "verify-capabilities $s scenario", + } +} + +func main() { + sdk.Serve(stubProvider{}, + sdk.WithBuildVersion(sdk.ResolveBuildVersion(Version)), + ) +} +GO + cat > "$d/go.mod" <<'MOD' +module github.com/test/PLACEHOLDER + +go 1.26.0 + +require github.com/GoCodeAlone/workflow v0.62.0 + +replace github.com/GoCodeAlone/workflow => ../../../../.. +MOD + sed -i.bak "s|PLACEHOLDER|$s|" "$d/go.mod" && rm -f "$d/go.mod.bak" +done +``` + +Note: relative `replace` is written DIRECTLY (no `$REPO_ROOT` indirection) so generators run on worktrees with spaces in their path. PLACEHOLDER substitution uses fixed-text sed (no shell-expansion of module name). + +**Step 2: Generate + tidy (writes go.sum in-place)** + +```bash +bash /tmp/gen-verify-fixtures.sh +for d in cmd/wfctl/testdata/verify_capabilities/*/; do + (cd "$d" && GOWORK=off go mod tidy) +done +``` + +**Step 3: Verify all 4 fixtures build standalone** + +```bash +for d in cmd/wfctl/testdata/verify_capabilities/*/; do + (cd "$d" && GOWORK=off go build -mod=readonly -o /tmp/p .) && echo "$d: ok" || { echo "$d: FAIL"; exit 1; } +done +``` +Expected: all 4 print `ok`. + +**Step 4: Create maintenance README** + +```bash +cat > cmd/wfctl/testdata/verify_capabilities/README.md <<'MD' +# verify_capabilities test fixtures + +Fixtures for `plugin_verify_capabilities_test.go` (workflow#765). + +Each scenario directory is a self-contained Go module. Tests build in-place +with `go build -mod=readonly`; binary emitted to `t.TempDir()`. + +## Maintenance + +When workflow SDK adds a new transitive dep that fixtures pick up, regenerate +each fixture's `go.sum`: + +```bash +for d in cmd/wfctl/testdata/verify_capabilities/*/; do + (cd "$d" && GOWORK=off go mod tidy) +done +git add cmd/wfctl/testdata/verify_capabilities/*/go.sum +``` + +The `replace github.com/GoCodeAlone/workflow => ../../../../..` directive +resolves 5-ups from each scenario directory to the workflow repo root. +DO NOT use an absolute path — it diverges across developer machines. + +## Scenarios + +- `good/` — plugin.json version=0.0.0, ldflag injects v0.1.0 → PASS (CI artifact case) +- `release-good/` — plugin.json version=1.2.3, ldflag injects v1.2.3 → PASS (release case) +- `missing-ldflag/` — plugin.json version=0.0.0, no ldflag (Version="dev" → ResolveBuildVersion returns "(devel) [@ sha]") → FAIL +- `version-drift/` — plugin.json version=1.2.3, ldflag injects v0.9.0 → FAIL +- `name-drift/` — plugin.json name="foo", binary advertises Name="bar" (see Task 6 — created separately) → FAIL + +## SDK semantics + +`sdk.ResolveBuildVersion` returns its argument unchanged UNLESS the arg is one +of `{"", "dev", "(devel)"}`, in which case it consults `debug.ReadBuildInfo()` +and returns `"(devel) [@ [.dirty]]"`. So: + +- Initial `var Version = "dev"` + no ldflag → wire Version is `"(devel) [@ sha]"` +- Initial `var Version = "dev"` + ldflag `-X .Version=v1.2.3` → wire Version is `"v1.2.3"` +- Initial `var Version = "0.0.0"` + no ldflag → wire Version is `"0.0.0"` (NOT a build-info fallback; `"0.0.0"` is NOT in the SDK's reset set) + +The `missing-ldflag` fixture uses `Version = "dev"` deliberately so it exercises +the `(devel)` fallback path, not the `"0.0.0"` pass-through. +MD +``` + +**Step 5: Commit** + +```bash +git add cmd/wfctl/testdata/verify_capabilities/ +git commit -m "test(wfctl): verify-capabilities fixtures (4 build-pass scenarios) (workflow#765)" +``` + +--- + +### Task 6: Create name-drift fixture (binary advertises different Name) + +**Change class:** Test fixture. + +**Files:** +- Create: `cmd/wfctl/testdata/verify_capabilities/name-drift/{plugin.json,main.go,go.mod,go.sum}` + +**Step 1: Generate the fixture** + +```bash +d=cmd/wfctl/testdata/verify_capabilities/name-drift +mkdir -p "$d" +cat > "$d/plugin.json" <<'JSON' +{ + "name": "verify-name-drift", + "version": "0.0.0", + "minEngineVersion": "v0.62.0", + "author": "test fixture", + "description": "verify-capabilities name-drift scenario" +} +JSON +cat > "$d/main.go" <<'GO' +package main + +import sdk "github.com/GoCodeAlone/workflow/plugin/external/sdk" + +var Version = "dev" + +type stubProvider struct{} + +// Manifest intentionally returns a DIFFERENT name than plugin.json declares. +// plugin.json says "verify-name-drift"; runtime says "verify-name-drift-binary". +func (stubProvider) Manifest() sdk.PluginManifest { + return sdk.PluginManifest{ + Name: "verify-name-drift-binary", + Version: "0.0.0", + Author: "test fixture", + Description: "verify-capabilities name-drift scenario", + } +} + +func main() { + sdk.Serve(stubProvider{}, + sdk.WithBuildVersion(sdk.ResolveBuildVersion(Version)), + ) +} +GO +cat > "$d/go.mod" <<'MOD' +module github.com/test/name-drift + +go 1.26.0 + +require github.com/GoCodeAlone/workflow v0.62.0 + +replace github.com/GoCodeAlone/workflow => ../../../../.. +MOD +(cd "$d" && GOWORK=off go mod tidy) +``` + +**Step 2: Verify fixture builds** + +Run: `(cd cmd/wfctl/testdata/verify_capabilities/name-drift && GOWORK=off go build -mod=readonly -o /tmp/p .)` +Expected: exit 0. + +**Step 3: Commit** + +```bash +git add cmd/wfctl/testdata/verify_capabilities/name-drift/ +git commit -m "test(wfctl): name-drift fixture (binary advertises mismatched Name) (workflow#765)" +``` + +--- + +### Task 7: Integration tests — 5 scenarios end-to-end + +**Change class:** Plugin / extension (exercises spawn + RPC + diff against real fixture binaries). + +**Files:** +- Modify: `cmd/wfctl/plugin_verify_capabilities_test.go` + +**Step 1: Add the fixture-build helper + 5 test cases** + +In `cmd/wfctl/plugin_verify_capabilities_test.go`: **Edit the existing SINGLE import block** to add `"fmt"` and `"os/exec"` (alongside existing `"os"`, `"path/filepath"`, `"strings"`, `"testing"`). DO NOT add a second `import (...)` declaration — golangci-lint will fail the build. Then append the helper + test functions below. + +```go +// buildFixtureBinaryForVerify builds the fixture scenario in-place and emits +// the binary to t.TempDir(). ldflag is the -X ...Version= value ("" = no flag, +// which makes ResolveBuildVersion fall back to "(devel) [@ sha]" for fixtures +// whose initial Version var is "dev"). +func buildFixtureBinaryForVerify(t *testing.T, scenario, ldflagTag string) string { + t.Helper() + binPath := filepath.Join(t.TempDir(), "p") + args := []string{"build", "-mod=readonly"} + if ldflagTag != "" { + args = append(args, "-ldflags", + fmt.Sprintf("-X github.com/test/%s.Version=%s", scenario, ldflagTag)) + } + args = append(args, "-o", binPath, ".") + cmd := exec.Command("go", args...) + cmd.Dir = filepath.Join("testdata", "verify_capabilities", scenario) + cmd.Env = append(os.Environ(), "GOWORK=off") + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("build %s: %v\n%s", scenario, err, out) + } + return binPath +} + +func TestVerifyCapabilities_Good(t *testing.T) { + bin := buildFixtureBinaryForVerify(t, "good", "v0.1.0") + if err := runPluginVerifyCapabilities([]string{"--binary", bin, "testdata/verify_capabilities/good"}); err != nil { + t.Fatalf("want PASS, got: %v", err) + } +} + +func TestVerifyCapabilities_ReleaseGood(t *testing.T) { + bin := buildFixtureBinaryForVerify(t, "release-good", "v1.2.3") + if err := runPluginVerifyCapabilities([]string{"--binary", bin, "testdata/verify_capabilities/release-good"}); err != nil { + t.Fatalf("want PASS, got: %v", err) + } +} + +func TestVerifyCapabilities_MissingLdflag(t *testing.T) { + // No ldflag → Version stays "dev" → ResolveBuildVersion("dev") → "(devel) [@ sha]" + bin := buildFixtureBinaryForVerify(t, "missing-ldflag", "") + err := runPluginVerifyCapabilities([]string{"--binary", bin, "testdata/verify_capabilities/missing-ldflag"}) + if err == nil { + t.Fatal("want FAIL, got nil") + } + if !strings.Contains(err.Error(), "mismatch") { + t.Errorf("want mismatch error, got: %v", err) + } +} + +func TestVerifyCapabilities_VersionDrift(t *testing.T) { + bin := buildFixtureBinaryForVerify(t, "version-drift", "v0.9.0") + err := runPluginVerifyCapabilities([]string{"--binary", bin, "testdata/verify_capabilities/version-drift"}) + if err == nil { + t.Fatal("want FAIL, got nil") + } + if !strings.Contains(err.Error(), "mismatch") { + t.Errorf("want mismatch error, got: %v", err) + } +} + +func TestVerifyCapabilities_NameDrift(t *testing.T) { + // Build with non-sentinel ldflag tag so Version PASSes — matrix row that + // fires: plugin.json="0.0.0" + binary="v0.0.0" → PASS via the + // `declared == "0.0.0"` branch returning early (isSentinel("v0.0.0")==false + // because the SDK sentinel set is {"", "dev", "0.0.0", "(devel)..."} — NOT + // "v0.0.0"). This ISOLATES Name as the sole failure under test, so a + // regression that breaks Name-diff while leaving Version-diff intact + // doesn't silently pass through a lenient `Contains("mismatch")` check. + bin := buildFixtureBinaryForVerify(t, "name-drift", "v0.0.0") + err := runPluginVerifyCapabilities([]string{"--binary", bin, "testdata/verify_capabilities/name-drift"}) + if err == nil { + t.Fatal("want FAIL, got nil") + } + // Tighter assertion: error must specifically mention "name:" prefix from the diff report. + if !strings.Contains(err.Error(), "name:") && !strings.Contains(fmt.Sprintf("%v", err), "name:") { + t.Errorf("want name-mismatch error, got: %v", err) + } +} +``` + +**Step 2: Run all integration tests** + +Run: `cd cmd/wfctl && go test -run TestVerifyCapabilities -count=1 -timeout 120s ./...` +Expected: 5 scenario tests + 8 unit tests from Tasks 1-3 = 13 PASS. + +**Step 3: Commit** + +```bash +git add cmd/wfctl/plugin_verify_capabilities_test.go +git commit -m "test(wfctl): verify-capabilities integration tests (5 scenarios) (workflow#765)" +``` + +--- + +### Task 8: Documentation update — PLUGIN_RELEASE_GATES.md + +**Change class:** Documentation. + +**Files:** +- Modify: `docs/PLUGIN_RELEASE_GATES.md` (append Verify-Capabilities section) + +**Step 1: Append section** + +```bash +cat >> docs/PLUGIN_RELEASE_GATES.md <<'MD' + +## Verify-Capabilities (workflow#765 — runtime truth-check) + +`wfctl plugin verify-capabilities` is the runtime sibling of `validate-contract`: +it spawns the plugin binary, calls `PluginService.GetManifest`, and verifies +the returned `Name` + `Version` match `plugin.json`. Catches the +**ldflag-missing truth-loop bug**: a plugin can pass `validate-contract` +(static check) and still ship a binary whose `Manifest.Version` is the +SDK's `(devel) [@ sha]` sentinel because the goreleaser ldflag never fired. + +### Synopsis + +``` +wfctl plugin verify-capabilities --binary +``` + +`--binary` REQUIRED (no build-from-source — operator builds via goreleaser +or `go build`). + +⚠ **Executes the binary** as a subprocess. Only run against artifacts you trust. + +### Local development + +```bash +go build -ldflags="-X github.com/GoCodeAlone/workflow-plugin-/internal.Version=v1.2.3" \ + -o /tmp/p ./cmd/ +wfctl plugin verify-capabilities --binary /tmp/p . +``` + +### CI integration (release.yml post-goreleaser, pre-publish) + +```yaml +- name: Verify capabilities (post-build runtime check) + run: | + RUNNER_ARCH=$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/') + BIN=$(jq -r --arg arch "$RUNNER_ARCH" \ + '[.[] | select(.type=="Binary" and .goos=="linux" and .goarch==$arch)] | .[0].path // ""' \ + dist/artifacts.json) + "${RUNNER_TEMP}/wfctl-bin/wfctl" plugin verify-capabilities --binary "$BIN" . +``` + +### Version diff matrix + +| plugin.json `version` | binary `Manifest.Version` | Outcome | +|---|---|---| +| `"0.0.0"` (sentinel) | non-sentinel (`"v1.2.3"`) | PASS — CI artifact under verification | +| `"0.0.0"` | sentinel (`""`, `"dev"`, `"0.0.0"`, `"(devel)..."`) | FAIL — ldflag missing | +| `"X.Y.Z"` (release) | `"vX.Y.Z"` or `"X.Y.Z"` | PASS — normalize leading v | +| `"X.Y.Z"` | sentinel | FAIL — ldflag missing | +| `"X.Y.Z"` | anything else | FAIL — version drift | + +### Non-goals + +- Does NOT walk per-type RPCs (`GetModuleTypes`/`GetStepTypes`/`GetTriggerTypes`) — IaC bridge returns Unimplemented. +- Does NOT diff `GetContractRegistry` — deferred to workflow#766 (requires `capabilities.iacServices` schema first). +- Does NOT build the binary — operator's responsibility. +- Does NOT verify `minEngineVersion` at runtime (not on `pb.Manifest`). + +See `docs/plans/2026-05-24-verify-capabilities-design.md` for full design. +MD +``` + +**Step 2: Verify no broken anchors** + +Run (best-effort; skip if tool absent): `markdown-link-check docs/PLUGIN_RELEASE_GATES.md 2>&1 | head` +Expected: no broken links (or tool-missing — acceptable; visual review of the append). + +**Step 3: Commit** + +```bash +git add docs/PLUGIN_RELEASE_GATES.md +git commit -m "docs: add Verify-Capabilities section to PLUGIN_RELEASE_GATES (workflow#765)" +``` + +--- + +## Final verification (post-Task-8) + +Before opening the PR: + +```bash +# 1. All tests pass (unit + integration) +cd cmd/wfctl && go test -count=1 -timeout 120s ./... + +# 2. Lint clean +go vet ./... +golangci-lint run ./cmd/wfctl/... + +# 3. Help text correct +go build -o /tmp/wfctl ./cmd/wfctl && /tmp/wfctl plugin verify-capabilities --help +# Expected: help text contains "REQUIRED: --binary", "WARNING: this command EXECUTES", jq example + +# 4. Conformance still works (no regression from inlined spawn — we did NOT touch conformance) +go test -run TestConformance -count=1 -timeout 300s ./cmd/wfctl/... + +# 5. End-to-end smoke against a real plugin (out-of-tree) +# cd /tmp && git clone --depth=1 git@github.com:GoCodeAlone/workflow-plugin-discord.git +# cd workflow-plugin-discord +# go build -ldflags="-X .../internal.Version=v0.1.1" -o /tmp/p ./cmd/workflow-plugin-discord +# /tmp/wfctl plugin verify-capabilities --binary /tmp/p . +# Expected: "OK workflow-plugin-discord v0.1.1 (plugin.json: 0.1.1)" +``` + +## Rollback + +This PR adds a CLI subcommand only — no shared-helper refactor, no schema migrations, no upstream consumer changes. Rollback path: +- `git revert ` removes the new subcommand and its tests + fixtures + doc append. +- No data migration, no schema change, no upstream consumer change. +- Backwards-compat: subcommand is purely additive; pre-PR wfctl callers continue to work. + +Scaffold-template release.yml wiring is a separate follow-up PR on scaffold-workflow-plugin (not in this PR's scope). diff --git a/docs/plans/2026-05-24-verify-capabilities.md.scope-lock b/docs/plans/2026-05-24-verify-capabilities.md.scope-lock new file mode 100644 index 00000000..5bc518dc --- /dev/null +++ b/docs/plans/2026-05-24-verify-capabilities.md.scope-lock @@ -0,0 +1 @@ +aa85b5960321a75b42c4c0bed5343c432aabcde0c2dedac42bdf17a57dd5d5bc