feat(wfctl): plugin verify-capabilities subcommand (workflow#765)#766
Merged
Conversation
Files a design doc for the live-deploy CI matrix deferred from the 2026-05-19 multi-repo QoL sweep. Schema-level validation is insufficient to promote a plugin to 'verified'; this design adds a weekly OIDC-driven GitHub Actions matrix that exercises each IaC plugin's examples/minimal/config.yaml against staging cloud accounts, auto-promotes on green, demotes on 2 consecutive REDs. Execution is gated on operator provisioning staging accounts + GitHub OIDC trust per provider. Document this as the next concrete step. Companion to workflow#725 (marketplace-verify subcommand). Closes #723. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…matrix" This reverts commit e4be573.
…IL → revised) Cycle-1 had 3 Critical findings: - F1: diff table fields not on pb.Manifest (only 6 scalars; type lists on separate RPCs) - F2: IaC bridge returns Unimplemented for type-list RPCs - F4: handshake export path wrong (external.Handshake, not sdk) Plus 4 Important + 1 Minor. Cycle-2 pivots: - Adopt reviewer Option 3: Manifest scalars + ContractRegistry only (drops per-type RPC walk; collapses F1+F2+F7) - Adopt reviewer Option 2: drop build-from-source default; require --binary (addresses F5+F8 + cycle-1 self-challenge) - Adopt reviewer Option 1 partial: extract spawnAndDial helper from plugin_conformance.go (kills F3 duplication) - Fix F4 import; F5 version-diff matrix; F7 contract-diff for declared-contract half
…2 review) Cycle-2 FAILed with 2 Critical + 4 Important. Cycle-3 pivots: - Drop contract-diff entirely (F-NEW-1: plugin.json has no iacResources key; F-NEW-2: BuildContractRegistry emits infra-internal noise). Defer to follow-up issue #766 (introduces capabilities.iacServices schema + server-side BuildContractRegistryForPlugin filter). - Fix version-diff matrix (F-NEW-3): sentinel set is {"", "(devel)...", "0.0.0"}, cite buildversion.go:36-42. Matrix's isSentinel() predicate covers all SDK-emitted forms. - Specify test fixtures concretely (F-NEW-5): testdata/verify_capabilities/{good,release-good,missing-ldflag,version-drift,name-drift}/ mirroring plugin_validate_contract precedent. - CI binary path via jq dist/artifacts.json lookup (F-NEW-6). Scope reduces to: Manifest.Name exact + Manifest.Version sentinel-pattern matrix. Catches the truth-loop bug (ldflag-missing) the user actually asked about. Contract-diff deferred to follow-up after plugin.json schema gains LHS field.
…+ fixture precedent)
Cycle-3 FAILed: 2 Critical (sentinel-set drift; fixture precedent miscited). Cycle-4 fixes:
- F-CYCLE3-1: isSentinel() superset includes 'dev' to match SDK sentinel set {"", "dev", "(devel)"} from buildversion.go:36-42 + plugin.json '0.0.0' + HasPrefix '(devel)'.
- F-CYCLE3-2: Cite correct precedent — plugin_conformance_test.go:buildFixtureBinary with per-fixture go.mod + replace directive + GOWORK=off. Validate-contract is pure-static; never builds. Per-fixture layout spec includes go.mod template.
- F-CYCLE3-3: Add preflight binary-path validation + security note explicit; jq fallback ('// ""') guards against null returns.
- F-CYCLE3-4: Rewrite Surface row honestly — conformance-flag (--mode manifest-verify) is technically viable since checkTypedIaCPlugin handles non-IaC too; pick new-subcommand on separation-of-concerns basis (truth-check vs typed-IaC scan).
- F-CYCLE3-5: Explicit fixture PluginManifest.Validate() prereqs documented (name + version + author + description all required).
…t + fixture path + jq filter) Cycle-4 FAIL: 3 Important. Cycle-5 fixes: - spawn-helper cut-point explicit (line 504 only; IaC-validation lines 505-513 stay in conformance caller) - fixture build is in-place (NOT copy-to-TempDir) + main.go at fixture root matching precedent - jq filter pins goos==linux + goarch matching runner (uname -m sed map x86_64→amd64, aarch64→arm64)
…es fully) Cycle-5 FAIL: 3 Important regressions on cycle-4 fixes. Cycle-6 fixes: - F5-1: jq filter (goos+goarch pin) now propagated to BOTH Synopsis AND CI integration blocks (was only in Synopsis); strike 'any arch — only needs ONE binary' rationale (false: wrong-arch binary won't exec). - F5-2: pin -mod=readonly + check in fixture go.sum; sidesteps both source-tree pollution (mod=mod writes go.sum) and copy-to-TempDir overhead; maintenance note for regen on SDK dep bumps. - F5-3: explicit defer-transform note — line 484's defer client.Kill() returned as func() cleanup, caller defers; literal extraction would either kill plugin too early OR leak processes. - Reconcile line-range citations: 462-504 throughout (was 462-515 in Behavior step 2).
9 tasks, 1 PR. Pairs with design doc cycle-6 PASS adversarial.
…le-1 review) Cycle 1 FAIL: 4 Critical (fictional EngineManifest signature; fixture wrong PluginManifest type; missing-ldflag mechanics wrong; fixture json shape diverges). Cycle 2: - Drop Task 1 (spawnAndDial extraction) per reviewer Option 3; inline ~40 LOC. Eliminates I2 verification-class mismatch on refactor-without-test. - C1: GetManifest direct via pb.NewPluginServiceClient(pluginClient.Conn()) — bypasses adapter precedence. - C2: fixture uses sdk.PluginManifest (sdk-package value), Manifest() returns value (no error). - C3: initial Version='dev' so ResolveBuildVersion falls back to '(devel)' for missing-ldflag. - C4: minimal plugin.json (only PluginManifest fields). 8 tasks now (was 9). Single PR.
… rebase + tighten test) Cycle 2 FAIL 3 Critical + 2 Important. Cycle 3: - Rebased onto current main (validate-contract + registry-sync now exist in dispatcher). - Restructured Tasks 2/3/4 'append imports' instructions: explicit 'Edit the SINGLE existing import block' with warnings; Task 4 Step 1 specifies final import-block shape end-to-end (eliminates C-2-2 + C-2-3 duplicate-import compile failures). - Fixture go-directive 1.24 → 1.26.0 to match workflow root (eliminates I-2-1). - Name-drift fixture ldflag tag changed v0.1.0 → v0.0.0 so plugin.json '0.0.0' + binary 'v0.0.0' Version matrix PASSes; Name is isolated failure under test. Test assertion tightened from 'mismatch' to 'name:' substring; verify-capabilities error now embeds joined failure list so test can assert without stderr capture (eliminates I-2-2).
… + generator hazards)
Cycle 3 FAIL: 1 Critical (Task 7 reintroduced duplicate-import-block; cycle-3 fix missed Task 7). Cycle 4:
- Task 7 Step 1: replace literal 'import (...)' snippet with explicit 'Edit existing single import block' instruction.
- Task 7 name-drift test comment: clarified which matrix row fires (declared=='0.0.0' branch returns PASS; isSentinel('v0.0.0') false).
- Task 5 generator: embed relative 'replace github.com/GoCodeAlone/workflow => ../../../../..' DIRECTLY in heredoc (no $REPO_ROOT indirection, no sed dance); PLACEHOLDER for module name substituted via fixed-text sed. Eliminates shell-quoting hazard on worktrees with spaces.
- Task 6 generator: same direct-replace pattern.
…k 7 missing fmt)
Cycle 4 FAIL: 2 Critical (compile blockers). Cycle 5:
- Task 1: ship test file with only strings + testing (was pre-staging os + path/filepath which Task 1 functions don't use → 'imported and not used').
- Task 2: explicitly says 'Edit the existing SINGLE import block to add os + path/filepath' when those imports are first used.
- Task 7: now adds 'fmt' AND 'os/exec' (was only os/exec; helper uses fmt.Sprintf and test uses fmt.Sprintf('%v', err) → undefined: fmt).
Each task's import additions match its actual code usage.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
⏱ Benchmark Results✅ No significant performance regressions detected. benchstat comparison (baseline → PR)
|
Merged
3 tasks
intel352
added a commit
that referenced
this pull request
May 24, 2026
…766) (workflow#765) (#769) * feat(wfctl): plugin verify-capabilities subcommand skeleton (workflow#765) * feat(wfctl): verify-capabilities preflight binary-path validation (workflow#765) * feat(wfctl): verify-capabilities sentinel-pattern Version diff matrix (workflow#765) * feat(wfctl): wire inline spawn + direct GetManifest + Name/Version diff (workflow#765) * test(wfctl): verify-capabilities fixtures (4 build-pass scenarios) (workflow#765) * test(wfctl): name-drift fixture (binary advertises mismatched Name) (workflow#765) * test(wfctl): verify-capabilities integration tests (5 scenarios) (workflow#765) * docs: add Verify-Capabilities section to PLUGIN_RELEASE_GATES (workflow#765)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes workflow#765. Runtime sibling to
validate-contract: spawns a plugin binary, callsPluginService.GetManifestdirectly via gRPC, diffsName+Versionagainstplugin.jsonwith sentinel-aware Version matrix. Catches the ldflag-missing truth-loop bug that closes the workflow#762/#764 contract.Architecture
wfctl plugin verify-capabilities --binary <path> <plugin-dir>.--binaryREQUIRED (no build-from-source — operator builds via goreleaser orgo build).pb.NewPluginServiceClient(pluginClient.Conn())to bypassExternalPluginAdapter.EngineManifest()'s precedence rules.Test plan
go test -count=1 -timeout 120s ./cmd/wfctl/...— all 15 verify-capabilities tests PASSgo vet ./...cleanwfctl plugin verify-capabilities --helpprints expected usageTestPluginConformanceregression suite still PASS (no helper extraction; conformance untouched)Design + plan
Iterated through 6 design cycles + 5 plan cycles + alignment-check + scope-lock per superpowers pipeline:
docs/plans/2026-05-24-verify-capabilities-design.md(cycle 6 PASS adversarial)docs/plans/2026-05-24-verify-capabilities.md(cycle 5 PASS adversarial + alignment PASS + Locked 2026-05-24T05:01:50Z)Implementer deviations
-X github.com/test/%s.Version→-X main.Version(fixture ispackage main; linker symbol ismain.Version). Plan body retained as-is (scope-locked); test code carries the corrected form with explanatory comment.TestConformance, actual test names areTestPluginConformance*. Ran with correct pattern; suite green.Non-goals (deferred)
GetContractRegistry— follow-up issue feat(wfctl): plugin verify-capabilities subcommand (workflow#765) #766 (requirescapabilities.iacServicesschema onPluginManifestfirst).GetModuleTypes/GetStepTypes/GetTriggerTypes) — IaC bridge returns Unimplemented.