Add plugin compatibility conformance evidence#629
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds an executable plugin compatibility “evidence” pipeline to wfctl (conformance generation → registry compatibility index publishing → install/update/lock compatibility-aware resolution) and documents the new CLI surfaces and CI workflow.
Changes:
- Added strict plugin compatibility evidence/index models plus registry fetching for
compatibility/<plugin>/index.json. - Introduced
wfctl plugin conformanceandwfctl plugin-registry compatibility update, and wired install/update/lock to resolve versions via compatibility evidence with enforce/warn/force behavior. - Extended
.wfctl-lock.yamlto record platform-scoped compatibility metadata.
Reviewed changes
Copilot reviewed 46 out of 46 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/WFCTL.md | Documents new plugin conformance, plugin-registry compatibility update, and new install/update/lock flags/env behavior. |
| docs/plans/2026-05-11-plugin-conformance-compat.md.scope-lock | Adds plan scope lock metadata for the implementation plan. |
| docs/plans/2026-05-11-plugin-conformance-compat.md | Adds the implementation plan for compatibility evidence/indexing and resolver wiring. |
| docs/plans/2026-05-11-plugin-conformance-compat.alignment-check-2.md | Adds PASS alignment report for plan coverage. |
| docs/plans/2026-05-11-plugin-conformance-compat.alignment-check-1.md | Adds initial FAIL alignment report capturing missing requirements found earlier. |
| docs/plans/2026-05-11-plugin-conformance-compat.adversarial-review-2.md | Adds PASS adversarial review report for the plan. |
| docs/plans/2026-05-11-plugin-conformance-compat.adversarial-review-1.md | Adds initial FAIL adversarial review report for the plan. |
| docs/plans/2026-05-11-plugin-conformance-compat-design.md | Adds the design doc for compatibility evidence/index model and resolver behavior. |
| docs/plans/2026-05-11-plugin-conformance-compat-design.adversarial-review-6.md | Adds PASS adversarial review report for the design. |
| docs/plans/2026-05-11-plugin-conformance-compat-design.adversarial-review-5.md | Adds earlier FAIL adversarial review report for the design. |
| docs/plans/2026-05-11-plugin-conformance-compat-design.adversarial-review-4.md | Adds earlier FAIL adversarial review report for the design. |
| docs/plans/2026-05-11-plugin-conformance-compat-design.adversarial-review-3.md | Adds earlier FAIL adversarial review report for the design. |
| docs/plans/2026-05-11-plugin-conformance-compat-design.adversarial-review-2.md | Adds earlier FAIL adversarial review report for the design. |
| docs/plans/2026-05-11-plugin-conformance-compat-design.adversarial-review-1.md | Adds earlier FAIL adversarial review report for the design. |
| decisions/0030-plugin-conformance-evidence-index.md | Adds ADR capturing the decision to use generated compatibility evidence and version indexes. |
| config/wfctl_lockfile.go | Adds platform-scoped compatibility metadata to the wfctl lockfile schema and writer. |
| config/wfctl_lockfile_test.go | Adds/extends lockfile tests to cover compatibility metadata round-trip and output. |
| cmd/wfctl/testdata/conformance/no-iac/plugin.json | Adds a conformance fixture plugin without typed IaC service. |
| cmd/wfctl/testdata/conformance/no-iac/main.go | Implements the “no typed IaC” conformance fixture binary. |
| cmd/wfctl/testdata/conformance/no-iac/go.mod | Adds module metadata for the “no-iac” conformance fixture. |
| cmd/wfctl/testdata/conformance/iac-pass/plugin.json | Adds a passing typed IaC conformance fixture manifest. |
| cmd/wfctl/testdata/conformance/iac-pass/main.go | Implements a minimal typed IaC provider fixture for conformance. |
| cmd/wfctl/testdata/conformance/iac-pass/go.mod | Adds module metadata for the “iac-pass” conformance fixture. |
| cmd/wfctl/testdata/conformance/iac-hang/plugin.json | Adds a hanging conformance fixture manifest. |
| cmd/wfctl/testdata/conformance/iac-hang/main.go | Implements a fixture that sleeps to exercise timeout handling. |
| cmd/wfctl/testdata/conformance/iac-hang/go.mod | Adds module metadata for the “iac-hang” conformance fixture. |
| cmd/wfctl/registry_source.go | Extends registry sources with FetchVersionIndex and adds a manifest→index synthesis fallback. |
| cmd/wfctl/registry_source_test.go | Adds tests for native/synthesized version index fetching (static + GitHub). |
| cmd/wfctl/registry_config.go | Adds compatibility config + per-registry evidence trust configuration and defaults. |
| cmd/wfctl/registry_compatibility.go | Adds wfctl plugin-registry compatibility update to validate evidence and atomically update index.json. |
| cmd/wfctl/registry_compatibility_test.go | Adds tests for compatibility index updating, sorting, stale marking, and range derivation. |
| cmd/wfctl/registry_cmd.go | Adds compatibility subcommand routing and updates plugin-registry usage text. |
| cmd/wfctl/plugin.go | Adds wfctl plugin conformance subcommand wiring and help text. |
| cmd/wfctl/plugin_lock.go | Wires wfctl plugin lock through compatibility resolver and writes lockfile compatibility metadata. |
| cmd/wfctl/plugin_lock_test.go | Adds tests for compatibility-aware lock selection and warn-mode forced metadata. |
| cmd/wfctl/plugin_install.go | Wires install/update flows through compatibility resolver and adds new flags. |
| cmd/wfctl/plugin_install_test.go | Adds tests covering install/update behavior with compatibility evidence (pass/fail selection, warn behavior). |
| cmd/wfctl/plugin_conformance.go | Implements wfctl plugin conformance with staging, hashing, plugin launch, and evidence output. |
| cmd/wfctl/plugin_conformance_test.go | Adds tests for conformance CLI behavior, evidence output, archive hashing, and timeout handling. |
| cmd/wfctl/plugin_compat_resolver.go | Implements the compatibility resolver used by install/update/lock (enforce/warn/force). |
| cmd/wfctl/plugin_compat_resolver_test.go | Adds resolver unit tests for pass/fail precedence, missing evidence policy, and advisory mode. |
| cmd/wfctl/plugin_compat_model.go | Adds evidence/index models, semver normalization, SHA normalization, trust parsing, and evidence digest computation. |
| cmd/wfctl/plugin_compat_model_test.go | Adds tests for version canonicalization, digest rules, SHA normalization, and trust config parsing. |
| cmd/wfctl/multi_registry.go | Adds multi-registry support for fetching version indexes and a manifest+index “same source” fetch. |
| cmd/wfctl/multi_registry_test.go | Adds tests for trust defaults and multi-registry version index selection/normalized fallback. |
⏱ Benchmark Results✅ No significant performance regressions detected. benchstat comparison (baseline → PR)
|
Welcome to Codecov 🎉Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests. ℹ️ You can also turn on project coverage checks and project coverage reporting on Pull Request comment Thanks for integrating Codecov - We've got you covered ☂️ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
wfctl plugin conformancefor typed-IaC artifact/source checkswfctl plugin-registry compatibility updatefor atomic registry compatibility indexessetup-wfctlVerification
GOWORK=off go test ./cmd/wfctl -run 'TestPluginConformance' -count=1GOWORK=off go test ./cmd/wfctl -run 'TestPluginRegistryCompatibility|TestPluginCompatResolver|TestRunPluginInstall.*Compat|TestRunPluginUpdate.*Compat|TestPluginLock' -count=1GOWORK=off go test ./cmd/wfctl ./config -run 'TestPluginLock|TestWfctlLockfile|TestPluginCompatResolver|TestRunPluginInstall.*Compat|TestRunPluginUpdate.*Compat' -count=1GOWORK=off go build -o /tmp/wfctl-compat ./cmd/wfctl/tmp/wfctl-compat plugin conformance --mode typed-iac --format json --engine-version v0.51.2 ./cmd/wfctl/testdata/conformance/iac-pass/tmp/wfctl-compat plugin conformance --mode typed-iac --artifact /tmp/wfctl-iac-pass.tar.gz --format json --engine-version v0.51.2 --output /tmp/wfctl-evidence.json/tmp/wfctl-compat plugin-registry compatibility update --registry-dir /tmp/wfctl-test-registry --plugin iac-pass --version v0.1.0 --evidence /tmp/wfctl-evidence.json --latest-engine v0.51.2Known unrelated local failures remain in
cmd/wfctlfull/focused package runs:TestConfigMigrate_DefaultWriterIsStderrTestInfraMultiEnv_E2E/staging_plan_excludes_dnsTestInfraMultiEnv_E2E/prod_plan_includes_dns_with_large_dbTestRunCIRunTestFallsBackToGoTestWhenNoConfiguredTestsprints an intentional failing fixture but does not fail itself. The broad./...run failed only because of the samecmd/wfctlfailures;dynamicprinted expected panic-recovery text without failing the package.