feat(e2e): customOperator, wait: imports, ./... discovery, shared install#155
Merged
Conversation
InstallOrUpgradeOrkestra and HelmInstall now capture stdout/stderr instead of piping to the terminal. On failure the captured output is included in the returned error. On success only the single status line is printed. WaitForResource used kubectl wait --for=condition=Ready for Deployments, which never succeeds (Deployments have no Ready condition). Now uses kubectl rollout status deployment/<name> instead.
…rators Adds spec.customOperator: true to E2ESpec. When set, bundle generation, Orkestra helm install, OCI pre-pull, and helm uninstall are all skipped. Cluster setup, CRD apply, setup manifests, CR apply, assertions, and CRD cleanup all run unchanged. resolveSource() now allows specs with spec.cr but no spec.katalog when customOperator is true. applyCRD guards against empty katalogFile. isPureAggregator() correctly excludes customOperator specs with a CR. checkAll() now runs commands before resources so action commands (e.g. kubectl delete) execute before the resource state is checked. ork validate shows 'mode: custom operator (Orkestra install skipped)' and no longer requires spec.katalog when customOperator: true.
Adds wait: string to E2EImport. The runner sleeps that duration before starting the import. Useful for webhook deregistration gaps, namespace termination, or cert-manager cert provisioning after setup.helm. Validated at load time with time.ParseDuration — bad values are caught before the cluster starts. ork validate shows '(wait: Xs)' in the imports list. Shorthand form (plain string) continues to work unchanged.
Adds recursive e2e discovery: ork e2e ./... walks the directory tree, finds all *e2e.yaml files that are not pure aggregators (imports-only), sorts them, builds a dynamic in-memory suite, and runs it. ork e2e ./... ork e2e ./examples/beginner/... ork e2e ./... --wait 2s # inject wait between each test ork e2e ./... --skip vendor,testdata,external/07-vault Pure aggregators are skipped to avoid double-running tests that are already covered by their root suite. --skip accepts comma-separated path patterns matched against the full file path.
04-imports.md: adds wait to the fields table and a 'When to use wait:' section covering webhook deregistration, namespace termination, and cert provisioning. 01-spec.md: notes that spec.katalog is optional when customOperator: true; adds spec.customOperator field entry with link to new page. 05-custom-operator.md: new page — what it does, how to activate, what is skipped, the setup.helm pattern, and four use cases (migration parity, third-party operators, two-operator composition, universal test harness).
01-pure-custom: cert-manager installed via setup.helm. Applies a self-signed Certificate CR, asserts the TLS Secret is created. customOperator: true — no Orkestra bundle or install. 3/3 confirmed passing. 02-side-by-side: AppCert CRD tested two ways. Orkestra side wraps cert-manager via customResources (ClusterIssuer + Certificate as children). Custom side applies cert-manager resources directly via command assertions. Identical assertions — when both pass, the two implementations are equivalent. Root README and root e2e.yaml aggregator included.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
spec.customOperator: true— skip Orkestra bundle+install, useork e2eas a test harness for any operator. Paired withsetup.helmto install the operator under test. Validation shows mode indicator;spec.katalogis optional.wait:on import declarations — sleep N before an import starts. Covers webhook deregistration, namespace termination, cert provisioning.runImportspre-installs once, each sub-runner syncs the bundle, coordinator uninstalls at the end. Eliminates per-import helm install/uninstall cycles (~3 min saved on a 5-import suite).ork e2e ./...discovery — walks a directory, finds all*e2e.yamlleaf files, runs them as a suite.--waitinjects wait between tests.--skipfilters paths.WaitForResourcefor Deployments — was usingkubectl wait --for=condition=Ready(no such condition); now useskubectl rollout status.checkAllorder — commands now run before resources so action commands execute before state is asserted.examples/use-cases/custom-operator/—01-pure-custom(cert-manager, 3/3 ✓) and02-side-by-side(AppCert parity via Orkestra vs direct cert-manager).04-imports.md(wait),01-spec.md(customOperator), new05-custom-operator.mdpage with use cases.Test plan
make orkcompiles cleanork e2e --use-currentpasses onexamples/use-cases/custom-operator/01-pure-custom— 3/3ork e2e ./examples/beginner/... --use-currentdiscovers 4 files, runs 4/4 with shared Orkestraork validateshowsmode: custom operatorand(wait: Xs)on imports with waitork e2e -f examples/use-cases/custom-operator/02-side-by-side/e2e-orkestra.yamlork e2e ./... --skip use-cases/custom-operatoron full examples tree