Showcase: automated template compatibility testing + AI debugging workflow alignment#280
Draft
kalanyuz wants to merge 212 commits intocre-reliability-load-testfrom
Draft
Showcase: automated template compatibility testing + AI debugging workflow alignment#280kalanyuz wants to merge 212 commits intocre-reliability-load-testfrom
kalanyuz wants to merge 212 commits intocre-reliability-load-testfrom
Conversation
…alues (#124) * Limit LogTrigger filtering / event encoding to indexed fields * Added sanitation for empty topic values * added missing nil reflect import for empty bindings
* feat: add preview-build pipeline * Potential fix for code scanning alert no. 7: Workflow does not contain permissions Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * update --------- Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* fix compile error * regenerate bindings
* feat: add update command * feat: add update command * fix: use commandPath for excluded commands * chore: gendoc * chore: address comments * chore: address comments * chore: address comments * chore: update maxExtractSize to 500MB for windows * improve message on windows * delay tmpDir cleanup
* update init cmd next step. Added RPC prompt for por template. * updated workflow settings template so we have different value per environment for config and workflow name * lint fix * Added rpc-url flag for init cmd * fix init test * chore: fix tests * fix cre init that would validate rpc-url flag * fix cre init that would validate rpc-url flag * fix cre_init test * update docs * remove comment --------- Co-authored-by: anirudhwarrier <12178754+anirudhwarrier@users.noreply.github.com> Co-authored-by: Akshay Aggarwal <aks.agg94@gmail.com>
* logic to check latest release from repo * Updated update check script + logger priority * added sync.WaitGroup so that coroutine for version check and telemetry doesn't get killed by process termination (may happen if command executed is very fast, such as cre version) * cleaned up comments * updated version check timeout to 6 secs * gendoc * fixed linter errors * fixed comments * Fix update display for sub-root command (i.e workflow, account, secrets) Make sure update warning is always display last * go mod tidy * gendocs * moved update package to internal/update * fixed order of sequence for update check message display * fix telemetry not being called * Revert waitgroup to time.sleep * fix emitter event function * revert telemetry change --------- Co-authored-by: Akshay Aggarwal <akshay.aggarwal@smartcontract.com> Co-authored-by: anirudhwarrier <12178754+anirudhwarrier@users.noreply.github.com>
* Limit LogTrigger filtering / event encoding to indexed fields * Added sanitation for empty topic values * added missing nil reflect import for empty bindings * Reworked log trigger encoding in bindings * bumped sdk version and regenerated example bindings
* verify ownership before execution * lint * fix lint issues --------- Co-authored-by: anirudhwarrier <12178754+anirudhwarrier@users.noreply.github.com>
increase timout on http requests
* add update cmd in the exclusion for update check * added version check for cre update to skip if already on the latest
* fix context cancel and improve user event detail * mod tidy * fix tests
* reorder and enrich * lint * cleanup
* reorder and enrich * lint * cleanup * Add flags and decrease sleep
fix: error on refresh token
Reduce sleep duration before exiting the command.
* Update TS templates to use the latest SDK * Point to beta release
* Validate credentials * fix tests
* Add workflowID to user event * clean up logs
Update default ts sdk version to beta.1
* add workflow language to runtimecontext * lint * review feedback
* test charm lib impelentation for cre init
* Add shared UI package with Charm ecosystem for styled CLI output
- Create internal/ui/ package with centralized Lipgloss styles (styles.go)
- Add output helpers for consistent styling: Title, Box, Success, Dim, etc.
- Implement Bubble Tea spinner with reference counting for async operations
- Add GlobalSpinner singleton for seamless spinner across CLI lifecycle
- Update PersistentPreRunE to show spinner during initialization
- Migrate cre init and cre whoami to use shared UI package
* Add spinner to cre init and fix layout issues
- Add spinner during file generation (copying, generating templates, contracts)
- Show spinner during Go dependencies installation
- Display dependencies in styled box after spinner completes
- Fix Next steps box spacing and formatting
- Refactor initializeGoModule to return deps instead of printing
* Style help page with Lipgloss
- Add styled template functions (styleTitle, styleSection, styleCommand, styleDim, styleSuccess, styleCode)
- Update help template to use Lipgloss styling
- Style section headers, command names, tips, and URLs
- Improve visual hierarchy and readability
* Apply Chainlink Blocks color palette to CLI
- Add complete Blocks palette constants (Gray, Blue, Green, Red, Orange, Yellow, Teal, Purple)
- Use high-contrast colors for dark terminal readability
- Style titles/commands with Blue 400-500 for visibility
- Style secondary info with Gray 500 (dimmed)
- Create custom Huh theme with Blocks colors for forms
- Update spinner to use Blue 500
* Modernize cre login command with Charm UI components
- Add styled title and welcome message using Chainlink theme
- Add Bubble Tea spinner with progress states throughout auth flow:
- Preparing authentication
- Opening browser
- Waiting for authentication
- Exchanging authorization code
- Saving credentials
- Show styled URL fallback when browser cannot open automatically
- Display success message with next steps in branded box
- Update spinner message during org membership retry flow
- Update tests to include spinner in handler instantiations
* Add styled error display for CLI using ui.Error()
- Add SilenceErrors: true to root command to disable Cobra's default error output
- Display all user-facing errors with styled ui.Error() in Execute()
- Errors now show with red color and ✗ prefix consistent with Chainlink theme
- Internal debug logging via zerolog remains unchanged
* Improve CLI error display with styled output and smart usage handling
- Add SilenceErrors: true to disable Cobra's default error output
- Display errors with styled ui.Error() (red with ✗ prefix)
- Set SilenceUsage in PersistentPreRunE to hide usage for runtime errors
- Keep usage/suggestions visible for command typos and flag errors
* Add login prompt when authentication is required
* Fix tests broken by Charm UI changes
* Fix cre logout to skip credential check and handle nil credentials
* Modernize cre version command with Charm UI styling
* Modernize cre update command with progress bar and Charm UI
* Fix creinit tests to provide all inputs via flags
* Update the list-key command to use the shared Charm UI components for
consistent styling across the CLI
* Update test assertions for Charm UI output changes
* Update the generate-bindings command to use the shared Charm UI
* Update the secrets list command to use the shared Charm UI component
* Update the secrets delete command to use the shared Charm UI components
* Modernize workflow pause, activate, delete with Charm UI
* updated styling for workflow deploy, pause, delete and transaction
* mondernized link and unlink command using charm lib
* updated missing charm console output in creinit.go
* fixed build output ui
* updated fmt.print with new ui that were not replaced
* mondernized simulate command with charm ui output
* 1. Updated internal/settings/settings_generate.go:
- Replaced prompt.YesNoPrompt with huh.NewConfirm forms
- Removed stdin io.Reader parameter
2. Updated cmd/creinit/creinit.go:
- Updated call sites to match new function signatures
3. Updated cmd/secrets/common/handler.go:
- Replaced ~25 fmt.Print* calls with ui.* functions
4. Updated cmd/workflow/simulate/telemetry_writer.go:
- Replaced fmt.Printf with ui.Printf
- Removed unused fmt import
5. Deleted internal/prompt/ directory:
- Removed entire old promptui-based package
6. Cleaned cmd/common/utils.go:
- Removed unused MustGetUserInputWithPrompt function
- Removed unused bufio and errors imports
7. Dependencies cleaned (go mod tidy):
- Removed github.com/manifoldco/promptui
- Removed github.com/chzyer/readline
* improved prompt behavior with autocomplete and default value
* added error helpers
* improved login command with cancellation / escape
* fixed logout issue that was not flushing credentials
* updated login to remove the esc logic
* updated secret tests that were failing
* fixed lint error: unused functions and returns
* Refactored credentials.go for deploy access status and added deploy access display to cre whoami command
* Updated gated message with the command to request access
* added new account access command
* added account access command to settings exclusion
* access command logic to submit form to zendesk
* added prompt to request access when running cre account access cmd
* Refactor access request logic into shared package and add deploy access check to workflow deploy command
* Fix background goroutine error appearing during deploy access prompt
* Update account command description to mention deploy access
* Show deploy access hint after successful workflow simulation
* Add deploy access hint to global help template for gated users
* Added prompt to describe use cases when request access request
* update code so that request is sent to a proxy that will take care of submitting the request to zendesk
* updated deploy request changes to be compatible with new charm lib refactor
* updated simulator deploy message to use box layout
* Yes is now selected by default for cre deploy access request prompt
* updated creinit to use bubbletea wizard
* fixed linter issues
* fixed weird spacing in creinit.go
* cre init: wizard now display files created in <workflow directory> and
- Contracts generated in <contracts directory>
* restored original comments in creinit.go
* fixed update cmd progress indicator issue
* Temp mock access request behavior before API implementation
* Added ascii logo in creinit wizard
* Spinner do not display in verbose mode to avoid stdout and stderr conflict and visual display issue in verbose mode
* replaced temp REST HTTP client with GraphQL client
* added tests for access cmd
* Replace embedded templates with dynamic GitHub-based template fetching
Templates are now discovered and downloaded from GitHub repositories at
runtime instead of being embedded in the CLI binary via go:embed. This
enables template updates without CLI releases and supports multiple
template sources.
New packages:
- internal/templaterepo/ - GitHub API client, file cache, and registry
for discovering, caching, and scaffolding templates
- internal/config/ - CLI config (~/.cre/config.yaml) for template
repository sources with env var and flag overrides
Changes:
- cmd/creinit: rewritten to use registry-based template fetching with
RegistryInterface for testability. Wizard simplified to 3 steps
(project name, template selection, workflow name)
- cmd/creinit/template/workflow/ removed (all embedded templates)
- cmd/creinit/go_module_init.go removed (templates are self-contained)
- SDK version constants moved to internal/constants/
- cmd/generate-bindings updated to import from constants package
- E2E tests updated: --template-id flag replaced with --template using
template name strings (kv-store-go, cre-custom-data-feed-go, etc.)
Template sources configurable via:
- --template-repo flag
- CRE_TEMPLATE_REPOS env var
- ~/.cre/config.yaml
- Default: smartcontractkit/cre-templates@main
* fix & log for extracing templates
* Built-in hello-world template — always available, even offline:
- internal/templaterepo/builtin/hello-world-go/ — embedded files (main.go, workflow.yaml, README.md, configs, secrets.yaml)
- internal/templaterepo/builtin.go — //go:embed, BuiltInTemplate var, ScaffoldBuiltIn() function
- TemplateSummary.BuiltIn bool field to distinguish embedded vs remote
- ListTemplates() always prepends the built-in template (no network needed)
- ScaffoldTemplate() uses ScaffoldBuiltIn() for built-in templates instead of downloading
* Remove template-repo flag from cre init, added template-id as a deprecated flag for backward compatibility
* Added back builtin TS hello world template, updated template-id behavior
* Added networks field to template metadata (internal/templaterepo/types.go) — An optional []string of chain names in template.yaml.
Dynamic project.yaml RPC generation (internal/settings/):
- project.yaml.tpl now uses a {{RPCsList}} placeholder instead of hardcoded chains
- BuildRPCsListYAML() generates the rpcs YAML block from networks + URLs
- Falls back to ethereum-testnet-sepolia with default public RPC when no networks specified
New wizard step (cmd/creinit/wizard.go):
- stepNetworkRPCs between template selection and workflow name
- Prompts for each network's RPC URL sequentially
- Empty input accepted (fill later), non-empty input validated as valid HTTP/HTTPS URL
- Step is skipped if template has no networks or all RPCs provided via flags
--rpc-url flag (cmd/creinit/creinit.go):
- Repeatable StringArray flag: --rpc-url chain-name=url
- Parsed in ResolveInputs(), merged with wizard results in Execute()
- Flag values override wizard input
* Patch RPC when template arleady have rpc configured in project.yaml
* Fixed go init for default embed template
* Fixed TS template default template (same as main branch now)
* Updated init wizard for better layout
* Add multi-workflow support and preserve template-shipped workflow.yaml
- Add WorkflowDirEntry struct and Workflows/PostInit fields to
TemplateMetadata in types.go
- Rewrite renameWorkflowDir in registry.go to branch on workflow count:
skip renaming for multi-workflow, rename single workflow, heuristic
fallback for legacy
- Update wizard to skip workflow name prompt for multi-workflow
templates and default placeholder from Workflows[0].Dir
- Default workflowName from Workflows[0].Dir in non-interactive mode
- Loop GenerateWorkflowSettingsFile for each declared workflow dir in
multi-workflow templates
- Skip workflow.yaml generation when template already ships one
(prevents overwriting custom config paths)
- Rewrite printSuccessMessage to show per-workflow summary and postInit
instructions
- Add 6 new test cases covering multi-workflow, single-workflow
defaults, rename with flag, and built-in backwards compat
* added in wizard check for cre init - folder and rpc url
* updated cre init list layout
* cre init search wizard stay displayed
* updated template.yaml local to .cre/template.yaml
* added a flag to pass new repository url, made ~/.cre/template.yaml source of truth for template source
* Root-level template repo extraction fix
* added cre templates list/add/remove to manage repo sources
* template.yaml now has projectDir to specify the cre project dir, updated copy logic
* Added prediction market repo as default
* --target is now explicit and prompted if ommitted
* Fixed log message (from log to debug)
* fix go mod tidy for dynamic template in go
* fix linter
* cre init create .env file if not present
* fix linter
---------
Co-authored-by: Tarcísio Zotelli Ferraz <tar-2008.ferraz@hotmail.com>
Added Pharos Atlantic support
* Fix gosec violations for golangci-lint v2.11.2 * pin lint version
…tegration with CI/CD pipelines (#302) * Allow build, upload, and deploy do be done on their own for better integration with CI/CD pipelines * Small cleanups, no need to be logged in to build, specify full path of build output * Allow larger path for deploys to account for URLs * Fix folder permissions * Fix more lint... * Lint again... * another lint fix attempt...
* chore: bumps http capability * chore: bump core version * match versions in test --------- Co-authored-by: Timothy Funnell <137069066+timothyF95@users.noreply.github.com>
…imulate would (#308) Fix a bug where build doesn't work on some edge cases where simulate does...
…ilds fully reproducible (#310) * Fix a bug where build doesn't work on some edge cases where simulate does... * Respect .env in all commands, and override other settings. Make go builds fully reproducible * Unify UX for env load, and have better messaging * Update docs * Use .env.public for public config * Gendoc ...
* Add hash command and display hashes during other commands * parity between deploy and hash * Fix docs --------- Co-authored-by: De Clercq Wentzel <10665586+wentzeld@users.noreply.github.com>
* Add the ability to have replace from env var in project.yaml, this enable RPC Secrets in urls * fix trailing slash --------- Co-authored-by: Tarcísio Zotelli Ferraz <tar-2008.ferraz@hotmail.com>
…is (#312) * Add feature to generate-bindings to allow processing JSON contract abis * Change abigen to read data from abi file based on extension, if .json then parses inner abi field, else does original input parsing * Regenerate docs for generate-bindings * optimize generate-bindings tests --------- Co-authored-by: Tarcísio Zotelli Ferraz <tar-2008.ferraz@hotmail.com>
* Added mock forwarders for new EVM chains * Lint * Removed celo sepolia * Added mock forwarder * Disabled gnosis chiado simulation * Added mainnet mock forwarders * Updated cre sdk go version
Wasm debug symbols by default for simulator
* updated metadata to support new field, category workflow is now the only installable template with the CLI * updated cre init to support multiple workflow dir for bun install instruction * fixed RPC url replacement issue on cre init * cre init wizard only show template selected card when passing --template param * fixed test/linter * updated error message when cre init --template is not found --------- Co-authored-by: Tarcísio Zotelli Ferraz <tar-2008.ferraz@hotmail.com>
* feat: add --limits flag to cre workflow simulate # Conflicts: # cmd/root.go # cmd/workflow/simulate/simulate.go # cmd/workflow/workflow.go * feat: add --limits flag to cre workflow simulate * feat: add --limits flag to cre workflow simulate * fix lint * fix implement interface * update docs * update text for sim limits * remove unused arg * unit tests * fallback to default if flag is empty * fix apply order * use brotli compression --------- Co-authored-by: Tarcisio Ferraz <tarcisio.ferraz@smartcontract.com>
…NodeJS APIs (#295) * Update built-in template * Restore proper versioning
Added hyperliquid mainnet and gnosis chiado support
* Add fetch for tenant config and write context file * Update based on new api changes * lint * Adjust for latest api changes * add support for api key * Clean up * review feedback * tenant -> user context
* Add fetch for tenant config and write context file * Update based on new api changes * lint * Adjust for latest api changes * add support for api key * Clean up * Extract orgID from claim
* Add fetch for tenant config and write context file * Update based on new api changes * lint * Adjust for latest api changes * add support for api key * Clean up * Extract orgID from claim * review feedback * tenant -> user context * Add secret auth flow flag * refactor tests
* Add secret auth api request * Add tests and refactor RPC check * Refactor duplicate code * Bind to workflow owner when not nil
* Refactor: extract oauth from login * Lint
* CLI AI UX improvements: - Show required networks in cre templates list output - Add --json flag to cre templates list for machine-parseable output - Show actionable error with missing --rpc-url flags when cre init fails without TTY - Add --non-interactive flag to cre init for CI/CD usage - Only show tip footer on root help page, not subcommands - Make cre init respect -R/--project-root flag for output location * update creinit * fix tests * gendoc * comment fix
* Refactor: extract oauth from login * Lint * complete vault secrets browser OAuth (callback + exchange) * Lint * Update token exchage flow
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR: Showcase automated template compatibility testing and AI debugging workflow alignment
Summary
Brief overview of what this PR implements:
1..5(including IDs3and5) with a drift canary.flowchart LR A[Template or creinit change] --> B[Path Filter Job] B --> C[TestTemplateCompatibility] C --> D[PR Signal] D --> E[QA + Skills + Docs Alignment]Changes
1. Template Compatibility Automation
File:
test/template_compatibility_test.goWhat changed:
TestTemplateCompatibilitytable-driven suite for template IDs1..5.TS_HelloWorld_Template3andTS_ConfHTTP_Template5.TestTemplateCompatibility_AllTemplatesCovered.Result: Template init/build/simulate regressions are caught deterministically and coverage drift is flagged early.
2. CI Enforcement Path
File:
.github/workflows/pull-request-main.ymlWhat changed:
template-compat-path-filter.ci-test-template-compaton Linux + Windows.go test -v -timeout 20m -run TestTemplateCompatibility ./test/when template-impacting paths change.Result: Compatibility checks become a targeted PR guardrail instead of ad-hoc local validation.
3. Template Repository Operational Setup
Files:
submodules.yamlscripts/setup-submodules.shMakefile.gitignoreWhat changed:
cre-templatesreference checkout.Result: External template workspace setup is reproducible without using Git submodules.
4. Agent and Framework Alignment
Files:
AGENTS.md.claude/skills/cre-add-template/**.claude/skills/cre-qa-runner/**.claude/skills/cre-cli-tui-testing/**.claude/skills/using-cre-cli/**testing-framework/*.md.claude/skills/skill-audit-report.mdWhat changed:
Result: Consistent operator + agent guidance across docs, skills, and CI expectations.
Impact
Modified (selected files)
test/template_compatibility_test.go- New unified compatibility suite + canary..github/workflows/pull-request-main.yml- Path filter + compatibility job.AGENTS.md- Source-mode architecture and maintenance workflow.submodules.yaml- External template repo relationship config.scripts/setup-submodules.sh- Deterministic clone/update/clean script..claude/skills/cre-add-template/SKILL.md- Template workflow and checklist boundaries..claude/skills/cre-qa-runner/SKILL.md- QA runbook/reporting + source provenance guidance.testing-framework/README.md- Embedded baseline + branch-gated dynamic framing.Affected (no runtime behavior change)
cmd/creinit/*runtime flow remains embedded-template baseline.Testing
Compatibility Suite
go test -v -timeout 20m -run TestTemplateCompatibility ./test/1,2,3,4,5and canary coverage guard.Skill/Docs Consistency Checks
rg "^description:" .claude/skills/*/SKILL.md).AGENTS.md,testing-framework/*.md, and updated skill docs.Deployment Notes
Pre-Deployment Requirements
Deployment Order
Backward Compatibility
Verification Checklist
1..5.Rollback Plan
If issues occur:
.github/workflows/pull-request-main.yml.test/template_compatibility_test.go.AGENTS.md,.claude/skills/**, andtesting-framework/*.md.