Skip to content

Add workflow_call contract test — catch new required secrets/inputs before release #260

@cbeaulieu-gt

Description

@cbeaulieu-gt

Problem

When a PR adds a new required: true secret or input to a reusable workflow's workflow_call: block, CI here passes green because this repo dogfoods via pull_request_target (direct secret access), bypassing the workflow_call path entirely. Consumers using the new tag then break silently with errors like:

##[error]No authentication token provided. Set app_id + app_private_key inputs.

This already shipped twice without detection:

The dogfood loop's blind spot is a structural one — pull_request_target reads the workflow definition from the base branch and uses the calling repo's secrets directly. The workflow_call form (where consumers must pass secrets through) is never exercised in CI.

Goal

Catch any change that breaks the workflow_call caller contract before merge, so consumers never see the silent-break-on-tag-bump pattern again.

Proposed approaches

Option A — Static contract diff (cheap, deterministic)

Add a CI job that diffs each .github/workflows/claude-*.yml file's workflow_call: block between PR head and main. Fail if:

  • A new secrets.<name> is added with required: true
  • A new inputs.<name> is added with required: true and no default:
  • An existing optional secrets.<name> is changed to required: true

Failures must be acknowledged either by:

  • Adding a BREAKING-CHANGES.md entry naming the affected workflow + the new requirement
  • A breaking-change-acknowledged label on the PR

Cost: small Python or jq script + a CI step. No live execution required.

Option B — Live workflow_call smoke test (more thorough)

Add a workflow_call-smoke.yml workflow inside this repo that, for each claude-*.yml reusable workflow, has a stub caller job using only the documented minimum (whatever the README/migration-guide tells consumers to pass). Run on every PR.

If a PR adds a new required secret or input without updating the stub callers, the smoke test fails. Forces the author to either:

  • Make the new requirement optional (preserve consumer compatibility), or
  • Update the stub callers + the consumer-facing docs/migration guide as part of the same PR.

Cost: one workflow_call-smoke.yml file (~50 lines), runs all reusable workflows on each PR. May need an org-secret carve-out if the smoke caller needs distinct test credentials.

Option C — Both

A is fast feedback (statically). B is true behavior verification. They complement: A catches the schema regression instantly; B catches semantic regressions A misses (e.g. a step inside the action gaining a new env-var dependency without a corresponding declared input).

Recommendation

Start with A — it's a deterministic gate, cheap to implement, catches the exact class of regression that bit us. Layer in B later if A misses cases.

Acceptance criteria

  • CI job runs on every PR touching .github/workflows/claude-*.yml
  • Job FAILs on any new required: true workflow_call secret/input without an acknowledgement (see Option A criteria)
  • README / consumer docs reference the contract — consumers can read what the minimum caller pattern is and what version it was last validated for
  • Self-test: a deliberately-bad PR (adds a new required: true secret without acknowledgement) FAILs the new gate

Context

🤖 Generated by Claude Code on behalf of @cbeaulieu-gt

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions