Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,13 @@ Configs that still reference the legacy types now fail to load with an actionabl

### Added

- **GitHub environment secret destinations for IaC output sync**: `secretStores` can now use
`provider: github` with `config.environment`, including `${WORKFLOW_ENV}`, so
`wfctl infra apply --env staging` writes to GitHub Actions environment secrets instead of
repository secrets.
- **`secrets.generate[].store` routing**: `infra_output` generators can name a store from
`secretStores`, allowing provisioning output such as database URLs to be piped directly into
the intended secret manager location after apply.
- **Engine-side sensitive-output routing** (v0.27.0): `ResourceDriver` outputs flagged with
`Sensitive: {key: true}` on Create/Update are routed through the configured `secrets.Provider`
and replaced in state with `secret_ref://<resource>_<key>` placeholders. Plugins remain
Expand Down
22 changes: 19 additions & 3 deletions cmd/wfctl/dsl-reference-embedded.md
Original file line number Diff line number Diff line change
Expand Up @@ -944,8 +944,8 @@ The optional `secretStores:` section declares named secret storage backends. Thi
### Fields

- `secretStores.<name>` (object) — a named store. Fields:
- `provider` (string, required) — backend provider: `env`, `vault`, `aws-secrets-manager`, `gcp-secret-manager`
- `config` (map) — provider-specific configuration (e.g., Vault address, AWS region)
- `provider` (string, required) — backend provider: `env`, `vault`, `aws-secrets-manager`, `gcp-secret-manager`, `github`
- `config` (map) — provider-specific configuration (e.g., Vault address, AWS region). GitHub supports `repo`, `token_env`, and optional `environment` for environment-scoped Actions secrets.

### Example

Expand All @@ -957,6 +957,12 @@ secretStores:
provider: aws-secrets-manager
config:
region: us-east-1
github-env:
provider: github
config:
repo: GoCodeAlone/example-app
token_env: GH_MANAGEMENT_TOKEN
environment: ${WORKFLOW_ENV}
```

### Relationship to Other Sections
Expand All @@ -975,7 +981,7 @@ The optional `secrets:` section declares the application's secret management con
### Fields

- `secrets.defaultStore` (string) — name of the default store from `secretStores`. When set, all secrets without an explicit `store` field use this store.
- `secrets.provider` (string) — legacy single-provider name (use `defaultStore` + `secretStores` for new configs). Supported: `env`, `vault`, `aws-secrets-manager`, `gcp-secret-manager`
- `secrets.provider` (string) — legacy single-provider name (use `defaultStore` + `secretStores` for new configs). Supported: `env`, `vault`, `aws-secrets-manager`, `gcp-secret-manager`, `github`
- `secrets.config` (map) — provider-specific configuration (used with legacy `provider` field)
- `secrets.rotation` (object) — default rotation policy:
- `enabled` (bool) — enable automatic rotation
Expand All @@ -986,6 +992,11 @@ The optional `secrets:` section declares the application's secret management con
- `description` (string) — human-readable description
- `store` (string) — name of a specific store from `secretStores`; overrides `defaultStore` and environment override
- `rotation` (object) — per-secret rotation override (same fields as `secrets.rotation`)
- `secrets.generate` (array) — secrets generated by `wfctl infra bootstrap` or synced by `wfctl infra apply`. Each entry:
- `key` (string, required) — target secret name
- `type` (string, required) — generator type, including `random_hex`, `random_base64`, `random_alphanumeric`, `provider_credential`, or `infra_output`
- `source` (string) — for `infra_output`, `<module>.<output>` copied from applied IaC state
- `store` (string) — optional named store from `secretStores` for `infra_output` entries; lets provisioning output pipe directly into a specific secret manager location

### Example (multi-store)

Expand All @@ -1012,6 +1023,11 @@ secrets:
- name: STRIPE_SECRET_KEY
description: Stripe payment API key
store: payment-vault
generate:
- key: DATABASE_URL
type: infra_output
source: main-db.uri
store: github-env
```

### Example (single provider, legacy)
Expand Down
13 changes: 6 additions & 7 deletions cmd/wfctl/infra.go
Original file line number Diff line number Diff line change
Expand Up @@ -1518,14 +1518,13 @@ func runInfraApply(args []string) error {
if err != nil {
return fmt.Errorf("load current state for infra_output sync: %w", err)
}
// Only reload the workflow config when env resolution is actually needed:
// it is needed only when --env is set AND at least one infra_output secret
// generator is configured (otherwise syncInfraOutputSecrets is a no-op for
// env resolution regardless).
// Only reload the workflow config when routing or env resolution is needed:
// store-scoped generators need secretStores, and --env needs module
// ResolveForEnv so bmw-database.uri can find bmw-staging-db in state.
var wfCfg *config.WorkflowConfig
if envName != "" {
for _, g := range secretsCfg.Generate {
if g.Type == "infra_output" {
for _, g := range secretsCfg.Generate {
if g.Type == "infra_output" {
if envName != "" || g.Store != "" {
var loadErr error
wfCfg, loadErr = config.LoadFromFile(cfgFile)
if loadErr != nil {
Expand Down
85 changes: 61 additions & 24 deletions cmd/wfctl/infra_output_secrets.go
Original file line number Diff line number Diff line change
Expand Up @@ -146,35 +146,28 @@ func syncInfraOutputSecrets(ctx context.Context, secretsCfg *SecretsConfig, prov

// Lazy List() cache — same pattern as bootstrapSecrets for write-only
// providers (GitHub Actions) that return ErrUnsupported on Get.
var listSet map[string]struct{}
var listErr error
var listDone bool
lookupViaList := func(key string) (bool, error) {
if !listDone {
names, err := provider.List(ctx)
listErr = err
if err == nil {
listSet = make(map[string]struct{}, len(names))
for _, n := range names {
listSet[n] = struct{}{}
}
}
listDone = true
}
if listErr != nil && !errors.Is(listErr, secrets.ErrUnsupported) {
return false, fmt.Errorf("list secrets to check %q: %w", key, listErr)
}
_, ok := listSet[key]
return ok, nil
}
listLookups := map[secrets.Provider]*providerListLookup{}

stateOutputs := buildStateOutputsMap(states)

for _, gen := range gens {
genProvider, err := providerForSecretGen(wfCfg, provider, gen, envName)
if err != nil {
return err
}
lookupViaList := func(key string) (bool, error) {
lookup, ok := listLookups[genProvider]
if !ok {
lookup = &providerListLookup{provider: genProvider}
listLookups[genProvider] = lookup
}
return lookup.exists(ctx, key)
}
Comment on lines +149 to +165

// Attempt to read the current value. This serves two purposes:
// 1. Existence check for readable providers.
// 2. Value comparison in refresh mode to avoid spurious updates.
currentVal, getErr := provider.Get(ctx, gen.Key)
currentVal, getErr := genProvider.Get(ctx, gen.Key)

var exists bool
var isReadable bool
Expand Down Expand Up @@ -212,16 +205,60 @@ func syncInfraOutputSecrets(ctx context.Context, secretsCfg *SecretsConfig, prov
fmt.Printf(" secret %q: unchanged\n", gen.Key)
continue
}
if err := provider.Set(ctx, gen.Key, newValue); err != nil {
if err := genProvider.Set(ctx, gen.Key, newValue); err != nil {
return fmt.Errorf("store secret %q: %w", gen.Key, err)
}
fmt.Printf(" secret %q: updated from infra output\n", gen.Key)
} else {
if err := provider.Set(ctx, gen.Key, newValue); err != nil {
if err := genProvider.Set(ctx, gen.Key, newValue); err != nil {
return fmt.Errorf("store secret %q: %w", gen.Key, err)
}
fmt.Printf(" secret %q: created from infra output\n", gen.Key)
}
}
return nil
}

type providerListLookup struct {
provider secrets.Provider
listSet map[string]struct{}
listErr error
listDone bool
}

func (l *providerListLookup) exists(ctx context.Context, key string) (bool, error) {
if !l.listDone {
names, err := l.provider.List(ctx)
l.listErr = err
if err == nil {
l.listSet = make(map[string]struct{}, len(names))
for _, n := range names {
l.listSet[n] = struct{}{}
}
}
l.listDone = true
}
if l.listErr != nil && !errors.Is(l.listErr, secrets.ErrUnsupported) {
return false, fmt.Errorf("list secrets to check %q: %w", key, l.listErr)
}
_, ok := l.listSet[key]
return ok, nil
}

func providerForSecretGen(wfCfg *config.WorkflowConfig, fallback secrets.Provider, gen SecretGen, envName string) (secrets.Provider, error) {
if gen.Store == "" {
return fallback, nil
}
if wfCfg == nil {
return nil, fmt.Errorf("secret %q references store %q, but workflow config is unavailable", gen.Key, gen.Store)
}
store, ok := wfCfg.SecretStores[gen.Store]
if !ok || store == nil {
return nil, fmt.Errorf("secret %q references unknown store %q", gen.Key, gen.Store)
}
provider, err := resolveSecretsProviderForEnv(secretsConfigFromStore(store), envName)
if err != nil {
return nil, fmt.Errorf("resolve store %q for secret %q: %w", gen.Store, gen.Key, err)
}
return provider, nil
}
32 changes: 32 additions & 0 deletions cmd/wfctl/infra_output_secrets_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,10 @@ package main
import (
"context"
"fmt"
"os"
"testing"

"github.com/GoCodeAlone/workflow/config"
"github.com/GoCodeAlone/workflow/interfaces"
"github.com/GoCodeAlone/workflow/secrets"
)
Expand Down Expand Up @@ -163,6 +165,36 @@ func TestSyncInfraOutputSecrets_WritesSecret(t *testing.T) {
}
}

func TestSyncInfraOutputSecrets_RoutesGeneratorToNamedStore(t *testing.T) {
envProvider := secrets.NewEnvProvider("ROUTED_")
_ = envProvider.Delete(context.Background(), "DATABASE_URL")
t.Cleanup(func() { _ = envProvider.Delete(context.Background(), "DATABASE_URL") })
p := newSimpleProvider()
cfg := &SecretsConfig{
Generate: []SecretGen{
{Key: "DATABASE_URL", Type: "infra_output", Source: "bmw-database.uri", Store: "github-env"},
},
}
wfCfg := &config.WorkflowConfig{
SecretStores: map[string]*config.SecretStoreConfig{
"github-env": {
Provider: "env",
Config: map[string]any{"prefix": "ROUTED_"},
},
},
}
err := syncInfraOutputSecrets(context.Background(), cfg, p, sampleStates(), wfCfg, "staging", nil, false)
if err != nil {
t.Fatalf("syncInfraOutputSecrets: %v", err)
}
if _, ok := p.data["DATABASE_URL"]; ok {
t.Fatalf("default provider received DATABASE_URL; wanted named store routing")
}
if got := os.Getenv("ROUTED_DATABASE_URL"); got != "postgres://user:pass@db.example.com:5432/app" {
t.Fatalf("ROUTED_DATABASE_URL = %q, want infra output", got)
}
}

func TestSyncInfraOutputSecrets_WritesMultiple(t *testing.T) {
p := newSimpleProvider()
cfg := &SecretsConfig{
Expand Down
74 changes: 68 additions & 6 deletions cmd/wfctl/infra_secrets.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package main
import (
"fmt"
"os"
"strings"

"github.com/GoCodeAlone/workflow/config"
"github.com/GoCodeAlone/workflow/secrets"
Expand Down Expand Up @@ -48,15 +49,19 @@ func parseInfraConfig(cfgFile string) (*InfraConfig, error) {
}

// resolveSecretsProvider constructs the appropriate secrets.Provider from cfg.
// ${VAR} / $VAR references in cfg.Config are expanded via os.ExpandEnv before
// the provider is constructed, so credentials can be passed through the environment
// (e.g. VAULT_TOKEN=s.xxx in CI) rather than hard-coded in YAML. cfg.Config is
// never mutated — expansion produces a deep copy.
// ${VAR} / $VAR references in cfg.Config are expanded before the provider is
// constructed, so credentials can be passed through the environment (e.g.
// VAULT_TOKEN=s.xxx in CI) rather than hard-coded in YAML. cfg.Config is never
// mutated — expansion produces a deep copy.
func resolveSecretsProvider(cfg *SecretsConfig) (secrets.Provider, error) {
return resolveSecretsProviderForEnv(cfg, "")
}

func resolveSecretsProviderForEnv(cfg *SecretsConfig, envName string) (secrets.Provider, error) {
if cfg == nil {
return nil, fmt.Errorf("no secrets config provided")
}
c := config.ExpandEnvInMap(cfg.Config)
c := expandSecretsConfigForEnv(cfg.Config, envName)
if c == nil {
c = map[string]any{}
}
Expand All @@ -67,7 +72,14 @@ func resolveSecretsProvider(cfg *SecretsConfig) (secrets.Provider, error) {
if tokenVar == "" {
tokenVar = "GITHUB_TOKEN" //nolint:gosec // G101: this is an env var name, not a credential
}
return secrets.NewGitHubSecretsProvider(repo, tokenVar)
provider, err := secrets.NewGitHubSecretsProvider(repo, tokenVar)
if err != nil {
return nil, err
}
if environment, _ := c["environment"].(string); environment != "" {
provider.SetEnvironment(environment)
}
return provider, nil

case "vault":
addr, _ := c["address"].(string)
Expand Down Expand Up @@ -101,6 +113,56 @@ func resolveSecretsProvider(cfg *SecretsConfig) (secrets.Provider, error) {
}
}

func expandSecretsConfigForEnv(m map[string]any, envName string) map[string]any {
if m == nil {
return nil
}
out := make(map[string]any, len(m))
for k, v := range m {
out[k] = expandSecretConfigValueForEnv(v, envName)
}
return out
}

func expandSecretConfigValueForEnv(v any, envName string) any {
switch val := v.(type) {
case string:
return os.Expand(val, func(key string) string {
if key == "WORKFLOW_ENV" {
return envName
}
return os.Getenv(key)
})
case map[string]any:
return expandSecretsConfigForEnv(val, envName)
case []any:
out := make([]any, len(val))
for i, item := range val {
out[i] = expandSecretConfigValueForEnv(item, envName)
}
return out
default:
return v
}
}

func secretsConfigFromStore(store *config.SecretStoreConfig) *SecretsConfig {
if store == nil {
return nil
}
provider := strings.TrimSpace(store.Provider)
switch provider {
case "aws-secrets-manager":
provider = "aws"
case "github-actions":
provider = "github"
}
return &SecretsConfig{
Provider: provider,
Config: store.Config,
}
}

// buildAdhocProvider constructs a secrets.Provider for ad-hoc operations without
// requiring an app.yaml secrets block. Supports keychain, env, and aws.
// vault and github require explicit config via the app.yaml secrets block.
Expand Down
23 changes: 23 additions & 0 deletions cmd/wfctl/infra_secrets_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,29 @@ func TestResolveSecretsProvider_GitHubProvider(t *testing.T) {
}
}

func TestResolveSecretsProvider_GitHubProviderEnvironmentScope(t *testing.T) {
t.Setenv("GH_TOKEN_TEST", "fake-token")
cfg := &SecretsConfig{
Provider: "github",
Config: map[string]any{
"repo": "owner/repo",
"token_env": "GH_TOKEN_TEST",
"environment": "${WORKFLOW_ENV}",
},
}
p, err := resolveSecretsProviderForEnv(cfg, "staging")
if err != nil {
t.Fatalf("resolveSecretsProviderForEnv github: %v", err)
}
gh, ok := p.(*secrets.GitHubSecretsProvider)
if !ok {
t.Fatalf("provider type = %T, want *secrets.GitHubSecretsProvider", p)
}
if gh.Environment() != "staging" {
t.Errorf("github environment = %q, want staging", gh.Environment())
}
}

func TestResolveSecretsProvider_UnknownProvider(t *testing.T) {
cfg := &SecretsConfig{Provider: "mystery"}
_, err := resolveSecretsProvider(cfg)
Expand Down
1 change: 1 addition & 0 deletions config/secrets_config.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ type SecretGen struct {
Length int `json:"length,omitempty" yaml:"length,omitempty"` // for random generators
Source string `json:"source,omitempty" yaml:"source,omitempty"` // for provider_credential
Name string `json:"name,omitempty" yaml:"name,omitempty"` // optional human-readable label
Store string `json:"store,omitempty" yaml:"store,omitempty"` // optional named store for infra_output sync
}

// SecretsConfig defines secret management for the application.
Expand Down
Loading
Loading