Cloud-SDK Plan #2 — PR 2: storage.s3 + step.s3_upload + aws.credentials + v1.1.0

CHANGELOG.md

@@ -0,0 +1,67 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is loosely based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
+This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [1.1.0] — 2026-05-15
+
+### Added
+
+- **`aws.credentials` standalone module** (`internal/modules/aws_credentials.go`):
+  optional DRY module that lets a config declare AWS credentials once and have
+  many sibling `storage.s3` / `step.s3_upload` modules reference them via
+  `credentials_ref:`. Backed by a process-local `credref` registry that rejects
+  duplicate names within a config.
+- **`storage.s3` standalone module** (`internal/modules/storage_s3.go`): the
+  S3-backed storage module, plugin-native via `IaCServeOptions.Modules`.
+  Credentials inline (`credentials:` sub-block) or `credentials_ref:` a
+  sibling `aws.credentials` module. Optional `endpoint` override (MinIO /
+  LocalStack via path-style addressing).
+- **`step.s3_upload` standalone step** (`internal/steps/s3_upload.go`):
+  pipeline step that uploads a base64-encoded body from a dot-path in the
+  pipeline context to S3 and returns `{url, key, bucket}` as step output.
+  Supports `{{ .field }}` / `{{ uuid }}` key templates and
+  `content_type_from` dot-path resolution. Plugin-native via
+  `IaCServeOptions.Steps`.
+- **`internal/awscreds.BuildAWSConfig`** — in-plugin AWS credential
+  resolution. Handles all 4 source paths from the YAML `credentials.type`
+  field: `static` (inline keys), `env` / `""` (aws-sdk-go-v2 default chain),
+  `profile` (shared-config profile via `config.WithSharedConfigProfile`),
+  `role_arn` (STS `AssumeRole` on top of base creds). Ports the SDK-bearing
+  resolver bodies from workflow core's `module/cloud_account_aws_creds.go`,
+  which the matched workflow-core change rewrites to declare-only markers.
+- **IaC provider credential path** now routes through `BuildAWSConfig` so a
+  `credentials:` sub-block on the IaC provider config honours all 4 source
+  paths (previously only inline static keys at top-level were recognised).
+- **`TestPluginJSONCapabilities_ModuleStep_Parity`** — host-conformance test
+  that asserts plugin.json `capabilities.moduleTypes` /
+  `capabilities.stepTypes` exactly match the providers wired into
+  `IaCServeOptions`.
+
+### Changed
+
+- **`plugin.json` `version`**: 1.0.0 → 1.1.0 (compatibility-marker minor bump
+  for the new module / step capabilities).
+- **`plugin.json` `minEngineVersion`**: 0.52.0 → 0.53.0 — requires workflow
+  v0.53.0+ for the `IaCServeOptions.Modules` / `.Steps` bridge wiring (the
+  plan-2 PR 1 SDK extension).
+- **`plugin.json` `capabilities.moduleTypes`**: adds `aws.credentials` and
+  `storage.s3` alongside the existing `iac.provider`.
+- **`plugin.json` `capabilities.stepTypes`**: adds `step.s3_upload`.
+- **`go.mod`** pins `github.com/GoCodeAlone/workflow v0.53.0`.
+
+### Notes
+
+- Phase-B core PR (workflow plan-2 Task 14/15) deletes in-core
+  `iac_state_spaces.go` and `s3_storage.go` / `pipeline_step_s3_upload.go`;
+  it is blocked on this release tag.
+- Runtime-launch validation transcript:
+  `docs/runtime-validation/aws-plugin-v1.1.0.md`.
+
+## [1.0.0] — earlier
+
+- Typed-IaC migration; baseline AWS provider surface (ECS / EKS / RDS /
+  ElastiCache / VPC / ALB / Route53 / ECR / API Gateway / SecurityGroup /
+  IAM / S3 / ACM / AutoScaling) + `iac.state s3` backend.

cmd/workflow-plugin-aws/main.go

@@ -16,5 +16,8 @@ import (
 )
 
 func main() {
-	sdk.ServeIaCPlugin(internal.NewIaCServer(), sdk.IaCServeOptions{})
+	sdk.ServeIaCPlugin(internal.NewIaCServer(), sdk.IaCServeOptions{
+		Modules: internal.ModuleProviders(),
+		Steps:   internal.StepProviders(),
+	})
 }

docs/runtime-validation/aws-plugin-v1.1.0.md

@@ -0,0 +1,87 @@
+# Runtime-launch validation — workflow-plugin-aws v1.1.0
+
+**Scope:** plan-2 PR 2 finishing task (Task 7) — wire `IaCServeOptions.Modules`
++ `.Steps` for `aws.credentials`, `storage.s3`, `step.s3_upload`; bump
+`plugin.json` `version` / `minEngineVersion`; release v1.1.0.
+
+**Change class:** plugin-loading path + version pin → runtime-launch
+validation per the cross-plan policy.
+
+## What was validated
+
+The plugin binary was built fresh from the branch HEAD and exercised as a
+go-plugin subprocess.
+
+### 1. Build
+
+```
+$ GOWORK=off go build -o /tmp/aws-plugin-v110/workflow-plugin-aws ./cmd/workflow-plugin-aws
+$ ls -la /tmp/aws-plugin-v110/workflow-plugin-aws
+-rwxr-xr-x  ... 184M ... workflow-plugin-aws
+```
+
+Build is clean (`BUILD_EXIT 0`); the binary is the standard ~184 MiB
+linked subprocess artifact.
+
+### 2. go-plugin handshake guard
+
+Running the binary outside the host process surfaces the canonical
+`go-plugin` self-identification message — proving `sdk.ServeIaCPlugin`
+wired the handshake correctly and the binary refuses to operate without a
+host-provided cookie/protocol exchange.
+
+```
+$ /tmp/aws-plugin-v110/workflow-plugin-aws
+This binary is a plugin. These are not meant to be executed directly.
+Please execute the program that consumes these plugins, which will
+load any plugins automatically
+```
+
+This is the go-plugin library's `ServeConfig`-rejection emission and
+demonstrates: (a) `sdk.ServeIaCPlugin` did not panic on the new
+`IaCServeOptions.Modules` / `.Steps` fields, (b) the bridge construction
+path completed successfully, (c) the host-or-nothing handshake guard is
+intact.
+
+### 3. In-process bridge parity (`go test ./internal/...`)
+
+The host-conformance parity tests build the same providers `main.go`
+wires into `IaCServeOptions` and assert the plugin.json declarations
+match exactly:
+
+- `TestPluginJSONCapabilities_ModuleStep_Parity` — plugin.json
+  `capabilities.moduleTypes` (minus the implicit `iac.provider`) ↔
+  `internal.ModuleProviders()` keys; `capabilities.stepTypes` ↔
+  `internal.StepProviders()` keys. Both bidirectional.
+- `TestCapabilityParity_IaCStateBackends` — pre-existing parity test
+  for the iac.state-backend capability surface.
+
+Both pass under `-race`.
+
+### 4. Full unit test suite
+
+```
+ok  github.com/GoCodeAlone/workflow-plugin-aws/drivers
+ok  github.com/GoCodeAlone/workflow-plugin-aws/internal
+ok  github.com/GoCodeAlone/workflow-plugin-aws/internal/awscreds
+ok  github.com/GoCodeAlone/workflow-plugin-aws/internal/credref
+ok  github.com/GoCodeAlone/workflow-plugin-aws/internal/modules
+ok  github.com/GoCodeAlone/workflow-plugin-aws/internal/statebackend
+ok  github.com/GoCodeAlone/workflow-plugin-aws/internal/steps
+ok  github.com/GoCodeAlone/workflow-plugin-aws/provider
+```
+
+All packages green under `GOWORK=off go test ./... -race`.
+
+## What was NOT validated here
+
+A full `wfctl plugin install <binary> && wfctl plugin list` end-to-end
+exercise was not run in this implementer session because the
+workflow-plugin-aws repo's CI does not bundle a wfctl binary — the
+shell-level handshake check + the in-process bridge parity tests are the
+canonical evidence in this repo. The full `wfctl`-driven host-load path
+is exercised by the workflow-core PR 1 integration tests (plan-2 Task 2)
+which were verified at the v0.53.0 tag this PR pins. PR 4 of plan-2
+(Phase B core deletion) is blocked on this release tag, so any regression
+in the host-load path surfaces at PR 4's CI before any in-core path is
+removed.

go.mod

@@ -3,7 +3,7 @@ module github.com/GoCodeAlone/workflow-plugin-aws
 go 1.26.0
 
 require (
-	github.com/GoCodeAlone/workflow v0.51.11-0.20260514225636-522748f35474
+	github.com/GoCodeAlone/workflow v0.53.0
 	github.com/aws/aws-sdk-go-v2 v1.41.7
 	github.com/aws/aws-sdk-go-v2/config v1.32.16
 	github.com/aws/aws-sdk-go-v2/credentials v1.19.15
@@ -21,6 +21,8 @@ require (
 	github.com/aws/aws-sdk-go-v2/service/rds v1.115.0
 	github.com/aws/aws-sdk-go-v2/service/route53 v1.62.5
 	github.com/aws/aws-sdk-go-v2/service/s3 v1.97.2
+	github.com/aws/aws-sdk-go-v2/service/sts v1.42.0
+	github.com/google/uuid v1.6.0
 	google.golang.org/grpc v1.80.0
 	google.golang.org/protobuf v1.36.11
 )
@@ -66,7 +68,6 @@ require (
 	github.com/aws/aws-sdk-go-v2/service/signin v1.0.10 // indirect
 	github.com/aws/aws-sdk-go-v2/service/sso v1.30.16 // indirect
 	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.20 // indirect
-	github.com/aws/aws-sdk-go-v2/service/sts v1.42.0 // indirect
 	github.com/aws/smithy-go v1.25.1 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/bits-and-blooms/bitset v1.24.4 // indirect
@@ -105,7 +106,6 @@ require (
 	github.com/golobby/cast v1.3.3 // indirect
 	github.com/google/btree v1.1.3 // indirect
 	github.com/google/s2a-go v0.1.9 // indirect
-	github.com/google/uuid v1.6.0 // indirect
 	github.com/googleapis/enterprise-certificate-proxy v0.3.14 // indirect
 	github.com/googleapis/gax-go/v2 v2.19.0 // indirect
 	github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674 // indirect

go.sum

@@ -44,8 +44,8 @@ github.com/GoCodeAlone/modular/modules/jsonschema v1.15.0 h1:xb1mI4NZkzvNKQ2F6nk
 github.com/GoCodeAlone/modular/modules/jsonschema v1.15.0/go.mod h1:hhGouwAVsonmJ4Lain4jINZ9nZCoc9l9eF3BHbmR8eE=
 github.com/GoCodeAlone/modular/modules/reverseproxy/v2 v2.8.0 h1:cvdLHbM/vzvygQTcAWSJsy+dAPzzwWyjzKMmTBFcFIo=
 github.com/GoCodeAlone/modular/modules/reverseproxy/v2 v2.8.0/go.mod h1:/9ipMG4qM2CHQ14BfXKdVlYRJelef6M8MFI5TbZv67M=
-github.com/GoCodeAlone/workflow v0.51.11-0.20260514225636-522748f35474 h1:C5Hi9BCtTDDP7k/++++LOfj2LxyaKP4YtgB0h5xgkeQ=
-github.com/GoCodeAlone/workflow v0.51.11-0.20260514225636-522748f35474/go.mod h1:L1kIOZqebO1WPriHXcqT7bg/uS7pExR8pOrWvurqhR4=
+github.com/GoCodeAlone/workflow v0.53.0 h1:+UjoWNHRB1aPiQfWJUltsXnZfupsqbGmItv9xZto4kY=
+github.com/GoCodeAlone/workflow v0.53.0/go.mod h1:L1kIOZqebO1WPriHXcqT7bg/uS7pExR8pOrWvurqhR4=
 github.com/GoCodeAlone/yaegi v0.17.2 h1:WK6Y6e0t1a6U7r+S2dN3CGWW1PizYD3zO0zneToZPxM=
 github.com/GoCodeAlone/yaegi v0.17.2/go.mod h1:z5Pr6Wse6QJcQvpgxTxzMAevFarH0N37TG88Y9dprx0=
 github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.32.0 h1:rIkQfkCOVKc1OiRCNcSDD8ml5RJlZbH/Xsq7lbpynwc=

internal/awscreds/awscreds.go

@@ -0,0 +1,162 @@
+// Package awscreds provides the in-plugin AWS credential resolution path.
+//
+// BuildAWSConfig is the single entry point: given a CredInput (parsed from
+// either a YAML `credentials:` block or a host-delivered CloudCredentials),
+// it returns a fully-resolved aws.Config. The 4 source paths are:
+//
+//   - "static": inline AccessKey/SecretKey/SessionToken;
+//   - "env" (or unset): aws-sdk-go-v2's default credential chain;
+//   - "profile": shared-config profile via config.WithSharedConfigProfile;
+//   - "role_arn": STS AssumeRole with optional ExternalID + base creds.
+//
+// The "profile" and "role_arn" SDK blocks are ported from workflow core's
+// module/cloud_account_aws_creds.go (awsProfileResolver, awsRoleARNResolver),
+// which Phase-B PR 4 (plan-2 Task 13) rewrites to *declare, don't resolve*.
+// The SDK-bearing resolution lives here, in the plugin.
+//
+// CredInput.Source MUST be populated by the call-site from the YAML
+// `credentials.type` field (the value the user wrote in their config). It
+// is NOT read from CloudAccount.Extra — that map never crosses the
+// host↔plugin gRPC boundary.
+package awscreds
+
+import (
+	"context"
+	"fmt"
+
+	"github.com/aws/aws-sdk-go-v2/aws"
+	"github.com/aws/aws-sdk-go-v2/config"
+	"github.com/aws/aws-sdk-go-v2/credentials"
+	"github.com/aws/aws-sdk-go-v2/service/sts"
+)
+
+// CredInput is the parsed-config shape BuildAWSConfig consumes. The call-site
+// (a Provider's CreateModule/CreateStep or the IaC provider's Initialize)
+// parses the `credentials:` YAML block — or the legacy top-level
+// access_key_id/secret_access_key keys — into this struct.
+type CredInput struct {
+	AccessKey    string
+	SecretKey    string
+	SessionToken string
+	Region       string
+	RoleARN      string
+	ExternalID   string
+	Profile      string
+	// Source mirrors the YAML `credentials.type` field — one of
+	// "static" | "env" | "profile" | "role_arn" | "" (default chain).
+	Source string
+	// SessionName is the STS AssumeRole session name. Defaults to
+	// "workflow-session" when empty. Honoured only when Source == "role_arn".
+	SessionName string
+}
+
+// stsAssumeRoleAPI is the subset of *sts.Client BuildAWSConfig calls. It
+// exists so tests can inject a fake STS implementation without spinning up
+// a real STS endpoint.
+type stsAssumeRoleAPI interface {
+	AssumeRole(ctx context.Context, in *sts.AssumeRoleInput, opts ...func(*sts.Options)) (*sts.AssumeRoleOutput, error)
+}
+
+// newSTSClient builds an STS client from the given base aws.Config. Tests
+// override this var to inject a fake STS API.
+var newSTSClient = func(cfg aws.Config) stsAssumeRoleAPI {
+	return sts.NewFromConfig(cfg)
+}
+
+// BuildAWSConfig returns a resolved aws.Config for the given CredInput.
+// See package doc for the source-path semantics.
+func BuildAWSConfig(ctx context.Context, c CredInput) (aws.Config, error) {
+	switch c.Source {
+	case "profile":
+		return loadProfile(ctx, c)
+	case "role_arn":
+		return loadRoleARN(ctx, c)
+	}
+	// "static" | "env" | "" — all flow through the default chain with
+	// optional static-credential override when both keys are supplied.
+	opts := baseLoadOptions(c)
+	if c.AccessKey != "" && c.SecretKey != "" {
+		opts = append(opts, config.WithCredentialsProvider(
+			credentials.NewStaticCredentialsProvider(c.AccessKey, c.SecretKey, c.SessionToken),
+		))
+	}
+	cfg, err := config.LoadDefaultConfig(ctx, opts...)
+	if err != nil {
+		return aws.Config{}, fmt.Errorf("aws creds: load default config: %w", err)
+	}
+	return cfg, nil
+}
+
+// loadProfile loads aws.Config from a named shared-config profile. Ported
+// from workflow core's awsProfileResolver (cloud_account_aws_creds.go).
+func loadProfile(ctx context.Context, c CredInput) (aws.Config, error) {
+	profile := c.Profile
+	if profile == "" {
+		profile = "default"
+	}
+	opts := baseLoadOptions(c)
+	opts = append(opts, config.WithSharedConfigProfile(profile))
+	cfg, err := config.LoadDefaultConfig(ctx, opts...)
+	if err != nil {
+		return aws.Config{}, fmt.Errorf("aws creds: load profile %q: %w", profile, err)
+	}
+	return cfg, nil
+}
+
+// loadRoleARN obtains temporary credentials via STS AssumeRole on top of
+// base credentials (inline static keys when supplied, otherwise the default
+// chain). Ported from workflow core's awsRoleARNResolver
+// (cloud_account_aws_creds.go).
+func loadRoleARN(ctx context.Context, c CredInput) (aws.Config, error) {
+	if c.RoleARN == "" {
+		return aws.Config{}, fmt.Errorf("aws creds: role_arn source requires non-empty RoleARN")
+	}
+	baseOpts := baseLoadOptions(c)
+	if c.AccessKey != "" && c.SecretKey != "" {
+		baseOpts = append(baseOpts, config.WithCredentialsProvider(
+			credentials.NewStaticCredentialsProvider(c.AccessKey, c.SecretKey, c.SessionToken),
+		))
+	}
+	baseCfg, err := config.LoadDefaultConfig(ctx, baseOpts...)
+	if err != nil {
+		return aws.Config{}, fmt.Errorf("aws creds: load base config for role_arn: %w", err)
+	}
+
+	sessionName := c.SessionName
+	if sessionName == "" {
+		sessionName = "workflow-session"
+	}
+	input := &sts.AssumeRoleInput{
+		RoleArn:         aws.String(c.RoleARN),
+		RoleSessionName: aws.String(sessionName),
+	}
+	if c.ExternalID != "" {
+		input.ExternalId = aws.String(c.ExternalID)
+	}
+
+	out, err := newSTSClient(baseCfg).AssumeRole(ctx, input)
+	if err != nil {
+		return aws.Config{}, fmt.Errorf("aws creds: AssumeRole %q: %w", c.RoleARN, err)
+	}
+	if out == nil || out.Credentials == nil {
+		return aws.Config{}, fmt.Errorf("aws creds: AssumeRole %q returned no credentials", c.RoleARN)
+	}
+
+	assumed := baseCfg.Copy()
+	assumed.Credentials = credentials.NewStaticCredentialsProvider(
+		aws.ToString(out.Credentials.AccessKeyId),
+		aws.ToString(out.Credentials.SecretAccessKey),
+		aws.ToString(out.Credentials.SessionToken),
+	)
+	return assumed, nil
+}
+
+// baseLoadOptions builds the LoadDefaultConfig options common to every
+// path — currently just Region when set.
+func baseLoadOptions(c CredInput) []func(*config.LoadOptions) error {
+	var opts []func(*config.LoadOptions) error
+	if c.Region != "" {
+		opts = append(opts, config.WithRegion(c.Region))
+	}
+	return opts
+}