feat: add KMS-backed master key encryption at rest

.env.example

@@ -20,11 +20,33 @@ METRICS_ENABLED=true
 METRICS_NAMESPACE=secrets
 
 # Master keys (Envelope Encryption)
-# Generate a new master key using: ./bin/app create-master-key
+# Two modes available: KMS Mode (recommended) or Legacy Mode (development only)
+#
+# === KMS MODE (RECOMMENDED FOR PRODUCTION) ===
+# Encrypts master keys at rest using external Key Management Service
+# Generate a new KMS master key using: ./bin/app create-master-key --kms-provider=<provider> --kms-key-uri=<uri>
+# Rotate master keys using: ./bin/app rotate-master-key --id=<new-key-id>
+#
+# KMS Providers:
+# - localsecrets: Local testing (base64key://<32-byte-base64-key>)
+# - gcpkms: Google Cloud KMS (gcpkms://projects/<project>/locations/<location>/keyRings/<ring>/cryptoKeys/<key>)
+# - awskms: AWS KMS (awskms:///<key-id> or awskms:///<alias>)
+# - azurekeyvault: Azure Key Vault (azurekeyvault://<vault-name>.vault.azure.net/keys/<key-name>)
+# - hashivault: HashiCorp Vault (hashivault:///<path>)
+#
+# KMS_PROVIDER=localsecrets
+# KMS_KEY_URI=base64key://smGbjm71Nxd1Ig5FS0wj9SlbzAIrnolCz9bQQ6uAhl4=
+# MASTER_KEYS=default:ARiEeAASDiXKAxzOQCw2NxQfrHAc33CPP/7SsvuVjVvq1olzRBudplPoXRkquRWUXQ+CnEXi15LACqXuPGszLS+anJUrdn04
+# ACTIVE_MASTER_KEY_ID=default
+#
+# === LEGACY MODE (DEVELOPMENT ONLY) ===
+# ⚠️ SECURITY WARNING: Master keys stored as plaintext base64 - USE ONLY FOR DEVELOPMENT
 # Each key must be exactly 32 bytes (256 bits), base64-encoded
 # Format: id1:base64key1,id2:base64key2 (comma-separated for multiple keys)
-# ⚠️ SECURITY WARNING: Store master keys in secrets manager in production
-# Never commit master keys to source control
+# Generate a new plaintext master key using: ./bin/app create-master-key
+#
+KMS_PROVIDER=
+KMS_KEY_URI=
 MASTER_KEYS=default:bEu+O/9NOFAsWf1dhVB9aprmumKhhBcE6o7UPVmI43Y=
 ACTIVE_MASTER_KEY_ID=default
 

AGENTS.md

@@ -1057,6 +1057,181 @@ func (s *Server) SetupRouter(
 
 **Reference:** `/internal/http/server.go` (SetupRouter method)
 
+## KMS Service Implementation
+
+The project supports KMS (Key Management Service) integration for encrypting master keys at rest using external providers. KMS functionality follows interface segregation principles with the domain layer defining minimal interfaces and the service layer providing concrete implementations.
+
+### Interface Segregation Pattern
+
+**Domain Layer** (`internal/crypto/domain/master_key.go`):
+```go
+// Minimal interfaces defined by domain - no external dependencies
+type KMSService interface {
+    OpenKeeper(ctx context.Context, keyURI string) (KMSKeeper, error)
+}
+
+type KMSKeeper interface {
+    Decrypt(ctx context.Context, ciphertext []byte) ([]byte, error)
+    Close() error
+}
+```
+
+**Service Layer** (`internal/crypto/service/kms_service.go`):
+- Implements `KMSService` using `gocloud.dev/secrets`
+- Imports all KMS provider drivers (gcpkms, awskms, azurekeyvault, hashivault, localsecrets)
+- Returns `*secrets.Keeper` which naturally implements `KMSKeeper` (duck typing)
+
+**Type Compatibility:**
+- `*secrets.Keeper` from gocloud.dev implements both `Decrypt()` and `Close()` methods
+- No wrapper types needed - direct type assertion works in implementation code
+
+**Reference:** `/internal/crypto/service/kms_service.go` and `/internal/crypto/domain/master_key.go:114-128`
+
+### Testing with localsecrets Provider
+
+**Always use `localsecrets` provider for tests** - no external dependencies or credentials required.
+
+**Generate test KMS key:**
+```go
+func generateLocalSecretsKMSKey(t *testing.T) string {
+    t.Helper()
+    key := make([]byte, 32)
+    _, err := rand.Read(key)
+    require.NoError(t, err)
+    return "base64key://" + base64.URLEncoding.EncodeToString(key)
+}
+```
+
+**Type assertion for Encrypt method** (not part of domain interface):
+```go
+keeperInterface, err := kmsService.OpenKeeper(ctx, kmsKeyURI)
+require.NoError(t, err)
+
+// Type assert to access Encrypt method for tests
+keeper, ok := keeperInterface.(*secrets.Keeper)
+require.True(t, ok, "keeper should be *secrets.Keeper")
+
+ciphertext, err := keeper.Encrypt(ctx, plaintext)
+```
+
+**Mock implementations must return copies** to avoid issues when ciphertext is zeroed:
+```go
+// BAD - returns slice of input (will be zeroed)
+func (m *MockKMSKeeper) Decrypt(ctx context.Context, ciphertext []byte) ([]byte, error) {
+    return ciphertext, nil
+}
+
+// GOOD - returns a copy
+func (m *MockKMSKeeper) Decrypt(ctx context.Context, ciphertext []byte) ([]byte, error) {
+    result := make([]byte, len(ciphertext))
+    copy(result, ciphertext)
+    return result, nil
+}
+```
+
+**Reference:** `/internal/crypto/service/kms_service_test.go` and `/test/integration/api_test.go` (KMS helpers)
+
+### Error Handling for Close() Calls
+
+**All `Close()` calls MUST check errors** (enforced by golangci-lint errcheck).
+
+**Production code pattern** (with logging):
+```go
+defer func() {
+    if closeErr := keeper.Close(); closeErr != nil {
+        logger.Error("failed to close KMS keeper", slog.Any("error", closeErr))
+    }
+}()
+```
+
+**Test code pattern** (with assertions):
+```go
+defer func() {
+    assert.NoError(t, keeper.Close())
+}()
+```
+
+**CLI code pattern** (with user-facing message):
+```go
+defer func() {
+    if closeErr := keeperInterface.Close(); closeErr != nil {
+        fmt.Printf("Warning: failed to close KMS keeper: %v\n", closeErr)
+    }
+}()
+```
+
+**Reference:** `/internal/crypto/domain/master_key.go:213-217` and `/cmd/app/commands/master_key.go:58-62`
+
+### Memory Safety and Performance
+
+**Startup-only decryption:**
+- KMS operations happen only at application startup
+- Master keys decrypted into memory once via `LoadMasterKeyChain()`
+- No per-operation KMS calls (performance optimization)
+
+**Memory cleanup:**
+- Master key zeroing handled by existing `MasterKeyChain.Close()`
+- KEK chain similarly zeroed via `KekChain.Close()`
+- No additional cleanup needed for KMS-decrypted keys
+
+**Ownership transfer:**
+- Decrypted key data ownership transfers to `MasterKeyChain`
+- Original slices can be safely reused by KMS keeper
+- Domain layer makes defensive copies when needed
+
+**Reference:** `/internal/crypto/domain/master_key.go:183-285` (loadMasterKeyChainFromKMS)
+
+### URI Masking for Security
+
+**Use `maskKeyURI()` to redact sensitive URI components in logs:**
+
+```go
+maskedURI := maskKeyURI(cfg.KMSKeyURI)
+logger.Info("opening KMS keeper",
+    slog.String("kms_provider", cfg.KMSProvider),
+    slog.String("kms_key_uri", maskedURI),
+)
+```
+
+**Masking examples:**
+- `gcpkms://projects/my-project/...` → `gcpkms://projects/***/...`
+- `awskms://key-id-123?region=us-east-1` → `awskms://***?region=us-east-1`
+- `azurekeyvault://vault.azure.net/keys/mykey` → `azurekeyvault://***`
+- `base64key://c2VjcmV0a2V5` → `base64key://***`
+
+**Purpose:**
+- Prevents sensitive key identifiers from appearing in logs
+- Preserves provider type and structure for debugging
+- Retains query parameters (e.g., region) that are not sensitive
+
+**Reference:** `/internal/crypto/domain/master_key.go:130-181` (maskKeyURI function)
+
+### Auto-Detection Mode
+
+**KMS vs Legacy mode determined by environment variables:**
+
+```go
+// KMS mode: both KMS_PROVIDER and KMS_KEY_URI must be set
+if cfg.KMSProvider != "" && cfg.KMSKeyURI != "" {
+    return loadMasterKeyChainFromKMS(ctx, cfg, kmsService, logger)
+}
+
+// Legacy mode: neither should be set
+if cfg.KMSProvider == "" && cfg.KMSKeyURI == "" {
+    return LoadMasterKeyChainFromEnv()
+}
+
+// Error: inconsistent configuration
+return ErrKMSProviderNotSet or ErrKMSKeyURINotSet
+```
+
+**Validation:**
+- Fail fast on inconsistent configuration (one set, one empty)
+- Clear error messages indicating which variable is missing
+- No silent fallbacks - explicit mode selection
+
+**Reference:** `/internal/crypto/domain/master_key.go:287-315` (LoadMasterKeyChain)
+
 ## See also
 
 - [Repository README](README.md)

CHANGELOG.md

@@ -5,6 +5,26 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.6.0] - 2026-02-19
+
+### Added
+- Added KMS-backed master key support with `KMS_PROVIDER` and `KMS_KEY_URI`
+- Added `rotate-master-key` CLI command for staged master key rotation
+- Added `create-master-key` KMS flags: `--kms-provider` and `--kms-key-uri`
+- Added gocloud-based KMS service support for `localsecrets`, Google Cloud KMS, AWS KMS, Azure Key Vault, and HashiCorp Vault
+
+### Changed
+- Master key loading now auto-detects KMS mode vs legacy mode and validates KMS configuration consistency at startup
+
+### Security
+- Added encrypted-at-rest master key workflow through external KMS providers
+- Added startup validation and error paths for incomplete KMS configuration and decryption failures
+
+### Documentation
+- Added `docs/releases/v0.6.0.md` release notes and `docs/releases/v0.6.0-upgrade.md` upgrade guide
+- Added KMS operations guide: `docs/operations/kms-setup.md`
+- Updated CLI and environment variable docs for KMS configuration and master key rotation workflows
+
 ## [0.5.1] - 2026-02-19
 
 ### Fixed

README.md

@@ -13,10 +13,10 @@ Secrets is inspired by **HashiCorp Vault** ❤️, but it is intentionally **muc
 The default way to run Secrets is the published Docker image:
 
 ```bash
-docker pull allisson/secrets:v0.5.1
+docker pull allisson/secrets:v0.6.0
 ```
 
-Use pinned tags for reproducible setups. `latest` is also available for fast iteration.
+Use pinned tags for reproducible setups. `latest` is available for dev-only fast iteration.
 
 Docs release/API metadata source: `docs/metadata.json`.
 
@@ -29,20 +29,21 @@ Then follow the Docker setup guide in [docs/getting-started/docker.md](docs/gett
 1. 🐳 **Run with Docker image (recommended)**: [docs/getting-started/docker.md](docs/getting-started/docker.md)
 2. 💻 **Run locally for development**: [docs/getting-started/local-development.md](docs/getting-started/local-development.md)
 
-## 🆕 What's New in v0.5.1
+## 🆕 What's New in v0.6.0
 
-- 🛠️ Fixed master key loading to preserve usable key material while zeroing temporary decoded buffers
-- 🧹 Hardened keychain teardown to zero in-memory master keys before clearing chain state
-- 🔒 Expanded regression coverage for master key memory lifecycle and close behavior
-- 📘 Added release notes: [docs/releases/v0.5.1.md](docs/releases/v0.5.1.md)
-- ⬆️ Added upgrade guide: [docs/releases/v0.5.1-upgrade.md](docs/releases/v0.5.1-upgrade.md)
-- 📦 Updated pinned Docker docs/examples to `allisson/secrets:v0.5.1`
+- ☁️ Added KMS integration for master key encryption at rest (`KMS_PROVIDER`, `KMS_KEY_URI`)
+- 🔁 Added `rotate-master-key` CLI command for safer master key lifecycle operations
+- 🧭 Added provider-specific KMS setup and migration runbook documentation
+- ✅ Added KMS migration checklist: [docs/operations/kms-migration-checklist.md](docs/operations/kms-migration-checklist.md)
+- 📘 Added release notes: [docs/releases/v0.6.0.md](docs/releases/v0.6.0.md)
+- ⬆️ Added upgrade guide: [docs/releases/v0.6.0-upgrade.md](docs/releases/v0.6.0-upgrade.md)
+- 📦 Updated pinned Docker docs/examples to `allisson/secrets:v0.6.0`
 
 Release history quick links:
 
-- Current: [v0.5.1 release notes](docs/releases/v0.5.1.md)
-- Previous: [v0.5.0 release notes](docs/releases/v0.5.0.md)
-- Previous upgrade guide: [v0.5.0 upgrade guide](docs/releases/v0.5.0-upgrade.md)
+- Current: [v0.6.0 release notes](docs/releases/v0.6.0.md)
+- Previous: [v0.5.1 release notes](docs/releases/v0.5.1.md)
+- Previous upgrade guide: [v0.5.1 upgrade guide](docs/releases/v0.5.1-upgrade.md)
 
 ## 📚 Docs Map
 
@@ -53,26 +54,28 @@ Release history quick links:
 - 🧰 **Troubleshooting**: [docs/getting-started/troubleshooting.md](docs/getting-started/troubleshooting.md)
 - ✅ **Smoke test script**: [docs/getting-started/smoke-test.md](docs/getting-started/smoke-test.md)
 - 🧪 **CLI commands reference**: [docs/cli/commands.md](docs/cli/commands.md)
-- 🚀 **v0.5.1 release notes**: [docs/releases/v0.5.1.md](docs/releases/v0.5.1.md)
-- ⬆️ **v0.5.1 upgrade guide**: [docs/releases/v0.5.1-upgrade.md](docs/releases/v0.5.1-upgrade.md)
+- 🚀 **v0.6.0 release notes**: [docs/releases/v0.6.0.md](docs/releases/v0.6.0.md)
+- ⬆️ **v0.6.0 upgrade guide**: [docs/releases/v0.6.0-upgrade.md](docs/releases/v0.6.0-upgrade.md)
 - 🔁 **Release compatibility matrix**: [docs/releases/compatibility-matrix.md](docs/releases/compatibility-matrix.md)
 
 - **By Topic**
-- ⚙️ **Environment variables**: [docs/configuration/environment-variables.md](docs/configuration/environment-variables.md)
-- 🏗️ **Architecture concepts**: [docs/concepts/architecture.md](docs/concepts/architecture.md)
-- 🔒 **Security model**: [docs/concepts/security-model.md](docs/concepts/security-model.md)
-- 📘 **Glossary**: [docs/concepts/glossary.md](docs/concepts/glossary.md)
-- 🔑 **Key management operations**: [docs/operations/key-management.md](docs/operations/key-management.md)
-- 🔐 **Security hardening**: [docs/operations/security-hardening.md](docs/operations/security-hardening.md)
-- 📊 **Monitoring and metrics**: [docs/operations/monitoring.md](docs/operations/monitoring.md)
-- 🧯 **Operator drills**: [docs/operations/operator-drills.md](docs/operations/operator-drills.md)
-- 🚀 **Production rollout golden path**: [docs/operations/production-rollout.md](docs/operations/production-rollout.md)
-- 🚑 **Failure playbooks**: [docs/operations/failure-playbooks.md](docs/operations/failure-playbooks.md)
-- 🏭 **Production deployment**: [docs/operations/production.md](docs/operations/production.md)
-- 🛠️ **Development and testing**: [docs/development/testing.md](docs/development/testing.md)
-- 🗺️ **Docs architecture map**: [docs/development/docs-architecture-map.md](docs/development/docs-architecture-map.md)
-- 🤝 **Docs contributing**: [docs/contributing.md](docs/contributing.md)
-- 🗒️ **Docs changelog**: [docs/CHANGELOG.md](docs/CHANGELOG.md)
+  - ⚙️ **Environment variables**: [docs/configuration/environment-variables.md](docs/configuration/environment-variables.md)
+  - 🏗️ **Architecture concepts**: [docs/concepts/architecture.md](docs/concepts/architecture.md)
+  - 🔒 **Security model**: [docs/concepts/security-model.md](docs/concepts/security-model.md)
+  - 📘 **Glossary**: [docs/concepts/glossary.md](docs/concepts/glossary.md)
+  - 🔑 **Key management operations**: [docs/operations/key-management.md](docs/operations/key-management.md)
+  - ☁️ **KMS setup guide**: [docs/operations/kms-setup.md](docs/operations/kms-setup.md)
+  - ✅ **KMS migration checklist**: [docs/operations/kms-migration-checklist.md](docs/operations/kms-migration-checklist.md)
+  - 🔐 **Security hardening**: [docs/operations/security-hardening.md](docs/operations/security-hardening.md)
+  - 📊 **Monitoring and metrics**: [docs/operations/monitoring.md](docs/operations/monitoring.md)
+  - 🧯 **Operator drills**: [docs/operations/operator-drills.md](docs/operations/operator-drills.md)
+  - 🚀 **Production rollout golden path**: [docs/operations/production-rollout.md](docs/operations/production-rollout.md)
+  - 🚑 **Failure playbooks**: [docs/operations/failure-playbooks.md](docs/operations/failure-playbooks.md)
+  - 🏭 **Production deployment**: [docs/operations/production.md](docs/operations/production.md)
+  - 🛠️ **Development and testing**: [docs/development/testing.md](docs/development/testing.md)
+  - 🗺️ **Docs architecture map**: [docs/development/docs-architecture-map.md](docs/development/docs-architecture-map.md)
+  - 🤝 **Docs contributing**: [docs/contributing.md](docs/contributing.md)
+  - 🗒️ **Docs changelog**: [docs/CHANGELOG.md](docs/CHANGELOG.md)
 
 Release note location:
 
@@ -102,6 +105,7 @@ All detailed guides include practical use cases and copy/paste-ready examples.
 ## ✨ What You Get
 
 - 🔐 Envelope encryption (`Master Key -> KEK -> DEK -> Secret Data`)
+- 🔑 **KMS Integration** for master key encryption at rest (supports Google Cloud KMS, AWS KMS, Azure Key Vault, HashiCorp Vault, and local secrets for testing)
 - 🚄 Transit encryption (`/v1/transit/keys/*`) for encrypt/decrypt as a service (decrypt input uses `<version>:<base64-ciphertext>`; see [Transit API docs](docs/api/transit.md), [create vs rotate](docs/api/transit.md#create-vs-rotate), and [error matrix](docs/api/transit.md#endpoint-error-matrix))
 - 🎫 Tokenization API (`/v1/tokenization/*`) for token generation, detokenization, validation, and revocation
 - 👤 Token-based authentication and policy-based authorization