feat: gate tests on license availability using BeforeDiscovery skip pattern#1727
feat: gate tests on license availability using BeforeDiscovery skip pattern#1727Mynster9361 wants to merge 11 commits into
Conversation
Add $__MtSession.Licenses hashtable populated once by Initialize-MtSession at startup. Get-MtLicenseInformation is now cache-first and lazy-loads SKU data only on a cache miss. Add Get-MtSessionLicenses for use in Pester BeforeDiscovery blocks to read the cache without additional Graph API calls. Clear-ModuleVariable resets the cache on each Invoke-Maester run. Co-authored-by: Copilot <copilot@github.com>
When -AutoFilterLicenses is set, Maester reads the pre-fetched session license cache and appends the appropriate License-* tags to ExcludeTag so tests requiring absent licenses are skipped cleanly at discovery time. Co-authored-by: Copilot <copilot@github.com>
Replace ambiguous -Error parameter with -SkippedBecause Error -SkippedError in six CA helper catch blocks (was causing InvalidResult in Pester). Move the $hasRiskCAPolicy skip in Test-MtCaMisconfiguredIDProtection outside the try-catch to prevent Pester's internal flow-control exception from being caught and re-thrown as a double Set-ItResult error (MT.1049). Replace MtRoleDefinition object casts with .ToString() plus stable GUID fallbacks in Test-MtCaExclusionForDirectorySyncAccount, and add a null guard to Get-MtRoleInfo for when the module role cache is unavailable (MT.1020). Co-authored-by: Copilot <copilot@github.com>
…attern
Add BeforeDiscovery { $Licenses = Get-MtSessionLicenses } and Describe-level
-Skip expressions to tests that require specific licenses:
- License-EntraP2: CA risk sign-in/user risk tests (MT.1012, 1013, 1049),
PIM alerts tests (MT.1029–1032)
- License-EntraGovernance: Entitlement Management tests (MT.1106–1110)
- License-Intune: MDE antivirus and Intune platform tests
Also fix MT.1033/1034 test numbering (array IndexOf on single-object
variables) and convert MT.1034 inline license check to -Skip expression.
Co-authored-by: Copilot <copilot@github.com>
…-MtSessionLicenses Co-authored-by: Copilot <copilot@github.com>
Up to standards ✅🟢 Issues
|
1. Pester test singular noun for Get-MTSessionLicenses -> Get-MTSessionLicens + all references 2. website build error with incomplete explicit mapping pair; a key node is missed; or followed by a non-tabulated empty line due to Helper: not being encapsulated
There was a problem hiding this comment.
Pull request overview
Adds a discovery-time license-gating pattern so Maester tests can skip cleanly on unlicensed tenants, while also prefetching license data in session state for reuse across test discovery and execution.
Changes:
- Added
BeforeDiscovery+-Skiplicense gating to selected Entra, Intune, Defender, and Governance test files. - Added session-level license caching/accessors and new
Invoke-Maesterauto-filter logic forLicense-*tags. - Updated docs for license gating and fixed a few related test/runtime edge cases.
Reviewed changes
Copilot reviewed 28 out of 28 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| website/docs/writing-tests/advanced-concepts.md | Documents the new license-gating pattern and tag taxonomy. |
| website/docs/commands/Test-MaesterResultValid.mdx | Quotes YAML front matter description. |
| website/docs/commands/Import-SingleResultFile.mdx | Quotes YAML front matter description. |
| tests/Maester/Intune/Test-MtIntunePlatform.Tests.ps1 | Adds Intune license-based Describe skip. |
| tests/Maester/Entra/Test-PrivilegedAssignments.Tests.ps1 | Adds P2/Governance gating to PIM alert tests. |
| tests/Maester/Entra/Test-MtEntitlementManagementValidResourceRoles.Tests.ps1 | Adds Governance license gating. |
| tests/Maester/Entra/Test-MtEntitlementManagementValidApprovers.Tests.ps1 | Adds Governance license gating. |
| tests/Maester/Entra/Test-MtEntitlementManagementOrphanedResources.Tests.ps1 | Adds Governance license gating. |
| tests/Maester/Entra/Test-MtEntitlementManagementInactivePolicies.Tests.ps1 | Adds Governance license gating. |
| tests/Maester/Entra/Test-MtEntitlementManagementDeletedGroups.Tests.ps1 | Adds Governance license gating. |
| tests/Maester/Entra/Test-ConditionalAccessWhatIf.Tests.ps1 | Uses cached license data and fixes IndexOf handling. |
| tests/Maester/Entra/Test-ConditionalAccessBaseline.Tests.ps1 | Adds cached Entra plan usage and new license tags/skips. |
| tests/Maester/Defender/Test-MtMdeAntivirusPolicy.Tests.ps1 | Adds Intune license gating for MDE policy tests. |
| powershell/public/maester/entra/Test-MtCaWIFBlockLegacyAuthentication.ps1 | Changes error handling to skipped-on-error. |
| powershell/public/maester/entra/Test-MtCaRequirePasswordChangeForHighUserRisk.ps1 | Changes error handling to skipped-on-error. |
| powershell/public/maester/entra/Test-MtCaReferencedGroupsExist.ps1 | Changes error handling to skipped-on-error. |
| powershell/public/maester/entra/Test-MtCaMisconfiguredIDProtection.ps1 | Refactors skip flow around try/catch and result setup. |
| powershell/public/maester/entra/Test-MtCaMfaForGuest.ps1 | Changes error handling to skipped-on-error. |
| powershell/public/maester/entra/Test-MtCaMfaForAllUsers.ps1 | Changes error handling to skipped-on-error. |
| powershell/public/maester/entra/Test-MtCaExclusionForDirectorySyncAccount.ps1 | Adds stable GUID fallback for directory sync roles. |
| powershell/public/Invoke-Maester.ps1 | Adds new auto license-tag exclusion switch and example. |
| powershell/public/Get-MtSessionLicens.ps1 | Adds a new public session-license accessor function. |
| powershell/public/Get-MtLicenseInformation.ps1 | Adds cache-aware license lookup and session writes. |
| powershell/Maester.psm1 | Extends session state with cached license data. |
| powershell/Maester.psd1 | Exports the new public session-license function. |
| powershell/internal/Initialize-MtSession.ps1 | Prefetches license products into session cache. |
| powershell/internal/Get-MtRoleInfo.ps1 | Adds a null guard and reformats role definitions. |
| powershell/internal/Clear-ModuleVariable.ps1 | Clears cached licenses during session reset. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 28 out of 28 changed files in this pull request and generated 5 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
|
@copilot resolve the merge conflicts in this pull request |
…test. Will take a look at that wednessday unless you do it before me @SamErde :)
|
@SamErde Got through most of the suggestions/recommendations. I have not made a unit test for it yet will properly first get arround to it at the earliest wednessday so if you want to give it a go you are welcome to do :) |
|
|
||
| Write-Information "Total Exchange Online licenses: $TotalLicenses" | ||
| return $TotalLicenses | ||
| $LicenseType = $TotalLicenses |
There was a problem hiding this comment.
Trying to remember how this function works: should $LicenseType be returning the total count of licenses? That sounds like a mismatch, but I could be wrong.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 29 out of 29 changed files in this pull request and generated 5 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # Automatically exclude tests that require licenses not present in the tenant. | ||
| # When set, Maester queries the tenant license information at startup and adds the | ||
| # appropriate License-* tags to ExcludeTag so unlicensed tests are skipped cleanly. | ||
| # This requires a Graph connection and is silently ignored when not connected. | ||
| [Parameter(HelpMessage = 'Skip tests that require licenses the tenant does not have.')] | ||
| [switch] $AutoFilterLicense |
| # Lazy-load SKU data only when a cache miss requires it. | ||
| if ($null -eq $SKUs) { | ||
| $SKUs = Invoke-MtGraphRequest -RelativeUri 'subscribedSkus' | Where-Object { $_.capabilityStatus -eq 'Enabled' } | ||
| $ServicePlans = $SKUs | Select-Object -ExpandProperty servicePlans | Select-Object -ExpandProperty servicePlanId | Sort-Object -Unique |
| # If -AutoFilterLicense is set, exclude tests whose required license is absent in the tenant. | ||
| if ($AutoFilterLicense.IsPresent) { | ||
| $tenantLicenses = Get-MtSessionLicense | ||
| if ($tenantLicenses.Count -gt 0) { | ||
| $licenseExclusions = [System.Collections.Generic.List[string]]::new() | ||
|
|
||
| # Entra ID tiers: Free excludes P1 + P2 + Governance; P1 excludes P2 + Governance | ||
| switch ($tenantLicenses['EntraID']) { | ||
| 'Free' { | ||
| $licenseExclusions.AddRange([string[]]@('License-EntraP1', 'License-EntraP2', 'License-EntraGovernance')) | ||
| } | ||
| 'P1' { | ||
| $licenseExclusions.AddRange([string[]]@('License-EntraP2', 'License-EntraGovernance')) | ||
| } | ||
| } | ||
|
|
||
| # Binary (licensed / not licensed) products | ||
| $binaryProducts = @{ | ||
| 'EntraWorkloadID' = 'License-EntraWorkloadID' | ||
| 'Eop' = 'License-Eop' | ||
| 'ExoDlp' = 'License-ExoDlp' | ||
| 'Mdo' = 'License-Mdo' | ||
| 'AdvAudit' = 'License-AdvAudit' | ||
| 'DefenderXDR' = 'License-DefenderXDR' | ||
| 'CustomerLockbox' = 'License-CustomerLockbox' | ||
| 'Intune' = 'License-Intune' | ||
| } | ||
| foreach ($entry in $binaryProducts.GetEnumerator()) { | ||
| if ($null -eq $tenantLicenses[$entry.Key]) { | ||
| $licenseExclusions.Add($entry.Value) | ||
| } | ||
| } | ||
|
|
||
| if ($licenseExclusions.Count -gt 0) { | ||
| $ExcludeTag = @($ExcludeTag | Where-Object { $_ }) + $licenseExclusions | ||
| Write-Verbose "AutoFilterLicense: excluding tags $($licenseExclusions -join ', ')" | ||
| } |
| Context 'Maester/Entra' -ForEach @( $RegularUsers ) { | ||
| # Regular users | ||
| It "MT.1033.$($RegularUsers.IndexOf($_)): User should be blocked from using legacy authentication ($($_.userPrincipalName))" -Tag 'MT.1033', 'CA', 'CAWhatIf', 'LongRunning', 'Maester' -Skip:( $EntraIDPlan -eq 'Free' ) { | ||
| It "MT.1033.$([array]::IndexOf(@($RegularUsers), $_)): User should be blocked from using legacy authentication ($($_.userPrincipalName))" -Tag 'MT.1033', 'CA', 'CAWhatIf', 'LongRunning', 'Maester' -Skip:( $EntraIDPlan -eq 'Free' ) { |
| } else { | ||
| Test-MtConditionalAccessWhatIf -UserId $id -IncludeApplications '00000002-0000-0ff1-ce00-000000000000' -ClientAppType exchangeActiveSync | Should -BeNullOrEmpty | ||
| } | ||
| It "MT.1034.$([array]::IndexOf(@($EmergencyAccessUsers), $_)): Emergency access users should not be blocked ($($_.userPrincipalName))" -Tag 'MT.1034' -Skip:($EntraIDPlan -eq 'Free') { |
|
Amazing PR, @Mynster9361! This has been asked for many times and will make a lot of people happy. Just a couple of minor changes requested. This is almost ready to merge! |
| # appropriate License-* tags to ExcludeTag so unlicensed tests are skipped cleanly. | ||
| # This requires a Graph connection and is silently ignored when not connected. | ||
| [Parameter(HelpMessage = 'Skip tests that require licenses the tenant does not have.')] | ||
| [switch] $AutoFilterLicense |
There was a problem hiding this comment.
We should automatically exclude tests that the tenant is not licensed for and not require a parameter for this.
The alternative should be opt-in: use a switch (e.g. IgnoreLicense) to perform all tests regardless of what licenses are detected.
| switch ($tenantLicenses['EntraID']) { | ||
| 'Free' { | ||
| $licenseExclusions.AddRange([string[]]@('License-EntraP1', 'License-EntraP2', 'License-EntraGovernance')) | ||
| } | ||
| 'P1' { | ||
| $licenseExclusions.AddRange([string[]]@('License-EntraP2', 'License-EntraGovernance')) | ||
| } | ||
| } | ||
|
|
||
| # Binary (licensed / not licensed) products | ||
| $binaryProducts = @{ | ||
| 'EntraWorkloadID' = 'License-EntraWorkloadID' | ||
| 'Eop' = 'License-Eop' | ||
| 'ExoDlp' = 'License-ExoDlp' | ||
| 'Mdo' = 'License-Mdo' | ||
| 'AdvAudit' = 'License-AdvAudit' | ||
| 'DefenderXDR' = 'License-DefenderXDR' | ||
| 'CustomerLockbox' = 'License-CustomerLockbox' | ||
| 'Intune' = 'License-Intune' | ||
| } | ||
| foreach ($entry in $binaryProducts.GetEnumerator()) { | ||
| if ($null -eq $tenantLicenses[$entry.Key]) { | ||
| $licenseExclusions.Add($entry.Value) | ||
| } | ||
| } | ||
|
|
||
| if ($licenseExclusions.Count -gt 0) { | ||
| $ExcludeTag = @($ExcludeTag | Where-Object { $_ }) + $licenseExclusions |
There was a problem hiding this comment.
Please align this tag structure with the way that the optional Severity:{Severity} tags are written. For example, License:Intune.
| Context 'Maester/Entra' -ForEach @( $RegularUsers ) { | ||
| # Regular users | ||
| It "MT.1033.$($RegularUsers.IndexOf($_)): User should be blocked from using legacy authentication ($($_.userPrincipalName))" -Tag 'MT.1033', 'CA', 'CAWhatIf', 'LongRunning', 'Maester' -Skip:( $EntraIDPlan -eq 'Free' ) { | ||
| It "MT.1033.$([array]::IndexOf(@($RegularUsers), $_)): User should be blocked from using legacy authentication ($($_.userPrincipalName))" -Tag 'MT.1033', 'CA', 'CAWhatIf', 'LongRunning', 'Maester' -Skip:( $EntraIDPlan -eq 'Free' ) { |
There was a problem hiding this comment.
Nice cleanup! 👍 For future reference, it is a good practice to use separate PRs for different types of changes/fixes so they can all be reviewed separately.
|
@Mynster9361 Thanks for this. Please see my comment at #1746 (comment) This is something I've been working on at Microsoft on the ZTAssessment. My plan was to bring that implementation over here since it solves some of the key problems in how we add tags for our current tests. |
That is awesome @merill does that mean we should close this one and keep the work going in #1746 @SamErde ? |
|
Yeah, I think we can merge into that one. There is a bit of work to do though including backfilling the licensing for all the existing tests. |
|
definently just keeping the PR open till i have some time to write down the tests i included in this one in regards to license requirements and will add that to #1746 |
|
closing this PR have added the license information from this pr to #1746 |
Add BeforeDiscovery { $Licenses = Get-MtSessionLicenses } and Describe-level -Skip expressions to tests that require specific licenses:
Also fix MT.1033/1034 test numbering (array IndexOf on single-object variables) and convert MT.1034 inline license check to -Skip expression.
Co-authored-by: Copilot copilot@github.com
📑 Description
This PR introduces a consistent, discovery-time license gating pattern across Maester test files. Previously, tests that required specific licenses either ran and produced misleading failures on unlicensed tenants, or used inline
if/elseguards that don't integrate cleanly with Pester's skip mechanism or the new-AutoFilterLicensestag filtering inInvoke-Maester.Pattern applied:
BeforeDiscovery { $Licenses = Get-MtSessionLicenses } Describe "..." -Tag "...", "License-EntraP2" -Skip:($Licenses.EntraID -notin 'P2', 'Governance') { It "MT.XXXX: ..." -Tag "MT.XXXX" { ... } }Get-MtSessionLicensesreturns the license map pre-fetched byInitialize-MtSessionat startup — no extra Graph API calls at discovery time.Tests updated:
License-EntraP2License-EntraGovernanceLicense-IntuneAdditional fixes:
$collection.IndexOf($_)on a potentially single-object variable replaced with[array]::IndexOf(@($collection), $_)to avoid calling.IndexOf()on a non-array type.if (Get-MtLicenseInformation)guard removed in favour of-Skip:($EntraIDPlan -eq 'Free')on theItblock, consistent with the rest of the file.Describeblock (MT.1029–1032): Inlineif/else+Add-MtTestResultDetail -SkippedBecause NotLicensedEntraIDP2replaced with a single Describe-level-Skip, removing duplicated boilerplate across four tests.Added a new Gating tests on license availability section to advanced-concepts.md covering Get-MtSessionLicenses, the BeforeDiscovery skip pattern, license value and skip expression reference tables, and the License-* tag taxonomy.
Closes #
#1071
#894
#812
✅ Checks
/powershell/tests/pester.ps1locally.ℹ️ Additional Information
Compatibility: The
-Skip:()expression inBeforeDiscoveryscope is evaluated at Pester discovery time. IfGet-MtSessionLicensesreturns an empty hashtable (e.g. when running tests directly withInvoke-Pesterwithout a Graph connection),$Licenses.EntraIDwill be$null, which means-Skip:($null -notin 'P2', 'Governance')evaluates to$true— the tests will be skipped rather than fail. This is the safe default for disconnected/offline runs.Works with
-AutoFilterLicenses: TheLicense-*tags added to eachDescribeblock are whatInvoke-Maester -AutoFilterLicensesuses to exclude unlicensed tests viaExcludeTagat the Pester filter level — a faster path that skips the tests before discovery rather than during it.How to Contribute
🏗️ Read our full contributing guide for the Maester project.
🧪 We also have additional instructions and a checklist for creating tests.
Join us at the Maester repository discussions or Entra Discord for more help and conversations!
While you wait for a review, why not spread some Maester love on social media? Thank you! 💖