feat(config): track per-test license requirements in maester-config.json

[SamErde]

📑 Description

Adds a Licenses array to every TestSettings entry in tests/maester-config.json, giving us a single index that answers: which licenses make this test's feature available?

Today, license requirements are scattered across Pester Describe tags ("Entra ID P1"), inline Get-MtLicenseInformation skip checks, the NotLicensed* skip-reason vocabulary in Get-MtSkippedReason.ps1, and prose in test docs. This change consolidates those signals into a structured field on each test, plus two scripts (one to populate the field, one to audit it) so future drift is detectable.

Relates to #1071 (filter tests by license type) and #627 (add license metadata).

Vocabulary

Token names mirror the existing Maester abstractions in Get-MtLicenseInformation.ps1 and Get-MtSkippedReason.ps1:

EntraIDP1, EntraIDP2, EntraIDGovernance, EntraWorkloadIDP1, EntraWorkloadIDP2, Eop, MdoP1, MdoP2, ExoDlp, AdvAudit, DefenderXDR, Intune, CustomerLockbox, plus AzureDevOps for AZDO.* tests.

Semantics:

• [] → no premium license required (baseline; matches today's implicit default).
• ["EntraIDP2"] → exactly this tier required.
• ["Eop", "MdoP1"] → any one of the listed licenses satisfies the requirement (matches the OR-semantics already used for Mdo / MdoV2 in Get-MtLicenseInformation).
• ["TBD"] → reserved for orphan entries (config rows with no backing .Tests.ps1).

Note for reviewers: I considered using Microsoft's official SKU/license display names (e.g. Microsoft Entra ID P1) but kept the existing internal shorthand for consistency with the rest of the codebase. Happy to rename in a follow-up if maintainers prefer.

Approach

build/Update-MaesterConfigLicenses.ps1 walks every tests/**/*.Tests.ps1, locates each It block by test ID, and extracts license signals in this priority order:

1. Add-MtTestResultDetail -SkippedBecause NotLicensed<Token> inside the It block.
2. Get-MtLicenseInformation -Product X comparisons (e.g. -ne 'P2' → EntraIDP2).
3. -Skip:( … license check … ) parameter on the It block.
4. The same patterns inside the underlying Test-* PowerShell function.
5. License-bearing Describe / Context / It tags ("Entra ID P1", "DefenderXDR", "Intune", "MDI", "Governance", "MdoP1", "MdoP2", "customerlockbox"). Forward-compatible: a future License:<Token> tag form is parsed too. The bare "Defender" tag is intentionally not treated as a signal because it's ambiguous (MDE / MDO / MDI / MDC / MDCA) — those tests are caught by signals 1–4.
6. Suite default: AZDO.* → ["AzureDevOps"]; everything else with no signal → [].

build/Test-MaesterConfigLicenses.ps1 independently re-derives signals from the test file, the underlying function, the website doc (website/docs/tests/<id>.md), and the function's companion help doc (powershell/public/**/<function>.md), then compares against the config. Verdicts: OK, OK_MD (config justified by markdown evidence only), BASELINE, ORPHAN, MISMATCH. Use -OnlyMismatches to filter.

Result

Verdict	Count	Meaning
OK	147	Config tokens fully backed by code signals (test or function).
OK_MD	1	MT.1071 — companion Test-MtCaAzureDevOps.md states "Microsoft Entra ID P1 or P2 is required".
BASELINE	251	Licenses: []; no premium signals found anywhere.
ORPHAN	6	Pre-existing config rows with no backing .Tests.ps1: CIS.M365.8.2.4, ORCA.1000–ORCA.1004. Marked ["TBD"] until a backing test is added or the row is removed.
MISMATCH	0	No entry's config contradicts what the code enforces.

Markdown-only signals reviewed and intentionally not actioned (false positives — phrases in remediation prose that aren't actual license gates):

• CIS.M365.2.1.{3,6,12,13} and ORCA.{109, 118.x, 231-233.x} — "EOP" appears in remediation prose, but EOP is bundled with any tenant that has Exchange Online (not a premium gate).
• CISA.MS.EXO.16.2 — "Defender XDR" is a portal-name reference (admin nav path), not a license claim. Function gates on MdoP1.
• CISA.MS.EXO.17.2 — deprecated CISA test that returns null; historical "Purview Audit (Premium)" context only.
• MT.1098 — markdown mentions Defender for Endpoint integration, but the MTD connector configuration is an Intune feature.

✅ Checks

• My pull request adheres to the code style of this project.
• My code requires changes to the documentation. (see "Out of scope" below — separate follow-up)
• I have updated the documentation as required.
• The build and unit tests pass after running /powershell/tests/pester.ps1 locally.

ℹ️ Additional Information

Files changed

File	Change
tests/maester-config.json	+810 / -405. Each TestSettings row gains one "Licenses": [...] line; the prior Title line gains a trailing comma. No other edits, no reordering. LF line endings preserved.
build/Update-MaesterConfigLicenses.ps1	New. Regenerable enricher; supports -DryRun.
build/Test-MaesterConfigLicenses.ps1	New. Independent verifier; supports -OnlyMismatches.

Out of scope (suggested follow-ups)

• Wiring the runtime to read Licenses from the config and skip tests accordingly. Today, skipping is still driven by inline Get-MtLicenseInformation checks; this PR only populates the index.
• Surfacing Licenses in the generated test docs from website/scripts/generate-test-docs.mjs.
• Adding a CI check that re-runs Update-MaesterConfigLicenses.ps1 -DryRun and fails on diff (similar to build/eidsca/Update-EidscaTests.ps1).
• Triaging the 6 pre-existing ORPHAN config rows.
• Renaming tokens to match Microsoft's official license names if maintainers prefer.

Reproducing locally

# Regenerate the Licenses field
pwsh build/Update-MaesterConfigLicenses.ps1 -DryRun     # preview
pwsh build/Update-MaesterConfigLicenses.ps1             # write

# Audit the result
pwsh build/Test-MaesterConfigLicenses.ps1               # full report
pwsh build/Test-MaesterConfigLicenses.ps1 -OnlyMismatches

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}

[Copilot]

Pull request overview

This PR adds structured per-test license metadata to the Maester test configuration so license requirements can be centrally indexed (and later used for filtering/skipping/reporting) rather than inferred from scattered code/doc signals.

Changes:

• Adds a "Licenses": [...] array to each TestSettings entry in tests/maester-config.json.
• Introduces build/Update-MaesterConfigLicenses.ps1 to derive/populate license tokens from tests and underlying functions.
• Introduces build/Test-MaesterConfigLicenses.ps1 to audit/verify config license tokens against independently re-derived signals (including markdown cross-checks).

Reviewed changes

Copilot reviewed 2 out of 4 changed files in this pull request and generated 4 comments.

File	Description
tests/maester-config.json	Adds a Licenses array to every test config entry.
build/Update-MaesterConfigLicenses.ps1	New script to derive and write license tokens into the config.
build/Test-MaesterConfigLicenses.ps1	New verifier script to detect drift/mismatches between config and signals in code/docs.
.gitignore	Ignores a local Claude Code settings file.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[cloudflare-workers-and-pages]

Deploying maester with    Cloudflare Pages

Latest commit:	0459a63
Status:	✅  Deploy successful!
Preview URL:	https://42c24b96.maester.pages.dev
Branch Preview URL:	https://feat-tracklicensemetadata.maester.pages.dev

View logs

[merill]

For licenses there are a few key requirements we need to take into account.

• We should use the product sku from Microsoft for the specific license sku as opposed to custom labels. The reference should be based on the Service Plan Name in this doc https://learn.microsoft.com/en-us/entra/identity/users/licensing-service-plan-reference. For e.g . the id should be AAD_PREMIUM and not EntraIDP1
• Licensing should also take into account tests that require on multiple licenses including license combinations where OR and AND should be supported. E.g. for a test might require SharePoint and Purview license if it's related to a Purview feature for SharePoint. Alternatively you could also have a Purview for Education and General Purview added to the mix. I like the implementation used in the zero trust assessment. https://github.com/microsoft/zerotrustassessment/blob/main/src/powershell/private/license/Test-ZtLicense.ps1

I would re-use the same functionality since it has been tested and is in use by ZeroTrust assessment and helps keep the logic the same Test-ZtLicense -CompatibleLicense @("LicenseA", "LicenseB&LicenseC") -CurrentLicense @("LicenseD", "LicenseB", "LicenseC")

[SamErde]

Thanks, I'll check out the ZTA implementation!

[merill]

The actual implementation in Maester is going to be a bit different to ZTAssess. Since here Pester is the engine running the tests.

I think the approach would be to run pester and make it skip the tests, then we go back and add all the skipped tests into testresults and include the .md content (as well as the info on why they were skipped). This way the results will let the user have the same experience they do today.

[SamErde]

Ah, so are there scenarios where even this structure would not suffice? Should I just close this PR?

[Mynster9361]

just adding this information from my pr #1727 in agreement with @merill

Pretty sure this does not cover everything that requires a license in Maester but i think it is a nice start

Test Skip Conditions

License: Intune

Test File	Skip Condition
tests/Maester/Defender/Test-MtMdeAntivirusPolicy.Tests.ps1	License-Intune
tests/Maester/Intune/Test-MtIntunePlatform.Tests.ps1	License-Intune

License: P2 / Governance

Test Files

Test File	Skip Condition
tests/Maester/Entra/Test-MtEntitlementManagementDeletedGroups.Tests.ps1	P2, Governance
tests/Maester/Entra/Test-MtEntitlementManagementInactivePolicies.Tests.ps1	P2, Governance
tests/Maester/Entra/Test-MtEntitlementManagementOrphanedResources.Tests.ps1	P2, Governance
tests/Maester/Entra/Test-MtEntitlementManagementValidApprovers.Tests.ps1	P2, Governance
tests/Maester/Entra/Test-MtEntitlementManagementValidResourceRoles.Tests.ps1	P2, Governance

Functions

Function	Skip Condition
Test-MtCaMfaForRiskySignIn	P2, Governance
Test-MtCaMisconfiguredIDProtection	P2, Governance
Test-MtCaRequirePasswordChangeForHighUserRisk	P2, Governance
Test-MtEntitlementManagementDeletedGroups	P2, Governance
Test-MtEntitlementManagementInactivePolicies	P2, Governance
Test-MtEntitlementManagementOrphanedResources	P2, Governance
Test-MtEntitlementManagementValidApprovers	P2, Governance
Test-MtEntitlementManagementValidResourceRoles	P2, Governance
Test-MtPimAlertsExists	P2, Governance

Entra ID Plan: Free

Function	Skip Condition
Test-MtConditionalAccessWhatIf	$EntraIDPlan -eq 'Free'
Test-MtCaWIFBlockLegacyAuthentication	$EntraIDPlan -eq 'Free'