Skip to content

Add DSPM for AI tests (MT.1172-MT.1176)#1703

Open
OfirGavish wants to merge 5 commits into
maester365:mainfrom
OfirGavish:dspm-ai-tests
Open

Add DSPM for AI tests (MT.1172-MT.1176)#1703
OfirGavish wants to merge 5 commits into
maester365:mainfrom
OfirGavish:dspm-ai-tests

Conversation

@OfirGavish
Copy link
Copy Markdown
Contributor

@OfirGavish OfirGavish commented Apr 27, 2026

Summary

Adds 5 Microsoft Purview Data Security Posture Management (DSPM) for AI governance tests covering Microsoft 365 Copilot oversharing, audit, and retention controls. Reserved in #697.

MT Function Severity Service
1172 Test-MtPurviewAiAuditLogIngestion High ExchangeOnline
1173 Test-MtPurviewAiSensitivityLabelsForFiles Medium SecurityCompliance
1174 Test-MtPurviewAiInsiderRiskPolicy Medium SecurityCompliance
1175 Test-MtPurviewAiDlpPolicy High ExchangeOnline + SecurityCompliance
1176 Test-MtPurviewAiRetentionPolicy Medium SecurityCompliance

Note on numbering: originally proposed as MT.1152-MT.1156, renumbered to MT.1172-MT.1176 to avoid collision with the Microsoft Defender antivirus metadata batch added upstream in 7ebade5.

Why

Maester currently has solid coverage for Copilot Studio agent governance (MT.1113-1122) and Intune controls (MT.1123, MT.1147-1151), but has no tests covering the Microsoft Purview DSPM for AI controls that govern Microsoft 365 Copilot itself. These 5 tests close that gap by checking the most impactful DSPM-for-AI prerequisites and protective controls a tenant should have configured before broadly enabling Copilot.

What each test does

  • MT.1172 verifies the unified audit log is enabled - this is the foundation for every other Purview AI control (DSPM for AI activity explorer, Risky AI usage Insider Risk policies, Copilot eDiscovery, Communication Compliance for Copilot all silently break without it).
  • MT.1173 verifies sensitivity labels are published and at least one is scoped to files. Without this, Copilot has nothing to honor or inherit onto AI-generated content.
  • MT.1174 verifies an Insider Risk Management policy from the Risky AI usage template is enabled. Defensive matching on InsiderRiskScenario / Name / PolicyTemplate for forward compatibility.
  • MT.1175 verifies a DLP policy targets Microsoft 365 Copilot. Detects coverage across all four shapes Microsoft currently exposes via Get-DlpCompliancePolicy: MicrosoftCopilotLocation, Workload (MicrosoftCopilot), Locations, and EnforcementPlanes (e.g. CopilotExperiences).
  • MT.1176 verifies a retention policy targets the Microsoft Copilot location for governed retention/disposal of Copilot prompts and AI-generated responses. Queries both retention surfaces — legacy Get-RetentionCompliancePolicy and the newer Get-AppRetentionCompliancePolicy (User:M365Copilot) — via the Get-MtExo cache wrapper. Passes if either surface protects Copilot.

Files

  • powershell/public/maester/purview/Test-MtPurviewAi*.ps1 (5 functions)
  • powershell/public/maester/purview/Test-MtPurviewAi*.md (5 companion remediation pages)
  • website/docs/tests/maester/MT.117[2-6].md (5 website docs)
  • tests/Maester/Purview/Test-MtPurviewAi.Tests.ps1 (consolidated Pester suite)
  • tests/maester-config.json (5 new entries)
  • powershell/Maester.psd1 (5 new functions added to FunctionsToExport)
  • powershell/public/cisa/exchange/Get-MtExo.ps1 (added RetentionCompliancePolicy + AppRetentionCompliancePolicy to the cached commands hashtable so MT.1176 can route through the standard Get-MtExo wrapper)

Conventions

All tests follow the standard Maester pattern: Test-MtConnection guard, Get-MtLicenseInformation where applicable (MT.1175 only - DLP requires Exo DLP licence), try/catch with NotAuthorized/Error skips, Add-MtTestResultDetail with Well done. prefix on pass, %TestResult% markdown table with risk callout on detail tests. EXO/SCC cmdlets are routed through Get-MtExo -Request <CmdletSuffix> so they participate in $__MtSession.ExoCache.

Validation

  • Invoke-ScriptAnalyzer -Severity Warning,Error clean on all 6 new PS1 files.
  • All files parse without errors.
  • maester-config.json parses (5 new entries verified).
  • Test-ModuleManifest succeeds.
  • All 5 files use UTF-8 BOM.
  • Test-MtPurviewAiSensitivityLabelsForFiles has [SuppressMessage('PSUseSingularNouns', ...)] with justification (matches existing pattern from Test-MtAIAgentMcpTools).
  • End-to-end tested against a live tenant: all 5 functions execute correctly through every code path (pass / fail-with-detail / license-skip / connection-skip). MT.1172 + MT.1173 returned True (PASS) on the test tenant; MT.1174 + MT.1175 + MT.1176 returned False with correctly-rendered detail markdown showing the empty-state row.

Notes for reviewers

  • MT.1173 originally planned to check SPO/OneDrive AIP integration via Get-SPOTenant, but Maester does not connect to the SharePoint Management Shell. Pivoted to checking the actual prerequisite (published file-scoped sensitivity labels) via Get-Label + Get-LabelPolicy.
  • MT.1174 license skip uses SkippedBecause Custom (with reason) since there is no specific NotLicensedIRM enum value in Add-MtTestResultDetail. Happy to add one in a follow-up if preferred.
  • MT.1175 detection covers all four current Copilot DLP schema shapes per @SamErde's review (MicrosoftCopilotLocation, Workload, Locations, EnforcementPlanes).
  • MT.1176 queries both Get-RetentionCompliancePolicy and Get-AppRetentionCompliancePolicy per @SamErde's review, with graceful fallback if the app-retention cmdlet isn't available in the connected SCC session. Each policy is tagged with a Surface column (Legacy / App) in the result detail.

@OfirGavish OfirGavish requested review from a team as code owners April 27, 2026 22:10
@codacy-production
Copy link
Copy Markdown

codacy-production Bot commented Apr 27, 2026

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@SamErde SamErde requested a review from Copilot April 27, 2026 22:12
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds Microsoft Purview DSPM for AI governance coverage to Maester by introducing five new MT.1152–MT.1156 tests (Copilot audit ingestion, sensitivity labels, insider risk, DLP, retention) along with their remediation and website documentation, config registration, test harness, and module exports.

Changes:

  • Add 5 new Purview-focused Maester PowerShell tests (MT.1152–MT.1156) with detailed markdown result output.
  • Add remediation markdown pages and Docusaurus website docs for each new test.
  • Register the tests in maester-config.json, add a consolidated Pester suite, and export the functions in Maester.psd1.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
website/docs/tests/maester/MT.1152.md Website doc for unified audit log ingestion prerequisite test.
website/docs/tests/maester/MT.1153.md Website doc for published file-scoped sensitivity labels prerequisite test.
website/docs/tests/maester/MT.1154.md Website doc for Insider Risk “Risky AI usage” policy test.
website/docs/tests/maester/MT.1155.md Website doc for Copilot-scoped DLP policy test.
website/docs/tests/maester/MT.1156.md Website doc for Copilot-scoped retention policy test.
tests/maester-config.json Adds MT.1152–MT.1156 entries to the test catalog (ID/severity/title).
tests/Maester/Purview/Test-MtPurviewAi.Tests.ps1 New Pester suite invoking the 5 Purview AI tests.
powershell/public/maester/purview/Test-MtPurviewAiAuditLogIngestion.ps1 Implements MT.1152 (unified audit log ingestion enabled).
powershell/public/maester/purview/Test-MtPurviewAiAuditLogIngestion.md Remediation/details page for MT.1152.
powershell/public/maester/purview/Test-MtPurviewAiSensitivityLabelsForFiles.ps1 Implements MT.1153 (published file-scoped sensitivity labels).
powershell/public/maester/purview/Test-MtPurviewAiSensitivityLabelsForFiles.md Remediation/details page for MT.1153.
powershell/public/maester/purview/Test-MtPurviewAiInsiderRiskPolicy.ps1 Implements MT.1154 (Risky AI usage Insider Risk policy enabled).
powershell/public/maester/purview/Test-MtPurviewAiInsiderRiskPolicy.md Remediation/details page for MT.1154.
powershell/public/maester/purview/Test-MtPurviewAiDlpPolicy.ps1 Implements MT.1155 (Copilot-targeted DLP policy enabled, non-simulation).
powershell/public/maester/purview/Test-MtPurviewAiDlpPolicy.md Remediation/details page for MT.1155.
powershell/public/maester/purview/Test-MtPurviewAiRetentionPolicy.ps1 Implements MT.1156 (Copilot-targeted retention policy enabled).
powershell/public/maester/purview/Test-MtPurviewAiRetentionPolicy.md Remediation/details page for MT.1156.
powershell/Maester.psd1 Exports the 5 new test functions from the module.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread powershell/public/maester/purview/Test-MtPurviewAiInsiderRiskPolicy.ps1 Outdated
OfirGavish added a commit to OfirGavish/maester that referenced this pull request Apr 28, 2026
MT.1153 (Test-MtPurviewAiSensitivityLabelsForFiles):
  Cross-reference Get-LabelPolicy output to verify a file-scoped label
  is actually included in a published policy, not merely that *some*
  file-scoped label exists alongside *some* published policy. Adds a
  third row to the result table showing the published file-scoped
  label count.

MT.1154 (Test-MtPurviewAiInsiderRiskPolicy):
  Narrow the 'cmdlet unavailable / not licensed' skip to
  CommandNotFoundException and matching 'is not recognized' /
  'is unavailable' messages. All other exceptions now fall back to
  -SkippedBecause Error -SkippedError so transient service errors,
  throttling, and module/session issues are no longer masked as
  licensing skips.
@OfirGavish OfirGavish requested a review from Copilot April 28, 2026 11:26
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread powershell/public/maester/purview/Test-MtPurviewAiSensitivityLabelsForFiles.ps1 Outdated
OfirGavish added a commit to OfirGavish/maester that referenced this pull request Apr 29, 2026
Address Copilot follow-up review on PR maester365#1703:
A label policy is now treated as 'published' only when Mode is 'Enforce'
AND (when the Enabled property is exposed by the SCC schema) Enabled is
true. The previous '-or' logic could classify a disabled Enforce-mode
policy or an enabled non-Enforce-mode policy as published, which is
inconsistent with the in-code comment and remediation docs.
@SamErde
Copy link
Copy Markdown
Contributor

SamErde commented May 11, 2026

@copilot resolve the merge conflicts in this pull request

@SamErde SamErde added enhancement New feature or request maester-test Related to a Maester test purview labels May 11, 2026
@SamErde
Copy link
Copy Markdown
Contributor

SamErde commented May 11, 2026

Thanks for putting this together — these DSPM for AI checks are a really useful addition, and the overall PR shape looks good: helper functions are exported, companion docs and website docs are present, the test IDs are wired through config, and the targeted PowerShell/Pester validation I ran did not show syntax or analyzer issues.

I found two areas that I think should be tightened before merge so the checks do not false-fail tenants that are configured through the current Microsoft Purview surfaces:

  1. MT.1155 DLP policy detection may miss valid Microsoft 365 Copilot DLP policies

    In powershell/public/maester/purview/Test-MtPurviewAiDlpPolicy.ps1, the check currently identifies Copilot-targeted DLP policies by looking at MicrosoftCopilotLocation or Workload -match 'Copilot'. Some current Purview-created / PowerShell-created Copilot DLP policies can surface through fields such as Locations and EnforcementPlanes (for example CopilotExperiences) instead.

    Potential solution: keep the existing matching for compatibility, but expand the detection to also treat policies as Copilot-targeted when their locations/enforcement-plane metadata indicates Copilot experiences. That should make the check resilient across tenant schema variations and current Purview policy creation paths.

  2. MT.1156 retention policy detection may be checking the older/general retention surface only

    In powershell/public/maester/purview/Test-MtPurviewAiRetentionPolicy.ps1, the check currently uses Get-RetentionCompliancePolicy and looks for MicrosoftCopilotLocation / Workload -match 'Copilot'. Microsoft’s current Copilot retention guidance describes separate Copilot interaction/app retention policy locations, including app retention policy cmdlets and application values such as User:M365Copilot.

    Potential solution: extend the helper to query the current Copilot/app retention policy surface as well, while preserving the existing Get-RetentionCompliancePolicy path for older or tenant-specific representations. The pass condition should succeed when either surface shows an enabled policy targeting Microsoft Copilot interactions.

Also, GitHub currently reports the PR as conflicting with main, so the branch will need a merge/rebase before it can be merged.

Thanks again for the contribution — once the policy-surface detection is broadened, these look much closer to being merge-ready.

Copy link
Copy Markdown
Contributor

@SamErde SamErde left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See previous comment. Thanks! 🙂

Adds five Maester tests covering Microsoft Purview Data Security
Posture Management for AI:

- MT.1172 Unified audit log ingestion is enabled for AI activity
- MT.1173 Sensitivity labels are published for files used by Microsoft 365 Copilot
- MT.1174 Insider Risk Management policy for Risky AI usage is enabled
- MT.1175 DLP policy is configured for the Microsoft 365 Copilot location
- MT.1176 Retention policy is configured for the Microsoft Copilot location

MT.1175 detects Copilot DLP coverage across the multiple shapes
exposed by Get-DlpCompliancePolicy: MicrosoftCopilotLocation,
Workload (MicrosoftCopilot), Locations, and EnforcementPlanes
(CopilotExperiences).

MT.1176 queries both retention surfaces -- legacy
Get-RetentionCompliancePolicy and the newer
Get-AppRetentionCompliancePolicy (User:M365Copilot) -- via the
Get-MtExo cache wrapper. Passes if either surface protects Copilot.

Test IDs originally proposed as MT.1152-MT.1156 were renumbered to
MT.1172-MT.1176 to avoid collision with the Defender antivirus
metadata batch added upstream in 7ebade5.
@OfirGavish OfirGavish changed the title Add DSPM for AI tests (MT.1152-MT.1156) Add DSPM for AI tests (MT.1172-MT.1176) May 12, 2026
@OfirGavish
Copy link
Copy Markdown
Contributor Author

Thanks for the careful review @SamErde — addressed both points and pushed bc9df176. The branch is now a single clean commit on top of main.

Renumbered to MT.1172-MT.1176
Upstream commit 7ebade5 (Added test meta data) claimed our original MT.1152-MT.1156 range for the Microsoft Defender antivirus configuration tests, so I bumped this PR's IDs to the next available contiguous range. PR title + description updated.

MT.1175 — broader Copilot DLP detection
The $copilotPolicies filter now matches a policy if any of these are true (defensive against the schema migration you flagged):

  • MicrosoftCopilotLocation is populated (current dedicated location)
  • Workload matches Copilot or contains MicrosoftCopilot
  • Locations references a Copilot location
  • EnforcementPlanes contains a Copilot plane (e.g. CopilotExperiences)

MT.1176 — dual retention surface
Added RetentionCompliancePolicy and AppRetentionCompliancePolicy to Get-MtExo so both surfaces are cached and the test stays consistent with the rest of Maester's EXO/SCC pattern. The test now:

  • Always queries Get-RetentionCompliancePolicy (legacy MicrosoftPurviewLocation / Copilot workload)
  • Best-effort queries Get-AppRetentionCompliancePolicy for the newer User:M365Copilot surface (graceful fallback if the cmdlet isn't available in the connected SCC session)
  • Tags each matching policy with a Surface column (Legacy / App) in the result detail
  • Passes if either surface protects Copilot

Companion .md docs updated with verification snippets for both surfaces. Validated against the test tenant — all five tests still execute through every code path correctly.

SamErde and others added 2 commits May 12, 2026 14:10
The build-validation Pester test "Should contain Write-Verbose logging"
in powershell/tests/functions/Common.Tests.ps1 requires every exported
public function to contain at least one Write-Verbose call. Four of the
five new DSPM for AI tests were missing this, causing 4 failures across
ubuntu-latest, macOS-latest and windows-latest runners.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@SamErde
Copy link
Copy Markdown
Contributor

SamErde commented May 12, 2026

Thanks for the PR @OfirGavish — really nice set of DSPM for AI tests! 🎉

The build-validation failures across all three runners were all the same Pester check from powershell/tests/functions/Common.Tests.ps1:

It "Should contain Write-Verbose logging" ... because we like information when troubleshooting

That test enforces a project-wide convention that every exported public function must contain at least one Write-Verbose call. Four of the five new functions were missing one — Test-MtPurviewAiRetentionPolicy.ps1 already had two, which is why it passed.

Since maintainerCanModify is enabled on this PR, I went ahead and pushed a small commit that adds a single Write-Verbose line just after the param() block in each of the four affected files:

  • Test-MtPurviewAiAuditLogIngestion.ps1
  • Test-MtPurviewAiDlpPolicy.ps1
  • Test-MtPurviewAiInsiderRiskPolicy.ps1
  • Test-MtPurviewAiSensitivityLabelsForFiles.ps1

That should turn the build-validation checks green. Feel free to amend / squash the commit into your branch however you prefer — and thanks again for the contribution!

@SamErde SamErde self-requested a review May 12, 2026 18:35
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated no new comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request maester-test Related to a Maester test purview

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants