Added CIS M365v5 SPO tests#1433
Conversation
|
I'm excited to see these in action, @HenrikPiecha! As you noted above, these will not work in some contexts. Can you add checks to the beginning of the Pester test for these so it will fail gracefully and return a relevant SkippedBecause reason to the results report? Let us know if you'd like any help with that!
|
There was a problem hiding this comment.
Pull request overview
This pull request adds 7 new SharePoint Online (SPO) tests based on CIS Microsoft 365 v5.0.0 benchmark recommendations. These tests validate SPO tenant configuration settings related to external sharing, guest access, custom scripts, and malware protection. All tests use the Microsoft.Online.SharePoint.PowerShell module and require interactive connection via Connect-SPOService.
Changes:
- Added 7 PowerShell test functions (Test-MtSpo*) for SPO tenant security checks
- Added corresponding Pester tests tagged with 'SpoTenant'
- Added website and PowerShell documentation for all 7 tests
- Updated maester-config.json with test metadata (MT.1113-MT.1119)
- Exported new functions in Maester.psd1 module manifest
Reviewed changes
Copilot reviewed 24 out of 24 changed files in this pull request and generated 16 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/Maester/Spo/Test-SpoTenant.Tests.ps1 | New Pester test file containing 7 SPO tenant configuration tests (MT.1113-MT.1119) |
| powershell/public/maester/spo/Test-MtSpoB2BIntegration.ps1 | Tests Microsoft Entra B2B integration (CIS 7.2.2) |
| powershell/public/maester/spo/Test-MtSpoCustomScriptExecutionOnSiteCollection.ps1 | Tests custom script restrictions on site collections (CIS 7.3.4) |
| powershell/public/maester/spo/Test-MtSpoDefaultSharingLink.ps1 | Tests default sharing link type restrictions (CIS 7.2.7) |
| powershell/public/maester/spo/Test-MtSpoDefaultSharingLinkPermission.ps1 | Tests default sharing link permissions (CIS 7.2.11) |
| powershell/public/maester/spo/Test-MtSpoGuestAccessExpiry.ps1 | Tests guest access expiration settings (CIS 7.2.9) |
| powershell/public/maester/spo/Test-MtSpoGuestCannotShareUnownedItem.ps1 | Tests guest resharing restrictions (CIS 7.2.5) |
| powershell/public/maester/spo/Test-MtSpoPreventDownloadMaliciousFile.ps1 | Tests malicious file download prevention (CIS 7.3.1) |
| powershell/public/maester/spo/*.md | Documentation files for each test function |
| website/docs/tests/maester/MT.111*.md | Website documentation for each test with CIS benchmark details |
| tests/maester-config.json | Configuration entries for all 7 new tests with severity levels |
| powershell/Maester.psd1 | Module manifest updated to export 7 new SPO functions |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
SamErde
added
enhancement
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
|
BTW, you do not need to specify a severity tag in the Pester test files if they are added to the |
|
@HenrikPiecha one catch I noticed with the SPO cmdlets is they don't work on non Windows devices like macOS. Do you know if these tests can make use of PnP PowerShell? I believe that module has been updated PS7. https://learn.microsoft.com/en-us/powershell/sharepoint/sharepoint-pnp/sharepoint-pnp-cmdlets I believe this would also solve the interactive issue because PnP supports app only as well. Do you know off the top of your head if these cmdlets are only available in SPO. Another addition we need to make is to If not, folks would never know that they need to connect to SPO/PnP to run thes |
|
@HenrikPiecha are you open to changing these SPO calls on PnP? Without it these tests are very unlikely to be run by a majority user Maester users that run on GitHub actions since these containers are all Linux. So none of the Windows PowerShell dependent modules like SPO will work. Thoughts? If you don't have bandwidth are you okay if I update them to PnP PS? |
|
@merill I'll give the PnP Option a try, for now I struggeled with getting this to work with existing App registrations. |
|
@merill It really wasn't as complicated as I thought.. The changes I did to Connect-Maester, I would revert to query the tenant domain via graph api "organization" via: $org = Invoke-MtGraphRequest -RelativeUri "organization"
$tenantDomain = ($org.verifiedDomains | Where-Object {$_.isInitial -eq $true}) | Select-Object -ExpandProperty Name
$spoUri = "https://$(($tenantDomain).Replace(".onmicrosoft.com",$null))-admin.sharepoint.com"To connect to PnP it would switch to following for non-interactive login. Connect-PnPOnline -Url $spoUri -Tenant $tenantDomain -ClientId $GraphClientId -Thumbprint $CertThumbprint -ErrorAction Stop # CertThumbprint variable unknown?!For Interactive logins we would require a custom app registration with redirect uri "mobile and desktop applications" listening on "http://localhost" and api permissions "SharePoint > Sites.FullControl.All" Connect-PnPOnline -Url $spoUri -ClientId $GraphClientId -InteractiveWithin all the tests we simply would run "Get-PnPTenant" instead of "Get-SpoTenant", performance wise we could add the results after first execution in a "CmdletCache" and keep on calling from there. Let me know what you think. Should I add a new PR to start from "scratch"? Some new tests i have been working on, checking the SharePoint Online default orgwide sharing link expiration that will be GA in the next weeks are currently not working with PnP as I can't query the property.
|
Up to standards ✅🟢 Issues
|
|
@HenrikPiecha yes let's do this. For the interactive, we don't really need a custom app since we already have the Graph PowerShell app which does have localhost on it. I believe all we need to do is add the SPO scope when calling connect-graph. For the ones that are not there, maybe opening an issue in the PnP repo would help us track so we know when its available to then use it in maester... |
|
Just adding a note on this one. Looking at the Appendix: Change History in the document there is the following updates/changes to Chapter 7
|
So are those all tests that can be removed or updated in our version? |
I can see none of the tests with changes 7.3.3, 7.3.4, 7.2.6 is included in this PR so my comments can be ignored :D But if 7.3.3, 7.3.4 was in it they could be removed and 7.2.6 would have needed an update for docs & maybe title and then validate the audit logic |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…dItem.ps1 Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…dItem.ps1 Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…usFile.ps1 Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 24 out of 24 changed files in this pull request and generated 8 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # Check for prerequisite before running the tests. If the prerequisites are not met, return $null to skip the tests. | ||
| # Following checks are performed to to current limitations of the Microsoft.Online.SharePoint.PowerShell module: | ||
| # - Run on Windows OS | ||
| # - Run in PowerShell 5 not 7 | ||
| # - Check if the module is installed and loaded | ||
| # - Check if the user is connected to SharePoint Online with Connect-SpoService | ||
| if (-not ($PSVersionTable.OS -match "Windows" -or [System.Environment]::OSVersion.VersionString -match "Windows")) { | ||
| Write-Host "SharePoint Online tests can only be run on Windows OS. Skipping tests..." -ForegroundColor Yellow | ||
| return $null | ||
| } | ||
| if ($PSVersionTable.PSVersion.Major -ne 5) { | ||
| Write-Host "SharePoint Online tests can only be run in PowerShell 5. Skipping tests..." -ForegroundColor Yellow | ||
| return $null | ||
| } | ||
| if (-not (Get-Module -Name Microsoft.Online.SharePoint.PowerShell -ListAvailable)) { | ||
| Write-Host "Microsoft.Online.SharePoint.PowerShell module is not installed. Skipping tests..." -ForegroundColor Yellow | ||
| return $null | ||
| } | ||
| if (-not (Get-Module -Name Microsoft.Online.SharePoint.PowerShell)) { | ||
| Write-Host "Microsoft.Online.SharePoint.PowerShell module is not imported. Skipping tests..." -ForegroundColor Yellow | ||
| return $null | ||
| } | ||
| if (-not (Get-SpoTenant)) { | ||
| Write-Host "Not connected to SharePoint Online. Please connect using Connect-SpoService before running the tests. Skipping tests..." -ForegroundColor Yellow | ||
| return $null | ||
| } | ||
| } | ||
|
|
||
| Describe 'Maester/SpoTenant' -Tag 'Maester', 'SpoTenant' { |
There was a problem hiding this comment.
The prerequisite checks in BeforeDiscovery return $null, but the Describe block is still defined unconditionally. In this pattern, returning from BeforeDiscovery won’t actually skip the tests; it can still attempt to run and/or fail during discovery. Consider computing a $skip boolean + reason in BeforeDiscovery and applying -Skip:(...) (as done in other test files) to the Describe/It blocks instead of relying on return $null here.
| # Check for prerequisite before running the tests. If the prerequisites are not met, return $null to skip the tests. | |
| # Following checks are performed to to current limitations of the Microsoft.Online.SharePoint.PowerShell module: | |
| # - Run on Windows OS | |
| # - Run in PowerShell 5 not 7 | |
| # - Check if the module is installed and loaded | |
| # - Check if the user is connected to SharePoint Online with Connect-SpoService | |
| if (-not ($PSVersionTable.OS -match "Windows" -or [System.Environment]::OSVersion.VersionString -match "Windows")) { | |
| Write-Host "SharePoint Online tests can only be run on Windows OS. Skipping tests..." -ForegroundColor Yellow | |
| return $null | |
| } | |
| if ($PSVersionTable.PSVersion.Major -ne 5) { | |
| Write-Host "SharePoint Online tests can only be run in PowerShell 5. Skipping tests..." -ForegroundColor Yellow | |
| return $null | |
| } | |
| if (-not (Get-Module -Name Microsoft.Online.SharePoint.PowerShell -ListAvailable)) { | |
| Write-Host "Microsoft.Online.SharePoint.PowerShell module is not installed. Skipping tests..." -ForegroundColor Yellow | |
| return $null | |
| } | |
| if (-not (Get-Module -Name Microsoft.Online.SharePoint.PowerShell)) { | |
| Write-Host "Microsoft.Online.SharePoint.PowerShell module is not imported. Skipping tests..." -ForegroundColor Yellow | |
| return $null | |
| } | |
| if (-not (Get-SpoTenant)) { | |
| Write-Host "Not connected to SharePoint Online. Please connect using Connect-SpoService before running the tests. Skipping tests..." -ForegroundColor Yellow | |
| return $null | |
| } | |
| } | |
| Describe 'Maester/SpoTenant' -Tag 'Maester', 'SpoTenant' { | |
| # Check for prerequisites before running the tests. If the prerequisites are not met, | |
| # mark the suite to be skipped during discovery. | |
| # Following checks are performed due to current limitations of the Microsoft.Online.SharePoint.PowerShell module: | |
| # - Run on Windows OS | |
| # - Run in PowerShell 5 not 7 | |
| # - Check if the module is installed and loaded | |
| # - Check if the user is connected to SharePoint Online with Connect-SpoService | |
| $script:skipSpoTenantTests = $false | |
| $script:skipSpoTenantTestsReason = $null | |
| if (-not ($PSVersionTable.OS -match "Windows" -or [System.Environment]::OSVersion.VersionString -match "Windows")) { | |
| $script:skipSpoTenantTests = $true | |
| $script:skipSpoTenantTestsReason = "SharePoint Online tests can only be run on Windows OS. Skipping tests..." | |
| } | |
| elseif ($PSVersionTable.PSVersion.Major -ne 5) { | |
| $script:skipSpoTenantTests = $true | |
| $script:skipSpoTenantTestsReason = "SharePoint Online tests can only be run in PowerShell 5. Skipping tests..." | |
| } | |
| elseif (-not (Get-Module -Name Microsoft.Online.SharePoint.PowerShell -ListAvailable)) { | |
| $script:skipSpoTenantTests = $true | |
| $script:skipSpoTenantTestsReason = "Microsoft.Online.SharePoint.PowerShell module is not installed. Skipping tests..." | |
| } | |
| elseif (-not (Get-Module -Name Microsoft.Online.SharePoint.PowerShell)) { | |
| $script:skipSpoTenantTests = $true | |
| $script:skipSpoTenantTestsReason = "Microsoft.Online.SharePoint.PowerShell module is not imported. Skipping tests..." | |
| } | |
| elseif (-not (Get-SpoTenant)) { | |
| $script:skipSpoTenantTests = $true | |
| $script:skipSpoTenantTestsReason = "Not connected to SharePoint Online. Please connect using Connect-SpoService before running the tests. Skipping tests..." | |
| } | |
| if ($script:skipSpoTenantTests) { | |
| Write-Host $script:skipSpoTenantTestsReason -ForegroundColor Yellow | |
| } | |
| } | |
| Describe 'Maester/SpoTenant' -Tag 'Maester', 'SpoTenant' -Skip:($script:skipSpoTenantTests) { |
| Write-Host "Microsoft.Online.SharePoint.PowerShell module is not imported. Skipping tests..." -ForegroundColor Yellow | ||
| return $null | ||
| } | ||
| if (-not (Get-SpoTenant)) { |
There was a problem hiding this comment.
if (-not (Get-SpoTenant)) is used as a connection check during discovery, but Get-SPOTenant typically throws when not connected; this can break test discovery rather than skipping cleanly. Wrap this call in try/catch (with -ErrorAction Stop) and set the skip condition based on whether the call succeeds.
| if (-not (Get-SpoTenant)) { | |
| $isSpoConnected = $true | |
| try { | |
| Get-SPOTenant -ErrorAction Stop | Out-Null | |
| } | |
| catch { | |
| $isSpoConnected = $false | |
| } | |
| if (-not $isSpoConnected) { |
| It 'MT.1114: Ensure Office 365 SharePoint infected files are disallowed for download' -Tag 'MT.1114', 'CIS', 'CIS M365v5', 'CIS 7.3.1', 'Severity:High' { | ||
| $result = Test-MtSpoPreventDownloadMaliciousFile | ||
| if ($null -ne $result) { | ||
| $result | Should -Be $true -Because 'Office 365 SharePoint infected files are disallowed for download.' | ||
| } |
There was a problem hiding this comment.
This test is labeled/tagged as MT.1114 but the title/CIS mapping and the invoked function (Test-MtSpoPreventDownloadMaliciousFile) correspond to the infected-file-download control documented as MT.1119 / CIS 7.3.1. Please correct the ID/tag to MT.1119, and add a separate MT.1114 test once Test-MtSpoCustomScriptExecutionOnSiteCollection exists (it’s currently referenced in the manifest/config).
|
@copilot resolve the merge conflicts in this pull request |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…dItem.ps1 Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
|
@copilot resolve the merge conflicts in this pull request |
Description
Added SharePoint Online admin center tests from CIS Microsoft 365 v5.0.0
All the tests are related to the PowerShell module Microsoft.Online.SharePoint.PowerShell.
@merill please check the invocation as discussed.
The tests with Tag
SpoTenantshould only be executed when running interactivly as the ps module currently does not support read-only permission or service principal authentication from powershell 7.Running the connection is mandatory:
Contribution Checklist
Before submitting this PR, please confirm you have completed the following:
/powershell/tests/pester.ps1on your local system.Join us at the Maester repository discussions 💬 or Entra Discord 🧑💻 for more help and conversations!