🙏 Handle tenant config as a custom config

[fflaten]

Describe the feature

I would like Maester to handle the new tenant-config as overrides like tests/Custom/maester-config.json. The tenant-config should either:

a) Replace the old custom-config completely. Breaking change, but easily fixed by renaming/copying tests/Custom/maester-config.json to tests/Custom/maester-config.{tenantId}.json
b) Merge settings using the following priority:

1. Tenant-specific overrides: tests/Custom/maester-config.{tenantId}.json
2. Global overrides: tests/Custom/maester-config.json
3. Baseline: tests/maester-config.json

Impact and importance

Simplify the config story after the introduction of multi-tenant report (#1558).

Before, the default config was tests/maester-config.json, with any user-specific overrides (only) placed in the custom config tests/Custom/maester-config.json.

Now, the full config can be either the tenant-specific tests/maester-config.{tenantId}.json or the global tests/maester-config.json, resulting in:

• Duplicate code. Likely 99% of every tenant-config
• Manual work required for all tenant-configs to merge updates from global config after running Update-MaesterTests
• tests/Custom/maester-config.json being in a weird place as it overrides all tenants. A common use-case was defining emergency accounts, but they are tenant-specific and must be set in the full tenant-config.
• Conflicting recommendations. Modifying a full config is not recommended. See warning here and related discussion Change Export Config button to only export non-default values? #1461

Alternatives you've considered

No response

Additional context

No response

[merill]

@fflaten Agree, this is a good way to solve it.

I prefer option b since it will allow orgs to configure settings that work across tenants and gradually step up from a single tenant to multi-tenant.

The export is going to be interesting.

We need the ux to show the three levels of settings.. and export accordingly...

[fflaten]

I prefer option b since it will allow orgs to configure settings that work across tenants and gradually step up from a single tenant to multi-tenant.

Thanks. Agreed - I expect most testsettings overrides as global (org-level) and not tenant-specific. Ex severity-changes and disabling inapplicable tests due to policy or alternative solutions (PAM vs PIM etc).

The export is going to be interesting.

We need the ux to show the three levels of settings.. and export accordingly...

Already noted in the related discussion 👍

[fflaten]

I prefer option b since it will allow orgs to configure settings that work across tenants and gradually step up from a single tenant to multi-tenant.

Thanks. Agreed - I expect most testsettings overrides as global (org-level) and not tenant-specific. Ex severity-changes and disabling inapplicable tests due to policy or alternative solutions (PAM vs PIM etc).

The export is going to be interesting.

We need the ux to show the three levels of settings.. and export accordingly...

Already noted in the related discussion 👍

[fflaten]

Be aware that option b) has the minor breaking change as well by moving the tenant-config to Custom-folder for consistency.