From 704d59a1c960087593a405c3d6229e63a3ef8483 Mon Sep 17 00:00:00 2001 From: Michael Flanakin Date: Sat, 4 Apr 2026 11:54:22 -0700 Subject: [PATCH 01/30] Update release template --- .../-internal-only--release-checklist.md | 185 ++++++++++-------- 1 file changed, 101 insertions(+), 84 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/-internal-only--release-checklist.md b/.github/ISSUE_TEMPLATE/-internal-only--release-checklist.md index a1ec9d21c..b016ef208 100644 --- a/.github/ISSUE_TEMPLATE/-internal-only--release-checklist.md +++ b/.github/ISSUE_TEMPLATE/-internal-only--release-checklist.md @@ -1,10 +1,9 @@ --- -name: "(Internal only) Release checklist" +name: '(Internal only) Release checklist' about: List of tasks needed to ship a new FinOps toolkit release title: "\U0001F4CB Mmmm yyyy (v#.#) release checklist" -labels: "Type: Release \U0001F680" +labels: "Skill: Deployment, Type: Release \U0001F680" assignees: '' - --- Complete the following tasks to publish a monthly release. @@ -18,8 +17,8 @@ Status icons: ## 🔄ī¸ Monthly updates -- [ ] Update open data files ([learn more](https://github.com/microsoft/finops-toolkit/tree/dev/src/open-data)). - - [ ] Pricing units +- ❌ Update open data files ([learn more](https://github.com/microsoft/finops-toolkit/tree/dev/src/open-data)). + - ❌ Pricing units 1. Run the extraction query. 2. Replace the contents of the corresponding [open data](https://github.com/microsoft/finops-toolkit/tree/dev/src/open-data) files. 3. Compare changes and revert any invalid updates. @@ -27,7 +26,7 @@ Status icons: 5. Review changes and revert unrelated files or unintended changes. 6. Document the added/updated rows in the changelog. 7. Publish a PR for updates - - [ ] Regions + - ❌ Regions 1. Run the extraction query. 2. Replace the contents of the corresponding [open data](https://github.com/microsoft/finops-toolkit/tree/dev/src/open-data) files. 3. Compare changes and revert any invalid updates. @@ -35,7 +34,7 @@ Status icons: 5. Review changes and revert unrelated files or unintended changes. 6. Document the added/updated rows in the changelog. 7. Publish a PR for updates - - [ ] Services + - ❌ Services 1. Run the extraction query. 2. Replace the contents of the corresponding [open data](https://github.com/microsoft/finops-toolkit/tree/dev/src/open-data) files. 3. Compare changes and revert any invalid updates. @@ -43,8 +42,17 @@ Status icons: 5. Review changes and revert unrelated files or unintended changes. 6. Document the added/updated rows in the changelog. 7. Publish a PR for updates - - [ ] Update resource types and re-gen internal PowerShell functions: `/src/scripts/Build-OpenData.ps1 -PowerShell -Test` - - [ ] Consider automating this process in [Build-OpenData](https://github.com/microsoft/finops-toolkit/tree/dev/src/scripts/Build-OpenData.ps1). + - ❌ Resource types + 1. Generate ResourceTypes.json: `/src/scripts/Build-OpenData.ps1 ResourceTypes -Json` + 2. Review changes and revert unrelated files or unintended changes. + - Consider automating any of the reverted checks to streamline the process next time. + 3. Generate ResourceTypes.csv: `/src/scripts/Build-OpenData.ps1 ResourceTypes -Csv` + 4. Review changes and revert unrelated files or unintended changes. + 5. Document the added/updated rows in the changelog. + 6. Publish a PR for updates + - ❌ Update FinOps hubs KQL functions: `/src/scripts/Build-OpenData.ps1 -Hubs` + - ❌ Re-gen internal PowerShell functions: `/src/scripts/Build-OpenData.ps1 -PowerShell -Test` + - ❌ Consider automating this process in [Build-OpenData](https://github.com/microsoft/finops-toolkit/tree/dev/src/scripts/Build-OpenData.ps1). - [ ] Update Bicep CLI ([docs](https://learn.microsoft.com/azure/azure-resource-manager/bicep/install#azure-cli)): `az bicep upgrade` --- @@ -57,94 +65,84 @@ Status icons: - [ ] Submit PRs for issues that can be resolved. - [ ] Complete [open PRs](https://github.com/microsoft/finops-toolkit/pulls) that are ready to be resolved. - [ ] Confirm there are no pending changes in dev. + ```powershell cd "" git checkout dev git status ``` + - [ ] Update feature branches and confirm if they're ready for release. - [ ] Auto-merge feature branch with `dev`: `/src/scripts/Merge-DevBranch.ps1 *` - - ~Manually update remaining feature branches: `/src/scripts/Merge-DevBranch.ps1 features/`~ - - ~features/TODO~ - - ~Commit the changes and push to the feature branch.~ - - [ ] Build and deploy templates: `/src/scripts/Build-Toolkit.ps1`. - - [ ] Confirm all tests pass: `/src/scripts/Test-PowerShell.ps1 -Unit -Integration -Build`. - - [ ] Confirm if all features are code complete and not missing any functionality required for release. - > _Once in `dev`, the feature is considered part of the next release and can be pushed out at any time. Any broken features will be reverted._ - - [ ] Confirm new or updated functionality is documented in the [changelog](https://github.com/microsoft/finops-toolkit/blob/dev/docs-mslearn/toolkit/changelog.md). - - [ ] Confirm new or updated functionality must be documented in the [documentation](https://github.com/microsoft/finops-toolkit/blob/dev/docs). - - [ ] If adding a new tool, update the [list of available tools](https://aka.ms/finops/toolkit#available-tools) on the documentation home page. + - ❌ Manually update remaining feature branches: `/src/scripts/Merge-DevBranch.ps1 features/` +- [ ] Build and deploy templates: `/src/scripts/Build-Toolkit.ps1`. +- [ ] Confirm all tests pass: `/src/scripts/Test-PowerShell.ps1 -Unit -Integration`. +- [ ] Confirm if all features are code complete and not missing any functionality required for release. + > _Once in `dev`, the feature is considered part of the next release and can be pushed out at any time. Any broken features will be reverted._ +- [ ] Confirm new or updated functionality is documented in the [changelog](https://github.com/microsoft/finops-toolkit/blob/dev/docs/changelog.md). + > _See the changelog for details about changelog requirements._ +- [ ] Confirm new or updated functionality must be documented in the [documentation](https://github.com/microsoft/finops-toolkit/blob/dev/docs). +- [ ] Summarize the monthly changes in the "what's new" section of the marketing pages for each tool. +- [ ] Update the FinOps hubs compatibility guide and upgrade guide. +- [ ] Do a global search and replace for "FTKx.x" for the previous version to replace it with the new version number. +- [ ] If adding a new tool: + - [ ] Update the [marketing site landing page](https://github.com/microsoft/finops-toolkit/tree/dev/docs/README.md). + - [ ] Add a new page the [marketing site](https://github.com/microsoft/finops-toolkit/tree/dev/docs). + - [ ] Add an overview page in [MS Learn documentation](https://github.com/microsoft/finops-toolkit/tree/dev/docs-mslearn/toolkit). + - [ ] Update the [MS Learn menu to add new page(s)](https://github.com/microsoft/finops-toolkit/tree/dev/docs-mslearn/TOC.yml). + - [ ] Add the new tool's tech lead to the [Advisor council doc](https://github.com/microsoft/finops-toolkit/tree/dev/docs-wiki/Advisory-council.md). - [ ] Merge any feature branches that are ready to `dev`. - - [ ] Create a PR to merge the feature branch into `dev`. - - [ ] Follow the normal PR process to merge the PR. + - Create a PR to merge the feature branch into `dev`. + - Follow the normal PR process to merge the PR. --- ## 🔜 Finalize release -- [ ] Review the [changelog](https://github.com/microsoft/finops-toolkit/blob/dev/docs-mslearn/toolkit/changelog.md) to ensure it encapsulates all changes. +- [ ] Review the [changelog](../docs/_resources/changelog.md) to ensure it encapsulates all changes. - Move all released changes to an official numbered version section. - If there are committed changes in a feature branch that you want to mention, add them to an "Unreleased" section. -- [ ] Update the version. - ```powershell - /src/scripts/Update-Version [-Major|Minor|Patch] - ``` -- [ ] Build all toolkit templates and resolve any issues. +- [ ] Update the version: `/src/scripts/Update-Version.ps1 [-Major|Minor|Patch]` +- [ ] Build all toolkit templates and resolve any issues: `/src/scripts/Build-Toolkit` > _This step is optional, but can catch issues earlier. You can also add the `-Build` parameter to the publish command in the next step._ - ```powershell - /src/scripts/Build-Toolkit - ``` -- [ ] Ensure all tests pass: - ```powershell - /src/scripts/Test-PowerShell - ``` -- [ ] Package all release files - - [ ] Run `Package-Toolkit -Build -PowerBI` script. - - [ ] For the Cost summary report: - - [ ] Save the file as a PBIX file to the release folder. - - [ ] Change the sensitivity to **Public**. If the option is disabled, close the file and reopen it. - > ⚠ī¸ _Power BI does not remember the sensitivity setting for Power BI projects so this needs to be done for each release. If not done, the report will not open for anyone outside of Microsoft._ - - [ ] Remove the following queries from Transform data: **Hubs\***, **InstanceSizeFlexibility**, **Reservation\***, **Storage\***. - - [ ] Verify each page has data. - - [ ] Select the Get started page and save the PBIX again in the release folder. - > ⚠ī¸ _**DO NOT** save the above changes back to the Power BI project files!_ - - [ ] Copy the first paragraph from the **Get started** page and export a template (PBIT file) in the release folder. Use the copied text for the description and add "Learn more at https://aka.ms/ftk/pbi/{report-name}" as a separate paragraph in the description. - - [ ] For the Data ingestion report: - - [ ] Save the file as a PBIX file to the release folder. - - [ ] Change the sensitivity to **Public**. If the option is disabled, close the file and reopen it. - > ⚠ī¸ _Power BI does not remember the sensitivity setting for Power BI projects so this needs to be done for each release. If not done, the report will not open for anyone outside of Microsoft._ - - [ ] Remove the following queries from Transform data: **InstanceSizeFlexibility**, **Prices**, **Reservation\***. - - [ ] Verify each page has data. - - [ ] Select the Get started page and save the PBIX again in the release folder. - > ⚠ī¸ _**DO NOT** save the above changes back to the Power BI project files!_ - - [ ] Copy the first paragraph from the **Get started** page and export a template (PBIT file) in the release folder. Use the copied text for the description and add "Learn more at https://aka.ms/ftk/pbi/{report-name}" as a separate paragraph in the description. - - [ ] For the Rate optimization report: - - [ ] Save the file as a PBIX file to the release folder. - - [ ] Change the sensitivity to **Public**. If the option is disabled, close the file and reopen it. - > ⚠ī¸ _Power BI does not remember the sensitivity setting for Power BI projects so this needs to be done for each release. If not done, the report will not open for anyone outside of Microsoft._ - - [ ] Remove the following queries from Transform data: **Hubs\***, **Storage\***. - - [ ] Verify each page has data. - - [ ] Select the Get started page and save the PBIX again in the release folder. - > ⚠ī¸ _**DO NOT** save the above changes back to the Power BI project files!_ - - [ ] Copy the first paragraph from the **Get started** page and export a template (PBIT file) in the release folder. Use the copied text for the description and add "Learn more at https://aka.ms/ftk/pbi/{report-name}" as a separate paragraph in the description. +- [ ] Ensure all tests pass: `/src/scripts/Test-PowerShell -Unit -Integration` +- [ ] Package all release files + - [ ] Run `Package-Toolkit.ps1 -Build -CopyFiles -OpenPBI` script. + - [ ] Save and close each Power BI project: + - Select the `/release/pbix` folder. + - Change the file extension to PBIX. + - When prompted, set the sensitivity to "Public". + - Manually remove unused queries based on what's documented in Build-PowerBI.ps1 ~line 120. + - Verify all pages, switch to the Get started page, and save again. + - [ ] Run `Package-Toolkit -ZipPBI` script. - [ ] Check the docs for broken links: - - [ ] Create a personal fork of the main repo. - - [ ] If you already have one, update it to the latest. - - [ ] Enable GitHub pages in your fork to use the `dev` branch `docs` folder. - - [ ] Verify the `pages build and deployment` action completes successfully. - - [ ] Open a dead link checker (e.g., [deadlinkchecker.com](https://www.deadlinkchecker.com/website-dead-link-checker.asp)) and check `https://{your-username}.github.io/finops-toolkit`. + - Create a personal fork of the main repo. + - If you already have one, update it to the latest. + - Enable GitHub pages in your fork to use the `dev` branch `docs` folder. + - Verify the `pages build and deployment` action completes successfully. + - Open a dead link checker (e.g., [deadlinkchecker.com](https://www.deadlinkchecker.com/website-dead-link-checker.asp)) and check `https://{your-username}.github.io/finops-toolkit`. - Ignore **link/href** errors for new pages that have not been released in the official `main` branch. - Ignore any **Download** errors for the new release or any new files that haven't been released in the official `main` branch. - Ignore the **Full changelog** error for the new release, which hasn't been released in the official `main` branch. - Ignore the 403 error for learn.finops.org. - Ignore any 429 errors from GitHub. These are caused due to all the contributor links that are checked. - - [ ] Fix any broken links, push changes, and rerun the tool. + - Fix any broken links, push changes, and rerun the tool. --- ## 🔜 Publish release - [ ] Publish the PowerShell module by running the [Publish PowerShell action](https://github.com/microsoft/finops-toolkit/actions/workflows/publish.yml). +- [ ] Submit PR to publish docs to the Microsoft Docs repo. + - [ ] Update your fork and local clone (main branch) of the Microsoft Docs repo. + - [ ] Publish a PR: + - Start PR: `/src/scripts/Publish-Toolkit.ps1 "docs" -Commit` + - Complete PR template requirements and set the name to `FinOps toolkit v#.# doc updates`. + - Review the code that's changed in the PR, verify that the changelog covers everything, and update as needed. + - If you make changes, re-publish without committing: `/src/scripts/Publish-Toolkit.ps1 "docs" -Build` + - Switch to the target repo folder and verify your changes were applied correctly. + - Commit and push your changes. + - Return to the PR URL and publish the PR. - [ ] Submit PRs to publish bicep modules in the Bicep Registry. - [ ] Update your fork and local clone of the Bicep Registry. > _Make sure you're in the main branch of the target repo when publishing a new template. If in another folder, the script will assume you're updating that branch and not create a new one._ @@ -188,7 +186,7 @@ Status icons: - Fix any issues in the FinOps toolkit codebase (not in the target repo). - Repeat publishing steps until all issues have been resolved. - [ ] Check back after 2-3 days to see if all your PRs were merged ([AQT](https://github.com/Azure/azure-quickstart-templates/pulls?q=is%3Apr+is%3Aopen+finops)). - > _Bicep Registry and Azure Quickstart Template repos are manually reviewed and closed 2-3x per week. If you don't see an update, contact the ARM team._ + > _Bicep Registry and Azure Quickstart Template repos are manually reviewed and closed 2-3x per week. If you don't see an update, contact the ARM team. The Docs repo is partially typically reviewed within a few hours but they may request blocking changes._ - [ ] Merge to main: `/src/scripts/Merge-DevBranch.ps1 main` - [ ] Update the [milestone](https://github.com/microsoft/finops-toolkit/milestones). - Review all issues in the milestone, move anything that needs to be pushed, and close any completed items. @@ -200,7 +198,7 @@ Status icons: - [ ] Set the **Target** to `main`. - [ ] Set the **Previous tag** to the previous release tag. - [ ] Set the name to `Mmm yyyy (v#.#)`. - - [ ] Copy the body from the previous release to use as a template. + - [ ] Copy the body from the [previous release](https://github.com/microsoft/finops-toolkit/releases) to use as a template. - [ ] Change the "New in" header to use the new version number. - [ ] Summarize changes from the changelog in the **New in** and **Updated** sections. - Simplify to only include one line per tool. @@ -224,28 +222,47 @@ Status icons: --- +## 🔜 Announcements + +- [ ] Publish release announcement on the [FinOps blog](https://aka.ms/finops/blog) +- [ ] Share on LinkedIn +- [ ] Share on Twitter/X +- [ ] Share on Azure Updates (TBD) +- [ ] Share on Engage (Azure Connection Program) +- [ ] Share on F2 Slack + +--- + ## 🔜 Post-release clean-up - [ ] Update the `Toolkit / Should return all known releases` PowerShell integration test based on the latest version. > _See `src/powershell/Tests/Integration/Toolkit.Tests.ps1` > `Get-FinOpsToolkitVersion` > `Should return all known releases`_ - - [ ] Add the latest public version to the `$expected` variable. - - [ ] Update the file checks to include/exclude any new/removed files. + - Add the latest public version to the `$expected` variable. + - Update the file checks to include/exclude any new/removed files. - [ ] Update the minor version and add the "dev" label: `/src/scripts/Update-Version.ps1 -Minor -Label dev` +- [ ] Create a placeholder in the changelog for the new release. + + ```markdown + ## vX.Y + + _Released Mmm yyyy_ + + > [!div class="nextstepaction"] > [Download](https://github.com/microsoft/finops-toolkit/releases/tag/vX.Y) > [!div class="nextstepaction"] > [Full changelog](https://github.com/microsoft/finops-toolkit/compare/vX.Y...vX.Y) + +
+ ``` + - [ ] Update remaining branches that were not merged with dev - [ ] TODO: Add branches - - [ ] features/adx - - [ ] features/alerts - - [ ] features/mslearn - - [ ] features/private - - [ ] features/savings - - [ ] features/services + - [ ] features/anomaly + - [ ] features/finley + - [ ] features/recs - [ ] features/ux -- [ ] Confirm the following branches are needed or should be deleted: - - [ ] features/aoe - - [ ] features/governance + - [ ] features/xcloud +- [ ] Copy any additional code from the following branches and delete them: - [ ] features/hack24 - [ ] features/hourly - - [ ] features/powershell - - [ ] features/privateendpoint +- [ ] Delete the following branches: + - [ ] features/alerts
From 7e77eb01c8e4227864af71c294c07bfe48664a38 Mon Sep 17 00:00:00 2001 From: Michael Flanakin Date: Sat, 4 Apr 2026 15:05:08 -0700 Subject: [PATCH 02/30] Initial Start-Release script --- .claude/commands/release.md | 24 ++++++ src/scripts/Start-Release.ps1 | 147 ++++++++++++++++++++++++++++++++++ 2 files changed, 171 insertions(+) create mode 100644 .claude/commands/release.md create mode 100644 src/scripts/Start-Release.ps1 diff --git a/.claude/commands/release.md b/.claude/commands/release.md new file mode 100644 index 000000000..04a4eb37f --- /dev/null +++ b/.claude/commands/release.md @@ -0,0 +1,24 @@ +--- +description: Walk through the release checklist interactively +allowed-tools: Read, Grep, Bash(pwsh -Command ./src/scripts/Start-Release*), Bash(gh issue view *) +--- + +# Release + +Assists in the FinOps toolkit release process. + +## Process + +### Phase 1: Setup + +Run `pwsh -Command ./src/scripts/Start-Release.ps1` and report the result to the user. + +### Future phases + +Additional phases will be added incrementally as we work through the release process: + +- **Phase 2: Monthly updates** — Open data, Bicep CLI +- **Phase 3: Core features** — Branch merges, build, test, version references +- **Phase 4: Finalize** — Changelog, version bump, packaging +- **Phase 5: Publish** — PowerShell, Bicep Registry, AQT, docs, GitHub release +- **Phase 6: Post-release** — Dev version bump, branch cleanup diff --git a/src/scripts/Start-Release.ps1 b/src/scripts/Start-Release.ps1 new file mode 100644 index 000000000..ddf3a31dc --- /dev/null +++ b/src/scripts/Start-Release.ps1 @@ -0,0 +1,147 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. + +<# + .SYNOPSIS + Starts or continues the release process. + + .DESCRIPTION + The Start-Release command determines the current release state and performs the next steps that can be automated. + Currently, it ensures a release tracking issue exists on GitHub. + + .PARAMETER DaysUntilRelease + Optional. Number of days until the expected release date. Used to determine the release month when running + near the end of a month. Default = 7. + + .EXAMPLE + ./Start-Release + + Detects the version from package.json, checks for an existing release issue, and creates one if needed. + Uses the date 7 days from now to determine the release month. + + .EXAMPLE + ./Start-Release -DaysUntilRelease 14 + + Uses the date 14 days from now to determine the release month. +#> +param( + [int] + $DaysUntilRelease = 7 +) + +$ErrorActionPreference = 'Stop' +$repoRoot = (Resolve-Path (Join-Path $PSScriptRoot '../..')).Path + +# Get version from package.json +$version = & (Join-Path $PSScriptRoot 'Get-Version.ps1') +$versionTag = & (Join-Path $PSScriptRoot 'Get-Version.ps1') -AsTag + +# Determine release month/year based on expected release date +$releaseDate = (Get-Date).AddDays($DaysUntilRelease) +$Month = $releaseDate.ToString('MMMM') +$Year = $releaseDate.Year + +Write-Host "Release: $versionTag ($Month $Year)" -ForegroundColor Cyan +Write-Host "" + +# --- Step 1: Ensure release issue exists --- + +# Read the issue template +$templatePath = Join-Path $repoRoot '.github/ISSUE_TEMPLATE/-internal-only--release-checklist.md' +if (-not (Test-Path $templatePath)) +{ + Write-Error "Issue template not found at $templatePath" + return +} + +$templateContent = Get-Content $templatePath -Raw + +# Parse YAML frontmatter +$frontmatterMatch = [regex]::Match($templateContent, '(?s)^---\r?\n(.*?)\r?\n---\r?\n(.*)') +if (-not $frontmatterMatch.Success) +{ + Write-Error "Failed to parse YAML frontmatter from issue template" + return +} + +$frontmatter = $frontmatterMatch.Groups[1].Value +$body = $frontmatterMatch.Groups[2].Value + +# Extract title pattern from frontmatter +$titleMatch = [regex]::Match($frontmatter, 'title:\s*"(.+)"') +if (-not $titleMatch.Success) +{ + Write-Error "Failed to extract title from issue template frontmatter" + return +} +$titlePattern = $titleMatch.Groups[1].Value + +# Extract labels from frontmatter +$labelsMatch = [regex]::Match($frontmatter, 'labels:\s*"(.+)"') +$labels = if ($labelsMatch.Success) { $labelsMatch.Groups[1].Value } else { '' } + +# Fill in placeholders +$title = $titlePattern ` + -replace 'Mmmm', $Month ` + -replace 'yyyy', $Year ` + -replace '#\.#', $version ` + -replace '\\U0001F4CB', ([char]::ConvertFromUtf32(0x1F4CB)) + +$title = $title -replace '\\U([0-9A-Fa-f]{8})', { [char]::ConvertFromUtf32([Convert]::ToInt32($_.Groups[1].Value, 16)) } + +# Check for existing release issue +Write-Host "Checking for existing release issue..." -ForegroundColor Gray +$existingIssues = gh issue list --state open --label "Type: Release 🚀" --search "$versionTag release checklist" --json number,title,url 2>&1 +if ($LASTEXITCODE -ne 0) +{ + Write-Error "Failed to search for existing issues: $existingIssues" + return +} + +$issues = $existingIssues | ConvertFrom-Json +if ($issues.Count -gt 0) +{ + $issue = $issues[0] + Write-Host "Found existing release issue: #$($issue.number) - $($issue.title)" -ForegroundColor Green + Write-Host " $($issue.url)" -ForegroundColor Gray + return @{ + Action = 'existing' + Number = $issue.number + Title = $issue.title + Url = $issue.url + } +} + +# Create the issue +Write-Host "Creating release issue..." -ForegroundColor Gray +Write-Host " Title: $title" -ForegroundColor Gray + +# Write body to a temp file to avoid shell escaping issues +$tempFile = [System.IO.Path]::GetTempFileName() +try +{ + Set-Content -Path $tempFile -Value $body -NoNewline + $result = gh issue create --title $title --label $labels --body-file $tempFile 2>&1 + if ($LASTEXITCODE -ne 0) + { + Write-Error "Failed to create issue: $result" + return + } + + # gh issue create returns the URL + $issueUrl = $result.Trim() + $issueNumber = [regex]::Match($issueUrl, '/issues/(\d+)').Groups[1].Value + + Write-Host "Created release issue: #$issueNumber" -ForegroundColor Green + Write-Host " $issueUrl" -ForegroundColor Gray + return @{ + Action = 'created' + Number = [int]$issueNumber + Title = $title + Url = $issueUrl + } +} +finally +{ + Remove-Item $tempFile -ErrorAction SilentlyContinue +} From 7f262f435e4f6c833435c763b8fc864f161640cc Mon Sep 17 00:00:00 2001 From: Michael Flanakin Date: Sat, 4 Apr 2026 15:51:23 -0700 Subject: [PATCH 03/30] Pull and triage issues/PRs --- .claude/commands/release.md | 59 ++++++++++-- src/scripts/Start-Release.ps1 | 164 ++++++++++++++++++++++++++-------- 2 files changed, 177 insertions(+), 46 deletions(-) diff --git a/.claude/commands/release.md b/.claude/commands/release.md index 04a4eb37f..9c5fe0229 100644 --- a/.claude/commands/release.md +++ b/.claude/commands/release.md @@ -1,6 +1,6 @@ --- description: Walk through the release checklist interactively -allowed-tools: Read, Grep, Bash(pwsh -Command ./src/scripts/Start-Release*), Bash(gh issue view *) +allowed-tools: Read, Grep, Bash(gh issue view *), Bash(pwsh -Command ./src/scripts/*) --- # Release @@ -11,14 +11,55 @@ Assists in the FinOps toolkit release process. ### Phase 1: Setup -Run `pwsh -Command ./src/scripts/Start-Release.ps1` and report the result to the user. +Run the setup script and capture its return object to a temp JSON file: -### Future phases +```bash +pwsh -Command "./src/scripts/Start-Release.ps1 | ConvertTo-Json -Depth 5 | Set-Content /tmp/ftk-release.json" +``` -Additional phases will be added incrementally as we work through the release process: +Then read `/tmp/ftk-release.json` with the Read tool to get the structured result. Report the release issue URL and milestone summary to the user. -- **Phase 2: Monthly updates** — Open data, Bicep CLI -- **Phase 3: Core features** — Branch merges, build, test, version references -- **Phase 4: Finalize** — Changelog, version bump, packaging -- **Phase 5: Publish** — PowerShell, Bicep Registry, AQT, docs, GitHub release -- **Phase 6: Post-release** — Dev version bump, branch cleanup +### Phase 2: Core features + +After setup completes, do the following in parallel: + +1. **Launch background build/test agents.** Use the Task tool to spin up subagents that run these commands. Each agent should capture the full output but ONLY return text to you if there was a failure. On success, return a single short confirmation line (e.g., "Build succeeded" or "Unit tests passed: 342 tests"). + + - **Build agent:** `pwsh -Command ./src/scripts/Build-Toolkit.ps1` + - **Unit test agent:** `pwsh -Command "./src/scripts/Test-PowerShell.ps1 -Unit"` + - **Lint test agent:** `pwsh -Command "./src/scripts/Test-PowerShell.ps1 -Lint"` + - **Integration test agent:** `pwsh -Command "./src/scripts/Test-PowerShell.ps1 -Integration"` + +2. **Triage milestone items** (while build/tests run). Use the milestone data from the JSON file to triage issues and PRs using the layered approach below. + +#### Milestone triage + +Group the milestone issues and PRs into logical categories based on their titles and labels. Then present them to the user in a layered AskUserQuestion flow: + +**Round 1: Bulk triage by group.** + +For each logical group of issues/PRs (e.g., "DevOps improvements", "Documentation updates", "Feature X"): + +- Write a concise question (max 100 words) summarizing what the group is about. Include issue/PR numbers, short titles, and their Summary field from the JSON data to give the user enough context to decide without reading each issue. +- Options: "Keep all in this release (Recommended)", "Push all to next release", "Review individually" +- Do NOT do deep analysis of each issue. Use the title, labels, and summary excerpt from the data. +- Present up to 4 groups per AskUserQuestion call. + +**Round 2: Individual review (only for groups where user chose "Review individually").** + +For each issue/PR in that group: + +- Summarize in one line what the issue is about. +- Options: "Keep in this release", "Push to next release", "Investigate further" +- If "Investigate further" is chosen, read the issue body and comments via `gh issue view {number}`, then provide a deeper analysis and re-present the decision. + +**After triage:** + +Report which items are staying and which are being pushed. Do NOT move milestones automatically — just report the recommendations for the user to act on. + +#### Build/test results + +After triage is complete (or if triage finishes quickly), check on the background agents. Report results: + +- If all passed, report a brief summary. +- If any failed, show the failure output and ask the user how to proceed. diff --git a/src/scripts/Start-Release.ps1 b/src/scripts/Start-Release.ps1 index ddf3a31dc..9f73017c8 100644 --- a/src/scripts/Start-Release.ps1 +++ b/src/scripts/Start-Release.ps1 @@ -7,7 +7,7 @@ .DESCRIPTION The Start-Release command determines the current release state and performs the next steps that can be automated. - Currently, it ensures a release tracking issue exists on GitHub. + Steps include: ensuring a release tracking issue exists, updating the Bicep CLI, and querying milestone data. .PARAMETER DaysUntilRelease Optional. Number of days until the expected release date. Used to determine the release month when running @@ -98,50 +98,140 @@ if ($LASTEXITCODE -ne 0) return } -$issues = $existingIssues | ConvertFrom-Json -if ($issues.Count -gt 0) +$matchingIssues = $existingIssues | ConvertFrom-Json +if ($matchingIssues.Count -gt 0) { - $issue = $issues[0] - Write-Host "Found existing release issue: #$($issue.number) - $($issue.title)" -ForegroundColor Green - Write-Host " $($issue.url)" -ForegroundColor Gray - return @{ - Action = 'existing' - Number = $issue.number - Title = $issue.title - Url = $issue.url + $releaseIssue = $matchingIssues[0] + Write-Host "Found existing release issue: #$($releaseIssue.number) - $($releaseIssue.title)" -ForegroundColor Green + Write-Host " $($releaseIssue.url)" -ForegroundColor Gray +} +else +{ + # Create the issue + Write-Host "Creating release issue..." -ForegroundColor Gray + Write-Host " Title: $title" -ForegroundColor Gray + + # Write body to a temp file to avoid shell escaping issues + $tempFile = [System.IO.Path]::GetTempFileName() + try + { + Set-Content -Path $tempFile -Value $body -NoNewline + $result = gh issue create --title $title --label $labels --body-file $tempFile 2>&1 + if ($LASTEXITCODE -ne 0) + { + Write-Error "Failed to create issue: $result" + return + } + + # gh issue create returns the URL + $issueUrl = $result.Trim() + $issueNumber = [regex]::Match($issueUrl, '/issues/(\d+)').Groups[1].Value + + Write-Host "Created release issue: #$issueNumber" -ForegroundColor Green + Write-Host " $issueUrl" -ForegroundColor Gray + } + finally + { + Remove-Item $tempFile -ErrorAction SilentlyContinue } } -# Create the issue -Write-Host "Creating release issue..." -ForegroundColor Gray -Write-Host " Title: $title" -ForegroundColor Gray +# --- Step 2: Update Bicep CLI --- + +Write-Host "" +Write-Host "Updating Bicep CLI..." -ForegroundColor Cyan +$bicepResult = az bicep upgrade 2>&1 +if ($LASTEXITCODE -ne 0) +{ + Write-Warning "Bicep CLI update failed: $bicepResult" +} +else +{ + Write-Host "Bicep CLI updated." -ForegroundColor Green +} + +# --- Step 3: Query milestone issues and PRs --- + +Write-Host "" +Write-Host "Querying milestone data..." -ForegroundColor Cyan + +# Find the milestone matching this version tag +$milestones = gh api 'repos/{owner}/{repo}/milestones?state=open' 2>&1 | ConvertFrom-Json +$milestone = $milestones | Where-Object { $_.title -eq $versionTag -or $_.title -eq $version } -# Write body to a temp file to avoid shell escaping issues -$tempFile = [System.IO.Path]::GetTempFileName() -try +if (-not $milestone) +{ + Write-Warning "No open milestone found matching '$versionTag' or '$version'. Skipping milestone query." +} +else { - Set-Content -Path $tempFile -Value $body -NoNewline - $result = gh issue create --title $title --label $labels --body-file $tempFile 2>&1 - if ($LASTEXITCODE -ne 0) + $milestoneNumber = $milestone.number + Write-Host "Found milestone: $($milestone.title) (#$milestoneNumber) — $($milestone.open_issues) open issue(s)" -ForegroundColor Green + + # Query all open issues and PRs in this milestone + $milestoneItems = gh api "repos/{owner}/{repo}/issues?milestone=$milestoneNumber&state=open&per_page=100" 2>&1 | ConvertFrom-Json + + $milestoneIssues = @() + $milestonePRs = @() + foreach ($item in $milestoneItems) { - Write-Error "Failed to create issue: $result" - return + # Extract a short summary from the body for triage context + $summary = '' + if ($item.body) + { + # Strip HTML comments, then grab first non-empty, non-header, non-checkbox line + $cleanBody = $item.body -replace '(?s)', '' + $lines = $cleanBody -split '\r?\n' | ForEach-Object { $_.Trim() } | Where-Object { + $_ -ne '' -and + $_ -notmatch '^#{1,3}\s' -and + $_ -notmatch '^[-*]\s*\[' -and + $_ -notmatch '^\*\*Changes' -and + $_ -notmatch '^[⚠ī¸|>|---]' + } + if ($lines.Count -gt 0) + { + $summary = ($lines | Select-Object -First 2) -join ' ' + if ($summary.Length -gt 200) { $summary = $summary.Substring(0, 200) + '...' } + } + } + + $entry = @{ + Number = $item.number + Title = $item.title + Summary = $summary + Labels = @($item.labels | ForEach-Object { $_.name }) + Assignees = @($item.assignees | ForEach-Object { $_.login }) + Url = $item.html_url + } + + if ($item.pull_request) + { + $milestonePRs += $entry + } + else + { + $milestoneIssues += $entry + } } - # gh issue create returns the URL - $issueUrl = $result.Trim() - $issueNumber = [regex]::Match($issueUrl, '/issues/(\d+)').Groups[1].Value - - Write-Host "Created release issue: #$issueNumber" -ForegroundColor Green - Write-Host " $issueUrl" -ForegroundColor Gray - return @{ - Action = 'created' - Number = [int]$issueNumber - Title = $title - Url = $issueUrl - } + Write-Host " Issues: $($milestoneIssues.Count) PRs: $($milestonePRs.Count)" -ForegroundColor Gray } -finally -{ - Remove-Item $tempFile -ErrorAction SilentlyContinue + +# --- Return result object --- + +return @{ + Version = $version + VersionTag = $versionTag + Month = $Month + Year = $Year + ReleaseIssue = if ($releaseIssue) { @{ Number = $releaseIssue.number; Title = $releaseIssue.title; Url = $releaseIssue.url } } else { $null } + Milestone = if ($milestone) { + @{ + Number = $milestoneNumber + Title = $milestone.title + OpenIssues = $milestone.open_issues + Issues = $milestoneIssues + PRs = $milestonePRs + } + } else { $null } } From 272000fca4598c76414350879f83db0816baa839 Mon Sep 17 00:00:00 2001 From: Michael Flanakin Date: Sat, 4 Apr 2026 17:31:09 -0700 Subject: [PATCH 04/30] Create future milestones --- src/scripts/Start-Release.ps1 | 31 +++++++++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-) diff --git a/src/scripts/Start-Release.ps1 b/src/scripts/Start-Release.ps1 index 9f73017c8..ee2391d21 100644 --- a/src/scripts/Start-Release.ps1 +++ b/src/scripts/Start-Release.ps1 @@ -163,7 +163,33 @@ if (-not $milestone) { Write-Warning "No open milestone found matching '$versionTag' or '$version'. Skipping milestone query." } -else + +# Ensure at least 2 future milestones exist (for pushing items to next or next+1 release) +$currentMajor = [int]($version -replace '\..*$', '') +$futureTitles = @("v$($currentMajor + 1)", "v$($currentMajor + 2)") +foreach ($futureTitle in $futureTitles) +{ + $existing = $milestones | Where-Object { $_.title -eq $futureTitle } + if ($existing) + { + Write-Host " $futureTitle already exists" -ForegroundColor Gray + } + else + { + Write-Host " Creating milestone: $futureTitle" -ForegroundColor Gray + gh api 'repos/{owner}/{repo}/milestones' --method POST --field title=$futureTitle 2>&1 | Out-Null + if ($LASTEXITCODE -eq 0) + { + Write-Host " Created milestone: $futureTitle" -ForegroundColor Green + } + else + { + Write-Warning "Failed to create milestone: $futureTitle" + } + } +} + +if ($milestone) { $milestoneNumber = $milestone.number Write-Host "Found milestone: $($milestone.title) (#$milestoneNumber) — $($milestone.open_issues) open issue(s)" -ForegroundColor Green @@ -186,7 +212,8 @@ else $_ -notmatch '^#{1,3}\s' -and $_ -notmatch '^[-*]\s*\[' -and $_ -notmatch '^\*\*Changes' -and - $_ -notmatch '^[⚠ī¸|>|---]' + $_ -notmatch '^[>⚠]' -and + $_ -notmatch '^---' } if ($lines.Count -gt 0) { From 2e665cd8a476f93f6f062eb72817001db1cf2ede Mon Sep 17 00:00:00 2001 From: Michael Flanakin Date: Mon, 6 Apr 2026 04:07:39 -0700 Subject: [PATCH 05/30] Triage open issues/PRs --- .claude/commands/release.md | 28 ++++++++++++---------------- .claude/settings.json | 3 +-- 2 files changed, 13 insertions(+), 18 deletions(-) diff --git a/.claude/commands/release.md b/.claude/commands/release.md index 9c5fe0229..7e6a25dac 100644 --- a/.claude/commands/release.md +++ b/.claude/commands/release.md @@ -34,28 +34,24 @@ After setup completes, do the following in parallel: #### Milestone triage -Group the milestone issues and PRs into logical categories based on their titles and labels. Then present them to the user in a layered AskUserQuestion flow: +Analyze all milestone items and recommend keep vs push for each. The release is expected within ~7 days, so push anything complex unless it's a bug fix or a feature explicitly targeting this release. Use labels, titles, and summaries to judge — don't over-explain your reasoning in questions. -**Round 1: Bulk triage by group.** +Group items by topic, then present via AskUserQuestion. Use the version tag from the JSON (e.g., "v14") and the next milestone title (e.g., "v15") in option labels. -For each logical group of issues/PRs (e.g., "DevOps improvements", "Documentation updates", "Feature X"): +**Round 1: Bulk triage by group** (up to 4 groups per AskUserQuestion call). -- Write a concise question (max 100 words) summarizing what the group is about. Include issue/PR numbers, short titles, and their Summary field from the JSON data to give the user enough context to decide without reading each issue. -- Options: "Keep all in this release (Recommended)", "Push all to next release", "Review individually" -- Do NOT do deep analysis of each issue. Use the title, labels, and summary excerpt from the data. -- Present up to 4 groups per AskUserQuestion call. +- **Header:** 1-3 word group name +- **Question:** Plain text, no formatting. "{count} items: #{number} {short title}, #{number} {short title}, ..." Titles 3-5 words each. +- **Options:** Recommendation first with "(Recommended)". Always include: "Keep all", "Push all", "Review individually". -**Round 2: Individual review (only for groups where user chose "Review individually").** +**Round 2: Individual review** (only for groups where user chose "Review individually"). -For each issue/PR in that group: +- **Header:** "#{number}: {1-3 word title}" +- **Question:** Plain text summary from JSON data, up to 100 words. State your recommendation and why. +- **Options:** "Keep in {version} (Recommended)" or "Push to {next version} (Recommended)" (whichever you recommend first), the other option, and "Investigate further". +- If "Investigate further" is chosen, fetch details via `gh issue view {number}`, provide deeper analysis, and re-present. -- Summarize in one line what the issue is about. -- Options: "Keep in this release", "Push to next release", "Investigate further" -- If "Investigate further" is chosen, read the issue body and comments via `gh issue view {number}`, then provide a deeper analysis and re-present the decision. - -**After triage:** - -Report which items are staying and which are being pushed. Do NOT move milestones automatically — just report the recommendations for the user to act on. +**After triage:** Report which items are staying and which are being pushed. Do NOT move milestones — just report for the user to act on. #### Build/test results diff --git a/.claude/settings.json b/.claude/settings.json index 34eacd484..057d883a9 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -1,6 +1,5 @@ { "permissions": { - "allow": [ - ] + "allow": ["Bash(pwsh -Command \"./src/scripts/:*)", "Bash(pwsh -Command './src/scripts/:*)"] } } From b7aa588ef5f552b7f132390faa0c0c59aae0978f Mon Sep 17 00:00:00 2001 From: Michael Flanakin Date: Mon, 6 Apr 2026 10:42:43 -0700 Subject: [PATCH 06/30] Add issue triage --- .claude/commands/release.md | 51 ++++++++------- .claude/settings.json | 7 ++- src/scripts/Start-Release.ps1 | 115 +++++++++++++++++++++++----------- 3 files changed, 111 insertions(+), 62 deletions(-) diff --git a/.claude/commands/release.md b/.claude/commands/release.md index 7e6a25dac..2f432f72b 100644 --- a/.claude/commands/release.md +++ b/.claude/commands/release.md @@ -1,38 +1,32 @@ --- description: Walk through the release checklist interactively -allowed-tools: Read, Grep, Bash(gh issue view *), Bash(pwsh -Command ./src/scripts/*) +allowed-tools: Read, Grep, Bash(gh issue view *), Bash(pwsh -Command ./src/scripts/*), Bash(pwsh -Command "./src/scripts/*) --- # Release -Assists in the FinOps toolkit release process. - -## Process - -### Phase 1: Setup - -Run the setup script and capture its return object to a temp JSON file: +## Phase 1: Setup ```bash -pwsh -Command "./src/scripts/Start-Release.ps1 | ConvertTo-Json -Depth 5 | Set-Content /tmp/ftk-release.json" +pwsh -Command "./src/scripts/Start-Release.ps1 -OutputFile (Join-Path ([System.IO.Path]::GetTempPath()) 'ftk-release.json')" ``` -Then read `/tmp/ftk-release.json` with the Read tool to get the structured result. Report the release issue URL and milestone summary to the user. +Read the JSON file from the temp path printed in the output. Report the release issue URL and milestone summary to the user. -### Phase 2: Core features +## Phase 2: Core features -After setup completes, do the following in parallel: +Do the following in parallel: -1. **Launch background build/test agents.** Use the Task tool to spin up subagents that run these commands. Each agent should capture the full output but ONLY return text to you if there was a failure. On success, return a single short confirmation line (e.g., "Build succeeded" or "Unit tests passed: 342 tests"). +1. **Launch background build/test agents.** Use the Task tool for each. Only return output on failure; on success return a one-line confirmation. - - **Build agent:** `pwsh -Command ./src/scripts/Build-Toolkit.ps1` - - **Unit test agent:** `pwsh -Command "./src/scripts/Test-PowerShell.ps1 -Unit"` - - **Lint test agent:** `pwsh -Command "./src/scripts/Test-PowerShell.ps1 -Lint"` - - **Integration test agent:** `pwsh -Command "./src/scripts/Test-PowerShell.ps1 -Integration"` + - `pwsh -Command ./src/scripts/Build-Toolkit.ps1` + - `pwsh -Command "./src/scripts/Test-PowerShell.ps1 -Unit"` + - `pwsh -Command "./src/scripts/Test-PowerShell.ps1 -Lint"` + - `pwsh -Command "./src/scripts/Test-PowerShell.ps1 -Integration"` -2. **Triage milestone items** (while build/tests run). Use the milestone data from the JSON file to triage issues and PRs using the layered approach below. +2. **Triage milestone items** using the layered approach below. -#### Milestone triage +### Milestone triage Analyze all milestone items and recommend keep vs push for each. The release is expected within ~7 days, so push anything complex unless it's a bug fix or a feature explicitly targeting this release. Use labels, titles, and summaries to judge — don't over-explain your reasoning in questions. @@ -53,9 +47,20 @@ Group items by topic, then present via AskUserQuestion. Use the version tag from **After triage:** Report which items are staying and which are being pushed. Do NOT move milestones — just report for the user to act on. -#### Build/test results +### Untriaged issues + +If `NeedsReview` in the JSON contains any issues, present them for quick triage using the same Round 1/Round 2 pattern. These are issues with the "Needs: Review 👀" label that haven't been triaged yet. + +### Build/test results + +After triage, check on the background agents. If all passed, report a brief summary. For each failure, show the error output and present via AskUserQuestion with options: "Investigate and fix", "Skip for now". + +### Next actions + +After all triage and build/test results are reported, analyze the kept milestone items and suggest concrete next actions: -After triage is complete (or if triage finishes quickly), check on the background agents. Report results: +- PRs from others awaiting the user's review +- The user's PRs that have reviewer feedback to address +- Issues with no open PRs and no one actively working them -- If all passed, report a brief summary. -- If any failed, show the failure output and ask the user how to proceed. +Present as a prioritized list — no AUQ needed, just a summary the user can act on. diff --git a/.claude/settings.json b/.claude/settings.json index 057d883a9..f7d549f67 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -1,5 +1,10 @@ { "permissions": { - "allow": ["Bash(pwsh -Command \"./src/scripts/:*)", "Bash(pwsh -Command './src/scripts/:*)"] + "allow": [ + "Bash(pwsh -Command \"./src/scripts/Start-Release.ps1 -OutputFile (Join-Path ([System.IO.Path]::GetTempPath()) 'ftk-release.json')\")", + "Bash(pwsh -Command \"./src/scripts/Test-PowerShell.ps1 :*)", + "Bash(pwsh -Command \"./src/scripts/:*)", + "Bash(pwsh -Command './src/scripts/:*)" + ] } } diff --git a/src/scripts/Start-Release.ps1 b/src/scripts/Start-Release.ps1 index ee2391d21..aced594a7 100644 --- a/src/scripts/Start-Release.ps1 +++ b/src/scripts/Start-Release.ps1 @@ -13,6 +13,9 @@ Optional. Number of days until the expected release date. Used to determine the release month when running near the end of a month. Default = 7. + .PARAMETER OutputFile + Optional. Path to write JSON output to. If not specified, results are only returned as a PowerShell object. + .EXAMPLE ./Start-Release @@ -26,12 +29,48 @@ #> param( [int] - $DaysUntilRelease = 7 + $DaysUntilRelease = 7, + + [string] + $OutputFile ) $ErrorActionPreference = 'Stop' $repoRoot = (Resolve-Path (Join-Path $PSScriptRoot '../..')).Path +function ConvertTo-ItemEntry($item) +{ + # Extract a short summary from the body for triage context + $summary = '' + if ($item.body) + { + # Strip HTML comments, then grab first non-empty, non-header, non-checkbox line + $cleanBody = $item.body -replace '(?s)', '' + $lines = $cleanBody -split '\r?\n' | ForEach-Object { $_.Trim() } | Where-Object { + $_ -ne '' -and + $_ -notmatch '^#{1,3}\s' -and + $_ -notmatch '^[-*]\s*\[' -and + $_ -notmatch '^\*\*Changes' -and + $_ -notmatch '^[>⚠]' -and + $_ -notmatch '^---' + } + if ($lines.Count -gt 0) + { + $summary = ($lines | Select-Object -First 2) -join ' ' + if ($summary.Length -gt 200) { $summary = $summary.Substring(0, 200) + '...' } + } + } + + return @{ + Number = $item.number + Title = $item.title + Summary = $summary + Labels = @($item.labels | ForEach-Object { $_.name }) + Assignees = @($item.assignees | ForEach-Object { $_.login }) + Url = $item.html_url + } +} + # Get version from package.json $version = & (Join-Path $PSScriptRoot 'Get-Version.ps1') $versionTag = & (Join-Path $PSScriptRoot 'Get-Version.ps1') -AsTag @@ -201,36 +240,7 @@ if ($milestone) $milestonePRs = @() foreach ($item in $milestoneItems) { - # Extract a short summary from the body for triage context - $summary = '' - if ($item.body) - { - # Strip HTML comments, then grab first non-empty, non-header, non-checkbox line - $cleanBody = $item.body -replace '(?s)', '' - $lines = $cleanBody -split '\r?\n' | ForEach-Object { $_.Trim() } | Where-Object { - $_ -ne '' -and - $_ -notmatch '^#{1,3}\s' -and - $_ -notmatch '^[-*]\s*\[' -and - $_ -notmatch '^\*\*Changes' -and - $_ -notmatch '^[>⚠]' -and - $_ -notmatch '^---' - } - if ($lines.Count -gt 0) - { - $summary = ($lines | Select-Object -First 2) -join ' ' - if ($summary.Length -gt 200) { $summary = $summary.Substring(0, 200) + '...' } - } - } - - $entry = @{ - Number = $item.number - Title = $item.title - Summary = $summary - Labels = @($item.labels | ForEach-Object { $_.name }) - Assignees = @($item.assignees | ForEach-Object { $_.login }) - Url = $item.html_url - } - + $entry = ConvertTo-ItemEntry $item if ($item.pull_request) { $milestonePRs += $entry @@ -244,15 +254,34 @@ if ($milestone) Write-Host " Issues: $($milestoneIssues.Count) PRs: $($milestonePRs.Count)" -ForegroundColor Gray } +# --- Step 4: Query untriaged issues --- + +Write-Host "" +Write-Host "Querying untriaged issues..." -ForegroundColor Cyan + +$untriagedLabel = [System.Uri]::EscapeDataString("Needs: Review 👀") +$untriagedItems = gh api "repos/{owner}/{repo}/issues?state=open&labels=$untriagedLabel&per_page=100" 2>&1 | ConvertFrom-Json +$untriagedIssues = @() +foreach ($item in $untriagedItems) +{ + if (-not $item.pull_request) + { + $untriagedIssues += ConvertTo-ItemEntry $item + } +} + +Write-Host " Untriaged issues: $($untriagedIssues.Count)" -ForegroundColor Gray + # --- Return result object --- -return @{ - Version = $version - VersionTag = $versionTag - Month = $Month - Year = $Year - ReleaseIssue = if ($releaseIssue) { @{ Number = $releaseIssue.number; Title = $releaseIssue.title; Url = $releaseIssue.url } } else { $null } - Milestone = if ($milestone) { +$result = @{ + Version = $version + VersionTag = $versionTag + Month = $Month + Year = $Year + ReleaseIssue = if ($releaseIssue) { @{ Number = $releaseIssue.number; Title = $releaseIssue.title; Url = $releaseIssue.url } } else { $null } + NeedsReview = $untriagedIssues + Milestone = if ($milestone) { @{ Number = $milestoneNumber Title = $milestone.title @@ -262,3 +291,13 @@ return @{ } } else { $null } } + +# Write JSON output if requested +if ($OutputFile) +{ + $result | ConvertTo-Json -Depth 5 | Set-Content $OutputFile + Write-Host "" + Write-Host "Results written to $OutputFile" -ForegroundColor Green +} + +return $result From 32e0597a116832006cd9f2ddeeaac630cd0b9adb Mon Sep 17 00:00:00 2001 From: Michael Flanakin Date: Mon, 6 Apr 2026 18:32:09 -0700 Subject: [PATCH 07/30] Add changelog review --- .claude/commands/release.md | 19 ++++++++++++++- .claude/settings.json | 1 + docs-wiki/Coding-guidelines.md | 43 ++++++++++++++++++++++++++++++++++ 3 files changed, 62 insertions(+), 1 deletion(-) diff --git a/.claude/commands/release.md b/.claude/commands/release.md index 2f432f72b..ba6ea3c7f 100644 --- a/.claude/commands/release.md +++ b/.claude/commands/release.md @@ -17,7 +17,7 @@ Read the JSON file from the temp path printed in the output. Report the release Do the following in parallel: -1. **Launch background build/test agents.** Use the Task tool for each. Only return output on failure; on success return a one-line confirmation. +1. **Launch background build/test commands.** Use Bash with `run_in_background` for each: - `pwsh -Command ./src/scripts/Build-Toolkit.ps1` - `pwsh -Command "./src/scripts/Test-PowerShell.ps1 -Unit"` @@ -55,6 +55,23 @@ If `NeedsReview` in the JSON contains any issues, present them for quick triage After triage, check on the background agents. If all passed, report a brief summary. For each failure, show the error output and present via AskUserQuestion with options: "Investigate and fix", "Skip for now". +### Changelog review + +Read @docs-mslearn/toolkit/changelog.md and review the current version section against the changelog standard in @docs-wiki/Coding-guidelines.md. Apply fixes directly — no AUQ needed. Specific checks: + +- Only one version section for this release (no duplicates). +- Unreleased section should be empty (we release from dev). Flag any content there. +- Tool sections match the order from the previous release. Flag any new tools or reordering. +- Categories in order: Added, Changed, Fixed, Deprecated, Removed. Empty categories omitted. +- Every entry starts with a past-tense verb, ends with a period, follows content guidelines. +- Entries are concise. Trim verbose or implementation-detail-heavy text. +- Issue links present where applicable, formatted as `([#{number}](url))`. +- Breaking changes prefixed with `**Breaking:**` and listed first in their category. +- No filler entries ("various bug fixes", "minor improvements"). +- Entries placed in the correct category (e.g., new things under Added, not Changed). + +After applying fixes, show a summary of what was changed so the user can review. + ### Next actions After all triage and build/test results are reported, analyze the kept milestone items and suggest concrete next actions: diff --git a/.claude/settings.json b/.claude/settings.json index f7d549f67..88ae1a159 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -2,6 +2,7 @@ "permissions": { "allow": [ "Bash(pwsh -Command \"./src/scripts/Start-Release.ps1 -OutputFile (Join-Path ([System.IO.Path]::GetTempPath()) 'ftk-release.json')\")", + "Bash(pwsh -Command \"./src/scripts/Test-PowerShell.ps1 -Unit\")", "Bash(pwsh -Command \"./src/scripts/Test-PowerShell.ps1 :*)", "Bash(pwsh -Command \"./src/scripts/:*)", "Bash(pwsh -Command './src/scripts/:*)" diff --git a/docs-wiki/Coding-guidelines.md b/docs-wiki/Coding-guidelines.md index a3a4e6f8d..b66a7f49a 100644 --- a/docs-wiki/Coding-guidelines.md +++ b/docs-wiki/Coding-guidelines.md @@ -4,6 +4,7 @@ On this page: - [ℹī¸ General guidelines](#ℹī¸-general-guidelines) - [🔤 Content (strings and microcopy)](#-content-strings-and-microcopy) +- [📋 Changelog](#-changelog) --- @@ -43,3 +44,45 @@ We adhere to the [Microsoft style guide](https://docs.microsoft.com/style-guide/ [Learn more](https://docs.microsoft.com/style-guide/welcome)
+ +## 📋 Changelog + +The [changelog](../docs-mslearn/toolkit/changelog.md) documents user-facing changes for each release. It follows [Keep a Changelog](https://keepachangelog.com) conventions adapted for this project's multi-tool structure. + +### Structure + +```markdown +## v{version} + +_Released {Month} {Year}_ + +### [{Tool name}]({doc-link}) v{version} + +- **Added** + - Entry text ([#{issue}](url)). +- **Changed** + - Entry text ([#{issue}](url)). +- **Fixed** + - Entry text ([#{issue}](url)). +- **Deprecated** + - Entry text ([#{issue}](url)). +- **Removed** + - Entry text ([#{issue}](url)). +``` + +### Rules + +- **One version section.** All changes for the upcoming release go in a single version section. Do not create duplicate sections. +- **Unreleased section.** Only for changes merged to feature branches that are not yet in `dev`. Once in `dev`, move to the version section. +- **Category order.** Added, Changed, Fixed, Deprecated, Removed. Omit empty categories. +- **Tool sections.** Group by tool using H3 headings with a link to the tool's doc page and the version number (e.g., `### [FinOps hubs](...) v14`). Match the tool order from previous releases. +- **Entry format.** Start with a past-tense verb (Added, Changed, Fixed, Removed, Updated). End with a period. Follow the [content guidelines](#-content-strings-and-microcopy). +- **Issue links.** Link to the GitHub issue when one exists: `([#{number}]({url}))`. Omit when no issue applies (e.g., minor doc fixes). +- **One line per change.** Each entry should be a single concise sentence. Sub-bullets can provide context but keep the overall entry brief. +- **No filler.** Omit entries like "Various bug fixes and improvements" or "Minor code cleanup." Every entry should describe a specific, user-facing change. +- **No implementation details.** Write for users, not developers. "Fixed dashboard freezes during large report generation" not "Fixed async loop timing in render pipeline." +- **Breaking changes.** Prefix with `**Breaking:**` and list first within the category. +- **Within-category ordering.** Breaking changes first, then by importance to users. +- **Brevity over verbosity.** Aim for the minimum words that convey the change and its impact. A changelog is a summary, not documentation. + +
From 0be980f8673fdb47c2017a8648a9a709cb4f7bae Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?H=C3=A9lder=20Pinto?= Date: Tue, 7 Apr 2026 15:52:24 +0100 Subject: [PATCH 08/30] Initial commit for migration to DCR-based ingestion --- .../Deploy-AzureOptimizationEngine.ps1 | 23 +- .../Setup-LogAnalyticsTablesAndDCRs.ps1 | 1152 +++++++++++++++++ .../azuredeploy-nested.bicep | 24 +- .../model/loganalyticsingestcontrol-table.sql | 11 +- ...t-OptimizationCSVExportsToLogAnalytics.ps1 | 127 +- .../Ingest-RecommendationsToLogAnalytics.ps1 | 107 +- .../Ingest-SuppressionsToLogAnalytics.ps1 | 134 +- src/optimization-engine/upgrade-manifest.json | 8 +- 8 files changed, 1362 insertions(+), 224 deletions(-) create mode 100644 src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 diff --git a/src/optimization-engine/Deploy-AzureOptimizationEngine.ps1 b/src/optimization-engine/Deploy-AzureOptimizationEngine.ps1 index cdedc639c..244097c5e 100644 --- a/src/optimization-engine/Deploy-AzureOptimizationEngine.ps1 +++ b/src/optimization-engine/Deploy-AzureOptimizationEngine.ps1 @@ -547,7 +547,8 @@ if ("N", "n" -contains $workspaceReuse) $laWorkspaceResourceGroup = $resourceGroupName $la = Get-AzOperationalInsightsWorkspace -ResourceGroupName $resourceGroupName -Name $laWorkspaceName -ErrorAction SilentlyContinue - if ($null -ne $la) { + if ($null -ne $la) + { Write-Host "(The Log Analytics Workspace was already deployed)" -ForegroundColor Green } } @@ -1406,6 +1407,26 @@ if ("Y", "y" -contains $continueInput) } #endregion + #region DCR-based ingestion setup + Write-Host "Setting up Log Analytics custom tables and Data Collection Rules for DCR-based ingestion..." -ForegroundColor Green + try + { + .\Setup-LogAnalyticsTablesAndDCRs.ps1 ` + -ResourceGroupName $resourceGroupName ` + -AutomationAccountName $automationAccountName ` + -WorkspaceName $laWorkspaceName ` + -WorkspaceResourceGroupName $laWorkspaceResourceGroup ` + -SqlServerName $sqlServerName ` + -SqlDatabaseName $sqlDatabaseName ` + -CloudEnvironment $cloudEnvironment + } + catch + { + Write-Host "Could not complete the Log Analytics tables and DCR setup: $($_.Exception.Message)" -ForegroundColor Yellow + Write-Host "You can re-run Setup-LogAnalyticsTablesAndDCRs.ps1 manually after resolving the issue." -ForegroundColor Yellow + } + #endregion + if (!$silentDeploy) { #region Grant Microsoft Entra ID role to AOE principal diff --git a/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 b/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 new file mode 100644 index 000000000..9c0597653 --- /dev/null +++ b/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 @@ -0,0 +1,1152 @@ +īģŋ<# +.SYNOPSIS +Creates or updates the custom Log Analytics tables and Data Collection Rules (DCRs) required +for DCR-based ingestion in the Azure Optimization Engine. + +.DESCRIPTION +This script provisions all custom Log Analytics tables with explicit schemas and one DCR per table, +replacing the legacy Log Analytics Data Collector API (SharedKey) ingestion. It must be run once +on both new and existing AOE installations. On existing installations, it also migrates the SQL +control table to store the DCR immutable IDs. + +The script: +- Creates a Data Collection Endpoint (DCE) if one is not yet referenced in the DCEIngestionEndpoint + automation variable (the DCE itself is expected to already be deployed via the Bicep template). +- Creates or updates each custom Log Analytics table schema. +- Creates or updates one DCR per table. +- Assigns the Monitoring Metrics Publisher role on every DCR to the Automation account managed identity. +- Writes the DCR immutable ID for each table into the SQL LogAnalyticsIngestControl table. +- Creates/updates the AzureOptimization_DCEIngestionEndpoint automation variable. +- Removes the AzureOptimization_LogAnalyticsWorkspaceKey automation variable (no longer needed). + +.PARAMETER ResourceGroupName +The resource group where the AOE Automation account and (optionally) Log Analytics workspace reside. + +.PARAMETER AutomationAccountName +The name of the AOE Automation account. + +.PARAMETER WorkspaceName +The name of the Log Analytics workspace where custom tables will be created. + +.PARAMETER WorkspaceResourceGroupName +The resource group of the Log Analytics workspace. Defaults to ResourceGroupName if not specified. + +.PARAMETER WorkspaceSubscriptionId +The subscription ID of the Log Analytics workspace. Defaults to the current subscription if not specified. + +.PARAMETER SqlServerName +The SQL Server hostname (FQDN or short name). If short name, '.database.windows.net' is appended. + +.PARAMETER SqlDatabaseName +The name of the AOE SQL database. + +.PARAMETER CloudEnvironment +The Azure cloud environment. Default is AzureCloud. Supported: AzureCloud, AzureChinaCloud, AzureUSGovernment. + +.PARAMETER AzureEnvironment +Alias for CloudEnvironment, for compatibility with the main deploy script. + +.EXAMPLE +.\Setup-LogAnalyticsTablesAndDCRs.ps1 ` + -ResourceGroupName "rg-aoe" ` + -AutomationAccountName "aa-aoe" ` + -WorkspaceName "la-aoe" ` + -SqlServerName "sql-aoe" ` + -SqlDatabaseName "db-aoe" + +.LINK +https://aka.ms/AzureOptimizationEngine/deployment +#> +param ( + [Parameter(Mandatory = $true)] + [string] $ResourceGroupName, + + [Parameter(Mandatory = $true)] + [string] $AutomationAccountName, + + [Parameter(Mandatory = $true)] + [string] $WorkspaceName, + + [Parameter(Mandatory = $false)] + [string] $WorkspaceResourceGroupName, + + [Parameter(Mandatory = $false)] + [string] $WorkspaceSubscriptionId, + + [Parameter(Mandatory = $true)] + [string] $SqlServerName, + + [Parameter(Mandatory = $true)] + [string] $SqlDatabaseName, + + [Parameter(Mandatory = $false)] + [string] $CloudEnvironment = "AzureCloud", + + [Parameter(Mandatory = $false)] + [string] $AzureEnvironment +) + +$ErrorActionPreference = "Stop" + +# AzureEnvironment is an alias for CloudEnvironment +if (-not [string]::IsNullOrEmpty($AzureEnvironment)) +{ + $CloudEnvironment = $AzureEnvironment +} + +if ([string]::IsNullOrEmpty($WorkspaceResourceGroupName)) +{ + $WorkspaceResourceGroupName = $ResourceGroupName +} + +#region Resolve Monitor audience URL per cloud +switch ($CloudEnvironment) +{ + "AzureChinaCloud" { $monitorAudience = "https://monitor.azure.cn/" } + "AzureUSGovernment" { $monitorAudience = "https://monitor.azure.us/" } + default { $monitorAudience = "https://monitor.azure.com/" } +} + +switch ($CloudEnvironment) +{ + "AzureChinaCloud" { $armEndpoint = "https://management.chinacloudapi.cn" } + "AzureUSGovernment" { $armEndpoint = "https://management.usgovcloudapi.net" } + default { $armEndpoint = "https://management.azure.com" } +} +#endregion + +#region Resolve SQL connection string +if (-not ($SqlServerName -like "*.*")) +{ + switch ($CloudEnvironment) + { + "AzureChinaCloud" { $SqlServerName = "$SqlServerName.database.chinacloudapi.cn" } + "AzureUSGovernment" { $SqlServerName = "$SqlServerName.database.usgovcloudapis.net" } + default { $SqlServerName = "$SqlServerName.database.windows.net" } + } +} +#endregion + +Write-Host "Setting up Log Analytics custom tables and DCRs for the Azure Optimization Engine..." -ForegroundColor Green + +#region Resolve workspace and current subscription context +$context = Get-AzContext +if ([string]::IsNullOrEmpty($WorkspaceSubscriptionId)) +{ + $WorkspaceSubscriptionId = $context.Subscription.Id +} +$currentSubscriptionId = $context.Subscription.Id + +$workspaceResourceId = "/subscriptions/$WorkspaceSubscriptionId/resourceGroups/$WorkspaceResourceGroupName/providers/Microsoft.OperationalInsights/workspaces/$WorkspaceName" +$workspaceLocation = (Get-AzOperationalInsightsWorkspace -ResourceGroupName $WorkspaceResourceGroupName -Name $WorkspaceName).Location +#endregion + +#region Get DCE endpoint and resource ID from automation variable +$dceVar = Get-AzAutomationVariable -ResourceGroupName $ResourceGroupName -AutomationAccountName $AutomationAccountName ` + -Name "AzureOptimization_DCEIngestionEndpoint" -ErrorAction SilentlyContinue +if ($null -eq $dceVar -or [string]::IsNullOrEmpty($dceVar.Value)) +{ + throw "AzureOptimization_DCEIngestionEndpoint automation variable not found. Ensure the Bicep template has been deployed first." +} +$dceLogsIngestionEndpoint = $dceVar.Value +Write-Host "DCE Logs Ingestion endpoint: $dceLogsIngestionEndpoint" -ForegroundColor Cyan + +# Discover the DCE resource ID by searching for DCEs in the resource group whose logsIngestion endpoint matches +$dceListUri = "$armEndpoint/subscriptions/$currentSubscriptionId/resourceGroups/$ResourceGroupName/providers/Microsoft.Insights/dataCollectionEndpoints?api-version=$dcrApiVersion" +$dceListResponse = Invoke-RestMethod -Method GET -Uri $dceListUri -Headers $armHeaders +$matchedDce = $dceListResponse.value | Where-Object { $_.properties.logsIngestion.endpoint -eq $dceLogsIngestionEndpoint } +if ($null -eq $matchedDce) +{ + throw "Could not find a Data Collection Endpoint in resource group $ResourceGroupName with logsIngestion endpoint '$dceLogsIngestionEndpoint'." +} +$dceResourceId = $matchedDce.id +Write-Host "DCE resource ID: $dceResourceId" -ForegroundColor Cyan +#endregion + +#region Get automation account managed identity principal ID +$automationAccount = Get-AzAutomationAccount -ResourceGroupName $ResourceGroupName -Name $AutomationAccountName +$automationPrincipalId = $automationAccount.Identity.PrincipalId +if ([string]::IsNullOrEmpty($automationPrincipalId)) +{ + throw "Could not retrieve the managed identity principal ID for the Automation account $AutomationAccountName." +} +Write-Host "Automation account managed identity: $automationPrincipalId" -ForegroundColor Cyan +#endregion + +#region Monitoring Metrics Publisher role definition ID +# Role definition: Monitoring Metrics Publisher (3913510d-42f4-4e42-8a64-420c390055eb) +$monitoringMetricsPublisherRoleId = "3913510d-42f4-4e42-8a64-420c390055eb" +#endregion + +#region Table schema definitions +# Each entry defines the columns that the ingest runbooks write. +# Column type mapping from the legacy _suffix convention: +# _s -> string +# _g -> string (GUIDs stored as plain strings in the typed column name) +# _t -> datetime +# _d -> real +# _b -> boolean +# TimeGenerated is mandatory for all custom tables. + +$tableSchemas = @{ + + "VMsV1" = @( + @{ name = "TimeGenerated"; type = "datetime" } + @{ name = "Timestamp_s"; type = "string" } + @{ name = "Cloud_s"; type = "string" } + @{ name = "TenantGuid_g"; type = "string" } + @{ name = "SubscriptionGuid_g"; type = "string" } + @{ name = "ResourceGroupName_s"; type = "string" } + @{ name = "Zones_s"; type = "string" } + @{ name = "Location_s"; type = "string" } + @{ name = "VMName_s"; type = "string" } + @{ name = "DeploymentModel_s"; type = "string" } + @{ name = "InstanceId_s"; type = "string" } + @{ name = "VMSize_s"; type = "string" } + @{ name = "CoresCount_d"; type = "real" } + @{ name = "MemoryMB_d"; type = "real" } + @{ name = "OSType_s"; type = "string" } + @{ name = "LicenseType_s"; type = "string" } + @{ name = "DataDiskCount_d"; type = "real" } + @{ name = "NicCount_d"; type = "real" } + @{ name = "UsesManagedDisks_s"; type = "string" } + @{ name = "AvailabilitySetId_s"; type = "string" } + @{ name = "BootDiagnosticsEnabled_s"; type = "string" } + @{ name = "BootDiagnosticsStorageAccount_s"; type = "string" } + @{ name = "StatusDate_s"; type = "string" } + @{ name = "PowerState_s"; type = "string" } + @{ name = "ImagePublisher_s"; type = "string" } + @{ name = "ImageOffer_s"; type = "string" } + @{ name = "ImageSku_s"; type = "string" } + @{ name = "ImageVersion_s"; type = "string" } + @{ name = "ImageExactVersion_s"; type = "string" } + @{ name = "OSName_s"; type = "string" } + @{ name = "OSVersion_s"; type = "string" } + @{ name = "Tags_s"; type = "string" } + ) + + "DisksV1" = @( + @{ name = "TimeGenerated"; type = "datetime" } + @{ name = "Timestamp_s"; type = "string" } + @{ name = "Cloud_s"; type = "string" } + @{ name = "TenantGuid_g"; type = "string" } + @{ name = "SubscriptionGuid_g"; type = "string" } + @{ name = "ResourceGroupName_s"; type = "string" } + @{ name = "DiskName_s"; type = "string" } + @{ name = "InstanceId_s"; type = "string" } + @{ name = "Location_s"; type = "string" } + @{ name = "OwnerVMId_s"; type = "string" } + @{ name = "DeploymentModel_s"; type = "string" } + @{ name = "DiskType_s"; type = "string" } + @{ name = "TimeCreated_s"; type = "string" } + @{ name = "DiskIOPS_d"; type = "real" } + @{ name = "DiskThroughput_d"; type = "real" } + @{ name = "DiskTier_s"; type = "string" } + @{ name = "DiskState_s"; type = "string" } + @{ name = "EncryptionType_s"; type = "string" } + @{ name = "Zones_s"; type = "string" } + @{ name = "Caching_s"; type = "string" } + @{ name = "DiskSizeGB_d"; type = "real" } + @{ name = "SKU_s"; type = "string" } + @{ name = "StatusDate_s"; type = "string" } + @{ name = "Tags_s"; type = "string" } + ) + + "VhdDisksV1" = @( + @{ name = "TimeGenerated"; type = "datetime" } + @{ name = "Timestamp_s"; type = "string" } + @{ name = "Cloud_s"; type = "string" } + @{ name = "TenantGuid_g"; type = "string" } + @{ name = "SubscriptionGuid_g"; type = "string" } + @{ name = "ResourceGroupName_s"; type = "string" } + @{ name = "DiskName_s"; type = "string" } + @{ name = "InstanceId_s"; type = "string" } + @{ name = "OwnerVMId_s"; type = "string" } + @{ name = "Location_s"; type = "string" } + @{ name = "DeploymentModel_s"; type = "string" } + @{ name = "DiskType_s"; type = "string" } + @{ name = "Caching_s"; type = "string" } + @{ name = "DiskSizeGB_d"; type = "real" } + @{ name = "StatusDate_s"; type = "string" } + @{ name = "Tags_s"; type = "string" } + ) + + "AvailSetsV1" = @( + @{ name = "TimeGenerated"; type = "datetime" } + @{ name = "Timestamp_s"; type = "string" } + @{ name = "Cloud_s"; type = "string" } + @{ name = "TenantGuid_g"; type = "string" } + @{ name = "SubscriptionGuid_g"; type = "string" } + @{ name = "ResourceGroupName_s"; type = "string" } + @{ name = "InstanceName_s"; type = "string" } + @{ name = "InstanceId_s"; type = "string" } + @{ name = "SkuName_s"; type = "string" } + @{ name = "Location_s"; type = "string" } + @{ name = "FaultDomains_d"; type = "real" } + @{ name = "UpdateDomains_d"; type = "real" } + @{ name = "VmCount_d"; type = "real" } + @{ name = "StatusDate_s"; type = "string" } + @{ name = "Tags_s"; type = "string" } + @{ name = "Zones_s"; type = "string" } + ) + + "AdvisorV1" = @( + @{ name = "TimeGenerated"; type = "datetime" } + @{ name = "Timestamp_s"; type = "string" } + @{ name = "Cloud_s"; type = "string" } + @{ name = "Category_s"; type = "string" } + @{ name = "Impact_s"; type = "string" } + @{ name = "ImpactedArea_s"; type = "string" } + @{ name = "Description_s"; type = "string" } + @{ name = "RecommendationText_s"; type = "string" } + @{ name = "RecommendationTypeId_g"; type = "string" } + @{ name = "InstanceId_s"; type = "string" } + @{ name = "InstanceName_s"; type = "string" } + @{ name = "Tags_s"; type = "string" } + @{ name = "AdditionalInfo_s"; type = "string" } + @{ name = "ResourceGroup_s"; type = "string" } + @{ name = "SubscriptionGuid_g"; type = "string" } + @{ name = "TenantGuid_g"; type = "string" } + ) + + "RemediationV1" = @( + @{ name = "TimeGenerated"; type = "datetime" } + @{ name = "Timestamp_s"; type = "string" } + @{ name = "Cloud_s"; type = "string" } + @{ name = "TenantGuid_g"; type = "string" } + @{ name = "SubscriptionGuid_g"; type = "string" } + @{ name = "ResourceGroupName_s"; type = "string" } + @{ name = "ResourceId_s"; type = "string" } + @{ name = "RemediationId_g"; type = "string" } + @{ name = "RemediationName_s"; type = "string" } + @{ name = "PolicyAssignmentId_s"; type = "string" } + @{ name = "PolicyDefinitionId_s"; type = "string" } + @{ name = "ProvisioningState_s"; type = "string" } + @{ name = "StatusDate_s"; type = "string" } + ) + + "ConsumptionV1" = @( + @{ name = "TimeGenerated"; type = "datetime" } + @{ name = "Timestamp_s"; type = "string" } + @{ name = "AdditionalInfo_s"; type = "string" } + @{ name = "BillingAccountId_s"; type = "string" } + @{ name = "BillingAccountName_s"; type = "string" } + @{ name = "BillingCurrencyCode_s"; type = "string" } + @{ name = "BillingPeriodEndDate_s"; type = "string" } + @{ name = "BillingPeriodStartDate_s"; type = "string" } + @{ name = "BillingProfileId_s"; type = "string" } + @{ name = "BillingProfileName_s"; type = "string" } + @{ name = "ChargeType_s"; type = "string" } + @{ name = "ConsumedService_s"; type = "string" } + @{ name = "CostAllocationRuleName_s"; type = "string" } + @{ name = "CostCenter_s"; type = "string" } + @{ name = "CostInBillingCurrency_s"; type = "string" } + @{ name = "Date_s"; type = "string" } + @{ name = "EffectivePrice_s"; type = "string" } + @{ name = "Frequency_s"; type = "string" } + @{ name = "InvoiceSectionId_s"; type = "string" } + @{ name = "InvoiceSectionName_s"; type = "string" } + @{ name = "IsAzureCreditEligible_s"; type = "string" } + @{ name = "MeterCategory_s"; type = "string" } + @{ name = "MeterId_g"; type = "string" } + @{ name = "MeterName_s"; type = "string" } + @{ name = "MeterRegion_s"; type = "string" } + @{ name = "MeterSubCategory_s"; type = "string" } + @{ name = "OfferId_s"; type = "string" } + @{ name = "PricingModel_s"; type = "string" } + @{ name = "ProductOrderId_s"; type = "string" } + @{ name = "ProductOrderName_s"; type = "string" } + @{ name = "PublisherName_s"; type = "string" } + @{ name = "PublisherType_s"; type = "string" } + @{ name = "Quantity_s"; type = "string" } + @{ name = "ReservationId_s"; type = "string" } + @{ name = "ReservationName_s"; type = "string" } + @{ name = "ResourceGroup_s"; type = "string" } + @{ name = "ResourceId_s"; type = "string" } + @{ name = "ResourceLocation_s"; type = "string" } + @{ name = "ServiceFamily_s"; type = "string" } + @{ name = "ServiceInfo1_s"; type = "string" } + @{ name = "ServiceInfo2_s"; type = "string" } + @{ name = "SubscriptionGuid_g"; type = "string" } + @{ name = "SubscriptionName_s"; type = "string" } + @{ name = "Tags_s"; type = "string" } + @{ name = "Term_s"; type = "string" } + @{ name = "UnitOfMeasure_s"; type = "string" } + @{ name = "UnitPrice_s"; type = "string" } + ) + + "AADObjectsV1" = @( + @{ name = "TimeGenerated"; type = "datetime" } + @{ name = "Timestamp_s"; type = "string" } + @{ name = "TenantGuid_g"; type = "string" } + @{ name = "Cloud_s"; type = "string" } + @{ name = "ObjectId_g"; type = "string" } + @{ name = "ObjectType_s"; type = "string" } + @{ name = "ObjectSubType_s"; type = "string" } + @{ name = "DisplayName_s"; type = "string" } + @{ name = "SecurityEnabled_s"; type = "string" } + @{ name = "ApplicationId_g"; type = "string" } + @{ name = "Keys_s"; type = "string" } + @{ name = "PrincipalNames_s"; type = "string" } + @{ name = "Owners_s"; type = "string" } + @{ name = "CreatedDate_s"; type = "string" } + @{ name = "DeletedDate_s"; type = "string" } + ) + + "LoadBalancersV1" = @( + @{ name = "TimeGenerated"; type = "datetime" } + @{ name = "Timestamp_s"; type = "string" } + @{ name = "Cloud_s"; type = "string" } + @{ name = "TenantGuid_g"; type = "string" } + @{ name = "SubscriptionGuid_g"; type = "string" } + @{ name = "ResourceGroupName_s"; type = "string" } + @{ name = "InstanceName_s"; type = "string" } + @{ name = "InstanceId_s"; type = "string" } + @{ name = "SkuName_s"; type = "string" } + @{ name = "SkuTier_s"; type = "string" } + @{ name = "Location_s"; type = "string" } + @{ name = "LbType_s"; type = "string" } + @{ name = "LbRulesCount_d"; type = "real" } + @{ name = "InboundNatRulesCount_d"; type = "real" } + @{ name = "OutboundRulesCount_d"; type = "real" } + @{ name = "FrontendIPsCount_d"; type = "real" } + @{ name = "BackendIPCount_d"; type = "real" } + @{ name = "BackendAddressesCount_d"; type = "real" } + @{ name = "InboundNatPoolsCount_d"; type = "real" } + @{ name = "BackendPoolsCount_d"; type = "real" } + @{ name = "ProbesCount_d"; type = "real" } + @{ name = "StatusDate_s"; type = "string" } + @{ name = "Tags_s"; type = "string" } + ) + + "AppGatewaysV1" = @( + @{ name = "TimeGenerated"; type = "datetime" } + @{ name = "Timestamp_s"; type = "string" } + @{ name = "Cloud_s"; type = "string" } + @{ name = "TenantGuid_g"; type = "string" } + @{ name = "SubscriptionGuid_g"; type = "string" } + @{ name = "ResourceGroupName_s"; type = "string" } + @{ name = "InstanceName_s"; type = "string" } + @{ name = "InstanceId_s"; type = "string" } + @{ name = "SkuName_s"; type = "string" } + @{ name = "SkuTier_s"; type = "string" } + @{ name = "SkuCapacity_d"; type = "real" } + @{ name = "Location_s"; type = "string" } + @{ name = "Zones_s"; type = "string" } + @{ name = "EnableHttp2_s"; type = "string" } + @{ name = "GatewayIPsCount_d"; type = "real" } + @{ name = "FrontendIPsCount_d"; type = "real" } + @{ name = "FrontendPortsCount_d"; type = "real" } + @{ name = "BackendIPCount_d"; type = "real" } + @{ name = "BackendAddressesCount_d"; type = "real" } + @{ name = "HttpSettingsCount_d"; type = "real" } + @{ name = "HttpListenersCount_d"; type = "real" } + @{ name = "BackendPoolsCount_d"; type = "real" } + @{ name = "ProbesCount_d"; type = "real" } + @{ name = "UrlPathMapsCount_d"; type = "real" } + @{ name = "RequestRoutingRulesCount_d"; type = "real" } + @{ name = "RewriteRulesCount_d"; type = "real" } + @{ name = "RedirectConfsCount_d"; type = "real" } + @{ name = "StatusDate_s"; type = "string" } + @{ name = "Tags_s"; type = "string" } + ) + + "ResourceContainersV1" = @( + @{ name = "TimeGenerated"; type = "datetime" } + @{ name = "Timestamp_s"; type = "string" } + @{ name = "Cloud_s"; type = "string" } + @{ name = "TenantGuid_g"; type = "string" } + @{ name = "SubscriptionGuid_g"; type = "string" } + @{ name = "Location_s"; type = "string" } + @{ name = "ContainerType_s"; type = "string" } + @{ name = "ContainerName_s"; type = "string" } + @{ name = "InstanceId_s"; type = "string" } + @{ name = "ResourceCount_d"; type = "real" } + @{ name = "ManagedBy_s"; type = "string" } + @{ name = "ContainerProperties_s"; type = "string" } + @{ name = "Tags_s"; type = "string" } + @{ name = "StatusDate_s"; type = "string" } + ) + + "RBACAssignmentsV1" = @( + @{ name = "TimeGenerated"; type = "datetime" } + @{ name = "Timestamp_s"; type = "string" } + @{ name = "TenantGuid_g"; type = "string" } + @{ name = "Cloud_s"; type = "string" } + @{ name = "Model_s"; type = "string" } + @{ name = "PrincipalId_s"; type = "string" } + @{ name = "Scope_s"; type = "string" } + @{ name = "RoleDefinition_s"; type = "string" } + ) + + "VNetsV1" = @( + @{ name = "TimeGenerated"; type = "datetime" } + @{ name = "Timestamp_s"; type = "string" } + @{ name = "Cloud_s"; type = "string" } + @{ name = "TenantGuid_g"; type = "string" } + @{ name = "SubscriptionGuid_g"; type = "string" } + @{ name = "ResourceGroupName_s"; type = "string" } + @{ name = "Location_s"; type = "string" } + @{ name = "VNetName_s"; type = "string" } + @{ name = "InstanceId_s"; type = "string" } + @{ name = "Model_s"; type = "string" } + @{ name = "VNetPrefixes_s"; type = "string" } + @{ name = "DNSServers_s"; type = "string" } + @{ name = "PeeringsCount_d"; type = "real" } + @{ name = "EnableDdosProtection_s"; type = "string" } + @{ name = "SubnetName_s"; type = "string" } + @{ name = "SubnetPrefix_s"; type = "string" } + @{ name = "SubnetDelegationsCount_d"; type = "real" } + @{ name = "SubnetTotalPrefixIPs_d"; type = "real" } + @{ name = "SubnetUsedIPs_d"; type = "real" } + @{ name = "SubnetNSGId_s"; type = "string" } + @{ name = "Tags_s"; type = "string" } + @{ name = "StatusDate_s"; type = "string" } + ) + + "NICsV1" = @( + @{ name = "TimeGenerated"; type = "datetime" } + @{ name = "Timestamp_s"; type = "string" } + @{ name = "Cloud_s"; type = "string" } + @{ name = "TenantGuid_g"; type = "string" } + @{ name = "SubscriptionGuid_g"; type = "string" } + @{ name = "ResourceGroupName_s"; type = "string" } + @{ name = "Location_s"; type = "string" } + @{ name = "Name_s"; type = "string" } + @{ name = "InstanceId_s"; type = "string" } + @{ name = "IsPrimary_s"; type = "string" } + @{ name = "EnableAcceleratedNetworking_s"; type = "string" } + @{ name = "EnableIPForwarding_s"; type = "string" } + @{ name = "TapConfigurationsCount_d"; type = "real" } + @{ name = "HostedWorkloadsCount_d"; type = "real" } + @{ name = "InternalDomainNameSuffix_s"; type = "string" } + @{ name = "AppliedDnsServers_s"; type = "string" } + @{ name = "DnsServers_s"; type = "string" } + @{ name = "OwnerVMId_s"; type = "string" } + @{ name = "OwnerPEId_s"; type = "string" } + @{ name = "MacAddress_s"; type = "string" } + @{ name = "NicType_s"; type = "string" } + @{ name = "NicNSGId_s"; type = "string" } + @{ name = "PrivateIPAddressVersion_s"; type = "string" } + @{ name = "PrivateIPAllocationMethod_s"; type = "string" } + @{ name = "IsIPConfigPrimary_s"; type = "string" } + @{ name = "PrivateIPAddress_s"; type = "string" } + @{ name = "PublicIPId_s"; type = "string" } + @{ name = "IPConfigName_s"; type = "string" } + @{ name = "SubnetId_s"; type = "string" } + @{ name = "Tags_s"; type = "string" } + @{ name = "StatusDate_s"; type = "string" } + ) + + "NSGsV1" = @( + @{ name = "TimeGenerated"; type = "datetime" } + @{ name = "Timestamp_s"; type = "string" } + @{ name = "Cloud_s"; type = "string" } + @{ name = "TenantGuid_g"; type = "string" } + @{ name = "SubscriptionGuid_g"; type = "string" } + @{ name = "ResourceGroupName_s"; type = "string" } + @{ name = "Location_s"; type = "string" } + @{ name = "NSGName_s"; type = "string" } + @{ name = "InstanceId_s"; type = "string" } + @{ name = "NicCount_d"; type = "real" } + @{ name = "SubnetCount_d"; type = "real" } + @{ name = "RuleName_s"; type = "string" } + @{ name = "RuleProtocol_s"; type = "string" } + @{ name = "RuleDirection_s"; type = "string" } + @{ name = "RulePriority_d"; type = "real" } + @{ name = "RuleAccess_s"; type = "string" } + @{ name = "RuleDestinationAddresses_s"; type = "string" } + @{ name = "RuleSourceAddresses_s"; type = "string" } + @{ name = "RuleDestinationPorts_s"; type = "string" } + @{ name = "RuleSourcePorts_s"; type = "string" } + @{ name = "Tags_s"; type = "string" } + @{ name = "StatusDate_s"; type = "string" } + ) + + "PublicIPsV1" = @( + @{ name = "TimeGenerated"; type = "datetime" } + @{ name = "Timestamp_s"; type = "string" } + @{ name = "Cloud_s"; type = "string" } + @{ name = "TenantGuid_g"; type = "string" } + @{ name = "SubscriptionGuid_g"; type = "string" } + @{ name = "ResourceGroupName_s"; type = "string" } + @{ name = "Location_s"; type = "string" } + @{ name = "Name_s"; type = "string" } + @{ name = "InstanceId_s"; type = "string" } + @{ name = "Model_s"; type = "string" } + @{ name = "SkuName_s"; type = "string" } + @{ name = "SkuTier_s"; type = "string" } + @{ name = "AllocationMethod_s"; type = "string" } + @{ name = "AddressVersion_s"; type = "string" } + @{ name = "AssociatedResourceId_s"; type = "string" } + @{ name = "PublicIpPrefixId_s"; type = "string" } + @{ name = "IPAddress_s"; type = "string" } + @{ name = "FQDN_s"; type = "string" } + @{ name = "Zones_s"; type = "string" } + @{ name = "Tags_s"; type = "string" } + @{ name = "StatusDate_s"; type = "string" } + ) + + "VMSSV1" = @( + @{ name = "TimeGenerated"; type = "datetime" } + @{ name = "Timestamp_s"; type = "string" } + @{ name = "Cloud_s"; type = "string" } + @{ name = "TenantGuid_g"; type = "string" } + @{ name = "SubscriptionGuid_g"; type = "string" } + @{ name = "ResourceGroupName_s"; type = "string" } + @{ name = "Zones_s"; type = "string" } + @{ name = "Location_s"; type = "string" } + @{ name = "VMSSName_s"; type = "string" } + @{ name = "ComputerNamePrefix_s"; type = "string" } + @{ name = "InstanceId_s"; type = "string" } + @{ name = "VMSSSize_s"; type = "string" } + @{ name = "CoresCount_d"; type = "real" } + @{ name = "MemoryMB_d"; type = "real" } + @{ name = "OSType_s"; type = "string" } + @{ name = "DataDiskCount_d"; type = "real" } + @{ name = "NicCount_d"; type = "real" } + @{ name = "StatusDate_s"; type = "string" } + @{ name = "Tags_s"; type = "string" } + @{ name = "Capacity_s"; type = "string" } + @{ name = "Priority_s"; type = "string" } + @{ name = "OSDiskSize_s"; type = "string" } + @{ name = "OSDiskCaching_s"; type = "string" } + @{ name = "OSDiskSKU_s"; type = "string" } + @{ name = "SinglePlacementGroup_s"; type = "string" } + @{ name = "UpgradePolicy_s"; type = "string" } + @{ name = "OverProvision_s"; type = "string" } + @{ name = "PlatformFaultDomainCount_s"; type = "string" } + @{ name = "ZoneBalance_s"; type = "string" } + @{ name = "UsesManagedDisks_s"; type = "string" } + @{ name = "ImagePublisher_s"; type = "string" } + @{ name = "ImageOffer_s"; type = "string" } + @{ name = "ImageSku_s"; type = "string" } + @{ name = "ImageVersion_s"; type = "string" } + @{ name = "ImageExactVersion_s"; type = "string" } + ) + + "SqlDbV1" = @( + @{ name = "TimeGenerated"; type = "datetime" } + @{ name = "Timestamp_s"; type = "string" } + @{ name = "Cloud_s"; type = "string" } + @{ name = "TenantGuid_g"; type = "string" } + @{ name = "SubscriptionGuid_g"; type = "string" } + @{ name = "ResourceGroupName_s"; type = "string" } + @{ name = "ZoneRedundant_s"; type = "string" } + @{ name = "Location_s"; type = "string" } + @{ name = "DBName_s"; type = "string" } + @{ name = "InstanceId_s"; type = "string" } + @{ name = "SkuName_s"; type = "string" } + @{ name = "SkuTier_s"; type = "string" } + @{ name = "SkuCapacity_d"; type = "real" } + @{ name = "ServiceObjectiveName_s"; type = "string" } + @{ name = "StorageAccountType_s"; type = "string" } + @{ name = "LicenseType_s"; type = "string" } + @{ name = "MaxSizeBytes_d"; type = "real" } + @{ name = "MaxLogSizeBytes_d"; type = "real" } + @{ name = "Tags_s"; type = "string" } + @{ name = "StatusDate_s"; type = "string" } + ) + + "MonitorMetricsV1" = @( + @{ name = "TimeGenerated"; type = "datetime" } + @{ name = "Timestamp_s"; type = "string" } + @{ name = "Cloud_s"; type = "string" } + @{ name = "TenantGuid_g"; type = "string" } + @{ name = "SubscriptionGuid_g"; type = "string" } + @{ name = "ResourceGroupName_s"; type = "string" } + @{ name = "ResourceName_s"; type = "string" } + @{ name = "ResourceId_s"; type = "string" } + @{ name = "MetricNames_s"; type = "string" } + @{ name = "AggregationType_s"; type = "string" } + @{ name = "AggregationOfType_s"; type = "string" } + @{ name = "MetricValue_d"; type = "real" } + @{ name = "TimeGrain_s"; type = "string" } + @{ name = "TimeSpan_s"; type = "string" } + ) + + "PolicyStatesV1" = @( + @{ name = "TimeGenerated"; type = "datetime" } + @{ name = "Timestamp_s"; type = "string" } + @{ name = "Cloud_s"; type = "string" } + @{ name = "TenantGuid_g"; type = "string" } + @{ name = "SubscriptionGuid_g"; type = "string" } + @{ name = "ResourceGroupName_s"; type = "string" } + @{ name = "ResourceId_s"; type = "string" } + @{ name = "ResourceType_s"; type = "string" } + @{ name = "ComplianceState_s"; type = "string" } + @{ name = "ComplianceReason_s"; type = "string" } + @{ name = "Effect_s"; type = "string" } + @{ name = "AssignmentId_s"; type = "string" } + @{ name = "AssignmentName_s"; type = "string" } + @{ name = "InitiativeId_s"; type = "string" } + @{ name = "InitiativeName_s"; type = "string" } + @{ name = "DefinitionId_s"; type = "string" } + @{ name = "DefinitionName_s"; type = "string" } + @{ name = "DefinitionReferenceId_s"; type = "string" } + @{ name = "EvaluatedOn_s"; type = "string" } + @{ name = "StatesCount_d"; type = "real" } + @{ name = "Tags_s"; type = "string" } + @{ name = "StatusDate_s"; type = "string" } + ) + + "RecommendationsV1" = @( + @{ name = "TimeGenerated"; type = "datetime" } + @{ name = "Timestamp_s"; type = "string" } + @{ name = "Cloud_s"; type = "string" } + @{ name = "Category_s"; type = "string" } + @{ name = "ImpactedArea_s"; type = "string" } + @{ name = "Impact_s"; type = "string" } + @{ name = "RecommendationType_s"; type = "string" } + @{ name = "RecommendationSubType_s"; type = "string" } + @{ name = "RecommendationSubTypeId_g"; type = "string" } + @{ name = "RecommendationDescription_s"; type = "string" } + @{ name = "RecommendationAction_s"; type = "string" } + @{ name = "InstanceId_s"; type = "string" } + @{ name = "InstanceName_s"; type = "string" } + @{ name = "AdditionalInfo_s"; type = "string" } + @{ name = "ResourceGroup_s"; type = "string" } + @{ name = "SubscriptionGuid_g"; type = "string" } + @{ name = "SubscriptionName_s"; type = "string" } + @{ name = "TenantGuid_g"; type = "string" } + @{ name = "FitScore_d"; type = "real" } + @{ name = "Tags_s"; type = "string" } + @{ name = "DetailsURL_s"; type = "string" } + @{ name = "GeneratedDate_s"; type = "string" } + ) + + "ReservationsUsageV1" = @( + @{ name = "TimeGenerated"; type = "datetime" } + @{ name = "Timestamp_s"; type = "string" } + @{ name = "ReservationResourceId_s"; type = "string" } + @{ name = "ReservationOrderId_s"; type = "string" } + @{ name = "ReservationId_g"; type = "string" } + @{ name = "DisplayName_s"; type = "string" } + @{ name = "SKUName_s"; type = "string" } + @{ name = "Location_s"; type = "string" } + @{ name = "ResourceType_s"; type = "string" } + @{ name = "AppliedScopeType_s"; type = "string" } + @{ name = "Term_s"; type = "string" } + @{ name = "ProvisioningState_s"; type = "string" } + @{ name = "RenewState_s"; type = "string" } + @{ name = "PurchaseDate_s"; type = "string" } + @{ name = "ExpiryDate_s"; type = "string" } + @{ name = "Archived_s"; type = "string" } + @{ name = "ReservedHours_d"; type = "real" } + @{ name = "UsedHours_d"; type = "real" } + @{ name = "UsageDate_s"; type = "string" } + @{ name = "MinUtilPercentage_d"; type = "real" } + @{ name = "AvgUtilPercentage_d"; type = "real" } + @{ name = "MaxUtilPercentage_d"; type = "real" } + @{ name = "PurchasedQuantity_d"; type = "real" } + @{ name = "RemainingQuantity_d"; type = "real" } + @{ name = "TotalReservedQuantity_d"; type = "real" } + @{ name = "UsedQuantity_d"; type = "real" } + @{ name = "UtilizedPercentage_d"; type = "real" } + @{ name = "UtilTrend_s"; type = "string" } + @{ name = "Util1Days_d"; type = "real" } + @{ name = "Util7Days_d"; type = "real" } + @{ name = "Util30Days_d"; type = "real" } + @{ name = "Scope_s"; type = "string" } + @{ name = "TenantGuid_g"; type = "string" } + @{ name = "Cloud_s"; type = "string" } + @{ name = "CollectedDate_s"; type = "string" } + ) + + "AppServicePlansV1" = @( + @{ name = "TimeGenerated"; type = "datetime" } + @{ name = "Timestamp_s"; type = "string" } + @{ name = "Cloud_s"; type = "string" } + @{ name = "TenantGuid_g"; type = "string" } + @{ name = "SubscriptionGuid_g"; type = "string" } + @{ name = "ResourceGroupName_s"; type = "string" } + @{ name = "ZoneRedundant_s"; type = "string" } + @{ name = "Location_s"; type = "string" } + @{ name = "AppServicePlanName_s"; type = "string" } + @{ name = "InstanceId_s"; type = "string" } + @{ name = "Kind_s"; type = "string" } + @{ name = "SkuName_s"; type = "string" } + @{ name = "SkuTier_s"; type = "string" } + @{ name = "SkuCapacity_d"; type = "real" } + @{ name = "SkuFamily_s"; type = "string" } + @{ name = "SkuSize_s"; type = "string" } + @{ name = "ComputeMode_s"; type = "string" } + @{ name = "NumberOfWorkers_d"; type = "real" } + @{ name = "CurrentNumberOfWorkers_d"; type = "real" } + @{ name = "MaximumNumberOfWorkers_d"; type = "real" } + @{ name = "NumberOfSites_d"; type = "real" } + @{ name = "PlanName_s"; type = "string" } + @{ name = "Tags_s"; type = "string" } + @{ name = "StatusDate_s"; type = "string" } + ) + + "PricesheetV1" = @( + @{ name = "TimeGenerated"; type = "datetime" } + @{ name = "Timestamp_s"; type = "string" } + @{ name = "MeterID_g"; type = "string" } + @{ name = "MeterName_s"; type = "string" } + @{ name = "MeterCategory_s"; type = "string" } + @{ name = "MeterSubCategory_s"; type = "string" } + @{ name = "MeterRegion_s"; type = "string" } + @{ name = "UnitOfMeasure_s"; type = "string" } + @{ name = "PartNumber_s"; type = "string" } + @{ name = "UnitPrice_s"; type = "string" } + @{ name = "CurrencyCode_s"; type = "string" } + @{ name = "IncludedQuantity_s"; type = "string" } + @{ name = "OfferId_s"; type = "string" } + @{ name = "Term_s"; type = "string" } + @{ name = "PriceType_s"; type = "string" } + ) + + "ReservationsPriceV1" = @( + @{ name = "TimeGenerated"; type = "datetime" } + @{ name = "Timestamp_s"; type = "string" } + @{ name = "productName_s"; type = "string" } + @{ name = "serviceName_s"; type = "string" } + @{ name = "skuName_s"; type = "string" } + @{ name = "skuId_s"; type = "string" } + @{ name = "armRegionName_s"; type = "string" } + @{ name = "location_s"; type = "string" } + @{ name = "effectiveStartDate_s"; type = "string" } + @{ name = "effectiveEndDate_s"; type = "string" } + @{ name = "unitPrice_s"; type = "string" } + @{ name = "currencyCode_s"; type = "string" } + @{ name = "unitOfMeasure_s"; type = "string" } + @{ name = "armSkuName_s"; type = "string" } + @{ name = "productId_s"; type = "string" } + @{ name = "reservationTerm_s"; type = "string" } + @{ name = "meterName_s"; type = "string" } + ) + + "SavingsPlansUsageV1" = @( + @{ name = "TimeGenerated"; type = "datetime" } + @{ name = "Timestamp_s"; type = "string" } + @{ name = "SavingsPlanResourceId_s"; type = "string" } + @{ name = "SavingsPlanOrderId_s"; type = "string" } + @{ name = "SavingsPlanId_g"; type = "string" } + @{ name = "DisplayName_s"; type = "string" } + @{ name = "SKUName_s"; type = "string" } + @{ name = "Term_s"; type = "string" } + @{ name = "ProvisioningState_s"; type = "string" } + @{ name = "AppliedScopeType_s"; type = "string" } + @{ name = "RenewState_s"; type = "string" } + @{ name = "PurchaseDate_s"; type = "string" } + @{ name = "BenefitStart_s"; type = "string" } + @{ name = "ExpiryDate_s"; type = "string" } + @{ name = "EffectiveDate_s"; type = "string" } + @{ name = "BillingScopeId_s"; type = "string" } + @{ name = "BillingAccountId_s"; type = "string" } + @{ name = "BillingProfileId_s"; type = "string" } + @{ name = "BillingPlan_s"; type = "string" } + @{ name = "CommitmentGrain_s"; type = "string" } + @{ name = "CommitmentCurrencyCode_s"; type = "string" } + @{ name = "CommitmentAmount_d"; type = "real" } + @{ name = "UtilTrend_s"; type = "string" } + @{ name = "Util1Days_d"; type = "real" } + @{ name = "Util7Days_d"; type = "real" } + @{ name = "Util30Days_d"; type = "real" } + @{ name = "Scope_s"; type = "string" } + @{ name = "TenantGuid_g"; type = "string" } + @{ name = "Cloud_s"; type = "string" } + @{ name = "CollectedDate_s"; type = "string" } + ) + + "SuppressionsV1" = @( + @{ name = "TimeGenerated"; type = "datetime" } + @{ name = "Timestamp_s"; type = "string" } + @{ name = "FilterId_g"; type = "string" } + @{ name = "RecommendationSubTypeId_g"; type = "string" } + @{ name = "FilterType_s"; type = "string" } + @{ name = "InstanceId_g"; type = "string" } + @{ name = "InstanceName_s"; type = "string" } + @{ name = "FilterStartDate_s"; type = "string" } + @{ name = "FilterEndDate_s"; type = "string" } + @{ name = "Author_s"; type = "string" } + @{ name = "Notes_s"; type = "string" } + ) +} + +# Storage container -> LogAnalyticsSuffix mapping used to update SQL control table +$containerSuffixMap = @{ + "argvmexports" = "VMsV1" + "argdiskexports" = "DisksV1" + "argvhdexports" = "VhdDisksV1" + "argavailsetexports" = "AvailSetsV1" + "advisorexports" = "AdvisorV1" + "remediationlogs" = "RemediationV1" + "consumptionexports" = "ConsumptionV1" + "aadobjectsexports" = "AADObjectsV1" + "arglbexports" = "LoadBalancersV1" + "argappgwexports" = "AppGatewaysV1" + "argrescontainersexports" = "ResourceContainersV1" + "rbacexports" = "RBACAssignmentsV1" + "argvnetexports" = "VNetsV1" + "argnicexports" = "NICsV1" + "argnsgexports" = "NSGsV1" + "argpublicipexports" = "PublicIPsV1" + "argvmssexports" = "VMSSV1" + "argsqldbexports" = "SqlDbV1" + "azmonitorexports" = "MonitorMetricsV1" + "policystateexports" = "PolicyStatesV1" + "recommendationsexports" = "RecommendationsV1" + "reservationsexports" = "ReservationsUsageV1" + "argappserviceplanexports" = "AppServicePlansV1" + "pricesheetexports" = "PricesheetV1" + "reservationspriceexports" = "ReservationsPriceV1" + "savingsplansexports" = "SavingsPlansUsageV1" +} +#endregion + +#region Get ARM access token (used for Log Analytics Tables API and DCR API) +$tableApiVersion = "2023-01-01-preview" +$dcrApiVersion = "2022-06-01" +$lognamePrefix = "AzureOptimization" + +$armToken = (Get-AzAccessToken -ResourceUrl "$armEndpoint/").Token +$armHeaders = @{ + "Authorization" = "Bearer $armToken" + "Content-Type" = "application/json" +} +#endregion + +#region Get Log Analytics workspace resource ID in the target subscription context +if ($WorkspaceSubscriptionId -ne $currentSubscriptionId) +{ + Write-Host "Switching to workspace subscription $WorkspaceSubscriptionId..." -ForegroundColor Cyan + Set-AzContext -SubscriptionId $WorkspaceSubscriptionId | Out-Null + $armToken = (Get-AzAccessToken -ResourceUrl "$armEndpoint/").Token + $armHeaders["Authorization"] = "Bearer $armToken" +} +#endregion + +#region Create/update custom Log Analytics tables and DCRs +Write-Host "Processing $($tableSchemas.Keys.Count) custom Log Analytics tables and DCRs..." -ForegroundColor Green + +$dcrSuffixToImmutableId = @{} + +foreach ($suffix in $tableSchemas.Keys) +{ + $tableName = "$lognamePrefix$suffix" + "_CL" + $streamName = "Custom-$lognamePrefix$suffix" + $dcrName = "AOE-DCR-$suffix" + + Write-Host " [$suffix] Setting up table $tableName..." -ForegroundColor Cyan + + #region Build table columns payload + $columns = @() + foreach ($col in $tableSchemas[$suffix]) + { + $columns += @{ name = $col.name; type = $col.type } + } + + $tablePayload = @{ + properties = @{ + schema = @{ + name = $tableName + columns = $columns + } + } + } | ConvertTo-Json -Depth 10 + + $tableUri = "$armEndpoint$workspaceResourceId/tables/$tableName`?api-version=$tableApiVersion" + + # Check if the table already exists and whether it is a classic (Data Collector API) table. + # Classic tables cannot be updated via PUT directly; they must first be migrated to DCR-based. + $existingTable = $null + try + { + $existingTable = Invoke-RestMethod -Method GET -Uri $tableUri -Headers $armHeaders -ErrorAction Stop + } + catch + { + # Table does not yet exist; the PUT below will create it as a DCR-based table. + } + + if ($null -ne $existingTable -and $existingTable.properties.schema.tableSubType -eq "ClassicAPI") + { + # Migrate the classic table to DCR-based before updating the schema. + # This call is idempotent; it has no effect if the table has already been converted. + Write-Host " Table $tableName is a classic (Data Collector API) table. Migrating to DCR-based..." -ForegroundColor Yellow + $migrateUri = "$armEndpoint$workspaceResourceId/tables/$tableName/migrate?api-version=2021-12-01-preview" + Invoke-RestMethod -Method POST -Uri $migrateUri -Headers $armHeaders -Body "{}" | Out-Null + Write-Host " Migration completed for $tableName." -ForegroundColor Gray + } + + $tableResponse = Invoke-RestMethod -Method PUT -Uri $tableUri -Headers $armHeaders -Body $tablePayload + Write-Host " Table $tableName created/updated (subType: $($tableResponse.properties.schema.tableSubType))" -ForegroundColor Gray + #endregion + + #region Create DCR for this table + Write-Host " [$suffix] Creating DCR $dcrName..." -ForegroundColor Cyan + + $dcrPayload = @{ + location = $workspaceLocation + properties = @{ + dataCollectionEndpointId = $dceResourceId + streamDeclarations = @{ + $streamName = @{ + columns = $columns + } + } + destinations = @{ + logAnalytics = @( + @{ + workspaceResourceId = $workspaceResourceId + name = "laDest" + } + ) + } + dataFlows = @( + @{ + streams = @($streamName) + destinations = @("laDest") + transformKql = "source" + outputStream = "Custom-$tableName" + } + ) + } + } | ConvertTo-Json -Depth 10 + + # Switch context back to automation account subscription for DCR creation + if ($WorkspaceSubscriptionId -ne $currentSubscriptionId) + { + Set-AzContext -SubscriptionId $currentSubscriptionId | Out-Null + $armToken = (Get-AzAccessToken -ResourceUrl "$armEndpoint/").Token + $armHeaders["Authorization"] = "Bearer $armToken" + } + + $dcrUri = "$armEndpoint/subscriptions/$currentSubscriptionId/resourceGroups/$ResourceGroupName/providers/Microsoft.Insights/dataCollectionRules/$dcrName`?api-version=$dcrApiVersion" + $dcrResponse = Invoke-RestMethod -Method PUT -Uri $dcrUri -Headers $armHeaders -Body $dcrPayload + $dcrImmutableId = $dcrResponse.properties.immutableId + $dcrResourceId = $dcrResponse.id + $dcrSuffixToImmutableId[$suffix] = $dcrImmutableId + Write-Host " DCR immutable ID: $dcrImmutableId" -ForegroundColor Gray + + # Switch back to workspace subscription if needed for subsequent table operations + if ($WorkspaceSubscriptionId -ne $currentSubscriptionId) + { + Set-AzContext -SubscriptionId $WorkspaceSubscriptionId | Out-Null + $armToken = (Get-AzAccessToken -ResourceUrl "$armEndpoint/").Token + $armHeaders["Authorization"] = "Bearer $armToken" + } + #endregion + + #region Assign Monitoring Metrics Publisher role on DCR to Automation account MI + Write-Host " [$suffix] Granting Monitoring Metrics Publisher on DCR to Automation MI..." -ForegroundColor Cyan + + # Switch to DCR subscription for role assignment + if ($WorkspaceSubscriptionId -ne $currentSubscriptionId) + { + Set-AzContext -SubscriptionId $currentSubscriptionId | Out-Null + } + + $existingAssignment = Get-AzRoleAssignment -ObjectId $automationPrincipalId ` + -RoleDefinitionId $monitoringMetricsPublisherRoleId ` + -Scope $dcrResourceId -ErrorAction SilentlyContinue + if ($null -eq $existingAssignment) + { + New-AzRoleAssignment -ObjectId $automationPrincipalId ` + -RoleDefinitionId $monitoringMetricsPublisherRoleId ` + -Scope $dcrResourceId | Out-Null + Write-Host " Role assigned." -ForegroundColor Gray + } + else + { + Write-Host " Role already assigned." -ForegroundColor Gray + } + + # Restore workspace subscription context for next iteration + if ($WorkspaceSubscriptionId -ne $currentSubscriptionId) + { + Set-AzContext -SubscriptionId $WorkspaceSubscriptionId | Out-Null + $armToken = (Get-AzAccessToken -ResourceUrl "$armEndpoint/").Token + $armHeaders["Authorization"] = "Bearer $armToken" + } + #endregion +} +#endregion + +#region Restore context to current subscription +if ($WorkspaceSubscriptionId -ne $currentSubscriptionId) +{ + Set-AzContext -SubscriptionId $currentSubscriptionId | Out-Null +} +#endregion + +#region Update LogAnalyticsIngestControl SQL table with DCR immutable IDs +Write-Host "Updating SQL LogAnalyticsIngestControl with DCR immutable IDs..." -ForegroundColor Green + +$sqlToken = (Get-AzAccessToken -ResourceUrl "https://database.windows.net/").Token +$sqlConnectionString = "Server=tcp:$SqlServerName,1433;Initial Catalog=$SqlDatabaseName;Persist Security Info=False;MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Authentication=Active Directory Default;" + +try +{ + $sqlConnection = New-Object System.Data.SqlClient.SqlConnection + $sqlConnection.ConnectionString = $sqlConnectionString + $sqlConnection.AccessToken = $sqlToken + $sqlConnection.Open() + + foreach ($container in $containerSuffixMap.Keys) + { + $suffix = $containerSuffixMap[$container] + if ($dcrSuffixToImmutableId.ContainsKey($suffix)) + { + $immutableId = $dcrSuffixToImmutableId[$suffix] + $sqlCommand = $sqlConnection.CreateCommand() + $sqlCommand.CommandText = "UPDATE [dbo].[LogAnalyticsIngestControl] SET DCRImmutableId = @id WHERE StorageContainerName = @container" + $sqlCommand.Parameters.AddWithValue("@id", $immutableId) | Out-Null + $sqlCommand.Parameters.AddWithValue("@container", $container) | Out-Null + $sqlCommand.ExecuteNonQuery() | Out-Null + } + } + + # Update the SuppressionsV1 DCR immutable ID (not in container map, keyed directly) + if ($dcrSuffixToImmutableId.ContainsKey("SuppressionsV1")) + { + # Suppressions are not blob-based; store the immutable ID under a placeholder container name + $suppImmutableId = $dcrSuffixToImmutableId["SuppressionsV1"] + $sqlCommand = $sqlConnection.CreateCommand() + $sqlCommand.CommandText = @" +IF NOT EXISTS (SELECT 1 FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'suppressions') +BEGIN + INSERT INTO [dbo].[LogAnalyticsIngestControl] (StorageContainerName, LastProcessedDateTime, LastProcessedLine, LogAnalyticsSuffix, CollectedType, DCRImmutableId) + VALUES ('suppressions', '1901-01-01T00:00:00Z', -1, 'SuppressionsV1', 'Suppressions', @id) +END +ELSE +BEGIN + UPDATE [dbo].[LogAnalyticsIngestControl] SET DCRImmutableId = @id WHERE StorageContainerName = 'suppressions' +END +"@ + $sqlCommand.Parameters.AddWithValue("@id", $suppImmutableId) | Out-Null + $sqlCommand.ExecuteNonQuery() | Out-Null + } + + $sqlConnection.Close() + Write-Host "SQL update completed." -ForegroundColor Green +} +catch +{ + Write-Host "Could not update SQL control table: $($_.Exception.Message)" -ForegroundColor Yellow + Write-Host "You may need to update the DCRImmutableId column manually or ensure the managed identity has db_datareader/db_datawriter access." -ForegroundColor Yellow +} +#endregion + +#region Remove the legacy workspace shared key automation variable +Write-Host "Removing legacy AzureOptimization_LogAnalyticsWorkspaceKey automation variable..." -ForegroundColor Green +$keyVar = Get-AzAutomationVariable -ResourceGroupName $ResourceGroupName -AutomationAccountName $AutomationAccountName ` + -Name "AzureOptimization_LogAnalyticsWorkspaceKey" -ErrorAction SilentlyContinue +if ($null -ne $keyVar) +{ + Remove-AzAutomationVariable -ResourceGroupName $ResourceGroupName -AutomationAccountName $AutomationAccountName ` + -Name "AzureOptimization_LogAnalyticsWorkspaceKey" + Write-Host "Legacy workspace key variable removed." -ForegroundColor Green +} +else +{ + Write-Host "Variable already absent." -ForegroundColor Gray +} +#endregion + +Write-Host "Setup-LogAnalyticsTablesAndDCRs completed successfully." -ForegroundColor Green +Write-Host " Tables created: $($tableSchemas.Keys.Count)" -ForegroundColor Green +Write-Host " DCRs created/updated: $($dcrSuffixToImmutableId.Keys.Count)" -ForegroundColor Green diff --git a/src/optimization-engine/azuredeploy-nested.bicep b/src/optimization-engine/azuredeploy-nested.bicep index 0e12a764d..e6ed4196a 100644 --- a/src/optimization-engine/azuredeploy-nested.bicep +++ b/src/optimization-engine/azuredeploy-nested.bicep @@ -1274,7 +1274,7 @@ var automationVariables = [ { name: 'AzureOptimization_LogAnalyticsChunkSize' description: 'The size (in rows) for each chunk of Log Analytics ingestion request' - value: 6000 + value: 150 } { name: 'AzureOptimization_StorageBlobsPageSize' @@ -1570,6 +1570,17 @@ resource logAnalyticsWorkspace 'microsoft.operationalinsights/workspaces@2020-08 } } +resource dataCollectionEndpoint 'Microsoft.Insights/dataCollectionEndpoints@2022-06-01' = { + name: '${automationAccountName}-dce' + location: projectLocation + tags: resourceTags + properties: { + networkAcls: { + publicNetworkAccess: 'Enabled' + } + } +} + resource storageAccount 'Microsoft.Storage/storageAccounts@2022-09-01' = { name: storageAccountName location: projectLocation @@ -1893,6 +1904,15 @@ resource automationVariables_LogAnalyticsWorkspaceKey 'Microsoft.Automation/auto } } +resource automationVariables_DCEIngestionEndpoint 'Microsoft.Automation/automationAccounts/variables@2020-01-13-preview' = { + parent: automationAccount + name: 'AzureOptimization_DCEIngestionEndpoint' + properties: { + description: 'The Logs Ingestion endpoint URL of the Data Collection Endpoint used for DCR-based ingestion' + value: '"${dataCollectionEndpoint.properties.logsIngestion.endpoint}"' + } +} + resource automationSchedules_csvExports 'Microsoft.Automation/automationAccounts/schedules@2020-01-13-preview' = [for item in csvExportsSchedules: { parent: automationAccount name: item.exportSchedule @@ -2156,3 +2176,5 @@ resource contributorRoleAssignmentGuid_resource 'Microsoft.Authorization/roleAss } output automationPrincipalId string = reference(automationAccount.id, '2019-06-01', 'Full').identity.principalId +output dceLogsIngestionEndpoint string = dataCollectionEndpoint.properties.logsIngestion.endpoint +output dceResourceId string = dataCollectionEndpoint.id diff --git a/src/optimization-engine/model/loganalyticsingestcontrol-table.sql b/src/optimization-engine/model/loganalyticsingestcontrol-table.sql index 23d87d061..81f2541c4 100644 --- a/src/optimization-engine/model/loganalyticsingestcontrol-table.sql +++ b/src/optimization-engine/model/loganalyticsingestcontrol-table.sql @@ -8,7 +8,8 @@ AND OBJECTPROPERTY(id, N'IsUserTable') = 1) [LastProcessedDateTime] [datetime] NULL, [LastProcessedLine] [int] NULL, [LogAnalyticsSuffix] [varchar](50) NOT NULL, - [CollectedType] [varchar](50) NULL + [CollectedType] [varchar](50) NULL, + [DCRImmutableId] [varchar](100) NULL ) ALTER TABLE [dbo].[LogAnalyticsIngestControl] ADD PRIMARY KEY CLUSTERED @@ -18,8 +19,12 @@ AND OBJECTPROPERTY(id, N'IsUserTable') = 1) END ELSE BEGIN - IF NOT EXISTS (SELECT * FROM sys.columns WHERE object_id = OBJECT_ID(N'[dbo].[LogAnalyticsIngestControl]') AND name = 'CollectedType' -) BEGIN + IF NOT EXISTS (SELECT * FROM sys.columns WHERE object_id = OBJECT_ID(N'[dbo].[LogAnalyticsIngestControl]') AND name = 'CollectedType') + BEGIN ALTER TABLE [dbo].[LogAnalyticsIngestControl] ADD [CollectedType] VARCHAR (50) NULL END + IF NOT EXISTS (SELECT * FROM sys.columns WHERE object_id = OBJECT_ID(N'[dbo].[LogAnalyticsIngestControl]') AND name = 'DCRImmutableId') + BEGIN + ALTER TABLE [dbo].[LogAnalyticsIngestControl] ADD [DCRImmutableId] VARCHAR (100) NULL + END END \ No newline at end of file diff --git a/src/optimization-engine/runbooks/data-collection/Ingest-OptimizationCSVExportsToLogAnalytics.ps1 b/src/optimization-engine/runbooks/data-collection/Ingest-OptimizationCSVExportsToLogAnalytics.ps1 index 2fdcb1bdf..63fc8bc27 100644 --- a/src/optimization-engine/runbooks/data-collection/Ingest-OptimizationCSVExportsToLogAnalytics.ps1 +++ b/src/optimization-engine/runbooks/data-collection/Ingest-OptimizationCSVExportsToLogAnalytics.ps1 @@ -26,8 +26,7 @@ if ([string]::IsNullOrEmpty($sqldatabase)) { $sqldatabase = "azureoptimization" } -$workspaceId = Get-AutomationVariable -Name "AzureOptimization_LogAnalyticsWorkspaceId" -$sharedKey = Get-AutomationVariable -Name "AzureOptimization_LogAnalyticsWorkspaceKey" +$dceEndpoint = Get-AutomationVariable -Name "AzureOptimization_DCEIngestionEndpoint" $LogAnalyticsChunkSize = [int] (Get-AutomationVariable -Name "AzureOptimization_LogAnalyticsChunkSize" -ErrorAction SilentlyContinue) if (-not($LogAnalyticsChunkSize -gt 0)) { @@ -70,80 +69,53 @@ switch ($authenticationOption) #region Functions -# Function to create the authorization signature -function Build-OMSSignature ($workspaceId, $sharedKey, $date, $contentLength, $method, $contentType, $resource) +# Sends data to a Log Analytics custom table via the DCR-based Logs Ingestion API. +# Uses the Automation account managed identity bearer token for authentication. +function Send-LogIngestionData($accessToken, $dceEndpoint, $dcrImmutableId, $streamName, $body) { - $xHeaders = "x-ms-date:" + $date - $stringToHash = $method + "`n" + $contentLength + "`n" + $contentType + "`n" + $xHeaders + "`n" + $resource - $bytesToHash = [Text.Encoding]::UTF8.GetBytes($stringToHash) - $keyBytes = [Convert]::FromBase64String($sharedKey) - $sha256 = New-Object System.Security.Cryptography.HMACSHA256 - $sha256.Key = $keyBytes - $calculatedHash = $sha256.ComputeHash($bytesToHash) - $encodedHash = [Convert]::ToBase64String($calculatedHash) - $authorization = 'SharedKey {0}:{1}' -f $workspaceId, $encodedHash - return $authorization + $uri = "$dceEndpoint/dataCollectionRules/$dcrImmutableId/streams/$streamName`?api-version=2023-01-01" + $headers = @{ + "Authorization" = "Bearer $accessToken" + "Content-Type" = "application/json" + } + $response = Invoke-WebRequest -Uri $uri -Method POST -Headers $headers -Body $body -UseBasicParsing -TimeoutSec 1000 + return $response.StatusCode } -# Function to create and post the request -function Post-OMSData($workspaceId, $sharedKey, $body, $logType, $TimeStampField, $AzureEnvironment) +# Converts CSV string values in a PSObject to the correct types expected by DCR typed columns. +# Columns ending in _d are cast to [double]; all others remain as strings. +function ConvertTo-TypedObject($obj) { - $method = "POST" - $contentType = "application/json" - $resource = "/api/logs" - $rfc1123date = [DateTime]::UtcNow.ToString("r") - $contentLength = $body.Length - $signature = Build-OMSSignature ` - -workspaceId $workspaceId ` - -sharedKey $sharedKey ` - -date $rfc1123date ` - -contentLength $contentLength ` - -method $method ` - -contentType $contentType ` - -resource $resource - - $uri = "https://" + $workspaceId + ".ods.opinsights.azure.com" + $resource + "?api-version=2016-04-01" - if ($AzureEnvironment -eq "AzureChinaCloud") + $typed = [ordered]@{} + foreach ($prop in $obj.PSObject.Properties) { - $uri = "https://" + $workspaceId + ".ods.opinsights.azure.cn" + $resource + "?api-version=2016-04-01" - } - if ($AzureEnvironment -eq "AzureUSGovernment") - { - $uri = "https://" + $workspaceId + ".ods.opinsights.azure.us" + $resource + "?api-version=2016-04-01" - } - if ($AzureEnvironment -eq "AzureGermanCloud") - { - throw "Azure Germany isn't supported for the Log Analytics Data Collector API" - } - - $OMSheaders = @{ - "Authorization" = $signature - "Log-Type" = $logType - "x-ms-date" = $rfc1123date - "time-generated-field" = $TimeStampField - } - - try - { - - $response = Invoke-WebRequest -Uri $uri -Method POST -ContentType $contentType -Headers $OMSheaders -Body $body -UseBasicParsing -TimeoutSec 1000 - } - catch - { - if ($_.Exception.Response.StatusCode.Value__ -eq 401) + $name = $prop.Name + $value = $prop.Value + if ($name -eq 'Timestamp') { - "REAUTHENTICATING" - - $response = Invoke-WebRequest -Uri $uri -Method POST -ContentType $contentType -Headers $OMSheaders -Body $body -UseBasicParsing -TimeoutSec 1000 + # Rename to TimeGenerated as required by Log Analytics custom tables + $typed['TimeGenerated'] = $value + } + elseif ($name.EndsWith('_d')) + { + $d = 0.0 + if ([double]::TryParse($value, [System.Globalization.NumberStyles]::Any, [System.Globalization.CultureInfo]::InvariantCulture, [ref]$d)) + { + $typed[$name] = $d + } + else + { + $typed[$name] = $null + } } else { - return $_.Exception.Response.StatusCode.Value__ + $typed[$name] = $value } } - - return $response.StatusCode + return [PSCustomObject]$typed } + #endregion Functions $cloudDetails = Get-AzEnvironment -Name $CloudEnvironment @@ -213,9 +185,25 @@ $controlRow = $controlRows[0] $lastProcessedLine = $controlRow.LastProcessedLine $lastProcessedDateTime = $controlRow.LastProcessedDateTime.ToString("yyyy'-'MM'-'dd'T'HH':'mm':'ss'.'fff'Z'") $LogAnalyticsSuffix = $controlRow.LogAnalyticsSuffix +$dcrImmutableId = $controlRow.DCRImmutableId $logname = $lognamePrefix + $LogAnalyticsSuffix +$streamName = "Custom-$lognamePrefix$LogAnalyticsSuffix" + +if ([string]::IsNullOrEmpty($dcrImmutableId)) +{ + throw "DCRImmutableId is not set for container $storageAccountSinkContainer. Run Setup-LogAnalyticsTablesAndDCRs.ps1 first." +} + +# Obtain a bearer token for the Logs Ingestion API using the Automation managed identity +switch ($cloudEnvironment) +{ + "AzureChinaCloud" { $monitorAudience = "https://monitor.azure.cn/" } + "AzureUSGovernment" { $monitorAudience = "https://monitor.azure.us/" } + default { $monitorAudience = "https://monitor.azure.com/" } +} +$monitorToken = (Get-AzAccessToken -ResourceUrl $monitorAudience).Token -Write-Output "Processing blobs modified after $lastProcessedDateTime (line $lastProcessedLine) and ingesting them into the $($logname)_CL table..." +Write-Output "Processing blobs modified after $lastProcessedDateTime (line $lastProcessedLine) and ingesting them into the $($logname)_CL table (stream $streamName)..." $newProcessedTime = $null @@ -268,11 +256,16 @@ foreach ($blob in $unprocessedBlobs) if (($lineCounter -eq $LogAnalyticsChunkSize -or $r.Peek() -lt 0) -and $linesProcessed -gt 0) { $csvObject = $chunkLines | ConvertFrom-Csv - $jsonObject = ConvertTo-Json -InputObject $csvObject + $typedObjects = $csvObject | ForEach-Object { ConvertTo-TypedObject $_ } + $jsonObject = ConvertTo-Json -InputObject @($typedObjects) -Depth 3 if ($null -ne $jsonObject) { - $res = Post-OMSData -workspaceId $workspaceId -sharedKey $sharedKey -body ([System.Text.Encoding]::UTF8.GetBytes($jsonObject)) -logType $logname -TimeStampField "Timestamp" -AzureEnvironment $cloudEnvironment + # Refresh token for long-running ingestion jobs + $monitorToken = (Get-AzAccessToken -ResourceUrl $monitorAudience).Token + $res = Send-LogIngestionData -accessToken $monitorToken -dceEndpoint $dceEndpoint ` + -dcrImmutableId $dcrImmutableId -streamName $streamName ` + -body ([System.Text.Encoding]::UTF8.GetBytes($jsonObject)) if ($res -ge 200 -and $res -lt 300) { diff --git a/src/optimization-engine/runbooks/recommendations/Ingest-RecommendationsToLogAnalytics.ps1 b/src/optimization-engine/runbooks/recommendations/Ingest-RecommendationsToLogAnalytics.ps1 index a28957e0c..0648b6dcc 100644 --- a/src/optimization-engine/runbooks/recommendations/Ingest-RecommendationsToLogAnalytics.ps1 +++ b/src/optimization-engine/runbooks/recommendations/Ingest-RecommendationsToLogAnalytics.ps1 @@ -21,8 +21,7 @@ if ([string]::IsNullOrEmpty($sqldatabase)) { $sqldatabase = "azureoptimization" } -$workspaceId = Get-AutomationVariable -Name "AzureOptimization_LogAnalyticsWorkspaceId" -$sharedKey = Get-AutomationVariable -Name "AzureOptimization_LogAnalyticsWorkspaceKey" +$dceEndpoint = Get-AutomationVariable -Name "AzureOptimization_DCEIngestionEndpoint" $LogAnalyticsChunkSize = [int] (Get-AutomationVariable -Name "AzureOptimization_LogAnalyticsChunkSize" -ErrorAction SilentlyContinue) if (-not($LogAnalyticsChunkSize -gt 0)) { @@ -53,80 +52,18 @@ if (-not($StorageBlobsPageSize -gt 0)) #region Functions -# Function to create the authorization signature -function Build-OMSSignature ($workspaceId, $sharedKey, $date, $contentLength, $method, $contentType, $resource) +# Sends data to a Log Analytics custom table via the DCR-based Logs Ingestion API. +function Send-LogIngestionData($accessToken, $dceEndpoint, $dcrImmutableId, $streamName, $body) { - $xHeaders = "x-ms-date:" + $date - $stringToHash = $method + "`n" + $contentLength + "`n" + $contentType + "`n" + $xHeaders + "`n" + $resource - $bytesToHash = [Text.Encoding]::UTF8.GetBytes($stringToHash) - $keyBytes = [Convert]::FromBase64String($sharedKey) - $sha256 = New-Object System.Security.Cryptography.HMACSHA256 - $sha256.Key = $keyBytes - $calculatedHash = $sha256.ComputeHash($bytesToHash) - $encodedHash = [Convert]::ToBase64String($calculatedHash) - $authorization = 'SharedKey {0}:{1}' -f $workspaceId, $encodedHash - return $authorization -} - -# Function to create and post the request -function Post-OMSData($workspaceId, $sharedKey, $body, $logType, $TimeStampField, $AzureEnvironment) -{ - $method = "POST" - $contentType = "application/json" - $resource = "/api/logs" - $rfc1123date = [DateTime]::UtcNow.ToString("r") - $contentLength = $body.Length - $signature = Build-OMSSignature ` - -workspaceId $workspaceId ` - -sharedKey $sharedKey ` - -date $rfc1123date ` - -contentLength $contentLength ` - -method $method ` - -contentType $contentType ` - -resource $resource - - $uri = "https://" + $workspaceId + ".ods.opinsights.azure.com" + $resource + "?api-version=2016-04-01" - if ($AzureEnvironment -eq "AzureChinaCloud") - { - $uri = "https://" + $workspaceId + ".ods.opinsights.azure.cn" + $resource + "?api-version=2016-04-01" - } - if ($AzureEnvironment -eq "AzureUSGovernment") - { - $uri = "https://" + $workspaceId + ".ods.opinsights.azure.us" + $resource + "?api-version=2016-04-01" - } - if ($AzureEnvironment -eq "AzureGermanCloud") - { - throw "Azure Germany isn't supported for the Log Analytics Data Collector API" + $uri = "$dceEndpoint/dataCollectionRules/$dcrImmutableId/streams/$streamName`?api-version=2023-01-01" + $headers = @{ + "Authorization" = "Bearer $accessToken" + "Content-Type" = "application/json" } - - $OMSheaders = @{ - "Authorization" = $signature - "Log-Type" = $logType - "x-ms-date" = $rfc1123date - "time-generated-field" = $TimeStampField - } - - try - { - - $response = Invoke-WebRequest -Uri $uri -Method POST -ContentType $contentType -Headers $OMSheaders -Body $body -UseBasicParsing -TimeoutSec 1000 - } - catch - { - if ($_.Exception.Response.StatusCode.Value__ -eq 401) - { - "REAUTHENTICATING" - - $response = Invoke-WebRequest -Uri $uri -Method POST -ContentType $contentType -Headers $OMSheaders -Body $body -UseBasicParsing -TimeoutSec 1000 - } - else - { - return $_.Exception.Response.StatusCode.Value__ - } - } - + $response = Invoke-WebRequest -Uri $uri -Method POST -Headers $headers -Body $body -UseBasicParsing -TimeoutSec 1000 return $response.StatusCode } + #endregion Functions @@ -215,9 +152,25 @@ $controlRow = $controlRows[0] $lastProcessedLine = $controlRow.LastProcessedLine $lastProcessedDateTime = $controlRow.LastProcessedDateTime.ToString("yyyy'-'MM'-'dd'T'HH':'mm':'ss'.'fff'Z'") $LogAnalyticsSuffix = $controlRow.LogAnalyticsSuffix +$dcrImmutableId = $controlRow.DCRImmutableId $logname = $lognamePrefix + $LogAnalyticsSuffix +$streamName = "Custom-$lognamePrefix$LogAnalyticsSuffix" + +if ([string]::IsNullOrEmpty($dcrImmutableId)) +{ + throw "DCRImmutableId is not set for container $storageAccountSinkContainer. Run Setup-LogAnalyticsTablesAndDCRs.ps1 first." +} + +# Obtain a bearer token for the Logs Ingestion API +switch ($cloudEnvironment) +{ + "AzureChinaCloud" { $monitorAudience = "https://monitor.azure.cn/" } + "AzureUSGovernment" { $monitorAudience = "https://monitor.azure.us/" } + default { $monitorAudience = "https://monitor.azure.com/" } +} +$monitorToken = (Get-AzAccessToken -ResourceUrl $monitorAudience).Token -Write-Output "Processing blobs modified after $lastProcessedDateTime (line $lastProcessedLine) and ingesting them into the $($logname)_CL table..." +Write-Output "Processing blobs modified after $lastProcessedDateTime (line $lastProcessedLine) and ingesting them into the $($logname)_CL table (stream $streamName)..." $newProcessedTime = $null @@ -289,10 +242,16 @@ foreach ($blob in $unprocessedBlobs) $jsonObjectSplitted[$j][$i].RecommendationAction = $jsonObjectSplitted[$j][$i].RecommendationAction.Replace("'", "") $jsonObjectSplitted[$j][$i].AdditionalInfo = $jsonObjectSplitted[$j][$i].AdditionalInfo | ConvertTo-Json -Compress $jsonObjectSplitted[$j][$i].Tags = $jsonObjectSplitted[$j][$i].Tags | ConvertTo-Json -Compress + # Rename Timestamp to TimeGenerated as required by Log Analytics custom tables + $jsonObjectSplitted[$j][$i] | Add-Member -MemberType NoteProperty -Name 'TimeGenerated' -Value $jsonObjectSplitted[$j][$i].Timestamp -Force } $jsonObject = ConvertTo-Json -InputObject $jsonObjectSplitted[$j] - $res = Post-OMSData -workspaceId $workspaceId -sharedKey $sharedKey -body ([System.Text.Encoding]::UTF8.GetBytes($jsonObject)) -logType $logname -TimeStampField "Timestamp" -AzureEnvironment $cloudEnvironment + # Refresh token before each chunk upload + $monitorToken = (Get-AzAccessToken -ResourceUrl $monitorAudience).Token + $res = Send-LogIngestionData -accessToken $monitorToken -dceEndpoint $dceEndpoint ` + -dcrImmutableId $dcrImmutableId -streamName $streamName ` + -body ([System.Text.Encoding]::UTF8.GetBytes($jsonObject)) if ($res -ge 200 -and $res -lt 300) { Write-Output "Successfully uploaded $currentObjectLines $LogAnalyticsSuffix rows to Log Analytics" diff --git a/src/optimization-engine/runbooks/recommendations/Ingest-SuppressionsToLogAnalytics.ps1 b/src/optimization-engine/runbooks/recommendations/Ingest-SuppressionsToLogAnalytics.ps1 index db867ee93..291950137 100644 --- a/src/optimization-engine/runbooks/recommendations/Ingest-SuppressionsToLogAnalytics.ps1 +++ b/src/optimization-engine/runbooks/recommendations/Ingest-SuppressionsToLogAnalytics.ps1 @@ -6,8 +6,7 @@ if ([string]::IsNullOrEmpty($cloudEnvironment)) $cloudEnvironment = "AzureCloud" } -$workspaceId = Get-AutomationVariable -Name "AzureOptimization_LogAnalyticsWorkspaceId" -$sharedKey = Get-AutomationVariable -Name "AzureOptimization_LogAnalyticsWorkspaceKey" +$dceEndpoint = Get-AutomationVariable -Name "AzureOptimization_DCEIngestionEndpoint" $LogAnalyticsChunkSize = [int] (Get-AutomationVariable -Name "AzureOptimization_LogAnalyticsChunkSize" -ErrorAction SilentlyContinue) if (-not($LogAnalyticsChunkSize -gt 0)) { @@ -41,80 +40,18 @@ $FiltersTable = "Filters" #region Functions -# Function to create the authorization signature -function Build-OMSSignature ($workspaceId, $sharedKey, $date, $contentLength, $method, $contentType, $resource) +# Sends data to a Log Analytics custom table via the DCR-based Logs Ingestion API. +function Send-LogIngestionData($accessToken, $dceEndpoint, $dcrImmutableId, $streamName, $body) { - $xHeaders = "x-ms-date:" + $date - $stringToHash = $method + "`n" + $contentLength + "`n" + $contentType + "`n" + $xHeaders + "`n" + $resource - $bytesToHash = [Text.Encoding]::UTF8.GetBytes($stringToHash) - $keyBytes = [Convert]::FromBase64String($sharedKey) - $sha256 = New-Object System.Security.Cryptography.HMACSHA256 - $sha256.Key = $keyBytes - $calculatedHash = $sha256.ComputeHash($bytesToHash) - $encodedHash = [Convert]::ToBase64String($calculatedHash) - $authorization = 'SharedKey {0}:{1}' -f $workspaceId, $encodedHash - return $authorization -} - -# Function to create and post the request -function Post-OMSData($workspaceId, $sharedKey, $body, $logType, $TimeStampField, $AzureEnvironment) -{ - $method = "POST" - $contentType = "application/json" - $resource = "/api/logs" - $rfc1123date = [DateTime]::UtcNow.ToString("r") - $contentLength = $body.Length - $signature = Build-OMSSignature ` - -workspaceId $workspaceId ` - -sharedKey $sharedKey ` - -date $rfc1123date ` - -contentLength $contentLength ` - -method $method ` - -contentType $contentType ` - -resource $resource - - $uri = "https://" + $workspaceId + ".ods.opinsights.azure.com" + $resource + "?api-version=2016-04-01" - if ($AzureEnvironment -eq "AzureChinaCloud") - { - $uri = "https://" + $workspaceId + ".ods.opinsights.azure.cn" + $resource + "?api-version=2016-04-01" - } - if ($AzureEnvironment -eq "AzureUSGovernment") - { - $uri = "https://" + $workspaceId + ".ods.opinsights.azure.us" + $resource + "?api-version=2016-04-01" - } - if ($AzureEnvironment -eq "AzureGermanCloud") - { - throw "Azure Germany isn't supported for the Log Analytics Data Collector API" - } - - $OMSheaders = @{ - "Authorization" = $signature - "Log-Type" = $logType - "x-ms-date" = $rfc1123date - "time-generated-field" = $TimeStampField + $uri = "$dceEndpoint/dataCollectionRules/$dcrImmutableId/streams/$streamName`?api-version=2023-01-01" + $headers = @{ + "Authorization" = "Bearer $accessToken" + "Content-Type" = "application/json" } - - try - { - - $response = Invoke-WebRequest -Uri $uri -Method POST -ContentType $contentType -Headers $OMSheaders -Body $body -UseBasicParsing -TimeoutSec 1000 - } - catch - { - if ($_.Exception.Response.StatusCode.Value__ -eq 401) - { - "REAUTHENTICATING" - - $response = Invoke-WebRequest -Uri $uri -Method POST -ContentType $contentType -Headers $OMSheaders -Body $body -UseBasicParsing -TimeoutSec 1000 - } - else - { - return $_.Exception.Response.StatusCode.Value__ - } - } - + $response = Invoke-WebRequest -Uri $uri -Method POST -Headers $headers -Body $body -UseBasicParsing -TimeoutSec 1000 return $response.StatusCode } + #endregion Functions "Logging in to Azure with $authenticationOption..." @@ -137,6 +74,14 @@ switch ($authenticationOption) $cloudDetails = Get-AzEnvironment -Name $CloudEnvironment $azureSqlDomain = $cloudDetails.SqlDatabaseDnsSuffix.Substring(1) +# Determine the Logs Ingestion API audience URL for this cloud +switch ($cloudEnvironment) +{ + "AzureChinaCloud" { $monitorAudience = "https://monitor.azure.cn/" } + "AzureUSGovernment" { $monitorAudience = "https://monitor.azure.us/" } + default { $monitorAudience = "https://monitor.azure.com/" } +} + Write-Output "Getting excluded recommendation sub-type IDs..." $tries = 0 @@ -246,12 +191,53 @@ foreach ($filter in $filters) $filterObjects += $filterObject } -$filtersJson = $filterObjects | ConvertTo-Json +$filtersJson = $filterObjects | ForEach-Object { + $_ | Add-Member -MemberType NoteProperty -Name 'TimeGenerated' -Value $_.Timestamp -Force -PassThru +} | ConvertTo-Json $LogAnalyticsSuffix = "SuppressionsV1" $logname = $lognamePrefix + $LogAnalyticsSuffix +$streamName = "Custom-$lognamePrefix$LogAnalyticsSuffix" + +# Retrieve DCR immutable ID from SQL control table +$dcrImmutableId = $null +$tries = 0 +$dcrQuerySuccess = $false +do +{ + $tries++ + try + { + $dbToken = Get-AzAccessToken -ResourceUrl "https://$azureSqlDomain/" + $dcrConn = New-Object System.Data.SqlClient.SqlConnection("Server=tcp:$sqlserver,1433;Database=$sqldatabase;Encrypt=True;Connection Timeout=$SqlTimeout;") + $dcrConn.AccessToken = $dbToken.Token + $dcrConn.Open() + $dcrCmd = New-Object system.Data.SqlClient.SqlCommand + $dcrCmd.Connection = $dcrConn + $dcrCmd.CommandTimeout = $SqlTimeout + $dcrCmd.CommandText = "SELECT DCRImmutableId FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'suppressions'" + $dcrImmutableId = $dcrCmd.ExecuteScalar() + $dcrConn.Close() + $dcrConn.Dispose() + $dcrQuerySuccess = $true + } + catch + { + Write-Output "Failed to retrieve DCR immutable ID from SQL at try $tries." + Write-Output $Error[0] + Start-Sleep -Seconds ($tries * 20) + } +} while (-not($dcrQuerySuccess) -and $tries -lt 3) + +if ([string]::IsNullOrEmpty($dcrImmutableId)) +{ + throw "DCRImmutableId is not set for suppressions. Run Setup-LogAnalyticsTablesAndDCRs.ps1 first." +} -$res = Post-OMSData -workspaceId $workspaceId -sharedKey $sharedKey -body ([System.Text.Encoding]::UTF8.GetBytes($filtersJson)) -logType $logname -TimeStampField "Timestamp" -AzureEnvironment $cloudEnvironment +$monitorToken = (Get-AzAccessToken -ResourceUrl $monitorAudience).Token +$res = Send-LogIngestionData -accessToken $monitorToken -dceEndpoint $dceEndpoint ` + -dcrImmutableId $dcrImmutableId -streamName $streamName ` + -body ([System.Text.Encoding]::UTF8.GetBytes($filtersJson)) if ($res -ge 200 -and $res -lt 300) { Write-Output "Successfully uploaded $($filterObjects.Count) $LogAnalyticsSuffix rows to Log Analytics" diff --git a/src/optimization-engine/upgrade-manifest.json b/src/optimization-engine/upgrade-manifest.json index 457e0c712..de27e8cce 100644 --- a/src/optimization-engine/upgrade-manifest.json +++ b/src/optimization-engine/upgrade-manifest.json @@ -304,7 +304,7 @@ { "runbook": { "name": "runbooks/data-collection/Ingest-OptimizationCSVExportsToLogAnalytics.ps1", - "version": "1.6.2.0" + "version": "2.0.0.0" }, "source": "dataCollection" }, @@ -319,7 +319,7 @@ { "runbook": { "name": "runbooks/recommendations/Ingest-RecommendationsToLogAnalytics.ps1", - "version": "1.1.1.0" + "version": "2.0.0.0" }, "source": "recommendations", "schedule": "AzureOptimization_IngestRecommendationsWeekly" @@ -327,7 +327,7 @@ { "runbook": { "name": "runbooks/recommendations/Ingest-SuppressionsToLogAnalytics.ps1", - "version": "1.1.0.0" + "version": "2.0.0.0" }, "source": "recommendations", "schedule": "AzureOptimization_IngestSuppressionsWeekly" @@ -1052,7 +1052,7 @@ "overwriteVariables": [ { "name": "AzureOptimization_LogAnalyticsChunkSize", - "value": 6000 + "value": 150 } ] } \ No newline at end of file From 5d6b389ecab51305c53cd4eb8ca9c368ce2e2067 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?H=C3=A9lder=20Pinto?= Date: Tue, 7 Apr 2026 18:52:51 +0100 Subject: [PATCH 09/30] Bug fixing --- .../Deploy-AzureOptimizationEngine.ps1 | 40 +++--- .../Setup-LogAnalyticsTablesAndDCRs.ps1 | 116 ++++++++++-------- .../loganalyticsingestcontrol-initialize.sql | 52 ++++---- 3 files changed, 114 insertions(+), 94 deletions(-) diff --git a/src/optimization-engine/Deploy-AzureOptimizationEngine.ps1 b/src/optimization-engine/Deploy-AzureOptimizationEngine.ps1 index 244097c5e..536960ef1 100644 --- a/src/optimization-engine/Deploy-AzureOptimizationEngine.ps1 +++ b/src/optimization-engine/Deploy-AzureOptimizationEngine.ps1 @@ -1366,6 +1366,26 @@ if ("Y", "y" -contains $continueInput) } #endregion + #region DCR-based ingestion setup + Write-Host "Setting up Log Analytics custom tables and Data Collection Rules for DCR-based ingestion..." -ForegroundColor Green + try + { + .\Setup-LogAnalyticsTablesAndDCRs.ps1 ` + -ResourceGroupName $resourceGroupName ` + -AutomationAccountName $automationAccountName ` + -WorkspaceName $laWorkspaceName ` + -WorkspaceResourceGroupName $laWorkspaceResourceGroup ` + -SqlServerName $sqlServerName ` + -SqlDatabaseName $sqlDatabaseName ` + -CloudEnvironment $cloudEnvironment + } + catch + { + Write-Host "Could not complete the Log Analytics tables and DCR setup: $($_.Exception.Message)" -ForegroundColor Yellow + Write-Host "You can re-run Setup-LogAnalyticsTablesAndDCRs.ps1 manually after resolving the issue." -ForegroundColor Yellow + } + #endregion + #region Close SQL Server firewall rule if (-not($sqlServerName -like "*.database.*")) { @@ -1407,26 +1427,6 @@ if ("Y", "y" -contains $continueInput) } #endregion - #region DCR-based ingestion setup - Write-Host "Setting up Log Analytics custom tables and Data Collection Rules for DCR-based ingestion..." -ForegroundColor Green - try - { - .\Setup-LogAnalyticsTablesAndDCRs.ps1 ` - -ResourceGroupName $resourceGroupName ` - -AutomationAccountName $automationAccountName ` - -WorkspaceName $laWorkspaceName ` - -WorkspaceResourceGroupName $laWorkspaceResourceGroup ` - -SqlServerName $sqlServerName ` - -SqlDatabaseName $sqlDatabaseName ` - -CloudEnvironment $cloudEnvironment - } - catch - { - Write-Host "Could not complete the Log Analytics tables and DCR setup: $($_.Exception.Message)" -ForegroundColor Yellow - Write-Host "You can re-run Setup-LogAnalyticsTablesAndDCRs.ps1 manually after resolving the issue." -ForegroundColor Yellow - } - #endregion - if (!$silentDeploy) { #region Grant Microsoft Entra ID role to AOE principal diff --git a/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 b/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 index 9c0597653..7af044dc5 100644 --- a/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 +++ b/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 @@ -1,4 +1,4 @@ -īģŋ<# +<# .SYNOPSIS Creates or updates the custom Log Analytics tables and Data Collection Rules (DCRs) required for DCR-based ingestion in the Azure Optimization Engine. @@ -80,10 +80,7 @@ param ( [string] $SqlDatabaseName, [Parameter(Mandatory = $false)] - [string] $CloudEnvironment = "AzureCloud", - - [Parameter(Mandatory = $false)] - [string] $AzureEnvironment + [string] $CloudEnvironment = "AzureCloud" ) $ErrorActionPreference = "Stop" @@ -100,13 +97,6 @@ if ([string]::IsNullOrEmpty($WorkspaceResourceGroupName)) } #region Resolve Monitor audience URL per cloud -switch ($CloudEnvironment) -{ - "AzureChinaCloud" { $monitorAudience = "https://monitor.azure.cn/" } - "AzureUSGovernment" { $monitorAudience = "https://monitor.azure.us/" } - default { $monitorAudience = "https://monitor.azure.com/" } -} - switch ($CloudEnvironment) { "AzureChinaCloud" { $armEndpoint = "https://management.chinacloudapi.cn" } @@ -152,14 +142,14 @@ $dceLogsIngestionEndpoint = $dceVar.Value Write-Host "DCE Logs Ingestion endpoint: $dceLogsIngestionEndpoint" -ForegroundColor Cyan # Discover the DCE resource ID by searching for DCEs in the resource group whose logsIngestion endpoint matches -$dceListUri = "$armEndpoint/subscriptions/$currentSubscriptionId/resourceGroups/$ResourceGroupName/providers/Microsoft.Insights/dataCollectionEndpoints?api-version=$dcrApiVersion" -$dceListResponse = Invoke-RestMethod -Method GET -Uri $dceListUri -Headers $armHeaders -$matchedDce = $dceListResponse.value | Where-Object { $_.properties.logsIngestion.endpoint -eq $dceLogsIngestionEndpoint } + +$dceListResponse = Get-AzDataCollectionEndpoint -ResourceGroupName $ResourceGroupName -ErrorAction SilentlyContinue +$matchedDce = $dceListResponse | Where-Object { $_.LogIngestionEndpoint -eq $dceLogsIngestionEndpoint } if ($null -eq $matchedDce) { throw "Could not find a Data Collection Endpoint in resource group $ResourceGroupName with logsIngestion endpoint '$dceLogsIngestionEndpoint'." } -$dceResourceId = $matchedDce.id +$dceResourceId = $matchedDce.Id Write-Host "DCE resource ID: $dceResourceId" -ForegroundColor Cyan #endregion @@ -301,6 +291,7 @@ $tableSchemas = @{ @{ name = "RecommendationText_s"; type = "string" } @{ name = "RecommendationTypeId_g"; type = "string" } @{ name = "InstanceId_s"; type = "string" } + @{ name = "InstanceName_g"; type = "string" } @{ name = "InstanceName_s"; type = "string" } @{ name = "Tags_s"; type = "string" } @{ name = "AdditionalInfo_s"; type = "string" } @@ -474,6 +465,7 @@ $tableSchemas = @{ @{ name = "TenantGuid_g"; type = "string" } @{ name = "Cloud_s"; type = "string" } @{ name = "Model_s"; type = "string" } + @{ name = "PrincipalId_g"; type = "string" } @{ name = "PrincipalId_s"; type = "string" } @{ name = "Scope_s"; type = "string" } @{ name = "RoleDefinition_s"; type = "string" } @@ -897,31 +889,24 @@ $containerSuffixMap = @{ } #endregion -#region Get ARM access token (used for Log Analytics Tables API and DCR API) -$tableApiVersion = "2023-01-01-preview" $dcrApiVersion = "2022-06-01" $lognamePrefix = "AzureOptimization" - -$armToken = (Get-AzAccessToken -ResourceUrl "$armEndpoint/").Token -$armHeaders = @{ - "Authorization" = "Bearer $armToken" - "Content-Type" = "application/json" -} -#endregion +$tableSchemaApiVersion = "2023-01-01-preview" # Tables API (GET/PUT schema) +$tableMigrateApiVersion = "2021-12-01-preview" # Tables migrate endpoint #region Get Log Analytics workspace resource ID in the target subscription context if ($WorkspaceSubscriptionId -ne $currentSubscriptionId) { Write-Host "Switching to workspace subscription $WorkspaceSubscriptionId..." -ForegroundColor Cyan Set-AzContext -SubscriptionId $WorkspaceSubscriptionId | Out-Null - $armToken = (Get-AzAccessToken -ResourceUrl "$armEndpoint/").Token - $armHeaders["Authorization"] = "Bearer $armToken" } #endregion #region Create/update custom Log Analytics tables and DCRs Write-Host "Processing $($tableSchemas.Keys.Count) custom Log Analytics tables and DCRs..." -ForegroundColor Green +$existingTables = Get-AzOperationalInsightsTable -ResourceGroupName $WorkspaceResourceGroupName -WorkspaceName $WorkspaceName + $dcrSuffixToImmutableId = @{} foreach ($suffix in $tableSchemas.Keys) @@ -948,32 +933,62 @@ foreach ($suffix in $tableSchemas.Keys) } } | ConvertTo-Json -Depth 10 - $tableUri = "$armEndpoint$workspaceResourceId/tables/$tableName`?api-version=$tableApiVersion" - - # Check if the table already exists and whether it is a classic (Data Collector API) table. - # Classic tables cannot be updated via PUT directly; they must first be migrated to DCR-based. $existingTable = $null try { - $existingTable = Invoke-RestMethod -Method GET -Uri $tableUri -Headers $armHeaders -ErrorAction Stop + Write-Host " Checking if table $tableName exists and its type..." -ForegroundColor Yellow + $existingTable = $existingTables | Where-Object { $_.Name -eq $tableName } } catch { # Table does not yet exist; the PUT below will create it as a DCR-based table. } - if ($null -ne $existingTable -and $existingTable.properties.schema.tableSubType -eq "ClassicAPI") + if ($null -ne $existingTable -and $existingTable.Schema.TableSubType -eq "Classic") { # Migrate the classic table to DCR-based before updating the schema. # This call is idempotent; it has no effect if the table has already been converted. Write-Host " Table $tableName is a classic (Data Collector API) table. Migrating to DCR-based..." -ForegroundColor Yellow - $migrateUri = "$armEndpoint$workspaceResourceId/tables/$tableName/migrate?api-version=2021-12-01-preview" - Invoke-RestMethod -Method POST -Uri $migrateUri -Headers $armHeaders -Body "{}" | Out-Null + $migrateUri = "$armEndpoint$workspaceResourceId/tables/$tableName/migrate?api-version=$tableMigrateApiVersion" + Invoke-AzRestMethod -Method POST -Uri $migrateUri -Payload "{}" | Out-Null Write-Host " Migration completed for $tableName." -ForegroundColor Gray } - $tableResponse = Invoke-RestMethod -Method PUT -Uri $tableUri -Headers $armHeaders -Body $tablePayload - Write-Host " Table $tableName created/updated (subType: $($tableResponse.properties.schema.tableSubType))" -ForegroundColor Gray + # Build the DCR transform KQL. DCR stream declarations do not support the 'guid' type, + # so _g columns are declared as 'string' in the stream. For existing tables whose _g columns + # are already typed as Guid (created by the legacy Data Collector API), we must cast each + # such column back to guid inside the transform so the output matches the table schema. + # For new tables (where we create the schema with 'string' for _g columns) no casting is needed. + $transformKql = "source" + if ($null -ne $existingTable) + { + $guidColumns = @($existingTable.Schema.Columns | Where-Object { $_.Type -eq "Guid" }) + if ($guidColumns.Count -gt 0) + { + $extends = ($guidColumns | ForEach-Object { "$($_.Name) = toguid($($_.Name))" }) -join ", " + $transformKql = "source | extend $extends" + Write-Host " Existing guid columns detected ($($guidColumns.Count)). Using toguid() transform." -ForegroundColor Gray + } + } + + # Only create/update the table schema for new (non-existing) tables. + # Existing tables preserve their schema to avoid column type conflicts + # (e.g. Guid columns cannot be changed to string). + $tablePath = "/subscriptions/$WorkspaceSubscriptionId/resourceGroups/$WorkspaceResourceGroupName/providers/Microsoft.OperationalInsights/workspaces/$WorkspaceName/tables/$tableName`?api-version=$tableSchemaApiVersion" + if ($null -eq $existingTable) + { + $tableResponse = Invoke-AzRestMethod -Method PUT -Path $tablePath -Payload $tablePayload + Write-Host " Table $tableName created with status code $($tableResponse.StatusCode)." -ForegroundColor Gray + if ($tableResponse.StatusCode -ne 200 -and $tableResponse.StatusCode -ne 201) + { + $errorMessage = $tableResponse.Content | ConvertFrom-Json | Select-Object -ExpandProperty error | Select-Object -ExpandProperty message + throw " Failed to create table ${tableName}: ${errorMessage}" + } + } + else + { + Write-Host " Table $tableName already exists, skipping schema update." -ForegroundColor Gray + } #endregion #region Create DCR for this table @@ -1000,7 +1015,7 @@ foreach ($suffix in $tableSchemas.Keys) @{ streams = @($streamName) destinations = @("laDest") - transformKql = "source" + transformKql = $transformKql outputStream = "Custom-$tableName" } ) @@ -1011,14 +1026,23 @@ foreach ($suffix in $tableSchemas.Keys) if ($WorkspaceSubscriptionId -ne $currentSubscriptionId) { Set-AzContext -SubscriptionId $currentSubscriptionId | Out-Null - $armToken = (Get-AzAccessToken -ResourceUrl "$armEndpoint/").Token - $armHeaders["Authorization"] = "Bearer $armToken" } $dcrUri = "$armEndpoint/subscriptions/$currentSubscriptionId/resourceGroups/$ResourceGroupName/providers/Microsoft.Insights/dataCollectionRules/$dcrName`?api-version=$dcrApiVersion" - $dcrResponse = Invoke-RestMethod -Method PUT -Uri $dcrUri -Headers $armHeaders -Body $dcrPayload - $dcrImmutableId = $dcrResponse.properties.immutableId - $dcrResourceId = $dcrResponse.id + $dcrResponse = Invoke-AzRestMethod -Method PUT -Uri $dcrUri -Payload $dcrPayload + Write-Host " DCR $dcrName creation request completed with status code $($dcrResponse.StatusCode)." -ForegroundColor Gray + if ($dcrResponse.StatusCode -ne 200 -and $dcrResponse.StatusCode -ne 201) + { + $errorMessage = $dcrResponse.Content | ConvertFrom-Json | Select-Object -ExpandProperty error | Select-Object -ExpandProperty message + throw " Failed to create DCR ${dcrName}: ${errorMessage}" + } + else + { + $dcrResponseContent = $dcrResponse.Content | ConvertFrom-Json + } + Write-Host " DCR $dcrName created." -ForegroundColor Gray + $dcrImmutableId = $dcrResponseContent.properties.immutableId + $dcrResourceId = $dcrResponseContent.id $dcrSuffixToImmutableId[$suffix] = $dcrImmutableId Write-Host " DCR immutable ID: $dcrImmutableId" -ForegroundColor Gray @@ -1026,8 +1050,6 @@ foreach ($suffix in $tableSchemas.Keys) if ($WorkspaceSubscriptionId -ne $currentSubscriptionId) { Set-AzContext -SubscriptionId $WorkspaceSubscriptionId | Out-Null - $armToken = (Get-AzAccessToken -ResourceUrl "$armEndpoint/").Token - $armHeaders["Authorization"] = "Bearer $armToken" } #endregion @@ -1059,8 +1081,6 @@ foreach ($suffix in $tableSchemas.Keys) if ($WorkspaceSubscriptionId -ne $currentSubscriptionId) { Set-AzContext -SubscriptionId $WorkspaceSubscriptionId | Out-Null - $armToken = (Get-AzAccessToken -ResourceUrl "$armEndpoint/").Token - $armHeaders["Authorization"] = "Bearer $armToken" } #endregion } @@ -1076,7 +1096,7 @@ if ($WorkspaceSubscriptionId -ne $currentSubscriptionId) #region Update LogAnalyticsIngestControl SQL table with DCR immutable IDs Write-Host "Updating SQL LogAnalyticsIngestControl with DCR immutable IDs..." -ForegroundColor Green -$sqlToken = (Get-AzAccessToken -ResourceUrl "https://database.windows.net/").Token +$sqlToken = (Get-AzAccessToken -ResourceUrl "https://database.windows.net/").Token | ConvertFrom-SecureString -AsPlainText $sqlConnectionString = "Server=tcp:$SqlServerName,1433;Initial Catalog=$SqlDatabaseName;Persist Security Info=False;MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Authentication=Active Directory Default;" try diff --git a/src/optimization-engine/model/loganalyticsingestcontrol-initialize.sql b/src/optimization-engine/model/loganalyticsingestcontrol-initialize.sql index b8b931c0c..f84933e98 100644 --- a/src/optimization-engine/model/loganalyticsingestcontrol-initialize.sql +++ b/src/optimization-engine/model/loganalyticsingestcontrol-initialize.sql @@ -1,155 +1,155 @@ IF NOT EXISTS (SELECT * FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'argvmexports') BEGIN - INSERT INTO [dbo].[LogAnalyticsIngestControl] + INSERT INTO [dbo].[LogAnalyticsIngestControl] (StorageContainerName, LastProcessedDateTime, LastProcessedLine, LogAnalyticsSuffix, CollectedType) VALUES ('argvmexports', '1901-01-01T00:00:00Z', -1, 'VMsV1', 'ARGVirtualMachine') END IF NOT EXISTS (SELECT * FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'argdiskexports') BEGIN - INSERT INTO [dbo].[LogAnalyticsIngestControl] + INSERT INTO [dbo].[LogAnalyticsIngestControl] (StorageContainerName, LastProcessedDateTime, LastProcessedLine, LogAnalyticsSuffix, CollectedType) VALUES ('argdiskexports', '1901-01-01T00:00:00Z', -1, 'DisksV1', 'ARGManagedDisk') END IF NOT EXISTS (SELECT * FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'argvhdexports') BEGIN - INSERT INTO [dbo].[LogAnalyticsIngestControl] + INSERT INTO [dbo].[LogAnalyticsIngestControl] (StorageContainerName, LastProcessedDateTime, LastProcessedLine, LogAnalyticsSuffix, CollectedType) VALUES ('argvhdexports', '1901-01-01T00:00:00Z', -1, 'VhdDisksV1', 'ARGUnmanagedDisk') END IF NOT EXISTS (SELECT * FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'argavailsetexports') BEGIN - INSERT INTO [dbo].[LogAnalyticsIngestControl] + INSERT INTO [dbo].[LogAnalyticsIngestControl] (StorageContainerName, LastProcessedDateTime, LastProcessedLine, LogAnalyticsSuffix, CollectedType) VALUES ('argavailsetexports', '1901-01-01T00:00:00Z', -1, 'AvailSetsV1', 'ARGAvailabilitySet') END IF NOT EXISTS (SELECT * FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'advisorexports') BEGIN - INSERT INTO [dbo].[LogAnalyticsIngestControl] + INSERT INTO [dbo].[LogAnalyticsIngestControl] (StorageContainerName, LastProcessedDateTime, LastProcessedLine, LogAnalyticsSuffix, CollectedType) VALUES ('advisorexports', '1901-01-01T00:00:00Z', -1, 'AdvisorV1', 'AzureAdvisor') END IF NOT EXISTS (SELECT * FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'remediationlogs') BEGIN - INSERT INTO [dbo].[LogAnalyticsIngestControl] + INSERT INTO [dbo].[LogAnalyticsIngestControl] (StorageContainerName, LastProcessedDateTime, LastProcessedLine, LogAnalyticsSuffix, CollectedType) VALUES ('remediationlogs', '1901-01-01T00:00:00Z', -1, 'RemediationV1', 'RemediationLogs') END IF NOT EXISTS (SELECT * FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'consumptionexports') BEGIN - INSERT INTO [dbo].[LogAnalyticsIngestControl] + INSERT INTO [dbo].[LogAnalyticsIngestControl] (StorageContainerName, LastProcessedDateTime, LastProcessedLine, LogAnalyticsSuffix, CollectedType) VALUES ('consumptionexports', '1901-01-01T00:00:00Z', -1, 'ConsumptionV1', 'AzureConsumption') END IF NOT EXISTS (SELECT * FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'aadobjectsexports') BEGIN - INSERT INTO [dbo].[LogAnalyticsIngestControl] + INSERT INTO [dbo].[LogAnalyticsIngestControl] (StorageContainerName, LastProcessedDateTime, LastProcessedLine, LogAnalyticsSuffix, CollectedType) VALUES ('aadobjectsexports', '1901-01-01T00:00:00Z', -1, 'AADObjectsV1', 'AADObjects') END IF NOT EXISTS (SELECT * FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'arglbexports') BEGIN - INSERT INTO [dbo].[LogAnalyticsIngestControl] + INSERT INTO [dbo].[LogAnalyticsIngestControl] (StorageContainerName, LastProcessedDateTime, LastProcessedLine, LogAnalyticsSuffix, CollectedType) VALUES ('arglbexports', '1901-01-01T00:00:00Z', -1, 'LoadBalancersV1', 'ARGLoadBalancer') END IF NOT EXISTS (SELECT * FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'argappgwexports') BEGIN - INSERT INTO [dbo].[LogAnalyticsIngestControl] + INSERT INTO [dbo].[LogAnalyticsIngestControl] (StorageContainerName, LastProcessedDateTime, LastProcessedLine, LogAnalyticsSuffix, CollectedType) VALUES ('argappgwexports', '1901-01-01T00:00:00Z', -1, 'AppGatewaysV1', 'ARGAppGateway') END IF NOT EXISTS (SELECT * FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'argrescontainersexports') BEGIN - INSERT INTO [dbo].[LogAnalyticsIngestControl] + INSERT INTO [dbo].[LogAnalyticsIngestControl] (StorageContainerName, LastProcessedDateTime, LastProcessedLine, LogAnalyticsSuffix, CollectedType) VALUES ('argrescontainersexports', '1901-01-01T00:00:00Z', -1, 'ResourceContainersV1', 'ARGResourceContainers') END IF NOT EXISTS (SELECT * FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'rbacexports') BEGIN - INSERT INTO [dbo].[LogAnalyticsIngestControl] + INSERT INTO [dbo].[LogAnalyticsIngestControl] (StorageContainerName, LastProcessedDateTime, LastProcessedLine, LogAnalyticsSuffix, CollectedType) VALUES ('rbacexports', '1901-01-01T00:00:00Z', -1, 'RBACAssignmentsV1', 'RBACAssignments') END IF NOT EXISTS (SELECT * FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'argvnetexports') BEGIN - INSERT INTO [dbo].[LogAnalyticsIngestControl] + INSERT INTO [dbo].[LogAnalyticsIngestControl] (StorageContainerName, LastProcessedDateTime, LastProcessedLine, LogAnalyticsSuffix, CollectedType) VALUES ('argvnetexports', '1901-01-01T00:00:00Z', -1, 'VNetsV1', 'ARGVirtualNetwork') END IF NOT EXISTS (SELECT * FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'argnicexports') BEGIN - INSERT INTO [dbo].[LogAnalyticsIngestControl] + INSERT INTO [dbo].[LogAnalyticsIngestControl] (StorageContainerName, LastProcessedDateTime, LastProcessedLine, LogAnalyticsSuffix, CollectedType) VALUES ('argnicexports', '1901-01-01T00:00:00Z', -1, 'NICsV1', 'ARGNetworkInterface') END IF NOT EXISTS (SELECT * FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'argnsgexports') BEGIN - INSERT INTO [dbo].[LogAnalyticsIngestControl] + INSERT INTO [dbo].[LogAnalyticsIngestControl] (StorageContainerName, LastProcessedDateTime, LastProcessedLine, LogAnalyticsSuffix, CollectedType) VALUES ('argnsgexports', '1901-01-01T00:00:00Z', -1, 'NSGsV1', 'ARGNSGRule') END IF NOT EXISTS (SELECT * FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'argpublicipexports') BEGIN - INSERT INTO [dbo].[LogAnalyticsIngestControl] + INSERT INTO [dbo].[LogAnalyticsIngestControl] (StorageContainerName, LastProcessedDateTime, LastProcessedLine, LogAnalyticsSuffix, CollectedType) VALUES ('argpublicipexports', '1901-01-01T00:00:00Z', -1, 'PublicIPsV1', 'ARGPublicIP') END IF NOT EXISTS (SELECT * FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'argvmssexports') BEGIN - INSERT INTO [dbo].[LogAnalyticsIngestControl] + INSERT INTO [dbo].[LogAnalyticsIngestControl] (StorageContainerName, LastProcessedDateTime, LastProcessedLine, LogAnalyticsSuffix, CollectedType) VALUES ('argvmssexports', '1901-01-01T00:00:00Z', -1, 'VMSSV1', 'ARGVMSS') END IF NOT EXISTS (SELECT * FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'argsqldbexports') BEGIN - INSERT INTO [dbo].[LogAnalyticsIngestControl] + INSERT INTO [dbo].[LogAnalyticsIngestControl] (StorageContainerName, LastProcessedDateTime, LastProcessedLine, LogAnalyticsSuffix, CollectedType) VALUES ('argsqldbexports', '1901-01-01T00:00:00Z', -1, 'SqlDbV1', 'ARGSqlDb') END IF NOT EXISTS (SELECT * FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'azmonitorexports') BEGIN - INSERT INTO [dbo].[LogAnalyticsIngestControl] + INSERT INTO [dbo].[LogAnalyticsIngestControl] (StorageContainerName, LastProcessedDateTime, LastProcessedLine, LogAnalyticsSuffix, CollectedType) VALUES ('azmonitorexports', '1901-01-01T00:00:00Z', -1, 'MonitorMetricsV1', 'MonitorMetrics') END IF NOT EXISTS (SELECT * FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'policystateexports') BEGIN - INSERT INTO [dbo].[LogAnalyticsIngestControl] + INSERT INTO [dbo].[LogAnalyticsIngestControl] (StorageContainerName, LastProcessedDateTime, LastProcessedLine, LogAnalyticsSuffix, CollectedType) VALUES ('policystateexports', '1901-01-01T00:00:00Z', -1, 'PolicyStatesV1', 'PolicyStates') END IF NOT EXISTS (SELECT * FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'recommendationsexports') BEGIN - INSERT INTO [dbo].[LogAnalyticsIngestControl] + INSERT INTO [dbo].[LogAnalyticsIngestControl] (StorageContainerName, LastProcessedDateTime, LastProcessedLine, LogAnalyticsSuffix, CollectedType) VALUES ('recommendationsexports', '2022-12-26T00:00:00Z', -1, 'RecommendationsV1', 'Recommendations') END IF NOT EXISTS (SELECT * FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'reservationsexports') BEGIN - INSERT INTO [dbo].[LogAnalyticsIngestControl] + INSERT INTO [dbo].[LogAnalyticsIngestControl] (StorageContainerName, LastProcessedDateTime, LastProcessedLine, LogAnalyticsSuffix, CollectedType) VALUES ('reservationsexports', '1901-01-01T00:00:00Z', -1, 'ReservationsUsageV1', 'ReservationsUsage') END IF NOT EXISTS (SELECT * FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'argappserviceplanexports') BEGIN - INSERT INTO [dbo].[LogAnalyticsIngestControl] + INSERT INTO [dbo].[LogAnalyticsIngestControl] (StorageContainerName, LastProcessedDateTime, LastProcessedLine, LogAnalyticsSuffix, CollectedType) VALUES ('argappserviceplanexports', '1901-01-01T00:00:00Z', -1, 'AppServicePlansV1', 'AppServicePlans') END IF NOT EXISTS (SELECT * FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'pricesheetexports') BEGIN - INSERT INTO [dbo].[LogAnalyticsIngestControl] + INSERT INTO [dbo].[LogAnalyticsIngestControl] (StorageContainerName, LastProcessedDateTime, LastProcessedLine, LogAnalyticsSuffix, CollectedType) VALUES ('pricesheetexports', '1901-01-01T00:00:00Z', -1, 'PricesheetV1', 'Pricesheet') END IF NOT EXISTS (SELECT * FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'reservationspriceexports') BEGIN - INSERT INTO [dbo].[LogAnalyticsIngestControl] + INSERT INTO [dbo].[LogAnalyticsIngestControl] (StorageContainerName, LastProcessedDateTime, LastProcessedLine, LogAnalyticsSuffix, CollectedType) VALUES ('reservationspriceexports', '1901-01-01T00:00:00Z', -1, 'ReservationsPriceV1', 'ReservationsPrice') END IF NOT EXISTS (SELECT * FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'savingsplansexports') BEGIN - INSERT INTO [dbo].[LogAnalyticsIngestControl] + INSERT INTO [dbo].[LogAnalyticsIngestControl] (StorageContainerName, LastProcessedDateTime, LastProcessedLine, LogAnalyticsSuffix, CollectedType) VALUES ('savingsplansexports', '1901-01-01T00:00:00Z', -1, 'SavingsPlansUsageV1', 'SavingsPlansUsage') END From d8aea1a8e0c55509ecfe4fafe9ce40139140f5ab Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?H=C3=A9lder=20Pinto?= Date: Tue, 7 Apr 2026 20:05:49 +0100 Subject: [PATCH 10/30] Solved schema and database connection issues --- .../Setup-LogAnalyticsTablesAndDCRs.ps1 | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 b/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 index 7af044dc5..c7d24aeaf 100644 --- a/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 +++ b/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 @@ -962,7 +962,7 @@ foreach ($suffix in $tableSchemas.Keys) $transformKql = "source" if ($null -ne $existingTable) { - $guidColumns = @($existingTable.Schema.Columns | Where-Object { $_.Type -eq "Guid" }) + $guidColumns = @($existingTable.Schema.Columns | Where-Object { $_.Type -eq "Guid" -and $_.Name -in $columns.name }) if ($guidColumns.Count -gt 0) { $extends = ($guidColumns | ForEach-Object { "$($_.Name) = toguid($($_.Name))" }) -join ", " @@ -1096,8 +1096,11 @@ if ($WorkspaceSubscriptionId -ne $currentSubscriptionId) #region Update LogAnalyticsIngestControl SQL table with DCR immutable IDs Write-Host "Updating SQL LogAnalyticsIngestControl with DCR immutable IDs..." -ForegroundColor Green -$sqlToken = (Get-AzAccessToken -ResourceUrl "https://database.windows.net/").Token | ConvertFrom-SecureString -AsPlainText -$sqlConnectionString = "Server=tcp:$SqlServerName,1433;Initial Catalog=$SqlDatabaseName;Persist Security Info=False;MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Authentication=Active Directory Default;" +$cloudDetails = Get-AzEnvironment -Name $CloudEnvironment +$azureSqlDomain = $cloudDetails.SqlDatabaseDnsSuffix.Substring(1) +$sqlToken = (Get-AzAccessToken -ResourceUrl "https://$azureSqlDomain/" -AsSecureString).Token | ConvertFrom-SecureString -AsPlainText + +$sqlConnectionString = "Server=tcp:$SqlServerName,1433;Database=$SqlDatabaseName;Encrypt=True;Connection Timeout=$SqlTimeout;" try { @@ -1127,14 +1130,14 @@ try $suppImmutableId = $dcrSuffixToImmutableId["SuppressionsV1"] $sqlCommand = $sqlConnection.CreateCommand() $sqlCommand.CommandText = @" -IF NOT EXISTS (SELECT 1 FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'suppressions') +IF NOT EXISTS (select 1 FROM [dbo].[LogAnalyticsIngestControl] WHERE StorageContainerName = 'suppressions') BEGIN - INSERT INTO [dbo].[LogAnalyticsIngestControl] (StorageContainerName, LastProcessedDateTime, LastProcessedLine, LogAnalyticsSuffix, CollectedType, DCRImmutableId) - VALUES ('suppressions', '1901-01-01T00:00:00Z', -1, 'SuppressionsV1', 'Suppressions', @id) +INSERT INTO [dbo].[LogAnalyticsIngestControl] (StorageContainerName, LastProcessedDateTime, LastProcessedLine, LogAnalyticsSuffix, CollectedType, DCRImmutableId) +VALUES ('suppressions', '1901-01-01T00:00:00Z', -1, 'SuppressionsV1', 'Suppressions', @id) END ELSE BEGIN - UPDATE [dbo].[LogAnalyticsIngestControl] SET DCRImmutableId = @id WHERE StorageContainerName = 'suppressions' +UPDATE [dbo].[LogAnalyticsIngestControl] SET DCRImmutableId = @id WHERE StorageContainerName = 'suppressions' END "@ $sqlCommand.Parameters.AddWithValue("@id", $suppImmutableId) | Out-Null From 6b4d56e5f3e1f1100ff6e1c295224242b8176ad7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?H=C3=A9lder=20Pinto?= Date: Tue, 12 May 2026 17:58:50 +0100 Subject: [PATCH 11/30] Fixed CSV vs. Log Analytics schema mismatch --- .../Setup-LogAnalyticsTablesAndDCRs.ps1 | 56 ++++++++++++++++--- 1 file changed, 47 insertions(+), 9 deletions(-) diff --git a/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 b/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 index c7d24aeaf..21d70ac47 100644 --- a/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 +++ b/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 @@ -924,6 +924,29 @@ foreach ($suffix in $tableSchemas.Keys) $columns += @{ name = $col.name; type = $col.type } } + # Build source stream columns from table schema columns by removing the legacy type suffix + # (_s, _g, _t, _d, _b). This keeps table column names backward-compatible while accepting + # CSV/source payload fields without suffixes. + $sourceColumnsByName = [ordered]@{} + $destinationToSource = @{} + foreach ($col in $tableSchemas[$suffix]) + { + $sourceName = $col.name + if ($sourceName -match '^(.*)_[sgtdb]$') + { + $sourceName = $Matches[1] + } + + if (-not $sourceColumnsByName.Contains($sourceName)) + { + $sourceColumnsByName[$sourceName] = @{ name = $sourceName; type = $col.type } + } + + $destinationToSource[$col.name] = $sourceName + } + + $sourceColumns = @($sourceColumnsByName.Values) + $tablePayload = @{ properties = @{ schema = @{ @@ -954,23 +977,38 @@ foreach ($suffix in $tableSchemas.Keys) Write-Host " Migration completed for $tableName." -ForegroundColor Gray } - # Build the DCR transform KQL. DCR stream declarations do not support the 'guid' type, - # so _g columns are declared as 'string' in the stream. For existing tables whose _g columns - # are already typed as Guid (created by the legacy Data Collector API), we must cast each - # such column back to guid inside the transform so the output matches the table schema. - # For new tables (where we create the schema with 'string' for _g columns) no casting is needed. - $transformKql = "source" + # Build the DCR transform KQL. + # Stream declarations use unsuffixed source fields, while output maps to suffixed table columns. + # For existing tables whose _g columns are already typed as Guid (created by the legacy Data + # Collector API), cast those specific outputs with toguid() so the transformed output matches + # the existing table schema. + $guidColumnNames = @() if ($null -ne $existingTable) { $guidColumns = @($existingTable.Schema.Columns | Where-Object { $_.Type -eq "Guid" -and $_.Name -in $columns.name }) if ($guidColumns.Count -gt 0) { - $extends = ($guidColumns | ForEach-Object { "$($_.Name) = toguid($($_.Name))" }) -join ", " - $transformKql = "source | extend $extends" + $guidColumnNames = @($guidColumns | ForEach-Object { $_.Name }) Write-Host " Existing guid columns detected ($($guidColumns.Count)). Using toguid() transform." -ForegroundColor Gray } } + $projectExpressions = @() + foreach ($col in $columns) + { + $destinationName = $col.name + $sourceName = $destinationToSource[$destinationName] + if ($destinationName -in $guidColumnNames) + { + $projectExpressions += "$destinationName = toguid($sourceName)" + } + else + { + $projectExpressions += "$destinationName = $sourceName" + } + } + $transformKql = "source | project " + ($projectExpressions -join ", ") + # Only create/update the table schema for new (non-existing) tables. # Existing tables preserve their schema to avoid column type conflicts # (e.g. Guid columns cannot be changed to string). @@ -1000,7 +1038,7 @@ foreach ($suffix in $tableSchemas.Keys) dataCollectionEndpointId = $dceResourceId streamDeclarations = @{ $streamName = @{ - columns = $columns + columns = $sourceColumns } } destinations = @{ From 4c8c6c18423363b733ac7df60e343fd06f2ef585 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?H=C3=A9lder=20Pinto?= Date: Wed, 13 May 2026 15:58:59 +0100 Subject: [PATCH 12/30] Reverted real columns to string --- .../Setup-LogAnalyticsTablesAndDCRs.ps1 | 152 +++++++++--------- 1 file changed, 76 insertions(+), 76 deletions(-) diff --git a/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 b/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 index 21d70ac47..a9994b2ad 100644 --- a/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 +++ b/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 @@ -193,12 +193,12 @@ $tableSchemas = @{ @{ name = "DeploymentModel_s"; type = "string" } @{ name = "InstanceId_s"; type = "string" } @{ name = "VMSize_s"; type = "string" } - @{ name = "CoresCount_d"; type = "real" } - @{ name = "MemoryMB_d"; type = "real" } + @{ name = "CoresCount_s"; type = "string" } + @{ name = "MemoryMB_s"; type = "string" } @{ name = "OSType_s"; type = "string" } @{ name = "LicenseType_s"; type = "string" } - @{ name = "DataDiskCount_d"; type = "real" } - @{ name = "NicCount_d"; type = "real" } + @{ name = "DataDiskCount_s"; type = "string" } + @{ name = "NicCount_s"; type = "string" } @{ name = "UsesManagedDisks_s"; type = "string" } @{ name = "AvailabilitySetId_s"; type = "string" } @{ name = "BootDiagnosticsEnabled_s"; type = "string" } @@ -229,14 +229,14 @@ $tableSchemas = @{ @{ name = "DeploymentModel_s"; type = "string" } @{ name = "DiskType_s"; type = "string" } @{ name = "TimeCreated_s"; type = "string" } - @{ name = "DiskIOPS_d"; type = "real" } - @{ name = "DiskThroughput_d"; type = "real" } + @{ name = "DiskIOPS_s"; type = "string" } + @{ name = "DiskThroughput_s"; type = "string" } @{ name = "DiskTier_s"; type = "string" } @{ name = "DiskState_s"; type = "string" } @{ name = "EncryptionType_s"; type = "string" } @{ name = "Zones_s"; type = "string" } @{ name = "Caching_s"; type = "string" } - @{ name = "DiskSizeGB_d"; type = "real" } + @{ name = "DiskSizeGB_s"; type = "string" } @{ name = "SKU_s"; type = "string" } @{ name = "StatusDate_s"; type = "string" } @{ name = "Tags_s"; type = "string" } @@ -256,7 +256,7 @@ $tableSchemas = @{ @{ name = "DeploymentModel_s"; type = "string" } @{ name = "DiskType_s"; type = "string" } @{ name = "Caching_s"; type = "string" } - @{ name = "DiskSizeGB_d"; type = "real" } + @{ name = "DiskSizeGB_s"; type = "string" } @{ name = "StatusDate_s"; type = "string" } @{ name = "Tags_s"; type = "string" } ) @@ -272,9 +272,9 @@ $tableSchemas = @{ @{ name = "InstanceId_s"; type = "string" } @{ name = "SkuName_s"; type = "string" } @{ name = "Location_s"; type = "string" } - @{ name = "FaultDomains_d"; type = "real" } - @{ name = "UpdateDomains_d"; type = "real" } - @{ name = "VmCount_d"; type = "real" } + @{ name = "FaultDomains_s"; type = "string" } + @{ name = "UpdateDomains_s"; type = "string" } + @{ name = "VmCount_s"; type = "string" } @{ name = "StatusDate_s"; type = "string" } @{ name = "Tags_s"; type = "string" } @{ name = "Zones_s"; type = "string" } @@ -397,15 +397,15 @@ $tableSchemas = @{ @{ name = "SkuTier_s"; type = "string" } @{ name = "Location_s"; type = "string" } @{ name = "LbType_s"; type = "string" } - @{ name = "LbRulesCount_d"; type = "real" } - @{ name = "InboundNatRulesCount_d"; type = "real" } - @{ name = "OutboundRulesCount_d"; type = "real" } - @{ name = "FrontendIPsCount_d"; type = "real" } - @{ name = "BackendIPCount_d"; type = "real" } - @{ name = "BackendAddressesCount_d"; type = "real" } - @{ name = "InboundNatPoolsCount_d"; type = "real" } - @{ name = "BackendPoolsCount_d"; type = "real" } - @{ name = "ProbesCount_d"; type = "real" } + @{ name = "LbRulesCount_s"; type = "string" } + @{ name = "InboundNatRulesCount_s"; type = "string" } + @{ name = "OutboundRulesCount_s"; type = "string" } + @{ name = "FrontendIPsCount_s"; type = "string" } + @{ name = "BackendIPCount_s"; type = "string" } + @{ name = "BackendAddressesCount_s"; type = "string" } + @{ name = "InboundNatPoolsCount_s"; type = "string" } + @{ name = "BackendPoolsCount_s"; type = "string" } + @{ name = "ProbesCount_s"; type = "string" } @{ name = "StatusDate_s"; type = "string" } @{ name = "Tags_s"; type = "string" } ) @@ -421,23 +421,23 @@ $tableSchemas = @{ @{ name = "InstanceId_s"; type = "string" } @{ name = "SkuName_s"; type = "string" } @{ name = "SkuTier_s"; type = "string" } - @{ name = "SkuCapacity_d"; type = "real" } + @{ name = "SkuCapacity_s"; type = "string" } @{ name = "Location_s"; type = "string" } @{ name = "Zones_s"; type = "string" } @{ name = "EnableHttp2_s"; type = "string" } - @{ name = "GatewayIPsCount_d"; type = "real" } - @{ name = "FrontendIPsCount_d"; type = "real" } - @{ name = "FrontendPortsCount_d"; type = "real" } - @{ name = "BackendIPCount_d"; type = "real" } - @{ name = "BackendAddressesCount_d"; type = "real" } - @{ name = "HttpSettingsCount_d"; type = "real" } - @{ name = "HttpListenersCount_d"; type = "real" } - @{ name = "BackendPoolsCount_d"; type = "real" } - @{ name = "ProbesCount_d"; type = "real" } - @{ name = "UrlPathMapsCount_d"; type = "real" } - @{ name = "RequestRoutingRulesCount_d"; type = "real" } - @{ name = "RewriteRulesCount_d"; type = "real" } - @{ name = "RedirectConfsCount_d"; type = "real" } + @{ name = "GatewayIPsCount_s"; type = "string" } + @{ name = "FrontendIPsCount_s"; type = "string" } + @{ name = "FrontendPortsCount_s"; type = "string" } + @{ name = "BackendIPCount_s"; type = "string" } + @{ name = "BackendAddressesCount_s"; type = "string" } + @{ name = "HttpSettingsCount_s"; type = "string" } + @{ name = "HttpListenersCount_s"; type = "string" } + @{ name = "BackendPoolsCount_s"; type = "string" } + @{ name = "ProbesCount_s"; type = "string" } + @{ name = "UrlPathMapsCount_s"; type = "string" } + @{ name = "RequestRoutingRulesCount_s"; type = "string" } + @{ name = "RewriteRulesCount_s"; type = "string" } + @{ name = "RedirectConfsCount_s"; type = "string" } @{ name = "StatusDate_s"; type = "string" } @{ name = "Tags_s"; type = "string" } ) @@ -452,7 +452,7 @@ $tableSchemas = @{ @{ name = "ContainerType_s"; type = "string" } @{ name = "ContainerName_s"; type = "string" } @{ name = "InstanceId_s"; type = "string" } - @{ name = "ResourceCount_d"; type = "real" } + @{ name = "ResourceCount_s"; type = "string" } @{ name = "ManagedBy_s"; type = "string" } @{ name = "ContainerProperties_s"; type = "string" } @{ name = "Tags_s"; type = "string" } @@ -484,13 +484,13 @@ $tableSchemas = @{ @{ name = "Model_s"; type = "string" } @{ name = "VNetPrefixes_s"; type = "string" } @{ name = "DNSServers_s"; type = "string" } - @{ name = "PeeringsCount_d"; type = "real" } + @{ name = "PeeringsCount_s"; type = "string" } @{ name = "EnableDdosProtection_s"; type = "string" } @{ name = "SubnetName_s"; type = "string" } @{ name = "SubnetPrefix_s"; type = "string" } - @{ name = "SubnetDelegationsCount_d"; type = "real" } - @{ name = "SubnetTotalPrefixIPs_d"; type = "real" } - @{ name = "SubnetUsedIPs_d"; type = "real" } + @{ name = "SubnetDelegationsCount_s"; type = "string" } + @{ name = "SubnetTotalPrefixIPs_s"; type = "string" } + @{ name = "SubnetUsedIPs_s"; type = "string" } @{ name = "SubnetNSGId_s"; type = "string" } @{ name = "Tags_s"; type = "string" } @{ name = "StatusDate_s"; type = "string" } @@ -509,8 +509,8 @@ $tableSchemas = @{ @{ name = "IsPrimary_s"; type = "string" } @{ name = "EnableAcceleratedNetworking_s"; type = "string" } @{ name = "EnableIPForwarding_s"; type = "string" } - @{ name = "TapConfigurationsCount_d"; type = "real" } - @{ name = "HostedWorkloadsCount_d"; type = "real" } + @{ name = "TapConfigurationsCount_s"; type = "string" } + @{ name = "HostedWorkloadsCount_s"; type = "string" } @{ name = "InternalDomainNameSuffix_s"; type = "string" } @{ name = "AppliedDnsServers_s"; type = "string" } @{ name = "DnsServers_s"; type = "string" } @@ -540,12 +540,12 @@ $tableSchemas = @{ @{ name = "Location_s"; type = "string" } @{ name = "NSGName_s"; type = "string" } @{ name = "InstanceId_s"; type = "string" } - @{ name = "NicCount_d"; type = "real" } - @{ name = "SubnetCount_d"; type = "real" } + @{ name = "NicCount_s"; type = "string" } + @{ name = "SubnetCount_s"; type = "string" } @{ name = "RuleName_s"; type = "string" } @{ name = "RuleProtocol_s"; type = "string" } @{ name = "RuleDirection_s"; type = "string" } - @{ name = "RulePriority_d"; type = "real" } + @{ name = "RulePriority_s"; type = "string" } @{ name = "RuleAccess_s"; type = "string" } @{ name = "RuleDestinationAddresses_s"; type = "string" } @{ name = "RuleSourceAddresses_s"; type = "string" } @@ -592,11 +592,11 @@ $tableSchemas = @{ @{ name = "ComputerNamePrefix_s"; type = "string" } @{ name = "InstanceId_s"; type = "string" } @{ name = "VMSSSize_s"; type = "string" } - @{ name = "CoresCount_d"; type = "real" } - @{ name = "MemoryMB_d"; type = "real" } + @{ name = "CoresCount_s"; type = "string" } + @{ name = "MemoryMB_s"; type = "string" } @{ name = "OSType_s"; type = "string" } - @{ name = "DataDiskCount_d"; type = "real" } - @{ name = "NicCount_d"; type = "real" } + @{ name = "DataDiskCount_s"; type = "string" } + @{ name = "NicCount_s"; type = "string" } @{ name = "StatusDate_s"; type = "string" } @{ name = "Tags_s"; type = "string" } @{ name = "Capacity_s"; type = "string" } @@ -630,12 +630,12 @@ $tableSchemas = @{ @{ name = "InstanceId_s"; type = "string" } @{ name = "SkuName_s"; type = "string" } @{ name = "SkuTier_s"; type = "string" } - @{ name = "SkuCapacity_d"; type = "real" } + @{ name = "SkuCapacity_s"; type = "string" } @{ name = "ServiceObjectiveName_s"; type = "string" } @{ name = "StorageAccountType_s"; type = "string" } @{ name = "LicenseType_s"; type = "string" } - @{ name = "MaxSizeBytes_d"; type = "real" } - @{ name = "MaxLogSizeBytes_d"; type = "real" } + @{ name = "MaxSizeBytes_s"; type = "string" } + @{ name = "MaxLogSizeBytes_s"; type = "string" } @{ name = "Tags_s"; type = "string" } @{ name = "StatusDate_s"; type = "string" } ) @@ -652,7 +652,7 @@ $tableSchemas = @{ @{ name = "MetricNames_s"; type = "string" } @{ name = "AggregationType_s"; type = "string" } @{ name = "AggregationOfType_s"; type = "string" } - @{ name = "MetricValue_d"; type = "real" } + @{ name = "MetricValue_s"; type = "string" } @{ name = "TimeGrain_s"; type = "string" } @{ name = "TimeSpan_s"; type = "string" } ) @@ -677,7 +677,7 @@ $tableSchemas = @{ @{ name = "DefinitionName_s"; type = "string" } @{ name = "DefinitionReferenceId_s"; type = "string" } @{ name = "EvaluatedOn_s"; type = "string" } - @{ name = "StatesCount_d"; type = "real" } + @{ name = "StatesCount_s"; type = "string" } @{ name = "Tags_s"; type = "string" } @{ name = "StatusDate_s"; type = "string" } ) @@ -701,7 +701,7 @@ $tableSchemas = @{ @{ name = "SubscriptionGuid_g"; type = "string" } @{ name = "SubscriptionName_s"; type = "string" } @{ name = "TenantGuid_g"; type = "string" } - @{ name = "FitScore_d"; type = "real" } + @{ name = "FitScore_s"; type = "string" } @{ name = "Tags_s"; type = "string" } @{ name = "DetailsURL_s"; type = "string" } @{ name = "GeneratedDate_s"; type = "string" } @@ -724,21 +724,21 @@ $tableSchemas = @{ @{ name = "PurchaseDate_s"; type = "string" } @{ name = "ExpiryDate_s"; type = "string" } @{ name = "Archived_s"; type = "string" } - @{ name = "ReservedHours_d"; type = "real" } - @{ name = "UsedHours_d"; type = "real" } + @{ name = "ReservedHours_s"; type = "string" } + @{ name = "UsedHours_s"; type = "string" } @{ name = "UsageDate_s"; type = "string" } - @{ name = "MinUtilPercentage_d"; type = "real" } - @{ name = "AvgUtilPercentage_d"; type = "real" } - @{ name = "MaxUtilPercentage_d"; type = "real" } - @{ name = "PurchasedQuantity_d"; type = "real" } - @{ name = "RemainingQuantity_d"; type = "real" } - @{ name = "TotalReservedQuantity_d"; type = "real" } - @{ name = "UsedQuantity_d"; type = "real" } - @{ name = "UtilizedPercentage_d"; type = "real" } + @{ name = "MinUtilPercentage_s"; type = "string" } + @{ name = "AvgUtilPercentage_s"; type = "string" } + @{ name = "MaxUtilPercentage_s"; type = "string" } + @{ name = "PurchasedQuantity_s"; type = "string" } + @{ name = "RemainingQuantity_s"; type = "string" } + @{ name = "TotalReservedQuantity_s"; type = "string" } + @{ name = "UsedQuantity_s"; type = "string" } + @{ name = "UtilizedPercentage_s"; type = "string" } @{ name = "UtilTrend_s"; type = "string" } - @{ name = "Util1Days_d"; type = "real" } - @{ name = "Util7Days_d"; type = "real" } - @{ name = "Util30Days_d"; type = "real" } + @{ name = "Util1Days_s"; type = "string" } + @{ name = "Util7Days_s"; type = "string" } + @{ name = "Util30Days_s"; type = "string" } @{ name = "Scope_s"; type = "string" } @{ name = "TenantGuid_g"; type = "string" } @{ name = "Cloud_s"; type = "string" } @@ -759,14 +759,14 @@ $tableSchemas = @{ @{ name = "Kind_s"; type = "string" } @{ name = "SkuName_s"; type = "string" } @{ name = "SkuTier_s"; type = "string" } - @{ name = "SkuCapacity_d"; type = "real" } + @{ name = "SkuCapacity_s"; type = "string" } @{ name = "SkuFamily_s"; type = "string" } @{ name = "SkuSize_s"; type = "string" } @{ name = "ComputeMode_s"; type = "string" } - @{ name = "NumberOfWorkers_d"; type = "real" } - @{ name = "CurrentNumberOfWorkers_d"; type = "real" } - @{ name = "MaximumNumberOfWorkers_d"; type = "real" } - @{ name = "NumberOfSites_d"; type = "real" } + @{ name = "NumberOfWorkers_s"; type = "string" } + @{ name = "CurrentNumberOfWorkers_s"; type = "string" } + @{ name = "MaximumNumberOfWorkers_s"; type = "string" } + @{ name = "NumberOfSites_s"; type = "string" } @{ name = "PlanName_s"; type = "string" } @{ name = "Tags_s"; type = "string" } @{ name = "StatusDate_s"; type = "string" } @@ -832,11 +832,11 @@ $tableSchemas = @{ @{ name = "BillingPlan_s"; type = "string" } @{ name = "CommitmentGrain_s"; type = "string" } @{ name = "CommitmentCurrencyCode_s"; type = "string" } - @{ name = "CommitmentAmount_d"; type = "real" } + @{ name = "CommitmentAmount_s"; type = "string" } @{ name = "UtilTrend_s"; type = "string" } - @{ name = "Util1Days_d"; type = "real" } - @{ name = "Util7Days_d"; type = "real" } - @{ name = "Util30Days_d"; type = "real" } + @{ name = "Util1Days_s"; type = "string" } + @{ name = "Util7Days_s"; type = "string" } + @{ name = "Util30Days_s"; type = "string" } @{ name = "Scope_s"; type = "string" } @{ name = "TenantGuid_g"; type = "string" } @{ name = "Cloud_s"; type = "string" } From 880b4adb69a5b2ea880904943c3d38fdda309281 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?H=C3=A9lder=20Pinto?= Date: Sun, 17 May 2026 19:51:59 +0100 Subject: [PATCH 13/30] Fixed table schemas --- .../Setup-LogAnalyticsTablesAndDCRs.ps1 | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 b/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 index a9994b2ad..add3144d0 100644 --- a/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 +++ b/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 @@ -284,7 +284,7 @@ $tableSchemas = @{ @{ name = "TimeGenerated"; type = "datetime" } @{ name = "Timestamp_s"; type = "string" } @{ name = "Cloud_s"; type = "string" } - @{ name = "Category_s"; type = "string" } + @{ name = "Category"; type = "string" } @{ name = "Impact_s"; type = "string" } @{ name = "ImpactedArea_s"; type = "string" } @{ name = "Description_s"; type = "string" } @@ -295,7 +295,7 @@ $tableSchemas = @{ @{ name = "InstanceName_s"; type = "string" } @{ name = "Tags_s"; type = "string" } @{ name = "AdditionalInfo_s"; type = "string" } - @{ name = "ResourceGroup_s"; type = "string" } + @{ name = "ResourceGroup"; type = "string" } @{ name = "SubscriptionGuid_g"; type = "string" } @{ name = "TenantGuid_g"; type = "string" } ) @@ -352,8 +352,8 @@ $tableSchemas = @{ @{ name = "Quantity_s"; type = "string" } @{ name = "ReservationId_s"; type = "string" } @{ name = "ReservationName_s"; type = "string" } - @{ name = "ResourceGroup_s"; type = "string" } - @{ name = "ResourceId_s"; type = "string" } + @{ name = "ResourceGroup"; type = "string" } + @{ name = "ResourceId"; type = "string" } @{ name = "ResourceLocation_s"; type = "string" } @{ name = "ServiceFamily_s"; type = "string" } @{ name = "ServiceInfo1_s"; type = "string" } @@ -648,7 +648,7 @@ $tableSchemas = @{ @{ name = "SubscriptionGuid_g"; type = "string" } @{ name = "ResourceGroupName_s"; type = "string" } @{ name = "ResourceName_s"; type = "string" } - @{ name = "ResourceId_s"; type = "string" } + @{ name = "ResourceId"; type = "string" } @{ name = "MetricNames_s"; type = "string" } @{ name = "AggregationType_s"; type = "string" } @{ name = "AggregationOfType_s"; type = "string" } @@ -664,7 +664,7 @@ $tableSchemas = @{ @{ name = "TenantGuid_g"; type = "string" } @{ name = "SubscriptionGuid_g"; type = "string" } @{ name = "ResourceGroupName_s"; type = "string" } - @{ name = "ResourceId_s"; type = "string" } + @{ name = "ResourceId"; type = "string" } @{ name = "ResourceType_s"; type = "string" } @{ name = "ComplianceState_s"; type = "string" } @{ name = "ComplianceReason_s"; type = "string" } @@ -686,7 +686,7 @@ $tableSchemas = @{ @{ name = "TimeGenerated"; type = "datetime" } @{ name = "Timestamp_s"; type = "string" } @{ name = "Cloud_s"; type = "string" } - @{ name = "Category_s"; type = "string" } + @{ name = "Category"; type = "string" } @{ name = "ImpactedArea_s"; type = "string" } @{ name = "Impact_s"; type = "string" } @{ name = "RecommendationType_s"; type = "string" } @@ -697,7 +697,7 @@ $tableSchemas = @{ @{ name = "InstanceId_s"; type = "string" } @{ name = "InstanceName_s"; type = "string" } @{ name = "AdditionalInfo_s"; type = "string" } - @{ name = "ResourceGroup_s"; type = "string" } + @{ name = "ResourceGroup"; type = "string" } @{ name = "SubscriptionGuid_g"; type = "string" } @{ name = "SubscriptionName_s"; type = "string" } @{ name = "TenantGuid_g"; type = "string" } From c3ff89730b29a7ccf37a7d61ba2b08eb714c3e80 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?H=C3=A9lder=20Pinto?= Date: Mon, 18 May 2026 10:29:52 +0100 Subject: [PATCH 14/30] Changed Metrics Publisher role assignment scope --- .../Setup-LogAnalyticsTablesAndDCRs.ps1 | 65 ++++++++++--------- 1 file changed, 33 insertions(+), 32 deletions(-) diff --git a/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 b/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 index add3144d0..3778111f5 100644 --- a/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 +++ b/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 @@ -14,7 +14,7 @@ The script: automation variable (the DCE itself is expected to already be deployed via the Bicep template). - Creates or updates each custom Log Analytics table schema. - Creates or updates one DCR per table. -- Assigns the Monitoring Metrics Publisher role on every DCR to the Automation account managed identity. +- Assigns the Monitoring Metrics Publisher role on the resource group containing the DCRs to the Automation account managed identity. - Writes the DCR immutable ID for each table into the SQL LogAnalyticsIngestControl table. - Creates/updates the AzureOptimization_DCEIngestionEndpoint automation variable. - Removes the AzureOptimization_LogAnalyticsWorkspaceKey automation variable (no longer needed). @@ -909,6 +909,38 @@ $existingTables = Get-AzOperationalInsightsTable -ResourceGroupName $WorkspaceRe $dcrSuffixToImmutableId = @{} +#region Assign Monitoring Metrics Publisher role at resource group scope +Write-Host "Granting Monitoring Metrics Publisher on resource group to Automation MI..." -ForegroundColor Green + +# Role assignment must be created in the DCR subscription/resource group context +if ($WorkspaceSubscriptionId -ne $currentSubscriptionId) +{ + Set-AzContext -SubscriptionId $currentSubscriptionId | Out-Null +} + +$dcrResourceGroupScope = "/subscriptions/$currentSubscriptionId/resourceGroups/$ResourceGroupName" +$existingAssignment = Get-AzRoleAssignment -ObjectId $automationPrincipalId ` + -RoleDefinitionId $monitoringMetricsPublisherRoleId ` + -Scope $dcrResourceGroupScope -ErrorAction SilentlyContinue +if ($null -eq $existingAssignment) +{ + New-AzRoleAssignment -ObjectId $automationPrincipalId ` + -RoleDefinitionId $monitoringMetricsPublisherRoleId ` + -Scope $dcrResourceGroupScope | Out-Null + Write-Host " Role assigned on resource group scope." -ForegroundColor Gray +} +else +{ + Write-Host " Role already assigned on resource group scope." -ForegroundColor Gray +} + +# Restore workspace subscription context for table operations +if ($WorkspaceSubscriptionId -ne $currentSubscriptionId) +{ + Set-AzContext -SubscriptionId $WorkspaceSubscriptionId | Out-Null +} +#endregion + foreach ($suffix in $tableSchemas.Keys) { $tableName = "$lognamePrefix$suffix" + "_CL" @@ -1080,7 +1112,6 @@ foreach ($suffix in $tableSchemas.Keys) } Write-Host " DCR $dcrName created." -ForegroundColor Gray $dcrImmutableId = $dcrResponseContent.properties.immutableId - $dcrResourceId = $dcrResponseContent.id $dcrSuffixToImmutableId[$suffix] = $dcrImmutableId Write-Host " DCR immutable ID: $dcrImmutableId" -ForegroundColor Gray @@ -1091,36 +1122,6 @@ foreach ($suffix in $tableSchemas.Keys) } #endregion - #region Assign Monitoring Metrics Publisher role on DCR to Automation account MI - Write-Host " [$suffix] Granting Monitoring Metrics Publisher on DCR to Automation MI..." -ForegroundColor Cyan - - # Switch to DCR subscription for role assignment - if ($WorkspaceSubscriptionId -ne $currentSubscriptionId) - { - Set-AzContext -SubscriptionId $currentSubscriptionId | Out-Null - } - - $existingAssignment = Get-AzRoleAssignment -ObjectId $automationPrincipalId ` - -RoleDefinitionId $monitoringMetricsPublisherRoleId ` - -Scope $dcrResourceId -ErrorAction SilentlyContinue - if ($null -eq $existingAssignment) - { - New-AzRoleAssignment -ObjectId $automationPrincipalId ` - -RoleDefinitionId $monitoringMetricsPublisherRoleId ` - -Scope $dcrResourceId | Out-Null - Write-Host " Role assigned." -ForegroundColor Gray - } - else - { - Write-Host " Role already assigned." -ForegroundColor Gray - } - - # Restore workspace subscription context for next iteration - if ($WorkspaceSubscriptionId -ne $currentSubscriptionId) - { - Set-AzContext -SubscriptionId $WorkspaceSubscriptionId | Out-Null - } - #endregion } #endregion From c967267c458c840ca447205241f0f8e18059e7a2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?H=C3=A9lder=20Pinto?= Date: Fri, 22 May 2026 16:25:40 +0100 Subject: [PATCH 15/30] Fixed FitScore schema --- src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 b/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 index 3778111f5..f4a790d47 100644 --- a/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 +++ b/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 @@ -701,7 +701,7 @@ $tableSchemas = @{ @{ name = "SubscriptionGuid_g"; type = "string" } @{ name = "SubscriptionName_s"; type = "string" } @{ name = "TenantGuid_g"; type = "string" } - @{ name = "FitScore_s"; type = "string" } + @{ name = "FitScore_d"; type = "real" } @{ name = "Tags_s"; type = "string" } @{ name = "DetailsURL_s"; type = "string" } @{ name = "GeneratedDate_s"; type = "string" } From a00a585be6f79482e80cf234936d14e90447a3ec Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?H=C3=A9lder=20Pinto?= Date: Tue, 2 Jun 2026 12:35:16 +0100 Subject: [PATCH 16/30] Fixed ingestion schema and improved SQL session management --- .../Setup-LogAnalyticsTablesAndDCRs.ps1 | 4 +- ...t-OptimizationCSVExportsToLogAnalytics.ps1 | 79 ++++++++--- .../Ingest-RecommendationsToLogAnalytics.ps1 | 77 ++++++++--- .../Ingest-RecommendationsToSQLServer.ps1 | 126 ++++++++++++------ .../Ingest-SuppressionsToLogAnalytics.ps1 | 10 +- 5 files changed, 209 insertions(+), 87 deletions(-) diff --git a/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 b/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 index f4a790d47..23649cb3c 100644 --- a/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 +++ b/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 @@ -358,7 +358,7 @@ $tableSchemas = @{ @{ name = "ServiceFamily_s"; type = "string" } @{ name = "ServiceInfo1_s"; type = "string" } @{ name = "ServiceInfo2_s"; type = "string" } - @{ name = "SubscriptionGuid_g"; type = "string" } + @{ name = "SubscriptionId"; type = "string" } @{ name = "SubscriptionName_s"; type = "string" } @{ name = "Tags_s"; type = "string" } @{ name = "Term_s"; type = "string" } @@ -572,7 +572,7 @@ $tableSchemas = @{ @{ name = "AddressVersion_s"; type = "string" } @{ name = "AssociatedResourceId_s"; type = "string" } @{ name = "PublicIpPrefixId_s"; type = "string" } - @{ name = "IPAddress_s"; type = "string" } + @{ name = "IPAddress"; type = "string" } @{ name = "FQDN_s"; type = "string" } @{ name = "Zones_s"; type = "string" } @{ name = "Tags_s"; type = "string" } diff --git a/src/optimization-engine/runbooks/data-collection/Ingest-OptimizationCSVExportsToLogAnalytics.ps1 b/src/optimization-engine/runbooks/data-collection/Ingest-OptimizationCSVExportsToLogAnalytics.ps1 index 63fc8bc27..22a094adb 100644 --- a/src/optimization-engine/runbooks/data-collection/Ingest-OptimizationCSVExportsToLogAnalytics.ps1 +++ b/src/optimization-engine/runbooks/data-collection/Ingest-OptimizationCSVExportsToLogAnalytics.ps1 @@ -116,6 +116,41 @@ function ConvertTo-TypedObject($obj) return [PSCustomObject]$typed } +function Close-SqlConnection() +{ + if ($null -ne $script:SqlConnection) + { + if ($script:SqlConnection.State -ne [System.Data.ConnectionState]::Closed) + { + $script:SqlConnection.Close() + } + $script:SqlConnection.Dispose() + $script:SqlConnection = $null + $script:SqlTokenExpiresOn = $null + } +} + +function Get-SqlConnection() +{ + $refreshWindow = (Get-Date).ToUniversalTime().AddMinutes(5) + if ($null -ne $script:SqlConnection -and + $script:SqlConnection.State -eq [System.Data.ConnectionState]::Open -and + $script:SqlTokenExpiresOn -gt $refreshWindow) + { + return $script:SqlConnection + } + + Close-SqlConnection + + $dbToken = Get-AzAccessToken -ResourceUrl "https://$azureSqlDomain/" + $script:SqlTokenExpiresOn = $dbToken.ExpiresOn.UtcDateTime + $script:SqlConnection = New-Object System.Data.SqlClient.SqlConnection("Server=tcp:$sqlserver,1433;Database=$sqldatabase;Encrypt=True;Connection Timeout=$SqlTimeout;Pooling=False") + $script:SqlConnection.AccessToken = $dbToken.Token + $script:SqlConnection.Open() + + return $script:SqlConnection +} + #endregion Functions $cloudDetails = Get-AzEnvironment -Name $CloudEnvironment @@ -145,10 +180,7 @@ do $tries++ try { - $dbToken = Get-AzAccessToken -ResourceUrl "https://$azureSqlDomain/" - $Conn = New-Object System.Data.SqlClient.SqlConnection("Server=tcp:$sqlserver,1433;Database=$sqldatabase;Encrypt=True;Connection Timeout=$SqlTimeout;") - $Conn.AccessToken = $dbToken.Token - $Conn.Open() + $Conn = Get-SqlConnection $Cmd = New-Object system.Data.SqlClient.SqlCommand $Cmd.Connection = $Conn $Cmd.CommandTimeout = $SqlTimeout @@ -162,20 +194,31 @@ do } catch { + Close-SqlConnection Write-Output "Failed to contact SQL at try $tries." Write-Output $Error[0] Start-Sleep -Seconds ($tries * 20) } + finally + { + if ($null -ne $sqlAdapter) + { + $sqlAdapter.Dispose() + } + if ($null -ne $Cmd) + { + $Cmd.Dispose() + } + } } while (-not($connectionSuccess) -and $tries -lt 3) +Close-SqlConnection + if (-not($connectionSuccess)) { throw "Could not establish connection to SQL." } -$Conn.Close() -$Conn.Dispose() - if ($controlRows.Count -eq 0 -or -not($controlRows[0].LastProcessedDateTime)) { throw "Could not find a valid ingestion control row for $storageAccountSinkContainer" @@ -302,17 +345,13 @@ foreach ($blob in $unprocessedBlobs) $lastProcessedDateTime = $updatedLastProcessedDateTime Write-Output "Updating last processed time / line to $($updatedLastProcessedDateTime) / $updatedLastProcessedLine" $sqlStatement = "UPDATE [$LogAnalyticsIngestControlTable] SET LastProcessedLine = $updatedLastProcessedLine, LastProcessedDateTime = '$updatedLastProcessedDateTime' WHERE StorageContainerName = '$storageAccountSinkContainer'" - $dbToken = Get-AzAccessToken -ResourceUrl "https://$azureSqlDomain/" - $Conn = New-Object System.Data.SqlClient.SqlConnection("Server=tcp:$sqlserver,1433;Database=$sqldatabase;Encrypt=True;Connection Timeout=$SqlTimeout;") - $Conn.AccessToken = $dbToken.Token - $Conn.Open() + $Conn = Get-SqlConnection $Cmd = New-Object system.Data.SqlClient.SqlCommand $Cmd.Connection = $Conn $Cmd.CommandText = $sqlStatement $Cmd.CommandTimeout = $SqlTimeout - $Cmd.ExecuteReader() - $Conn.Close() - $Conn.Dispose() + $Cmd.ExecuteNonQuery() | Out-Null + $Cmd.Dispose() $chunkLines = @() $chunkLines += $header @@ -332,17 +371,13 @@ foreach ($blob in $unprocessedBlobs) $updatedLastProcessedDateTime = $newProcessedTime Write-Output "Updating last processed time / line to $($updatedLastProcessedDateTime) / $updatedLastProcessedLine" $sqlStatement = "UPDATE [$LogAnalyticsIngestControlTable] SET LastProcessedLine = $updatedLastProcessedLine, LastProcessedDateTime = '$updatedLastProcessedDateTime' WHERE StorageContainerName = '$storageAccountSinkContainer'" - $dbToken = Get-AzAccessToken -ResourceUrl "https://$azureSqlDomain/" - $Conn = New-Object System.Data.SqlClient.SqlConnection("Server=tcp:$sqlserver,1433;Database=$sqldatabase;Encrypt=True;Connection Timeout=$SqlTimeout;") - $Conn.AccessToken = $dbToken.Token - $Conn.Open() + $Conn = Get-SqlConnection $Cmd = New-Object system.Data.SqlClient.SqlCommand $Cmd.Connection = $Conn $Cmd.CommandText = $sqlStatement $Cmd.CommandTimeout = $SqlTimeout - $Cmd.ExecuteReader() - $Conn.Close() - $Conn.Dispose() + $Cmd.ExecuteNonQuery() | Out-Null + $Cmd.Dispose() } else { @@ -352,4 +387,6 @@ foreach ($blob in $unprocessedBlobs) Remove-Item -Path $blobFilePath -Force } +Close-SqlConnection + Write-Output "DONE" \ No newline at end of file diff --git a/src/optimization-engine/runbooks/recommendations/Ingest-RecommendationsToLogAnalytics.ps1 b/src/optimization-engine/runbooks/recommendations/Ingest-RecommendationsToLogAnalytics.ps1 index 0648b6dcc..fc2e927b0 100644 --- a/src/optimization-engine/runbooks/recommendations/Ingest-RecommendationsToLogAnalytics.ps1 +++ b/src/optimization-engine/runbooks/recommendations/Ingest-RecommendationsToLogAnalytics.ps1 @@ -25,7 +25,7 @@ $dceEndpoint = Get-AutomationVariable -Name "AzureOptimization_DCEIngestionEndp $LogAnalyticsChunkSize = [int] (Get-AutomationVariable -Name "AzureOptimization_LogAnalyticsChunkSize" -ErrorAction SilentlyContinue) if (-not($LogAnalyticsChunkSize -gt 0)) { - $LogAnalyticsChunkSize = 6000 + $LogAnalyticsChunkSize = 150 } $lognamePrefix = Get-AutomationVariable -Name "AzureOptimization_LogAnalyticsLogPrefix" -ErrorAction SilentlyContinue if ([string]::IsNullOrEmpty($lognamePrefix)) @@ -64,6 +64,41 @@ function Send-LogIngestionData($accessToken, $dceEndpoint, $dcrImmutableId, $str return $response.StatusCode } +function Close-SqlConnection() +{ + if ($null -ne $script:SqlConnection) + { + if ($script:SqlConnection.State -ne [System.Data.ConnectionState]::Closed) + { + $script:SqlConnection.Close() + } + $script:SqlConnection.Dispose() + $script:SqlConnection = $null + $script:SqlTokenExpiresOn = $null + } +} + +function Get-SqlConnection() +{ + $refreshWindow = (Get-Date).ToUniversalTime().AddMinutes(5) + if ($null -ne $script:SqlConnection -and + $script:SqlConnection.State -eq [System.Data.ConnectionState]::Open -and + $script:SqlTokenExpiresOn -gt $refreshWindow) + { + return $script:SqlConnection + } + + Close-SqlConnection + + $dbToken = Get-AzAccessToken -ResourceUrl "https://$azureSqlDomain/" + $script:SqlTokenExpiresOn = $dbToken.ExpiresOn.UtcDateTime + $script:SqlConnection = New-Object System.Data.SqlClient.SqlConnection("Server=tcp:$sqlserver,1433;Database=$sqldatabase;Encrypt=True;Connection Timeout=$SqlTimeout;Pooling=False") + $script:SqlConnection.AccessToken = $dbToken.Token + $script:SqlConnection.Open() + + return $script:SqlConnection +} + #endregion Functions @@ -112,10 +147,7 @@ do $tries++ try { - $dbToken = Get-AzAccessToken -ResourceUrl "https://$azureSqlDomain/" - $Conn = New-Object System.Data.SqlClient.SqlConnection("Server=tcp:$sqlserver,1433;Database=$sqldatabase;Encrypt=True;Connection Timeout=$SqlTimeout;") - $Conn.AccessToken = $dbToken.Token - $Conn.Open() + $Conn = Get-SqlConnection $Cmd = New-Object system.Data.SqlClient.SqlCommand $Cmd.Connection = $Conn $Cmd.CommandTimeout = $SqlTimeout @@ -129,20 +161,31 @@ do } catch { + Close-SqlConnection Write-Output "Failed to contact SQL at try $tries." Write-Output $Error[0] Start-Sleep -Seconds ($tries * 20) } + finally + { + if ($null -ne $sqlAdapter) + { + $sqlAdapter.Dispose() + } + if ($null -ne $Cmd) + { + $Cmd.Dispose() + } + } } while (-not($connectionSuccess) -and $tries -lt 3) +Close-SqlConnection + if (-not($connectionSuccess)) { throw "Could not establish connection to SQL." } -$Conn.Close() -$Conn.Dispose() - if ($controlRows.Count -eq 0 -or -not($controlRows[0].LastProcessedDateTime)) { throw "Could not find a valid ingestion control row for $storageAccountSinkContainer" @@ -164,9 +207,9 @@ if ([string]::IsNullOrEmpty($dcrImmutableId)) # Obtain a bearer token for the Logs Ingestion API switch ($cloudEnvironment) { - "AzureChinaCloud" { $monitorAudience = "https://monitor.azure.cn/" } - "AzureUSGovernment" { $monitorAudience = "https://monitor.azure.us/" } - default { $monitorAudience = "https://monitor.azure.com/" } + "AzureChinaCloud" { $monitorAudience = "https://monitor.azure.cn/" } + "AzureUSGovernment" { $monitorAudience = "https://monitor.azure.us/" } + default { $monitorAudience = "https://monitor.azure.com/" } } $monitorToken = (Get-AzAccessToken -ResourceUrl $monitorAudience).Token @@ -274,17 +317,13 @@ foreach ($blob in $unprocessedBlobs) $lastProcessedDateTime = $updatedLastProcessedDateTime Write-Output "Updating last processed time / line to $($updatedLastProcessedDateTime) / $updatedLastProcessedLine" $sqlStatement = "UPDATE [$LogAnalyticsIngestControlTable] SET LastProcessedLine = $updatedLastProcessedLine, LastProcessedDateTime = '$updatedLastProcessedDateTime' WHERE StorageContainerName = '$storageAccountSinkContainer'" - $dbToken = Get-AzAccessToken -ResourceUrl "https://$azureSqlDomain/" - $Conn = New-Object System.Data.SqlClient.SqlConnection("Server=tcp:$sqlserver,1433;Database=$sqldatabase;Encrypt=True;Connection Timeout=$SqlTimeout;") - $Conn.AccessToken = $dbToken.Token - $Conn.Open() + $Conn = Get-SqlConnection $Cmd = New-Object system.Data.SqlClient.SqlCommand $Cmd.Connection = $Conn $Cmd.CommandText = $sqlStatement $Cmd.CommandTimeout = $SqlTimeout - $Cmd.ExecuteReader() - $Conn.Close() - $Conn.Dispose() + $Cmd.ExecuteNonQuery() | Out-Null + $Cmd.Dispose() } else { @@ -303,4 +342,6 @@ foreach ($blob in $unprocessedBlobs) Remove-Item -Path $blob.Name -Force } +Close-SqlConnection + Write-Output "DONE" \ No newline at end of file diff --git a/src/optimization-engine/runbooks/recommendations/Ingest-RecommendationsToSQLServer.ps1 b/src/optimization-engine/runbooks/recommendations/Ingest-RecommendationsToSQLServer.ps1 index eb9d9e963..b95adb40e 100644 --- a/src/optimization-engine/runbooks/recommendations/Ingest-RecommendationsToSQLServer.ps1 +++ b/src/optimization-engine/runbooks/recommendations/Ingest-RecommendationsToSQLServer.ps1 @@ -32,7 +32,8 @@ $storageAccountSink = Get-AutomationVariable -Name "AzureOptimization_StorageSi $storageAccountSinkContainer = Get-AutomationVariable -Name "AzureOptimization_RecommendationsContainer" -ErrorAction SilentlyContinue -if ([string]::IsNullOrEmpty($storageAccountSinkContainer)) { +if ([string]::IsNullOrEmpty($storageAccountSinkContainer)) +{ $storageAccountSinkContainer = "recommendationsexports" } $StorageBlobsPageSize = [int] (Get-AutomationVariable -Name "AzureOptimization_StorageBlobsPageSize" -ErrorAction SilentlyContinue) @@ -41,14 +42,54 @@ if (-not($StorageBlobsPageSize -gt 0)) $StorageBlobsPageSize = 1000 } +function Close-SqlConnection() +{ + if ($null -ne $script:SqlConnection) + { + if ($script:SqlConnection.State -ne [System.Data.ConnectionState]::Closed) + { + $script:SqlConnection.Close() + } + $script:SqlConnection.Dispose() + $script:SqlConnection = $null + $script:SqlTokenExpiresOn = $null + } +} + +function Get-SqlConnection() +{ + $refreshWindow = (Get-Date).ToUniversalTime().AddMinutes(5) + if ($null -ne $script:SqlConnection -and + $script:SqlConnection.State -eq [System.Data.ConnectionState]::Open -and + $script:SqlTokenExpiresOn -gt $refreshWindow) + { + return $script:SqlConnection + } + + Close-SqlConnection + + $dbToken = Get-AzAccessToken -ResourceUrl "https://$azureSqlDomain/" + $script:SqlTokenExpiresOn = $dbToken.ExpiresOn.UtcDateTime + $script:SqlConnection = New-Object System.Data.SqlClient.SqlConnection("Server=tcp:$sqlserver,1433;Database=$sqldatabase;Encrypt=True;Connection Timeout=$SqlTimeout;Pooling=False") + $script:SqlConnection.AccessToken = $dbToken.Token + $script:SqlConnection.Open() + + return $script:SqlConnection +} + + "Logging in to Azure with $authenticationOption..." -switch ($authenticationOption) { - "UserAssignedManagedIdentity" { +switch ($authenticationOption) +{ + "UserAssignedManagedIdentity" + { Connect-AzAccount -Identity -EnvironmentName $cloudEnvironment -AccountId $uamiClientID break } - Default { #ManagedIdentity + Default + { + #ManagedIdentity Connect-AzAccount -Identity -EnvironmentName $cloudEnvironment break } @@ -71,7 +112,7 @@ do $blobs = Get-AzStorageBlob -Container $storageAccountSinkContainer -MaxCount $StorageBlobsPageSize -ContinuationToken $continuationToken -Context $saCtx | Sort-Object -Property LastModified if ($blobs.Count -le 0) { break } $allblobs += $blobs - $continuationToken = $blobs[$blobs.Count -1].ContinuationToken; + $continuationToken = $blobs[$blobs.Count - 1].ContinuationToken } While ($null -ne $continuationToken) @@ -81,14 +122,13 @@ $recommendationsTable = "Recommendations" $tries = 0 $connectionSuccess = $false -do { +do +{ $tries++ - try { - $dbToken = Get-AzAccessToken -ResourceUrl "https://$azureSqlDomain/" - $Conn = New-Object System.Data.SqlClient.SqlConnection("Server=tcp:$sqlserver,1433;Database=$sqldatabase;Encrypt=True;Connection Timeout=$SqlTimeout;") - $Conn.AccessToken = $dbToken.Token - $Conn.Open() - $Cmd=new-object system.Data.SqlClient.SqlCommand + try + { + $Conn = Get-SqlConnection + $Cmd = New-Object system.Data.SqlClient.SqlCommand $Cmd.Connection = $Conn $Cmd.CommandTimeout = $SqlTimeout $Cmd.CommandText = "SELECT * FROM [dbo].[$SqlServerIngestControlTable] WHERE StorageContainerName = '$storageAccountSinkContainer' and SqlTableName = '$recommendationsTable'" @@ -99,13 +139,17 @@ do { $sqlAdapter.Fill($controlRows) | Out-Null $connectionSuccess = $true } - catch { + catch + { + Close-SqlConnection Write-Output "Failed to contact SQL at try $tries." Write-Output $Error[0] Start-Sleep -Seconds ($tries * 20) } } while (-not($connectionSuccess) -and $tries -lt 3) +Close-SqlConnection + if (-not($connectionSuccess)) { throw "Could not establish connection to SQL." @@ -120,8 +164,6 @@ $controlRow = $controlRows[0] $lastProcessedLine = $controlRow.LastProcessedLine $lastProcessedDateTime = $controlRow.LastProcessedDateTime.ToString("yyyy'-'MM'-'dd'T'HH':'mm':'ss'.'fff'Z'") -$Conn.Close() -$Conn.Dispose() Write-Output "Processing blobs modified after $lastProcessedDateTime (line $lastProcessedLine) and ingesting them into the Recommendations SQL table..." @@ -129,10 +171,12 @@ $newProcessedTime = $null $unprocessedBlobs = @() -foreach ($blob in $allblobs) { +foreach ($blob in $allblobs) +{ $blobLastModified = $blob.LastModified.UtcDateTime.ToString("yyyy'-'MM'-'dd'T'HH':'mm':'ss'.'fff'Z'") if ($lastProcessedDateTime -lt $blobLastModified -or ` - ($lastProcessedDateTime -eq $blobLastModified -and $lastProcessedLine -gt 0)) { + ($lastProcessedDateTime -eq $blobLastModified -and $lastProcessedLine -gt 0)) + { Write-Output "$($blob.Name) found (modified on $blobLastModified)" $unprocessedBlobs += $blob } @@ -142,7 +186,8 @@ $unprocessedBlobs = $unprocessedBlobs | Sort-Object -Property LastModified Write-Output "Found $($unprocessedBlobs.Count) new blobs to process..." -foreach ($blob in $unprocessedBlobs) { +foreach ($blob in $unprocessedBlobs) +{ $newProcessedTime = $blob.LastModified.UtcDateTime.ToString("yyyy'-'MM'-'dd'T'HH':'mm':'ss'.'fff'Z'") Write-Output "About to process $($blob.Name)..." Get-AzStorageBlobContent -CloudBlob $blob.ICloudBlob -Context $saCtx -Force @@ -167,8 +212,9 @@ foreach ($blob in $unprocessedBlobs) { if ($recCount -gt 1) { - for ($i = 0; $i -lt $recCount; $i += $ChunkSize) { - $jsonObjectSplitted += , @($jsonObject[$i..($i + ($ChunkSize - 1))]); + for ($i = 0; $i -lt $recCount; $i += $ChunkSize) + { + $jsonObjectSplitted += , @($jsonObject[$i..($i + ($ChunkSize - 1))]) } } else @@ -223,60 +269,58 @@ foreach ($blob in $unprocessedBlobs) { $sqlStatement += ", '$($jsonObjectSplitted[$j][$i].InstanceId)', '$($jsonObjectSplitted[$j][$i].InstanceName)', '$additionalInfoString'" $sqlStatement += ", $resourceGroup, $subscriptionGuid, $subscriptionName, '$($jsonObjectSplitted[$j][$i].TenantGuid)'" $sqlStatement += ", $($jsonObjectSplitted[$j][$i].FitScore), '$tagsString', '$($jsonObjectSplitted[$j][$i].DetailsURL)')" - if ($i -ne ($jsonObjectSplitted[$j].Count-1)) + if ($i -ne ($jsonObjectSplitted[$j].Count - 1)) { $sqlStatement += "," } } - $dbToken = Get-AzAccessToken -ResourceUrl "https://$azureSqlDomain/" - $Conn2 = New-Object System.Data.SqlClient.SqlConnection("Server=tcp:$sqlserver,1433;Database=$sqldatabase;Encrypt=True;Connection Timeout=$SqlTimeout;") - $Conn2.AccessToken = $dbToken.Token - $Conn2.Open() - - $Cmd=new-object system.Data.SqlClient.SqlCommand - $Cmd.Connection = $Conn2 + $Conn = Get-SqlConnection + $Cmd = New-Object system.Data.SqlClient.SqlCommand + $Cmd.Connection = $Conn $Cmd.CommandText = $sqlStatement $Cmd.CommandTimeout = $SqlTimeout try { - $Cmd.ExecuteReader() + $Cmd.ExecuteNonQuery() | Out-Null } catch { Write-Output "Failed statement: $sqlStatement" throw } - - $Conn2.Close() + finally + { + $Cmd.Dispose() + } $linesProcessed += $currentObjectLines Write-Output "Processed $linesProcessed lines..." - if ($j -eq ($jsonObjectSplitted.Count - 1)) { + if ($j -eq ($jsonObjectSplitted.Count - 1)) + { $lastProcessedLine = -1 } - else { + else + { $lastProcessedLine = $linesProcessed - 1 } $updatedLastProcessedLine = $lastProcessedLine $updatedLastProcessedDateTime = $lastProcessedDateTime - if ($j -eq ($jsonObjectSplitted.Count - 1)) { + if ($j -eq ($jsonObjectSplitted.Count - 1)) + { $updatedLastProcessedDateTime = $newProcessedTime } $lastProcessedDateTime = $updatedLastProcessedDateTime Write-Output "Updating last processed time / line to $($updatedLastProcessedDateTime) / $updatedLastProcessedLine" $sqlStatement = "UPDATE [$SqlServerIngestControlTable] SET LastProcessedLine = $updatedLastProcessedLine, LastProcessedDateTime = '$updatedLastProcessedDateTime' WHERE StorageContainerName = '$storageAccountSinkContainer'" - $dbToken = Get-AzAccessToken -ResourceUrl "https://$azureSqlDomain/" - $Conn = New-Object System.Data.SqlClient.SqlConnection("Server=tcp:$sqlserver,1433;Database=$sqldatabase;Encrypt=True;Connection Timeout=$SqlTimeout;") - $Conn.AccessToken = $dbToken.Token - $Conn.Open() - $Cmd=new-object system.Data.SqlClient.SqlCommand + $Conn = Get-SqlConnection + $Cmd = New-Object system.Data.SqlClient.SqlCommand $Cmd.Connection = $Conn $Cmd.CommandText = $sqlStatement $Cmd.CommandTimeout = $SqlTimeout - $Cmd.ExecuteReader() - $Conn.Close() + $Cmd.ExecuteNonQuery() | Out-Null + $Cmd.Dispose() } else { diff --git a/src/optimization-engine/runbooks/recommendations/Ingest-SuppressionsToLogAnalytics.ps1 b/src/optimization-engine/runbooks/recommendations/Ingest-SuppressionsToLogAnalytics.ps1 index 291950137..5699adfff 100644 --- a/src/optimization-engine/runbooks/recommendations/Ingest-SuppressionsToLogAnalytics.ps1 +++ b/src/optimization-engine/runbooks/recommendations/Ingest-SuppressionsToLogAnalytics.ps1 @@ -10,7 +10,7 @@ $dceEndpoint = Get-AutomationVariable -Name "AzureOptimization_DCEIngestionEndp $LogAnalyticsChunkSize = [int] (Get-AutomationVariable -Name "AzureOptimization_LogAnalyticsChunkSize" -ErrorAction SilentlyContinue) if (-not($LogAnalyticsChunkSize -gt 0)) { - $LogAnalyticsChunkSize = 6000 + $LogAnalyticsChunkSize = 150 } $lognamePrefix = Get-AutomationVariable -Name "AzureOptimization_LogAnalyticsLogPrefix" -ErrorAction SilentlyContinue if ([string]::IsNullOrEmpty($lognamePrefix)) @@ -77,9 +77,9 @@ $azureSqlDomain = $cloudDetails.SqlDatabaseDnsSuffix.Substring(1) # Determine the Logs Ingestion API audience URL for this cloud switch ($cloudEnvironment) { - "AzureChinaCloud" { $monitorAudience = "https://monitor.azure.cn/" } - "AzureUSGovernment" { $monitorAudience = "https://monitor.azure.us/" } - default { $monitorAudience = "https://monitor.azure.com/" } + "AzureChinaCloud" { $monitorAudience = "https://monitor.azure.cn/" } + "AzureUSGovernment" { $monitorAudience = "https://monitor.azure.us/" } + default { $monitorAudience = "https://monitor.azure.com/" } } Write-Output "Getting excluded recommendation sub-type IDs..." @@ -197,7 +197,7 @@ $filtersJson = $filterObjects | ForEach-Object { $LogAnalyticsSuffix = "SuppressionsV1" $logname = $lognamePrefix + $LogAnalyticsSuffix -$streamName = "Custom-$lognamePrefix$LogAnalyticsSuffix" +$streamName = "Custom-$logname" # Retrieve DCR immutable ID from SQL control table $dcrImmutableId = $null From 7087f07b475fb445fc34918c73cf5c1ae3b5d3ed Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?H=C3=A9lder=20Pinto?= Date: Tue, 2 Jun 2026 15:52:37 +0100 Subject: [PATCH 17/30] Improved support for legacy and non-enterprise subscriptions --- .../Export-ConsumptionToBlobStorage.ps1 | 421 +++++++++--------- 1 file changed, 221 insertions(+), 200 deletions(-) diff --git a/src/optimization-engine/runbooks/data-collection/Export-ConsumptionToBlobStorage.ps1 b/src/optimization-engine/runbooks/data-collection/Export-ConsumptionToBlobStorage.ps1 index 11dabcb0d..cda9b13ab 100644 --- a/src/optimization-engine/runbooks/data-collection/Export-ConsumptionToBlobStorage.ps1 +++ b/src/optimization-engine/runbooks/data-collection/Export-ConsumptionToBlobStorage.ps1 @@ -22,26 +22,32 @@ $ErrorActionPreference = "Stop" $global:hadErrors = $false $global:scopesWithErrors = @() -function Authenticate-AzureWithOption { +function Authenticate-AzureWithOption +{ param ( [string] $authOption = "ManagedIdentity", [string] $cloudEnv = "AzureCloud", [string] $clientID ) - switch ($authOption) { - "UserAssignedManagedIdentity" { + switch ($authOption) + { + "UserAssignedManagedIdentity" + { Connect-AzAccount -Identity -EnvironmentName $cloudEnv -AccountId $clientID break } - Default { #ManagedIdentity + Default + { + #ManagedIdentity Connect-AzAccount -Identity -EnvironmentName $cloudEnv break } } } -function Generate-CostDetails { +function Generate-CostDetails +{ param ( [string] $ScopeId, [string] $ScopeName @@ -54,7 +60,7 @@ function Generate-CostDetails { $body = "{ `"metric`": `"$consumptionMetric`", `"timePeriod`": { `"start`": `"$targetStartDate`", `"end`": `"$targetEndDate`" } }" $result = Invoke-AzRestMethod -Path $CostDetailsApiPath -Method POST -Payload $body $requestResultPath = $result.Headers.Location.PathAndQuery - if ($result.StatusCode -in (200,202)) + if ($result.StatusCode -in (200, 202)) { $tries = 0 $requestSuccess = $false @@ -103,53 +109,54 @@ function Generate-CostDetails { # header normalization between MCA and EA $headerConversion = @{ - additionalInfo = "AdditionalInfo"; - billingAccountId = "BillingAccountId"; - billingAccountName = "BillingAccountName"; - billingCurrency = "BillingCurrencyCode"; - billingPeriodEndDate = "BillingPeriodEndDate"; - billingPeriodStartDate = "BillingPeriodStartDate"; - billingProfileId = "BillingProfileId"; - billingProfileName = "BillingProfileName"; - chargeType = "ChargeType"; - consumedService = "ConsumedService"; - costAllocationRuleName = "CostAllocationRuleName"; - costCenter = "CostCenter"; - costInBillingCurrency = "CostInBillingCurrency"; - date = "Date"; - effectivePrice = "EffectivePrice"; - frequency = "Frequency"; - invoiceSectionId = "InvoiceSectionId"; - invoiceSectionName = "InvoiceSectionName"; - isAzureCreditEligible = "IsAzureCreditEligible"; - meterCategory = "MeterCategory"; - meterId = "MeterId"; - meterName = "MeterName"; - meterRegion = "MeterRegion"; - meterSubCategory = "MeterSubCategory"; - offerId = "OfferId"; - pricingModel = "PricingModel"; - productOrderId = "ProductOrderId"; - productOrderName = "ProductOrderName"; - publisherName = "PublisherName"; - publisherType = "PublisherType"; - quantity = "Quantity"; - reservationId = "ReservationId"; - reservationName = "ReservationName"; - resourceGroupName = "ResourceGroup"; - resourceLocation = "ResourceLocation"; - serviceFamily = "ServiceFamily"; - serviceInfo1 = "ServiceInfo1"; - serviceInfo2 = "ServiceInfo2"; - subscriptionName = "SubscriptionName"; - tags = "Tags"; - term = "Term"; - unitOfMeasure = "UnitOfMeasure"; - unitPrice = "UnitPrice" + additionalInfo = "AdditionalInfo" + billingAccountId = "BillingAccountId" + billingAccountName = "BillingAccountName" + billingCurrency = "BillingCurrencyCode" + billingPeriodEndDate = "BillingPeriodEndDate" + billingPeriodStartDate = "BillingPeriodStartDate" + billingProfileId = "BillingProfileId" + billingProfileName = "BillingProfileName" + chargeType = "ChargeType" + consumedService = "ConsumedService" + costAllocationRuleName = "CostAllocationRuleName" + costCenter = "CostCenter" + costInBillingCurrency = "CostInBillingCurrency" + date = "Date" + effectivePrice = "EffectivePrice" + frequency = "Frequency" + invoiceSectionId = "InvoiceSectionId" + invoiceSectionName = "InvoiceSectionName" + isAzureCreditEligible = "IsAzureCreditEligible" + meterCategory = "MeterCategory" + meterId = "MeterId" + meterName = "MeterName" + meterRegion = "MeterRegion" + meterSubCategory = "MeterSubCategory" + offerId = "OfferId" + pricingModel = "PricingModel" + productOrderId = "ProductOrderId" + productOrderName = "ProductOrderName" + publisherName = "PublisherName" + publisherType = "PublisherType" + quantity = "Quantity" + reservationId = "ReservationId" + reservationName = "ReservationName" + resourceGroupName = "ResourceGroup" + resourceLocation = "ResourceLocation" + serviceFamily = "ServiceFamily" + serviceInfo1 = "ServiceInfo1" + serviceInfo2 = "ServiceInfo2" + subscriptionName = "SubscriptionName" + tags = "Tags" + term = "Term" + unitOfMeasure = "UnitOfMeasure" + unitPrice = "UnitPrice" } $lineCounter = 0 - while ($r.Peek() -ge 0) { + while ($r.Peek() -ge 0) + { $line = $r.ReadLine() $lineCounter++ if ($lineCounter -eq 1) @@ -178,7 +185,7 @@ function Generate-CostDetails { $w.Close() $csvBlobName = [System.IO.Path]::GetFileName($finalCsvExportPath) - $csvProperties = @{"ContentType" = "text/csv"}; + $csvProperties = @{"ContentType" = "text/csv" } Set-AzStorageBlobContent -File $finalCsvExportPath -Container $storageAccountSinkContainer -Properties $csvProperties -Blob $csvBlobName -Context $saCtx -Force $now = (Get-Date).ToUniversalTime().ToString("yyyy'-'MM'-'dd'T'HH':'mm':'ss'.'fff'Z'") @@ -414,12 +421,12 @@ else # for each subscription, get billing data -$datetime = (get-date).ToUniversalTime() +$datetime = (Get-Date).ToUniversalTime() $timestamp = $datetime.ToString("yyyy-MM-ddTHH:mm:00.000Z") if ($cloudEnvironment -eq "AzureChinaCloud" -and -not([string]::IsNullOrEmpty($chinaEAEnrollment)) -and -not([string]::IsNullOrEmpty($chinaEAKey))) { - $targetMonth = $targetStartDate.Substring(0,7) + $targetMonth = $targetStartDate.Substring(0, 7) $consumption = $null $billingEntries = @() @@ -427,7 +434,7 @@ if ($cloudEnvironment -eq "AzureChinaCloud" -and -not([string]::IsNullOrEmpty($c $PricesheetApiUri = "https://ea.azure.cn/rest/$chinaEAEnrollment/usage-report?month=$targetMonth&type=pricesheet&fmt=Csv" $Headers = @{} - $Headers.Add("Authorization","Bearer $chinaEAKey") + $Headers.Add("Authorization", "Bearer $chinaEAKey") Write-Output "Getting pricesheet for month $targetMonth (EA enrollment $chinaEAEnrollment)..." @@ -443,12 +450,12 @@ if ($cloudEnvironment -eq "AzureChinaCloud" -and -not([string]::IsNullOrEmpty($c Write-Output "Removed pricesheet-$targetMonth.csv from local disk..." - $csvFile2 = $csvFile[2..($csvFile.Count-1)] + $csvFile2 = $csvFile[2..($csvFile.Count - 1)] $headerLine = $csvFile2[0] $columnHeaders = $headerLine.Split(",") for ($i = 0; $i -lt $columnHeaders.Count; $i++) { - if($columnHeaders[$i] -match '.+\((?.+)\)') + if ($columnHeaders[$i] -match '.+\((?.+)\)') { $columnHeaders[$i] = $Matches.ColumnName } @@ -465,7 +472,8 @@ if ($cloudEnvironment -eq "AzureChinaCloud" -and -not([string]::IsNullOrEmpty($c $requestSuccess = $false do { - try { + try + { $tries++ Invoke-RestMethod -Method Get -Uri $BillingApiUri -Headers $Headers -OutFile "usagedetails-$targetStartDate.csv" @@ -479,12 +487,12 @@ if ($cloudEnvironment -eq "AzureChinaCloud" -and -not([string]::IsNullOrEmpty($c Write-Output "Removed usagedetails-$targetStartDate.csv from local disk..." - $csvFile2 = $csvFile[2..($csvFile.Count-1)] + $csvFile2 = $csvFile[2..($csvFile.Count - 1)] $headerLine = $csvFile2[0] $columnHeaders = $headerLine.Split(",") for ($i = 0; $i -lt $columnHeaders.Count; $i++) { - if($columnHeaders[$i] -match '.+\((?.+)\)') + if ($columnHeaders[$i] -match '.+\((?.+)\)') { $columnHeaders[$i] = $Matches.ColumnName } @@ -496,7 +504,8 @@ if ($cloudEnvironment -eq "AzureChinaCloud" -and -not([string]::IsNullOrEmpty($c $consumption = $csvFile2 | ConvertFrom-Csv $requestSuccess = $true } - catch { + catch + { $ErrorMessage = $_.Exception.Message Write-Warning "Error getting consumption data: $ErrorMessage. $tries of 3 tries. Waiting 60 seconds..." Start-Sleep -s 60 @@ -523,7 +532,7 @@ if ($cloudEnvironment -eq "AzureChinaCloud" -and -not([string]::IsNullOrEmpty($c { $instanceId = $consumptionLine.'Instance ID'.ToLower() $idParts = $consumptionLine.'Instance ID'.Split("/") - $instanceName = $idParts[$idParts.Count-1].ToLower() + $instanceName = $idParts[$idParts.Count - 1].ToLower() } $rgName = $null @@ -569,31 +578,31 @@ if ($cloudEnvironment -eq "AzureChinaCloud" -and -not([string]::IsNullOrEmpty($c } $billingEntry = New-Object PSObject -Property @{ - Timestamp = $timestamp - SubscriptionId = $consumptionLine.SubscriptionGuid - ResourceGroup = $rgName - ResourceName = $instanceName - ResourceId = $instanceId - Date = $consumptionLine.Date - Tags = $consumptionLine.Tags - AdditionalInfo = $consumptionLine.AdditionalInfo - BillingCurrencyCode = "CNY" - ChargeType = "Usage" - ConsumedService = $consumptionLine.'Consumed Service' + Timestamp = $timestamp + SubscriptionId = $consumptionLine.SubscriptionGuid + ResourceGroup = $rgName + ResourceName = $instanceName + ResourceId = $instanceId + Date = $consumptionLine.Date + Tags = $consumptionLine.Tags + AdditionalInfo = $consumptionLine.AdditionalInfo + BillingCurrencyCode = "CNY" + ChargeType = "Usage" + ConsumedService = $consumptionLine.'Consumed Service' CostInBillingCurrency = $convertedCost - EffectivePrice = $convertedPrice - Frequency = "UsageBased" - MeterCategory = $consumptionLine.'Meter Category' - MeterId = $consumptionLine.'Meter ID' - MeterName = $consumptionLine.'Meter Name' - MeterSubCategory = $consumptionLine.'Meter Sub-Category' - PartNumber = $partNumber - ProductName = $consumptionLine.Product - Quantity = $consumptionLine.'Consumed Quantity' - UnitOfMeasure = $consumptionLine.'Unit of Measure' - UnitPrice = $unitPrice - ResourceLocation = $consumptionLine.'Resource Location' - AccountOwnerId = $consumptionLine.AccountOwnerId + EffectivePrice = $convertedPrice + Frequency = "UsageBased" + MeterCategory = $consumptionLine.'Meter Category' + MeterId = $consumptionLine.'Meter ID' + MeterName = $consumptionLine.'Meter Name' + MeterSubCategory = $consumptionLine.'Meter Sub-Category' + PartNumber = $partNumber + ProductName = $consumptionLine.Product + Quantity = $consumptionLine.'Consumed Quantity' + UnitOfMeasure = $consumptionLine.'Unit of Measure' + UnitPrice = $unitPrice + ResourceLocation = $consumptionLine.'Resource Location' + AccountOwnerId = $consumptionLine.AccountOwnerId } $billingEntries += $billingEntry @@ -612,7 +621,7 @@ if ($cloudEnvironment -eq "AzureChinaCloud" -and -not([string]::IsNullOrEmpty($c Write-Output "Exported $($billingEntries.Count) entries as CSV to $csvExportPath" $csvBlobName = $csvExportPath - $csvProperties = @{"ContentType" = "text/csv"}; + $csvProperties = @{"ContentType" = "text/csv" } Set-AzStorageBlobContent -File $csvExportPath -Container $storageAccountSinkContainer -Properties $csvProperties -Blob $csvBlobName -Context $saCtx -Force Write-Output "Uploaded to blob storage!" @@ -625,8 +634,8 @@ else { if ($consumptionScope -eq "Subscription") { - $CostDetailsSupportedQuotaIDs = @('EnterpriseAgreement_2014-09-01','Internal_2014-09-01','CSP_2015-05-01') - $ConsumptionSupportedQuotaIDs = @('PayAsYouGo_2014-09-01','MSDN_2014-09-01') + $CostDetailsSupportedQuotaIDs = @('PayAsYouGo_2014-09-01', 'EnterpriseAgreement_2014-09-01', 'Internal_2014-09-01', 'CSP_2015-05-01') + $ConsumptionSupportedQuotaIDs = @('MSDN_2014-09-01') foreach ($subscription in $subscriptions) { @@ -651,12 +660,14 @@ else $requestSuccess = $false do { - try { + try + { $tries++ $consumption = (Invoke-AzRestMethod -Path $ConsumptionApiPath -Method GET).Content | ConvertFrom-Json $requestSuccess = $true } - catch { + catch + { $ErrorMessage = $_.Exception.Message Write-Warning "Error getting consumption data: $ErrorMessage. $tries of 3 tries. Waiting 60 seconds..." Start-Sleep -s 60 @@ -676,126 +687,136 @@ else $tags = $null } - if ([string]::IsNullOrEmpty($consumptionLine.properties.billingProfileId)) + if ($consumptionLine.kind -eq "legacy") { # legacy consumption schema $billingEntry = New-Object PSObject -Property @{ - Timestamp = $timestamp - AccountName = $consumptionLine.properties.accountName - AccountOwnerId = $consumptionLine.properties.accountOwnerId - AdditionalInfo = $consumptionLine.properties.additionalInfo - benefitId = $consumptionLine.properties.benefitId - benefitName = $consumptionLine.properties.benefitName - BillingAccountId = $consumptionLine.properties.billingAccountId - BillingAccountName = $consumptionLine.properties.billingAccountName - BillingCurrencyCode = $consumptionLine.properties.billingCurrency - BillingPeriodEndDate= $consumptionLine.properties.billingPeriodEndDate - BillingPeriodStartDate= $consumptionLine.properties.billingPeriodStartDate - BillingProfileId = $consumptionLine.properties.billingProfileId - BillingProfileName= $consumptionLine.properties.billingProfileName - ChargeType = $consumptionLine.properties.chargeType - ConsumedService = $consumptionLine.properties.consumedService + Timestamp = $timestamp + AccountName = $consumptionLine.properties.accountName + AccountOwnerId = $consumptionLine.properties.accountOwnerId + AdditionalInfo = $consumptionLine.properties.additionalInfo + benefitId = $consumptionLine.properties.benefitId + benefitName = $consumptionLine.properties.benefitName + BillingAccountId = $consumptionLine.properties.billingAccountId + BillingAccountName = $consumptionLine.properties.billingAccountName + BillingCurrencyCode = $consumptionLine.properties.billingCurrency + BillingPeriodEndDate = $consumptionLine.properties.billingPeriodEndDate + BillingPeriodStartDate = $consumptionLine.properties.billingPeriodStartDate + BillingProfileId = $consumptionLine.properties.billingProfileId + BillingProfileName = $consumptionLine.properties.billingProfileName + ChargeType = $consumptionLine.properties.chargeType + ConsumedService = $consumptionLine.properties.consumedService CostAllocationRuleName = $consumptionLine.properties.costAllocationRuleName - CostCenter = $consumptionLine.properties.costCenter - CostInBillingCurrency = $consumptionLine.properties.cost - Date = (Get-Date $consumptionLine.properties.date).ToString("MM/dd/yyyy") - EffectivePrice = $consumptionLine.properties.effectivePrice - Frequency = $consumptionLine.properties.frequency - InvoiceSectionName = $consumptionLine.properties.invoiceSection - IsAzureCreditEligible = $consumptionLine.properties.isAzureCreditEligible - MeterCategory = $consumptionLine.properties.meterDetails.meterCategory - MeterId = $consumptionLine.properties.meterId - MeterName = $consumptionLine.properties.meterDetails.meterName - MeterRegion = $consumptionLine.properties.meterDetails.meterRegion - MeterSubCategory = $consumptionLine.properties.meterDetails.meterSubCategory - OfferId = $consumptionLine.properties.offerId - PartNumber = $consumptionLine.properties.partNumber - PayGPrice = $consumptionLine.properties.PayGPrice - PlanName = $consumptionLine.properties.planName - PricingModel = $consumptionLine.properties.pricingModel - ProductName = $consumptionLine.properties.product - PublisherName = $consumptionLine.properties.publisherName - PublisherType = $consumptionLine.properties.publisherType - Quantity = $consumptionLine.properties.quantity - ReservationId = $consumptionLine.properties.reservationId - ReservationName = $consumptionLine.properties.reservationName - ResourceGroup = $consumptionLine.properties.resourceGroup - ResourceId = $consumptionLine.properties.resourceId - ResourceLocation = $consumptionLine.properties.resourceLocation - ResourceName = $consumptionLine.properties.resourceName - ServiceFamily = $consumptionLine.properties.meterDetails.serviceFamily - SubscriptionId = $consumptionLine.properties.subscriptionId - SubscriptionName = $consumptionLine.properties.subscriptionName - Tags = $tags - Term = $consumptionLine.properties.term - UnitOfMeasure = $consumptionLine.properties.meterDetails.unitOfMeasure - UnitPrice = $consumptionLine.properties.unitPrice + CostCenter = $consumptionLine.properties.costCenter + CostInBillingCurrency = $consumptionLine.properties.cost + Date = (Get-Date $consumptionLine.properties.date).ToString("MM/dd/yyyy") + EffectivePrice = $consumptionLine.properties.effectivePrice + Frequency = $consumptionLine.properties.frequency + InvoiceSectionName = $consumptionLine.properties.invoiceSection + IsAzureCreditEligible = $consumptionLine.properties.isAzureCreditEligible + MeterCategory = $consumptionLine.properties.meterDetails.meterCategory + MeterId = $consumptionLine.properties.meterId + MeterName = $consumptionLine.properties.meterDetails.meterName + MeterRegion = $consumptionLine.properties.meterDetails.meterRegion + MeterSubCategory = $consumptionLine.properties.meterDetails.meterSubCategory + OfferId = $consumptionLine.properties.offerId + PartNumber = $consumptionLine.properties.partNumber + PayGPrice = $consumptionLine.properties.PayGPrice + PlanName = $consumptionLine.properties.planName + PricingModel = $consumptionLine.properties.pricingModel + ProductName = $consumptionLine.properties.product + PublisherName = $consumptionLine.properties.publisherName + PublisherType = $consumptionLine.properties.publisherType + Quantity = $consumptionLine.properties.quantity + ReservationId = $consumptionLine.properties.reservationId + ReservationName = $consumptionLine.properties.reservationName + ResourceGroup = $consumptionLine.properties.resourceGroup + ResourceId = $consumptionLine.properties.resourceId + ResourceLocation = $consumptionLine.properties.resourceLocation + ResourceName = $consumptionLine.properties.resourceName + ServiceFamily = $consumptionLine.properties.meterDetails.serviceFamily + SubscriptionId = $consumptionLine.properties.subscriptionId + SubscriptionName = $consumptionLine.properties.subscriptionName + Tags = $tags + Term = $consumptionLine.properties.term + UnitOfMeasure = $consumptionLine.properties.meterDetails.unitOfMeasure + UnitPrice = $consumptionLine.properties.unitPrice } } - else + elseif ($consumptionLine.kind -eq "modern") { # MCA consumption schema $billingEntry = New-Object PSObject -Property @{ - Timestamp = $timestamp - AdditionalInfo = $consumptionLine.properties.additionalInfo - benefitId = $consumptionLine.properties.benefitId - benefitName = $consumptionLine.properties.benefitName - BillingAccountId = $consumptionLine.properties.billingAccountId - BillingAccountName = $consumptionLine.properties.billingAccountName - BillingCurrencyCode = $consumptionLine.properties.billingCurrencyCode - BillingPeriodEndDate= $consumptionLine.properties.billingPeriodEndDate - BillingPeriodStartDate= $consumptionLine.properties.billingPeriodStartDate - BillingProfileId = $consumptionLine.properties.billingProfileId - BillingProfileName= $consumptionLine.properties.billingProfileName - ChargeType = $consumptionLine.properties.chargeType - ConsumedService = $consumptionLine.properties.consumedService - CostAllocationRuleName = $consumptionLine.properties.costAllocationRuleName - CostCenter = $consumptionLine.properties.costCenter - CostInBillingCurrency = $consumptionLine.properties.costInBillingCurrency - costInPricingCurrency = $consumptionLine.properties.costInPricingCurrency - costInUSD = $consumptionLine.properties.costInUSD - customerName = $consumptionLine.properties.customerName - Date = (Get-Date $consumptionLine.properties.date).ToString("MM/dd/yyyy") - EffectivePrice = $consumptionLine.properties.effectivePrice - exchangeRate = $consumptionLine.properties.exchangeRate - exchangeRateDate = $consumptionLine.properties.exchangeRateDate + Timestamp = $timestamp + AdditionalInfo = $consumptionLine.properties.additionalInfo + benefitId = $consumptionLine.properties.benefitId + benefitName = $consumptionLine.properties.benefitName + BillingAccountId = $consumptionLine.properties.billingAccountId + BillingAccountName = $consumptionLine.properties.billingAccountName + BillingCurrencyCode = $consumptionLine.properties.billingCurrencyCode + BillingPeriodEndDate = $consumptionLine.properties.billingPeriodEndDate + BillingPeriodStartDate = $consumptionLine.properties.billingPeriodStartDate + BillingProfileId = $consumptionLine.properties.billingProfileId + BillingProfileName = $consumptionLine.properties.billingProfileName + ChargeType = $consumptionLine.properties.chargeType + ConsumedService = $consumptionLine.properties.consumedService + CostAllocationRuleName = $consumptionLine.properties.costAllocationRuleName + CostCenter = $consumptionLine.properties.costCenter + CostInBillingCurrency = $consumptionLine.properties.costInBillingCurrency + costInPricingCurrency = $consumptionLine.properties.costInPricingCurrency + costInUSD = $consumptionLine.properties.costInUSD + customerName = $consumptionLine.properties.customerName + Date = (Get-Date $consumptionLine.properties.date).ToString("MM/dd/yyyy") + EffectivePrice = $consumptionLine.properties.effectivePrice + exchangeRate = $consumptionLine.properties.exchangeRate + exchangeRateDate = $consumptionLine.properties.exchangeRateDate exchangeRatePricingToBilling = $consumptionLine.properties.exchangeRatePricingToBilling - Frequency = $consumptionLine.properties.frequency - invoiceSectionId = $consumptionLine.properties.invoiceSectionId - InvoiceSectionName = $consumptionLine.properties.invoiceSectionName - IsAzureCreditEligible = $consumptionLine.properties.isAzureCreditEligible - MeterCategory = $consumptionLine.properties.meterCategory - MeterId = $consumptionLine.properties.meterId - MeterName = $consumptionLine.properties.meterName - MeterRegion = $consumptionLine.properties.meterRegion - MeterSubCategory = $consumptionLine.properties.meterSubCategory - PartNumber = $consumptionLine.properties.partNumber - paygCostInBillingCurrency = $consumptionLine.properties.paygCostInBillingCurrency - paygCostInUSD = $consumptionLine.properties.paygCostInUSD - PayGPrice = $consumptionLine.properties.payGPrice - PlanName = $consumptionLine.properties.planName - pricingCurrencyCode = $consumptionLine.properties.pricingCurrencyCode - PricingModel = $consumptionLine.properties.pricingModel - ProductName = $consumptionLine.properties.product - productIdentifier = $consumptionLine.properties.productIdentifier - PublisherName = $consumptionLine.properties.publisherName - PublisherType = $consumptionLine.properties.publisherType - Quantity = $consumptionLine.properties.quantity - ReservationId = $consumptionLine.properties.reservationId - ReservationName = $consumptionLine.properties.reservationName - ResourceGroup = $consumptionLine.properties.resourceGroup - ResourceId = $consumptionLine.properties.instanceName - ResourceLocation = $consumptionLine.properties.resourceLocation - resourceLocationNormalized = $consumptionLine.properties.resourceLocationNormalized - ServiceFamily = $consumptionLine.properties.serviceFamily - SubscriptionId = $consumptionLine.properties.subscriptionGuid - SubscriptionName = $consumptionLine.properties.subscriptionName - Tags = $tags - Term = $consumptionLine.properties.term - UnitOfMeasure = $consumptionLine.properties.unitOfMeasure - UnitPrice = $consumptionLine.properties.unitPrice + Frequency = $consumptionLine.properties.frequency + invoiceSectionId = $consumptionLine.properties.invoiceSectionId + InvoiceSectionName = $consumptionLine.properties.invoiceSectionName + IsAzureCreditEligible = $consumptionLine.properties.isAzureCreditEligible + MeterCategory = $consumptionLine.properties.meterCategory + MeterId = $consumptionLine.properties.meterId + MeterName = $consumptionLine.properties.meterName + MeterRegion = $consumptionLine.properties.meterRegion + MeterSubCategory = $consumptionLine.properties.meterSubCategory + PartNumber = $consumptionLine.properties.partNumber + paygCostInBillingCurrency = $consumptionLine.properties.paygCostInBillingCurrency + paygCostInUSD = $consumptionLine.properties.paygCostInUSD + PayGPrice = $consumptionLine.properties.payGPrice + PlanName = $consumptionLine.properties.planName + pricingCurrencyCode = $consumptionLine.properties.pricingCurrencyCode + PricingModel = $consumptionLine.properties.pricingModel + ProductName = $consumptionLine.properties.product + productIdentifier = $consumptionLine.properties.productIdentifier + PublisherName = $consumptionLine.properties.publisherName + PublisherType = $consumptionLine.properties.publisherType + Quantity = $consumptionLine.properties.quantity + ReservationId = $consumptionLine.properties.reservationId + ReservationName = $consumptionLine.properties.reservationName + ResourceGroup = $consumptionLine.properties.resourceGroup + ResourceId = $consumptionLine.properties.instanceName + ResourceLocation = $consumptionLine.properties.resourceLocation + resourceLocationNormalized = $consumptionLine.properties.resourceLocationNormalized + ServiceFamily = $consumptionLine.properties.serviceFamily + SubscriptionId = $consumptionLine.properties.subscriptionGuid + SubscriptionName = $consumptionLine.properties.subscriptionName + Tags = $tags + Term = $consumptionLine.properties.term + UnitOfMeasure = $consumptionLine.properties.unitOfMeasure + UnitPrice = $consumptionLine.properties.unitPrice + } + } + else + { + $global:hadErrors = $true + $global:scopesWithErrors += $ScopeName + if (-not($global:scopesWithErrors -contains ($ScopeName))) + { + Write-Warning "Unknown consumption line kind: $($consumptionLine.kind)" } + continue } $billingEntries += $billingEntry } @@ -822,7 +843,7 @@ else $billingEntries | Export-Csv -Path $csvExportPath -NoTypeInformation $csvBlobName = $csvExportPath - $csvProperties = @{"ContentType" = "text/csv"}; + $csvProperties = @{"ContentType" = "text/csv" } Set-AzStorageBlobContent -File $csvExportPath -Container $storageAccountSinkContainer -Properties $csvProperties -Blob $csvBlobName -Context $saCtx -Force $now = (Get-Date).ToUniversalTime().ToString("yyyy'-'MM'-'dd'T'HH':'mm':'ss'.'fff'Z'") @@ -858,7 +879,7 @@ else if ($consumptionScope -eq "BillingAccount") { "Starting cost details export process from $targetStartDate to $targetEndDate for Billing Account ID $BillingAccountID..." - Generate-CostDetails -ScopeId "/providers/Microsoft.Billing/billingAccounts/$BillingAccountID" -ScopeName $BillingAccountID.Replace(":","_") + Generate-CostDetails -ScopeId "/providers/Microsoft.Billing/billingAccounts/$BillingAccountID" -ScopeName $BillingAccountID.Replace(":", "_") } if ($consumptionScope -eq "BillingProfile") { From b0e282196ae306e7b80508179c39d6c8313be70a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?H=C3=A9lder=20Pinto?= Date: Tue, 2 Jun 2026 22:22:58 +0100 Subject: [PATCH 18/30] Schema fixes --- src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 b/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 index 23649cb3c..e2887d363 100644 --- a/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 +++ b/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 @@ -380,7 +380,7 @@ $tableSchemas = @{ @{ name = "Keys_s"; type = "string" } @{ name = "PrincipalNames_s"; type = "string" } @{ name = "Owners_s"; type = "string" } - @{ name = "CreatedDate_s"; type = "string" } + @{ name = "CreatedDate_t"; type = "datetime" } @{ name = "DeletedDate_s"; type = "string" } ) @@ -851,8 +851,8 @@ $tableSchemas = @{ @{ name = "FilterType_s"; type = "string" } @{ name = "InstanceId_g"; type = "string" } @{ name = "InstanceName_s"; type = "string" } - @{ name = "FilterStartDate_s"; type = "string" } - @{ name = "FilterEndDate_s"; type = "string" } + @{ name = "FilterStartDate_t"; type = "datetime" } + @{ name = "FilterEndDate_t"; type = "datetime" } @{ name = "Author_s"; type = "string" } @{ name = "Notes_s"; type = "string" } ) From 4d1f14153939f096226a112aa130f0267cb9d39d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?H=C3=A9lder=20Pinto?= Date: Wed, 3 Jun 2026 09:49:56 +0100 Subject: [PATCH 19/30] Pumped up runbook versions --- src/optimization-engine/azuredeploy-nested.bicep | 10 +++++----- src/optimization-engine/upgrade-manifest.json | 4 ++-- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/src/optimization-engine/azuredeploy-nested.bicep b/src/optimization-engine/azuredeploy-nested.bicep index e6ed4196a..a3df8afe2 100644 --- a/src/optimization-engine/azuredeploy-nested.bicep +++ b/src/optimization-engine/azuredeploy-nested.bicep @@ -949,7 +949,7 @@ var runbooks = [ } { name: consumptionExportsRunbookName - version: '2.1.1.0' + version: '2.1.2.0' description: 'Exports Azure Consumption events to Blob Storage using Azure Consumption API' type: 'PowerShell' scriptUri: uri(templateLocation, 'runbooks/data-collection/${consumptionExportsRunbookName}.ps1') @@ -1075,7 +1075,7 @@ var runbooks = [ } { name: csvIngestRunbookName - version: '1.6.2.0' + version: '2.0.0.0' description: 'Ingests CSV blobs as custom logs to Log Analytics' type: 'PowerShell' scriptUri: uri(templateLocation, 'runbooks/data-collection/${csvIngestRunbookName}.ps1') @@ -1187,21 +1187,21 @@ var runbooks = [ } { name: recommendationsIngestRunbookName - version: '1.7.1.0' + version: '1.7.2.0' description: 'Ingests JSON-based recommendations into an Azure SQL Database' type: 'PowerShell' scriptUri: uri(templateLocation, 'runbooks/recommendations/${recommendationsIngestRunbookName}.ps1') } { name: recommendationsLogAnalyticsIngestRunbookName - version: '1.1.1.0' + version: '2.0.0.0' description: 'Ingests JSON-based recommendations into Log Analytics' type: 'PowerShell' scriptUri: uri(templateLocation, 'runbooks/recommendations/${recommendationsLogAnalyticsIngestRunbookName}.ps1') } { name: suppressionsLogAnalyticsIngestRunbookName - version: '1.1.0.0' + version: '2.0.0.0' description: 'Ingests suppressions into Log Analytics' type: 'PowerShell' scriptUri: uri(templateLocation, 'runbooks/recommendations/${suppressionsLogAnalyticsIngestRunbookName}.ps1') diff --git a/src/optimization-engine/upgrade-manifest.json b/src/optimization-engine/upgrade-manifest.json index de27e8cce..2cb53a951 100644 --- a/src/optimization-engine/upgrade-manifest.json +++ b/src/optimization-engine/upgrade-manifest.json @@ -311,7 +311,7 @@ { "runbook": { "name": "runbooks/recommendations/Ingest-RecommendationsToSQLServer.ps1", - "version": "1.7.1.0" + "version": "1.7.2.0" }, "source": "recommendations", "schedule": "AzureOptimization_IngestRecommendationsWeekly" @@ -521,7 +521,7 @@ { "runbook": { "name": "runbooks/data-collection/Export-ConsumptionToBlobStorage.ps1", - "version": "2.1.1.0" + "version": "2.1.2.0" }, "container": "consumptionexports", "requiredVariables": [ From 8e467a64f4af5204a0452ee2b7f5549dfb3c4aab Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?H=C3=A9lder=20Pinto?= Date: Wed, 3 Jun 2026 12:10:17 +0100 Subject: [PATCH 20/30] Added support for DCE creation in upgrade scenarios and anticipated price ingests --- .../Deploy-AzureOptimizationEngine.ps1 | 28 ++++++++++++++++++- .../azuredeploy-nested.bicep | 6 ++-- src/optimization-engine/upgrade-manifest.json | 6 ++-- 3 files changed, 33 insertions(+), 7 deletions(-) diff --git a/src/optimization-engine/Deploy-AzureOptimizationEngine.ps1 b/src/optimization-engine/Deploy-AzureOptimizationEngine.ps1 index 536960ef1..d6135cf7c 100644 --- a/src/optimization-engine/Deploy-AzureOptimizationEngine.ps1 +++ b/src/optimization-engine/Deploy-AzureOptimizationEngine.ps1 @@ -1124,6 +1124,32 @@ if ("Y", "y" -contains $continueInput) #endregion } + #region Ensuring Data Collection Endpoint was created (in case of partial upgrade) + $dceName = "$automationAccountName-dce" + $dce = Get-AzDataCollectionEndpoint -ResourceGroupName $ResourceGroupName -Name $dceName -ErrorAction SilentlyContinue + if ($null -eq $dce) + { + Write-Host "Creating Data Collection Endpoint $dceName..." -ForegroundColor Green + $dce = New-AzDataCollectionEndpoint -ResourceGroupName $ResourceGroupName -Name $dceName -Location $targetLocation ` + -Tag $resourceTags -NetworkAclsPublicNetworkAccess Enabled + } + if ($null -ne $dce.LogIngestionEndpoint) + { + $dceIngestionEndpointVariableName = "AzureOptimization_DCEIngestionEndpoint" + $dceIngestionEndpointVariable = Get-AzAutomationVariable -ResourceGroupName $resourceGroupName -AutomationAccountName $automationAccountName -Name $dceIngestionEndpointVariableName -ErrorAction SilentlyContinue + if ($null -eq $dceIngestionEndpointVariable) + { + Write-Host "Creating Automation Variable with Data Collection Endpoint ingestion endpoint..." -ForegroundColor Green + New-AzAutomationVariable -Name $dceIngestionEndpointVariableName -ResourceGroupName $resourceGroupName -AutomationAccountName $automationAccountName -Value $dce.LogIngestionEndpoint -Encrypted $false | Out-Null + } + } + else + { + throw "Data Collection Endpoint $dceName does not have a log ingestion endpoint yet. Please, check it was created successfully." + } + #endregion + + #region Schedules reset if ($upgradingSchedules) { @@ -1168,7 +1194,7 @@ if ("Y", "y" -contains $continueInput) $deploymentDate = (Get-Date).ToUniversalTime().ToString("yyyy-MM-dd") Write-Host "Setting initial deployment date ($deploymentDate)..." -ForegroundColor Green New-AzAutomationVariable -Name $deploymentDateVariableName -Description "The date of the initial engine deployment" ` - -ResourceGroupName $resourceGroupName -AutomationAccountName $automationAccountName -Value $deploymentDate -Encrypted $false + -ResourceGroupName $resourceGroupName -AutomationAccountName $automationAccountName -Value $deploymentDate -Encrypted $false | Out-Null } #endregion diff --git a/src/optimization-engine/azuredeploy-nested.bicep b/src/optimization-engine/azuredeploy-nested.bicep index a3df8afe2..8b5cb755b 100644 --- a/src/optimization-engine/azuredeploy-nested.bicep +++ b/src/optimization-engine/azuredeploy-nested.bicep @@ -257,7 +257,7 @@ var csvExportsSchedules = [ { exportSchedule: priceExportsScheduleName exportDescription: 'Weekly Pricesheet and Reservation Prices exports' - exportTimeOffset: 'PT1H35M' + exportTimeOffset: 'PT1H05M' exportFrequency: 'Week' } { @@ -562,7 +562,7 @@ var csvExports = [ variableDescription: 'The Storage Account container where Pricesheet exports are dumped to' ingestSchedule: 'AzureOptimization_IngestPricesheetWeekly' ingestDescription: 'Weekly Pricesheet ingests' - ingestTimeOffset: 'PT2H' + ingestTimeOffset: 'PT1H35M' ingestFrequency: 'Week' ingestJobId: pricesheetIngestJobId exportSchedule: priceExportsScheduleName @@ -576,7 +576,7 @@ var csvExports = [ variableDescription: 'The Storage Account container where Reservations Prices exports are dumped to' ingestSchedule: 'AzureOptimization_IngestReservationsPriceWeekly' ingestDescription: 'Weekly Reservations Prices ingests' - ingestTimeOffset: 'PT2H' + ingestTimeOffset: 'PT1H35M' ingestFrequency: 'Week' ingestJobId: reservationPricesIngestJobId exportSchedule: priceExportsScheduleName diff --git a/src/optimization-engine/upgrade-manifest.json b/src/optimization-engine/upgrade-manifest.json index 2cb53a951..3b0b1f339 100644 --- a/src/optimization-engine/upgrade-manifest.json +++ b/src/optimization-engine/upgrade-manifest.json @@ -266,17 +266,17 @@ }, { "name": "AzureOptimization_ExportPricesWeekly", - "offset": "PT1H35M", + "offset": "PT1H05M", "frequency": "Week" }, { "name": "AzureOptimization_IngestPricesheetWeekly", - "offset": "PT2H10M", + "offset": "PT1H35M", "frequency": "Week" }, { "name": "AzureOptimization_IngestReservationsPriceWeekly", - "offset": "PT2H10M", + "offset": "PT1H35M", "frequency": "Week" }, { From 46140d75b1bf6bef2251466ef0073acecb1be105 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?H=C3=A9lder=20Pinto?= Date: Wed, 3 Jun 2026 14:36:55 +0100 Subject: [PATCH 21/30] Schema fixes --- src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 b/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 index e2887d363..5677e749c 100644 --- a/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 +++ b/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 @@ -320,6 +320,8 @@ $tableSchemas = @{ @{ name = "TimeGenerated"; type = "datetime" } @{ name = "Timestamp_s"; type = "string" } @{ name = "AdditionalInfo_s"; type = "string" } + @{ name = "benefitId_s"; type = "string" } + @{ name = "benefitName_s"; type = "string" } @{ name = "BillingAccountId_s"; type = "string" } @{ name = "BillingAccountName_s"; type = "string" } @{ name = "BillingCurrencyCode_s"; type = "string" } @@ -716,7 +718,7 @@ $tableSchemas = @{ @{ name = "DisplayName_s"; type = "string" } @{ name = "SKUName_s"; type = "string" } @{ name = "Location_s"; type = "string" } - @{ name = "ResourceType_s"; type = "string" } + @{ name = "ResourceType"; type = "string" } @{ name = "AppliedScopeType_s"; type = "string" } @{ name = "Term_s"; type = "string" } @{ name = "ProvisioningState_s"; type = "string" } From be627254d6a37ec124f64b31212fae7c2c903f8a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?H=C3=A9lder=20Pinto?= Date: Thu, 18 Jun 2026 10:38:15 +0100 Subject: [PATCH 22/30] Improved support for UAMI scenarios --- .../Deploy-AzureOptimizationEngine.ps1 | 165 ++++++++++-------- .../Setup-LogAnalyticsTablesAndDCRs.ps1 | 43 +++-- 2 files changed, 115 insertions(+), 93 deletions(-) diff --git a/src/optimization-engine/Deploy-AzureOptimizationEngine.ps1 b/src/optimization-engine/Deploy-AzureOptimizationEngine.ps1 index d6135cf7c..a69fd685b 100644 --- a/src/optimization-engine/Deploy-AzureOptimizationEngine.ps1 +++ b/src/optimization-engine/Deploy-AzureOptimizationEngine.ps1 @@ -1212,12 +1212,24 @@ if ("Y", "y" -contains $continueInput) $sa = Get-AzStorageAccount -ResourceGroupName $resourceGroupName -Name $storageAccountName $auto = Get-AzAutomationAccount -Name $automationAccountName -ResourceGroupName $resourceGroupName $spnId = $auto.Identity.PrincipalId - $storageBlobContributorRoleId = "ba92f5b4-2d11-453d-a403-e96b0029c9fe" # Storage Blob Data Contributor role - $roleAssignment = Get-AzRoleAssignment -ObjectId $spnId -Scope $sa.Id -ErrorAction SilentlyContinue - if (-not($roleAssignment) -or -not($roleAssignment.RoleDefinitionId -contains $storageBlobContributorRoleId)) + + if ($spnId) + { + $storageBlobContributorRoleId = "ba92f5b4-2d11-453d-a403-e96b0029c9fe" # Storage Blob Data Contributor role + $roleAssignment = Get-AzRoleAssignment -ObjectId $spnId -Scope $sa.Id -ErrorAction SilentlyContinue + if (-not($roleAssignment) -or -not($roleAssignment.RoleDefinitionId -contains $storageBlobContributorRoleId)) + { + Write-Host "Granting Storage Blob Data Contributor role to the Automation Account identity..." -ForegroundColor Green + New-AzRoleAssignment -ObjectId $spnId -RoleDefinitionId $storageBlobContributorRoleId -Scope $sa.Id | Out-Null + } + else + { + Write-Host "Automation Account identity already has the Storage Blob Data Contributor role at the Storage Account level." -ForegroundColor Green + } + } + else { - Write-Host "Granting Storage Blob Data Contributor role to the Automation Account identity..." -ForegroundColor Green - New-AzRoleAssignment -ObjectId $spnId -RoleDefinitionId $storageBlobContributorRoleId -Scope $sa.Id | Out-Null + Write-Host "Automation Account does not have a system-assigned identity. Manually grant the Storage Blob Data Contributor role to the managed identity used by AOE on the $($sa.Id) resource." -ForegroundColor Yellow } #region Open SQL Server firewall rule @@ -1359,17 +1371,25 @@ if ("Y", "y" -contains $continueInput) $Cmd.ExecuteReader() $Conn.Close() - $Conn = New-Object System.Data.SqlClient.SqlConnection("Server=tcp:$sqlServerEndpoint,1433;Database=$databaseName;Encrypt=True;Connection Timeout=$SqlTimeout;") - $Conn.AccessToken = $dbToken - $Conn.Open() - - $createUserQuery = (Get-Content -Path "./model/automation-user.sql").Replace("", $automationAccountName) - $Cmd = New-Object system.Data.SqlClient.SqlCommand - $Cmd.Connection = $Conn - $Cmd.CommandTimeout = $SqlTimeout - $Cmd.CommandText = $createUserQuery - $Cmd.ExecuteReader() - $Conn.Close() + if ($spnId) + { + $Conn = New-Object System.Data.SqlClient.SqlConnection("Server=tcp:$sqlServerEndpoint,1433;Database=$databaseName;Encrypt=True;Connection Timeout=$SqlTimeout;") + $Conn.AccessToken = $dbToken + $Conn.Open() + + $createUserQuery = (Get-Content -Path "./model/automation-user.sql").Replace("", $automationAccountName) + $Cmd = New-Object system.Data.SqlClient.SqlCommand + $Cmd.Connection = $Conn + $Cmd.CommandTimeout = $SqlTimeout + $Cmd.CommandText = $createUserQuery + $Cmd.ExecuteReader() + $Conn.Close() + } + else + { + Write-Host "Automation Account does not have a system-assigned identity. Please, create a SQL user for the identity used by AOE with db_datareader and db_datawriter roles on the $databaseName database." -ForegroundColor Yellow + Write-Host "You can use the T-SQL script in model/automation-user.sql as a template, replacing with the name of the AOE Managed Identity." -ForegroundColor Yellow + } $connectionSuccess = $true } @@ -1456,95 +1476,86 @@ if ("Y", "y" -contains $continueInput) if (!$silentDeploy) { #region Grant Microsoft Entra ID role to AOE principal - if ($null -eq $spnId) + if ($spnId) { - $auto = Get-AzAutomationAccount -Name $automationAccountName -ResourceGroupName $resourceGroupName - $spnId = $auto.Identity.PrincipalId - if ($null -eq $spnId) + try { - $runAsConnection = Get-AzAutomationConnection -ResourceGroupName $resourceGroupName -AutomationAccountName $automationAccountName -Name AzureRunAsConnection -ErrorAction SilentlyContinue - $runAsAppId = $runAsConnection.FieldDefinitionValues.ApplicationId - if ($runAsAppId) - { - $runAsServicePrincipal = Get-AzADServicePrincipal -ApplicationId $runAsAppId - $spnId = $runAsServicePrincipal.Id - } - } - } - - try - { - Import-Module Microsoft.Graph.Authentication - Import-Module Microsoft.Graph.Identity.DirectoryManagement + Import-Module Microsoft.Graph.Authentication + Import-Module Microsoft.Graph.Identity.DirectoryManagement - Write-Host "Granting Microsoft Entra ID Global Reader role to the Automation Account (requires administrative permissions in Microsoft Entra and MS Graph PowerShell SDK >= 2.4.0)..." -ForegroundColor Green + Write-Host "Granting Microsoft Entra ID Global Reader role to the Automation Account (requires administrative permissions in Microsoft Entra and MS Graph PowerShell SDK >= 2.4.0)..." -ForegroundColor Green - #workaround for https://github.com/microsoftgraph/msgraph-sdk-powershell/issues/888 - $localPath = [System.Environment]::GetFolderPath([System.Environment+SpecialFolder]::UserProfile) - if (-not(Get-Item "$localPath\.graph\" -ErrorAction SilentlyContinue)) - { - New-Item -Type Directory "$localPath\.graph" - } - - switch ($cloudEnvironment) - { - "AzureUSGovernment" - { - $graphEnvironment = "USGov" - break - } - "AzureChinaCloud" - { - $graphEnvironment = "China" - break - } - "AzureGermanCloud" + #workaround for https://github.com/microsoftgraph/msgraph-sdk-powershell/issues/888 + $localPath = [System.Environment]::GetFolderPath([System.Environment+SpecialFolder]::UserProfile) + if (-not(Get-Item "$localPath\.graph\" -ErrorAction SilentlyContinue)) { - $graphEnvironment = "Germany" - break + New-Item -Type Directory "$localPath\.graph" } - default + + switch ($cloudEnvironment) { - $graphEnvironment = "Global" + "AzureUSGovernment" + { + $graphEnvironment = "USGov" + break + } + "AzureChinaCloud" + { + $graphEnvironment = "China" + break + } + "AzureGermanCloud" + { + $graphEnvironment = "Germany" + break + } + default + { + $graphEnvironment = "Global" + } } - } - Connect-MgGraph -Scopes "RoleManagement.ReadWrite.Directory", "Directory.Read.All" -UseDeviceAuthentication -Environment $graphEnvironment -NoWelcome + Connect-MgGraph -Scopes "RoleManagement.ReadWrite.Directory", "Directory.Read.All" -UseDeviceAuthentication -Environment $graphEnvironment -NoWelcome - $globalReaderRole = Get-MgDirectoryRole -ExpandProperty Members -Property Id, Members, DisplayName, RoleTemplateId ` - | Where-Object { $_.RoleTemplateId -eq "f2ef992c-3afb-46b9-b7cf-a126ee74c451" } - $globalReaders = $globalReaderRole.Members.Id - if (-not($globalReaders -contains $spnId)) - { - New-MgDirectoryRoleMemberByRef -DirectoryRoleId $globalReaderRole.Id -BodyParameter @{"@odata.id" = "https://graph.microsoft.com/v1.0/directoryObjects/$spnId" } - Start-Sleep -Seconds 5 $globalReaderRole = Get-MgDirectoryRole -ExpandProperty Members -Property Id, Members, DisplayName, RoleTemplateId ` | Where-Object { $_.RoleTemplateId -eq "f2ef992c-3afb-46b9-b7cf-a126ee74c451" } $globalReaders = $globalReaderRole.Members.Id - if ($globalReaders -contains $spnId) + if (-not($globalReaders -contains $spnId)) { - Write-Host "Role granted." -ForegroundColor Green + New-MgDirectoryRoleMemberByRef -DirectoryRoleId $globalReaderRole.Id -BodyParameter @{"@odata.id" = "https://graph.microsoft.com/v1.0/directoryObjects/$spnId" } + Start-Sleep -Seconds 5 + $globalReaderRole = Get-MgDirectoryRole -ExpandProperty Members -Property Id, Members, DisplayName, RoleTemplateId ` + | Where-Object { $_.RoleTemplateId -eq "f2ef992c-3afb-46b9-b7cf-a126ee74c451" } + $globalReaders = $globalReaderRole.Members.Id + if ($globalReaders -contains $spnId) + { + Write-Host "Role granted." -ForegroundColor Green + } + else + { + throw "Error when trying to grant Global Reader role" + } } else { - throw "Error when trying to grant Global Reader role" + Write-Host "Role was already granted before." -ForegroundColor Green } } - else + catch { - Write-Host "Role was already granted before." -ForegroundColor Green + Write-Host $Error[0] -ForegroundColor Yellow + Write-Host "Could not grant role. If you want Microsoft Entra-based recommendations, please grant the Global Reader role manually to the $automationAccountName managed identity or, for previous versions of AOE, to the Run As Account principal." -ForegroundColor Red } } - catch + else { - Write-Host $Error[0] -ForegroundColor Yellow - Write-Host "Could not grant role. If you want Microsoft Entra-based recommendations, please grant the Global Reader role manually to the $automationAccountName managed identity or, for previous versions of AOE, to the Run As Account principal." -ForegroundColor Red + Write-Host "Automation Account does not have a system-assigned identity. Please grant the Global Reader role manually to the managed identity used by AOE." -ForegroundColor Yellow } #endregion } else { - Write-Host "Could not grant role. If you want Microsoft Entra-based recommendations, please grant the Global Reader role manually to the $automationAccountName managed identity or, for previous versions of AOE, to the Run As Account principal." -ForegroundColor Red + Write-Host "Could not grant role. If you want Microsoft Entra-based recommendations, please grant the Global Reader role manually to the $automationAccountName managed identity." -ForegroundColor Red } Write-Host "Azure Optimization Engine deployment completed! We're almost there..." -ForegroundColor Green diff --git a/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 b/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 index 5677e749c..e4f199973 100644 --- a/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 +++ b/src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1 @@ -158,9 +158,13 @@ $automationAccount = Get-AzAutomationAccount -ResourceGroupName $ResourceGroupNa $automationPrincipalId = $automationAccount.Identity.PrincipalId if ([string]::IsNullOrEmpty($automationPrincipalId)) { - throw "Could not retrieve the managed identity principal ID for the Automation account $AutomationAccountName." + Write-Host "Could not retrieve the managed identity principal ID for the Automation account $AutomationAccountName." -ForegroundColor Yellow + Write-Host "Grant the Metrics Publisher role to the AOE's managed identity on the resource group containing the DCRs after this script completes." -ForegroundColor Yellow +} +else +{ + Write-Host "Retrieved managed identity principal ID for Automation account: $automationPrincipalId" -ForegroundColor Cyan } -Write-Host "Automation account managed identity: $automationPrincipalId" -ForegroundColor Cyan #endregion #region Monitoring Metrics Publisher role definition ID @@ -914,26 +918,33 @@ $dcrSuffixToImmutableId = @{} #region Assign Monitoring Metrics Publisher role at resource group scope Write-Host "Granting Monitoring Metrics Publisher on resource group to Automation MI..." -ForegroundColor Green -# Role assignment must be created in the DCR subscription/resource group context -if ($WorkspaceSubscriptionId -ne $currentSubscriptionId) +if ($automationPrincipalId) { - Set-AzContext -SubscriptionId $currentSubscriptionId | Out-Null -} + # Role assignment must be created in the DCR subscription/resource group context + if ($WorkspaceSubscriptionId -ne $currentSubscriptionId) + { + Set-AzContext -SubscriptionId $currentSubscriptionId | Out-Null + } -$dcrResourceGroupScope = "/subscriptions/$currentSubscriptionId/resourceGroups/$ResourceGroupName" -$existingAssignment = Get-AzRoleAssignment -ObjectId $automationPrincipalId ` - -RoleDefinitionId $monitoringMetricsPublisherRoleId ` - -Scope $dcrResourceGroupScope -ErrorAction SilentlyContinue -if ($null -eq $existingAssignment) -{ - New-AzRoleAssignment -ObjectId $automationPrincipalId ` + $dcrResourceGroupScope = "/subscriptions/$currentSubscriptionId/resourceGroups/$ResourceGroupName" + $existingAssignment = Get-AzRoleAssignment -ObjectId $automationPrincipalId ` -RoleDefinitionId $monitoringMetricsPublisherRoleId ` - -Scope $dcrResourceGroupScope | Out-Null - Write-Host " Role assigned on resource group scope." -ForegroundColor Gray + -Scope $dcrResourceGroupScope -ErrorAction SilentlyContinue + if ($null -eq $existingAssignment) + { + New-AzRoleAssignment -ObjectId $automationPrincipalId ` + -RoleDefinitionId $monitoringMetricsPublisherRoleId ` + -Scope $dcrResourceGroupScope | Out-Null + Write-Host " Role assigned on resource group scope." -ForegroundColor Gray + } + else + { + Write-Host " Role already assigned on resource group scope." -ForegroundColor Gray + } } else { - Write-Host " Role already assigned on resource group scope." -ForegroundColor Gray + Write-Host " Automation MI not found; skipping role assignment." -ForegroundColor Yellow } # Restore workspace subscription context for table operations From 98ffb2207c29e0ade5ddb35b1e362fa186e3ba81 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?H=C3=A9lder=20Pinto?= Date: Wed, 24 Jun 2026 18:02:58 +0100 Subject: [PATCH 23/30] Added note to change log --- docs-mslearn/toolkit/changelog.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs-mslearn/toolkit/changelog.md b/docs-mslearn/toolkit/changelog.md index 30f3c252f..ae81dcf22 100644 --- a/docs-mslearn/toolkit/changelog.md +++ b/docs-mslearn/toolkit/changelog.md @@ -30,6 +30,11 @@ The following section lists features and enhancements that are currently in deve - Added ability to ingest recommendations from Azure Advisor and custom Azure Resource Graph queries. - Custom recommendations include stopped VMs, unattached disks, underutilized resources, and more. +### [Optimization engine](optimization-engine/overview.md) updates + +- **Changed** + - Migrated Log Analytics ingestion from the Data Collection API (to be deprecated in September 2026) to a Data Collection Rule, Ingestion API-based solution. +