AOE: migrate to DCR-based ingestion

[helderpinto]

🛠️ Description

This PR consolidates the past weeks of work on AOE DCR ingestion and hardening.

Main changes:

• Migrated ingestion flow to DCR-based ingestion (Data Collection API is deprecating in September 2026)
• Resolved SQL session management issues
• Improved support for legacy and non-enterprise subscriptions
• Improved support for UAMI scenarios

Why:

• Increase ingestion reliability
• Improve compatibility across subscription and identity scenarios
• Support smoother upgrade paths

📋 Checklist

🔬 How did you test this change?

• 🤏 Lint tests
• 🤞 PS -WhatIf / az validate
• 👍 Manually deployed + verified
• 💪 Unit tests
• 🙌 Integration tests

📦 Deploy to test?

• Hubs + ADX (managed)
• Hubs + Fabric (manual) — URI:
• Hubs (manual)
• Hubs (no data)
• Workbooks
• Alerts

🙋‍♀️ Do any of the following that apply?

• 🚨 This is a breaking change.
• 🤏 The change is less than 20 lines of code.

📑 Did you update docs/changelog.md?

• ✅ Updated changelog (required for dev PRs)
• ➡️ Will add log in a future PR (feature branch PRs only)
• ❎ Log not needed (small/internal change)

📖 Did you update documentation?

• ✅ Public docs in docs (required for dev)
• ✅ Public docs in docs-mslearn (required for dev)
• ✅ Internal dev docs in docs-wiki (required for dev)
• ✅ Internal dev docs in src (required for dev)
• ➡️ Will add docs in a future PR (feature branch PRs only)
• ❎ Docs not needed (small/internal change)

[Copilot]

Pull request overview

This PR advances the Azure Optimization Engine (AOE) migration from the Log Analytics Data Collector API to DCR-based Logs Ingestion by introducing DCR/table provisioning, wiring runbooks to the Logs Ingestion API, and updating deployment artifacts/schema to support the new flow. It also includes additional release-process automation/docs updates (release checklist/template, a Start-Release script, and Claude command/permissions config).

Changes:

• Added DCR/table setup automation (custom table schema + per-table DCR creation) and updated deployment to ensure DCE/DCR prerequisites exist.
• Updated ingestion runbooks and SQL control-table schema to use/store DCR immutable IDs and send data via the Logs Ingestion API.
• Added/updated release process assets (Start-Release script, release issue template, changelog guidance, Claude command/config).

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 14 comments.

Show a summary per file

File	Description
src/scripts/Start-Release.ps1	New script to bootstrap/continue monthly release workflow (issue + milestone + untriaged queries, Bicep CLI update).
src/optimization-engine/upgrade-manifest.json	Updates runbook versions, schedule offsets, and chunk size defaults for the DCR ingestion rollout.
src/optimization-engine/Setup-LogAnalyticsTablesAndDCRs.ps1	New end-to-end provisioning script for LA custom tables + DCRs + SQL control table DCRImmutableId updates.
src/optimization-engine/runbooks/recommendations/Ingest-SuppressionsToLogAnalytics.ps1	Migrates suppressions ingestion to Logs Ingestion API and retrieves DCR immutable ID from SQL.
src/optimization-engine/runbooks/recommendations/Ingest-RecommendationsToSQLServer.ps1	Hardens SQL connection/token lifecycle and adjusts SQL execution patterns.
src/optimization-engine/runbooks/recommendations/Ingest-RecommendationsToLogAnalytics.ps1	Migrates recommendations ingestion to Logs Ingestion API and uses DCRImmutableId from SQL.
src/optimization-engine/runbooks/data-collection/Ingest-OptimizationCSVExportsToLogAnalytics.ps1	Migrates CSV ingestion to Logs Ingestion API and introduces typed conversion for DCR ingestion.
src/optimization-engine/runbooks/data-collection/Export-ConsumptionToBlobStorage.ps1	Refactors/cleans up formatting and extends legacy/modern subscription handling logic.
src/optimization-engine/model/loganalyticsingestcontrol-table.sql	Adds DCRImmutableId column to ingestion control table schema.
src/optimization-engine/model/loganalyticsingestcontrol-initialize.sql	Makes initialization inserts explicit and consistent with schema evolution.
src/optimization-engine/Deploy-AzureOptimizationEngine.ps1	Ensures DCE exists in upgrade scenarios and invokes DCR/table setup during deployment.
src/optimization-engine/azuredeploy-nested.bicep	Adds DCE + automation variable for DCE ingestion endpoint; bumps runbook versions and chunk size.
docs-wiki/Coding-guidelines.md	Documents changelog structure/rules.
docs-mslearn/toolkit/changelog.md	Adds an Optimization engine changelog entry for the ingestion migration.
.github/ISSUE_TEMPLATE/-internal-only--release-checklist.md	Updates internal release checklist template (structure, labels, checklist content).
.claude/settings.json	Updates Claude tool execution allow-list for release/testing workflows.
.claude/commands/release.md	Adds a Claude “release” command to walk through the release checklist.

[flanakin]

🤖 [AI][Claude Code] PR Review

Summary: Solid migration from the deprecating Data Collector (SharedKey) API to DCR-based Logs Ingestion. Bicep builds clean with no new lint warnings, the SQL schema migration is idempotent, function naming uses approved verbs, and the cross-subscription/UAMI/legacy-subscription hardening is well thought out. One data-correctness bug in the CSV ingestion path needs fixing before merge, plus a shared error-handling regression in the new ingestion helper.

🚫 Blockers (1)

1. ConvertTo-TypedObject drops the Timestamp field, so the DCR transform's Timestamp_s = Timestamp projection yields null for every CSV-ingested row across all tables (VMs, Disks, Consumption, etc.).

⚠️ Should fix (1)

1. Send-LogIngestionData returns $response.StatusCode, but Invoke-WebRequest throws on non-2xx responses — so on a failed ingestion the caller's status-code branch is dead code and the runbook terminates without its intended error logging/cleanup. Affects all three ingestion runbooks.

💡 Suggestions (1)

1. Setup-LogAnalyticsTablesAndDCRs.ps1 hardcodes the log-name prefix while the runbooks read it from the AzureOptimization_LogAnalyticsLogPrefix automation variable — table/stream names would mismatch if a user ever customized the prefix.