From 2c7e32ccfc0c75e15343b626db048e6b29410461 Mon Sep 17 00:00:00 2001 From: msbrett Date: Wed, 17 Jun 2026 14:09:50 -0700 Subject: [PATCH 01/51] feat(finops-hub-local): two-DB ftklocal foundation (T-5000.1 + T-5000.2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Local install of FinOps Hubs: runs the toolkit's ADX KQL against a local Kusto Emulator (Kustainer) with the real two-database topology (Ingestion + Hub), so dashboards / Power BI / the query catalog run unchanged against a local endpoint. T-5000.1 — load-ftk-kql.ps1 creates the Ingestion + Hub databases and loads them by combining the toolkit's own finops-hub/.build.config fabric-setup bundles (real v1_0 included; database('Ingestion'). cross-DB references preserved; zero object renames — strict name parity). No hand-coded load order, no strip hack, no v1_0 stubs. T-5000.2 — ingest.ps1 -> Ingestion; ftk.ps1 -> Hub; run-parity-checks.ps1 is DB-aware (raw/final checks on Ingestion, Hub-view checks 9/10 on Hub) and adds a name-parity check. Verified end-to-end on R&D data: Costs and Prices ingested at exact parity (BilledCost preserved to $0.00; Prices raw==final 12,735,587), Hub cross-DB queries resolve, parity suite 10 pass / 0 fail / 2 manual. Prototype docs/notes are intentionally not migrated here yet — they are authored cleanly (sanitized, two-DB-accurate) in the docs task (T-5000.7). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- src/templates/finops-hub-local/.env.example | 21 + src/templates/finops-hub-local/.gitignore | 19 + src/templates/finops-hub-local/Makefile | 153 +++ .../finops-hub-local/docker-compose.yml | 32 + src/templates/finops-hub-local/ftkver.txt | 1 + .../scripts/chunked-prices-backfill.ps1 | 48 + .../finops-hub-local/scripts/ftk.ps1 | 483 ++++++++++ .../finops-hub-local/scripts/ingest.ps1 | 896 ++++++++++++++++++ .../finops-hub-local/scripts/load-ftk-kql.ps1 | 494 ++++++++++ .../scripts/run-parity-checks.ps1 | 514 ++++++++++ .../finops-hub-local/tests/parity-checks.kql | 142 +++ 11 files changed, 2803 insertions(+) create mode 100644 src/templates/finops-hub-local/.env.example create mode 100644 src/templates/finops-hub-local/.gitignore create mode 100644 src/templates/finops-hub-local/Makefile create mode 100644 src/templates/finops-hub-local/docker-compose.yml create mode 100644 src/templates/finops-hub-local/ftkver.txt create mode 100755 src/templates/finops-hub-local/scripts/chunked-prices-backfill.ps1 create mode 100644 src/templates/finops-hub-local/scripts/ftk.ps1 create mode 100755 src/templates/finops-hub-local/scripts/ingest.ps1 create mode 100644 src/templates/finops-hub-local/scripts/load-ftk-kql.ps1 create mode 100644 src/templates/finops-hub-local/scripts/run-parity-checks.ps1 create mode 100644 src/templates/finops-hub-local/tests/parity-checks.kql diff --git a/src/templates/finops-hub-local/.env.example b/src/templates/finops-hub-local/.env.example new file mode 100644 index 000000000..366549e42 --- /dev/null +++ b/src/templates/finops-hub-local/.env.example @@ -0,0 +1,21 @@ +# ftklocal tunables — copy to `.env` and edit as needed. +# `.env` is gitignored; `.env.example` is the documented template. + +# Host port to expose the Kustainer HTTP endpoint on. +# IMPORTANT: do NOT use 8080 — that port is reserved for maenifold (the MCP daemon). +HOST_PORT=8082 + +# Memory ceiling for the Kustainer container. +# +# 16g is the recommended default. The full Prices_transform_v1_2() backfill +# over 32M rows OOM'd at 8g on amd64-emulation (Apple Silicon under Rosetta). +# Costs-only workflows or smaller datasets work at 8g, but reliability across +# both Costs (13M rows) and Prices (32M rows) requires 16g. +MEM_LIMIT=16g + +# Directory on the host that holds the FOCUS parquet exports. +# Mounted read-only at /data/export inside the container. +# Default is ./export (relative to the Makefile / docker-compose.yml directory). +# Point to an absolute path to reuse an existing export without copying files: +# EXPORT_DIR=/Users/you/src/ftklocal/export +EXPORT_DIR=./export diff --git a/src/templates/finops-hub-local/.gitignore b/src/templates/finops-hub-local/.gitignore new file mode 100644 index 000000000..2ae74dd9c --- /dev/null +++ b/src/templates/finops-hub-local/.gitignore @@ -0,0 +1,19 @@ +# Ephemeral state and secrets — never commit. +.gate/ +kustainer-data/ +*.duckdb +.env +.DS_Store +*.log +__pycache__/ +*.pyc + +# Customer data — never commit. The raw FOCUS exports are customer-confidential, +# and so is any analysis derived from them. +export/ +notes/figures/ +notes/*refund* +notes/*sp-* +notes/*savings* +notes/*misalloc* +notes/*unused* diff --git a/src/templates/finops-hub-local/Makefile b/src/templates/finops-hub-local/Makefile new file mode 100644 index 000000000..a08a3bf11 --- /dev/null +++ b/src/templates/finops-hub-local/Makefile @@ -0,0 +1,153 @@ +# ftklocal — Makefile +# +# Targets: +# make up Bring Kustainer up and block until healthcheck passes. +# make down Stop and remove the container (keeps the data volume). +# make nuke Down + remove kustainer-data/ (destructive; full reset). +# make logs Tail Kustainer logs. +# make kql QUERY='...' Run a one-shot KQL query (default DB: Ingestion; override DB=Hub). +# +# DB topology: +# Ingestion raw tables (Costs_raw, Prices_raw), transform functions, final tables, +# open-data lookups, and Ingest_Manifest. +# Used by: ingest, ingest-status, checks 1–8 + 11, kql (default). +# Hub view functions (Costs_v1_2, Prices_v1_2, Costs, Prices, etc.) that +# cross-reference database('Ingestion').*. +# Used by: ftk query/run/schema, parity checks 9–10. +# +# Notes: +# - The Kustainer HTTP endpoint is on host port $(HOST_PORT) (default 8082). +# Port 8080 is intentionally avoided — that belongs to maenifold. +# - `kql` talks directly to Kustainer's REST API. Kustainer is NOT an MCP +# server, so a plain curl is the right tool here. + +SHELL := /bin/bash + +# Load .env if present so HOST_PORT / MEM_LIMIT / EXPORT_DIR are visible to +# both `docker compose` and the recipes below. +ifneq (,$(wildcard ./.env)) +include .env +export +endif + +HOST_PORT ?= 8082 +# Default DB for ad-hoc kql queries: Ingestion (raw + final tables). +# For Hub view functions use: make kql DB=Hub QUERY='Costs_v1_2 | count' +# For ftk catalog queries use: pwsh scripts/ftk.ps1 query "..." (targets Hub by default) +DB ?= Ingestion +COMPOSE ?= docker compose + +.PHONY: up down nuke logs kql ftk ftk-list wait-healthy load-ftk-kql ingest ingest-status chunked-prices-backfill parity help + +help: + @echo "Targets:" + @echo " up Bring Kustainer up and block until healthy." + @echo " down Stop the container (keeps data volume)." + @echo " nuke Destructive full reset (drops kustainer-data/)." + @echo " logs Tail Kustainer logs." + @echo " kql QUERY='...' One-shot KQL against $(DB)." + @echo " load-ftk-kql Load FTK analytics KQL + open-data lookups." + @echo " ingest Bulk-ingest ALL parquet under export/." + @echo " ingest SCOPE= Ingest only one scope." + @echo " ingest SCOPE= PERIOD=20260501-20260531" + @echo " Filter scope + period." + @echo " ingest-status Show Ingest_Manifest summary." + @echo " parity Run T-1.4 parity check suite." + @echo " chunked-prices-backfill Recovery path when Prices_transform OOMs." +up: + $(COMPOSE) up -d + @$(MAKE) --no-print-directory wait-healthy + +wait-healthy: + @echo "Waiting for kustainer to become healthy (up to 90s)..." + @for i in $$(seq 1 90); do \ + status=$$(docker inspect -f '{{.State.Health.Status}}' kustainer 2>/dev/null || echo "missing"); \ + if [ "$$status" = "healthy" ]; then \ + echo "kustainer is healthy."; \ + exit 0; \ + fi; \ + if [ "$$status" = "missing" ]; then \ + echo "kustainer container not found yet..."; \ + fi; \ + sleep 1; \ + done; \ + echo "ERROR: kustainer did not become healthy in 90s. Last status: $$status" >&2; \ + docker inspect -f '{{json .State.Health}}' kustainer 2>/dev/null | head -c 2000 >&2; \ + exit 1 + +down: + $(COMPOSE) down + +nuke: + $(COMPOSE) down -v + rm -rf kustainer-data + +logs: + $(COMPOSE) logs -f kustainer + +# Run a one-shot KQL query against $(DB). Override QUERY on the command line: +# make kql QUERY='.show tables' +# make kql QUERY='Costs_raw | count' +kql: + @if [ -z "$(QUERY)" ]; then \ + echo "usage: make kql QUERY=''" >&2; exit 2; \ + fi + @QUERY="$(QUERY)" DB="$(DB)" pwsh -NoProfile -Command '@{ db = $$env:DB; csl = $$env:QUERY } | ConvertTo-Json -Compress' | \ + curl -sS --fail-with-body --max-time 60 \ + -H 'Content-Type: application/json' \ + -d @- \ + "http://localhost:$(HOST_PORT)/v1/rest/$$(echo '$(QUERY)' | grep -q '^\.' && echo mgmt || echo query)" + @echo + +# ftk CLI passthrough: `make ftk ARGS='run savings-summary-report'`. +# For everyday use call `pwsh scripts/ftk.ps1 ...` directly. +ftk: + @pwsh scripts/ftk.ps1 $(ARGS) + +# List the FinOps Toolkit catalog queries discoverable locally. +ftk-list: + @pwsh scripts/ftk.ps1 list + +# Load the FTK analytics KQL into the two-DB topology (Ingestion + Hub). +# Reads src/templates/finops-hub/.build.config combineKql bundles and +# concatenates the listed .kql files in dependency order. Idempotent. +load-ftk-kql: + @pwsh scripts/load-ftk-kql.ps1 + +# Bulk-ingest parquet under ./export/ into Costs_raw / Prices_raw. Picks the +# latest run-uuid per (scope, type, period) by manifest.runInfo.submittedTime, +# enforces OverwritePreviousReport semantics (drops superseded extents by +# tag), and is idempotent via the Ingest_Manifest table. +# make ingest +# make ingest SCOPE= +# make ingest SCOPE= PERIOD=20260501-20260531 +INGEST_FILTERS := +ifneq ($(SCOPE),) +INGEST_FILTERS += --scope $(SCOPE) +endif +ifneq ($(PERIOD),) +INGEST_FILTERS += --period $(PERIOD) +endif +ingest: + @pwsh scripts/ingest.ps1 $(INGEST_FILTERS) + +# Quick summary of Ingest_Manifest: files + rows per (scope, export_type). +# Targets Ingestion DB (DB=Ingestion is the default). +ingest-status: + @$(MAKE) --no-print-directory kql \ + QUERY="Ingest_Manifest | summarize files=count(), rows=sum(rows_ingested) by scope, export_type | order by scope asc, export_type asc" + +# Run the T-1.4 parity check suite. Reads tests/parity-checks.kql, executes +# each numbered check against Kustainer's query endpoint, evaluates the EVAL +# rule, and prints a pass/fail report. Exit code 0 only if all checks pass. +parity: + @pwsh scripts/run-parity-checks.ps1 --endpoint http://localhost:$(HOST_PORT) --db $(DB) + +# Recovery path for the Prices_transform_v1_2() backfill when the single-pass +# .set-or-append OOMs. Processes Prices_raw one source extent at a time +# (bounded memory). Use after `make ingest` if Prices_final_v1_2 ends up +# partially populated or empty. Wall-clock ~15 min on Apple Silicon. +chunked-prices-backfill: + @pwsh scripts/chunked-prices-backfill.ps1 + +# (Customer-specific savings-plan analysis targets removed — see T-3.5.) diff --git a/src/templates/finops-hub-local/docker-compose.yml b/src/templates/finops-hub-local/docker-compose.yml new file mode 100644 index 000000000..f5401e9d5 --- /dev/null +++ b/src/templates/finops-hub-local/docker-compose.yml @@ -0,0 +1,32 @@ +services: + kustainer: + image: mcr.microsoft.com/azuredataexplorer/kustainer-linux:latest + platform: linux/amd64 + container_name: kustainer + ports: + # Host 8082 -> container 8080. Host port 8080 is reserved for maenifold. + - "${HOST_PORT:-8082}:8080" + environment: + ACCEPT_EULA: "Y" + volumes: + # Parquet exports — read-only, the engine only ingests from here. + - "${EXPORT_DIR:-./export}:/data/export:ro" + # Persistent engine state (databases, metadata, extents). + - ./kustainer-data:/kustodata + mem_limit: ${MEM_LIMIT:-8g} + restart: unless-stopped + healthcheck: + # Use the in-container curl to hit the internal mgmt endpoint. + # `.show version` is the cheapest mgmt command and proves the engine + # is actually answering, not just listening. + test: + - CMD-SHELL + - >- + curl -fsS --max-time 10 + -H 'Content-Type: application/json' + -d '{"db":"NetDefaultDB","csl":".show version"}' + http://localhost:8080/v1/rest/mgmt > /dev/null || exit 1 + interval: 10s + timeout: 15s + retries: 12 + start_period: 30s diff --git a/src/templates/finops-hub-local/ftkver.txt b/src/templates/finops-hub-local/ftkver.txt new file mode 100644 index 000000000..b9dc6d407 --- /dev/null +++ b/src/templates/finops-hub-local/ftkver.txt @@ -0,0 +1 @@ +15.0-dev diff --git a/src/templates/finops-hub-local/scripts/chunked-prices-backfill.ps1 b/src/templates/finops-hub-local/scripts/chunked-prices-backfill.ps1 new file mode 100755 index 000000000..1bce6856b --- /dev/null +++ b/src/templates/finops-hub-local/scripts/chunked-prices-backfill.ps1 @@ -0,0 +1,48 @@ +#!/usr/bin/env pwsh +<# +.SYNOPSIS + Standalone chunked backfill for Prices_final_v1_2. + +.DESCRIPTION + Recovery backfill for Prices_transform_v1_2. Dot-sources ingest.ps1 and + backfills Prices_final_v1_2 from Prices_raw one extent at a time, with the same + update-policy disable/restore and Rosetta crash retry behavior as ingest.ps1. +#> +[CmdletBinding()] +param() + +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +. (Join-Path $PSScriptRoot 'ingest.ps1') + +function Get-TableCount { + param([Parameter(Mandatory)] [string] $TableName) + $postResult = Invoke-KustoPost -CslText "$TableName | count" -EndpointUrl $script:KustainerQuery -TimeoutSec 60 + if ($postResult.Status -ge 200 -and $postResult.Status -lt 300) { + $jsonObject = Get-KustoJson -BodyText $postResult.Body + return [int64](Get-PrimaryRows $jsonObject)[0][0] + } + return -1 +} + +function Invoke-ChunkedPricesBackfillMain { + Write-InfoLine ("Before: Prices_final_v1_2 = {0:N0}" -f (Get-TableCount -TableName 'Prices_final_v1_2')) + $savedPolicies = Disable-UpdatePolicies + Write-InfoLine "Disabled update policies on $($savedPolicies.Count) table(s)" + $stopwatch = [Diagnostics.Stopwatch]::StartNew() + try { + $result = Invoke-ChunkedBackfillFinalTable -RawTableName 'Prices_raw' -FinalTableName 'Prices_final_v1_2' -TransformFunctionName 'Prices_transform_v1_2' -ExtentsPerBatch 1 + } + finally { + Restore-UpdatePolicies -SavedPolicies $savedPolicies + } + $stopwatch.Stop() + $afterCount = Get-TableCount -TableName 'Prices_final_v1_2' + Write-InfoLine ("After: Prices_final_v1_2 = {0:N0}" -f $afterCount) + Write-InfoLine ("Wall-clock: {0:N1}s ({1:N1} min). OK={2}, reported_rows={3:N0}" -f $stopwatch.Elapsed.TotalSeconds, ($stopwatch.Elapsed.TotalSeconds / 60.0), $result.Ok, $result.Rows) + if ($result.Ok) { return 0 } + return 1 +} + +exit (Invoke-ChunkedPricesBackfillMain) diff --git a/src/templates/finops-hub-local/scripts/ftk.ps1 b/src/templates/finops-hub-local/scripts/ftk.ps1 new file mode 100644 index 000000000..8bcd3f5db --- /dev/null +++ b/src/templates/finops-hub-local/scripts/ftk.ps1 @@ -0,0 +1,483 @@ +#!/usr/bin/env pwsh +<# +.SYNOPSIS + ftk - a tiny CLI for the local FinOps Toolkit (Kustainer) stack. + +.DESCRIPTION + Runs KQL against the local Kusto Emulator (Kustainer) and runs the FinOps + Toolkit published query catalog locally by adapting the upstream .kql files + on read (no forked copies). + + Subcommands: + ftk query "" Run ad-hoc KQL (.-prefixed = mgmt command). + ftk schema [-Tables|-Functions] + List tables and/or functions in the database. + ftk tables Show one table's column schema (cslschema). + ftk list List the catalog queries discovered on disk. + ftk run [opts] Adapt + run a named catalog query locally. + + Connection (overridable with -Endpoint / -Database): + KUSTAINER_QUERY full query URL (default http://localhost:/v1/rest/query) + HOST_PORT port (default 8082; read from .env if present) + FTK_DB database name (default Hub) + Never point this at port 8080 (that is the maenifold MCP gateway). + + Catalog (FTK_CATALOG_PATH overrides; else FTK_REPO / ../finops-toolkit): + 1. $FTK_CATALOG_PATH + 2. /release/agent-skills/finops-toolkit/references/queries/catalog + 3. /src/queries/catalog + +.EXAMPLE + pwsh scripts/ftk.ps1 query "Costs() | summarize sum(EffectiveCost) by ServiceName | top 10 by sum_EffectiveCost" + +.EXAMPLE + pwsh scripts/ftk.ps1 run savings-summary-report --format csv +#> +[CmdletBinding()] +param( + [Parameter(Position = 0)] + [string] $Command, + + [Parameter(Position = 1)] + [string] $Arg1, + + [Parameter(ValueFromRemainingArguments = $true)] + [string[]] $Rest +) + +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +$script:RepoRoot = Split-Path -Parent (Split-Path -Parent $PSCommandPath) + +# --------------------------------------------------------------------------- # +# Connection resolution +# --------------------------------------------------------------------------- # +function Get-HostPort { + if ($env:HOST_PORT) { return $env:HOST_PORT } + $envFile = Join-Path $script:RepoRoot '.env' + if (Test-Path $envFile) { + foreach ($line in Get-Content $envFile) { + $t = $line.Trim() + if ($t -like 'HOST_PORT=*') { return ($t -split '=', 2)[1].Trim() } + } + } + return '8082' +} + +function Get-QueryUrl { + param([string] $Override) + if ($Override) { return $Override } + if ($env:KUSTAINER_QUERY) { return $env:KUSTAINER_QUERY } + return "http://localhost:$(Get-HostPort)/v1/rest/query" +} + +function Get-MgmtUrl { + param([string] $Override) + return (Get-QueryUrl $Override) -replace '/v1/rest/query', '/v1/rest/mgmt' +} + +function Get-Database { + param([string] $Override) + if ($Override) { return $Override } + if ($env:FTK_DB) { return $env:FTK_DB } + return 'Hub' +} + +# --------------------------------------------------------------------------- # +# REST client (same REST pattern used across the ftklocal tooling) +# --------------------------------------------------------------------------- # +function Invoke-Kusto { + <# + Run a query (or .mgmt command) and return a PSCustomObject with + .Columns (string[]) and .Rows (object[][]). + #> + param( + [Parameter(Mandatory)] [string] $Csl, + [string] $Endpoint, + [string] $Database, + [int] $TimeoutSec = 120 + ) + $db = Get-Database $Database + $isMgmt = $Csl.TrimStart().StartsWith('.') + $url = if ($isMgmt) { Get-MgmtUrl $Endpoint } else { Get-QueryUrl $Endpoint } + $body = @{ db = $db; csl = $Csl } | ConvertTo-Json -Compress + $headers = @{ + 'Content-Type' = 'application/json' + 'x-ms-client-version' = 'Kusto.Python.Client:1.0.0' # unlocks full JSON error bodies + } + try { + $resp = Invoke-RestMethod -Method Post -Uri $url -Headers $headers -Body $body ` + -TimeoutSec $TimeoutSec + } + catch { + throw (ConvertTo-KustoError $_ $url) + } + if (-not $resp.Tables) { + return [pscustomobject]@{ Columns = @(); Rows = @() } + } + $table = $resp.Tables[0] + $cols = @($table.Columns | ForEach-Object { $_.ColumnName }) + return [pscustomobject]@{ Columns = $cols; Rows = $table.Rows } +} + +function ConvertTo-KustoError { + param($ErrorRecord, [string] $Url) + $raw = $null + try { $raw = $ErrorRecord.ErrorDetails.Message } catch { } + if (-not $raw) { + try { + $stream = $ErrorRecord.Exception.Response.GetResponseStream() + $raw = (New-Object IO.StreamReader($stream)).ReadToEnd() + } catch { } + } + if (-not $raw) { + return "Cannot reach Kustainer at $Url ($($ErrorRecord.Exception.Message)). Is the container up? Try ``make up``." + } + try { + $obj = $raw | ConvertFrom-Json + foreach ($key in 'error', 'Error') { + if ($obj.PSObject.Properties.Name -contains $key -and $obj.$key) { + $err = $obj.$key + $msg = $err.'@message'; if (-not $msg) { $msg = $err.message }; if (-not $msg) { $msg = $err.'@type' } + $extra = $null + if ($err.PSObject.Properties.Name -contains '@context' -and $err.'@context') { + $extra = $err.'@context'.'@message' + } + if ($extra) { return "Kusto error: $msg | $extra" } + return "Kusto error: $msg" + } + } + } catch { } + return "Kusto error: $($raw.Substring(0, [Math]::Min(400, $raw.Length)))" +} + +function Get-Scalar { + param([string] $Csl, [string] $Endpoint, [string] $Database) + $r = Invoke-Kusto -Csl $Csl -Endpoint $Endpoint -Database $Database + if ($r.Rows -and $r.Rows.Count -gt 0 -and $r.Rows[0].Count -gt 0) { return $r.Rows[0][0] } + return $null +} + +# --------------------------------------------------------------------------- # +# Output formatting +# --------------------------------------------------------------------------- # +function Format-Result { + param([string[]] $Columns, $Rows, [string] $Format) + + if ($Format -eq 'json') { + $list = foreach ($row in $Rows) { + $o = [ordered]@{} + for ($i = 0; $i -lt $Columns.Count; $i++) { $o[$Columns[$i]] = $row[$i] } + [pscustomobject]$o + } + return ($list | ConvertTo-Json -Depth 10) + } + if ($Format -eq 'csv') { + $sb = [Text.StringBuilder]::new() + [void]$sb.AppendLine(($Columns -join ',')) + foreach ($row in $Rows) { + $cells = for ($i = 0; $i -lt $Columns.Count; $i++) { + $v = if ($null -eq $row[$i]) { '' } else { [string]$row[$i] } + if ($v -match '[",\n]') { '"' + ($v -replace '"', '""') + '"' } else { $v } + } + [void]$sb.AppendLine(($cells -join ',')) + } + return $sb.ToString().TrimEnd("`r", "`n") + } + # table (default) + if (-not $Columns -or $Columns.Count -eq 0) { return '(no columns)' } + $widths = @($Columns | ForEach-Object { $_.Length }) + $strRows = @() + foreach ($row in $Rows) { + $sr = for ($i = 0; $i -lt $Columns.Count; $i++) { if ($null -eq $row[$i]) { '' } else { [string]$row[$i] } } + $sr = @($sr) + $strRows += , $sr + for ($i = 0; $i -lt $Columns.Count; $i++) { if ($sr[$i].Length -gt $widths[$i]) { $widths[$i] = $sr[$i].Length } } + } + $sep = ' ' + $lines = @() + $lines += (0..($Columns.Count - 1) | ForEach-Object { $Columns[$_].PadRight($widths[$_]) }) -join $sep + $lines += (0..($Columns.Count - 1) | ForEach-Object { '-' * $widths[$_] }) -join $sep + foreach ($sr in $strRows) { + $lines += (0..($Columns.Count - 1) | ForEach-Object { $sr[$_].PadRight($widths[$_]) }) -join $sep + } + $n = @($Rows).Count + $lines += '' + $lines += "($n row$(if ($n -ne 1) { 's' }))" + return ($lines -join "`n") +} + +# --------------------------------------------------------------------------- # +# Catalog discovery +# --------------------------------------------------------------------------- # +$script:LibraryQueries = @('costs-enriched-base') + +function Get-FtkRepo { + if ($env:FTK_REPO) { return $env:FTK_REPO } + return (Resolve-Path (Join-Path (Split-Path -Parent $script:RepoRoot) 'finops-toolkit') -ErrorAction SilentlyContinue) ` + ?? (Join-Path (Split-Path -Parent $script:RepoRoot) 'finops-toolkit') +} + +function Get-CatalogDir { + $candidates = @() + if ($env:FTK_CATALOG_PATH) { $candidates += $env:FTK_CATALOG_PATH } + $repo = Get-FtkRepo + $candidates += (Join-Path $repo 'release/agent-skills/finops-toolkit/references/queries/catalog') + $candidates += (Join-Path $repo 'src/queries/catalog') + foreach ($c in $candidates) { if (Test-Path $c -PathType Container) { return $c } } + return $null +} + +function Get-CatalogQueries { + $d = Get-CatalogDir + if (-not $d) { return @() } + return @(Get-ChildItem -Path $d -Filter '*.kql' | Sort-Object Name) +} + +function Get-QueryDescription { + param([string] $Path) + $lines = Get-Content -Path $Path + for ($i = 0; $i -lt $lines.Count; $i++) { + if ($lines[$i] -match '//\s*Description:') { + for ($j = $i + 1; $j -lt [Math]::Min($i + 4, $lines.Count); $j++) { + if ($lines[$j] -match '^\s*//\s*(.+)') { $t = $Matches[1].Trim(); if ($t) { return $t } } + } + } + } + foreach ($line in $lines) { + if ($line -match '^\s*//\s*Query:\s*(.+)') { return $Matches[1].Trim() } + } + return '' +} + +# --------------------------------------------------------------------------- # +# Query adapter (the only local-compat logic) +# --------------------------------------------------------------------------- # +$script:SchemaCache = @{} + +function Get-FunctionSchema { + param([string] $Fn, [string] $Endpoint, [string] $Database) + $key = "$(Get-Database $Database)::$Fn" + if ($script:SchemaCache.ContainsKey($key)) { return $script:SchemaCache[$key] } + $cols = @{} + try { + $r = Invoke-Kusto -Csl "$Fn() | getschema | project ColumnName" -Endpoint $Endpoint -Database $Database + foreach ($row in $r.Rows) { $cols[$row[0]] = $true } + } catch { } + $script:SchemaCache[$key] = $cols + return $cols +} + +function ConvertTo-KqlLiteral { + param([string] $Value) + $v = $Value.Trim() + if ($v -match '^-?\d+$') { return $v } + if ($v -match '^-?\d*\.\d+$') { return $v } + if ($v -in 'true', 'false') { return $v.ToLower() } + if ($v -match '^\d{4}-\d{2}-\d{2}([ T].*)?$') { return "datetime($($v -replace ' ', 'T'))" } + return "'" + ($v -replace "'", "\'") + "'" +} + +function Get-InferredSpanMonths { + param([string] $Rhs) + if ($Rhs -match 'ago\((\d+)d\)') { return [Math]::Max(1, [Math]::Round([int]$Matches[1] / 30.0)) } + if ($Rhs -match 'monthsago\((\d+)\)') { return [Math]::Max(1, [int]$Matches[1]) } + return 1 +} + +function Resolve-Window { + param([string] $Kql, [string] $Start, [string] $End, [string] $Endpoint, [string] $Database) + if ($Start -or $End) { + $s = if ($Start) { ConvertTo-KqlLiteral $Start } else { $null } + $e = if ($End) { ConvertTo-KqlLiteral $End } else { $null } + return @($s, $e) + } + $span = 1 + if ($Kql -match 'let\s+startDate\s*=\s*([^;]+);') { $span = Get-InferredSpanMonths $Matches[1] } + $maxRaw = $null + try { $maxRaw = Get-Scalar -Csl 'Costs() | summarize max(ChargePeriodStart)' -Endpoint $Endpoint -Database $Database } catch { $maxRaw = $null } + if (-not $maxRaw) { return @($null, $null) } + $maxd = ([datetime]$maxRaw).ToUniversalTime() + $endDt = (Get-Date -Year $maxd.Year -Month $maxd.Month -Day 1 -Hour 0 -Minute 0 -Second 0).AddMonths(1) + $startDt = $endDt.AddMonths(-$span) + $fmt = { param($d) "datetime($($d.ToString('yyyy-MM-ddTHH:mm:ssZ')))" } + return @((& $fmt $startDt), (& $fmt $endDt)) +} + +function Invoke-ProjectAwayTolerance { + param([string] $Kql, [string] $Endpoint, [string] $Database) + if ($Kql -notmatch 'project-away') { return $Kql } + if ($Kql -notmatch '\b(Costs|Prices|Recommendations|Transactions)\(\)') { return $Kql } + $fn = $Matches[1] + $schema = Get-FunctionSchema -Fn $fn -Endpoint $Endpoint -Database $Database + if ($schema.Count -eq 0) { return $Kql } + return [regex]::Replace($Kql, '(\|\s*project-away\s+)([^\n|]+)', { + param($m) + $prefix = $m.Groups[1].Value + $names = $m.Groups[2].Value -split ',' | ForEach-Object { $_.Trim() } | Where-Object { $_ } + $kept = @($names | Where-Object { $schema.ContainsKey($_) }) + if ($kept.Count -eq 0) { return '' } + return $prefix + ($kept -join ', ') + }) +} + +function Convert-CatalogQuery { + param( + [string] $Kql, + [hashtable] $Params, + [string] $Start, + [string] $End, + [string] $Endpoint, + [string] $Database + ) + $out = $Kql + + # 1. Date window + $win = Resolve-Window -Kql $out -Start $Start -End $End -Endpoint $Endpoint -Database $Database + if ($null -ne $win[0]) { + $out = [regex]::Replace($out, '(let\s+startDate\s*=\s*)([^;]+)(;)', { param($m) $m.Groups[1].Value + $win[0] + $m.Groups[3].Value }, 1) + } + if ($null -ne $win[1]) { + $out = [regex]::Replace($out, '(let\s+endDate\s*=\s*)([^;]+)(;)', { param($m) $m.Groups[1].Value + $win[1] + $m.Groups[3].Value }, 1) + } + + # 1b. Arbitrary --param overrides on top-level let bindings (typed literals). + if ($Params) { + foreach ($k in $Params.Keys) { + $lit = ConvertTo-KqlLiteral $Params[$k] + $pat = "(let\s+$([regex]::Escape($k))\s*=\s*)([^;]+)(;)" + if ($out -notmatch $pat) { + throw "--param '$k' does not match a top-level ``let $k = ...;`` in the query" + } + $out = [regex]::Replace($out, $pat, { param($m) $m.Groups[1].Value + $lit + $m.Groups[3].Value }, 1) + } + } + + # 2. decimal -> real normalization (the verified Kustainer compatibility fix) + $out = $out -replace "todecimal\(\s*''\s*\)", 'real(null)' + $out = $out -replace '\btodecimal\s*\(', 'toreal(' + $out = $out -replace '\bdecimal\s*\(', 'real(' + + # 3. cross-db reference strip — catalog queries that explicitly name database('Ingestion'). + # are stripped when running against Hub, which already resolves these via its view + # functions. This is a no-op for standard catalog queries that call Costs_v1_2() etc. + $out = $out -replace "database\(\s*'Ingestion'\s*\)\.", '' + + # 4. project-away tolerance: drop columns the local schema doesn't expose + $out = Invoke-ProjectAwayTolerance -Kql $out -Endpoint $Endpoint -Database $Database + + return $out +} + +# --------------------------------------------------------------------------- # +# Option parsing for the remaining args (mirrors the CLI flag surface) +# --------------------------------------------------------------------------- # +function Read-Options { + param([string[]] $Tokens) + $opts = @{ format = 'table'; params = @{}; positional = @() } + $i = 0 + while ($i -lt $Tokens.Count) { + switch -Regex ($Tokens[$i]) { + '^--format$|^-Format$' { $opts.format = $Tokens[++$i] } + '^--param$|^-Param$' { $kv = $Tokens[++$i]; $p = $kv -split '=', 2; if ($p.Count -ne 2) { throw "--param must be key=value, got '$kv'" }; $opts.params[$p[0].Trim()] = $p[1] } + '^--start$|^-Start$' { $opts.start = $Tokens[++$i] } + '^--end$|^-End$' { $opts.end = $Tokens[++$i] } + '^--show$|^-Show$' { $opts.show = $true } + '^--tables$|^-Tables$' { $opts.tables = $true } + '^--functions$|^-Functions$' { $opts.functions = $true } + '^--endpoint$|^-Endpoint$' { $opts.endpoint = $Tokens[++$i] } + '^--database$|^-Database$' { $opts.database = $Tokens[++$i] } + default { $opts.positional += $Tokens[$i] } + } + $i++ + } + return $opts +} + +# --------------------------------------------------------------------------- # +# Subcommands +# --------------------------------------------------------------------------- # +function Show-Usage { + @' +ftk - local FinOps Toolkit query CLI + + ftk query "" [--format table|json|csv] + ftk schema [--tables] [--functions] + ftk tables + ftk list + ftk run [--param k=v] [--start ] [--end ] [--show] [--format ...] + + Global: [--endpoint ] [--database ] +'@ +} + +$argv = @() +if ($Arg1) { $argv += $Arg1 } +if ($Rest) { $argv += $Rest } +$opt = Read-Options $argv + +$endpoint = if ($opt.ContainsKey('endpoint')) { $opt.endpoint } else { $null } +$db = if ($opt.ContainsKey('database')) { $opt.database } else { $null } + +try { + switch ($Command) { + 'query' { + $kql = $opt.positional[0] + if (-not $kql) { throw 'usage: ftk query ""' } + $r = Invoke-Kusto -Csl $kql -Endpoint $endpoint -Database $db + Write-Output (Format-Result -Columns $r.Columns -Rows $r.Rows -Format $opt.format) + } + 'schema' { + $wantTables = $opt.ContainsKey('tables') -or -not $opt.ContainsKey('functions') + $wantFuncs = $opt.ContainsKey('functions') -or -not $opt.ContainsKey('tables') + if ($wantTables) { + $r = Invoke-Kusto -Csl '.show tables | project TableName | order by TableName asc' -Endpoint $endpoint -Database $db + Write-Output 'Tables:' + foreach ($row in $r.Rows) { Write-Output " $($row[0])" } + } + if ($wantFuncs) { + $r = Invoke-Kusto -Csl '.show functions | project Name | order by Name asc' -Endpoint $endpoint -Database $db + Write-Output 'Functions:' + foreach ($row in $r.Rows) { Write-Output " $($row[0])" } + } + } + 'tables' { + $name = $opt.positional[0] + if (-not $name) { throw 'usage: ftk tables ' } + $r = Invoke-Kusto -Csl ".show table $name cslschema | project Schema" -Endpoint $endpoint -Database $db + if ($r.Rows) { Write-Output $r.Rows[0][0] } + } + 'list' { + $d = Get-CatalogDir + if (-not $d) { Write-Error 'No catalog found. Set FTK_CATALOG_PATH or FTK_REPO (looked under ../finops-toolkit).'; exit 1 } + Write-Output "Catalog: $d`n" + foreach ($p in Get-CatalogQueries) { + $nm = [IO.Path]::GetFileNameWithoutExtension($p.Name) + $tag = if ($script:LibraryQueries -contains $nm) { ' [library]' } else { '' } + $desc = Get-QueryDescription $p.FullName + Write-Output (' {0,-42}{1} {2}' -f $nm, $tag, $desc) + } + } + 'run' { + $name = $opt.positional[0] + if (-not $name) { throw 'usage: ftk run ' } + if ($script:LibraryQueries -contains $name) { throw "'$name' is a library building block, not a runnable query." } + $path = $null + foreach ($p in Get-CatalogQueries) { if ([IO.Path]::GetFileNameWithoutExtension($p.Name) -eq $name) { $path = $p.FullName; break } } + if (-not $path) { throw "Catalog query '$name' not found. Try ``ftk list``." } + $raw = Get-Content -Raw -Path $path + $start = if ($opt.ContainsKey('start')) { $opt.start } else { $null } + $end = if ($opt.ContainsKey('end')) { $opt.end } else { $null } + $adapted = Convert-CatalogQuery -Kql $raw -Params $opt.params -Start $start -End $end -Endpoint $endpoint -Database $db + if ($opt.ContainsKey('show')) { Write-Output $adapted; break } + $r = Invoke-Kusto -Csl $adapted -Endpoint $endpoint -Database $db + Write-Output (Format-Result -Columns $r.Columns -Rows $r.Rows -Format $opt.format) + } + default { Show-Usage; if ($Command) { exit 2 } } + } +} +catch { + Write-Error "error: $($_.Exception.Message)" + exit 2 +} diff --git a/src/templates/finops-hub-local/scripts/ingest.ps1 b/src/templates/finops-hub-local/scripts/ingest.ps1 new file mode 100755 index 000000000..9276acaef --- /dev/null +++ b/src/templates/finops-hub-local/scripts/ingest.ps1 @@ -0,0 +1,896 @@ +#!/usr/bin/env pwsh +<# +.SYNOPSIS + Bulk-ingest FOCUS Cost + Price Sheet parquet exports into local Kustainer raw tables. + +.DESCRIPTION + Bulk-ingests FOCUS parquet exports. Walks export/{scope}/{type}/{period}/{run-uuid}, + selects the latest manifest run per (scope,type,period), tracks idempotency in + Ingest_Manifest by checksum, enforces overwrite semantics for superseded runs, disables + transactional final-table update policies during bulk ingest, and explicitly backfills finals. + +.PARAMETER Scope + Filter to one scope subfolder under export/. + +.PARAMETER Period + Filter to one period (YYYYMMDD-YYYYMMDD). + +.PARAMETER DryRun + Plan and print work, including manifest skip decisions, without mutating Kustainer. + +.PARAMETER ForcePolicyRecapture + Allow capturing update policies that are already disabled. Dangerous; mirrors the ingest policy-capture guard. + +.EXAMPLE + pwsh scripts/ingest.ps1 --dry-run + +.EXAMPLE + pwsh scripts/ingest.ps1 --scope ea --period 20260501-20260531 +#> +[CmdletBinding()] +param( + [Parameter(ValueFromRemainingArguments = $true)] + [string[]] $CliArgs +) + +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +$script:RepoRoot = Split-Path -Parent (Split-Path -Parent $PSCommandPath) + +function Get-ExportDir { + if ($env:EXPORT_DIR) { return $env:EXPORT_DIR } + $envFilePath = Join-Path $script:RepoRoot '.env' + if (Test-Path -LiteralPath $envFilePath) { + foreach ($lineText in Get-Content -LiteralPath $envFilePath) { + $trimmed = $lineText.Trim() + if ($trimmed -like 'EXPORT_DIR=*') { return ($trimmed -split '=', 2)[1].Trim() } + } + } + return (Join-Path $script:RepoRoot 'export') +} + +$script:ExportDirHost = Get-ExportDir +$script:ExportDirContainer = '/data/export' +$script:DbName = if ($env:FTK_DB) { $env:FTK_DB } else { 'Ingestion' } +$script:IngestManifestTable = 'Ingest_Manifest' +$script:RowCountTolerance = 0.001 +$script:IngestTimeoutSec = 1800 +$script:DatasetTableMap = @{ + 'ms--focus-cost' = [pscustomobject]@{ Table = 'Costs_raw'; Mapping = 'Costs_raw_mapping' } + 'ms--pricesheet' = [pscustomobject]@{ Table = 'Prices_raw'; Mapping = 'Prices_raw_mapping' } +} +$script:RawToFinalPolicy = @{ + 'Costs_raw' = [pscustomobject]@{ FinalTable = 'Costs_final_v1_2'; TransformFn = 'Costs_transform_v1_2' } + 'Prices_raw' = [pscustomobject]@{ FinalTable = 'Prices_final_v1_2'; TransformFn = 'Prices_transform_v1_2' } +} + +function Get-HostPort { + if ($env:HOST_PORT) { return $env:HOST_PORT } + $envFilePath = Join-Path $script:RepoRoot '.env' + if (Test-Path -LiteralPath $envFilePath) { + foreach ($lineText in Get-Content -LiteralPath $envFilePath) { + $trimmed = $lineText.Trim() + if ($trimmed -like 'HOST_PORT=*') { return ($trimmed -split '=', 2)[1].Trim() } + } + } + return '8082' +} + +function Get-KustainerMgmtUrl { + if ($env:KUSTAINER_MGMT) { return $env:KUSTAINER_MGMT } + return "http://localhost:$(Get-HostPort)/v1/rest/mgmt" +} + +function Get-KustainerQueryUrl { + return (Get-KustainerMgmtUrl) -replace '/mgmt$', '/query' +} + +$script:KustainerMgmt = Get-KustainerMgmtUrl +$script:KustainerQuery = Get-KustainerQueryUrl + +function New-PostResult { + param([int] $StatusCode, [string] $BodyText) + [pscustomobject]@{ Status = $StatusCode; Body = $BodyText } +} + +function Write-InfoLine { + param([string] $Message = '') + [Console]::Out.WriteLine($Message) +} + +function Sort-ByNameOrdinal { + param([object[]] $Items) + $arr = @($Items) + [Array]::Sort($arr, [Comparison[object]] { param($left, $right) [StringComparer]::Ordinal.Compare($left.Name, $right.Name) }) + return $arr +} + +function Invoke-KustoPost { + param( + [Parameter(Mandatory)] [string] $CslText, + [string] $EndpointUrl = $script:KustainerMgmt, + [int] $TimeoutSec = 60 + ) + $payload = @{ db = $script:DbName; csl = $CslText } | ConvertTo-Json -Compress + $headers = @{ + 'Content-Type' = 'application/json' + 'x-ms-client-version' = 'Kusto.Python.Client:1.0.0' + 'Accept' = 'application/json' + } + try { + $response = Invoke-WebRequest -Method Post -Uri $EndpointUrl -Headers $headers -Body $payload ` + -TimeoutSec $TimeoutSec -SkipHttpErrorCheck + return (New-PostResult -StatusCode ([int]$response.StatusCode) -BodyText ([string]$response.Content)) + } + catch { + $bodyText = $null + try { $bodyText = $_.ErrorDetails.Message } catch { } + if (-not $bodyText) { + try { + $stream = $_.Exception.Response.GetResponseStream() + $bodyText = (New-Object IO.StreamReader($stream)).ReadToEnd() + } catch { } + } + if ($_.Exception.Response -and $bodyText) { + try { return (New-PostResult -StatusCode ([int]$_.Exception.Response.StatusCode) -BodyText $bodyText) } catch { } + } + $messageText = "Connection error: $($_.Exception.GetType().Name): $($_.Exception.Message)" + if ($_.Exception.GetType().Name -match 'WebException|HttpRequestException|IOException') { + return (New-PostResult -StatusCode 599 -BodyText $messageText) + } + return (New-PostResult -StatusCode 599 -BodyText $messageText) + } +} + +function Get-KustoJson { + param([Parameter(Mandatory)] [string] $BodyText) + return ($BodyText | ConvertFrom-Json) +} + +function Get-PrimaryRows { + param($JsonObject) + if (-not $JsonObject.Tables -or @($JsonObject.Tables).Count -eq 0) { return ,@() } + return ,@($JsonObject.Tables[0].Rows) +} + +function Get-PrimaryColumnNames { + param($JsonObject) + if (-not $JsonObject.Tables -or @($JsonObject.Tables).Count -eq 0) { return ,@() } + return ,@($JsonObject.Tables[0].Columns | ForEach-Object { $_.ColumnName }) +} + +function Get-QueryScalar { + param([Parameter(Mandatory)] [string] $CslText) + $postResult = Invoke-KustoPost -CslText $CslText -EndpointUrl $script:KustainerQuery -TimeoutSec 60 + if ($postResult.Status -lt 200 -or $postResult.Status -ge 300) { return $null } + try { + $jsonObject = Get-KustoJson -BodyText $postResult.Body + $rows = Get-PrimaryRows $jsonObject + if ($rows.Count -eq 0) { return $null } + return $rows[0][0] + } + catch { return $null } +} + +function Parse-KustoError { + param([string] $BodyText) + try { + $jsonObject = $BodyText | ConvertFrom-Json + $err = $null + if ($jsonObject.PSObject.Properties.Name -contains 'error') { $err = $jsonObject.error } + if ($err) { + $errType = $err.'@type'; if (-not $errType) { $errType = $err.code }; if (-not $errType) { $errType = 'Unknown' } + $errMsg = $err.'@message'; if (-not $errMsg) { $errMsg = $err.message }; if (-not $errMsg) { $errMsg = $BodyText.Substring(0, [Math]::Min(200, $BodyText.Length)) } + return @([string]$errType, [string]$errMsg) + } + } + catch { } + return @('RawText', $BodyText.Substring(0, [Math]::Min(300, $BodyText.Length))) +} + +function Get-JsonPropertyValue { + param($ObjectValue, [string] $Name, $DefaultValue = $null) + if ($null -eq $ObjectValue) { return $DefaultValue } + if ($ObjectValue.PSObject.Properties.Name -contains $Name) { return $ObjectValue.$Name } + return $DefaultValue +} + +function Get-SubmittedTime { + param($ManifestObject) + $runInfo = Get-JsonPropertyValue -ObjectValue $ManifestObject -Name 'runInfo' + $submitted = Get-JsonPropertyValue -ObjectValue $runInfo -Name 'submittedTime' + if ($submitted) { return [string]$submitted } + $created = Get-JsonPropertyValue -ObjectValue $runInfo -Name 'createdDate' + if ($created) { return [string]$created } + return $null +} + +function Read-RunManifest { + param([Parameter(Mandatory)] [string] $RunDirPath) + $manifestPath = Join-Path $RunDirPath 'manifest.json' + if (-not (Test-Path -LiteralPath $manifestPath -PathType Leaf)) { return $null } + try { return (Get-Content -LiteralPath $manifestPath -Raw -Encoding UTF8 | ConvertFrom-Json) } + catch { [Console]::Error.WriteLine(" WARN: cannot read $manifestPath`: $($_.Exception.Message)"); return $null } +} + +function New-RunPlan { + param( + [string] $ScopeName, + [string] $ExportTypeName, + [string] $PeriodName, + [string] $RunUuidValue, + [string] $RunDirPath, + [string] $SubmittedTimeValue, + [int64] $ExpectedRowsValue, + [object[]] $ParquetFilesValue, + [string[]] $SupersededRunUuidsValue + ) + [pscustomobject]@{ + Scope = $ScopeName + ExportType = $ExportTypeName + Period = $PeriodName + RunUuid = $RunUuidValue + RunDir = $RunDirPath + SubmittedTime = $SubmittedTimeValue + ExpectedRows = $ExpectedRowsValue + ParquetFiles = @($ParquetFilesValue) + SupersededRunUuids = @($SupersededRunUuidsValue) + } +} + +function Build-Plan { + param([string] $ExportRootPath, [string] $ScopeFilter, [string] $PeriodFilter) + $plans = @() + if (-not (Test-Path -LiteralPath $ExportRootPath -PathType Container)) { throw "export root not found: $ExportRootPath" } + foreach ($scopeDir in @(Sort-ByNameOrdinal @(Get-ChildItem -LiteralPath $ExportRootPath -Directory))) { + if ($ScopeFilter -and $scopeDir.Name -ne $ScopeFilter) { continue } + foreach ($typeDir in @(Sort-ByNameOrdinal @(Get-ChildItem -LiteralPath $scopeDir.FullName -Directory))) { + if (-not $script:DatasetTableMap.ContainsKey($typeDir.Name)) { + [Console]::Error.WriteLine(" skip: unknown dataset $($scopeDir.Name)/$($typeDir.Name)") + continue + } + foreach ($periodDir in @(Sort-ByNameOrdinal @(Get-ChildItem -LiteralPath $typeDir.FullName -Directory))) { + if ($PeriodFilter -and $periodDir.Name -ne $PeriodFilter) { continue } + $runs = @() + foreach ($runDir in @(Sort-ByNameOrdinal @(Get-ChildItem -LiteralPath $periodDir.FullName -Directory))) { + $manifestObject = Read-RunManifest -RunDirPath $runDir.FullName + if ($null -eq $manifestObject) { continue } + $runs += [pscustomobject]@{ + RunDir = $runDir + Submitted = Get-SubmittedTime -ManifestObject $manifestObject + Manifest = $manifestObject + SubmittedSort = if (Get-SubmittedTime -ManifestObject $manifestObject) { Get-SubmittedTime -ManifestObject $manifestObject } else { '' } + Mtime = $runDir.LastWriteTimeUtc.Ticks + RunName = $runDir.Name + } + } + if ($runs.Count -eq 0) { continue } + $sortedRuns = @($runs | Sort-Object SubmittedSort, Mtime, RunName) + $latest = $sortedRuns[$sortedRuns.Count - 1] + $superseded = @() + if ($sortedRuns.Count -gt 1) { $superseded = @($sortedRuns[0..($sortedRuns.Count - 2)] | ForEach-Object { $_.RunDir.Name }) } + $parts = @(Sort-ByNameOrdinal @(Get-ChildItem -LiteralPath $latest.RunDir.FullName -Filter '*.parquet' -File)) + if ($parts.Count -eq 0) { + [Console]::Error.WriteLine(" WARN: latest run has no parquet parts: $($latest.RunDir.FullName)") + continue + } + $expectedRaw = Get-JsonPropertyValue -ObjectValue $latest.Manifest -Name 'dataRowCount' -DefaultValue 0 + $plans += New-RunPlan -ScopeName $scopeDir.Name -ExportTypeName $typeDir.Name -PeriodName $periodDir.Name ` + -RunUuidValue $latest.RunDir.Name -RunDirPath $latest.RunDir.FullName -SubmittedTimeValue $latest.Submitted ` + -ExpectedRowsValue ([int64]$expectedRaw) -ParquetFilesValue $parts -SupersededRunUuidsValue $superseded + } + } + } + return @($plans) +} + +function Get-FileSha256Lower { + param([Parameter(Mandatory)] [string] $PathValue) + return (Get-FileHash -LiteralPath $PathValue -Algorithm SHA256).Hash.ToLowerInvariant() +} + +function Join-ManifestKey { + param([string] $ScopeName, [string] $ExportTypeName, [string] $PeriodName, [string] $RunUuidValue, [string] $FileNameValue) + $sep = [string][char]31 + return @($ScopeName, $ExportTypeName, $PeriodName, $RunUuidValue, $FileNameValue) -join $sep +} + +function Initialize-IngestManifestTable { + $schema = 'scope:string, export_type:string, period:string, run_uuid:string, file_name:string, file_size:long, rows_ingested:long, checksum_sha256:string, ingested_at:datetime' + $cslText = ".create-merge table $script:IngestManifestTable ($schema)" + $postResult = Invoke-KustoPost -CslText $cslText -TimeoutSec 60 + if ($postResult.Status -lt 200 -or $postResult.Status -ge 300) { + $errParts = Parse-KustoError -BodyText $postResult.Body + throw "failed to ensure $script:IngestManifestTable table: $($errParts[0]): $($errParts[1])" + } +} + +function Fetch-ManifestIndex { + $cslText = "$script:IngestManifestTable | project scope, export_type, period, run_uuid, file_name, file_size, rows_ingested, checksum_sha256, ingested_at" + $postResult = Invoke-KustoPost -CslText $cslText -EndpointUrl $script:KustainerQuery -TimeoutSec 60 + if ($postResult.Status -lt 200 -or $postResult.Status -ge 300) { + $errParts = Parse-KustoError -BodyText $postResult.Body + $msgLower = $errParts[1].ToLowerInvariant() + if ($msgLower.Contains('not found') -or $msgLower.Contains('failed to resolve') -or $msgLower.Contains('could not be resolved') -or $errParts[0].Contains('EntityNotFound') -or $errParts[1].Contains('SEM0100')) { return @{} } + throw "failed to read $script:IngestManifestTable`: $($errParts[0]): $($errParts[1])" + } + $jsonObject = Get-KustoJson -BodyText $postResult.Body + $rows = Get-PrimaryRows $jsonObject + $cols = Get-PrimaryColumnNames $jsonObject + $index = @{} + foreach ($row in $rows) { + $rec = [ordered]@{} + for ($idx = 0; $idx -lt $cols.Count; $idx++) { $rec[$cols[$idx]] = $row[$idx] } + $key = Join-ManifestKey -ScopeName $rec['scope'] -ExportTypeName $rec['export_type'] -PeriodName $rec['period'] -RunUuidValue $rec['run_uuid'] -FileNameValue $rec['file_name'] + $index[$key] = [pscustomobject]$rec + } + return $index +} + +function ConvertTo-CsvQuotedField { + param($Value) + $textValue = if ($null -eq $Value) { '' } else { [string]$Value } + return '"' + ($textValue -replace '"', '""') + '"' +} + +function Insert-ManifestRow { + param([Parameter(Mandatory)] $RecordObject) + $fields = @( + $RecordObject.scope, $RecordObject.export_type, $RecordObject.period, $RecordObject.run_uuid, + $RecordObject.file_name, [string]$RecordObject.file_size, [string]$RecordObject.rows_ingested, + $RecordObject.checksum_sha256, $RecordObject.ingested_at + ) | ForEach-Object { ConvertTo-CsvQuotedField $_ } + $csvLine = $fields -join ',' + $cslText = ".ingest inline into table $script:IngestManifestTable with (format='csv') <|`n$csvLine" + $postResult = Invoke-KustoPost -CslText $cslText -TimeoutSec 60 + if ($postResult.Status -lt 200 -or $postResult.Status -ge 300) { + $errParts = Parse-KustoError -BodyText $postResult.Body + throw "failed to insert $script:IngestManifestTable row: $($errParts[0]): $($errParts[1])" + } +} + +function Get-UpdatePolicy { + param([Parameter(Mandatory)] [string] $FinalTableName) + $postResult = Invoke-KustoPost -CslText ".show table $FinalTableName policy update" -TimeoutSec 30 + if ($postResult.Status -lt 200 -or $postResult.Status -ge 300) { return $null } + try { + $jsonObject = Get-KustoJson -BodyText $postResult.Body + $rows = Get-PrimaryRows $jsonObject + if ($rows.Count -eq 0) { return $null } + $bodyText = $rows[0][2] + if (-not $bodyText) { return $null } + return @($bodyText | ConvertFrom-Json) + } + catch { return $null } +} + +function Set-UpdatePolicy { + param([Parameter(Mandatory)] [string] $FinalTableName, [Parameter(Mandatory)] [object[]] $PolicyObject) + # Kusto expects a JSON array of policy objects. Serialize each element and + # join, so a single-element collection isn't emitted as a bare JSON object. + $elements = @($PolicyObject | ForEach-Object { $_ | ConvertTo-Json -Depth 20 -Compress }) + $bodyText = '[' + ($elements -join ',') + ']' + $cslText = '.alter table ' + $FinalTableName + ' policy update ```' + $bodyText + '```' + $postResult = Invoke-KustoPost -CslText $cslText -TimeoutSec 60 + if ($postResult.Status -lt 200 -or $postResult.Status -ge 300) { + $errParts = Parse-KustoError -BodyText $postResult.Body + throw "failed to set update policy on ${FinalTableName}: $($errParts[0]): $($errParts[1])" + } +} + +function Disable-UpdatePolicies { + param([bool] $ForceRecapture = $false) + $saved = @{} + foreach ($rawTableName in @('Costs_raw', 'Prices_raw')) { + $finalTableName = $script:RawToFinalPolicy[$rawTableName].FinalTable + $policy = @(Get-UpdatePolicy -FinalTableName $finalTableName) + if ($policy.Count -eq 0) { + [Console]::Error.WriteLine(" NOTE: no update policy on $finalTableName; skipping disable") + continue + } + $alreadyDisabled = $false + foreach ($polItem in $policy) { + $enabledValue = Get-JsonPropertyValue -ObjectValue $polItem -Name 'IsEnabled' -DefaultValue $true + if (-not [bool]$enabledValue) { $alreadyDisabled = $true; break } + } + if ($alreadyDisabled -and -not $ForceRecapture) { + throw "$finalTableName update policy is ALREADY disabled (IsEnabled=false). A previous ingest run likely crashed between disable/restore. Refusing to capture the disabled state as 'original' — that would silently bake the broken state into future runs.`n`nResolution: manually re-enable the policy via`n make kql QUERY=`".alter table $finalTableName policy update ''`"`nThen re-run this script. Or override with --force-policy-recapture if you know what you're doing." + } + $saved[$finalTableName] = @($policy | ConvertTo-Json -Depth 20 | ConvertFrom-Json) + $disabled = @($policy | ConvertTo-Json -Depth 20 | ConvertFrom-Json) + foreach ($polItem in $disabled) { $polItem.IsEnabled = $false } + Set-UpdatePolicy -FinalTableName $finalTableName -PolicyObject $disabled + $firstPolicy = $policy[0] + $wasEnabled = Get-JsonPropertyValue -ObjectValue $firstPolicy -Name 'IsEnabled' + $wasTransactional = Get-JsonPropertyValue -ObjectValue $firstPolicy -Name 'IsTransactional' + Write-InfoLine " PRE-INGEST: disabled update policy on $finalTableName (was: IsEnabled=$wasEnabled, IsTransactional=$wasTransactional)" + } + return $saved +} + +function Restore-UpdatePolicies { + param([System.Collections.IDictionary] $SavedPolicies) + foreach ($finalTableName in $SavedPolicies.Keys) { + try { + Set-UpdatePolicy -FinalTableName $finalTableName -PolicyObject @($SavedPolicies[$finalTableName]) + Write-InfoLine " POST-INGEST: restored update policy on $finalTableName" + } + catch { [Console]::Error.WriteLine(" ERROR: could not restore update policy on $finalTableName`: $($_.Exception.Message)") } + } +} + +function Get-RowsFromSetOrAppendResponse { + param([string] $BodyText) + $rowsAdded = 0L + try { + $jsonObject = Get-KustoJson -BodyText $BodyText + $cols = Get-PrimaryColumnNames $jsonObject + $rcIdx = [Array]::IndexOf([object[]]$cols, 'RowCount') + if ($rcIdx -ge 0) { + foreach ($row in (Get-PrimaryRows $jsonObject)) { $rowsAdded += [int64]($row[$rcIdx] ?? 0) } + } + } + catch { } + return $rowsAdded +} + +function Invoke-BackfillFinalTable { + param([string] $FinalTableName, [string] $TransformFunctionName) + $cslText = ".set-or-append $FinalTableName <| $TransformFunctionName()" + Write-InfoLine " BACKFILL: running $cslText ..." + $startTime = [Diagnostics.Stopwatch]::StartNew() + $postResult = Invoke-KustoPost -CslText $cslText -TimeoutSec $script:IngestTimeoutSec + $startTime.Stop() + if ($postResult.Status -lt 200 -or $postResult.Status -ge 300) { + $errParts = Parse-KustoError -BodyText $postResult.Body + [Console]::Error.WriteLine(" BACKFILL FAIL: $($errParts[0]): $($errParts[1].Substring(0, [Math]::Min(300, $errParts[1].Length)))") + return [pscustomobject]@{ Ok = $false; Rows = 0L } + } + $rowsAdded = Get-RowsFromSetOrAppendResponse -BodyText $postResult.Body + Write-InfoLine (" BACKFILL OK: {0:N0} rows in {1:N1}s" -f $rowsAdded, $startTime.Elapsed.TotalSeconds) + return [pscustomobject]@{ Ok = $true; Rows = $rowsAdded } +} + +function Get-TableExtents { + param([Parameter(Mandatory)] [string] $TableName) + $postResult = Invoke-KustoPost -CslText ".show table $TableName extents | project ExtentId" -TimeoutSec 120 + if ($postResult.Status -lt 200 -or $postResult.Status -ge 300) { return @() } + try { + $jsonObject = Get-KustoJson -BodyText $postResult.Body + return @((Get-PrimaryRows $jsonObject) | ForEach-Object { [string]$_[0] }) + } + catch { return @() } +} + +function Wait-ForHealth { + param([int] $TimeoutSec = 60) + $deadline = [DateTimeOffset]::UtcNow.AddSeconds($TimeoutSec) + while ([DateTimeOffset]::UtcNow -lt $deadline) { + $postResult = Invoke-KustoPost -CslText '.show version' -TimeoutSec 5 + if ($postResult.Status -ge 200 -and $postResult.Status -lt 300) { return $true } + Start-Sleep -Seconds 2 + } + return $false +} + +function Invoke-ChunkedBackfillFinalTable { + param( + [Parameter(Mandatory)] [string] $RawTableName, + [Parameter(Mandatory)] [string] $FinalTableName, + [Parameter(Mandatory)] [string] $TransformFunctionName, + [int] $ExtentsPerBatch = 2 + ) + $extents = @(Get-TableExtents -TableName $RawTableName) + if ($extents.Count -eq 0) { + Write-InfoLine " CHUNKED BACKFILL: no extents in $RawTableName, nothing to do" + return [pscustomobject]@{ Ok = $true; Rows = 0L } + } + $totalRows = 0L + $batchCount = [int][Math]::Ceiling($extents.Count / [double]$ExtentsPerBatch) + Write-InfoLine " CHUNKED BACKFILL: $($extents.Count) extents → $batchCount batches of up to $ExtentsPerBatch extents each" + for ($batchIndex = 0; $batchIndex -lt $batchCount; $batchIndex++) { + $startIdx = $batchIndex * $ExtentsPerBatch + $endIdx = [Math]::Min($startIdx + $ExtentsPerBatch - 1, $extents.Count - 1) + $chunk = @($extents[$startIdx..$endIdx]) + $extentGuids = ($chunk | ForEach-Object { "guid($_)" }) -join ', ' + $cslText = ".set-or-append $FinalTableName <|`nlet $RawTableName = __table(`"$RawTableName`", 'All', 'AllButRowStore')`n | where extent_id() in ($extentGuids);`n$TransformFunctionName()" + if ($batchIndex -gt 0) { Start-Sleep -Seconds 5 } + $maxAttempts = 4 + $attempt = 0 + $rowsAdded = 0L + $postResult = $null + $preExtentSet = @{} + foreach ($extentId in @(Get-TableExtents -TableName $FinalTableName)) { $preExtentSet[$extentId] = $true } + while ($attempt -lt $maxAttempts) { + $attempt++ + $stopwatch = [Diagnostics.Stopwatch]::StartNew() + $postResult = Invoke-KustoPost -CslText $cslText -TimeoutSec $script:IngestTimeoutSec + $stopwatch.Stop() + if ($postResult.Status -ge 200 -and $postResult.Status -lt 300) { break } + $errParts = Parse-KustoError -BodyText $postResult.Body + $errType = [string]$errParts[0] + $errMsg = [string]$errParts[1] + $isTransient = ($errMsg.Contains('Connection') -or $errMsg.Contains('RemoteDisconnected') -or $errType.Contains('Internal service') -or $errType.Contains('LowMemoryCondition') -or $errMsg.Contains('LowMemoryCondition') -or $errMsg.ToLowerInvariant().Contains('memory') -or $errType -eq 'Unknown' -or $postResult.Status -eq 599) + if ($isTransient -and $attempt -lt $maxAttempts) { + $waitSeconds = 15 * $attempt + [Console]::Error.WriteLine(" CHUNK $($batchIndex + 1)/$batchCount attempt $attempt/$maxAttempts`: transient error ($($errType.Substring(0, [Math]::Min(40, $errType.Length))): $($errMsg.Substring(0, [Math]::Min(80, $errMsg.Length)))), waiting ${waitSeconds}s for engine recovery...") + Start-Sleep -Seconds $waitSeconds + [void](Wait-ForHealth -TimeoutSec 120) + $postExtentSet = @{} + foreach ($extentId in @(Get-TableExtents -TableName $FinalTableName)) { $postExtentSet[$extentId] = $true } + $partial = @($postExtentSet.Keys | Where-Object { -not $preExtentSet.ContainsKey($_) } | Sort-Object) + if ($partial.Count -gt 0) { + $preview = ($partial | Select-Object -First 3) -join ', ' + if ($partial.Count -gt 3) { $preview += '...' } + [Console]::Error.WriteLine(" CHUNK $($batchIndex + 1)/$batchCount`: dropping $($partial.Count) partial extent(s) from failed attempt before retry: $preview") + $partialGuids = ($partial | ForEach-Object { "guid($_)" }) -join ', ' + $dropCsl = ".drop extents <| .show table $FinalTableName extents where ExtentId in ($partialGuids)" + $dropResult = Invoke-KustoPost -CslText $dropCsl -TimeoutSec 60 + if ($dropResult.Status -lt 200 -or $dropResult.Status -ge 300) { + $dropErr = Parse-KustoError -BodyText $dropResult.Body + [Console]::Error.WriteLine(" CHUNK $($batchIndex + 1)/$batchCount ABORT: could not drop partial extents ($($dropErr[0]): $($dropErr[1].Substring(0, [Math]::Min(120, $dropErr[1].Length))))") + return [pscustomobject]@{ Ok = $false; Rows = $totalRows } + } + } + continue + } + [Console]::Error.WriteLine(" CHUNK $($batchIndex + 1)/$batchCount FAIL after $attempt attempt(s): $errType`: $($errMsg.Substring(0, [Math]::Min(200, $errMsg.Length)))") + return [pscustomobject]@{ Ok = $false; Rows = $totalRows } + } + if ($null -eq $postResult -or $postResult.Status -lt 200 -or $postResult.Status -ge 300) { return [pscustomobject]@{ Ok = $false; Rows = $totalRows } } + $rowsAdded = Get-RowsFromSetOrAppendResponse -BodyText $postResult.Body + $totalRows += $rowsAdded + Write-InfoLine (" CHUNK {0}/{1}: +{2:N0} rows in {3:N1}s" -f ($batchIndex + 1), $batchCount, $rowsAdded, $stopwatch.Elapsed.TotalSeconds) + } + return [pscustomobject]@{ Ok = $true; Rows = $totalRows } +} + +function Invoke-BackfillFinalsPerPeriod { + param([object[]] $Plans) + $appended = @{} + foreach ($rawTableName in @('Costs_raw', 'Prices_raw')) { + $policyInfo = $script:RawToFinalPolicy[$rawTableName] + $rawCount = Get-QueryScalar -CslText "$rawTableName | count" + if (-not $rawCount) { + Write-InfoLine " $($policyInfo.FinalTable): $rawTableName is empty; skipping" + $appended[$policyInfo.FinalTable] = 0 + continue + } + $result = Invoke-BackfillFinalTable -FinalTableName $policyInfo.FinalTable -TransformFunctionName $policyInfo.TransformFn + if ($result.Ok) { $appended[$policyInfo.FinalTable] = $result.Rows; continue } + [Console]::Error.WriteLine(" $($policyInfo.FinalTable): single-pass backfill failed, falling back to per-extent chunked backfill...") + [void](Wait-ForHealth -TimeoutSec 60) + $chunked = Invoke-ChunkedBackfillFinalTable -RawTableName $rawTableName -FinalTableName $policyInfo.FinalTable -TransformFunctionName $policyInfo.TransformFn -ExtentsPerBatch 1 + $appended[$policyInfo.FinalTable] = if ($chunked.Ok) { $chunked.Rows } else { -1 } + if (-not $chunked.Ok) { [Console]::Error.WriteLine(" ERROR: $($policyInfo.FinalTable) still incomplete; manual intervention required (raise Kustainer mem_limit or shrink extents).") } + } + return $appended +} + +function Drop-SupersededExtents { + param( + [string] $TableName, + [string] $ScopeName, + [string] $ExportTypeName, + [string] $PeriodName, + [string] $KeepRunUuid, + [hashtable] $ManifestIndex + ) + $supersededRuns = @{} + foreach ($entry in $ManifestIndex.GetEnumerator()) { + $rec = $entry.Value + if ($rec.scope -eq $ScopeName -and $rec.export_type -eq $ExportTypeName -and $rec.period -eq $PeriodName -and $rec.run_uuid -ne $KeepRunUuid) { + if (-not $supersededRuns.ContainsKey($rec.run_uuid)) { $supersededRuns[$rec.run_uuid] = @() } + $supersededRuns[$rec.run_uuid] = @($supersededRuns[$rec.run_uuid]) + $rec + } + } + if ($supersededRuns.Count -eq 0) { return 0L } + $totalRows = 0L + foreach ($runUuidValue in @($supersededRuns.Keys)) { + $records = @($supersededRuns[$runUuidValue]) + foreach ($rec in $records) { $totalRows += [int64]($rec.rows_ingested ?? 0) } + $dropCsl = ".drop extents <| .show table $TableName extents where tags has 'run:$runUuidValue'" + $dropResult = Invoke-KustoPost -CslText $dropCsl -TimeoutSec 120 + if ($dropResult.Status -lt 200 -or $dropResult.Status -ge 300) { + $errParts = Parse-KustoError -BodyText $dropResult.Body + [Console]::Error.WriteLine(" WARN: drop superseded extents for $TableName/$runUuidValue failed: $($errParts[0]): $($errParts[1])") + continue + } + $deleteCsl = ".delete table $script:IngestManifestTable records <| $script:IngestManifestTable | where scope == '$ScopeName' and export_type == '$ExportTypeName' and period == '$PeriodName' and run_uuid == '$runUuidValue'" + $deleteResult = Invoke-KustoPost -CslText $deleteCsl -TimeoutSec 120 + if ($deleteResult.Status -lt 200 -or $deleteResult.Status -ge 300) { + $errParts = Parse-KustoError -BodyText $deleteResult.Body + [Console]::Error.WriteLine(" WARN: delete superseded manifest rows for $runUuidValue failed: $($errParts[0]): $($errParts[1])") + } + $keysToDrop = @($ManifestIndex.Keys | Where-Object { + $rec = $ManifestIndex[$_] + $rec.scope -eq $ScopeName -and $rec.export_type -eq $ExportTypeName -and $rec.period -eq $PeriodName -and $rec.run_uuid -eq $runUuidValue + }) + foreach ($keyValue in $keysToDrop) { $ManifestIndex.Remove($keyValue) } + } + return $totalRows +} + +function Get-ContainerPath { + param([Parameter(Mandatory)] [string] $HostPathValue) + $rootFull = [IO.Path]::GetFullPath($script:ExportDirHost).TrimEnd([IO.Path]::DirectorySeparatorChar, [IO.Path]::AltDirectorySeparatorChar) + $pathFull = [IO.Path]::GetFullPath($HostPathValue) + if (-not $pathFull.StartsWith($rootFull + [IO.Path]::DirectorySeparatorChar, [StringComparison]::Ordinal)) { + throw "parquet file $HostPathValue is outside $script:ExportDirHost" + } + $relPath = $pathFull.Substring($rootFull.Length + 1).Replace([IO.Path]::DirectorySeparatorChar, '/') + return "$script:ExportDirContainer/$relPath" +} + +function Escape-KqlHUri { + param([string] $UriText) + return ($UriText -replace '"', '\"') +} + +function Invoke-DoIngest { + param( + [string] $TableName, + [string] $MappingReference, + [string] $HostPathValue, + [string] $RunUuidValue, + [string] $ScopeName, + [string] $ExportTypeName, + [string] $PeriodName + ) + $uriText = Escape-KqlHUri -UriText (Get-ContainerPath -HostPathValue $HostPathValue) + $tagItems = @("run:$RunUuidValue", "scope:$ScopeName", "type:$ExportTypeName", "period:$PeriodName") + $tagsLiteral = '[' + (($tagItems | ForEach-Object { "'$_'" }) -join ',') + ']' + $cslText = ".ingest into table $TableName (h@`"$uriText`") with (format='parquet', ingestionMappingReference='$MappingReference', tags=`"$tagsLiteral`")" + $stopwatch = [Diagnostics.Stopwatch]::StartNew() + $postResult = Invoke-KustoPost -CslText $cslText -TimeoutSec $script:IngestTimeoutSec + $stopwatch.Stop() + if ($postResult.Status -lt 200 -or $postResult.Status -ge 300) { + $errParts = Parse-KustoError -BodyText $postResult.Body + throw "ingest failed for $([IO.Path]::GetFileName($HostPathValue)): HTTP $($postResult.Status) [$($errParts[0])] $($errParts[1])" + } + try { + $jsonObject = Get-KustoJson -BodyText $postResult.Body + $rows = Get-PrimaryRows $jsonObject + if ($rows.Count -gt 0) { + foreach ($row in $rows) { if ($row[3]) { throw "ingest reported HasErrors=true for $([IO.Path]::GetFileName($HostPathValue)): $($rows | ConvertTo-Json -Compress)" } } + } + } + catch { throw "could not parse ingest response: $($_.Exception.Message); body=$($postResult.Body.Substring(0, [Math]::Min(300, $postResult.Body.Length)))" } + return [pscustomobject]@{ Rows = 0; Seconds = $stopwatch.Elapsed.TotalSeconds } +} + +function New-IngestStats { [pscustomobject]@{ files_ingested = 0; files_skipped = 0; files_failed = 0; bytes_ingested = 0L; rows_ingested = 0L; expected_rows = 0L; seconds = 0.0 } } + +function Get-ManifestRowsForBlob { + param($ManifestObject, [string] $BlobFileName) + $blobs = Get-JsonPropertyValue -ObjectValue $ManifestObject -Name 'blobs' + foreach ($blob in @($blobs)) { + $blobName = [string](Get-JsonPropertyValue -ObjectValue $blob -Name 'blobName' -DefaultValue '') + if ($blobName.EndsWith('/' + $BlobFileName, [StringComparison]::Ordinal) -or $blobName -eq $BlobFileName) { + return [int64](Get-JsonPropertyValue -ObjectValue $blob -Name 'dataRowCount' -DefaultValue 0) + } + } + return $null +} + +function Invoke-RunIngest { + param([object[]] $Plans, [bool] $DryRun = $false, [bool] $ForcePolicyRecapture = $false) + if (-not $DryRun) { Initialize-IngestManifestTable } + $manifestIndex = Fetch-ManifestIndex + if ($manifestIndex.Count -gt 0) { [Console]::Error.WriteLine("INFO: Ingest_Manifest has $($manifestIndex.Count) existing rows") } + if ($manifestIndex.Count -eq 0) { + $rawTotal = 0L + foreach ($rawTableName in @('Costs_raw', 'Prices_raw')) { + try { + $postResult = Invoke-KustoPost -CslText "$rawTableName | count" -EndpointUrl $script:KustainerQuery -TimeoutSec 30 + if ($postResult.Status -ge 200 -and $postResult.Status -lt 300) { + $jsonObject = Get-KustoJson -BodyText $postResult.Body + $rawTotal += [int64](Get-PrimaryRows $jsonObject)[0][0] + } + } catch { } + } + if ($rawTotal -gt 0) { + throw "Raw tables have $($rawTotal.ToString('N0')) rows but Ingest_Manifest is empty. The container was likely recreated and lost the manifest while leaving raw data orphaned.`n`nResolution: either drop all raw extents to start fresh`n make kql QUERY=`".clear table Costs_raw data`"`n make kql QUERY=`".clear table Prices_raw data`"`n…then re-run, OR re-populate Ingest_Manifest from a backup. Refusing to proceed — silently treating every file as new would double-ingest the orphaned data." + } + } + + $fileUnits = @() + foreach ($plan in $Plans) { + $manifestObject = Read-RunManifest -RunDirPath $plan.RunDir + foreach ($part in @($plan.ParquetFiles)) { + $perBlob = Get-ManifestRowsForBlob -ManifestObject $manifestObject -BlobFileName $part.Name + if ($null -eq $perBlob) { $perBlob = 0L } + $fileUnits += [pscustomobject]@{ Plan = $plan; HostPath = $part; ExpectedRows = [int64]$perBlob } + } + } + $totalFiles = $fileUnits.Count + if ($totalFiles -eq 0) { Write-InfoLine 'no parquet files matched filters; nothing to do'; return @{} } + Write-InfoLine "PLAN: $($Plans.Count) (scope,type,period) tuples → $totalFiles parquet parts" + foreach ($plan in $Plans) { + $sup = if ($plan.SupersededRunUuids.Count -gt 0) { " (supersedes $($plan.SupersededRunUuids.Count) prior run-uuids)" } else { '' } + Write-InfoLine (" - {0}/{1}/{2} → run {3}… {4} part(s) · {5:N0} rows expected{6}" -f $plan.Scope, $plan.ExportType, $plan.Period, $plan.RunUuid.Substring(0, [Math]::Min(8, $plan.RunUuid.Length)), $plan.ParquetFiles.Count, $plan.ExpectedRows, $sup) + } + + if (-not $DryRun) { + foreach ($plan in $Plans) { + $tableName = $script:DatasetTableMap[$plan.ExportType].Table + $droppedRows = Drop-SupersededExtents -TableName $tableName -ScopeName $plan.Scope -ExportTypeName $plan.ExportType -PeriodName $plan.Period -KeepRunUuid $plan.RunUuid -ManifestIndex $manifestIndex + if ($droppedRows) { Write-InfoLine (" OVERWRITE: dropped {0:N0} rows from {1} for superseded run(s) of {2}/{3}/{4}" -f $droppedRows, $tableName, $plan.Scope, $plan.ExportType, $plan.Period) } + } + } + + $statsByDataset = @{} + $overall = [Diagnostics.Stopwatch]::StartNew() + $savedPolicies = @{} + if (-not $DryRun) { $savedPolicies = Disable-UpdatePolicies -ForceRecapture $ForcePolicyRecapture } + $paceFloorSec = if ($env:INGEST_PACE_FLOOR_S) { [double]$env:INGEST_PACE_FLOOR_S } else { 1.5 } + $pacePerMbSec = if ($env:INGEST_PACE_PER_MB_S) { [double]$env:INGEST_PACE_PER_MB_S } else { 0.05 } + try { + for ($unitIndex = 0; $unitIndex -lt $fileUnits.Count; $unitIndex++) { + $unit = $fileUnits[$unitIndex] + $plan = $unit.Plan + $mapInfo = $script:DatasetTableMap[$plan.ExportType] + $bucketKey = "$($plan.Scope)$([char]31)$($plan.ExportType)" + if (-not $statsByDataset.ContainsKey($bucketKey)) { $statsByDataset[$bucketKey] = New-IngestStats } + $bucket = $statsByDataset[$bucketKey] + $bucket.expected_rows += [int64]$unit.ExpectedRows + $sizeBytes = [int64]$unit.HostPath.Length + $bucket.bytes_ingested += $sizeBytes + $sha = Get-FileSha256Lower -PathValue $unit.HostPath.FullName + $manifestKey = Join-ManifestKey -ScopeName $plan.Scope -ExportTypeName $plan.ExportType -PeriodName $plan.Period -RunUuidValue $plan.RunUuid -FileNameValue $unit.HostPath.Name + $prior = if ($manifestIndex.ContainsKey($manifestKey)) { $manifestIndex[$manifestKey] } else { $null } + $displayIndex = $unitIndex + 1 + if ($prior -and $prior.checksum_sha256 -eq $sha) { + $bucket.files_skipped++ + Write-InfoLine ("[{0,3}/{1}] SKIP {2}/{3}/{4}/{5} ({6} KB, {7:N0} rows) already in Ingest_Manifest with matching checksum" -f $displayIndex, $totalFiles, $plan.Scope, $plan.ExportType, $plan.Period, $unit.HostPath.Name, [Math]::Floor($sizeBytes / 1024), [int64]$unit.ExpectedRows) + $bucket.rows_ingested += [int64]($prior.rows_ingested ?? 0) + continue + } + $progressPrefix = ("[{0,3}/{1}] ingesting {2}/{3}/{4}/{5} ({6} KB, {7:N0} rows expected)" -f $displayIndex, $totalFiles, $plan.Scope, $plan.ExportType, $plan.Period, $unit.HostPath.Name, [Math]::Floor($sizeBytes / 1024), [int64]$unit.ExpectedRows) + Write-Host -NoNewline ($progressPrefix + ' ... ') + if ($DryRun) { Write-InfoLine 'dry-run skip'; continue } + $paceSec = $paceFloorSec + ($sizeBytes / (1024.0 * 1024.0)) * $pacePerMbSec + if ($displayIndex -gt 1 -and $paceSec -gt 0) { Start-Sleep -Seconds $paceSec } + try { + $ingestResult = Invoke-DoIngest -TableName $mapInfo.Table -MappingReference $mapInfo.Mapping -HostPathValue $unit.HostPath.FullName -RunUuidValue $plan.RunUuid -ScopeName $plan.Scope -ExportTypeName $plan.ExportType -PeriodName $plan.Period + } + catch { + $bucket.files_failed++ + Write-InfoLine "FAIL ($($_.Exception.Message))" + if ($_.Exception.Message.Contains('Connection reset') -or $_.Exception.Message.Contains('RemoteDisconnected')) { + Write-InfoLine ' (engine may have crashed; waiting 30s for recovery)' + Start-Sleep -Seconds 30 + } + continue + } + $bucket.files_ingested++ + $bucket.rows_ingested += [int64]$unit.ExpectedRows + $bucket.seconds += [double]$ingestResult.Seconds + Write-InfoLine ("{0:N1}s OK" -f $ingestResult.Seconds) + $recordObject = [pscustomobject]@{ + scope = $plan.Scope; export_type = $plan.ExportType; period = $plan.Period; run_uuid = $plan.RunUuid + file_name = $unit.HostPath.Name; file_size = $sizeBytes; rows_ingested = [int64]$unit.ExpectedRows + checksum_sha256 = $sha; ingested_at = (Get-Date).ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ssZ') + } + try { Insert-ManifestRow -RecordObject $recordObject; $manifestIndex[$manifestKey] = $recordObject } + catch { [Console]::Error.WriteLine(" WARN: could not record Ingest_Manifest row: $($_.Exception.Message)") } + } + } + finally { + if ($savedPolicies.Count -gt 0) { Restore-UpdatePolicies -SavedPolicies $savedPolicies } + } + $overall.Stop() + Write-InfoLine ("`nINGEST WALL-CLOCK: {0:N1}s ({1:N1} min)" -f $overall.Elapsed.TotalSeconds, ($overall.Elapsed.TotalSeconds / 60.0)) + $anyIngested = $false + foreach ($statsItem in $statsByDataset.Values) { if ($statsItem.files_ingested -gt 0) { $anyIngested = $true; break } } + if (-not $DryRun -and $anyIngested) { + Write-InfoLine "`n=== Backfilling final tables via FTK transforms ===" + [void](Invoke-BackfillFinalsPerPeriod -Plans $Plans) + } + return $statsByDataset +} + +function Invoke-VerifyAndSummarize { + param([object[]] $Plans, [hashtable] $StatsByDataset) + Write-InfoLine ('') + Write-InfoLine ('=' * 78) + Write-InfoLine 'INGEST SUMMARY' + Write-InfoLine ('=' * 78) + $anyMismatch = $false + foreach ($bucketKey in @($StatsByDataset.Keys | Sort-Object)) { + $parts = $bucketKey -split [string][char]31, 2 + $scopeName = $parts[0] + $exportTypeName = $parts[1] + $stats = $StatsByDataset[$bucketKey] + $tableName = $script:DatasetTableMap[$exportTypeName].Table + Write-InfoLine (" {0,10} / {1,-16} → {2,-14} files ok={3} skip={4} fail={5} manifest-rows={6,11:N0} size={7,7:N1} MB ingest-time={8:N1}s" -f $scopeName, $exportTypeName, $tableName, $stats.files_ingested, $stats.files_skipped, $stats.files_failed, $stats.expected_rows, ($stats.bytes_ingested / 1024.0 / 1024.0), $stats.seconds) + if ($stats.files_failed) { $anyMismatch = $true } + } + $declared = @{} + $bucketFilters = @{} + foreach ($bucketKey in $StatsByDataset.Keys) { + $parts = $bucketKey -split [string][char]31, 2 + $scopeName = $parts[0] + $exportTypeName = $parts[1] + $tableName = $script:DatasetTableMap[$exportTypeName].Table + if (-not $declared.ContainsKey($tableName)) { $declared[$tableName] = 0L; $bucketFilters[$tableName] = @() } + $declared[$tableName] += [int64]$StatsByDataset[$bucketKey].expected_rows + $bucketFilters[$tableName] = @($bucketFilters[$tableName]) + [pscustomobject]@{ Scope = $scopeName; ExportType = $exportTypeName } + } + Write-InfoLine '' + foreach ($tableName in @($declared.Keys | Sort-Object)) { + $bucketPairs = (@($bucketFilters[$tableName]) | ForEach-Object { "(scope=='$($_.Scope)' and export_type=='$($_.ExportType)')" }) -join ', ' + $actualCsl = "$script:IngestManifestTable | where $bucketPairs | summarize sum(rows_ingested)" + $got = Get-QueryScalar -CslText $actualCsl + if ($null -eq $got) { Write-InfoLine " $tableName`: count query failed (cannot verify)"; $anyMismatch = $true; continue } + $want = [int64]$declared[$tableName] + $diff = [Math]::Abs([int64]$got - $want) + $pct = if ($want -gt 0) { $diff / [double]$want } else { 0.0 } + $status = if ($pct -le $script:RowCountTolerance) { 'OK' } else { 'MISMATCH' } + if ($status -eq 'MISMATCH') { $anyMismatch = $true } + Write-InfoLine (" {0}: ingested={1,11:N0} manifest-declared={2,11:N0} diff={3,7:N0} ({4:N4}%) {5}" -f $tableName, [int64]$got, $want, $diff, ($pct * 100.0), $status) + $full = Get-QueryScalar -CslText "$tableName | count" + if ($null -ne $full) { Write-InfoLine (" (table total across all runs: {0:N0})" -f [int64]$full) } + } + Write-InfoLine '' + $summaryCsl = "$script:IngestManifestTable | summarize files=count(), rows=sum(rows_ingested) by scope, export_type | order by scope asc, export_type asc" + $postResult = Invoke-KustoPost -CslText $summaryCsl -EndpointUrl $script:KustainerQuery -TimeoutSec 60 + if ($postResult.Status -ge 200 -and $postResult.Status -lt 300) { + $jsonObject = Get-KustoJson -BodyText $postResult.Body + $cols = Get-PrimaryColumnNames $jsonObject + $rows = Get-PrimaryRows $jsonObject + Write-InfoLine 'Ingest_Manifest summary:' + Write-InfoLine (' ' + (($cols | ForEach-Object { '{0,14}' -f $_ }) -join ' | ')) + foreach ($row in $rows) { Write-InfoLine (' ' + (($row | ForEach-Object { '{0,14}' -f ([string]$_) }) -join ' | ')) } + } + return (-not $anyMismatch) +} + +function Read-IngestOptions { + param([string[]] $Tokens) + $Tokens = @($Tokens | Where-Object { $null -ne $_ -and $_ -ne '' }) + $opts = @{ Scope = $null; Period = $null; DryRun = $false; ForcePolicyRecapture = $false } + $idx = 0 + while ($idx -lt $Tokens.Count) { + switch -Regex ($Tokens[$idx]) { + '^--scope$|^-Scope$' { $idx++; if ($idx -ge $Tokens.Count) { throw 'argument --scope: expected one argument' }; $opts.Scope = $Tokens[$idx] } + '^--period$|^-Period$' { $idx++; if ($idx -ge $Tokens.Count) { throw 'argument --period: expected one argument' }; $opts.Period = $Tokens[$idx] } + '^--dry-run$|^-DryRun$' { $opts.DryRun = $true } + '^--force-policy-recapture$|^-ForcePolicyRecapture$' { $opts.ForcePolicyRecapture = $true } + '^--help$|^-h$|^-\?$' { $opts.Help = $true } + default { throw "unrecognized arguments: $($Tokens[$idx])" } + } + $idx++ + } + return $opts +} + +function Show-IngestUsage { + @' +usage: ingest.ps1 [--scope SCOPE] [--period PERIOD] [--dry-run] [--force-policy-recapture] + +Bulk-ingest FOCUS Cost + Price Sheet parquet exports under ./export/ into Kustainer raw tables. +'@ +} + +function Invoke-IngestMain { + param([string[]] $Arguments) + try { $opts = Read-IngestOptions -Tokens @($Arguments) } + catch { [Console]::Error.WriteLine("error: $($_.Exception.Message)"); return 2 } + if ($opts.ContainsKey('Help') -and $opts.Help) { Show-IngestUsage; return 0 } + try { + $plans = Build-Plan -ExportRootPath $script:ExportDirHost -ScopeFilter $opts.Scope -PeriodFilter $opts.Period + if ($plans.Count -eq 0) { Write-InfoLine 'no (scope,type,period) tuples matched filters; nothing to do'; return 0 } + $stats = Invoke-RunIngest -Plans $plans -DryRun ([bool]$opts.DryRun) -ForcePolicyRecapture ([bool]$opts.ForcePolicyRecapture) + if ($opts.DryRun) { Write-InfoLine "`n(dry-run; no actual ingest occurred)"; return 0 } + $ok = Invoke-VerifyAndSummarize -Plans $plans -StatsByDataset $stats + if ($ok) { return 0 } + return 2 + } + catch { [Console]::Error.WriteLine("error: $($_.Exception.Message)"); return 1 } +} + +if ($MyInvocation.InvocationName -ne '.') { + exit (Invoke-IngestMain -Arguments @($CliArgs)) +} diff --git a/src/templates/finops-hub-local/scripts/load-ftk-kql.ps1 b/src/templates/finops-hub-local/scripts/load-ftk-kql.ps1 new file mode 100644 index 000000000..165b80066 --- /dev/null +++ b/src/templates/finops-hub-local/scripts/load-ftk-kql.ps1 @@ -0,0 +1,494 @@ +#!/usr/bin/env pwsh +<# +.SYNOPSIS + Load the FinOps Toolkit analytics KQL into local Kustainer (two-DB topology). + +.DESCRIPTION + Reads the combineKql bundles from src/templates/finops-hub/.build.config, + concatenates the listed .kql files in dependency order, and submits each + bundle to its target database. + + Two persisted databases are created idempotently: + Ingestion -- raw tables, transforms, final tables, open-data lookups + Hub -- view functions referencing database('Ingestion').* + + The $$rawRetentionInDays$$ macro in the Ingestion bundle is replaced with + RawRetentionDays (default 3650). The Hub bundle is submitted verbatim -- + database('Ingestion'). cross-DB references are preserved and resolve once + both databases exist. + + Open-data CSVs (PricingUnits/Regions/ResourceTypes/Services) are ingested + into the Ingestion database. + + Connection: + KUSTAINER_MGMT full mgmt URL (default http://localhost:/v1/rest/mgmt) + HOST_PORT port (default 8082; read from .env if present) + + FTK source paths (all derived from repo root by default): + FTK_BUILD_CONFIG path to finops-hub .build.config + FTK_OPEN_DATA overrides src/open-data path + FTK_REPO repo root override + +.EXAMPLE + pwsh scripts/load-ftk-kql.ps1 + +.EXAMPLE + pwsh scripts/load-ftk-kql.ps1 --dry-run +#> +[CmdletBinding()] +param( + [switch] $DryRun, + [switch] $SkipOpenData, + [switch] $SkipKql, + [Parameter(ValueFromRemainingArguments = $true)] + [string[]] $RemainingArgs +) + +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +if ($RemainingArgs) { + for ($argIndex = 0; $argIndex -lt $RemainingArgs.Count; $argIndex++) { + switch ($RemainingArgs[$argIndex]) { + '--dry-run' { $DryRun = $true } + '--skip-open-data' { $SkipOpenData = $true } + '--skip-kql' { $SkipKql = $true } + default { throw "unrecognized argument: $($RemainingArgs[$argIndex])" } + } + } +} + +# Script lives at /src/templates/finops-hub-local/scripts/ +$script:FtkLocalHome = Split-Path -Parent (Split-Path -Parent $PSCommandPath) +$script:RepoRoot = Split-Path -Parent (Split-Path -Parent (Split-Path -Parent $script:FtkLocalHome)) +$script:RawRetentionDays = '3650' +$script:TopLevelCommandRegex = [regex]'^\.[a-z]' +$script:IdempotentOkTypes = @('Kusto.Common.Svc.Exceptions.EntityNameAlreadyExistsException') + +# --------------------------------------------------------------------------- # +# Configuration / path resolution +# --------------------------------------------------------------------------- # +function Get-HostPort { + if ($env:HOST_PORT) { return $env:HOST_PORT } + $envFile = Join-Path $script:FtkLocalHome '.env' + if (Test-Path $envFile) { + foreach ($line in Get-Content -Path $envFile) { + $trimmed = $line.Trim() + if ($trimmed -like 'HOST_PORT=*') { return ($trimmed -split '=', 2)[1].Trim() } + } + } + return '8082' +} + +function Get-RepoRoot { + if ($env:FTK_REPO) { return $env:FTK_REPO } + return $script:RepoRoot +} + +function Get-BuildConfigPath { + if ($env:FTK_BUILD_CONFIG) { return $env:FTK_BUILD_CONFIG } + return Join-Path (Get-RepoRoot) 'src/templates/finops-hub/.build.config' +} + +function Get-OpenDataPath { + if ($env:FTK_OPEN_DATA) { return $env:FTK_OPEN_DATA } + return Join-Path (Get-RepoRoot) 'src/open-data' +} + +function Get-MgmtUrl { + if ($env:KUSTAINER_MGMT) { return $env:KUSTAINER_MGMT } + return "http://localhost:$(Get-HostPort)/v1/rest/mgmt" +} + +function Get-QueryUrl { + return ((Get-MgmtUrl) -replace '/mgmt$', '/query') +} + +$script:OpenData = Get-OpenDataPath +$script:MgmtUrl = Get-MgmtUrl + +# --------------------------------------------------------------------------- # +# Bundle builder -- reads .build.config and concatenates .kql files in order +# --------------------------------------------------------------------------- # +function Build-KqlBundle { + param([Parameter(Mandatory)] [string] $BundleName) + $configPath = Get-BuildConfigPath + $config = Get-Content -Raw -Path $configPath | ConvertFrom-Json + $entry = $config.combineKql | Where-Object { $_.name -eq $BundleName } | Select-Object -First 1 + if (-not $entry) { throw "combineKql entry '$BundleName' not found in $(Split-Path -Leaf $configPath)" } + $templateDir = Split-Path -Parent $configPath + $parts = foreach ($relPath in $entry.files) { + Get-Content -Raw -Path (Join-Path $templateDir $relPath) + } + return ($parts -join "`n") +} + +function Invoke-SubstMacros { + param([Parameter(Mandatory)] [string] $Text) + return $Text.Replace('$$rawRetentionInDays$$', $script:RawRetentionDays) +} + +# --------------------------------------------------------------------------- # +# KQL command splitter +# --------------------------------------------------------------------------- # +function Split-KqlCommands { + param([Parameter(Mandatory)] [string] $ScriptText) + $lines = $ScriptText -split "(?<=`n)", 0 + if ($lines.Count -gt 0 -and $lines[-1] -eq '') { $lines = $lines[0..($lines.Count - 2)] } + + $blocks = [System.Collections.Generic.List[object]]::new() + $current = [System.Collections.Generic.List[string]]::new() + foreach ($line in $lines) { + if ($script:TopLevelCommandRegex.IsMatch($line)) { + if ($current.Count -gt 0) { [void]$blocks.Add([string[]]$current.ToArray()) } + $current = [System.Collections.Generic.List[string]]::new() + [void]$current.Add($line) + } + else { [void]$current.Add($line) } + } + if ($current.Count -gt 0) { [void]$blocks.Add([string[]]$current.ToArray()) } + + $cleaned = [System.Collections.Generic.List[string]]::new() + foreach ($blockObject in $blocks) { + $block = [System.Collections.Generic.List[string]]::new() + foreach ($entry in [string[]]$blockObject) { [void]$block.Add($entry) } + $hasCommand = $false + foreach ($entry in $block) { + if ($script:TopLevelCommandRegex.IsMatch($entry)) { $hasCommand = $true; break } + } + if (-not $hasCommand) { continue } + while ($block.Count -gt 0) { + $last = $block[$block.Count - 1] + if ($last.Trim() -eq '' -or $last.TrimStart().StartsWith('//')) { $block.RemoveAt($block.Count - 1) } + else { break } + } + $text = ([string]::Concat([string[]]$block.ToArray())).TrimEnd() + if ($text) { [void]$cleaned.Add($text) } + } + return [string[]]$cleaned.ToArray() +} + +# --------------------------------------------------------------------------- # +# Kustainer REST poster +# --------------------------------------------------------------------------- # +function ConvertTo-BodyString { + param($Value) + if ($null -eq $Value) { return '' } + if ($Value -is [string]) { return $Value } + return ($Value | ConvertTo-Json -Depth 100 -Compress) +} + +function Post-Kql { + param( + [Parameter(Mandatory)] [string] $Csl, + [Parameter(Mandatory)] [string] $DatabaseName, + [int] $TimeoutSec = 60 + ) + $payload = @{ db = $DatabaseName; csl = $Csl } | ConvertTo-Json -Compress + $headers = @{ + 'Content-Type' = 'application/json' + 'x-ms-client-version' = 'Kusto.Python.Client:1.0.0' + 'Accept' = 'application/json' + } + try { + $statusCodeValue = $null + $responseBody = Invoke-RestMethod -Method Post -Uri $script:MgmtUrl -Headers $headers -Body $payload ` + -TimeoutSec $TimeoutSec -SkipHttpErrorCheck -StatusCodeVariable statusCodeValue + return [pscustomobject]@{ Status = [int]$statusCodeValue; Body = (ConvertTo-BodyString $responseBody) } + } + catch { + return [pscustomobject]@{ Status = 599; Body = "URL error: $($_.Exception.GetType().FullName): $($_.Exception.Message)" } + } +} + +function Parse-KustoError { + param([Parameter(Mandatory)] [string] $Body) + try { + $json = $Body | ConvertFrom-Json + $err = $null + if ($json.PSObject.Properties.Name -contains 'error') { $err = $json.error } + if ($null -ne $err) { + $typeValue = if ($err.PSObject.Properties.Name -contains '@type' -and $err.'@type') { $err.'@type' } + elseif ($err.PSObject.Properties.Name -contains 'code' -and $err.code) { $err.code } + else { 'Unknown' } + $messageValue = if ($err.PSObject.Properties.Name -contains '@message' -and $err.'@message') { $err.'@message' } + elseif ($err.PSObject.Properties.Name -contains 'message' -and $err.message) { $err.message } + else { $Body.Substring(0, [Math]::Min(200, $Body.Length)) } + return @([string]$typeValue, [string]$messageValue) + } + return @('Unknown', $Body.Substring(0, [Math]::Min(200, $Body.Length))) + } + catch { return @('RawText', $Body.Substring(0, [Math]::Min(300, $Body.Length))) } +} + +function Test-IdempotentSuccess { + param([Parameter(Mandatory)] [string] $Body) + return ($script:IdempotentOkTypes -contains (Parse-KustoError -Body $Body)[0]) +} + +# --------------------------------------------------------------------------- # +# Step reporting and execution +# --------------------------------------------------------------------------- # +function New-StepReport { + param([Parameter(Mandatory)] [string] $Label) + return [pscustomobject]@{ + Label = $Label + Total = 0 + Ok = 0 + IdempotentOk = 0 + Dropped = [System.Collections.Generic.List[object]]::new() + Failed = [System.Collections.Generic.List[object]]::new() + } +} + +function Get-FirstCommandLine { + param([Parameter(Mandatory)] [string] $CommandText) + foreach ($line in ($CommandText -split "`n")) { + if ($script:TopLevelCommandRegex.IsMatch($line)) { return $line.Trim() } + } + $lines = $CommandText -split "`n" + if ($lines.Count -gt 0) { return $lines[0] } + return '' +} + +function Invoke-BundleLoad { + param( + [Parameter(Mandatory)] [string] $Label, + [Parameter(Mandatory)] [string] $BundleText, + [Parameter(Mandatory)] [string] $DatabaseName, + [bool] $IsDryRun = $false + ) + $report = New-StepReport -Label $Label + $commands = @(Split-KqlCommands -ScriptText $BundleText) + $report.Total = $commands.Count + Write-Host "" + Write-Host "=== $Label [$DatabaseName] ($($report.Total) commands) ===" + for ($i = 0; $i -lt $commands.Count; $i++) { + $commandText = $commands[$i] + $firstLine = Get-FirstCommandLine -CommandText $commandText + $preview = if ($firstLine.Length -gt 110) { $firstLine.Substring(0, 110) } else { $firstLine } + if ($IsDryRun) { + Write-Host (' [{0,3}] dry {1}' -f ($i + 1), $preview) + continue + } + $result = Post-Kql -Csl $commandText -DatabaseName $DatabaseName + if ($result.Status -ge 200 -and $result.Status -lt 300) { + $report.Ok++ + Write-Host (' [{0,3}] OK {1}' -f ($i + 1), $preview) + } + elseif (Test-IdempotentSuccess -Body $result.Body) { + $report.IdempotentOk++ + $parsed = Parse-KustoError -Body $result.Body + Write-Host (' [{0,3}] OK* {1} -- {2}' -f ($i + 1), $preview, $parsed[1]) + } + else { + $parsed = Parse-KustoError -Body $result.Body + $short = "[$($parsed[0])] $($parsed[1])" + if ($short.Length -gt 400) { $short = $short.Substring(0, 400) } + [void]$report.Failed.Add([pscustomobject]@{ Command = $preview; Error = "HTTP $($result.Status): $short" }) + Write-Host (' [{0,3}] FAIL {1}' -f ($i + 1), $preview) + Write-Host " HTTP $($result.Status): $short" + } + } + return $report +} + +# --------------------------------------------------------------------------- # +# Open-data CSV ingestion (into Ingestion database) +# --------------------------------------------------------------------------- # +function ConvertTo-InlineCsvBlock { + param([Parameter(Mandatory)] [object[]] $Rows) + $builder = [System.Text.StringBuilder]::new() + foreach ($row in $Rows) { + $cells = foreach ($cell in $row) { + $value = if ($null -eq $cell) { '' } else { [string]$cell } + '"' + ($value -replace '"', '""') + '"' + } + [void]$builder.AppendLine(($cells -join ',')) + } + return $builder.ToString() +} + +function Read-CsvRows { + param([Parameter(Mandatory)] [string] $Path) + $csvRows = @(Import-Csv -Path $Path) + if ($csvRows.Count -gt 0) { + $headers = [string[]]@($csvRows[0].PSObject.Properties.Name) + } + else { + $headerLine = Get-Content -Path $Path -TotalCount 1 + $headers = if ($headerLine) { [string[]]($headerLine -split ',' | ForEach-Object { $_.Trim().Trim('"') }) } else { [string[]]@() } + } + return [pscustomobject]@{ Headers = $headers; Rows = $csvRows } +} + +function Invoke-OpenDataIngestion { + $db = 'Ingestion' + $report = New-StepReport -Label 'Open-data CSV ingestion (PricingUnits/Regions/ResourceTypes/Services)' + $plans = @( + [pscustomobject]@{ Table = 'PricingUnits'; Path = (Join-Path $script:OpenData 'PricingUnits.csv'); Columns = @( + [pscustomobject]@{ Target = 'x_PricingUnitDescription'; Source = 'UnitOfMeasure'; Type = 'string' }, + [pscustomobject]@{ Target = 'x_PricingBlockSize'; Source = 'PricingBlockSize'; Type = 'real' }, + [pscustomobject]@{ Target = 'PricingUnit'; Source = 'DistinctUnits'; Type = 'string' } + ) }, + [pscustomobject]@{ Table = 'Regions'; Path = (Join-Path $script:OpenData 'Regions.csv'); Columns = @( + [pscustomobject]@{ Target = 'ResourceLocation'; Source = 'OriginalValue'; Type = 'string' }, + [pscustomobject]@{ Target = 'RegionId'; Source = 'RegionId'; Type = 'string' }, + [pscustomobject]@{ Target = 'RegionName'; Source = 'RegionName'; Type = 'string' } + ) }, + [pscustomobject]@{ Table = 'ResourceTypes'; Path = (Join-Path $script:OpenData 'ResourceTypes.csv'); Columns = @( + [pscustomobject]@{ Target = 'x_ResourceType'; Source = 'ResourceType'; Type = 'string' }, + [pscustomobject]@{ Target = 'SingularDisplayName'; Source = 'SingularDisplayName'; Type = 'string' }, + [pscustomobject]@{ Target = 'PluralDisplayName'; Source = 'PluralDisplayName'; Type = 'string' }, + [pscustomobject]@{ Target = 'LowerSingularDisplayName'; Source = 'LowerSingularDisplayName'; Type = 'string' }, + [pscustomobject]@{ Target = 'LowerPluralDisplayName'; Source = 'LowerPluralDisplayName'; Type = 'string' }, + [pscustomobject]@{ Target = 'IsPreview'; Source = 'IsPreview'; Type = 'bool' }, + [pscustomobject]@{ Target = 'Description'; Source = 'Description'; Type = 'string' }, + [pscustomobject]@{ Target = 'IconUri'; Source = 'Icon'; Type = 'string' } + ) }, + [pscustomobject]@{ Table = 'Services'; Path = (Join-Path $script:OpenData 'Services.csv'); Columns = @( + [pscustomobject]@{ Target = 'x_ConsumedService'; Source = 'ConsumedService'; Type = 'string' }, + [pscustomobject]@{ Target = 'x_ResourceType'; Source = 'ResourceType'; Type = 'string' }, + [pscustomobject]@{ Target = 'ServiceName'; Source = 'ServiceName'; Type = 'string' }, + [pscustomobject]@{ Target = 'ServiceCategory'; Source = 'ServiceCategory'; Type = 'string' }, + [pscustomobject]@{ Target = 'ServiceSubcategory'; Source = 'ServiceSubcategory'; Type = 'string' }, + [pscustomobject]@{ Target = 'PublisherName'; Source = 'PublisherName'; Type = 'string' }, + [pscustomobject]@{ Target = 'x_PublisherCategory'; Source = 'PublisherType'; Type = 'string' }, + [pscustomobject]@{ Target = 'x_Environment'; Source = 'Environment'; Type = 'string' }, + [pscustomobject]@{ Target = 'x_ServiceModel'; Source = 'ServiceModel'; Type = 'string' } + ) } + ) + + Write-Host "" + Write-Host "=== $($report.Label) [$db] ===" + foreach ($plan in $plans) { + $report.Total++ + try { + $csvData = Read-CsvRows -Path $plan.Path + foreach ($col in $plan.Columns) { + if ($csvData.Headers -notcontains $col.Source) { throw "'$($col.Source)' not found in CSV headers" } + } + } + catch { + [void]$report.Failed.Add([pscustomobject]@{ Command = $plan.Table; Error = "CSV header mismatch: $($_.Exception.Message)" }) + Write-Host " FAIL $($plan.Table): header mismatch $($_.Exception.Message)" + continue + } + + $projectedRows = foreach ($csvRow in $csvData.Rows) { + $rowValues = foreach ($col in $plan.Columns) { + $prop = $csvRow.PSObject.Properties[$col.Source] + if ($null -eq $prop -or $null -eq $prop.Value) { '' } else { [string]$prop.Value } + } + ,([object[]]$rowValues) + } + $csvBody = ConvertTo-InlineCsvBlock -Rows @($projectedRows) + + $clearResult = Post-Kql -Csl ".clear table $($plan.Table) data" -DatabaseName $db + if (-not ($clearResult.Status -ge 200 -and $clearResult.Status -lt 300)) { + [void]$report.Failed.Add([pscustomobject]@{ Command = $plan.Table; Error = "clear failed: HTTP $($clearResult.Status): $($clearResult.Body.Substring(0, [Math]::Min(200, $clearResult.Body.Length)))" }) + Write-Host " FAIL $($plan.Table): clear failed $($clearResult.Status)" + continue + } + + $ingestCommand = ".ingest inline into table $($plan.Table) with (format='csv') <|`n$($csvBody.TrimEnd("`r", "`n"))" + $ingestResult = Post-Kql -Csl $ingestCommand -DatabaseName $db -TimeoutSec 180 + if (-not ($ingestResult.Status -ge 200 -and $ingestResult.Status -lt 300)) { + $shortBody = $ingestResult.Body.Substring(0, [Math]::Min(200, $ingestResult.Body.Length)) + [void]$report.Failed.Add([pscustomobject]@{ Command = $plan.Table; Error = "ingest failed: HTTP $($ingestResult.Status): $shortBody" }) + Write-Host " FAIL $($plan.Table): ingest failed $($ingestResult.Status): $shortBody" + continue + } + + try { + $countPayload = @{ db = $db; csl = "$($plan.Table) | count" } | ConvertTo-Json -Compress + $queryResponse = Invoke-RestMethod -Method Post -Uri (Get-QueryUrl) -Headers @{ 'Content-Type' = 'application/json' } -Body $countPayload -TimeoutSec 30 + $count = [int]$queryResponse.Tables[0].Rows[0][0] + $expected = @($csvData.Rows).Count + if ($count -ne $expected) { + [void]$report.Failed.Add([pscustomobject]@{ Command = $plan.Table; Error = "row count $count != expected $expected" }) + Write-Host " FAIL $($plan.Table): row count $count != expected $expected" + continue + } + $report.Ok++ + Write-Host " OK $($plan.Table): $count rows" + } + catch { + [void]$report.Failed.Add([pscustomobject]@{ Command = $plan.Table; Error = "count verification failed: $($_.Exception.Message)" }) + Write-Host " FAIL $($plan.Table): count verification $($_.Exception.Message)" + } + } + return $report +} + +# --------------------------------------------------------------------------- # +# Summary / main +# --------------------------------------------------------------------------- # +function Write-Summary { + param([Parameter(Mandatory)] [object[]] $Reports) + Write-Host "" + Write-Host ('=' * 78) + Write-Host 'SUMMARY' + Write-Host ('=' * 78) + $anyFailed = $false + foreach ($rpt in $Reports) { + Write-Host (" $($rpt.Label): total=$($rpt.Total) ok=$($rpt.Ok) idempotent_ok=$($rpt.IdempotentOk) dropped=$($rpt.Dropped.Count) failed=$($rpt.Failed.Count)") + if ($rpt.Failed.Count -gt 0) { + $anyFailed = $true + foreach ($failure in $rpt.Failed) { + Write-Host " FAIL: $($failure.Command)" + Write-Host " $($failure.Error)" + } + } + } + return (-not $anyFailed) +} + +try { + $reports = [System.Collections.Generic.List[object]]::new() + + if (-not $SkipKql) { + # 1. Bootstrap: create Ingestion and Hub databases (idempotent) + foreach ($dbName in @('Ingestion', 'Hub')) { + $csl = ".create database $dbName persist (@'/kustodata/dbs/$dbName/md', @'/kustodata/dbs/$dbName/data')" + if ($DryRun) { + Write-Host " BOOTSTRAP dry .create database $dbName persist ..." + } + else { + $result = Post-Kql -Csl $csl -DatabaseName 'NetDefaultDB' + if ($result.Status -ge 200 -and $result.Status -lt 300) { + Write-Host " BOOTSTRAP Created database '$dbName'." + } + elseif (Test-IdempotentSuccess -Body $result.Body) { + Write-Host " BOOTSTRAP Database '$dbName' already exists (idempotent)." + } + else { + $parsed = Parse-KustoError -Body $result.Body + throw "Failed to create database '$dbName': [$($parsed[0])] $($parsed[1])" + } + } + } + + # 2. Ingestion bundle -- apply $$rawRetentionInDays$$ macro substitution + $ingestionText = Invoke-SubstMacros -Text (Build-KqlBundle -BundleName 'finops-hub-fabric-setup-Ingestion.kql') + [void]$reports.Add((Invoke-BundleLoad -Label 'finops-hub-fabric-setup-Ingestion.kql' ` + -BundleText $ingestionText -DatabaseName 'Ingestion' -IsDryRun:$DryRun.IsPresent)) + + # 3. Hub bundle -- submitted verbatim; database('Ingestion'). refs preserved + $hubText = Build-KqlBundle -BundleName 'finops-hub-fabric-setup-Hub.kql' + [void]$reports.Add((Invoke-BundleLoad -Label 'finops-hub-fabric-setup-Hub.kql' ` + -BundleText $hubText -DatabaseName 'Hub' -IsDryRun:$DryRun.IsPresent)) + } + + if (-not $SkipOpenData -and -not $DryRun) { + [void]$reports.Add((Invoke-OpenDataIngestion)) + } + + $ok = Write-Summary -Reports ([object[]]$reports.ToArray()) + if ($ok) { exit 0 } + exit 1 +} +catch { + Write-Error "error: $($_.Exception.Message)" + exit 2 +} diff --git a/src/templates/finops-hub-local/scripts/run-parity-checks.ps1 b/src/templates/finops-hub-local/scripts/run-parity-checks.ps1 new file mode 100644 index 000000000..01fcf3689 --- /dev/null +++ b/src/templates/finops-hub-local/scripts/run-parity-checks.ps1 @@ -0,0 +1,514 @@ +#!/usr/bin/env pwsh +<# +.SYNOPSIS + Runs FTK-Local parity checks against the local Kustainer databases. + +.DESCRIPTION + Runs the ftklocal parity-check suite against the two-DB topology + (Ingestion + Hub). Reads tests/parity-checks.kql, executes each numbered + CHECK block against the appropriate database, evaluates the EVAL rule from + the block's comment header, and prints a pass/fail report. + + Database routing (enforced per-check): + Checks 1–8, 11 → Ingestion (raw tables, final tables) + Checks 9, 10 → Hub (Costs_v1_2 and other Hub view functions) + + After the file-based checks, a synthetic NAME-PARITY check (CHECK 12) + verifies that no object in either database has a FtkLocal-prefixed name and + that all expected core tables / functions are present. + + Exit code 0 only if ALL checks pass. Non-zero means at least one parity + violation. + + Connection: + KUSTAINER_QUERY full query URL (default http://localhost:/v1/rest/query) + HOST_PORT port (default 8082; read from .env if present) + FTK_DB default database for Ingestion-targeted checks + (default Ingestion; Hub checks are always routed to Hub) + +.EXAMPLE + pwsh scripts/run-parity-checks.ps1 + +.EXAMPLE + pwsh scripts/run-parity-checks.ps1 --check 3 --check 5 +#> + +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +$script:RepoRoot = Split-Path -Parent (Split-Path -Parent $PSCommandPath) +$script:ParityFile = Join-Path $script:RepoRoot 'tests/parity-checks.kql' + +function Get-HostPort { + if ($env:HOST_PORT) { return $env:HOST_PORT } + $envFile = Join-Path $script:RepoRoot '.env' + if (Test-Path $envFile) { + foreach ($line in Get-Content -Path $envFile) { + $trimmed = $line.Trim() + if ($trimmed -like 'HOST_PORT=*') { return ($trimmed -split '=', 2)[1].Trim() } + } + } + return '8082' +} + +function Get-QueryUrl { + param([string] $Override) + if ($Override) { return $Override } + if ($env:KUSTAINER_QUERY) { + if ($env:KUSTAINER_QUERY -match '/v1/rest/(query|mgmt)$') { return $env:KUSTAINER_QUERY } + return "$($env:KUSTAINER_QUERY.TrimEnd('/'))/v1/rest/query" + } + return "http://localhost:$(Get-HostPort)/v1/rest/query" +} + +function Get-Database { + param([string] $Override) + if ($Override) { return $Override } + if ($env:FTK_DB) { return $env:FTK_DB } + return 'Ingestion' +} + +class ParityCheck { + [int] $Number + [string] $Name + [string] $Expect + [string] $Evaluator + [string] $Kql + [string] $EvalKind = 'manual' + [Nullable[double]] $EvalArg + [string] $EvalCol + # Which Kustainer database to run this check against. Defaults to the + # caller-supplied baseline (Ingestion). Hub checks (9, 10) are overridden + # to 'Hub' after parsing so they resolve Costs_v1_2 and other Hub functions. + [string] $Database = '' + + ParityCheck([int] $number, [string] $name, [string] $expect, [string] $evaluator, [string] $kql) { + $this.Number = $number + $this.Name = $name + $this.Expect = $expect + $this.Evaluator = $evaluator + $this.Kql = $kql + } +} + +function Set-Evaluator { + param([Parameter(Mandatory)] [ParityCheck] $CheckItem) + + $evaluatorText = $CheckItem.Evaluator.ToLowerInvariant() + if ($evaluatorText.Contains('result == 0') -or $evaluatorText.Contains('should be 0')) { + $CheckItem.EvalKind = 'zero' + } + elseif ($evaluatorText.Contains('abs(') -and ($evaluatorText.Contains('<=') -or $evaluatorText.Contains('<'))) { + $match = [regex]::Match($evaluatorText, '<=?\s*([0-9.]+)') + if ($match.Success) { + $CheckItem.EvalKind = 'within_pct' + $value = [double]$match.Groups[1].Value + if ($evaluatorText.Contains('pct') -or $value -lt 1.0) { + $CheckItem.EvalArg = $value * 100 + } + else { + $CheckItem.EvalArg = $value + } + } + } + elseif ($evaluatorText.Contains('at least one row') -or $evaluatorText.Contains('non-empty')) { + $CheckItem.EvalKind = 'nonzero' + } + else { + $CheckItem.EvalKind = 'manual' + } +} + +function Get-ParityChecks { + param([Parameter(Mandatory)] [string] $Text) + + $parts = [regex]::Split($Text, '(?m)^// CHECK ') + if ($parts.Count -lt 2) { + throw 'No CHECK blocks found in parity-checks.kql' + } + + $checks = @() + foreach ($block in @($parts | Select-Object -Skip 1)) { + $headerMatch = [regex]::Match($block, '(\d+):\s*(.+)') + if (-not $headerMatch.Success) { continue } + + $number = [int]$headerMatch.Groups[1].Value + $name = $headerMatch.Groups[2].Value.Trim() + + $expectMatch = [regex]::Match($block, '(?m)^// EXPECT:\s*(.+?)$') + $evalMatch = [regex]::Match($block, '(?m)^// EVAL:\s*(.+?)$') + $expect = if ($expectMatch.Success) { $expectMatch.Groups[1].Value.Trim() } else { '' } + $evaluator = if ($evalMatch.Success) { $evalMatch.Groups[1].Value.Trim() } else { '' } + + $kqlLines = @() + $inKql = $false + foreach ($line in ($block -split "`r?`n")) { + if (-not $inKql) { + if ($line.StartsWith('//') -or $line -match '^\d+:' -or -not $line.Trim()) { + continue + } + $inKql = $true + } + $kqlLines += $line + } + $kql = ($kqlLines -join "`n").Trim() + + $checkItem = [ParityCheck]::new($number, $name, $expect, $evaluator, $kql) + Set-Evaluator -CheckItem $checkItem + $checks += $checkItem + } + + return $checks +} + +function Invoke-KustoQuery { + param( + [Parameter(Mandatory)] [string] $Endpoint, + [Parameter(Mandatory)] [string] $Database, + [Parameter(Mandatory)] [string] $Kql + ) + + $url = if ($Endpoint -match '/v1/rest/query$') { $Endpoint } else { "$($Endpoint.TrimEnd('/'))/v1/rest/query" } + $body = @{ db = $Database; csl = $Kql } | ConvertTo-Json -Compress + $headers = @{ + 'Content-Type' = 'application/json' + 'x-ms-client-version' = 'Kusto.Python.Client:1.0.0' + } + + try { + return Invoke-RestMethod -Method Post -Uri $url -Headers $headers -Body $body -TimeoutSec 60 + } + catch { + $raw = $null + try { $raw = $_.ErrorDetails.Message } catch { } + if (-not $raw) { + try { + $stream = $_.Exception.Response.GetResponseStream() + $raw = (New-Object IO.StreamReader($stream)).ReadToEnd() + } catch { } + } + if (-not $raw) { $raw = $_.Exception.Message } + + $status = $null + try { $status = [int]$_.Exception.Response.StatusCode } catch { } + if ($status) { + throw "Kusto HTTP ${status}: $($raw.Substring(0, [Math]::Min(500, $raw.Length)))" + } + throw $raw + } +} + +function Get-PrimaryResult { + param([Parameter(Mandatory)] $Response) + + if (-not ($Response.PSObject.Properties.Name -contains 'Tables') -or -not $Response.Tables) { + $json = $Response | ConvertTo-Json -Depth 20 -Compress + throw "Unexpected response shape: $($json.Substring(0, [Math]::Min(200, $json.Length)))" + } + $table = $Response.Tables[0] + $cols = @($table.Columns | ForEach-Object { $_.ColumnName }) + $rows = @($table.Rows) + return [pscustomobject]@{ Columns = $cols; Rows = $rows } +} + +function Format-PythonValue { + param($Value) + + if ($null -eq $Value) { return 'None' } + if ($Value -is [string]) { return "'" + ($Value -replace "'", "\'") + "'" } + if ($Value -is [bool]) { return $(if ($Value) { 'True' } else { 'False' }) } + return [string]$Value +} + +function Format-PythonList { + param($Value) + + if ($null -eq $Value) { return 'None' } + if ($Value -is [System.Array]) { + $items = foreach ($item in $Value) { Format-PythonList $item } + return '[' + ($items -join ', ') + ']' + } + return (Format-PythonValue $Value) +} + +function Get-Cell { + param($Row, [int] $Index) + + if ($Row -is [System.Array]) { return $Row[$Index] } + return $Row[$Index] +} + +function Get-RowCount { + param($Rows) + if ($null -eq $Rows) { return 0 } + return @($Rows).Count +} + +function Get-RowSlice { + param($Rows, [int] $Count) + $allRows = @($Rows) + if ($allRows.Count -eq 0) { return @() } + $slice = @() + for ($i = 0; $i -lt [Math]::Min($Count, $allRows.Count); $i++) { + $slice += , $allRows[$i] + } + return , $slice +} + +function Test-ParityCheck { + param( + [Parameter(Mandatory)] [ParityCheck] $CheckItem, + [Parameter(Mandatory)] [string[]] $Columns, + [Parameter(Mandatory)] $Rows + ) + + $rowCount = Get-RowCount $Rows + + if ($CheckItem.EvalKind -eq 'zero') { + if ($rowCount -eq 0) { + return [pscustomobject]@{ Status = 'FAIL'; Explanation = 'no rows returned (expected one with value 0)' } + } + $value = Get-Cell -Row $Rows[0] -Index 0 + if ($null -eq $value -or $value -eq 0 -or $value -eq 0.0) { + return [pscustomobject]@{ Status = 'PASS'; Explanation = "result = $(Format-PythonValue $value)" } + } + return [pscustomobject]@{ Status = 'FAIL'; Explanation = "expected 0, got $value" } + } + + if ($CheckItem.EvalKind -eq 'within_pct') { + if (($Columns -contains 'delta_pct') -and $rowCount -gt 0) { + $delta = [Math]::Abs([double](Get-Cell -Row $Rows[0] -Index ([array]::IndexOf($Columns, 'delta_pct')))) + $tolerance = if ($null -ne $CheckItem.EvalArg) { [double]$CheckItem.EvalArg } else { 5.0 } + if ($delta -le $tolerance) { + return [pscustomobject]@{ Status = 'PASS'; Explanation = ('delta = {0:F4}% within {1}%' -f $delta, $tolerance) } + } + return [pscustomobject]@{ Status = 'FAIL'; Explanation = ('delta = {0:F4}% exceeds {1}%' -f $delta, $tolerance) } + } + if (($Columns -contains 'delta_abs') -and $rowCount -gt 0) { + $delta = [Math]::Abs([double](Get-Cell -Row $Rows[0] -Index ([array]::IndexOf($Columns, 'delta_abs')))) + if ($delta -lt 1.0) { + return [pscustomobject]@{ Status = 'PASS'; Explanation = ('delta = ${0:F6}' -f $delta) } + } + return [pscustomobject]@{ Status = 'FAIL'; Explanation = ('delta = ${0:F2} (too large for rounding)' -f $delta) } + } + $firstRow = if ($rowCount -gt 0) { Format-PythonList $Rows[0] } else { 'empty' } + return [pscustomobject]@{ Status = 'MANUAL'; Explanation = "cols=$(Format-PythonList $Columns), first_row=$firstRow" } + } + + if ($CheckItem.EvalKind -eq 'nonzero') { + if ($rowCount -eq 0) { + return [pscustomobject]@{ Status = 'FAIL'; Explanation = 'no rows returned (expected at least one)' } + } + for ($colIndex = 0; $colIndex -lt $Columns.Count; $colIndex++) { + try { + if ([double](Get-Cell -Row $Rows[0] -Index $colIndex) -gt 0) { + return [pscustomobject]@{ Status = 'PASS'; Explanation = "top row $($Columns[$colIndex]) = $(Get-Cell -Row $Rows[0] -Index $colIndex)" } + } + } + catch { + continue + } + } + return [pscustomobject]@{ Status = 'FAIL'; Explanation = "no positive numeric in top row: $(Format-PythonList $Rows[0])" } + } + + $preview = if ($rowCount -gt 0) { Get-RowSlice -Rows $Rows -Count 5 } else { @() } + return [pscustomobject]@{ Status = 'MANUAL'; Explanation = "cols=$(Format-PythonList $Columns), rows[:5]=$(Format-PythonList $preview)" } +} + +function Read-Options { + param([string[]] $Tokens) + + $opts = @{ checks = @() } + $i = 0 + while ($i -lt $Tokens.Count) { + switch ($Tokens[$i]) { + '--db' { $opts.db = $Tokens[++$i] } + '-Db' { $opts.db = $Tokens[++$i] } + '--endpoint' { $opts.endpoint = $Tokens[++$i] } + '-Endpoint' { $opts.endpoint = $Tokens[++$i] } + '--check' { $opts.checks += [int]$Tokens[++$i] } + '-Check' { $opts.checks += [int]$Tokens[++$i] } + default { throw "unrecognized arguments: $($Tokens[$i])" } + } + $i++ + } + return $opts +} + +function Test-NameParity { + param( + [Parameter(Mandatory)] [string] $Endpoint, + [Parameter(Mandatory)] [string] $IngestionDb, + [Parameter(Mandatory)] [string] $HubDb + ) + + $ingestionQueryUrl = if ($Endpoint -match '/v1/rest/query$') { $Endpoint } else { "$($Endpoint.TrimEnd('/'))/v1/rest/query" } + + function Invoke-NameQuery { + param([string] $Db, [string] $Csl) + $body = @{ db = $Db; csl = $Csl } | ConvertTo-Json -Compress + $headers = @{ 'Content-Type' = 'application/json'; 'x-ms-client-version' = 'Kusto.Python.Client:1.0.0' } + try { + $resp = Invoke-RestMethod -Method Post -Uri $ingestionQueryUrl -Headers $headers -Body $body -TimeoutSec 30 + $cols = @($resp.Tables[0].Columns | ForEach-Object { $_.ColumnName }) + $rows = @($resp.Tables[0].Rows) + return @{ Cols = $cols; Rows = $rows; Error = $null } + } + catch { return @{ Cols = @(); Rows = @(); Error = $_.Exception.Message } } + } + + $ingTables = Invoke-NameQuery -Db $IngestionDb -Csl '.show tables | project TableName' + if ($ingTables.Error) { + return [pscustomobject]@{ Status = 'FAIL'; Explanation = "cannot query Ingestion tables: $($ingTables.Error)" } + } + + $hubFunctions = Invoke-NameQuery -Db $HubDb -Csl '.show functions | project Name' + if ($hubFunctions.Error) { + return [pscustomobject]@{ Status = 'FAIL'; Explanation = "cannot query Hub functions: $($hubFunctions.Error)" } + } + + $tableNames = @($ingTables.Rows | ForEach-Object { [string]$_[0] }) + $funcNames = @($hubFunctions.Rows | ForEach-Object { [string]$_[0] }) + + # Assert no FtkLocal-prefixed or -suffixed objects exist + $ftkLocalObjects = @($tableNames | Where-Object { $_ -imatch 'FtkLocal' }) + + @($funcNames | Where-Object { $_ -imatch 'FtkLocal' }) + if ($ftkLocalObjects.Count -gt 0) { + return [pscustomobject]@{ Status = 'FAIL'; Explanation = "FtkLocal-named objects found: $($ftkLocalObjects -join ', ')" } + } + + # Assert expected core Ingestion tables are present + $requiredTables = @('Costs_raw','Prices_raw','Costs_final_v1_2','Prices_final_v1_2') + $missingTables = @($requiredTables | Where-Object { $tableNames -notcontains $_ }) + if ($missingTables.Count -gt 0) { + return [pscustomobject]@{ Status = 'FAIL'; Explanation = "Ingestion missing expected tables: $($missingTables -join ', ')" } + } + + # Assert expected core Hub functions are present + $requiredFuncs = @('Costs_v1_2','Prices_v1_2','Costs','Prices') + $missingFuncs = @($requiredFuncs | Where-Object { $funcNames -notcontains $_ }) + if ($missingFuncs.Count -gt 0) { + return [pscustomobject]@{ Status = 'FAIL'; Explanation = "Hub missing expected functions: $($missingFuncs -join ', ')" } + } + + return [pscustomobject]@{ Status = 'PASS'; Explanation = "Ingestion: $($tableNames.Count) table(s), Hub: $($funcNames.Count) function(s); no FtkLocal objects; all core names present" } +} + +function Main { + param([string[]] $Tokens) + + $opts = Read-Options -Tokens $Tokens + $endpoint = Get-QueryUrl $(if ($opts.ContainsKey('endpoint')) { $opts.endpoint } else { $null }) + $database = Get-Database $(if ($opts.ContainsKey('db')) { $opts.db } else { $null }) + + $text = Get-Content -Raw -Path $script:ParityFile + $checks = @(Get-ParityChecks -Text $text) + + # Route Hub-view checks to the Hub database regardless of --db default. + # CHECK 9: Costs_v1_2 aggregation (Hub function) + # CHECK 10: Costs_v1_2 vs Costs_final_v1_2 reconciliation (Hub, with cross-DB ref) + foreach ($checkItem in $checks) { + if ($checkItem.Number -in @(9, 10)) { + $checkItem.Database = 'Hub' + } + } + + if ($opts.checks.Count -gt 0) { + $selected = @{} + foreach ($checkNumber in $opts.checks) { $selected[$checkNumber] = $true } + $checks = @($checks | Where-Object { $selected.ContainsKey($_.Number) }) + } + + Write-Host "Loaded $($checks.Count) parity check(s) from $script:ParityFile`n" + + $passCount = 0 + $failCount = 0 + $manualCount = 0 + $failures = @() + + foreach ($checkItem in $checks) { + # Per-check database: Hub checks use Hub; all others use the $database default (Ingestion). + $checkDb = if ($checkItem.Database) { $checkItem.Database } else { $database } + Write-Host "━━━ CHECK $($checkItem.Number): $($checkItem.Name) [DB: $checkDb] ━━━" + Write-Host " EXPECT: $($checkItem.Expect)" + try { + $response = Invoke-KustoQuery -Endpoint $endpoint -Database $checkDb -Kql $checkItem.Kql + $result = Get-PrimaryResult -Response $response + $columns = $result.Columns + $rows = $result.Rows + $evaluation = Test-ParityCheck -CheckItem $checkItem -Columns $columns -Rows $rows + $status = $evaluation.Status + $explanation = $evaluation.Explanation + } + catch { + $status = 'FAIL' + $explanation = "query error: $($_.Exception.Message)" + $columns = @() + $rows = @() + } + + $marker = @{ PASS = '✓'; FAIL = '✗'; MANUAL = '?' }[$status] + Write-Host " $marker ${status}: $explanation" + if ((Get-RowCount $rows) -gt 0 -and $status -ne 'PASS') { + $preview = Get-RowSlice -Rows $rows -Count 3 + Write-Host " rows[:3] = $(Format-PythonList $preview)" + } + Write-Host '' + + if ($status -eq 'PASS') { + $passCount++ + } + elseif ($status -eq 'FAIL') { + $failCount++ + $failures += [pscustomobject]@{ Check = $checkItem; Explanation = $explanation; Columns = $columns; Rows = $rows } + } + else { + $manualCount++ + } + } + + # Synthetic CHECK 12: name-parity — Ingestion tables and Hub functions. + # Only skip if the user filtered to specific checks and didn't include 12. + $runNameParity = $opts.checks.Count -eq 0 -or ($opts.checks -contains 12) + if ($runNameParity) { + Write-Host "━━━ CHECK 12: Name parity — Ingestion tables + Hub functions [DB: Ingestion+Hub] ━━━" + Write-Host " EXPECT: No FtkLocal-named objects; required core tables and functions present." + try { + $evaluation = Test-NameParity -Endpoint $endpoint -IngestionDb $database -HubDb 'Hub' + $status = $evaluation.Status + $explanation = $evaluation.Explanation + } + catch { + $status = 'FAIL' + $explanation = "name-parity check error: $($_.Exception.Message)" + } + $marker = @{ PASS = '✓'; FAIL = '✗'; MANUAL = '?' }[$status] + Write-Host " $marker ${status}: $explanation" + Write-Host '' + if ($status -eq 'PASS') { $passCount++ } + elseif ($status -eq 'FAIL') { + $failCount++ + $dummyCheck = [ParityCheck]::new(12, 'Name parity', '', '', '') + $failures += [pscustomobject]@{ Check = $dummyCheck; Explanation = $explanation; Columns = @(); Rows = @() } + } + else { $manualCount++ } + } + + Write-Host ('━' * 60) + Write-Host "SUMMARY: $passCount pass, $failCount fail, $manualCount manual (of $($checks.Count + ($runNameParity ? 1 : 0)) total)" + if ($failures.Count -gt 0) { + Write-Host "`nFAILURES:" + foreach ($failure in $failures) { + Write-Host " CHECK $($failure.Check.Number) ($($failure.Check.Name)): $($failure.Explanation)" + } + Write-Host "`nWrite a gap report at notes/parity-gaps.md for each failure before re-running." + return 1 + } + if ($manualCount -gt 0) { + Write-Host "`nManual checks above require human inspection but did not block the run." + } + return 0 +} + +exit (Main -Tokens $args) diff --git a/src/templates/finops-hub-local/tests/parity-checks.kql b/src/templates/finops-hub-local/tests/parity-checks.kql new file mode 100644 index 000000000..1704388f4 --- /dev/null +++ b/src/templates/finops-hub-local/tests/parity-checks.kql @@ -0,0 +1,142 @@ +// ============================================================================ +// FTK-Local Parity Checks — two-DB topology (Ingestion + Hub) +// ============================================================================ +// Validates that data flowing Costs_raw → Costs_transform_v1_2 → Costs_final_v1_2 +// matches FTK ADX semantics. Each numbered check has an EXPECTED rule documented +// in the comment header. scripts/run-parity-checks.ps1 reads this file, executes +// each check against the appropriate database (Ingestion or Hub), evaluates the +// EVAL rule, and produces a pass/fail report. +// +// ROUTING (enforced by run-parity-checks.ps1 per-check DB assignment): +// Checks 1–8, 11 → Ingestion (raw tables, final tables) +// Checks 9, 10 → Hub (view functions cross-referencing Ingestion) +// +// CONVENTIONS +// - Each check is a block: // CHECK : , // EXPECT: , then the KQL. +// - The runner splits checks on the `// CHECK ` boundary. +// - Each block must end with a single tabular result; no multi-statement KQL. +// ============================================================================ + +// CHECK 1: row count parity +// EXPECT: final_count within 5% of raw_count (some rows may be filtered by transform). +// EVAL: abs(final_count - raw_count) / raw_count <= 0.05 +let raw_count = toscalar(Costs_raw | count); +let final_count = toscalar(Costs_final_v1_2 | count); +print + raw_count = raw_count, + final_count = final_count, + delta_pct = todouble(final_count - raw_count) / todouble(raw_count) * 100.0 + +// CHECK 2: BilledCost preservation +// EXPECT: sum(BilledCost) in final equals sum(BilledCost) in raw (no silent drops). +// EVAL: abs(raw_sum - final_sum) / abs(raw_sum) < 0.001 (allow rounding noise) +let raw_sum = toscalar(Costs_raw | summarize sum(BilledCost)); +let final_sum = toscalar(Costs_final_v1_2 | summarize sum(BilledCost)); +print + raw_sum_BilledCost = raw_sum, + final_sum_BilledCost = final_sum, + delta_abs = abs(raw_sum - final_sum) + +// CHECK 3: EffectiveCost zeroed for commitment-discount purchases +// EXPECT: sum(EffectiveCost) == 0 for ChargeCategory='Purchase' AND isnotempty(CommitmentDiscountId). +// This is an FTK bug-fix: Azure exports incorrectly set a non-zero EffectiveCost on +// the purchase row of a reservation; amortization spreads cost across usage rows. +// EVAL: result == 0 +Costs_final_v1_2 +| where ChargeCategory == 'Purchase' and isnotempty(CommitmentDiscountId) +| summarize sum_EffectiveCost = sum(EffectiveCost) + +// CHECK 4: data-quality flag dictionary populates +// EXPECT: x_SourceChanges contains documented codes (MissingContractedCost, +// InvalidChargeFrequency, etc.). Print top 10 codes by frequency. +// EVAL: print only — surface what flags fired. Failure mode is "no flags at all" +// which would mean the dictionary wiring is broken. +Costs_final_v1_2 +| where isnotempty(x_SourceChanges) +| extend first_code = tostring(split(x_SourceChanges, ',')[0]) +| summarize Count = count() by first_code +| order by Count desc +| take 10 + +// CHECK 5: AmortizationClass labels amortized usage +// EXPECT: ZERO rows where ChargeCategory='Usage' AND isnotempty(CommitmentDiscountId) +// AND x_AmortizationClass != 'Amortized Charge'. The transform should label every +// reservation-covered usage row as Amortized Charge. +// EVAL: result == 0 +Costs_final_v1_2 +| where ChargeCategory == 'Usage' + and isnotempty(CommitmentDiscountId) + and x_AmortizationClass != 'Amortized Charge' +| count + +// CHECK 6: TotalSavings populated for services with reservations +// EXPECT: sum(x_TotalSavings) > 0 for services that typically have RIs/SPs +// (Virtual Machines, Storage Accounts). Top 10 by sum_TotalSavings. +// EVAL: at least one row with sum > 0 for ServiceName in {Virtual Machines, +// Storage Accounts, Azure Reservations, Azure Savings Plan for Compute}. +Costs_final_v1_2 +| summarize sum_TotalSavings = sum(x_TotalSavings) by ServiceName +| order by sum_TotalSavings desc +| take 10 + +// CHECK 7: parse_resourceid populates x_ResourceGroupName when ARM path has one +// EXPECT: ZERO rows where isempty(x_ResourceGroupName) AND isnotempty(ResourceId) +// AND ResourceId contains '/resourcegroups/'. parse_resourceid is supposed to +// extract the RG from any ARM path that contains one. +// EVAL: result == 0 +Costs_final_v1_2 +| where isempty(x_ResourceGroupName) + and isnotempty(ResourceId) + and ResourceId contains '/resourcegroups/' +| count + +// CHECK 8: PublisherName normalized (no 'Microsoft Corporation') +// EXPECT: 'Microsoft Corporation' should be normalized to 'Microsoft' by the +// transform; the raw string should never appear in the final table. +// EVAL: result == 0 +Costs_final_v1_2 +| where PublisherName == 'Microsoft Corporation' +| count + +// CHECK 9: Hub view function returns plausible totals +// EXPECT: Costs_v1_2 | summarize sum(EffectiveCost) by ServiceCategory returns +// non-empty result. The Hub view is what consumers query; verifies it's wired. +// EVAL: at least one row with sum_EffectiveCost > 0. +// DB: Hub +Costs_v1_2 +| summarize sum_EffectiveCost = sum(EffectiveCost) by ServiceCategory +| order by sum_EffectiveCost desc + +// CHECK 10: Costs_v1_2 vs Costs_final_v1_2 BilledCost reconciliation +// EXPECT: sum(BilledCost) from Costs_v1_2 (Hub view) == sum from Costs_final_v1_2 +// (Ingestion table). The Hub view should not silently filter or duplicate. +// EVAL: abs(view_sum - final_sum) / abs(final_sum) < 0.001 +// DB: Hub — uses database('Ingestion').Costs_final_v1_2 cross-DB reference +let view_sum = toscalar(Costs_v1_2 | summarize sum(BilledCost)); +let final_sum_check10 = toscalar(database('Ingestion').Costs_final_v1_2 | summarize sum(BilledCost)); +print + view_sum_BilledCost = view_sum, + final_sum_BilledCost = final_sum_check10, + delta_abs = abs(view_sum - final_sum_check10) + +// CHECK 11: Prices row count parity per PriceType +// EXPECT: Consumption + ReservedInstance counts in Prices_final match Prices_raw +// exactly; SavingsPlan is allowed to drift by up to 2% per the documented FTK +// upstream lookup-fanout (see notes/parity-gaps.md §"T-2.3 root cause analysis"). +// Failure mode is Consumption or RI drift, which would indicate a real defect. +// EVAL: manual — print delta per PriceType; the runner flags it as MANUAL but +// the human-readable PASS criterion is: Consumption delta=0, RI delta=0, +// SP delta within +/-5% of SP raw count. +let raw_by = Prices_raw | summarize raw_n=count() by PriceType; +let final_by = Prices_final_v1_2 | summarize final_n=count() by PriceType=x_SkuPriceType; +raw_by +| join kind=fullouter final_by on PriceType +| project + PriceType = coalesce(PriceType, PriceType1), + raw_n = coalesce(raw_n, 0), + final_n = coalesce(final_n, 0), + delta = coalesce(final_n, 0) - coalesce(raw_n, 0), + delta_pct = iff(coalesce(raw_n, 0) > 0, + round(todouble(coalesce(final_n, 0) - coalesce(raw_n, 0)) / todouble(coalesce(raw_n, 1)) * 100.0, 4), + 0.0) +| order by raw_n desc From f6108e8381a8e79a9919b35be3251cbf478319ac Mon Sep 17 00:00:00 2001 From: msbrett Date: Wed, 17 Jun 2026 14:18:46 -0700 Subject: [PATCH 02/51] fix(finops-hub-local): make kql defaults to Hub; split KQL_DB/PARITY_DB Red-team caught that `make kql` defaulted to Ingestion, so Hub view-function queries (e.g. Costs_v1_2) failed with 400 unless DB=Hub was passed. The DB var was shared with the parity recipe, which needs Ingestion as its base DB. Split into KQL_DB (?= Hub, used by `make kql`) and PARITY_DB (?= Ingestion, passed to run-parity-checks.ps1). ingest-status now passes KQL_DB=Ingestion explicitly since it inspects Ingest_Manifest in Ingestion. Header/help/topology comments updated. ingest unaffected (ingest.ps1 defaults to Ingestion). Verified: `make kql QUERY='Costs_v1_2 | count'` -> 1350561 (Hub) with no override; KQL_DB=Ingestion override works; make parity still 10/0/2 + name parity PASS; make ingest-status reads Ingest_Manifest on Ingestion. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- src/templates/finops-hub-local/Makefile | 37 ++++++++++++++----------- 1 file changed, 21 insertions(+), 16 deletions(-) diff --git a/src/templates/finops-hub-local/Makefile b/src/templates/finops-hub-local/Makefile index a08a3bf11..2dd51ce4c 100644 --- a/src/templates/finops-hub-local/Makefile +++ b/src/templates/finops-hub-local/Makefile @@ -5,15 +5,15 @@ # make down Stop and remove the container (keeps the data volume). # make nuke Down + remove kustainer-data/ (destructive; full reset). # make logs Tail Kustainer logs. -# make kql QUERY='...' Run a one-shot KQL query (default DB: Ingestion; override DB=Hub). +# make kql QUERY='...' Run a one-shot KQL query (default DB: Hub; override e.g. KQL_DB=Ingestion). # # DB topology: # Ingestion raw tables (Costs_raw, Prices_raw), transform functions, final tables, # open-data lookups, and Ingest_Manifest. -# Used by: ingest, ingest-status, checks 1–8 + 11, kql (default). +# Used by: ingest, ingest-status, checks 1–8 + 11, parity (base DB). # Hub view functions (Costs_v1_2, Prices_v1_2, Costs, Prices, etc.) that # cross-reference database('Ingestion').*. -# Used by: ftk query/run/schema, parity checks 9–10. +# Used by: ftk query/run/schema, parity checks 9–10, kql (default). # # Notes: # - The Kustainer HTTP endpoint is on host port $(HOST_PORT) (default 8082). @@ -30,12 +30,16 @@ include .env export endif -HOST_PORT ?= 8082 -# Default DB for ad-hoc kql queries: Ingestion (raw + final tables). -# For Hub view functions use: make kql DB=Hub QUERY='Costs_v1_2 | count' -# For ftk catalog queries use: pwsh scripts/ftk.ps1 query "..." (targets Hub by default) -DB ?= Ingestion -COMPOSE ?= docker compose +HOST_PORT ?= 8082 +# KQL_DB: default database for ad-hoc `make kql` queries. +# Defaults to Hub so view functions (Costs_v1_2, etc.) work without an override. +# Override for raw-table inspection: make kql KQL_DB=Ingestion QUERY='Costs_raw | count' +KQL_DB ?= Hub +# PARITY_DB: base database passed to run-parity-checks.ps1 --db. +# Checks 1-8/11 run against PARITY_DB (Ingestion); the script internally forces +# checks 9/10 to Hub regardless of this value. +PARITY_DB ?= Ingestion +COMPOSE ?= docker compose .PHONY: up down nuke logs kql ftk ftk-list wait-healthy load-ftk-kql ingest ingest-status chunked-prices-backfill parity help @@ -45,7 +49,7 @@ help: @echo " down Stop the container (keeps data volume)." @echo " nuke Destructive full reset (drops kustainer-data/)." @echo " logs Tail Kustainer logs." - @echo " kql QUERY='...' One-shot KQL against $(DB)." + @echo " kql QUERY='...' One-shot KQL against Hub (override: KQL_DB=Ingestion)." @echo " load-ftk-kql Load FTK analytics KQL + open-data lookups." @echo " ingest Bulk-ingest ALL parquet under export/." @echo " ingest SCOPE= Ingest only one scope." @@ -85,14 +89,15 @@ nuke: logs: $(COMPOSE) logs -f kustainer -# Run a one-shot KQL query against $(DB). Override QUERY on the command line: +# Run a one-shot KQL query against $(KQL_DB) (default: Hub). Override on the command line: # make kql QUERY='.show tables' -# make kql QUERY='Costs_raw | count' +# make kql QUERY='Costs_v1_2 | count' (Hub — default) +# make kql KQL_DB=Ingestion QUERY='Costs_raw | count' kql: @if [ -z "$(QUERY)" ]; then \ echo "usage: make kql QUERY=''" >&2; exit 2; \ fi - @QUERY="$(QUERY)" DB="$(DB)" pwsh -NoProfile -Command '@{ db = $$env:DB; csl = $$env:QUERY } | ConvertTo-Json -Compress' | \ + @QUERY="$(QUERY)" DB="$(KQL_DB)" pwsh -NoProfile -Command '@{ db = $$env:DB; csl = $$env:QUERY } | ConvertTo-Json -Compress' | \ curl -sS --fail-with-body --max-time 60 \ -H 'Content-Type: application/json' \ -d @- \ @@ -132,16 +137,16 @@ ingest: @pwsh scripts/ingest.ps1 $(INGEST_FILTERS) # Quick summary of Ingest_Manifest: files + rows per (scope, export_type). -# Targets Ingestion DB (DB=Ingestion is the default). +# Targets Ingestion DB explicitly (Ingest_Manifest lives there, not in Hub). ingest-status: - @$(MAKE) --no-print-directory kql \ + @$(MAKE) --no-print-directory kql KQL_DB=Ingestion \ QUERY="Ingest_Manifest | summarize files=count(), rows=sum(rows_ingested) by scope, export_type | order by scope asc, export_type asc" # Run the T-1.4 parity check suite. Reads tests/parity-checks.kql, executes # each numbered check against Kustainer's query endpoint, evaluates the EVAL # rule, and prints a pass/fail report. Exit code 0 only if all checks pass. parity: - @pwsh scripts/run-parity-checks.ps1 --endpoint http://localhost:$(HOST_PORT) --db $(DB) + @pwsh scripts/run-parity-checks.ps1 --endpoint http://localhost:$(HOST_PORT) --db $(PARITY_DB) # Recovery path for the Prices_transform_v1_2() backfill when the single-pass # .set-or-append OOMs. Processes Prices_raw one source extent at a time From 0fa2a96b9eedac8f9cedaa765f37b8725e10914b Mon Sep 17 00:00:00 2001 From: msbrett Date: Wed, 17 Jun 2026 14:56:22 -0700 Subject: [PATCH 03/51] docs(finops-hub-local): export-config + manual staging contract (T-5000.4) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Document the data-in path: Cost Management export configuration via the toolkit's New-/Get-/Start-FinOpsCostExport cmdlets (Azure-only; ftklocal consumes the produced FOCUS parquet), and the local staging contract that ingest.ps1 consumes — export/{scope}/{type}/{period}/{run-uuid}/ with manifest.json, the ms--focus-cost / ms--pricesheet dataset tokens, latest-run selection, SHA-256 idempotency, and supersede semantics (correct/replace data by staging a NEW run-uuid; same run-uuid + changed file double-ingests). Maps the msexports/ (no hub) and ingestion/ (hub deployed) source modes; auto-download bridge deferred (T-5000.9). Contract cross-checked against ingest.ps1. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../notes/staging-contract.md | 361 ++++++++++++++++++ 1 file changed, 361 insertions(+) create mode 100644 src/templates/finops-hub-local/notes/staging-contract.md diff --git a/src/templates/finops-hub-local/notes/staging-contract.md b/src/templates/finops-hub-local/notes/staging-contract.md new file mode 100644 index 000000000..a71a56a5a --- /dev/null +++ b/src/templates/finops-hub-local/notes/staging-contract.md @@ -0,0 +1,361 @@ +# ftklocal staging contract + +This document defines the data-in story for ftklocal: how Cost Management exports are +configured (Part 1) and how the resulting parquet files are staged locally for ingestion +into the Kusto emulator (Part 2). + +--- + +## Part 1 — Export configuration (Azure-only) + +### Honest design constraint + +Cost Management exports are **Azure-only**. `New-FinOpsCostExport` writes parquet to an +Azure Storage account; there is no local export mode. Export *configuration* lives in Azure; +ftklocal *consumes* the parquet that configuration produces. No Azure export was created as +part of authoring this document. + +### Relevant PowerShell cmdlets + +All three cmdlets are part of the FinOps Toolkit PowerShell module under +`src/powershell/Public/`. + +--- + +#### `New-FinOpsCostExport` + +**File:** `src/powershell/Public/New-FinOpsCostExport.ps1` + +Creates a Cost Management export that writes FOCUS parquet to Azure Storage. + +| Parameter | Required | Notes | +|-----------|----------|-------| +| `-Name` | Yes | Unique name within the scope. | +| `-Scope` | Yes | Azure resource ID of the billing scope (subscription, billing account, etc.). | +| `-Dataset` | No | Default: `FocusCost`. Use `FocusCost` for cost data or `PriceSheet` for prices. | +| `-DatasetVersion` | No | Default: `1.2-preview` (FocusCost). Use `1.2-preview` to get the latest FOCUS schema. | +| `-Format` | No | Default: `Csv`. **Set to `Parquet`** for ftklocal (ingest.ps1 expects parquet). | +| `-CompressionMode` | No | Default: `None`. `Snappy` is also accepted but adds a decompression step not tested with ftklocal. | +| `-StorageAccountId` | Yes | Resource ID of the target Azure Storage account. | +| `-StorageContainer` | No | Default: `cost-management`. Use `msexports` to match hub conventions, or any container for ftklocal staging. | +| `-StoragePath` | No | Subfolder path prefix within the container. Defaults to the scope ID. | +| `-DoNotPartition` | No | Omit this flag (leave partitioning **enabled**). Partitioning is recommended for reliability. | +| `-DoNotOverwrite` | No | Default: overwrite enabled. For ftklocal, overwrite is acceptable; for FinOps hubs, use `-DoNotOverwrite` to simplify troubleshooting. | +| `-Monthly` | No | Monthly cadence (mutually exclusive with `-OneTime`). | +| `-OneTime` | No | Single run over an explicit date range. | +| `-StartDate` / `-EndDate` | No | Date range for the export. Defaults for scheduled and one-time differ; see cmdlet help. | +| `-Backfill` | No | Number of prior months to export immediately at creation time (scheduled exports only). | +| `-Execute` | No | Run the export once immediately after creation. | + +**Minimal FOCUS/Parquet example:** + +```powershell +New-FinOpsCostExport ` + -Name 'focus-parquet-monthly' ` + -Scope '/subscriptions/' ` + -Dataset FocusCost ` + -DatasetVersion '1.2-preview' ` + -Format Parquet ` + -StorageAccountId '/subscriptions//resourceGroups//providers/Microsoft.Storage/storageAccounts/' ` + -StorageContainer 'msexports' ` + -Monthly +``` + +--- + +#### `Get-FinOpsCostExport` + +**File:** `src/powershell/Public/Get-FinOpsCostExport.ps1` + +Lists or filters existing Cost Management exports for a scope. + +| Parameter | Notes | +|-----------|-------| +| `-Scope` | Azure resource ID of the scope. Defaults to current Az context subscription. | +| `-Name` | Filter by export name. Supports wildcards. | +| `-Dataset` | Filter by dataset type (e.g., `FocusCost`, `PriceSheet`). | +| `-DatasetVersion` | Filter by schema version. | +| `-StorageAccountId` | Filter by destination storage account. | +| `-StorageContainer` | Filter by container name. Supports wildcards. | +| `-RunHistory` | Switch. When present, the API response is expanded to include run history for each export. This is the enumeration primitive a future auto-download bridge (T-5000.9) would use to discover available runs and their storage paths. | + +**Enumerate run history (future auto-bridge use):** + +```powershell +Get-FinOpsCostExport -Scope '/subscriptions/' -Dataset FocusCost -RunHistory +``` + +--- + +#### `Start-FinOpsCostExport` + +**File:** `src/powershell/Public/Start-FinOpsCostExport.ps1` + +Triggers an on-demand run of an existing export using the Cost Management Run API. + +| Parameter | Notes | +|-----------|-------| +| `-Name` | Required. Name of the export to run. | +| `-Scope` | Azure resource ID. Defaults to current Az context subscription. | +| `-StartDate` / `-EndDate` | Optional date range override. | +| `-Backfill` | Number of prior months to run in sequence. | + +--- + +### Auto-download bridge (deferred) + +The automatic storage-to-local-folder sync is **deferred** and tracked in **T-5000.9**. In +v1, parquet files are staged manually (see Part 2). `Get-FinOpsCostExport -RunHistory` is +the intended enumeration primitive for that future bridge. + +--- + +## Part 2 — Local staging contract + +### Overview + +`ingest.ps1` walks a local directory tree and ingests parquet files into the Kusto emulator. +The contract below is the exact interface that script expects. It applies regardless of where +the parquet originated. + +### Folder layout + +``` +export/ +└── / + └── / + └── / + ├── / + │ ├── manifest.json + │ └── *.parquet + └── / ← older run; superseded if present + ├── manifest.json + └── *.parquet +``` + +- **`export/`** — root staging directory. Default: `export/` relative to the ftklocal + checkout root. Override via the `EXPORT_DIR` environment variable or `.env` file. + (`ingest.ps1`, `Get-ExportDir`, lines 41–51.) +- **``** — a short, user-defined label for the billing scope (e.g., `ea`, + `sub-prod`, `mca-`). This becomes the `scope` column in + `Ingest_Manifest`. +- **``** — dataset type token. Must be a key in `$script:DatasetTableMap` + (`ingest.ps1`, lines 59–62). See [Dataset type tokens](#dataset-type-tokens) below. +- **``** — export billing period in `YYYYMMDD-YYYYMMDD` format + (e.g., `20260501-20260531`). Must match this format; it becomes the `period` column. +- **``** — a UUID that identifies one export run. Multiple run directories + under the same `(scope, type, period)` are allowed; `ingest.ps1` selects only the + latest. Becomes `run_uuid` in `Ingest_Manifest`. + +### Dataset type tokens + +Defined in `$script:DatasetTableMap` at `ingest.ps1` lines 59–62: + +| Token | Raw table | Ingestion mapping | Cost Management dataset | +|-------|-----------|------------------|------------------------| +| `ms--focus-cost` | `Costs_raw` | `Costs_raw_mapping` | `FocusCost` | +| `ms--pricesheet` | `Prices_raw` | `Prices_raw_mapping` | `PriceSheet` | + +Any directory name under `/` that is not one of these tokens is silently skipped +with a warning (`ingest.ps1`, lines 248–252). + +### `manifest.json` per run + +Each `/` directory must contain a `manifest.json` file. The script reads +this file in `Read-RunManifest` (`ingest.ps1`, lines 209–215) to obtain sort and +row-count metadata. Fields consumed by `ingest.ps1`: + +| JSON path | Required | Purpose | +|-----------|----------|---------| +| `runInfo.submittedTime` | Recommended | Primary sort key for latest-run selection. ISO 8601 string. | +| `runInfo.createdDate` | Fallback | Used when `submittedTime` is absent. | +| `dataRowCount` | Recommended | Total expected row count for the run (shown in the plan summary). | +| `blobs[].blobName` | Recommended | Matched by filename suffix to resolve per-file row counts. | +| `blobs[].dataRowCount` | Recommended | Expected rows for each parquet file (used in verify summary). | + +**Minimum viable manifest for manual staging:** + +```json +{ + "runInfo": { + "submittedTime": "", + "runId": "" + }, + "dataRowCount": , + "blobs": [ + { + "blobName": "/.parquet", + "dataRowCount": + } + ] +} +``` + +A manifest.json produced directly by Cost Management (from an `msexports/` export) already +contains all these fields and can be used without modification. + +### Parquet files + +One or more `.parquet` files in the run directory. File names must match the `blobName` +suffix pattern in `manifest.json` for per-file row counts to resolve correctly, though +mismatches are tolerated (the per-file count defaults to 0; the run is still ingested). +Files are sorted lexicographically by name before ingest (`ingest.ps1`, +`Sort-ByNameOrdinal`, lines 102–107 and 273). + +### Latest-run selection + +For each `(scope, type, period)` triple, `ingest.ps1` collects all run directories that +contain a valid `manifest.json`, then picks the **one latest run** using a three-key sort +(ascending): `submittedTime` → `LastWriteTimeUtc` → directory name ordinal. The last entry +after sorting is the winner. (`ingest.ps1`, `Build-Plan`, lines 269–272.) + +All other runs in the same `(scope, type, period)` are **superseded**. Their previously +ingested extents are dropped from the raw table and their rows are removed from +`Ingest_Manifest` before the winning run is ingested. (`ingest.ps1`, +`Drop-SupersededExtents`, lines 571–613.) + +### Idempotency via `Ingest_Manifest` + +Ingest is idempotent at the file level. Before ingesting a parquet file, `ingest.ps1` +looks up the composite key `(scope, export_type, period, run_uuid, file_name)` in the +`Ingest_Manifest` table. If the key exists and the stored SHA-256 checksum matches the +file, the file is skipped. (`ingest.ps1`, lines 741–748.) + +`Ingest_Manifest` schema: + +| Column | Type | Notes | +|--------|------|-------| +| `scope` | string | Matches `` folder name. | +| `export_type` | string | Matches `` folder name. | +| `period` | string | Matches `` folder name. | +| `run_uuid` | string | Matches `` folder name. | +| `file_name` | string | Parquet file name (base name only). | +| `file_size` | long | File size in bytes at ingest time. | +| `rows_ingested` | long | Expected rows (from manifest, or 0 if not available). | +| `checksum_sha256` | string | SHA-256 hex of the parquet file. | +| `ingested_at` | datetime | UTC timestamp of ingest. | + +Defined at `ingest.ps1`, `Initialize-IngestManifestTable` / `Insert-ManifestRow` +(lines 299–351). + +### Overwrite semantics summary + +| Condition | Behavior | +|-----------|----------| +| Same `(scope, type, period, run-uuid)`, same file checksum | **Skip.** File is already in `Ingest_Manifest` with a matching SHA-256; no ingest, no new row. (`ingest.ps1` lines 744–748.) | +| Same `(scope, type, period, run-uuid)`, **different** file checksum | **Double-ingest — do not do this.** The old `Ingest_Manifest` row is not deleted and the prior extents are not dropped; a second ingest call is issued and a second `Ingest_Manifest` row is appended for the same key. The raw table will contain duplicate data. (`ingest.ps1` lines 744–776.) | +| Same `(scope, type, period)`, **new** `run-uuid` | **Safe replace.** `Drop-SupersededExtents` drops extents tagged with older run-uuids from the raw table and deletes their `Ingest_Manifest` rows before the new run is ingested. (`ingest.ps1` lines 571–613.) | +| Different `(scope, type, period)` | **Independent.** No interaction with other tuples. | + +> **How to correct or replace data:** stage the replacement parquet under a **new ``** +> directory for the same `(scope, type, period)`. Give it a `manifest.json` with a later +> `runInfo.submittedTime` than the run being replaced. `ingest.ps1` will then select it as +> the latest run, drop the old run's extents and manifest rows, and ingest the new files +> cleanly. Never mutate files in an already-ingested run directory — that path has no safe +> overwrite; it only appends a duplicate. + +--- + +## Source modes — mapping to the local contract + +### Mode A: `msexports/` container (no hub deployed) + +When FinOps hubs is **not** deployed, Cost Management exports write directly to the +`msexports` Azure Storage container. The raw container path is: + +``` +msexports/{path}/{date-range}/{export-name}/{export-time}/{guid}/{file} +``` + +The `manifest.json` produced here contains `exportConfig.type` (dataset), +`runInfo.submittedTime`, `runInfo.runId`, and `blobs[].blobName` / `blobs[].dataRowCount`. +All fields that `ingest.ps1` consumes are present. + +**To use with ftklocal (v1 — manual staging):** + +1. Create the staging directory: `export//ms--focus-cost///` +2. Copy the parquet file(s) from the `msexports` run folder into this directory. +3. Copy the `manifest.json` from the same run folder. No field changes are needed. +4. Run `make ingest` (or `pwsh scripts/ingest.ps1`). + +**Notes:** +- If the export was configured with `-Format Csv` (the default), the data is CSV, not + parquet, and cannot be ingested directly. **Always configure `-Format Parquet`.** +- The dataset type token (`ms--focus-cost`) corresponds to the `FocusCost` export type. + Cost Management uses a different name in `exportConfig.type` (e.g., `FocusCost`); the + `ms--focus-cost` token is ftklocal's local convention for the staging directory name. + +### Mode B: `ingestion/` container (hub already deployed) + +When FinOps hubs **is** deployed, the hub ETL pipeline converts raw `msexports/` data into +normalized FOCUS parquet and writes it to the `ingestion/` container: + +``` +ingestion/{dataset}/{date-folder-path}/{scope-id-path}/{ingestion-id}__{original-file-name}.parquet +``` + +The `ingestion/` parquet has already been normalized by the hub's FTK transforms (including +`x_SourceValues` and other derived columns). **This mode maps most cleanly to ftklocal** +because: + +- The parquet schema matches what the FTK KQL transform functions loaded via + `load-ftk-kql.ps1` expect. +- Row counts and manifest metadata are already reconciled by the hub pipeline. + +**To use with ftklocal (v1 — manual staging):** + +1. Identify the relevant `ingestion/` run folder(s) in Azure Storage. +2. Stage files: `export//ms--focus-cost///` +3. Construct a `manifest.json` (minimum viable form above) referencing the parquet files. +4. Run `make ingest`. + +### Auto-download (deferred — T-5000.9) + +In v1, **all storage-to-local staging is manual**. The automatic bridge that would use +`Get-FinOpsCostExport -RunHistory` to enumerate runs, download parquet from the storage +container, and arrange files into the `export/{scope}/{type}/{period}/{run}/` layout is +tracked in **T-5000.9** and is not implemented in v1. + +--- + +## Worked example directory tree + +The following illustrates a fully staged `export/` tree for two scopes and two dataset types: + +``` +export/ +├── ea/ +│ ├── ms--focus-cost/ +│ │ └── 20260501-20260531/ +│ │ └── a1b2c3d4-0000-0000-0000-000000000001/ +│ │ ├── manifest.json +│ │ └── focus_2026_05.parquet +│ └── ms--pricesheet/ +│ └── 20260501-20260531/ +│ └── a1b2c3d4-0000-0000-0000-000000000002/ +│ ├── manifest.json +│ └── pricesheet_2026_05.parquet +└── sub-prod/ + └── ms--focus-cost/ + └── 20260501-20260531/ + ├── a1b2c3d4-0000-0000-0000-000000000003/ ← superseded + │ ├── manifest.json + │ └── focus_v1.parquet + └── a1b2c3d4-0000-0000-0000-000000000004/ ← latest (ingested) + ├── manifest.json + └── focus_v2.parquet +``` + +In this tree, `ingest.ps1` ingests four runs: +- `ea / ms--focus-cost / 20260501-20260531` → one run, ingested into `Costs_raw`. +- `ea / ms--pricesheet / 20260501-20260531` → one run, ingested into `Prices_raw`. +- `sub-prod / ms--focus-cost / 20260501-20260531` → two runs; the `_004` run is latest, + `_003` is superseded and its extents are dropped. + +--- + +## gitignore note + +The `export/` directory is already listed in +`src/templates/finops-hub-local/.gitignore` and is never committed to the repository. +All staged parquet and manifest files remain local only. From a4f61062f217a226484ce4269f37f6c64d777bc7 Mon Sep 17 00:00:00 2001 From: msbrett Date: Wed, 17 Jun 2026 14:56:48 -0700 Subject: [PATCH 04/51] feat(finops-hub-local): local settings.json + config-container decision (T-5000.10) Add a sanitized settings.example.json (HubInstance schema: scopes + retention) and scripts/settings.ps1 (init/show/get) for local-hub fidelity; gitignore the real settings.json. Wire load-ftk-kql.ps1 to derive the raw soft-delete window ($$rawRetentionInDays$$) from settings.json retention.raw.days with a 3650 fallback and a guard (0/negative/non-numeric -> fallback) plus a warning when a positive value is surprisingly short (<90 days) for local analysis. Document the config-container decision: Mode A (ingestion container) needs no config; Mode B (msexports) would need the msexports->ingestion conversion + the in-repo Exports/schemas (deferred to the bridge, T-5000.9). The ADX/KQL layer consumes no config schemas; load-ftk-kql.ps1 reads settings.json only for retention. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- src/templates/finops-hub-local/.gitignore | 1 + .../notes/config-container.md | 105 ++++++++++++++++++ .../finops-hub-local/scripts/load-ftk-kql.ps1 | 15 ++- .../finops-hub-local/scripts/settings.ps1 | 99 +++++++++++++++++ .../finops-hub-local/settings.example.json | 15 +++ 5 files changed, 234 insertions(+), 1 deletion(-) create mode 100644 src/templates/finops-hub-local/notes/config-container.md create mode 100644 src/templates/finops-hub-local/scripts/settings.ps1 create mode 100644 src/templates/finops-hub-local/settings.example.json diff --git a/src/templates/finops-hub-local/.gitignore b/src/templates/finops-hub-local/.gitignore index 2ae74dd9c..129589d3d 100644 --- a/src/templates/finops-hub-local/.gitignore +++ b/src/templates/finops-hub-local/.gitignore @@ -3,6 +3,7 @@ kustainer-data/ *.duckdb .env +settings.json .DS_Store *.log __pycache__/ diff --git a/src/templates/finops-hub-local/notes/config-container.md b/src/templates/finops-hub-local/notes/config-container.md new file mode 100644 index 000000000..94db042e4 --- /dev/null +++ b/src/templates/finops-hub-local/notes/config-container.md @@ -0,0 +1,105 @@ +# Config container — decision record + +## Context + +A deployed FinOps Hub writes two artifacts to the Azure Storage `config` +container: + +| Artifact | Location in storage | Schema | +|---|---|---| +| `settings.json` | `config/settings.json` | `https://aka.ms/finops/hubs/settings-schema` | +| Export schemas | `config/schemas/*.json` | Internal column-typing contract | + +The `settings.json` file drives **scope registration** and **retention +configuration** for the Azure Data Factory (ADF) orchestration layer. The +`config/schemas/*.json` files drive the ADF **msexports → ingestion +conversion** step — column typing and normalization of the raw Cost Management +CSV exports into Parquet. + +ftklocal does not run ADF. The question this note answers is: which of these +artifacts does ftklocal need, and when? + +--- + +## Mode A — ingestion container (already-converted Parquet) ✅ v1 default + +**What ftklocal does today.** + +ftklocal ingests the already-converted `ingestion/`-container Parquet directly +into Kusto (`Costs_raw`, `Prices_raw`). At this point the msexports → ingestion +conversion has already been performed by ADF in the deployed hub. + +**Config container requirement:** none. + +- `settings.json` — not required for the msexports → ingestion conversion path + (that is an ADF concern). However, `load-ftk-kql.ps1` does optionally read a + local `settings.json` for one value — `retention.raw.days` — to set the Kusto + raw-table soft-delete policy via the `$$rawRetentionInDays$$` macro; it falls + back to 3650 days when the file is absent or when `raw.days` is 0. The + scopes array in `settings.json` is an ADF concern and is not read by the + ADX/KQL layer. See `settings.example.json` and `scripts/settings.ps1`. +- `config/schemas/*.json` — not required. The Parquet files in the ingestion + container are already normalized; the column-typing step that uses the schemas + has already run. + +--- + +## Mode B — msexports container (raw Cost Management exports) 🔮 future / deferred + +**Not implemented in v1.** + +If ftklocal were extended to ingest directly from the raw Cost Management CSV +exports (`msexports/` container), it would need to perform the same +msexports → ingestion conversion that ADF does. That conversion uses the schema +files to type, deduplicate, and normalize each export. + +**Config container requirement:** + +- `config/schemas/*.json` — required for the conversion step. **These must + always be sourced from the in-repo copies** at + `src/templates/finops-hub/modules/Microsoft.CostManagement/Exports/schemas/` + (e.g. `focuscost_1.2.json`, `focuscost_1.2-preview.json`, + `pricesheet_2023-05-01_ea.json`). Never copy schemas from a customer's + storage; the in-repo versions are the canonical, version-controlled + definitions. +- `settings.json` — would be needed to resolve the scope list and retention + parameters for the conversion orchestration. + +**Status:** the full local msexports converter is deferred to the bridge work +tracked in T-5000.9. Mode A remains the v1 happy path. + +--- + +## ADX / KQL layer — `config/schemas/*.json` not consumed; `settings.json` read by `load-ftk-kql.ps1` for one value + +The `config/schemas/*.json` files (the msexports → ingestion column-typing +schemas) are not consumed by any component of the ADX/KQL layer. `settings.json` +is partially consumed: `load-ftk-kql.ps1` optionally reads it for +`retention.raw.days` only; everything else reads neither. Evidence: + +- **`scripts/load-ftk-kql.ps1`** — optionally reads a local `settings.json` for + `retention.raw.days` (with a 3650-day fallback) and passes it to the + `$$rawRetentionInDays$$` macro in the Ingestion bundle. It does not read + scopes, version, or any other `settings.json` field. It does not read + `config/schemas/*.json` at all. +- **`scripts/ftk.ps1`** — reads neither `settings.json` nor + `config/schemas/*.json`. It does use the word "schema" extensively, but only + for querying live Kusto table schemas via KQL (`.show table ... cslschema`, + `getschema`) — not for any on-disk config file. +- **Hub / Ingestion KQL queries** — the `.kql` files assembled by + `src/templates/finops-hub/.build.config` `combineKql` bundles contain no + reference to `settings.json` or `config/schemas/*.json`. +- The schema files in + `src/templates/finops-hub/modules/Microsoft.CostManagement/Exports/schemas/` + are consumed by the ADF pipeline, not by KQL. + +For v1, `config/schemas/*.json` is therefore **not required** (the +msexports → ingestion conversion is an ADF concern). `settings.json` is +**optional** — absent means 3650-day raw retention; present lets the operator +set it explicitly. + +> **Local analysis note:** `retention.raw.days` in `settings.json` drives the +> Kusto raw-table soft-delete window (the `$$rawRetentionInDays$$` macro in +> `load-ftk-kql.ps1`). Keep this value high (3650 days is the default) for +> local analysis — unlike Azure Storage, local raw tables ARE the source data; +> early soft-deletion means re-running a full ingest to recover. diff --git a/src/templates/finops-hub-local/scripts/load-ftk-kql.ps1 b/src/templates/finops-hub-local/scripts/load-ftk-kql.ps1 index 165b80066..b84611a47 100644 --- a/src/templates/finops-hub-local/scripts/load-ftk-kql.ps1 +++ b/src/templates/finops-hub-local/scripts/load-ftk-kql.ps1 @@ -61,7 +61,20 @@ if ($RemainingArgs) { # Script lives at /src/templates/finops-hub-local/scripts/ $script:FtkLocalHome = Split-Path -Parent (Split-Path -Parent $PSCommandPath) $script:RepoRoot = Split-Path -Parent (Split-Path -Parent (Split-Path -Parent $script:FtkLocalHome)) -$script:RawRetentionDays = '3650' +$script:RawRetentionDays = '3650' # default; overridden by settings.json retention.raw.days when > 0 +$settingsFile = Join-Path $script:FtkLocalHome 'settings.json' +if (Test-Path $settingsFile) { + try { + $rawDays = (Get-Content -Raw -Path $settingsFile | ConvertFrom-Json).retention.raw.days + if ($null -ne $rawDays -and [int]$rawDays -gt 0) { + $script:RawRetentionDays = [string][int]$rawDays + if ([int]$rawDays -lt 90) { + Write-Warning "retention.raw.days=$rawDays (settings.json): local raw tables will soft-delete data older than $rawDays days. Re-ingest restores it. Consider setting raw.days >= 3650 for local analysis." + } + } + } + catch { <# silently fall back to the 3650 default above #> } +} $script:TopLevelCommandRegex = [regex]'^\.[a-z]' $script:IdempotentOkTypes = @('Kusto.Common.Svc.Exceptions.EntityNameAlreadyExistsException') diff --git a/src/templates/finops-hub-local/scripts/settings.ps1 b/src/templates/finops-hub-local/scripts/settings.ps1 new file mode 100644 index 000000000..d929b47d6 --- /dev/null +++ b/src/templates/finops-hub-local/scripts/settings.ps1 @@ -0,0 +1,99 @@ +#!/usr/bin/env pwsh +<# +.SYNOPSIS + Manage the local ftklocal settings.json (hub fidelity layer). + +.DESCRIPTION + settings.json mirrors the shape written by the deployed FinOps Hub to the + Azure Storage `config` container (schema https://aka.ms/finops/hubs/settings-schema). + It is data-like and gitignored so that scope / billing-account identifiers + are never committed. settings.example.json is the sanitized template. + + Subcommands (default: show): + show Print the resolved settings (from settings.json or built-in defaults). + init Copy settings.example.json → settings.json if the file is absent. + get Return a single dot-path value, e.g. 'retention.raw.days'. + +.EXAMPLE + pwsh scripts/settings.ps1 init + +.EXAMPLE + pwsh scripts/settings.ps1 show + +.EXAMPLE + pwsh scripts/settings.ps1 get retention.raw.days +#> +[CmdletBinding()] +param( + [Parameter(Position = 0)] + [ValidateSet('show', 'init', 'get')] + [string] $Subcommand = 'show', + + [Parameter(Position = 1)] + [string] $DotPath +) + +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +$script:FtkLocalHome = Split-Path -Parent (Split-Path -Parent $PSCommandPath) +$script:SettingsPath = Join-Path $script:FtkLocalHome 'settings.json' +$script:ExamplePath = Join-Path $script:FtkLocalHome 'settings.example.json' + +# Default settings (matches Copy-FileToAzureBlob.ps1 defaults, lines 66-86) +$script:Defaults = [ordered]@{ + '$schema' = 'https://aka.ms/finops/hubs/settings-schema' + type = 'HubInstance' + version = '0.0.0' + learnMore = 'https://aka.ms/finops/hubs' + scopes = @() + retention = [ordered]@{ + msexports = @{ days = 0 } + ingestion = @{ months = 13 } + raw = @{ days = 3650 } + final = @{ months = 13 } + } +} + +function Read-Settings { + if (Test-Path $script:SettingsPath) { + return Get-Content -Raw -Path $script:SettingsPath | ConvertFrom-Json + } + Write-Warning "settings.json not found — using built-in defaults. Run 'pwsh scripts/settings.ps1 init' to create it." + return $script:Defaults | ConvertTo-Json -Depth 10 | ConvertFrom-Json +} + +switch ($Subcommand) { + 'init' { + if (Test-Path $script:SettingsPath) { + Write-Host "settings.json already exists at: $script:SettingsPath" + Write-Host "Edit it to add your billing-account scopes and adjust retention values." + } + else { + if (-not (Test-Path $script:ExamplePath)) { + throw "settings.example.json not found at: $script:ExamplePath" + } + Copy-Item -Path $script:ExamplePath -Destination $script:SettingsPath + Write-Host "Created settings.json from settings.example.json." + Write-Host "Edit $script:SettingsPath to add real billing-account scopes if needed." + Write-Host "Note: settings.json is gitignored — it will not be committed." + } + } + + 'show' { + $s = Read-Settings + $s | ConvertTo-Json -Depth 10 + } + + 'get' { + if (-not $DotPath) { throw "Usage: settings.ps1 get e.g. 'retention.raw.days'" } + $s = Read-Settings + $parts = $DotPath -split '\.' + $value = $s + foreach ($part in $parts) { + if ($null -eq $value) { throw "Path '$DotPath' not found in settings (null at '$part')" } + $value = $value.$part + } + Write-Output $value + } +} diff --git a/src/templates/finops-hub-local/settings.example.json b/src/templates/finops-hub-local/settings.example.json new file mode 100644 index 000000000..e9cface46 --- /dev/null +++ b/src/templates/finops-hub-local/settings.example.json @@ -0,0 +1,15 @@ +{ + "$schema": "https://aka.ms/finops/hubs/settings-schema", + "type": "HubInstance", + "version": "0.0.0", + "learnMore": "https://aka.ms/finops/hubs", + "scopes": [ + { "scope": "/providers/Microsoft.Billing/billingAccounts/" } + ], + "retention": { + "msexports": { "days": 0 }, + "ingestion": { "months": 13 }, + "raw": { "days": 3650 }, + "final": { "months": 13 } + } +} From 1f90c55fd29511df3540db20c806c2085cb20715 Mon Sep 17 00:00:00 2001 From: msbrett Date: Wed, 17 Jun 2026 16:33:49 -0700 Subject: [PATCH 05/51] feat(finops-hub-local): add localized ADX dashboard artifact (T-5000.5) Localize the FinOps hub ADX dashboard for the local Kusto emulator: point the single data source at http://localhost:8082 / database Hub (135 tiles, 87 queries unchanged from upstream). Add notes/dashboard.md documenting the proven ftk.ps1 CLI path, the ADX web UI as best-effort with the HTTPS->HTTP mixed-content caveat, and an ngrok HTTPS-tunnel workaround. Power BI dropped (local Mac, no AAD endpoint). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- src/templates/finops-hub-local/dashboard.json | 5559 +++++++++++++++++ .../finops-hub-local/notes/dashboard.md | 133 + 2 files changed, 5692 insertions(+) create mode 100644 src/templates/finops-hub-local/dashboard.json create mode 100644 src/templates/finops-hub-local/notes/dashboard.md diff --git a/src/templates/finops-hub-local/dashboard.json b/src/templates/finops-hub-local/dashboard.json new file mode 100644 index 000000000..521dec787 --- /dev/null +++ b/src/templates/finops-hub-local/dashboard.json @@ -0,0 +1,5559 @@ +{ + "$schema": "https://dataexplorer.azure.com/static/d/schema/60/dashboard.json", + "id": "ftklocal-adx-dashboard", + "eTag": "", + "title": "FinOps hub (local)", + "schema_version": "60", + "tiles": [ + { + "id": "acf2ca3d-9d8f-477f-b259-f3937bf938d4", + "title": "Cost last month", + "visualType": "multistat", + "pageId": "f416685d-f559-4514-8e45-5e0e09aec286", + "layout": { + "x": 0, + "y": 3, + "width": 4, + "height": 8 + }, + "queryRef": { + "kind": "query", + "queryId": "152f2041-bbc1-41e4-b155-271b2e0cf6e9" + }, + "visualOptions": { + "multiStat__textSize": "large", + "multiStat__valueColumn": null, + "colorRulesDisabled": true, + "colorStyle": "light", + "multiStat__displayOrientation": "horizontal", + "multiStat__labelColumn": null, + "multiStat__slot": { + "width": 1, + "height": 2 + }, + "colorRules": [] + } + }, + { + "id": "595e39e5-02ff-4ee7-99f9-68694ffd0495", + "title": "Cost over time", + "visualType": "column", + "pageId": "f416685d-f559-4514-8e45-5e0e09aec286", + "layout": { + "x": 4, + "y": 3, + "width": 17, + "height": 4 + }, + "queryRef": { + "kind": "query", + "queryId": "bc24e050-f2b9-4b4a-a08d-69fc4a4bb95e" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": false, + "legendLocation": "left", + "xColumnTitle": "", + "xColumn": null, + "yColumns": null, + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "efd2b03d-c80d-4d78-b77a-efa3f932efdb", + "title": "Change over time", + "visualType": "column", + "pageId": "f416685d-f559-4514-8e45-5e0e09aec286", + "layout": { + "x": 4, + "y": 7, + "width": 17, + "height": 4 + }, + "queryRef": { + "kind": "query", + "queryId": "0d91ea4a-c81d-4a21-b708-b6af37be1eec" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": false, + "legendLocation": "left", + "xColumnTitle": "", + "xColumn": null, + "yColumns": null, + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "9c5e208a-f0ce-4e2e-9969-35130eef22c1", + "title": "Counts", + "visualType": "multistat", + "pageId": "f416685d-f559-4514-8e45-5e0e09aec286", + "layout": { + "x": 0, + "y": 11, + "width": 12, + "height": 4 + }, + "queryRef": { + "kind": "query", + "queryId": "f2cecbb0-13f8-4642-afa4-bbcc0558f777" + }, + "visualOptions": { + "multiStat__textSize": "auto", + "multiStat__valueColumn": null, + "colorRulesDisabled": false, + "colorStyle": "light", + "multiStat__displayOrientation": "horizontal", + "multiStat__labelColumn": null, + "multiStat__slot": { + "width": 4, + "height": 1 + }, + "colorRules": [] + } + }, + { + "id": "2a1ee947-f1e3-447e-9388-91e318068608", + "title": "Counts (last n days)", + "visualType": "multistat", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 15, + "width": 6, + "height": 8 + }, + "queryRef": { + "kind": "query", + "queryId": "5d63e04d-1a11-4307-96f1-ebbf68e09be0" + }, + "visualOptions": { + "multiStat__textSize": "auto", + "multiStat__valueColumn": null, + "colorRulesDisabled": true, + "colorStyle": "light", + "multiStat__displayOrientation": "vertical", + "multiStat__labelColumn": null, + "multiStat__slot": { + "width": 2, + "height": 2 + }, + "colorRules": [ + { + "id": "414e8053-b30c-40dd-9f3f-510910d54d83", + "ruleType": "colorByCondition", + "applyToColumn": null, + "hideText": false, + "applyTo": "cells", + "conditions": [ + { + "column": null, + "operator": ">", + "values": [ + "" + ] + } + ], + "chainingOperator": "and", + "colorStyle": "bold", + "color": null, + "tag": "", + "icon": null, + "ruleName": "", + "visualType": "multistat" + } + ] + } + }, + { + "id": "49534f2e-97ea-4b41-87c2-5e0f4bc26db8", + "title": "Effective cost", + "visualType": "column", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 6, + "y": 2, + "width": 16, + "height": 9 + }, + "queryRef": { + "kind": "query", + "queryId": "290e7eab-8159-4338-8531-85e2718cedb1" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "Effective cost", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [ + { + "id": "e7c54e29-4843-465c-ab63-75e816da8fe0", + "columns": [ + "Change" + ], + "label": "% change", + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + } + ], + "showMultiplePanels": true + }, + "hideLegend": true, + "legendLocation": "left", + "xColumnTitle": "", + "xColumn": null, + "yColumns": null, + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "8e17c9f7-c513-4b93-bab8-44d3b0e48e9c", + "title": "", + "visualType": "markdownCard", + "pageId": "969ddf4c-8f2a-4ec4-9588-bb2f39473c9f", + "layout": { + "x": 0, + "y": 0, + "width": 18, + "height": 6 + }, + "markdownText": "# FinOps hubs $$ftkver$$ ($$build-month$$)\nThe FinOps hub dashboard provides several summaries of your cost and usage in aligment with the [FinOps Framework](https://aka.ms/finops/fx). All data formatted based on the [FinOps Open Cost and Usage Specification (FOCUS)](https://aka.ms/finops/focus).\n\nMost of this dashboard is designed to work with effective (amortized) cost rather than billed (actual) cost. Amortization breaks down reservation and savings plan purchases and allocates costs to the resources that received the benefit to facilitate chargeback reporting. Effective costs will not match your invoice. For invoice reconciliation, refer to the **Invoicing** page.\n\n[Learn more](https://aka.ms/finops/hubs)     [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Summary)\n", + "visualOptions": {} + }, + { + "id": "f5fc42ae-fb77-4df7-9407-5c9c3588dd84", + "title": "About the FinOps toolkit", + "visualType": "markdownCard", + "pageId": "969ddf4c-8f2a-4ec4-9588-bb2f39473c9f", + "layout": { + "x": 18, + "y": 0, + "width": 4, + "height": 6 + }, + "markdownText": "FinOps hubs are part of the [FinOps toolkit](https://aka.ms/finops/toolkit), an open-source collection of FinOps solutions that help you manage and optimize your cost, usage, and carbon.\n\nTo contribute to the FinOps toolkit, [join us on GitHub](https://aka.ms/ftk).\n", + "visualOptions": {} + }, + { + "id": "570e40a5-f67a-4c4b-84f6-c207ed021812", + "title": "3-month running total trend", + "visualType": "column", + "pageId": "5838c918-4541-44fe-90d2-77306ef1e241", + "layout": { + "x": 0, + "y": 14, + "width": 16, + "height": 7 + }, + "queryRef": { + "kind": "query", + "queryId": "d5ed469a-45ea-49a2-b305-b841050213cf" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": false, + "legendLocation": "bottom", + "xColumnTitle": "", + "xColumn": null, + "yColumns": null, + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "1ac02a9f-3c12-4828-a4dd-60a0d411f60c", + "title": "Running total - This month and last", + "visualType": "area", + "pageId": "5c57f940-6f0f-4e39-930b-b6e89bb758ad", + "layout": { + "x": 0, + "y": 3, + "width": 16, + "height": 8 + }, + "queryRef": { + "kind": "query", + "queryId": "10300876-866a-40d7-836d-b3a8783ecece" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": false, + "legendLocation": "bottom", + "xColumnTitle": "", + "xColumn": null, + "yColumns": null, + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "27ff6f92-2e6f-431f-bf3b-f36aad59b5f5", + "title": "3-month daily trend", + "visualType": "column", + "pageId": "5838c918-4541-44fe-90d2-77306ef1e241", + "layout": { + "x": 0, + "y": 21, + "width": 16, + "height": 5 + }, + "queryRef": { + "kind": "query", + "queryId": "9e41a624-d5f9-40c6-b47f-fef4b50ec3dd" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": false, + "legendLocation": "bottom", + "xColumnTitle": "", + "xColumn": null, + "yColumns": null, + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "97df5060-929c-4457-a7e9-47005c845f16", + "title": "3-month daily trend by subscription", + "visualType": "stackedcolumn", + "pageId": "5838c918-4541-44fe-90d2-77306ef1e241", + "layout": { + "x": 0, + "y": 26, + "width": 16, + "height": 7 + }, + "queryRef": { + "kind": "query", + "queryId": "f26e2204-270e-4219-8f68-5acef1c9393f" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": true, + "legendLocation": "right", + "xColumnTitle": "", + "xColumn": null, + "yColumns": null, + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "9", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "dad96fbb-750c-47a3-bd2e-e2bee74ebd22", + "title": "Cost + savings running total (this month and last)", + "visualType": "stackedarea", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 6, + "y": 11, + "width": 16, + "height": 7 + }, + "queryRef": { + "kind": "query", + "queryId": "dcc1f533-e5f9-4855-9e4c-21e21fcdf943" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "Cost + savings", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": false, + "legendLocation": "bottom", + "xColumnTitle": "", + "xColumn": null, + "yColumns": [ + "EffectiveCostRunningTotal", + "NegotiatedDiscountSavingsRunningTotal", + "CommitmentDiscountSavingsRunningTotal" + ], + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "66dcd731-a141-47b2-b36a-53a138cb75f2", + "title": "Summary", + "visualType": "multistat", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 2, + "width": 6, + "height": 13 + }, + "queryRef": { + "kind": "query", + "queryId": "e8b343dc-7430-4487-8d44-ef48ac454f2d" + }, + "visualOptions": { + "multiStat__textSize": "auto", + "multiStat__valueColumn": null, + "colorRulesDisabled": false, + "colorStyle": "light", + "multiStat__displayOrientation": "vertical", + "multiStat__labelColumn": null, + "multiStat__slot": { + "width": 2, + "height": 4 + }, + "colorRules": [ + { + "id": "a45d5307-a815-4baa-9743-d142253dc699", + "ruleType": "colorByCondition", + "applyToColumn": null, + "hideText": false, + "applyTo": "cells", + "conditions": [ + { + "operator": "==", + "column": "IsThisMonth", + "values": [ + "true" + ] + } + ], + "chainingOperator": "and", + "colorStyle": "light", + "color": "blue", + "tag": "This month", + "icon": null, + "ruleName": "This month", + "visualType": "multistat" + }, + { + "id": "a1307a22-3b48-48a9-9c1b-ef6b6a228796", + "ruleType": "colorByCondition", + "applyToColumn": null, + "hideText": false, + "applyTo": "cells", + "conditions": [ + { + "operator": "==", + "column": "IsThisMonth", + "values": [ + "false" + ] + } + ], + "chainingOperator": "and", + "colorStyle": "light", + "color": "green", + "tag": "Last month", + "icon": null, + "ruleName": "Last month", + "visualType": "multistat" + } + ] + } + }, + { + "id": "04cb63d9-0a92-488f-b863-815466d6725c", + "title": "Summary", + "visualType": "table", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 23, + "width": 22, + "height": 5 + }, + "queryRef": { + "kind": "query", + "queryId": "c077a6d4-719f-42fe-b39b-36f77dc68976" + }, + "visualOptions": { + "table__enableRenderLinks": true, + "colorRulesDisabled": false, + "colorStyle": "light", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "table__renderLinks": [], + "colorRules": [] + } + }, + { + "id": "4faba5ce-4190-45f9-b41e-5d9c6a9c8098", + "title": "Summary", + "visualType": "table", + "pageId": "5c57f940-6f0f-4e39-930b-b6e89bb758ad", + "layout": { + "x": 0, + "y": 11, + "width": 22, + "height": 5 + }, + "queryRef": { + "kind": "query", + "queryId": "3d947a7c-edca-46b5-a862-de77a725c85f" + }, + "visualOptions": { + "table__enableRenderLinks": true, + "colorRulesDisabled": false, + "colorStyle": "light", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "table__renderLinks": [], + "colorRules": [] + } + }, + { + "id": "890e0366-a9a5-4001-bc6e-f1b7f23fa8a7", + "title": "Effective cost by subscription", + "visualType": "table", + "pageId": "5c57f940-6f0f-4e39-930b-b6e89bb758ad", + "layout": { + "x": 0, + "y": 16, + "width": 22, + "height": 5 + }, + "queryRef": { + "kind": "query", + "queryId": "3886b5cd-34a8-42d7-9e16-33ea4d236953" + }, + "visualOptions": { + "table__enableRenderLinks": true, + "colorRulesDisabled": false, + "colorStyle": "light", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "table__renderLinks": [], + "colorRules": [] + } + }, + { + "id": "e47767ba-1884-4ca7-8458-24025b92464a", + "title": "", + "visualType": "markdownCard", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 58, + "width": 22, + "height": 4 + }, + "markdownText": "# Regions\r\nEffective cost breakdown by region\r\n\r\n\u2b06\ufe0f [Top](?tile=e3ac3bbc-a9ef-44dd-828f-226afbcfda33)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.Summary.Regions)", + "visualOptions": {} + }, + { + "id": "d3e265ff-2c5a-4a03-a8b8-e0b4e10cf362", + "title": "Monthly trend by region", + "visualType": "stackedcolumn", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 62, + "width": 22, + "height": 5 + }, + "queryRef": { + "kind": "query", + "queryId": "d5224d06-c8e7-4dd9-afec-595b39712f5a" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": true, + "legendLocation": "right", + "xColumnTitle": "", + "xColumn": null, + "yColumns": null, + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "9", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "c54de4ff-e793-49b7-b33d-db22fccd0290", + "title": "Monthly trend by region", + "visualType": "table", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 67, + "width": 22, + "height": 8 + }, + "queryRef": { + "kind": "query", + "queryId": "0dcf2c54-7f1d-45e9-a53b-d598a24493a4" + }, + "visualOptions": { + "table__enableRenderLinks": true, + "colorRulesDisabled": false, + "colorStyle": "light", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "table__renderLinks": [], + "colorRules": [] + } + }, + { + "id": "b2916ba0-cb54-433d-89a1-8905b8d8618a", + "title": "Monthly trend by service name", + "visualType": "stackedcolumn", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 45, + "width": 22, + "height": 5 + }, + "queryRef": { + "kind": "query", + "queryId": "30644718-defd-4c6c-9ffa-a9c2cd1f871f" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": true, + "legendLocation": "right", + "xColumnTitle": "", + "xColumn": null, + "yColumns": null, + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "9", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "6208961b-6249-4fd4-a2d6-16b5e58b9baa", + "title": "Monthly trend by service name", + "visualType": "table", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 50, + "width": 22, + "height": 8 + }, + "queryRef": { + "kind": "query", + "queryId": "6259f773-593c-4953-898c-15aa5ff6e53a" + }, + "visualOptions": { + "table__enableRenderLinks": true, + "colorRulesDisabled": false, + "colorStyle": "light", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "table__renderLinks": [], + "colorRules": [] + } + }, + { + "id": "4956ec68-b338-4926-bd72-ae6ca9e886d6", + "title": "", + "visualType": "markdownCard", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 75, + "width": 22, + "height": 4 + }, + "markdownText": "# Subscriptions\r\nEffective cost breakdown by subscription (subaccount)\r\n\r\n\u2b06\ufe0f [Top](?tile=e3ac3bbc-a9ef-44dd-828f-226afbcfda33)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.Summary.Subscriptions)", + "visualOptions": {} + }, + { + "id": "b28897e5-0606-4112-a13a-0cd814055ea7", + "title": "Monthly trend by subscription", + "visualType": "stackedcolumn", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 79, + "width": 22, + "height": 5 + }, + "queryRef": { + "kind": "query", + "queryId": "21a87abe-19e2-44ec-8298-ba872e66c162" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": true, + "legendLocation": "right", + "xColumnTitle": "", + "xColumn": null, + "yColumns": null, + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "9", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "7b24b810-3cd3-4b7c-92cc-6b40f071b24d", + "title": "Monthly trend by subscription", + "visualType": "table", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 84, + "width": 22, + "height": 8 + }, + "queryRef": { + "kind": "query", + "queryId": "90502e9a-2d0d-4ae4-8d9d-cc21f9b72d5d" + }, + "visualOptions": { + "table__enableRenderLinks": true, + "colorRulesDisabled": false, + "colorStyle": "light", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "table__renderLinks": [], + "colorRules": [] + } + }, + { + "id": "f56a2d38-7d69-4dfd-8cc8-6b7a292758b1", + "title": "Monthly trend by resource group", + "visualType": "table", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 101, + "width": 22, + "height": 8 + }, + "queryRef": { + "kind": "query", + "queryId": "c2c65ec0-e57d-4834-8a6b-b5975afeb9a0" + }, + "visualOptions": { + "table__enableRenderLinks": true, + "colorRulesDisabled": false, + "colorStyle": "light", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "table__renderLinks": [], + "colorRules": [] + } + }, + { + "id": "d2672640-0e48-47e5-a253-de927a77a637", + "title": "Monthly trend by resource group", + "visualType": "stackedcolumn", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 96, + "width": 22, + "height": 5 + }, + "queryRef": { + "kind": "query", + "queryId": "61b26784-6a73-4e80-85b2-9c5cfbd2dd06" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": true, + "legendLocation": "right", + "xColumnTitle": "", + "xColumn": null, + "yColumns": null, + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "9", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "6750ce15-ec4e-4c60-8079-b00a4b462d63", + "title": "", + "visualType": "markdownCard", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 92, + "width": 22, + "height": 4 + }, + "markdownText": "# Resource groups\r\nEffective cost breakdown by resource group\r\n\r\n\u2b06\ufe0f [Top](?tile=e3ac3bbc-a9ef-44dd-828f-226afbcfda33)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.Summary.ResourceGroups)", + "visualOptions": {} + }, + { + "id": "52055726-2788-4216-b813-9d91714c5953", + "title": "", + "visualType": "markdownCard", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 109, + "width": 22, + "height": 4 + }, + "markdownText": "# Resources\r\nEffective cost breakdown by resource\r\n\r\n\u2b06\ufe0f [Top](?tile=e3ac3bbc-a9ef-44dd-828f-226afbcfda33)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.Summary.Resources)", + "visualOptions": {} + }, + { + "id": "d84a014f-0b68-490b-a522-fba66fee1e0f", + "title": "Monthly trend by resource", + "visualType": "stackedcolumn", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 113, + "width": 22, + "height": 5 + }, + "queryRef": { + "kind": "query", + "queryId": "3924981c-23e4-464d-a872-045df1752750" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": true, + "legendLocation": "right", + "xColumnTitle": "", + "xColumn": null, + "yColumns": null, + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "9", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "586ebe0b-ff5c-4794-91a1-a8ec845cad3a", + "title": "Monthly trend by resource", + "visualType": "table", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 123, + "width": 22, + "height": 8 + }, + "queryRef": { + "kind": "query", + "queryId": "c7be613e-deb4-4779-ad75-4445e8d5e01f" + }, + "visualOptions": { + "table__enableRenderLinks": true, + "colorRulesDisabled": false, + "colorStyle": "light", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "table__renderLinks": [], + "colorRules": [] + } + }, + { + "id": "c03c1e57-5e66-4eb1-a20b-10c887eb2140", + "title": "Daily trend by resource", + "visualType": "stackedcolumn", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 118, + "width": 22, + "height": 5 + }, + "queryRef": { + "kind": "query", + "queryId": "49e24ee0-91de-4b1c-973f-036a3c060aca" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": true, + "legendLocation": "right", + "xColumnTitle": "", + "xColumn": null, + "yColumns": null, + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "9", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "d9960334-1cff-4686-9d4a-e6afc5574aa7", + "title": "", + "visualType": "markdownCard", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 28, + "width": 22, + "height": 4 + }, + "markdownText": "# Services\r\nEffective cost breakdown by service category and service name\r\n\r\n\u2b06\ufe0f [Top](?tile=e3ac3bbc-a9ef-44dd-828f-226afbcfda33)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.Summary.Services)", + "visualOptions": {} + }, + { + "id": "700f736d-56b1-4194-8282-891683f63134", + "title": "Monthly trend by service category", + "visualType": "stackedcolumn", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 32, + "width": 22, + "height": 5 + }, + "queryRef": { + "kind": "query", + "queryId": "4c7a7614-9b8c-415b-a4e1-d2d53c023d31" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": true, + "legendLocation": "right", + "xColumnTitle": "", + "xColumn": null, + "yColumns": null, + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "9", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "97872267-01c6-44f9-ad0d-34940d4b58fa", + "title": "Monthly trend by service category", + "visualType": "table", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 37, + "width": 22, + "height": 8 + }, + "queryRef": { + "kind": "query", + "queryId": "1d273b55-d2ea-427c-8a5f-01f6c240e98a" + }, + "visualOptions": { + "table__enableRenderLinks": true, + "colorRulesDisabled": false, + "colorStyle": "light", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "table__renderLinks": [], + "colorRules": [] + } + }, + { + "id": "a8edc317-353b-4648-9d32-e73e0d0a2894", + "title": "", + "visualType": "markdownCard", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 131, + "width": 22, + "height": 4 + }, + "markdownText": "# Resource inventory\r\nSummary of resources and their costs\r\n\r\n\u2b06\ufe0f [Top](?tile=e3ac3bbc-a9ef-44dd-828f-226afbcfda33)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.Summary.ResourceInventory)", + "visualOptions": {} + }, + { + "id": "e26eb29d-8021-406c-a84e-5568ddd234f4", + "title": "Monthly trend by resource type", + "visualType": "stackedcolumn", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 154, + "width": 22, + "height": 5 + }, + "queryRef": { + "kind": "query", + "queryId": "90e30901-a931-4a1d-b81e-0a1821032c3c" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": true, + "legendLocation": "right", + "xColumnTitle": "", + "xColumn": null, + "yColumns": null, + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "9", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "8d63f5d3-b1d8-4117-a358-bce7939a13ae", + "title": "Daily trend by resource type (last n days)", + "visualType": "stackedcolumn", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 141, + "width": 22, + "height": 5 + }, + "queryRef": { + "kind": "query", + "queryId": "ebe41e27-e9f9-478e-ab90-fd1f87906766" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": true, + "legendLocation": "right", + "xColumnTitle": "", + "xColumn": null, + "yColumns": null, + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "9", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "8caa438b-10a8-418c-ab3b-5c5aee53d3e6", + "title": "Monthly trend by resource", + "visualType": "table", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 159, + "width": 22, + "height": 8 + }, + "queryRef": { + "kind": "query", + "queryId": "7f7b08a9-ae15-46f8-8b0f-767280375add" + }, + "visualOptions": { + "table__enableRenderLinks": true, + "colorRulesDisabled": false, + "colorStyle": "light", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "table__renderLinks": [], + "colorRules": [] + } + }, + { + "id": "011c2b2c-a232-4459-acf4-8db4055f4f70", + "title": "Most cost (last n days)", + "visualType": "bar", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 17, + "y": 146, + "width": 5, + "height": 8 + }, + "queryRef": { + "kind": "query", + "queryId": "a35b4c05-8ecb-4c51-9aaf-980af3d7923e" + }, + "description": "Uses the Daily trend parameter for the lookback period", + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": true, + "legendLocation": "right", + "xColumnTitle": "", + "xColumn": null, + "yColumns": [ + "EffectiveCost", + "ResourceCount" + ], + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "9", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "4495e71e-1005-4224-91f4-8103f06981da", + "title": "Most used (last n days)", + "visualType": "bar", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 12, + "y": 146, + "width": 5, + "height": 8 + }, + "queryRef": { + "kind": "query", + "queryId": "9c5d3cf0-b6cb-45a2-acfd-b19df425bddf" + }, + "description": "Uses the Daily trend parameter for the lookback period", + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": true, + "legendLocation": "right", + "xColumnTitle": "", + "xColumn": null, + "yColumns": [ + "EffectiveCost", + "ResourceCount" + ], + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "9", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "38ffb57b-4646-4944-a92b-6966a3a6a807", + "title": "Resource type summary (last n days)", + "visualType": "table", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 146, + "width": 12, + "height": 8 + }, + "queryRef": { + "kind": "query", + "queryId": "2d7b6447-2769-40b8-958a-f252dab68b1e" + }, + "visualOptions": { + "table__enableRenderLinks": true, + "colorRulesDisabled": false, + "colorStyle": "light", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "table__renderLinks": [], + "colorRules": [] + } + }, + { + "id": "a83872cb-44ef-402a-8512-b9d5ba04744e", + "title": "Resource inventory summary (last n days)", + "visualType": "multistat", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 135, + "width": 22, + "height": 6 + }, + "queryRef": { + "kind": "query", + "queryId": "f062533b-8c94-412f-bf83-0eb5cd06063d" + }, + "hideTitle": true, + "visualOptions": { + "multiStat__textSize": "auto", + "multiStat__valueColumn": "Value", + "colorRulesDisabled": false, + "colorStyle": "light", + "multiStat__displayOrientation": "horizontal", + "multiStat__labelColumn": null, + "multiStat__slot": { + "width": 4, + "height": 2 + }, + "colorRules": [] + } + }, + { + "id": "5bd3258c-034c-4e23-b543-7cdf87bd153d", + "title": "", + "visualType": "markdownCard", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 0, + "y": 114, + "width": 22, + "height": 4 + }, + "markdownText": "# SKU prices\r\nSummary of the SKUs used and their applicable discounts\r\n\r\n\u2b06\ufe0f [Top](?tile=c0bd4bc4-a910-4a74-89a9-d4ae0f995563)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.RateOptimization.SkuPrices)", + "visualOptions": {} + }, + { + "id": "9d9838ca-63b2-4239-b2a7-346273bc2b86", + "title": "", + "visualType": "markdownCard", + "pageId": "ea47329e-0bc9-4c11-b110-534878dbb3ad", + "layout": { + "x": 0, + "y": 0, + "width": 22, + "height": 5 + }, + "markdownText": "# Understand uage and cost\nThe **Understand usage and cost** domain is focused on data acquisition, reporting, analysis, and alerting on top of your cost, usage, and carbon consumption. This domain focuses on observability and business intelligence. It involves gathering data (ingestion), organizing it for the organization (allocation), generating reports (reporting), and monitoring to proactively identify and address issues (anomalies).\n\n[Learn more](https://learn.microsoft.com/cloud-computing/finops/framework/understand/understand-cloud-usage-cost)     [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand)\n", + "visualOptions": {} + }, + { + "id": "679acf08-65b5-434c-a0fc-29e178c958c3", + "title": "", + "visualType": "markdownCard", + "pageId": "ea47329e-0bc9-4c11-b110-534878dbb3ad", + "layout": { + "x": 0, + "y": 5, + "width": 5, + "height": 7 + }, + "markdownText": "### Data ingestion\nData ingestion refers to the process of collecting, transforming, and organizing data from various sources into a single, easily accessible repository. With this capability, you identify data sources needed to support your FinOps practice and facilitate FinOps tasks.\n\n \n\n \n\n\ud83d\udcca [View report](#9e099251-9658-48da-b416-80422a2a47c7)\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/ingestion)", + "visualOptions": {} + }, + { + "id": "0d15a5a6-6e0e-4825-8cad-77f660b000da", + "title": "", + "visualType": "markdownCard", + "pageId": "ea47329e-0bc9-4c11-b110-534878dbb3ad", + "layout": { + "x": 5, + "y": 5, + "width": 6, + "height": 7 + }, + "markdownText": "### Allocation\nAllocation is the process of attributing, assigning, and redistributing shared costs and usage by using accounts, tags, and other metadata. It helps establish accountability among teams and projects within an organization. With this capability, you identify the metadata needed to map cost, usage, and carbon data to the teams responsible. This capability is critical to support organizational reporting and driving the right level of accountability.\n\n \n\n\ud83d\udcca Coming soon\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/allocation)\n", + "visualOptions": {} + }, + { + "id": "c249daec-c935-4669-b6fd-00d1b3fc8818", + "title": "", + "visualType": "markdownCard", + "pageId": "ea47329e-0bc9-4c11-b110-534878dbb3ad", + "layout": { + "x": 11, + "y": 5, + "width": 6, + "height": 7 + }, + "markdownText": "### Reporting + analytics\nReporting and analytics involve analyzing cloud data and creating reports to understand usage and spending patterns. It helps identify opportunities for improvement and supports informed decision-making about cloud resources. With this capability, you identify and build the reports and reporting infrastructure needed to facilitate other FinOps capabilities. This capability is critical to ensuring proper visibility and transparency and unblocks monitoring and optimization at scale across the organization.\n\n \n\n\ud83d\udcca [View report](#f8ee3008-df7e-442f-825e-2e3b46e4c185)\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/reporting)\n", + "visualOptions": {} + }, + { + "id": "652a4347-742c-42d2-aa43-39459b1e71c6", + "title": "", + "visualType": "markdownCard", + "pageId": "ea47329e-0bc9-4c11-b110-534878dbb3ad", + "layout": { + "x": 17, + "y": 5, + "width": 5, + "height": 7 + }, + "markdownText": "### Anomaly management\nAnomaly management refers to the practice of detecting and addressing abnormal or unexpected cost and usage patterns in a timely manner. With this capability, you identify, triage, and address unexpected changes in cost, usage, and carbon. This capability is especially important with usage-based billing and sustainability models, where small fluctuations in usage can result in large fiscal and environmental impact.\n\n\ud83d\udcca [View report](#5838c918-4541-44fe-90d2-77306ef1e241)\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/anomalies)\n", + "visualOptions": {} + }, + { + "id": "6ea8b37d-617b-47dd-add0-8e3c0f708540", + "title": "", + "visualType": "markdownCard", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 0, + "y": 103, + "width": 22, + "height": 4 + }, + "markdownText": "# Purchases\r\nCosts from commitment discount, Marketplace, and other purchases\r\n\r\n\u2b06\ufe0f [Top](?tile=c0bd4bc4-a910-4a74-89a9-d4ae0f995563)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.RateOptimization.Purchases)", + "visualOptions": {} + }, + { + "id": "e3ac3bbc-a9ef-44dd-828f-226afbcfda33", + "title": "On this page", + "visualType": "markdownCard", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 0, + "width": 22, + "height": 2 + }, + "markdownText": "  [Services](?tile=d9960334-1cff-4686-9d4a-e6afc5574aa7)\r\n\u2022 [Regions](?tile=e47767ba-1884-4ca7-8458-24025b92464a)\r\n\u2022 [Subscriptions](?tile=4956ec68-b338-4926-bd72-ae6ca9e886d6)\r\n\u2022 [Resource groups](?tile=6750ce15-ec4e-4c60-8079-b00a4b462d63)\r\n\u2022 [Resources](?tile=52055726-2788-4216-b813-9d91714c5953)\r\n\u2022 [Inventory](?tile=a8edc317-353b-4648-9d32-e73e0d0a2894)\r\n\u2022 [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.Summary.Nav)", + "visualOptions": {} + }, + { + "id": "00b9ea3a-3902-4234-806f-1a7bf87b3992", + "title": "Daily trend", + "visualType": "column", + "pageId": "5838c918-4541-44fe-90d2-77306ef1e241", + "layout": { + "x": 0, + "y": 4, + "width": 16, + "height": 10 + }, + "queryRef": { + "kind": "query", + "queryId": "1d9c166d-22b6-48fd-9a90-9f983083ecc7" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "Effective cost", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [ + { + "id": "e7c54e29-4843-465c-ab63-75e816da8fe0", + "columns": [ + "Change" + ], + "label": "% change", + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + } + ], + "showMultiplePanels": true + }, + "hideLegend": true, + "legendLocation": "left", + "xColumnTitle": "", + "xColumn": null, + "yColumns": null, + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "c0bd4bc4-a910-4a74-89a9-d4ae0f995563", + "title": "On this page", + "visualType": "markdownCard", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 0, + "y": 0, + "width": 22, + "height": 2 + }, + "markdownText": "  [Savings](?tile=fa177e0f-9a74-4475-b64d-a399224e96e0)\r\n\u2022 [Commitment discounts](?tile=cad5d666-02e8-4319-bbad-1bdf46ca2961)\r\n\u2022 [Purchases](?tile=6ea8b37d-617b-47dd-add0-8e3c0f708540)\r\n\u2022 [SKU prices](?tile=5bd3258c-034c-4e23-b543-7cdf87bd153d)\r\n\u2022 [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.RateOptimization.Nav)\r\n", + "visualOptions": {} + }, + { + "id": "72aa5a62-e129-4ffa-be62-82c833282d27", + "title": "SKUs with negotiated discounts (last n days)", + "visualType": "table", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 0, + "y": 118, + "width": 14, + "height": 6 + }, + "queryRef": { + "kind": "query", + "queryId": "af6424d4-8f73-4b81-bfdd-f0287ebcaaca" + }, + "visualOptions": { + "table__enableRenderLinks": true, + "colorRulesDisabled": false, + "colorStyle": "light", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "table__renderLinks": [], + "colorRules": [] + } + }, + { + "id": "3d7688f7-0a1c-4564-b851-f380a22817bf", + "title": "SKUs with commitment discounts (last n days)", + "visualType": "table", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 0, + "y": 124, + "width": 14, + "height": 6 + }, + "queryRef": { + "kind": "query", + "queryId": "c4f6542d-a9cb-4284-bd4e-b9f94ad02192" + }, + "visualOptions": { + "table__enableRenderLinks": true, + "colorRulesDisabled": false, + "colorStyle": "light", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "table__renderLinks": [], + "colorRules": [] + } + }, + { + "id": "b2c9dcef-b40d-4dad-a0a3-cb5fe2c77156", + "title": "SKUs with no discounts (last n days)", + "visualType": "table", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 0, + "y": 130, + "width": 14, + "height": 6 + }, + "queryRef": { + "kind": "query", + "queryId": "00ae3917-c783-45f6-a04e-9113e4c5d445" + }, + "visualOptions": { + "table__enableRenderLinks": true, + "colorRulesDisabled": false, + "colorStyle": "light", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "table__renderLinks": [], + "colorRules": [] + } + }, + { + "id": "a3d814b8-ff20-4919-ae2a-f1caec997edf", + "title": "Most used SKUs (last n days)", + "visualType": "bar", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 14, + "y": 118, + "width": 8, + "height": 18 + }, + "queryRef": { + "kind": "query", + "queryId": "d8a4634e-7ebe-47ce-83b5-ff7f50f6bee6" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": true, + "legendLocation": "bottom", + "xColumnTitle": "", + "xColumn": "SKU", + "yColumns": [ + "EffectiveCost" + ], + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "18517864-8a43-4f0c-80e3-863554f1b189", + "title": "", + "visualType": "markdownCard", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 167, + "width": 22, + "height": 4 + }, + "markdownText": "# Purchases\r\nCosts from commitment discount, Marketplace, and other purchases\r\n\r\n\u2b06\ufe0f [Top](?tile=c0bd4bc4-a910-4a74-89a9-d4ae0f995563)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.Summary.Purchases)", + "visualOptions": {} + }, + { + "id": "4383c893-8839-4678-88c6-9a6393595d0a", + "title": "Purchases", + "visualType": "table", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 0, + "y": 171, + "width": 15, + "height": 7 + }, + "queryRef": { + "kind": "query", + "queryId": "98c9b9b5-bebf-41f8-8319-5fa2523f9dd0" + }, + "visualOptions": { + "table__enableRenderLinks": true, + "colorRulesDisabled": false, + "colorStyle": "light", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "table__renderLinks": [], + "colorRules": [] + } + }, + { + "id": "84da605b-6a0d-4a2a-aac3-a4160f9de10d", + "title": "Purchases", + "visualType": "table", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 0, + "y": 107, + "width": 22, + "height": 7 + }, + "queryRef": { + "kind": "query", + "queryId": "35a3a2b3-4ab0-4449-96f7-c78db789089e" + }, + "visualOptions": { + "table__enableRenderLinks": true, + "colorRulesDisabled": false, + "colorStyle": "light", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "table__renderLinks": [], + "colorRules": [] + } + }, + { + "id": "ea97e18e-5253-4a9d-9eed-0f2fc45b3c5b", + "title": "Cost summary", + "description": "List cost is based on retail prices without any discounts, contracted cost includes negotiated discounts, and effective cost includes commitment and negotiated discounts.", + "visualType": "bar", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 6, + "y": 18, + "width": 8, + "height": 5 + }, + "queryRef": { + "kind": "query", + "queryId": "5ff29428-de83-4a2c-8f86-d8beebe68750" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": false, + "legendLocation": "bottom", + "xColumnTitle": "", + "xColumn": null, + "yColumns": null, + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "d9959281-dc89-44fc-a2b3-b4f5227e0179", + "title": "Savings summary", + "description": "Negotiated discounts are based on your contractual agreement and represented in the \"contracted cost\". Commitment discounts are based on reservations and savings plans and represented in the \"effective cost\".", + "visualType": "bar", + "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "layout": { + "x": 14, + "y": 18, + "width": 8, + "height": 5 + }, + "queryRef": { + "kind": "query", + "queryId": "f5f240a8-a818-4f27-bdfb-e96fcfe433bd" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": false, + "legendLocation": "bottom", + "xColumnTitle": "", + "xColumn": null, + "yColumns": null, + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "cad5d666-02e8-4319-bbad-1bdf46ca2961", + "title": "", + "visualType": "markdownCard", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 0, + "y": 63, + "width": 22, + "height": 4 + }, + "markdownText": "# Commitment discounts\r\nCommitment discount usage and costs\r\n\r\n\u2b06\ufe0f [Top](?tile=c0bd4bc4-a910-4a74-89a9-d4ae0f995563)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.RateOptimization.CommitmentDiscounts)", + "visualOptions": {} + }, + { + "id": "36c1dede-49c9-4ae3-a19a-a230f87e9197", + "title": "Commitment discount usage (last n days)", + "visualType": "table", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 0, + "y": 67, + "width": 13, + "height": 6 + }, + "queryRef": { + "kind": "query", + "queryId": "d7f31381-ba44-46a1-ad3e-dc6b8826706d" + }, + "visualOptions": { + "table__enableRenderLinks": true, + "colorRulesDisabled": false, + "colorStyle": "light", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "table__renderLinks": [], + "colorRules": [] + } + }, + { + "id": "9513d9d7-be4a-468c-aed0-f0076c9bf3c4", + "title": "Savings summary (last n months)", + "visualType": "multistat", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 0, + "y": 8, + "width": 15, + "height": 9 + }, + "queryRef": { + "kind": "query", + "queryId": "e17346d1-0227-4aa4-8765-f9b4bf43bed5" + }, + "visualOptions": { + "multiStat__textSize": "auto", + "multiStat__valueColumn": "Value", + "colorRulesDisabled": false, + "colorStyle": "light", + "multiStat__displayOrientation": "horizontal", + "multiStat__labelColumn": "Label", + "multiStat__slot": { + "width": 5, + "height": 3 + }, + "colorRules": [ + { + "id": "a45d5307-a815-4baa-9743-d142253dc699", + "ruleType": "colorByCondition", + "applyToColumn": null, + "hideText": false, + "applyTo": "cells", + "conditions": [ + { + "operator": "==", + "column": "Type", + "values": [ + "List" + ] + } + ], + "chainingOperator": "and", + "colorStyle": "light", + "color": "red", + "tag": "List cost", + "icon": null, + "ruleName": "List cost", + "visualType": "multistat" + }, + { + "id": "a1307a22-3b48-48a9-9c1b-ef6b6a228796", + "ruleType": "colorByCondition", + "applyToColumn": null, + "hideText": false, + "applyTo": "cells", + "conditions": [ + { + "operator": "==", + "column": "Type", + "values": [ + "Contracted" + ] + } + ], + "chainingOperator": "and", + "colorStyle": "light", + "color": "yellow", + "tag": "Contracted cost", + "icon": null, + "ruleName": "Contracted cost", + "visualType": "multistat" + }, + { + "id": "65c6cac8-64f6-4029-bca0-b09dbfef9516", + "ruleType": "colorByCondition", + "applyToColumn": null, + "hideText": false, + "applyTo": "cells", + "conditions": [ + { + "operator": "==", + "column": "Type", + "values": [ + "Effective" + ] + } + ], + "chainingOperator": "and", + "colorStyle": "light", + "color": "green", + "tag": "Effective cost", + "icon": null, + "ruleName": "Effective cost", + "visualType": "multistat" + }, + { + "id": "33f99591-c8a6-4aae-be3f-a613bbc85c28", + "ruleType": "colorByCondition", + "applyToColumn": null, + "hideText": false, + "applyTo": "cells", + "conditions": [ + { + "operator": "==", + "column": "Type", + "values": [ + "PartialSavings" + ] + } + ], + "chainingOperator": "and", + "colorStyle": "bold", + "color": "blue", + "tag": "Partial savings", + "icon": null, + "ruleName": "Partial savings", + "visualType": "multistat" + }, + { + "id": "55430568-c3ef-4ff7-84b9-f3f82fe1a05a", + "ruleType": "colorByCondition", + "applyToColumn": null, + "hideText": false, + "applyTo": "cells", + "conditions": [ + { + "operator": "==", + "column": "Type", + "values": [ + "TotalSavings" + ] + } + ], + "chainingOperator": "and", + "colorStyle": "bold", + "color": "green", + "tag": "Total savings", + "icon": null, + "ruleName": "Total savings", + "visualType": "multistat" + } + ] + } + }, + { + "id": "9aec9f5c-f7cc-48da-8f0d-303137623945", + "title": "Savings summary (last n days)", + "visualType": "multistat", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 0, + "y": 44, + "width": 15, + "height": 9 + }, + "queryRef": { + "kind": "query", + "queryId": "f1ef29df-7a1d-4dd0-8619-0c0164707b31" + }, + "visualOptions": { + "multiStat__textSize": "auto", + "multiStat__valueColumn": "Value", + "colorRulesDisabled": false, + "colorStyle": "light", + "multiStat__displayOrientation": "horizontal", + "multiStat__labelColumn": "Label", + "multiStat__slot": { + "width": 5, + "height": 3 + }, + "colorRules": [ + { + "id": "a45d5307-a815-4baa-9743-d142253dc699", + "ruleType": "colorByCondition", + "applyToColumn": null, + "hideText": false, + "applyTo": "cells", + "conditions": [ + { + "operator": "==", + "column": "Type", + "values": [ + "List" + ] + } + ], + "chainingOperator": "and", + "colorStyle": "light", + "color": "red", + "tag": "List cost", + "icon": null, + "ruleName": "List cost", + "visualType": "multistat" + }, + { + "id": "a1307a22-3b48-48a9-9c1b-ef6b6a228796", + "ruleType": "colorByCondition", + "applyToColumn": null, + "hideText": false, + "applyTo": "cells", + "conditions": [ + { + "operator": "==", + "column": "Type", + "values": [ + "Contracted" + ] + } + ], + "chainingOperator": "and", + "colorStyle": "light", + "color": "yellow", + "tag": "Contracted cost", + "icon": null, + "ruleName": "Contracted cost", + "visualType": "multistat" + }, + { + "id": "65c6cac8-64f6-4029-bca0-b09dbfef9516", + "ruleType": "colorByCondition", + "applyToColumn": null, + "hideText": false, + "applyTo": "cells", + "conditions": [ + { + "operator": "==", + "column": "Type", + "values": [ + "Effective" + ] + } + ], + "chainingOperator": "and", + "colorStyle": "light", + "color": "green", + "tag": "Effective cost", + "icon": null, + "ruleName": "Effective cost", + "visualType": "multistat" + }, + { + "id": "33f99591-c8a6-4aae-be3f-a613bbc85c28", + "ruleType": "colorByCondition", + "applyToColumn": null, + "hideText": false, + "applyTo": "cells", + "conditions": [ + { + "operator": "==", + "column": "Type", + "values": [ + "PartialSavings" + ] + } + ], + "chainingOperator": "and", + "colorStyle": "bold", + "color": "blue", + "tag": "Partial savings", + "icon": null, + "ruleName": "Partial savings", + "visualType": "multistat" + }, + { + "id": "55430568-c3ef-4ff7-84b9-f3f82fe1a05a", + "ruleType": "colorByCondition", + "applyToColumn": null, + "hideText": false, + "applyTo": "cells", + "conditions": [ + { + "operator": "==", + "column": "Type", + "values": [ + "TotalSavings" + ] + } + ], + "chainingOperator": "and", + "colorStyle": "bold", + "color": "green", + "tag": "Total savings", + "icon": null, + "ruleName": "Total savings", + "visualType": "multistat" + } + ] + } + }, + { + "id": "e8018da4-7269-4e91-9f32-4bcf390746af", + "title": "Commitment discount breakdown (last n months)", + "visualType": "multistat", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 15, + "y": 8, + "width": 7, + "height": 9 + }, + "queryRef": { + "kind": "query", + "queryId": "7d0c2c1f-338b-4534-b173-36b284779131" + }, + "visualOptions": { + "multiStat__textSize": "auto", + "multiStat__valueColumn": "Value", + "colorRulesDisabled": true, + "colorStyle": "light", + "multiStat__displayOrientation": "horizontal", + "multiStat__labelColumn": "Label", + "multiStat__slot": { + "width": 2, + "height": 2 + }, + "colorRules": [] + } + }, + { + "id": "8dd83ee9-6692-4b94-a8c3-68caa7adbb64", + "title": "Commitment discount breakdown (last n days)", + "visualType": "multistat", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 15, + "y": 44, + "width": 7, + "height": 9 + }, + "queryRef": { + "kind": "query", + "queryId": "84ecad69-79ac-45b9-a8af-60be28dcc748" + }, + "visualOptions": { + "multiStat__textSize": "auto", + "multiStat__valueColumn": "Value", + "colorRulesDisabled": true, + "colorStyle": "light", + "multiStat__displayOrientation": "horizontal", + "multiStat__labelColumn": "Label", + "multiStat__slot": { + "width": 2, + "height": 2 + }, + "colorRules": [] + } + }, + { + "id": "e69fe03b-606c-4b93-b9b5-6f1048b24deb", + "title": "Monthly trend by service category", + "visualType": "table", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 0, + "y": 34, + "width": 22, + "height": 8 + }, + "queryRef": { + "kind": "query", + "queryId": "85320475-730b-411f-98d5-fbc345bf5e68" + }, + "visualOptions": { + "table__enableRenderLinks": true, + "colorRulesDisabled": false, + "colorStyle": "light", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "table__renderLinks": [], + "colorRules": [] + } + }, + { + "id": "df06246e-5feb-4fd1-bcb0-c04448ac4c80", + "title": "Monthly savings trend", + "visualType": "stackedcolumn", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 0, + "y": 21, + "width": 15, + "height": 6 + }, + "queryRef": { + "kind": "query", + "queryId": "5bbb5369-ac95-45fc-853c-a6a2ce6a9e7b" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": false, + "legendLocation": "right", + "xColumnTitle": "", + "xColumn": null, + "yColumns": null, + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "9", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "fa177e0f-9a74-4475-b64d-a399224e96e0", + "title": "", + "visualType": "markdownCard", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 0, + "y": 2, + "width": 22, + "height": 4 + }, + "markdownText": "# Savings\r\nSummary of negotiated and commitment discount savings\r\n\r\n\u2b06\ufe0f [Top](?tile=c0bd4bc4-a910-4a74-89a9-d4ae0f995563)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.RateOptimization.Savings)", + "visualOptions": {} + }, + { + "id": "8f92a18b-9000-48f8-952a-5f21c535105c", + "title": "Total savings (last n months)", + "visualType": "pie", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 15, + "y": 17, + "width": 7, + "height": 10 + }, + "queryRef": { + "kind": "query", + "queryId": "93f0eb9f-fa4d-4c54-96c2-8c1d3387d13a" + }, + "visualOptions": { + "hideLegend": false, + "legendLocation": "bottom", + "xColumn": "Type", + "yColumns": null, + "seriesColumns": null, + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "labelDisabled": false, + "pie__label": [ + "name", + "percentage" + ], + "tooltipDisabled": false, + "pie__tooltip": [ + "name", + "percentage", + "value" + ], + "pie__orderBy": "size", + "pie__kind": "pie", + "pie__topNSlices": null, + "crossFilter": [], + "drillthrough": [] + } + }, + { + "id": "b28fdb79-8d7a-414a-9da4-75ab9bffe134", + "title": "Daily savings trend", + "visualType": "stackedcolumn", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 0, + "y": 57, + "width": 15, + "height": 6 + }, + "queryRef": { + "kind": "query", + "queryId": "5c442903-65b0-4b53-9e7c-3ea9c1af7be5" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": false, + "legendLocation": "right", + "xColumnTitle": "", + "xColumn": null, + "yColumns": null, + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "9", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "088944a4-6cc6-4320-a8eb-82d35ee5b5d6", + "title": "Total savings (last n days)", + "visualType": "pie", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 15, + "y": 53, + "width": 7, + "height": 10 + }, + "queryRef": { + "kind": "query", + "queryId": "9a9ee4b0-a37d-475f-bf1d-f51573243491" + }, + "visualOptions": { + "hideLegend": false, + "legendLocation": "bottom", + "xColumn": "Type", + "yColumns": null, + "seriesColumns": null, + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "labelDisabled": false, + "pie__label": [ + "name", + "percentage" + ], + "tooltipDisabled": false, + "pie__tooltip": [ + "name", + "percentage", + "value" + ], + "pie__orderBy": "size", + "pie__kind": "pie", + "pie__topNSlices": null, + "crossFilter": [], + "drillthrough": [] + } + }, + { + "id": "bd461109-e42e-4329-a4b9-16a83c1ad9b3", + "title": "", + "visualType": "markdownCard", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 0, + "y": 6, + "width": 22, + "height": 2 + }, + "markdownText": "## Monthly trend (last n months)", + "visualOptions": {} + }, + { + "id": "40a4e89e-e521-4fb1-bdca-95166b7c9267", + "title": "", + "visualType": "markdownCard", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 0, + "y": 42, + "width": 22, + "height": 2 + }, + "markdownText": "## Daily trend (last n days)", + "visualOptions": {} + }, + { + "id": "cee8e33c-bb08-4f48-9c8e-c785985d16e5", + "title": "", + "visualType": "markdownCard", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 0, + "y": 73, + "width": 22, + "height": 2 + }, + "markdownText": "## Subscription chargeback (last n months)", + "visualOptions": {} + }, + { + "id": "fad3f76a-1c81-4c4f-8787-99da226100d1", + "title": "", + "visualType": "markdownCard", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 0, + "y": 83, + "width": 22, + "height": 2 + }, + "markdownText": "## Resource group chargeback (last n months)", + "visualOptions": {} + }, + { + "id": "b2c319c7-0ef0-477b-a80f-9b6e022b7e5e", + "title": "Commitment discount chargeback by resource group", + "visualType": "table", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 0, + "y": 85, + "width": 22, + "height": 8 + }, + "queryRef": { + "kind": "query", + "queryId": "377e3693-0738-45d7-97d9-4b6e71ec5b36" + }, + "visualOptions": { + "table__enableRenderLinks": true, + "colorRulesDisabled": false, + "colorStyle": "light", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "table__renderLinks": [], + "colorRules": [] + } + }, + { + "id": "e22f49e5-766c-4b35-8b4c-b99bef142d42", + "title": "Commitment discount chargeback by subscription", + "visualType": "table", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 0, + "y": 75, + "width": 22, + "height": 8 + }, + "queryRef": { + "kind": "query", + "queryId": "13cad52d-91e7-4ab4-aad5-4aae21c1019a" + }, + "visualOptions": { + "table__enableRenderLinks": true, + "colorRulesDisabled": false, + "colorStyle": "light", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "table__renderLinks": [], + "colorRules": [] + } + }, + { + "id": "c7e80308-bbb1-4aa8-a3d7-ce9eb0145f92", + "title": "Commitment discount usage by resource (last n days)", + "visualType": "table", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 0, + "y": 95, + "width": 22, + "height": 8 + }, + "queryRef": { + "kind": "query", + "queryId": "dcf69b47-233e-4bc4-b914-3f6f09f4cb82" + }, + "visualOptions": { + "table__enableRenderLinks": true, + "colorRulesDisabled": false, + "colorStyle": "light", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "table__renderLinks": [], + "colorRules": [] + } + }, + { + "id": "6e4a3743-e023-4fbc-a34d-43c08f201c6e", + "title": "", + "visualType": "markdownCard", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 0, + "y": 93, + "width": 22, + "height": 2 + }, + "markdownText": "## Resources usage (last n days)", + "visualOptions": {} + }, + { + "id": "ab300a6a-3bbd-450b-942c-7f30113cba15", + "title": "On this page", + "visualType": "markdownCard", + "pageId": "9e099251-9658-48da-b416-80422a2a47c7", + "layout": { + "x": 0, + "y": 0, + "width": 22, + "height": 2 + }, + "markdownText": "  [FinOps hubs](?tile=1e089e2c-d94a-4bfa-95e7-5f8b1249c982)\r\n\u2022 [Ingested data](?tile=10d7d4d3-0201-4cdf-977b-4814152fda24)\r\n\u2022 [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.DataIngestion.Nav)", + "visualOptions": {} + }, + { + "id": "1e089e2c-d94a-4bfa-95e7-5f8b1249c982", + "title": "", + "visualType": "markdownCard", + "pageId": "9e099251-9658-48da-b416-80422a2a47c7", + "layout": { + "x": 0, + "y": 5, + "width": 22, + "height": 4 + }, + "markdownText": "# FinOps hubs\r\nSummary of the cost and usage of FinOps hubs infrastructure\r\n\r\n\u2b06\ufe0f [Top](?tile=ab300a6a-3bbd-450b-942c-7f30113cba15)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.DataIngestion.FinOpsHubs)", + "visualOptions": {} + }, + { + "id": "05764729-51e4-4048-8cf8-2ac56eb3e643", + "title": "Monthly trend by hub instance", + "visualType": "stackedcolumn", + "pageId": "9e099251-9658-48da-b416-80422a2a47c7", + "layout": { + "x": 0, + "y": 13, + "width": 22, + "height": 5 + }, + "queryRef": { + "kind": "query", + "queryId": "d7a9ff96-da17-4826-9fb5-b23f4c7b938d" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": true, + "legendLocation": "right", + "xColumnTitle": "", + "xColumn": null, + "yColumns": null, + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "9", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "37a8bef0-7dfb-42b9-aeab-55f8c9c178b8", + "title": "Daily trend by hub instance", + "visualType": "stackedcolumn", + "pageId": "9e099251-9658-48da-b416-80422a2a47c7", + "layout": { + "x": 0, + "y": 18, + "width": 22, + "height": 5 + }, + "queryRef": { + "kind": "query", + "queryId": "13813a00-e634-4428-9eac-ea255fd1eaca" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": true, + "legendLocation": "right", + "xColumnTitle": "", + "xColumn": null, + "yColumns": null, + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "9", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "c48e07c5-f0a5-43e2-8a13-ee43ed1b92f9", + "title": "Monthly trend by hub instance", + "visualType": "table", + "pageId": "9e099251-9658-48da-b416-80422a2a47c7", + "layout": { + "x": 0, + "y": 23, + "width": 22, + "height": 8 + }, + "queryRef": { + "kind": "query", + "queryId": "62bbceee-c089-45db-822c-0b4745358aa4" + }, + "visualOptions": { + "table__enableRenderLinks": true, + "colorRulesDisabled": false, + "colorStyle": "light", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "table__renderLinks": [], + "colorRules": [] + } + }, + { + "id": "10d7d4d3-0201-4cdf-977b-4814152fda24", + "title": "", + "visualType": "markdownCard", + "pageId": "9e099251-9658-48da-b416-80422a2a47c7", + "layout": { + "x": 0, + "y": 31, + "width": 22, + "height": 4 + }, + "markdownText": "# Ingested data\r\nSummary of the data that's been ingested into FinOps hubs\r\n\r\n\u2b06\ufe0f [Top](?tile=ab300a6a-3bbd-450b-942c-7f30113cba15)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.DataIngestion.IngestedData)", + "visualOptions": {} + }, + { + "id": "0b192258-9639-4bd1-bffb-f0fbdcd14d1f", + "title": "Ingested data by table", + "visualType": "stackedcolumn", + "pageId": "9e099251-9658-48da-b416-80422a2a47c7", + "layout": { + "x": 0, + "y": 38, + "width": 22, + "height": 5 + }, + "queryRef": { + "kind": "query", + "queryId": "b98c9ed7-47b3-41c1-a596-fd9d32725b33" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": false, + "legendLocation": "right", + "xColumnTitle": "", + "xColumn": null, + "yColumns": [ + "Rows" + ], + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "9", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "db78922e-5215-4b93-9762-212eb0e32bf3", + "title": "Ingested cost data by scope", + "visualType": "stackedcolumn", + "pageId": "9e099251-9658-48da-b416-80422a2a47c7", + "layout": { + "x": 0, + "y": 43, + "width": 22, + "height": 5 + }, + "queryRef": { + "kind": "query", + "queryId": "187e22fe-3d78-45a2-a2bf-090ece144fd1" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": false, + "legendLocation": "right", + "xColumnTitle": "", + "xColumn": null, + "yColumns": [ + "Rows" + ], + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "9", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "c636ea6c-8356-4a69-a4d6-61e6596a3c4e", + "title": "Ingested scopes", + "visualType": "card", + "pageId": "9e099251-9658-48da-b416-80422a2a47c7", + "layout": { + "x": 0, + "y": 35, + "width": 4, + "height": 3 + }, + "queryRef": { + "kind": "query", + "queryId": "d2c522dc-ef4b-4714-a473-09350e275557" + }, + "visualOptions": { + "multiStat__textSize": "auto", + "multiStat__valueColumn": null, + "colorRulesDisabled": false, + "colorStyle": "light", + "colorRules": [] + } + }, + { + "id": "379b25b8-7188-4657-b82f-dee14b344bec", + "title": "Ingested months", + "visualType": "card", + "pageId": "9e099251-9658-48da-b416-80422a2a47c7", + "layout": { + "x": 4, + "y": 35, + "width": 4, + "height": 3 + }, + "queryRef": { + "kind": "query", + "queryId": "6fe11b78-82ef-4e9a-8411-a65e5b0d8bba" + }, + "visualOptions": { + "multiStat__textSize": "auto", + "multiStat__valueColumn": null, + "colorRulesDisabled": false, + "colorStyle": "light", + "colorRules": [] + } + }, + { + "id": "71b29fd4-2999-45dc-9e4b-14152c48a4f4", + "title": "Ingested price data by scope", + "visualType": "stackedcolumn", + "pageId": "9e099251-9658-48da-b416-80422a2a47c7", + "layout": { + "x": 0, + "y": 48, + "width": 22, + "height": 5 + }, + "queryRef": { + "kind": "query", + "queryId": "b3621977-59be-4f98-a034-a94479612115" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": false, + "legendLocation": "right", + "xColumnTitle": "", + "xColumn": null, + "yColumns": [ + "Rows" + ], + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "9", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "29f1c705-131d-44b9-9bd6-368c93aedcdf", + "title": "Ingested recommendation data by scope", + "visualType": "stackedcolumn", + "pageId": "9e099251-9658-48da-b416-80422a2a47c7", + "layout": { + "x": 0, + "y": 53, + "width": 22, + "height": 5 + }, + "queryRef": { + "kind": "query", + "queryId": "4a2a77fe-e819-4059-b027-dab0a1770a0a" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": false, + "legendLocation": "right", + "xColumnTitle": "", + "xColumn": null, + "yColumns": [ + "Rows" + ], + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "9", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "47471816-9569-4ff3-816b-98fbc47b5c17", + "title": "Ingested transaction data by scope", + "visualType": "stackedcolumn", + "pageId": "9e099251-9658-48da-b416-80422a2a47c7", + "layout": { + "x": 0, + "y": 58, + "width": 22, + "height": 5 + }, + "queryRef": { + "kind": "query", + "queryId": "8f1ad1ef-6f5e-4f20-a1c4-3941e871d0cd" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": false, + "legendLocation": "right", + "xColumnTitle": "", + "xColumn": null, + "yColumns": [ + "Rows" + ], + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "9", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "fcedcd3b-cd11-42ab-8363-de9619fc2018", + "title": "Ingested commitment discount usage data by scope", + "visualType": "stackedcolumn", + "pageId": "9e099251-9658-48da-b416-80422a2a47c7", + "layout": { + "x": 0, + "y": 63, + "width": 22, + "height": 5 + }, + "queryRef": { + "kind": "query", + "queryId": "e9a9a135-6a74-4463-b69f-eac6930ff1f2" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": false, + "legendLocation": "right", + "xColumnTitle": "", + "xColumn": null, + "yColumns": [ + "Rows" + ], + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "9", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "e29123b0-320a-4d36-a5bd-14c1757af8b8", + "title": "About this domain", + "visualType": "markdownCard", + "pageId": "9e099251-9658-48da-b416-80422a2a47c7", + "layout": { + "x": 0, + "y": 2, + "width": 22, + "height": 3 + }, + "markdownText": "Data ingestion refers to the process of collecting, transforming, and organizing data from various sources into a single, easily accessible repository. With this capability, you identify data sources needed to support your FinOps practice and facilitate FinOps tasks. [Learn more](http://aka.ms/ftk/fx/ingestion)", + "visualOptions": {} + }, + { + "id": "fe3644dd-bb95-4fee-b212-5e75374b3100", + "title": "", + "visualType": "markdownCard", + "pageId": "8beab65c-f5ec-4661-bc67-37b10baffb16", + "layout": { + "x": 0, + "y": 0, + "width": 22, + "height": 5 + }, + "markdownText": "# Quantify business value\nThe **Quantify business value** domain focuses on analyzing cost, usage, and sustainability to align with organizational plans and measure the return on investment from cloud computing efforts. This domain is all about measuring and maximizing the business value each team and workload gets from the cloud to maximize future potential.\n\n[Learn more](https://learn.microsoft.com/cloud-computing/finops/framework/quantify/quantify-business-value)     [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Quantify)\n", + "visualOptions": {} + }, + { + "id": "353712a3-e590-4d4e-a041-54c4b3105c99", + "title": "", + "visualType": "markdownCard", + "pageId": "8beab65c-f5ec-4661-bc67-37b10baffb16", + "layout": { + "x": 0, + "y": 5, + "width": 4, + "height": 8 + }, + "markdownText": "### Planning and estimating\nPlanning and estimating involve predicting the cost and usage of new and existing workloads based on potential architectural changes and shifting business priorities. With this capability, you establish baseline expectations for new cloud workloads and prepare for changes to existing workloads. This capability is an important part of establishing and maintaining organizational budgets.\n\n \n\n\ud83d\udcca Coming soon\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/planning)", + "visualOptions": {} + }, + { + "id": "cbc699f4-0197-420a-a439-3c869bf48227", + "title": "", + "visualType": "markdownCard", + "pageId": "8beab65c-f5ec-4661-bc67-37b10baffb16", + "layout": { + "x": 4, + "y": 5, + "width": 4, + "height": 8 + }, + "markdownText": "### Forecasting\nForecasting involves analyzing historical trends and future plans to predict costs, understand the impact on current budgets, and influence future budgets. With this capability, you'll project future cost, usage, and carbon emissions based on historical trends. This capability is a major part of establishing accurate budgets for each team.\n\n \n\n\ud83d\udcca Coming soon\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/forecasting)", + "visualOptions": {} + }, + { + "id": "5206bf80-0c56-4d07-bedb-ab0dc15bffb5", + "title": "", + "visualType": "markdownCard", + "pageId": "8beab65c-f5ec-4661-bc67-37b10baffb16", + "layout": { + "x": 8, + "y": 5, + "width": 5, + "height": 8 + }, + "markdownText": "### Budgeting\nBudgeting is the process of monitoring and managing financial plans and limits over a specific period to control spending effectively. This capability allows you to use established estimates and forecasts to allocate funds to each team for fiscal year planning and set up alerts to ensure budget accountability. This capability plays a key role in staying within established fiscal constraints on an ongoing basis.\n\n \n\n\ud83d\udcca [View report](#5c57f940-6f0f-4e39-930b-b6e89bb758ad)\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/budgeting)", + "visualOptions": {} + }, + { + "id": "566ad60b-4c09-45e4-aa2f-e6817c181a61", + "title": "", + "visualType": "markdownCard", + "pageId": "8beab65c-f5ec-4661-bc67-37b10baffb16", + "layout": { + "x": 13, + "y": 5, + "width": 4, + "height": 8 + }, + "markdownText": "### Benchmarking\nBenchmarking is a systematic process of evaluating the performance and value of cloud services using efficiency metrics, either within an organization or against industry peers. This capability allows you to measure and track key performance indicators to identify and accelerate the adoption of successful initiatives, maximizing cloud return on investment (ROI) across teams and external organizations when applicable.\n\n \n\n\ud83d\udcca Coming soon\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/benchmarking)", + "visualOptions": {} + }, + { + "id": "6282be37-d78a-4555-91e4-9c71dee8fcb7", + "title": "", + "visualType": "markdownCard", + "pageId": "8beab65c-f5ec-4661-bc67-37b10baffb16", + "layout": { + "x": 17, + "y": 5, + "width": 5, + "height": 8 + }, + "markdownText": "### Unit economics\nUnit economics refers to the process of calculating the cost and carbon emissions of a single unit of a business that can show the business value of the cloud. With this capability, you break down cost, usage, and carbon emissions and compare that with the business value per unit to identify high- and low-value workloads. This capability helps you identify areas that can be scaled out to accelerate business value through specific technology decisions.\n\n \n\n\ud83d\udcca Coming soon\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/uniteconomics)", + "visualOptions": {} + }, + { + "id": "c19ca143-e39b-4407-9592-cbaf87a37529", + "title": "", + "visualType": "markdownCard", + "pageId": "d01a8154-8a60-4d22-96ea-54b45b1417fe", + "layout": { + "x": 0, + "y": 0, + "width": 22, + "height": 5 + }, + "markdownText": "# Optimize usage and cost\nThe **Optimize usage and cost** domain focused on designing and optimizing solutions for efficiency to ensure you get the most out of your cloud investments.\n\n[Learn more](https://learn.microsoft.com/cloud-computing/finops/framework/optimize/optimize-cloud-usage-cost)     [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Quantify)\n", + "visualOptions": {} + }, + { + "id": "4ce93188-d865-432f-8488-c5d5758d0f14", + "title": "", + "visualType": "markdownCard", + "pageId": "d01a8154-8a60-4d22-96ea-54b45b1417fe", + "layout": { + "x": 0, + "y": 5, + "width": 5, + "height": 8 + }, + "markdownText": "### Architecting for the cloud\nArchitecting for the cloud involves designing and implementing cloud infrastructure and applications to optimize cost, performance, scalability, and reliability. Simultaneously, you align with business objectives.\n\nWith this capability, you ensure solutions are designed for efficiency in early development and migration stages to ensure they follow proven practices to maximize value and reduce waste. This capability is an important step towards reducing post-deployment optimization efforts, which often require much time and effort.\n\n\ud83d\udcca Coming soon\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/architecting)", + "visualOptions": {} + }, + { + "id": "87e197cb-12de-4eca-86e6-ee1d469c8990", + "title": "", + "visualType": "markdownCard", + "pageId": "d01a8154-8a60-4d22-96ea-54b45b1417fe", + "layout": { + "x": 5, + "y": 5, + "width": 5, + "height": 8 + }, + "markdownText": "### Workload optimization\nWorkload optimization refers to the process of ensuring cloud services are utilized and tuned to maximize business value and minimize wasteful usage and spending. With this capability, you analyze cost, usage, and carbon emissions for cloud workloads to identify opportunities to maximize efficiency. This capability usually starts with recommendations and expands into more nuanced optimization efforts based on detailed resource utilization analysis. This capability can be time and effort intensive as each cloud service has its different optimization opportunities.\n\n\ud83d\udcca Coming soon\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/workloads)", + "visualOptions": {} + }, + { + "id": "a60dda7f-8f14-429d-b7ff-a2db94feb809", + "title": "", + "visualType": "markdownCard", + "pageId": "d01a8154-8a60-4d22-96ea-54b45b1417fe", + "layout": { + "x": 10, + "y": 5, + "width": 4, + "height": 8 + }, + "markdownText": "### Rate optimization\nRate optimization is the practice of obtaining reduced rates on cloud services, often by committing to a certain level of usage or spend over a specific period. With this capability, you analyze SKU usage patterns to inform rate negotiation efforts and commitment discount planning. This capability can lead to impressive discounts based on consistent usage patterns.\n\n\ud83d\udcca [View report](#306fef9a-c760-4559-a326-7c25d196b616)\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/rates)", + "visualOptions": {} + }, + { + "id": "aa339c66-412d-4619-9737-06d5e4c282fe", + "title": "", + "visualType": "markdownCard", + "pageId": "d01a8154-8a60-4d22-96ea-54b45b1417fe", + "layout": { + "x": 14, + "y": 5, + "width": 4, + "height": 8 + }, + "markdownText": "### Licensing and SaaS\nLicensing & SaaS involves managing and optimizing the financial aspects of software licensing agreements and Software as a Service (SaaS) investments within an organization. With this capability, you track and manage software licenses and prepaid SaaS products to ensure they're being fully utilized. This capability also covers bring your own license offers, like Azure Hybrid Benefit.\n\n\ud83d\udcca [View report](#a4ec1d55-5b6e-49af-bb9e-4f3d136cdf05)\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/licensing)", + "visualOptions": {} + }, + { + "id": "2ba70b12-ac64-41b9-b13f-d3c91dab6df4", + "title": "On this page", + "visualType": "markdownCard", + "pageId": "a4ec1d55-5b6e-49af-bb9e-4f3d136cdf05", + "layout": { + "x": 0, + "y": 0, + "width": 22, + "height": 2 + }, + "markdownText": "  [Hybrid Benefit](?tile=805cac8c-a189-45cb-8ba3-fbb1a06dfba0)\r\n\u2022 [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.Licensing.Nav)", + "visualOptions": {} + }, + { + "id": "805cac8c-a189-45cb-8ba3-fbb1a06dfba0", + "title": "", + "visualType": "markdownCard", + "pageId": "a4ec1d55-5b6e-49af-bb9e-4f3d136cdf05", + "layout": { + "x": 0, + "y": 5, + "width": 22, + "height": 4 + }, + "markdownText": "# Azure Hybrid Benefit\r\nSummary of the Hybrid Benefit coverage and utilization\r\n\r\n\u2b06\ufe0f [Top](?tile=2ba70b12-ac64-41b9-b13f-d3c91dab6df4)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.Licensing.HybridBenefit)", + "visualOptions": {} + }, + { + "id": "d2e4a321-9c2c-411b-8a20-19fd86dab3fe", + "title": "Hybrid Benefit summary (last n days)", + "visualType": "multistat", + "pageId": "a4ec1d55-5b6e-49af-bb9e-4f3d136cdf05", + "layout": { + "x": 0, + "y": 9, + "width": 6, + "height": 15 + }, + "queryRef": { + "kind": "query", + "queryId": "1a400b6c-9204-434d-88ff-2dc5260a95bf" + }, + "visualOptions": { + "multiStat__textSize": "auto", + "multiStat__valueColumn": "Value", + "colorRulesDisabled": true, + "colorStyle": "light", + "multiStat__displayOrientation": "vertical", + "multiStat__labelColumn": "Label", + "multiStat__slot": { + "width": 2, + "height": 3 + }, + "colorRules": [] + } + }, + { + "id": "80e1ecb4-69f6-451d-9e1c-f90d17b7f865", + "title": "Underutilized vCPU capacity (last n days)", + "visualType": "table", + "pageId": "a4ec1d55-5b6e-49af-bb9e-4f3d136cdf05", + "layout": { + "x": 6, + "y": 9, + "width": 16, + "height": 5 + }, + "queryRef": { + "kind": "query", + "queryId": "c2ad5370-f4df-4a13-ac31-2c095f88754d" + }, + "visualOptions": { + "table__enableRenderLinks": true, + "colorRulesDisabled": true, + "colorStyle": "light", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "table__renderLinks": [], + "colorRules": [] + } + }, + { + "id": "59a1756c-7586-4364-8047-604aa7d6f09a", + "title": "Fully utilized vCPU capacity (last n days)", + "visualType": "table", + "pageId": "a4ec1d55-5b6e-49af-bb9e-4f3d136cdf05", + "layout": { + "x": 6, + "y": 14, + "width": 16, + "height": 5 + }, + "queryRef": { + "kind": "query", + "queryId": "4614acfb-2288-4a71-97f7-32845a9ddcb2" + }, + "visualOptions": { + "table__enableRenderLinks": true, + "colorRulesDisabled": true, + "colorStyle": "light", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "table__renderLinks": [], + "colorRules": [] + } + }, + { + "id": "9197f822-bdf3-42af-8b43-5609c9b91f00", + "title": "Eligible resources (last n days)", + "visualType": "table", + "pageId": "a4ec1d55-5b6e-49af-bb9e-4f3d136cdf05", + "layout": { + "x": 6, + "y": 19, + "width": 16, + "height": 5 + }, + "queryRef": { + "kind": "query", + "queryId": "99d44f1f-c53a-40ee-b749-8e5557ec58c7" + }, + "visualOptions": { + "table__enableRenderLinks": true, + "colorRulesDisabled": true, + "colorStyle": "light", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "table__renderLinks": [], + "colorRules": [] + } + }, + { + "id": "d9d6d2e0-6403-418f-8aa0-1e2eb4ce2822", + "title": "", + "visualType": "markdownCard", + "pageId": "d01a8154-8a60-4d22-96ea-54b45b1417fe", + "layout": { + "x": 18, + "y": 5, + "width": 4, + "height": 8 + }, + "markdownText": "### Cloud sustainability\nCloud sustainability balances environmental and financial efficiency in cloud optimization, ensuring alignment with strategic objectives. With this capability, you focus explicitly on optimizing carbon emissions through usage and workload analysis. This capability relies heavily on understanding the carbon impact of different services across different regions to identify the right cost/carbon balance for your organizational goals.\n\n\ud83d\udcca Coming soon\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/sustainability)", + "visualOptions": {} + }, + { + "id": "43e7bf02-fa40-4a35-9b41-e45496d0404b", + "title": "", + "visualType": "markdownCard", + "pageId": "11f019cd-e0cc-4aa7-a125-4dc852eb3f10", + "layout": { + "x": 0, + "y": 0, + "width": 22, + "height": 5 + }, + "markdownText": "# Manage the FinOps practice\nThe **Manage the FinOps practice** domain focuses on establishing a clear and consistent vision of FinOps and driving cultural adoption across your organization. Other domains are focused on the FinOps tasks you perform to drive efficiency and maximize value. This domain is focused more on how you run your FinOps practice and supporting those efforts.\n\n[Learn more](https://learn.microsoft.com/cloud-computing/finops/framework/manage/manage-finops)     [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Manage)\n", + "visualOptions": {} + }, + { + "id": "9c05ec17-5d60-4ed3-bc8c-6214e8955312", + "title": "", + "visualType": "markdownCard", + "pageId": "11f019cd-e0cc-4aa7-a125-4dc852eb3f10", + "layout": { + "x": 0, + "y": 5, + "width": 6, + "height": 7 + }, + "markdownText": "### FinOps education and enablement\nFinOps education and enablement involve refers to the process of providing training, resources, and support to help individuals and teams within an organization adopt FinOps practices. This capability ensures that stakeholders across the organization have the resources they need to:\n\n- Understand FinOps\n- Understand how it can help them achieve their goals\n- Understand how to perform FinOps tasks using the available tools and services.\n\n\ud83e\udded [Open FinOps guide](https://aka.ms/finops/guide)\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/education)", + "visualOptions": {} + }, + { + "id": "abf6cf66-48b8-4d86-826d-64ebc9ddcc90", + "title": "", + "visualType": "markdownCard", + "pageId": "11f019cd-e0cc-4aa7-a125-4dc852eb3f10", + "layout": { + "x": 6, + "y": 5, + "width": 6, + "height": 7 + }, + "markdownText": "### FinOps practice operations\nFinOps practice operations refer to the process of:\n\n- Building and managing a robust FinOps team\n- Defining clear cross-functional responsibilities\n- Integrating FinOps practices into organizational processes to manage cloud cost and usage effectively.\n\nWith this capability, you establish and manage your FinOps practice, focusing on supporting the needs of stakeholders across the organization. This capability is pivotal to driving organizational adoption.\n\n\ud83d\udcca Coming soon\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/operations)", + "visualOptions": {} + }, + { + "id": "20c47c65-228a-4c70-9d7c-418ad1c8c5cf", + "title": "", + "visualType": "markdownCard", + "pageId": "11f019cd-e0cc-4aa7-a125-4dc852eb3f10", + "layout": { + "x": 12, + "y": 5, + "width": 5, + "height": 7 + }, + "markdownText": "### Onboarding workloads\nOnboarding workloads refers to the process of bringing new and existing applications into the cloud based on their financial and technical feasibility. With this capability, you:\n\n- Understand requirements for new workloads\n- Identify if there are any architectural requirements or changes that should be considered\n- Validate that the budget is available to support their needs\n\n\ud83d\udcca Coming soon\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/onboarding)", + "visualOptions": {} + }, + { + "id": "31c7508b-6298-4257-9c35-a7d9cd7c84e2", + "title": "", + "visualType": "markdownCard", + "pageId": "11f019cd-e0cc-4aa7-a125-4dc852eb3f10", + "layout": { + "x": 17, + "y": 5, + "width": 5, + "height": 7 + }, + "markdownText": "### Cloud policy and governance\nCloud policy and governance refer to the process of defining, implementing, and monitoring a framework of rules that guide an organization's FinOps efforts. With this capability, you identify and implement policies to support organizational goals by promoting or limiting the use of:\n\n- Specific SKUs\n- Resource configurations\n- Other practices that might affect cost, usage, and carbon growth\n\n\ud83d\udcca Coming soon\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/policy)", + "visualOptions": {} + }, + { + "id": "a000abed-3a39-4675-806c-9423ff344d2e", + "title": "", + "visualType": "markdownCard", + "pageId": "11f019cd-e0cc-4aa7-a125-4dc852eb3f10", + "layout": { + "x": 0, + "y": 12, + "width": 6, + "height": 7 + }, + "markdownText": "### Invoicing and chargeback\nInvoicing and chargeback refer to the process of receiving, reconciling, and paying provider invoices, and then billing internal teams for their respective cloud costs using existing internal finance tools and processes. With this capability, you break down invoices and cross-charge portions of these invoices to the teams responsible for the charges.\n\n\ud83d\udcca [View report](#f416685d-f559-4514-8e45-5e0e09aec286)\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/chargeback)", + "visualOptions": {} + }, + { + "id": "fc3beac9-e524-4262-bdc5-39f7168e11ff", + "title": "", + "visualType": "markdownCard", + "pageId": "11f019cd-e0cc-4aa7-a125-4dc852eb3f10", + "layout": { + "x": 6, + "y": 12, + "width": 6, + "height": 7 + }, + "markdownText": "### FinOps assessment\nFinOps assessment refers to the process of measuring the effectiveness of a FinOps practice, aligning activities with organizational goals, and identifying areas for improvement to drive FinOps maturity. With this capability, you assess and benchmark the maturity of different teams to identify focus areas for key milestones. This capability is important to set clear, quantitative expectations with teams that are backed up with recommendations to drive targeted improvements and prioritize organizational goals.\n\n\ud83d\udccb [Start a FinOps review](https://aka.ms/finops/review)\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/assessment)", + "visualOptions": {} + }, + { + "id": "d525fe75-1afb-43f3-89c3-b8ccfbc8f570", + "title": "", + "visualType": "markdownCard", + "pageId": "11f019cd-e0cc-4aa7-a125-4dc852eb3f10", + "layout": { + "x": 12, + "y": 12, + "width": 5, + "height": 7 + }, + "markdownText": "### FinOps tools and services\nFinOps tools and services encapsulate identifying, configuring, and integrating tools and services that meet the needs of FinOps capabilities. They also enable the FinOps practice at scale throughout the organization. With this capability, you identify the tools and services needed to accelerate FinOps tasks and support your FinOps practice.\n\n\ud83d\udcca Coming soon\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/tools)", + "visualOptions": {} + }, + { + "id": "c2452fca-3ba3-40f6-9ccb-94ea6d075a12", + "title": "", + "visualType": "markdownCard", + "pageId": "11f019cd-e0cc-4aa7-a125-4dc852eb3f10", + "layout": { + "x": 17, + "y": 12, + "width": 5, + "height": 7 + }, + "markdownText": "### Intersecting frameworks\nFinOps and intersecting frameworks refers to integrating FinOps practices with other frameworks and methodologies used by an organization. With this capability, you'll partner with stakeholders in other disciplines to understand each other's others priorities to achieve an acceptable balance compared to cost, usage, and carbon efficiency goals.\n\n\ud83d\udcca Coming soon\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/frameworks)", + "visualOptions": {} + }, + { + "id": "876b81e5-4f71-4c69-bd62-8d470f50bf31", + "title": "About this capability", + "visualType": "markdownCard", + "pageId": "f416685d-f559-4514-8e45-5e0e09aec286", + "layout": { + "x": 0, + "y": 0, + "width": 21, + "height": 3 + }, + "markdownText": "Invoicing and chargeback refer to the process of receiving, reconciling, and paying provider invoices, and then billing internal teams for their respective cloud costs using existing internal finance tools and processes. With this capability, you break down invoices and cross-charge portions of these invoices to the teams responsible for the charges.\r\n\r\n[Learn more](http://aka.ms/ftk/fx/chargeback)     [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Manage.Invoicing)", + "visualOptions": {} + }, + { + "id": "a66f0b2b-6bce-4bfb-ad3e-13f7eabcb586", + "title": "About this capability", + "visualType": "markdownCard", + "pageId": "5c57f940-6f0f-4e39-930b-b6e89bb758ad", + "layout": { + "x": 0, + "y": 0, + "width": 22, + "height": 3 + }, + "markdownText": "Budgeting is the process of monitoring and managing financial plans and limits over a specific period to control spending effectively. This capability allows you to use established estimates and forecasts to allocate funds to each team for fiscal year planning and set up alerts to ensure budget accountability. This capability plays a key role in staying within established fiscal constraints on an ongoing basis.\r\n\r\n[Learn more](http://aka.ms/ftk/fx/budgeting)     [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.Quantify.Budgeting.Nav)", + "visualOptions": {} + }, + { + "id": "08c92e20-6563-464b-893f-81d80dd3caa1", + "title": "About this capability", + "visualType": "markdownCard", + "pageId": "5838c918-4541-44fe-90d2-77306ef1e241", + "layout": { + "x": 0, + "y": 0, + "width": 16, + "height": 4 + }, + "markdownText": "Anomaly management refers to the practice of detecting and addressing abnormal or unexpected cost and usage patterns in a timely manner. With this capability, you identify, triage, and address unexpected changes in cost, usage, and carbon. This capability is especially important with usage-based billing and sustainability models, where small fluctuations in usage can result in large fiscal and environmental impact.\n\n[Learn more](http://aka.ms/ftk/fx/anomalies)     [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.Anomalies.Nav)\n", + "visualOptions": {} + }, + { + "id": "6fb4c25e-ae84-495b-88ae-6dd2035500d1", + "title": "Forecast (next n days)", + "visualType": "timechart", + "pageId": "5838c918-4541-44fe-90d2-77306ef1e241", + "layout": { + "x": 0, + "y": 33, + "width": 9, + "height": 7 + }, + "queryRef": { + "kind": "query", + "queryId": "5fd3ea2a-8882-4438-9103-fa3945240604" + }, + "visualOptions": { + "multipleYAxes": { + "base": { + "id": "-1", + "label": "", + "columns": [], + "yAxisMaximumValue": null, + "yAxisMinimumValue": null, + "yAxisScale": "linear", + "horizontalLines": [] + }, + "additional": [], + "showMultiplePanels": false + }, + "hideLegend": false, + "legendLocation": "bottom", + "xColumnTitle": "", + "xColumn": null, + "yColumns": null, + "seriesColumns": null, + "xAxisScale": "linear", + "verticalLine": "", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "selectedDataOnLoad": { + "all": true, + "limit": 10 + }, + "dataPointsTooltip": { + "all": false, + "limit": 1 + } + } + }, + { + "id": "961b8980-213f-4bc7-9512-8a08614229e0", + "title": "Effective Savings Rate (last n months)", + "visualType": "multistat", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 0, + "y": 17, + "width": 15, + "height": 4 + }, + "queryRef": { + "kind": "query", + "queryId": "e1bc2d51-44af-4dd9-8b0d-a71088b551f5" + }, + "visualOptions": { + "multiStat__textSize": "auto", + "multiStat__valueColumn": "Value", + "colorRulesDisabled": false, + "colorStyle": "light", + "multiStat__displayOrientation": "horizontal", + "multiStat__labelColumn": "Label", + "multiStat__slot": { + "width": 5, + "height": 1 + }, + "colorRules": [ + { + "id": "a45d5307-a815-4baa-9743-d142253dc699", + "ruleType": "colorByCondition", + "applyToColumn": null, + "hideText": false, + "applyTo": "cells", + "conditions": [ + { + "operator": "==", + "column": "Type", + "values": [ + "List" + ] + } + ], + "chainingOperator": "and", + "colorStyle": "light", + "color": "red", + "tag": "List cost", + "icon": null, + "ruleName": "List cost", + "visualType": "multistat" + }, + { + "id": "a1307a22-3b48-48a9-9c1b-ef6b6a228796", + "ruleType": "colorByCondition", + "applyToColumn": null, + "hideText": false, + "applyTo": "cells", + "conditions": [ + { + "operator": "==", + "column": "Type", + "values": [ + "Contracted" + ] + } + ], + "chainingOperator": "and", + "colorStyle": "light", + "color": "yellow", + "tag": "Contracted cost", + "icon": null, + "ruleName": "Contracted cost", + "visualType": "multistat" + }, + { + "id": "65c6cac8-64f6-4029-bca0-b09dbfef9516", + "ruleType": "colorByCondition", + "applyToColumn": null, + "hideText": false, + "applyTo": "cells", + "conditions": [ + { + "operator": "==", + "column": "Type", + "values": [ + "Effective" + ] + } + ], + "chainingOperator": "and", + "colorStyle": "light", + "color": "green", + "tag": "Effective cost", + "icon": null, + "ruleName": "Effective cost", + "visualType": "multistat" + }, + { + "id": "33f99591-c8a6-4aae-be3f-a613bbc85c28", + "ruleType": "colorByCondition", + "applyToColumn": null, + "hideText": false, + "applyTo": "cells", + "conditions": [ + { + "operator": "==", + "column": "Type", + "values": [ + "PartialSavings" + ] + } + ], + "chainingOperator": "and", + "colorStyle": "bold", + "color": "blue", + "tag": "Partial savings", + "icon": null, + "ruleName": "Partial savings", + "visualType": "multistat" + }, + { + "id": "55430568-c3ef-4ff7-84b9-f3f82fe1a05a", + "ruleType": "colorByCondition", + "applyToColumn": null, + "hideText": false, + "applyTo": "cells", + "conditions": [ + { + "operator": "==", + "column": "Type", + "values": [ + "TotalSavings" + ] + } + ], + "chainingOperator": "and", + "colorStyle": "bold", + "color": "green", + "tag": "Total savings", + "icon": null, + "ruleName": "Total savings", + "visualType": "multistat" + }, + { + "id": "852c731e-90ec-4438-8bb7-00be0127b802", + "ruleType": "colorByCondition", + "applyToColumn": null, + "hideText": false, + "applyTo": "cells", + "conditions": [ + { + "operator": "==", + "column": "Type", + "values": [ + "EffectiveSavingsRate" + ] + } + ], + "chainingOperator": "and", + "colorStyle": "bold", + "color": "yellow", + "tag": "ESR", + "icon": null, + "ruleName": "ESR", + "visualType": "multistat" + } + ] + } + }, + { + "id": "6367a5d7-99ca-4afc-b9d4-4c77ec259a07", + "title": "Savings breakdown by month", + "visualType": "table", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 0, + "y": 27, + "width": 10, + "height": 7 + }, + "queryRef": { + "kind": "query", + "queryId": "3b3f0a58-2d84-4e3e-bebc-3e747a7d5ede" + }, + "visualOptions": { + "table__enableRenderLinks": true, + "colorRulesDisabled": false, + "colorStyle": "light", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "table__renderLinks": [], + "colorRules": [] + } + }, + { + "id": "69e3a7e2-9477-4b63-8c86-9d66a782501b", + "title": "Effective Savings Rate (last n days)", + "visualType": "multistat", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 0, + "y": 53, + "width": 15, + "height": 4 + }, + "queryRef": { + "kind": "query", + "queryId": "cb34f22b-9370-460c-9658-3e73d220bbc7" + }, + "visualOptions": { + "multiStat__textSize": "auto", + "multiStat__valueColumn": "Value", + "colorRulesDisabled": false, + "colorStyle": "light", + "multiStat__displayOrientation": "horizontal", + "multiStat__labelColumn": "Label", + "multiStat__slot": { + "width": 5, + "height": 1 + }, + "colorRules": [ + { + "id": "a45d5307-a815-4baa-9743-d142253dc699", + "ruleType": "colorByCondition", + "applyToColumn": null, + "hideText": false, + "applyTo": "cells", + "conditions": [ + { + "operator": "==", + "column": "Type", + "values": [ + "List" + ] + } + ], + "chainingOperator": "and", + "colorStyle": "light", + "color": "red", + "tag": "List cost", + "icon": null, + "ruleName": "List cost", + "visualType": "multistat" + }, + { + "id": "a1307a22-3b48-48a9-9c1b-ef6b6a228796", + "ruleType": "colorByCondition", + "applyToColumn": null, + "hideText": false, + "applyTo": "cells", + "conditions": [ + { + "operator": "==", + "column": "Type", + "values": [ + "Contracted" + ] + } + ], + "chainingOperator": "and", + "colorStyle": "light", + "color": "yellow", + "tag": "Contracted cost", + "icon": null, + "ruleName": "Contracted cost", + "visualType": "multistat" + }, + { + "id": "65c6cac8-64f6-4029-bca0-b09dbfef9516", + "ruleType": "colorByCondition", + "applyToColumn": null, + "hideText": false, + "applyTo": "cells", + "conditions": [ + { + "operator": "==", + "column": "Type", + "values": [ + "Effective" + ] + } + ], + "chainingOperator": "and", + "colorStyle": "light", + "color": "green", + "tag": "Effective cost", + "icon": null, + "ruleName": "Effective cost", + "visualType": "multistat" + }, + { + "id": "33f99591-c8a6-4aae-be3f-a613bbc85c28", + "ruleType": "colorByCondition", + "applyToColumn": null, + "hideText": false, + "applyTo": "cells", + "conditions": [ + { + "operator": "==", + "column": "Type", + "values": [ + "PartialSavings" + ] + } + ], + "chainingOperator": "and", + "colorStyle": "bold", + "color": "blue", + "tag": "Partial savings", + "icon": null, + "ruleName": "Partial savings", + "visualType": "multistat" + }, + { + "id": "55430568-c3ef-4ff7-84b9-f3f82fe1a05a", + "ruleType": "colorByCondition", + "applyToColumn": null, + "hideText": false, + "applyTo": "cells", + "conditions": [ + { + "operator": "==", + "column": "Type", + "values": [ + "TotalSavings" + ] + } + ], + "chainingOperator": "and", + "colorStyle": "bold", + "color": "green", + "tag": "Total savings", + "icon": null, + "ruleName": "Total savings", + "visualType": "multistat" + }, + { + "id": "852c731e-90ec-4438-8bb7-00be0127b802", + "ruleType": "colorByCondition", + "applyToColumn": null, + "hideText": false, + "applyTo": "cells", + "conditions": [ + { + "operator": "==", + "column": "Type", + "values": [ + "EffectiveSavingsRate" + ] + } + ], + "chainingOperator": "and", + "colorStyle": "bold", + "color": "yellow", + "tag": "ESR", + "icon": null, + "ruleName": "ESR", + "visualType": "multistat" + } + ] + } + }, + { + "id": "fc2e27cf-57be-486f-afa5-6a93d6e9c6a7", + "title": "Ingested months", + "visualType": "multistat", + "pageId": "9e099251-9658-48da-b416-80422a2a47c7", + "layout": { + "x": 0, + "y": 9, + "width": 9, + "height": 4 + }, + "queryRef": { + "kind": "query", + "queryId": "8de47213-8327-44da-9d1b-8ba5de74c44a" + }, + "visualOptions": { + "multiStat__textSize": "auto", + "multiStat__valueColumn": "Value", + "colorRulesDisabled": false, + "colorStyle": "light", + "multiStat__displayOrientation": "horizontal", + "multiStat__labelColumn": "Label", + "multiStat__slot": { + "width": 3, + "height": 1 + }, + "colorRules": [] + } + }, + { + "id": "2e9d6a7f-6393-41ad-83df-51faddbe3b8e", + "title": "Effective cost breakdown by month", + "visualType": "table", + "pageId": "306fef9a-c760-4559-a326-7c25d196b616", + "layout": { + "x": 10, + "y": 27, + "width": 12, + "height": 7 + }, + "queryRef": { + "kind": "query", + "queryId": "0341b3e4-eccc-4924-9555-9835b128c543" + }, + "visualOptions": { + "table__enableRenderLinks": true, + "colorRulesDisabled": false, + "colorStyle": "light", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "table__renderLinks": [], + "colorRules": [] + } + }, + { + "id": "daa1b2bd-0ccd-43ff-ad2c-1337d247805a", + "title": "", + "visualType": "markdownCard", + "pageId": "9e099251-9658-48da-b416-80422a2a47c7", + "layout": { + "x": 0, + "y": 68, + "width": 22, + "height": 4 + }, + "markdownText": "# Data quality\r\nData quality checks that may impact your reports.\r\n\r\n\u2b06\ufe0f [Top](?tile=ab300a6a-3bbd-450b-942c-7f30113cba15)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.DataIngestion.DataQuality)", + "visualOptions": {} + }, + { + "id": "d5ea61ef-d4f5-423e-bfa4-d1ab473867c2", + "title": "Summary of list, contracted, and effective cost alignment", + "visualType": "table", + "pageId": "9e099251-9658-48da-b416-80422a2a47c7", + "layout": { + "x": 0, + "y": 72, + "width": 22, + "height": 8 + }, + "queryRef": { + "kind": "query", + "queryId": "56fd8707-bbb3-4f7e-8e37-8dd90ada3baa" + }, + "description": "This table summarizes rows where cost columns are not set or may be incorrect in the data provided by Cost Management.", + "visualOptions": { + "table__enableRenderLinks": true, + "colorRulesDisabled": true, + "colorStyle": "light", + "crossFilterDisabled": false, + "drillthroughDisabled": false, + "crossFilter": [], + "drillthrough": [], + "table__renderLinks": [], + "colorRules": [] + } + } + ], + "baseQueries": [ + { + "id": "58764bcb-2ba0-4c7f-b018-e5d7f6cff688", + "queryId": "43612ae4-c475-4f22-bb50-ce9d995abb8f", + "variableName": "CostsThisMonth" + }, + { + "id": "21512220-154c-4646-a188-72e8c437d8ed", + "queryId": "cb1f5404-c0b1-42fd-99fb-3cff7b08daaa", + "variableName": "CostsLastMonth" + }, + { + "id": "5fa73857-4e92-4a16-8179-f94563e5f605", + "queryId": "4ce0f587-2d45-436c-8f79-102c6b382439", + "variableName": "CostsByMonth" + }, + { + "id": "4880346a-0a24-48f9-bf25-b7427df29d69", + "queryId": "6b598467-8c31-4693-b1eb-7ed683fcfc3a", + "variableName": "CostsByDay" + }, + { + "id": "8ca40660-3fbc-4a08-b12e-92e7382d9449", + "queryId": "4a1973bf-08e9-4e82-b8e6-6edff81cf0a5", + "variableName": "CostsByDayAHB" + }, + { + "id": "48ebd897-d085-490f-8b4b-9b43d4fb2efc", + "queryId": "eb9259cc-05b7-4441-a66d-a29026fe371b", + "variableName": "CostsPlus" + } + ], + "parameters": [ + { + "kind": "int", + "id": "50e8acdc-5d90-4937-96cb-53a9b0ad047f", + "displayName": "Monthly trend", + "description": "How many months should be shown in monthly trend charts and tables?", + "variableName": "numberOfMonths", + "selectionType": "scalar", + "includeAllOption": false, + "defaultValue": { + "kind": "query-result" + }, + "dataSource": { + "kind": "query", + "columns": { + "value": "Value", + "label": "Label" + }, + "queryRef": { + "kind": "query", + "queryId": "c9039243-968d-4e75-9899-8d4ab51a9896" + } + }, + "showOnPages": { + "kind": "all" + } + }, + { + "kind": "int", + "selectionType": "freetext", + "id": "c016c954-1d37-41fb-8c4e-7607a9be2e96", + "displayName": "Daily trend", + "variableName": "numberOfDays", + "description": "How many days should be shown in daily trend charts and tables? Does not apply to \"this month and last\" queries.", + "defaultValue": { + "kind": "value", + "value": 28 + }, + "showOnPages": { + "kind": "all" + } + }, + { + "kind": "int", + "selectionType": "freetext", + "id": "55463317-2576-4d98-875e-85a1fa032c12", + "displayName": "Max group count", + "variableName": "maxGroupCount", + "description": "How many groups should be shown in charts? Remaining will be in an \"others\" group.", + "defaultValue": { + "kind": "value", + "value": 9 + }, + "showOnPages": { + "kind": "all" + } + }, + { + "kind": "string", + "id": "7c3e4a2b-1d8f-4e9a-b5c7-a9d6f3e2b4c1", + "displayName": "Currency", + "description": "Filter costs to a single billing currency. Aggregating values across currencies without conversion produces incorrect totals and savings rates.", + "variableName": "selectedBillingCurrency", + "selectionType": "scalar", + "includeAllOption": false, + "defaultValue": { + "kind": "query-result" + }, + "dataSource": { + "kind": "query", + "columns": { + "value": "BillingCurrency", + "label": "Label" + }, + "queryRef": { + "kind": "query", + "queryId": "f2a8c4d6-3b5e-4a7f-9c2d-8e5b1f4a7d9c" + } + }, + "showOnPages": { + "kind": "all" + } + } + ], + "dataSources": [ + { + "id": "23540be2-ffc9-4b61-8c4c-05e493e682a6", + "kind": "manual-kusto", + "scopeId": "kusto", + "name": "Hub", + "clusterUri": "http://localhost:8082", + "database": "Hub" + } + ], + "pages": [ + { + "name": "About", + "id": "969ddf4c-8f2a-4ec4-9588-bb2f39473c9f" + }, + { + "name": "UNDERSTAND", + "id": "ea47329e-0bc9-4c11-b110-534878dbb3ad" + }, + { + "id": "f8ee3008-df7e-442f-825e-2e3b46e4c185", + "name": "- Summary" + }, + { + "name": "- Anomaly management", + "id": "5838c918-4541-44fe-90d2-77306ef1e241" + }, + { + "name": "- Data ingestion", + "id": "9e099251-9658-48da-b416-80422a2a47c7" + }, + { + "name": "OPTIMIZE", + "id": "d01a8154-8a60-4d22-96ea-54b45b1417fe" + }, + { + "name": "- Rate optimization", + "id": "306fef9a-c760-4559-a326-7c25d196b616" + }, + { + "id": "a4ec1d55-5b6e-49af-bb9e-4f3d136cdf05", + "name": "- Licensing + SaaS" + }, + { + "name": "QUANTIFY", + "id": "8beab65c-f5ec-4661-bc67-37b10baffb16" + }, + { + "name": "- Budgeting", + "id": "5c57f940-6f0f-4e39-930b-b6e89bb758ad" + }, + { + "name": "MANAGE", + "id": "11f019cd-e0cc-4aa7-a125-4dc852eb3f10" + }, + { + "name": "- Invoicing + chargeback", + "id": "f416685d-f559-4514-8e45-5e0e09aec286" + } + ], + "queries": [ + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let monthname = dynamic(['(ignore)', 'Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']);\nlet costs = materialize(\n CostsLastMonth\n | summarize BilledCost = round(sum(BilledCost), 2), EffectiveCost = round(sum(EffectiveCost), 2) by BillingPeriodStart = startofmonth(BillingPeriodStart)\n | extend json = todynamic(strcat('[{\"type\":\"Billed cost\", \"Cost\":', BilledCost, '}, {\"type\":\"Effective cost\", \"Cost\":', EffectiveCost, '}]'))\n | mv-expand json\n | project Type = strcat(json.type, ' (', monthname[monthofyear(BillingPeriodStart)], ' ', format_datetime(BillingPeriodStart, 'yyyy'), ')'), Cost = todouble(json.Cost)\n);\ncosts", + "id": "152f2041-bbc1-41e4-b155-271b2e0cf6e9", + "usedVariables": [ + "CostsLastMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "CostsByMonth\n| summarize BilledCost = round(sum(BilledCost), 2), EffectiveCost = round(sum(EffectiveCost), 2) by BillingPeriodStart = startofmonth(BillingPeriodStart)\n| render timechart", + "id": "bc24e050-f2b9-4b4a-a08d-69fc4a4bb95e", + "usedVariables": [ + "CostsByMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "CostsByMonth\n// | summarize BilledCost = round(sum(BilledCost), 2), EffectiveCost = round(sum(EffectiveCost), 2) by BillingPeriodStart = startofmonth(BillingPeriodStart)\n// | render timechart\n| summarize BilledCost = sum(BilledCost), EffectiveCost = sum(EffectiveCost) by BillingPeriodStart = startofmonth(BillingPeriodStart)\n| order by BillingPeriodStart asc\n| extend PreviousBilledCost = prev(BilledCost)\n| extend PreviousEffectiveCost = prev(EffectiveCost)\n| project BillingPeriodStart\n , BilledCost = iif(isempty(PreviousBilledCost), todouble(0), todouble((BilledCost - PreviousBilledCost) * 100.0 / PreviousBilledCost))\n , EffectiveCost = iif(isempty(PreviousEffectiveCost), todouble(0), todouble((EffectiveCost - PreviousEffectiveCost) * 100.0 / PreviousEffectiveCost))\n", + "id": "0d91ea4a-c81d-4a21-b708-b6af37be1eec", + "usedVariables": [ + "CostsByMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let data = materialize(\n CostsLastMonth\n | summarize\n Subscriptions = dcount(SubAccountId),\n ResourceGroups = dcount(strcat(SubAccountId, x_ResourceGroupName)),\n Resources = dcount(ResourceId),\n Services = dcount(ServiceName)\n | project json = todynamic(strcat('[{ \"Type\":\"Subscriptions\", \"Count\":', Subscriptions, ' }, { \"Type\":\"Resource groups\", \"Count\":', ResourceGroups, ' }, { \"Type\":\"Resources\", \"Count\":', Resources, ' }, { \"Type\":\"Services\", \"Count\":', Services, ' }]'))\n | mv-expand json\n | project Label = tostring(json.Type), Count = tolong(json.Count)\n);\ndata", + "id": "f2cecbb0-13f8-4642-afa4-bbcc0558f777", + "usedVariables": [ + "CostsLastMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let data = materialize(\n CostsByDay\n | summarize\n Subscriptions = dcount(SubAccountId),\n ResourceGroups = dcount(strcat(SubAccountId, x_ResourceGroupName)),\n Resources = dcount(ResourceId),\n Services = dcount(ServiceName)\n | project json = todynamic(strcat('[{ \"Type\":\"Subscriptions\", \"Count\":', Subscriptions, ' }, { \"Type\":\"Resource groups\", \"Count\":', ResourceGroups, ' }, { \"Type\":\"Resources\", \"Count\":', Resources, ' }, { \"Type\":\"Services\", \"Count\":', Services, ' }]'))\n | mv-expand json\n | project Label = tostring(json.Type), Count = tolong(json.Count)\n);\ndata", + "id": "5d63e04d-1a11-4307-96f1-ebbf68e09be0", + "usedVariables": [ + "CostsByDay" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "CostsByMonth\n| summarize EffectiveCost = sum(EffectiveCost) by BillingPeriodStart\n| order by BillingPeriodStart asc\n| extend PreviousEffectiveCost = prev(EffectiveCost)\n| project BillingPeriodStart, EffectiveCost, Change = iif(isempty(PreviousEffectiveCost), todouble(0), todouble((EffectiveCost - PreviousEffectiveCost) / PreviousEffectiveCost)) * 100\n", + "id": "290e7eab-8159-4338-8531-85e2718cedb1", + "usedVariables": [ + "CostsByMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nCostsPlus\n| where ChargePeriodStart >= startofmonth(ago(90d))\n| summarize EffectiveCost = sum(EffectiveCost) by ChargePeriodStart, Day = dayofmonth(ChargePeriodStart), Month = strcat(format_datetime(ChargePeriodStart, 'MM '), monthname[monthofyear(ChargePeriodStart)])\n| order by ChargePeriodStart asc\n| extend EffectiveCostRunningTotal = row_cumsum(EffectiveCost, prev(Month) != Month)\n| project Day, EffectiveCostRunningTotal, Month\n| render areachart ", + "id": "d5ed469a-45ea-49a2-b305-b841050213cf", + "usedVariables": [ + "CostsPlus" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nCostsPlus\n| where ChargePeriodStart >= startofmonth(startofmonth(now()) - 1d)\n| summarize EffectiveCost = sum(EffectiveCost) by ChargePeriodStart, Month = strcat(format_datetime(ChargePeriodStart, 'MM '), monthname[monthofyear(ChargePeriodStart)])\n| order by ChargePeriodStart asc\n| extend EffectiveCostRunningTotal = row_cumsum(EffectiveCost, prev(Month) != Month)\n| project ChargePeriodStart, EffectiveCostRunningTotal, Month\n| render areachart ", + "id": "10300876-866a-40d7-836d-b3a8783ecece", + "usedVariables": [ + "CostsPlus" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nCostsPlus\n| where ChargePeriodStart >= startofmonth(ago(90d))\n| summarize EffectiveCost = sum(EffectiveCost) by ChargePeriodStart, Day = dayofmonth(ChargePeriodStart), Month = strcat(format_datetime(ChargePeriodStart, 'MM '), monthname[monthofyear(ChargePeriodStart)])\n| order by ChargePeriodStart asc\n| project Day, EffectiveCost, Month\n| render columnchart", + "id": "9e41a624-d5f9-40c6-b47f-fef4b50ec3dd", + "usedVariables": [ + "CostsPlus" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let costs = CostsPlus | where ChargePeriodStart >= startofmonth(ago(90d));\nlet all = costs | summarize sum(EffectiveCost) by SubAccountId;\nlet count = toscalar(all | order by sum_EffectiveCost desc | count);\nlet topX = all | order by sum_EffectiveCost desc | limit maxGroupCount;\nlet otherId = '(others)';\ncosts\n//\n// Group rows after max count\n| extend inTopX = SubAccountId in (topX)\n| extend SubAccountId = iff(inTopX, SubAccountId, otherId)\n| extend SubAccountName = iff(inTopX, SubAccountName, strcat('(', (count - maxGroupCount), ' others)'))\n//\n| summarize \n EffectiveCost = round(sum(EffectiveCost), 2),\n SubAccountName = take_any(SubAccountName)\n by\n ChargePeriodStart,\n SubAccountId\n| project ChargePeriodStart, EffectiveCost, Sub = iff(SubAccountId == otherId, SubAccountName, strcat(SubAccountName, ' (', split(SubAccountId, '/')[2], ')'))\n| order by EffectiveCost desc\n| render columnchart", + "id": "f26e2204-270e-4219-8f68-5acef1c9393f", + "usedVariables": [ + "CostsPlus", + "maxGroupCount" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nCostsPlus\n| where startofmonth(ChargePeriodStart) >= startofmonth(now(), -1)\n| summarize \n EffectiveCost = sum(EffectiveCost),\n ContractedCost = sum(ContractedCost),\n ListCost = sum(ListCost)\n by\n ChargePeriodStart,\n Month = strcat(format_datetime(ChargePeriodStart, 'MM '), monthname[monthofyear(ChargePeriodStart)])\n| extend CommitmentDiscountSavings = ContractedCost - EffectiveCost\n| extend NegotiatedDiscountSavings = ListCost - ContractedCost\n| order by ChargePeriodStart asc\n| extend EffectiveCostRunningTotal = row_cumsum(EffectiveCost, prev(Month) != Month)\n| extend CommitmentDiscountSavingsRunningTotal = row_cumsum(CommitmentDiscountSavings, prev(Month) != Month)\n| extend NegotiatedDiscountSavingsRunningTotal = row_cumsum(NegotiatedDiscountSavings, prev(Month) != Month)\n| project ChargePeriodStart, CommitmentDiscountSavingsRunningTotal, NegotiatedDiscountSavingsRunningTotal, EffectiveCostRunningTotal, Month\n| render areachart ", + "id": "dcc1f533-e5f9-4855-9e4c-21e21fcdf943", + "usedVariables": [ + "CostsPlus" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet data = materialize(\n CostsByMonth\n | where ChargePeriodStart >= startofmonth(now(), -1)\n | summarize \n BilledCost = sum(BilledCost),\n EffectiveCost = sum(EffectiveCost),\n ContractedCost = sum(ContractedCost),\n ListCost = sum(ListCost)\n by\n ChargePeriodStart\n | order by ChargePeriodStart asc\n | extend CommitmentDiscountSavings = ContractedCost - EffectiveCost\n | extend NegotiatedDiscountSavings = ListCost - ContractedCost\n | extend Month = monthname[monthofyear(ChargePeriodStart)]\n | project json = todynamic(strcat('[{ \"Type\":\"Billed cost\", \"Count\":', BilledCost, ' }, { \"Type\":\"Effective cost\", \"Count\":', EffectiveCost, ' }, { \"Type\":\"Commitment savings\", \"Count\":', CommitmentDiscountSavings, ' }, { \"Type\":\"Negotiated savings\", \"Count\":', NegotiatedDiscountSavings, ' }]')), Month, IsThisMonth = ChargePeriodStart >= startofmonth(now())\n | mv-expand json\n | project Label = strcat(json.Type, ' (', Month, ')'), Count = tolong(json.Count), IsThisMonth\n);\ndata", + "id": "e8b343dc-7430-4487-8d44-ef48ac454f2d", + "usedVariables": [ + "CostsByMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet data = (\n CostsByMonth\n | summarize \n BilledCost = todouble(round(sum(BilledCost), 2)),\n EffectiveCost = todouble(round(sum(EffectiveCost), 2)),\n ContractedCost = todouble(round(sum(ContractedCost), 2)),\n ListCost = todouble(round(sum(ListCost), 2))\n by\n ChargePeriodStart\n | order by ChargePeriodStart asc\n | extend CommitmentDiscountSavings = todouble(round(ContractedCost - EffectiveCost, 2))\n | extend NegotiatedDiscountSavings = todouble(round(ListCost - ContractedCost, 2))\n | extend ChargePeriod = strcat(format_datetime(ChargePeriodStart, 'yyyy-MM - '), monthname[monthofyear(ChargePeriodStart)])\n);\ndata | extend Data = 'Billed cost' | evaluate pivot(ChargePeriod, sum(BilledCost), Data)\n| union (data | extend Data = 'Effective cost' | evaluate pivot(ChargePeriod, sum(EffectiveCost), Data))\n| union (data | extend Data = 'Negotiated savings' | evaluate pivot(ChargePeriod, sum(NegotiatedDiscountSavings), Data))\n| union (data | extend Data = 'Commitment savings' | evaluate pivot(ChargePeriod, sum(CommitmentDiscountSavings), Data))\n", + "id": "c077a6d4-719f-42fe-b39b-36f77dc68976", + "usedVariables": [ + "CostsByMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet data = (\n CostsByMonth\n | summarize \n BilledCost = todouble(round(sum(BilledCost), 2)),\n EffectiveCost = todouble(round(sum(EffectiveCost), 2)),\n ContractedCost = todouble(round(sum(ContractedCost), 2)),\n ListCost = todouble(round(sum(ListCost), 2))\n by\n ChargePeriodStart = startofmonth(ChargePeriodStart)\n | order by ChargePeriodStart asc\n | extend CommitmentDiscountSavings = todouble(round(ContractedCost - EffectiveCost, 2))\n | extend NegotiatedDiscountSavings = todouble(round(ListCost - ContractedCost, 2))\n | extend ChargePeriod = strcat(format_datetime(ChargePeriodStart, 'yyyy-MM - '), monthname[monthofyear(ChargePeriodStart)])\n);\ndata | extend Data = 'Billed cost' | evaluate pivot(ChargePeriod, sum(BilledCost), Data)\n| union (data | extend Data = 'Effective cost' | evaluate pivot(ChargePeriod, sum(EffectiveCost), Data))\n| union (data | extend Data = 'Negotiated savings' | evaluate pivot(ChargePeriod, sum(NegotiatedDiscountSavings), Data))\n| union (data | extend Data = 'Commitment savings' | evaluate pivot(ChargePeriod, sum(CommitmentDiscountSavings), Data))\n//, BilledCost, NegotiatedDiscountSavings, CommitmentDiscountSavings)\n//| project ChargePeriod, BilledCost = round(BilledCost, 2), EffectiveCost = round(EffectiveCost, 2), NegotiatedDiscountSavings, CommitmentDiscountSavings\n", + "id": "3d947a7c-edca-46b5-a862-de77a725c85f", + "usedVariables": [ + "CostsByMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet data = (\n CostsByMonth\n | summarize \n BilledCost = sum(BilledCost),\n EffectiveCost = sum(EffectiveCost),\n ContractedCost = sum(ContractedCost),\n ListCost = sum(ListCost),\n SubAccountName = take_any(SubAccountName)\n by\n ChargePeriodStart = startofmonth(ChargePeriodStart),\n SubAccountId\n | as per\n | union (\n per\n | summarize \n BilledCost = sum(BilledCost),\n EffectiveCost = sum(EffectiveCost),\n ContractedCost = sum(ContractedCost),\n ListCost = sum(ListCost),\n SubAccountName = take_any(SubAccountName)\n by\n SubAccountId\n )\n | order by ChargePeriodStart asc\n | extend BilledCost = todouble(round(BilledCost, 2))\n | extend EffectiveCost = todouble(round(EffectiveCost, 2))\n | extend ContractedCost = todouble(round(ContractedCost, 2))\n | extend ListCost = todouble(round(ListCost, 2))\n | extend CommitmentDiscountSavings = todouble(round(ContractedCost - EffectiveCost, 2))\n | extend NegotiatedDiscountSavings = todouble(round(ListCost - ContractedCost, 2))\n | extend ChargePeriod = iff(isempty(ChargePeriodStart), strcat('Total'), strcat(format_datetime(ChargePeriodStart, 'yyyy-MM - '), monthname[monthofyear(ChargePeriodStart)]))\n);\ndata | evaluate pivot(ChargePeriod, sum(EffectiveCost), SubAccountName)\n| order by Total desc", + "id": "3886b5cd-34a8-42d7-9e16-33ea4d236953", + "usedVariables": [ + "CostsByMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let costs = CostsByMonth;\nlet all = costs | where isnotempty(RegionName) | summarize sum(EffectiveCost) by RegionName;\nlet count = toscalar(all | order by sum_EffectiveCost desc | count);\nlet topX = all | order by sum_EffectiveCost desc | limit iff(count - maxGroupCount > 1, maxGroupCount, count);\nlet otherId = '(others)';\ncosts\n//\n// Group rows after max count\n| extend inTopX = RegionName in (topX)\n| extend RegionName = iff(inTopX, RegionName, strcat('(', count - maxGroupCount, ' others)'))\n//\n| summarize \n EffectiveCost = round(sum(EffectiveCost), 2)\n by\n ChargePeriodStart = startofmonth(ChargePeriodStart),\n RegionName\n| project ChargePeriodStart, EffectiveCost, Region = RegionName\n| order by EffectiveCost desc", + "id": "d5224d06-c8e7-4dd9-afec-595b39712f5a", + "usedVariables": [ + "CostsByMonth", + "maxGroupCount" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet data = (\n CostsByMonth\n | summarize \n EffectiveCost = sum(EffectiveCost)\n by\n ChargePeriodStart = startofmonth(ChargePeriodStart),\n RegionName\n | as per\n | union (\n per\n | summarize \n EffectiveCost = sum(EffectiveCost)\n by\n RegionName\n )\n | order by ChargePeriodStart asc\n | extend EffectiveCost = todouble(round(EffectiveCost, 2))\n | extend ChargePeriod = iff(isempty(ChargePeriodStart), strcat('Total'), strcat(format_datetime(ChargePeriodStart, 'yyyy-MM - '), monthname[monthofyear(ChargePeriodStart)]))\n);\ndata | evaluate pivot(ChargePeriod, sum(EffectiveCost), RegionName)\n| order by Total desc", + "id": "0dcf2c54-7f1d-45e9-a53b-d598a24493a4", + "usedVariables": [ + "CostsByMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let costs = CostsByMonth;\nlet all = costs | where isnotempty(ServiceName) | summarize sum(EffectiveCost) by ServiceName;\nlet count = toscalar(all | order by sum_EffectiveCost desc | count);\nlet topX = all | order by sum_EffectiveCost desc | limit maxGroupCount;\nlet otherId = '(others)';\ncosts\n//\n// Group rows after max count\n| extend inTopX = ServiceName in (topX)\n| extend ServiceName = iff(inTopX, ServiceName, otherId)\n//\n| summarize \n EffectiveCost = round(sum(EffectiveCost), 2)\n by\n ChargePeriodStart,\n ServiceName\n| project ChargePeriodStart, EffectiveCost, Category = ServiceName\n| order by EffectiveCost desc\n| render columnchart", + "id": "30644718-defd-4c6c-9ffa-a9c2cd1f871f", + "usedVariables": [ + "CostsByMonth", + "maxGroupCount" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet data = (\n CostsByMonth\n | summarize \n EffectiveCost = sum(EffectiveCost)\n by\n ChargePeriodStart,\n ServiceCategory,\n ServiceName\n | as per\n | union (\n per\n | summarize \n EffectiveCost = sum(EffectiveCost)\n by\n ServiceCategory,\n ServiceName\n )\n | order by ChargePeriodStart asc\n | extend EffectiveCost = todouble(round(EffectiveCost, 2))\n | extend ChargePeriod = iff(isempty(ChargePeriodStart), strcat('Total'), strcat(format_datetime(ChargePeriodStart, 'yyyy-MM - '), monthname[monthofyear(ChargePeriodStart)]))\n);\ndata | evaluate pivot(ChargePeriod, sum(EffectiveCost), ServiceName, ServiceCategory)\n| order by Total desc", + "id": "6259f773-593c-4953-898c-15aa5ff6e53a", + "usedVariables": [ + "CostsByMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let costs = CostsByMonth;\nlet all = costs | where isnotempty(SubAccountId) | summarize sum(EffectiveCost) by SubAccountId;\nlet count = toscalar(all | order by sum_EffectiveCost desc | count);\nlet topX = all | order by sum_EffectiveCost desc | limit maxGroupCount;\nlet otherId = '(others)';\ncosts\n//\n// Group rows after max count\n| extend inTopX = SubAccountId in (topX)\n| extend SubAccountId = iff(inTopX, SubAccountId, otherId)\n| extend SubAccountName = iff(inTopX, SubAccountName, strcat('(', (count - maxGroupCount), ' others)'))\n//\n| summarize \n EffectiveCost = round(sum(EffectiveCost), 2),\n SubAccountName = take_any(SubAccountName)\n by\n ChargePeriodStart = startofmonth(ChargePeriodStart),\n SubAccountId\n| project ChargePeriodStart, EffectiveCost, Sub = iff(SubAccountId == otherId, SubAccountName, strcat(SubAccountName, ' (', split(SubAccountId, '/')[2], ')'))\n| order by EffectiveCost desc\n| render columnchart", + "id": "21a87abe-19e2-44ec-8298-ba872e66c162", + "usedVariables": [ + "CostsByMonth", + "maxGroupCount" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet data = (\n CostsByMonth\n | summarize \n BilledCost = sum(BilledCost),\n EffectiveCost = sum(EffectiveCost),\n ContractedCost = sum(ContractedCost),\n ListCost = sum(ListCost),\n SubAccountName = take_any(SubAccountName)\n by\n ChargePeriodStart,\n SubAccountId\n | as per\n | union (\n per\n | summarize \n BilledCost = sum(BilledCost),\n EffectiveCost = sum(EffectiveCost),\n ContractedCost = sum(ContractedCost),\n ListCost = sum(ListCost),\n SubAccountName = take_any(SubAccountName)\n by\n SubAccountId\n )\n | order by ChargePeriodStart asc\n | extend BilledCost = todouble(round(BilledCost, 2))\n | extend EffectiveCost = todouble(round(EffectiveCost, 2))\n | extend ContractedCost = todouble(round(ContractedCost, 2))\n | extend ListCost = todouble(round(ListCost, 2))\n | extend CommitmentDiscountSavings = todouble(round(ContractedCost - EffectiveCost, 2))\n | extend NegotiatedDiscountSavings = todouble(round(ListCost - ContractedCost, 2))\n | extend ChargePeriod = iff(isempty(ChargePeriodStart), strcat('Total'), strcat(format_datetime(ChargePeriodStart, 'yyyy-MM - '), monthname[monthofyear(ChargePeriodStart)]))\n);\ndata | evaluate pivot(ChargePeriod, sum(EffectiveCost), SubAccountName)\n| order by Total desc", + "id": "90502e9a-2d0d-4ae4-8d9d-cc21f9b72d5d", + "usedVariables": [ + "CostsByMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet data = (\n CostsByMonth\n | extend x_ResourceGroupId = strcat(SubAccountId, '/resourcegroups/', x_ResourceGroupName)\n | summarize \n EffectiveCost = sum(EffectiveCost),\n SubAccountName = take_any(SubAccountName),\n x_ResourceGroupName = take_any(x_ResourceGroupName)\n by\n ChargePeriodStart,\n x_ResourceGroupId\n | as per\n | union (\n per\n | summarize \n EffectiveCost = sum(EffectiveCost),\n SubAccountName = take_any(SubAccountName),\n x_ResourceGroupName = take_any(x_ResourceGroupName)\n by\n x_ResourceGroupId\n )\n | order by ChargePeriodStart asc\n | extend EffectiveCost = todouble(round(EffectiveCost, 2))\n | extend ChargePeriod = iff(isempty(ChargePeriodStart), strcat('Total'), strcat(format_datetime(ChargePeriodStart, 'yyyy-MM - '), monthname[monthofyear(ChargePeriodStart)]))\n);\ndata | evaluate pivot(ChargePeriod, sum(EffectiveCost), x_ResourceGroupName, SubAccountName)\n| order by Total desc", + "id": "c2c65ec0-e57d-4834-8a6b-b5975afeb9a0", + "usedVariables": [ + "CostsByMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let costs = CostsByMonth | extend x_ResourceGroupId = strcat(SubAccountId, '/resourcegroups/', x_ResourceGroupName);\nlet all = costs | where isnotempty(x_ResourceGroupId) | summarize sum(EffectiveCost) by x_ResourceGroupId;\nlet count = toscalar(all | order by sum_EffectiveCost desc | count);\nlet topX = all | order by sum_EffectiveCost desc | limit maxGroupCount;\nlet otherId = '(others)';\ncosts\n//\n// Group rows after max count\n| extend inTopX = x_ResourceGroupId in (topX)\n| extend x_ResourceGroupId = iff(inTopX, x_ResourceGroupId, otherId)\n| extend x_ResourceGroupName = iff(inTopX, x_ResourceGroupName, strcat('(', (count - maxGroupCount), ' others)'))\n//\n| summarize \n EffectiveCost = round(sum(EffectiveCost), 2),\n SubAccountName = take_any(SubAccountName),\n x_ResourceGroupName = take_any(x_ResourceGroupName)\n by\n ChargePeriodStart,\n x_ResourceGroupId\n| project ChargePeriodStart, EffectiveCost, RG = iff(x_ResourceGroupId == otherId, x_ResourceGroupName, strcat(x_ResourceGroupName, ' (', SubAccountName, ')'))\n| order by EffectiveCost desc\n| render columnchart", + "id": "61b26784-6a73-4e80-85b2-9c5cfbd2dd06", + "usedVariables": [ + "CostsByMonth", + "maxGroupCount" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let costs = CostsByMonth;\nlet all = costs | where isnotempty(ResourceId) | summarize sum(EffectiveCost) by ResourceId;\nlet count = toscalar(all | order by sum_EffectiveCost desc | count);\nlet topX = all | order by sum_EffectiveCost desc | limit maxGroupCount;\nlet otherId = '(others)';\ncosts\n//\n// Group rows after max count\n| extend inTopX = ResourceId in (topX)\n| extend ResourceId = iff(inTopX, ResourceId, otherId)\n| extend ResourceName = iff(inTopX, ResourceName, strcat('(', (count - maxGroupCount), ' others)'))\n//\n| summarize \n EffectiveCost = round(sum(EffectiveCost), 2),\n ResourceType = take_any(ResourceType),\n ResourceName = take_any(ResourceName)\n by\n ChargePeriodStart,\n ResourceId\n| project ChargePeriodStart, EffectiveCost, RG = iff(ResourceId == otherId, ResourceName, strcat(ResourceName, ' (', ResourceType, ')'))\n| order by EffectiveCost desc\n| render columnchart", + "id": "3924981c-23e4-464d-a872-045df1752750", + "usedVariables": [ + "CostsByMonth", + "maxGroupCount" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet data = (\n CostsByMonth\n | summarize \n EffectiveCost = sum(EffectiveCost),\n ResourceName = take_any(ResourceName),\n ResourceType = take_any(ResourceType),\n RegionName = take_any(RegionName),\n x_ResourceGroupName = take_any(x_ResourceGroupName),\n SubAccountName = take_any(SubAccountName)\n by\n ChargePeriodStart,\n ResourceId\n | as per\n | union (\n per\n | summarize \n EffectiveCost = sum(EffectiveCost)\n by\n ResourceId,\n ResourceName,\n ResourceType,\n RegionName,\n x_ResourceGroupName,\n SubAccountName\n )\n | order by ChargePeriodStart asc\n | extend EffectiveCost = todouble(round(EffectiveCost, 2))\n | extend ChargePeriod = iff(isempty(ChargePeriodStart), strcat('Total'), strcat(format_datetime(ChargePeriodStart, 'yyyy-MM - '), monthname[monthofyear(ChargePeriodStart)]))\n);\ndata | evaluate pivot(ChargePeriod, sum(EffectiveCost), ResourceName, ResourceType, RegionName, x_ResourceGroupName, SubAccountName)\n| order by Total desc", + "id": "c7be613e-deb4-4779-ad75-4445e8d5e01f", + "usedVariables": [ + "CostsByMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let costs = CostsByDay;\nlet all = costs | summarize sum(EffectiveCost) by ResourceId;\nlet count = toscalar(all | order by sum_EffectiveCost desc | count);\nlet topX = all | order by sum_EffectiveCost desc | limit maxGroupCount;\nlet otherId = '(others)';\ncosts\n//\n// Group rows after max count\n| extend inTopX = ResourceId in (topX)\n| extend ResourceId = iff(inTopX, ResourceId, otherId)\n| extend ResourceName = iff(inTopX, ResourceName, strcat('(', (count - maxGroupCount), ' others)'))\n//\n| summarize \n EffectiveCost = round(sum(EffectiveCost), 2),\n ResourceType = take_any(ResourceType),\n ResourceName = take_any(ResourceName)\n by\n ChargePeriodStart,\n ResourceId\n| project ChargePeriodStart, EffectiveCost, RG = iff(ResourceId == otherId, ResourceName, strcat(ResourceName, ' (', ResourceType, ')'))\n| order by EffectiveCost desc\n| render columnchart", + "id": "49e24ee0-91de-4b1c-973f-036a3c060aca", + "usedVariables": [ + "CostsByDay", + "maxGroupCount" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let costs = CostsByMonth;\nlet all = costs | where isnotempty(ServiceCategory) | summarize sum(EffectiveCost) by ServiceCategory;\nlet count = toscalar(all | order by sum_EffectiveCost desc | count);\nlet topX = all | order by sum_EffectiveCost desc | limit maxGroupCount;\nlet otherId = '(others)';\ncosts\n//\n// Group rows after max count\n| extend inTopX = ServiceCategory in (topX)\n| extend ServiceCategory = iff(inTopX, ServiceCategory, otherId)\n//\n| summarize \n EffectiveCost = round(sum(EffectiveCost), 2)\n by\n ChargePeriodStart,\n ServiceCategory\n| project ChargePeriodStart, EffectiveCost, Category = ServiceCategory\n| order by EffectiveCost desc\n| render columnchart", + "id": "4c7a7614-9b8c-415b-a4e1-d2d53c023d31", + "usedVariables": [ + "CostsByMonth", + "maxGroupCount" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet data = (\n CostsByMonth\n | summarize \n EffectiveCost = sum(EffectiveCost)\n by\n ChargePeriodStart,\n ServiceCategory,\n ServiceName\n | as per\n | union (\n per\n | summarize \n EffectiveCost = sum(EffectiveCost)\n by\n ServiceCategory,\n ServiceName\n )\n | order by ChargePeriodStart asc\n | extend EffectiveCost = todouble(round(EffectiveCost, 2))\n | extend ChargePeriod = iff(isempty(ChargePeriodStart), strcat('Total'), strcat(format_datetime(ChargePeriodStart, 'yyyy-MM - '), monthname[monthofyear(ChargePeriodStart)]))\n);\ndata | evaluate pivot(ChargePeriod, sum(EffectiveCost), ServiceCategory)\n| order by Total desc", + "id": "1d273b55-d2ea-427c-8a5f-01f6c240e98a", + "usedVariables": [ + "CostsByMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let costs = CostsByMonth;\nlet all = costs | where isnotempty(ResourceType) | summarize sum(EffectiveCost) by ResourceType;\nlet count = toscalar(all | order by sum_EffectiveCost desc | count);\nlet topX = all | order by sum_EffectiveCost desc | limit maxGroupCount;\nlet otherId = '(others)';\ncosts\n//\n// Group rows after max count\n| extend inTopX = ResourceType in (topX)\n| extend ResourceType = iff(inTopX, ResourceType, strcat('(', (count - maxGroupCount), ' others)'))\n//\n| summarize \n EffectiveCost = round(sum(EffectiveCost), 2)\n by\n ChargePeriodStart,\n ResourceType\n| project ChargePeriodStart, EffectiveCost, Type = ResourceType\n| order by EffectiveCost desc\n| render columnchart", + "id": "90e30901-a931-4a1d-b81e-0a1821032c3c", + "usedVariables": [ + "CostsByMonth", + "maxGroupCount" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let costs = CostsByDay;\nlet all = costs | where isnotempty(ResourceType) | summarize sum(EffectiveCost) by ResourceType;\nlet count = toscalar(all | order by sum_EffectiveCost desc | count);\nlet topX = all | order by sum_EffectiveCost desc | limit maxGroupCount;\nlet otherId = '(others)';\ncosts\n| where isnotempty(ResourceType)\n//\n// Group rows after max count\n| extend inTopX = ResourceType in (topX)\n| extend ResourceType = iff(inTopX, ResourceType, strcat('(', (count - maxGroupCount), ' others)'))\n//\n| summarize \n EffectiveCost = round(sum(EffectiveCost), 2)\n by\n ChargePeriodStart = startofday(ChargePeriodStart),\n ResourceType\n| project ChargePeriodStart, EffectiveCost, Type = ResourceType\n| order by EffectiveCost desc\n| render columnchart", + "id": "ebe41e27-e9f9-478e-ab90-fd1f87906766", + "usedVariables": [ + "CostsByDay", + "maxGroupCount" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet data = (\n CostsByMonth\n | summarize \n EffectiveCost = sum(EffectiveCost),\n ResourceName = take_any(ResourceName),\n ResourceType = take_any(ResourceType),\n RegionName = take_any(RegionName),\n x_ResourceGroupName = take_any(x_ResourceGroupName),\n SubAccountName = take_any(SubAccountName)\n by\n ChargePeriodStart,\n ResourceId\n | as per\n | union (\n per\n | summarize \n EffectiveCost = sum(EffectiveCost)\n by\n ResourceId,\n ResourceName,\n ResourceType,\n RegionName,\n x_ResourceGroupName,\n SubAccountName\n )\n | order by ChargePeriodStart asc\n | extend EffectiveCost = todouble(round(EffectiveCost, 2))\n | extend ChargePeriod = iff(isempty(ChargePeriodStart), strcat('Total'), strcat(format_datetime(ChargePeriodStart, 'yyyy-MM - '), monthname[monthofyear(ChargePeriodStart)]))\n);\ndata | evaluate pivot(ChargePeriod, sum(EffectiveCost), ResourceName, ResourceType, RegionName, x_ResourceGroupName, SubAccountName)\n| order by Total desc", + "id": "7f7b08a9-ae15-46f8-8b0f-767280375add", + "usedVariables": [ + "CostsByMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let costs = CostsByDay | where isnotempty(ResourceType);\nlet all = costs | summarize sum(EffectiveCost) by ResourceType;\nlet count = toscalar(all | order by sum_EffectiveCost desc | count);\nlet topX = all | order by sum_EffectiveCost desc | limit maxGroupCount;\nlet otherId = '(others)';\ncosts\n| where isnotempty(ResourceType)\n//\n// Group rows after max count\n| extend inTopX = ResourceType in (topX)\n| extend ResourceType = iff(inTopX, ResourceType, strcat('(', (count - maxGroupCount), ' others)'))\n//\n| summarize \n EffectiveCost = round(sum(EffectiveCost), 2)\n by\n ResourceType\n| order by EffectiveCost desc\n| render columnchart", + "id": "a35b4c05-8ecb-4c51-9aaf-980af3d7923e", + "usedVariables": [ + "CostsByDay", + "maxGroupCount" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let costs = CostsByDay | where isnotempty(ResourceType);\nlet all = costs | summarize ResourceCount = dcount(ResourceId) by ResourceType;\nlet count = toscalar(all | order by ResourceCount desc | count);\nlet topX = all | order by ResourceCount desc | limit maxGroupCount;\nlet otherId = '(others)';\ncosts\n| where isnotempty(ResourceType)\n//\n// Group rows after max count\n| extend inTopX = ResourceType in (topX)\n| extend ResourceType = iff(inTopX, ResourceType, strcat('(', (count - maxGroupCount), ' others)'))\n//\n| summarize \n ResourceCount = dcount(ResourceId)\n by\n ResourceType\n| order by ResourceCount desc\n| render columnchart", + "id": "9c5d3cf0-b6cb-45a2-acfd-b19df425bddf", + "usedVariables": [ + "CostsByDay", + "maxGroupCount" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nCostsByDay\n| where isnotempty(ResourceType)\n| summarize \n Count = dcount(ResourceId),\n EffectiveCost = sum(EffectiveCost),\n ListCost = sum(ListCost),\n ContractedCost = sum(ContractedCost)\n by\n ResourceType\n| order by Count desc\n| project \n Type = ResourceType,\n Count,\n Cost = round(EffectiveCost, 2),\n // NegotiatedSavings = round(ListCost - ContractedCost, 2),\n // CommitmentSavings = round(ContractedCost - EffectiveCost, 2),\n Savings = round(ListCost - EffectiveCost, 2),\n [\"Cost / Resource\"] = round(EffectiveCost / Count, 2)\n", + "id": "2d7b6447-2769-40b8-958a-f252dab68b1e", + "usedVariables": [ + "CostsByDay" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let costs = CostsByDay\n| where isnotempty(ResourceType);\ncosts | summarize Value = tostring(dcount(ResourceId)) by Label = \"Resources\", Order = 10\n| union (costs | summarize Value = tostring(dcount(ResourceType)) by Label = \"Resource types\", Order = 11)\n| union (costs | summarize Count = dcount(ResourceId) by Value = ResourceType | order by Count desc | limit 1 | extend Label = \"Most used\", Order = 21)\n| union (costs | summarize sum(EffectiveCost) by Value = ResourceType | order by sum_EffectiveCost desc | limit 1 | extend Label = \"Most cost\", Order = 22)\n| union (costs | summarize AllTypes = dcount(ResourceType), CommittedTypes = dcountif(ResourceType, isnotempty(CommitmentDiscountType)) by Label = \"Covered by commitment discounts\", Order = 31 | extend Value = strcat(round(1.0 * CommittedTypes / AllTypes * 100, 1), '%'))\n| union (costs | summarize Savings = sum(ListCost - EffectiveCost) by Value = ResourceType | order by Savings desc | limit 1 | extend Label = \"Most savings\", Order = 32)\n| union (costs | summarize CostPerResource = sum(EffectiveCost) / dcount(ResourceId) by Value = ResourceType | order by CostPerResource desc | limit 1 | extend Label = \"Most expensive (cost / resource)\", Order = 41)\n| union (costs | where ResourceType !in (CostsPlus | where ChargePeriodStart < ago(numberOfDays * 1d) - 1d | distinct ResourceType) | summarize Count = dcount(ResourceId) by ResourceType | order by Count desc | as d | count | extend Label = \"New in last n days\", Value = case(Count == 0, '(none)', Count == 1, toscalar(d | project ResourceType), strcat(toscalar(d | limit 1 | project ResourceType), ' and ', (Count - 1), ' more')), Order = 42)\n| order by Order asc\n", + "id": "f062533b-8c94-412f-bf83-0eb5cd06063d", + "usedVariables": [ + "CostsByDay", + "CostsPlus", + "numberOfDays" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "CostsByDay\n| summarize EffectiveCost = sum(EffectiveCost) by ChargePeriodStart = startofday(ChargePeriodStart)\n| order by ChargePeriodStart asc\n| extend PreviousEffectiveCost = prev(EffectiveCost)\n| project ChargePeriodStart, EffectiveCost, Change = iif(isempty(PreviousEffectiveCost), todouble(0), todouble((EffectiveCost - PreviousEffectiveCost) / PreviousEffectiveCost)) * 100\n", + "id": "1d9c166d-22b6-48fd-9a90-9f983083ecc7", + "usedVariables": [ + "CostsByDay" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "CostsByDay\n| where isnotempty(x_SkuDescription)\n//\n// Only include SKUs with effective prices\n| where x_EffectiveUnitPrice != 0 and isnotempty(x_EffectiveUnitPrice)\n//\n// Only include SKUs with negotiated discounts and not commitment discounts\n| where ListUnitPrice > x_EffectiveUnitPrice\n| where isempty(CommitmentDiscountStatus)\n//\n| summarize \n x_ResourceCount = dcount(ResourceId),\n EffectiveCost = round(sum(EffectiveCost), 2),\n BilledCost = round(sum(BilledCost), 2),\n ListUnitPrice = round(take_any(ListUnitPrice), 4),\n ContractedUnitPrice = round(take_any(ContractedUnitPrice), 4),\n x_EffectiveUnitPrice = round(take_any(x_EffectiveUnitPrice), 4),\n PricingQuantity = sum(PricingQuantity)\n by\n x_SkuDescription,\n PricingUnit,\n CommitmentDiscountType,\n x_SkuTerm\n| order by EffectiveCost desc\n| project \n x_SkuDescription,\n // CommitmentDiscountType,\n // x_SkuTerm,\n Quantity = round(PricingQuantity, 4),\n Unit = PricingUnit,\n // Resources = x_ResourceCount,\n List = ListUnitPrice,\n // ContractedUnitPrice,\n // x_EffectiveUnitPrice,\n Discount = round(ListUnitPrice - x_EffectiveUnitPrice, 4),\n // BilledCost,\n Cost = EffectiveCost\n| where Discount > 0\n", + "id": "af6424d4-8f73-4b81-bfdd-f0287ebcaaca", + "usedVariables": [ + "CostsByDay" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "CostsByDay\n| where isnotempty(x_SkuDescription)\n//\n// Only include SKUs with effective prices\n| where x_EffectiveUnitPrice != 0 and isnotempty(x_EffectiveUnitPrice)\n//\n// Only include SKUs with commitment discounts\n| where isnotempty(CommitmentDiscountStatus)\n//\n| summarize \n x_ResourceCount = dcount(ResourceId),\n EffectiveCost = round(sum(EffectiveCost), 2),\n BilledCost = round(sum(BilledCost), 2),\n ListUnitPrice = round(take_any(ListUnitPrice), 4),\n ContractedUnitPrice = round(take_any(ContractedUnitPrice), 4),\n x_EffectiveUnitPrice = round(take_any(x_EffectiveUnitPrice), 4),\n PricingQuantity = sum(PricingQuantity)\n by\n x_SkuDescription,\n PricingUnit,\n CommitmentDiscountType,\n x_SkuTerm\n| order by EffectiveCost desc\n| project \n x_SkuDescription,\n CommitmentDiscountType,\n Term = case(\n isempty(x_SkuTerm) or x_SkuTerm <= 0, '',\n x_SkuTerm < 12, strcat(x_SkuTerm, ' month', iff(x_SkuTerm != 1, 's', '')),\n strcat(x_SkuTerm / 12, ' year', iff(x_SkuTerm != 12, 's', ''))\n ),\n Quantity = round(PricingQuantity, 4),\n Unit = PricingUnit,\n // Resources = x_ResourceCount,\n List = ListUnitPrice,\n // ContractedUnitPrice,\n // x_EffectiveUnitPrice,\n Discount = round(ListUnitPrice - x_EffectiveUnitPrice, 4),\n // BilledCost,\n Cost = EffectiveCost\n//| where Discount > 0\n", + "id": "c4f6542d-a9cb-4284-bd4e-b9f94ad02192", + "usedVariables": [ + "CostsByDay" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "CostsByDay\n| where isnotempty(x_SkuDescription)\n//\n// Only include SKUs with effective prices\n| where x_EffectiveUnitPrice != 0 and isnotempty(x_EffectiveUnitPrice)\n//\n// Only include SKUs without any discounts\n| where ListUnitPrice == x_EffectiveUnitPrice\n| where isempty(CommitmentDiscountStatus)\n//\n| summarize \n x_ResourceCount = dcount(ResourceId),\n EffectiveCost = round(sum(EffectiveCost), 2),\n BilledCost = round(sum(BilledCost), 2),\n ListUnitPrice = round(take_any(ListUnitPrice), 4),\n ContractedUnitPrice = round(take_any(ContractedUnitPrice), 4),\n x_EffectiveUnitPrice = round(take_any(x_EffectiveUnitPrice), 4),\n PricingQuantity = sum(PricingQuantity)\n by\n x_SkuDescription,\n PricingUnit,\n CommitmentDiscountType,\n x_SkuTerm\n| order by EffectiveCost desc\n| project \n x_SkuDescription,\n // CommitmentDiscountType,\n // x_SkuTerm,\n Quantity = round(PricingQuantity, 4),\n Unit = PricingUnit,\n // Resources = x_ResourceCount,\n List = ListUnitPrice,\n // ContractedUnitPrice,\n // x_EffectiveUnitPrice,\n Discount = round(ListUnitPrice - x_EffectiveUnitPrice, 4),\n // BilledCost,\n Cost = EffectiveCost\n| where Discount == 0\n", + "id": "00ae3917-c783-45f6-a04e-9113e4c5d445", + "usedVariables": [ + "CostsByDay" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let doubleGroupCount = maxGroupCount * 2;\nlet costs = CostsByDay\n| where isnotempty(x_SkuDescription)\n;\nlet all = costs | summarize EffectiveCost = round(sum(EffectiveCost), 2) by x_SkuDescription;\nlet count = toscalar(all | count);\nlet topX = all | order by EffectiveCost desc | limit doubleGroupCount;\ncosts\n//\n// Group rows after max count\n| extend inTopX = x_SkuDescription in (topX)\n| extend x_SkuDescription = iff(inTopX, x_SkuDescription, strcat('(', (count - doubleGroupCount), ' others)'))\n//\n| summarize EffectiveCost = round(sum(EffectiveCost), 2) by SKU = x_SkuDescription\n| order by EffectiveCost desc\n", + "id": "d8a4634e-7ebe-47ce-83b5-ff7f50f6bee6", + "usedVariables": [ + "CostsByDay", + "maxGroupCount" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nCostsByMonth\n| where ChargeCategory == 'Purchase'\n| project\n ChargePeriodStart = substring(ChargePeriodStart, 0, 10),\n x_SkuDescription,\n CommitmentDiscountType,\n Term = case(isempty(x_SkuTerm) or x_SkuTerm <= 0, '', x_SkuTerm < 12, strcat(x_SkuTerm, ' month', iff(x_SkuTerm != 1, 's', '')), strcat(x_SkuTerm / 12, ' year', iff(x_SkuTerm != 12, 's', ''))),\n PricingQuantity,\n BilledCost,\n BillingCurrency\n| order by ChargePeriodStart desc\n", + "id": "98c9b9b5-bebf-41f8-8319-5fa2523f9dd0", + "usedVariables": [ + "CostsByMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nCostsByMonth\n| as costs\n| where ChargeCategory == 'Purchase'\n| where isnotempty(CommitmentDiscountType)\n| project\n ChargePeriodStart = substring(ChargePeriodStart, 0, 10),\n x_SkuDescription,\n CommitmentDiscountType,\n Term = case(isempty(x_SkuTerm) or x_SkuTerm <= 0, '', x_SkuTerm < 12, strcat(x_SkuTerm, ' month', iff(x_SkuTerm != 1, 's', '')), strcat(x_SkuTerm / 12, ' year', iff(x_SkuTerm != 12, 's', ''))),\n ChargeFrequency,\n ChargeClass,\n PricingQuantity,\n Utilization = real(0),\n BilledCost,\n x_SkuOrderId\n| join kind=leftouter (\n costs\n | where ChargeCategory == 'Usage'\n | where isnotempty(CommitmentDiscountId)\n //\n | extend x_CommitmentDiscountUtilizationPotential = case(\n ProviderName == 'Microsoft', EffectiveCost,\n CommitmentDiscountCategory == 'Usage', ConsumedQuantity,\n CommitmentDiscountCategory == 'Spend', EffectiveCost,\n real(0)\n )\n | extend x_CommitmentDiscountUtilizationAmount = iff(CommitmentDiscountStatus == 'Used', x_CommitmentDiscountUtilizationPotential, real(0))\n //\n | summarize\n x_CommitmentDiscountUtilizationAmount = sum(x_CommitmentDiscountUtilizationAmount),\n x_CommitmentDiscountUtilizationPotential = sum(x_CommitmentDiscountUtilizationPotential)\n by\n x_SkuOrderId\n | project\n x_SkuOrderId,\n Utilization = round(x_CommitmentDiscountUtilizationAmount / x_CommitmentDiscountUtilizationPotential * 100, 1)\n) on x_SkuOrderId\n| extend Utilization = coalesce(Utilization1, real(0))\n| project-away x_SkuOrderId, x_SkuOrderId1, Utilization1\n| order by ChargePeriodStart desc\n\n\n", + "id": "35a3a2b3-4ab0-4449-96f7-c78db789089e", + "usedVariables": [ + "CostsByMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let data = materialize(\n CostsByMonth | extend Period = 'Last n months'\n | union (CostsByDay | extend Period = 'Last n days')\n | summarize \n ListCost = round(sum(ListCost), 2),\n ContractedCost = round(sum(ContractedCost), 2),\n EffectiveCost = round(sum(EffectiveCost), 2)\n by\n Period\n | project Period, json = todynamic(strcat('[{ \"Label\":\"List\", \"Value\":', ListCost, ' }, { \"Label\":\"Contracted\", \"Value\":', ContractedCost, ' }, { \"Label\":\"Effective\", \"Value\":', EffectiveCost, ' }]'))\n | mv-expand json\n | project Label = tostring(json.Label), Value = tolong(json.Value), Period\n);\ndata", + "id": "5ff29428-de83-4a2c-8f86-d8beebe68750", + "usedVariables": [ + "CostsByDay", + "CostsByMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let data = materialize(\n CostsByMonth | extend Period = 'Last n months'\n | union (CostsByDay | extend Period = 'Last n days')\n | summarize\n ListCost = round(sum(ListCost), 2),\n ContractedCost = round(sum(ContractedCost), 2),\n EffectiveCost = round(sum(EffectiveCost), 2)\n by\n Period\n | project Period, json = todynamic(strcat('[{ \"Label\":\"Total\", \"Value\":', ListCost - EffectiveCost, ' }, { \"Label\":\"Negotiated\", \"Value\":', ListCost - ContractedCost, ' }, { \"Label\":\"Commitment\", \"Value\":', ContractedCost - EffectiveCost, ' }]'))\n | mv-expand json\n | project Label = tostring(json.Label), Value = tolong(json.Value), Period\n);\ndata\n", + "id": "f5f240a8-a818-4f27-bdfb-e96fcfe433bd", + "usedVariables": [ + "CostsByDay", + "CostsByMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nCostsByDay\n| where ChargeCategory == 'Usage'\n| where isnotempty(CommitmentDiscountId)\n//\n| extend x_CommitmentDiscountUtilizationPotential = case(\n ProviderName == 'Microsoft', EffectiveCost,\n CommitmentDiscountCategory == 'Usage', ConsumedQuantity,\n CommitmentDiscountCategory == 'Spend', EffectiveCost,\n real(0)\n)\n| extend x_CommitmentDiscountUtilizationAmount = iff(CommitmentDiscountStatus == 'Used', x_CommitmentDiscountUtilizationPotential, real(0))\n//\n| summarize\n CommitmentDiscountName = take_any(CommitmentDiscountName),\n CommitmentDiscountType = take_any(CommitmentDiscountType),\n x_SkuTerm = take_any(x_SkuTerm),\n ListCost = sum(ListCost),\n ContractedCost = sum(ContractedCost),\n EffectiveCost = sum(EffectiveCost),\n x_CommitmentDiscountUtilizationAmount = sum(x_CommitmentDiscountUtilizationAmount),\n x_CommitmentDiscountUtilizationPotential = sum(x_CommitmentDiscountUtilizationPotential)\n by\n CommitmentDiscountId\n| order by EffectiveCost desc\n| project\n CommitmentDiscountName,\n CommitmentDiscountType,\n Term = case(isempty(x_SkuTerm) or x_SkuTerm <= 0, '', x_SkuTerm < 12, strcat(x_SkuTerm, ' month', iff(x_SkuTerm != 1, 's', '')), strcat(x_SkuTerm / 12, ' year', iff(x_SkuTerm != 12, 's', ''))),\n Utilization = round(x_CommitmentDiscountUtilizationAmount / x_CommitmentDiscountUtilizationPotential * 100, 1),\n Cost = round(EffectiveCost, 2),\n Savings = round(ListCost - EffectiveCost, 2)\n", + "id": "d7f31381-ba44-46a1-ad3e-dc6b8826706d", + "usedVariables": [ + "CostsByDay" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let data = materialize(\n CostsByMonth\n //\n // Don't double-count commitment discount purchases\n | where x_AmortizationClass != 'Principal'\n //\n | summarize \n ListCost = sum(ListCost),\n ContractedCost = sum(ContractedCost),\n EffectiveCost = sum(EffectiveCost)\n | extend CommitmentDiscountSavings = ContractedCost - EffectiveCost\n | extend NegotiatedDiscountSavings = ListCost - ContractedCost\n | extend TotalSavings = ListCost - EffectiveCost\n | project json = todynamic(strcat('[',\n '{ \"order\":11, \"type\":\"List\", \"label\":\"Cost without discounts\", \"value\":\"', numberstring(round(ListCost, 2)), '\" },',\n '{ \"order\":12, \"type\":\"\", \"label\":\"\", \"value\":\"\u2796\" },',\n '{ \"order\":13, \"type\":\"Contracted\", \"label\":\"After negotiated discounts\", \"value\":\"', numberstring(round(ContractedCost, 2)), '\" },',\n '{ \"order\":14, \"type\":\"\", \"label\":\"\", \"value\":\"\ud83d\udff0\" },',\n '{ \"order\":15, \"type\":\"PartialSavings\", \"label\":\"Negotiated savings\", \"value\":\"', numberstring(round(NegotiatedDiscountSavings, 2)), '\" },',\n //\n '{ \"order\":21, \"type\":\"Contracted\", \"label\":\"After negotiated discounts\", \"value\":\"', numberstring(round(ContractedCost, 2)), '\" },',\n '{ \"order\":22, \"type\":\"\", \"label\":\"\", \"value\":\"\u2796\" },',\n '{ \"order\":23, \"type\":\"Effective\", \"label\":\"After commitment discounts\", \"value\":\"', numberstring(round(EffectiveCost, 2)), '\" },',\n '{ \"order\":24, \"type\":\"\", \"label\":\"\", \"value\":\"\ud83d\udff0\" },',\n '{ \"order\":25, \"type\":\"PartialSavings\", \"label\":\"Commitment savings\", \"value\":\"', numberstring(round(CommitmentDiscountSavings, 2)), '\" },',\n //\n '{ \"order\":31, \"type\":\"List\", \"label\":\"Cost without discounts\", \"value\":\"', numberstring(round(ListCost, 2)), '\" },',\n '{ \"order\":32, \"type\":\"\", \"label\":\"\", \"value\":\"\u2796\" },',\n '{ \"order\":33, \"type\":\"Effective\", \"label\":\"After commitment discounts\", \"value\":\"', numberstring(round(EffectiveCost, 2)), '\" },',\n '{ \"order\":34, \"type\":\"\", \"label\":\"\", \"value\":\"\ud83d\udff0\" },',\n '{ \"order\":35, \"type\":\"TotalSavings\", \"label\":\"Total savings\", \"value\":\"', numberstring(round(TotalSavings, 2)), '\" }',\n ']'))\n | mv-expand json\n | order by toint(json.order) asc\n | project Label = tostring(json.label), Value = tostring(json.value), Type = tostring(json.type)\n);\ndata", + "id": "e17346d1-0227-4aa4-8765-f9b4bf43bed5", + "usedVariables": [ + "CostsByMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let data = materialize(\n CostsByDay\n //\n // Don't double-count commitment discount purchases\n | where ChargeCategory == 'Usage' or isempty(CommitmentDiscountId)\n //\n | summarize \n ListCost = sum(ListCost),\n ContractedCost = sum(ContractedCost),\n EffectiveCost = sum(EffectiveCost)\n | extend CommitmentDiscountSavings = ContractedCost - EffectiveCost\n | extend NegotiatedDiscountSavings = ListCost - ContractedCost\n | extend TotalSavings = ListCost - EffectiveCost\n | project json = todynamic(strcat('[',\n '{ \"order\":11, \"type\":\"List\", \"label\":\"Cost without discounts\", \"value\":\"', numberstring(round(ListCost, 2)), '\" },',\n '{ \"order\":12, \"type\":\"\", \"label\":\"\", \"value\":\"\u2796\" },',\n '{ \"order\":13, \"type\":\"Contracted\", \"label\":\"After negotiated discounts\", \"value\":\"', numberstring(round(ContractedCost, 2)), '\" },',\n '{ \"order\":14, \"type\":\"\", \"label\":\"\", \"value\":\"\ud83d\udff0\" },',\n '{ \"order\":15, \"type\":\"PartialSavings\", \"label\":\"Negotiated savings\", \"value\":\"', numberstring(round(NegotiatedDiscountSavings, 2)), '\" },',\n //\n '{ \"order\":21, \"type\":\"Contracted\", \"label\":\"After negotiated discounts\", \"value\":\"', numberstring(round(ContractedCost, 2)), '\" },',\n '{ \"order\":22, \"type\":\"\", \"label\":\"\", \"value\":\"\u2796\" },',\n '{ \"order\":23, \"type\":\"Effective\", \"label\":\"After commitment discounts\", \"value\":\"', numberstring(round(EffectiveCost, 2)), '\" },',\n '{ \"order\":24, \"type\":\"\", \"label\":\"\", \"value\":\"\ud83d\udff0\" },',\n '{ \"order\":25, \"type\":\"PartialSavings\", \"label\":\"Commitment savings\", \"value\":\"', numberstring(round(CommitmentDiscountSavings, 2)), '\" },',\n //\n '{ \"order\":31, \"type\":\"List\", \"label\":\"Cost without discounts\", \"value\":\"', numberstring(round(ListCost, 2)), '\" },',\n '{ \"order\":32, \"type\":\"\", \"label\":\"\", \"value\":\"\u2796\" },',\n '{ \"order\":33, \"type\":\"Effective\", \"label\":\"After commitment discounts\", \"value\":\"', numberstring(round(EffectiveCost, 2)), '\" },',\n '{ \"order\":34, \"type\":\"\", \"label\":\"\", \"value\":\"\ud83d\udff0\" },',\n '{ \"order\":35, \"type\":\"TotalSavings\", \"label\":\"Total savings\", \"value\":\"', numberstring(round(TotalSavings, 2)), '\" }',\n ']'))\n | mv-expand json\n | order by toint(json.order) asc\n | project Label = tostring(json.label), Value = tostring(json.value), Type = tostring(json.type)\n);\ndata", + "id": "f1ef29df-7a1d-4dd0-8619-0c0164707b31", + "usedVariables": [ + "CostsByDay" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let data = materialize(\n CostsByMonth\n //\n | where isnotempty(CommitmentDiscountStatus)\n //\n // Guarantee there's a row for every combination\n | union (\n print json = dynamic([\n {\"order\": 11, \"CommitmentDiscountType\": \"Reservation\", \"CommitmentDiscountStatus\": \"Used\"},\n {\"order\": 12, \"CommitmentDiscountType\": \"Reservation\", \"CommitmentDiscountStatus\": \"Unused\"},\n {\"order\": 21, \"CommitmentDiscountType\": \"Savings Plan\", \"CommitmentDiscountStatus\": \"Used\"},\n {\"order\": 22, \"CommitmentDiscountType\": \"Savings Plan\", \"CommitmentDiscountStatus\": \"Unused\"}\n ])\n | mv-expand json\n | evaluate bag_unpack(json)\n | extend EffectiveCost = toreal(0)\n )\n //\n | summarize Value = sum(EffectiveCost), order = sum(order) by CommitmentDiscountStatus, CommitmentDiscountType\n | order by order asc\n | project Label = strcat(CommitmentDiscountStatus, ' ', tolower(CommitmentDiscountType), 's'), Value\n);\ndata", + "id": "7d0c2c1f-338b-4534-b173-36b284779131", + "usedVariables": [ + "CostsByMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let data = materialize(\n CostsByDay\n //\n // Don't double-count commitment discount purchases\n | where isnotempty(CommitmentDiscountStatus)\n //\n // Guarantee there's a row for every combination\n | union (\n print json = dynamic([\n {\"order\": 11, \"CommitmentDiscountType\": \"Reservation\", \"CommitmentDiscountStatus\": \"Used\"},\n {\"order\": 12, \"CommitmentDiscountType\": \"Reservation\", \"CommitmentDiscountStatus\": \"Unused\"},\n {\"order\": 21, \"CommitmentDiscountType\": \"Savings Plan\", \"CommitmentDiscountStatus\": \"Used\"},\n {\"order\": 22, \"CommitmentDiscountType\": \"Savings Plan\", \"CommitmentDiscountStatus\": \"Unused\"}\n ])\n | mv-expand json\n | evaluate bag_unpack(json)\n | extend EffectiveCost = toreal(0)\n )\n //\n | summarize Value = sum(EffectiveCost), order = sum(order) by CommitmentDiscountStatus, CommitmentDiscountType\n | order by order asc\n | project Label = strcat(CommitmentDiscountStatus, ' ', tolower(CommitmentDiscountType), 's'), Value\n);\ndata", + "id": "84ecad69-79ac-45b9-a8af-60be28dcc748", + "usedVariables": [ + "CostsByDay" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet data = (\n CostsByMonth\n | summarize \n EffectiveCost = sum(EffectiveCost)\n by\n ChargePeriodStart = startofmonth(ChargePeriodStart),\n ServiceCategory,\n ServiceName\n | as per\n | union (\n per\n | summarize \n EffectiveCost = sum(EffectiveCost)\n by\n ServiceCategory,\n ServiceName\n )\n | order by ChargePeriodStart asc\n | extend EffectiveCost = todouble(round(EffectiveCost, 2))\n | extend ChargePeriod = iff(isempty(ChargePeriodStart), strcat('Total'), strcat(format_datetime(ChargePeriodStart, 'yyyy-MM - '), monthname[monthofyear(ChargePeriodStart)]))\n);\ndata | evaluate pivot(ChargePeriod, sum(EffectiveCost), ServiceCategory)\n| order by Total desc", + "id": "85320475-730b-411f-98d5-fbc345bf5e68", + "usedVariables": [ + "CostsByMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "CostsByMonth\n//\n// Filter out commitment discount purchases\n| where ChargeCategory == 'Usage' or isempty(CommitmentDiscountId)\n//\n| summarize \n ListCost = sum(ListCost),\n ContractedCost = sum(ContractedCost),\n EffectiveCost = sum(EffectiveCost)\n by\n ChargePeriodStart = startofmonth(ChargePeriodStart),\n CommitmentDiscountType\n| project\n ChargePeriodStart,\n Savings = round(ListCost - EffectiveCost, 2),\n Type = case(\n isnotempty(CommitmentDiscountType), CommitmentDiscountType,\n 'Negotiated'\n )", + "id": "5bbb5369-ac95-45fc-853c-a6a2ce6a9e7b", + "usedVariables": [ + "CostsByMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "CostsByMonth\n//\n// Filter out commitment discount purchases\n| where ChargeCategory == 'Usage' or isempty(CommitmentDiscountId)\n//\n| summarize \n Savings = round(sum(ListCost - EffectiveCost), 2)\n by\n Type = case(\n isnotempty(CommitmentDiscountType), CommitmentDiscountType,\n 'Negotiated'\n )", + "id": "93f0eb9f-fa4d-4c54-96c2-8c1d3387d13a", + "usedVariables": [ + "CostsByMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "CostsByDay\n//\n// Filter out commitment discount purchases\n| where ChargeCategory == 'Usage' or isempty(CommitmentDiscountId)\n//\n| summarize \n ListCost = sum(ListCost),\n ContractedCost = sum(ContractedCost),\n EffectiveCost = sum(EffectiveCost)\n by\n ChargePeriodStart = startofday(ChargePeriodStart),\n CommitmentDiscountType\n| project\n ChargePeriodStart,\n Savings = round(ListCost - EffectiveCost, 2),\n Type = case(\n isnotempty(CommitmentDiscountType), CommitmentDiscountType,\n 'Negotiated'\n )", + "id": "5c442903-65b0-4b53-9e7c-3ea9c1af7be5", + "usedVariables": [ + "CostsByDay" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "CostsByDay\n//\n// Filter out commitment discount purchases\n| where ChargeCategory == 'Usage' or isempty(CommitmentDiscountId)\n//\n| summarize \n Savings = round(sum(ListCost - EffectiveCost), 2)\n by\n Type = case(\n isnotempty(CommitmentDiscountType), CommitmentDiscountType,\n 'Negotiated'\n )", + "id": "9a9ee4b0-a37d-475f-bf1d-f51573243491", + "usedVariables": [ + "CostsByDay" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet data = (\n CostsByMonth\n | where ChargeCategory == 'Usage'\n | where isnotempty(CommitmentDiscountId)\n | where EffectiveCost != 0\n | extend x_ResourceGroupId = strcat(SubAccountId, '/resourcegroups/', x_ResourceGroupName)\n | summarize \n EffectiveCost = sum(EffectiveCost),\n SubAccountName = take_anyif(SubAccountName, isnotempty(SubAccountId)),\n x_ResourceGroupName = take_any(x_ResourceGroupName),\n CommitmentDiscountName = take_any(CommitmentDiscountName),\n CommitmentDiscountType = take_any(CommitmentDiscountType)\n by\n ChargePeriodStart,\n SubAccountId,\n x_ResourceGroupId,\n CommitmentDiscountId,\n CommitmentDiscountStatus\n | as per\n | union (\n per\n | summarize \n EffectiveCost = sum(EffectiveCost),\n SubAccountName = take_anyif(SubAccountName, isnotempty(SubAccountId)),\n x_ResourceGroupName = take_any(x_ResourceGroupName),\n CommitmentDiscountName = take_any(CommitmentDiscountName),\n CommitmentDiscountType = take_any(CommitmentDiscountType)\n by\n x_ResourceGroupId,\n CommitmentDiscountId,\n CommitmentDiscountStatus\n )\n | order by ChargePeriodStart asc\n | extend EffectiveCost = todouble(round(EffectiveCost, 2))\n | extend ChargePeriod = iff(isempty(ChargePeriodStart), strcat('Total'), strcat(format_datetime(ChargePeriodStart, 'yyyy-MM - '), monthname[monthofyear(ChargePeriodStart)]))\n | extend x_ResourceGroupName = iff(isempty(x_ResourceGroupName) and CommitmentDiscountStatus == 'Unused', '(Unused)', x_ResourceGroupName)\n | extend SubAccountName = iff(isempty(SubAccountName) and CommitmentDiscountStatus == 'Unused', '(Unused)', SubAccountName)\n);\npercent((\n data | evaluate pivot(ChargePeriod, sum(EffectiveCost), x_ResourceGroupName, SubAccountName, CommitmentDiscountName, CommitmentDiscountType)\n | extend Count = tolong(Total * 1000)\n))\n| project-away Count\n| order by x_ResourceGroupName asc, SubAccountName asc, Total desc", + "id": "377e3693-0738-45d7-97d9-4b6e71ec5b36", + "usedVariables": [ + "CostsByMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet data = (\n CostsByMonth\n | where ChargeCategory == 'Usage'\n | where isnotempty(CommitmentDiscountId)\n | where EffectiveCost != 0\n | summarize \n BilledCost = sum(BilledCost),\n EffectiveCost = sum(EffectiveCost),\n ContractedCost = sum(ContractedCost),\n ListCost = sum(ListCost),\n SubAccountName = take_anyif(SubAccountName, isnotempty(SubAccountId)),\n CommitmentDiscountName = take_any(CommitmentDiscountName),\n CommitmentDiscountType = take_any(CommitmentDiscountType)\n by\n ChargePeriodStart,\n SubAccountId,\n CommitmentDiscountId,\n CommitmentDiscountStatus\n | as per\n | union (\n per\n | summarize \n BilledCost = sum(BilledCost),\n EffectiveCost = sum(EffectiveCost),\n ContractedCost = sum(ContractedCost),\n ListCost = sum(ListCost),\n SubAccountName = take_anyif(SubAccountName, isnotempty(SubAccountId)),\n CommitmentDiscountName = take_any(CommitmentDiscountName),\n CommitmentDiscountType = take_any(CommitmentDiscountType)\n by\n SubAccountId,\n CommitmentDiscountId,\n CommitmentDiscountStatus\n )\n | order by ChargePeriodStart asc\n | extend BilledCost = todouble(round(BilledCost, 2))\n | extend EffectiveCost = todouble(round(EffectiveCost, 2))\n | extend ContractedCost = todouble(round(ContractedCost, 2))\n | extend ListCost = todouble(round(ListCost, 2))\n | extend CommitmentDiscountSavings = todouble(round(ContractedCost - EffectiveCost, 2))\n | extend NegotiatedDiscountSavings = todouble(round(ListCost - ContractedCost, 2))\n | extend ChargePeriod = iff(isempty(ChargePeriodStart), strcat('Total'), strcat(format_datetime(ChargePeriodStart, 'yyyy-MM - '), monthname[monthofyear(ChargePeriodStart)]))\n | extend SubAccountName = iff(isempty(SubAccountName) and CommitmentDiscountStatus == 'Unused', '(Unused)', SubAccountName)\n);\npercent((\n data | evaluate pivot(ChargePeriod, sum(EffectiveCost), SubAccountName, CommitmentDiscountName, CommitmentDiscountType)\n | extend Count = tolong(Total * 1000)\n))\n| project-away Count\n| order by SubAccountName asc, Total desc", + "id": "13cad52d-91e7-4ab4-aad5-4aae21c1019a", + "usedVariables": [ + "CostsByMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "CostsByDay\n| where ChargeCategory == 'Usage'\n| where isnotempty(CommitmentDiscountId)\n| where EffectiveCost != 0\n| summarize \n EffectiveCost = sum(EffectiveCost),\n ResourceName = take_any(ResourceName),\n ResourceType = take_any(ResourceType),\n RegionName = take_any(RegionName),\n x_ResourceGroupName = take_any(x_ResourceGroupName),\n SubAccountName = take_any(SubAccountName),\n CommitmentDiscountName = take_any(CommitmentDiscountName),\n CommitmentDiscountType = take_any(CommitmentDiscountType)\n by\n ResourceId,\n CommitmentDiscountId\n| project \n CommitmentDiscountType,\n CommitmentDiscountName,\n ResourceName,\n ResourceType,\n RegionName,\n x_ResourceGroupName,\n SubAccountName,\n EffectiveCost = round(EffectiveCost, 2)\n| order by CommitmentDiscountType asc, CommitmentDiscountName asc, ResourceName asc, ResourceType asc, x_ResourceGroupName asc, SubAccountName asc, EffectiveCost desc", + "id": "dcf69b47-233e-4bc4-b914-3f6f09f4cb82", + "usedVariables": [ + "CostsByDay" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let costs = CostsByMonth\n| extend x_ToolkitTool = tostring(Tags['ftk-tool'])\n| where x_ToolkitTool == 'FinOps hubs'\n| extend x_ToolkitVersion = tostring(Tags['ftk-version'])\n| extend x_ResourceParentId = tostring(Tags['cm-resource-parent'])\n| extend x_ResourceParentName = database('Ingestion').parse_resourceid(x_ResourceParentId).ResourceName\n;\nlet all = costs | summarize sum(EffectiveCost) by x_ResourceParentId;\nlet count = toscalar(all | order by sum_EffectiveCost desc | count);\nlet topX = all | order by sum_EffectiveCost desc | limit maxGroupCount;\nlet otherId = '(others)';\ncosts\n//\n// Group rows after max count\n| extend inTopX = x_ResourceParentId in (topX)\n| extend x_ResourceParentId = iff(inTopX, x_ResourceParentId, otherId)\n| extend x_ResourceParentName = iff(inTopX, x_ResourceParentName, strcat('(', (count - maxGroupCount), ' others)'))\n//\n| summarize \n EffectiveCost = round(sum(EffectiveCost), 2),\n ResourceType = take_any(ResourceType),\n x_ResourceParentName = take_any(x_ResourceParentName),\n x_ResourceGroupName = take_any(x_ResourceGroupName),\n SubAccountName = take_any(SubAccountName)\n by\n ChargePeriodStart,\n x_ResourceParentId\n| project ChargePeriodStart, EffectiveCost, Hub = iff(x_ResourceParentId == otherId, x_ResourceParentName, strcat(x_ResourceParentName, ' (', x_ResourceGroupName, ' / ', SubAccountName, ')'))\n| order by EffectiveCost desc\n", + "id": "d7a9ff96-da17-4826-9fb5-b23f4c7b938d", + "usedVariables": [ + "CostsByMonth", + "maxGroupCount" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let costs = CostsByDay\n| extend x_ToolkitTool = tostring(Tags['ftk-tool'])\n| where x_ToolkitTool == 'FinOps hubs'\n| extend x_ToolkitVersion = tostring(Tags['ftk-version'])\n| extend x_ResourceParentId = tostring(Tags['cm-resource-parent'])\n| extend x_ResourceParentName = database('Ingestion').parse_resourceid(x_ResourceParentId).ResourceName\n;\nlet all = costs | summarize sum(EffectiveCost) by x_ResourceParentId;\nlet count = toscalar(all | order by sum_EffectiveCost desc | count);\nlet topX = all | order by sum_EffectiveCost desc | limit maxGroupCount;\nlet otherId = '(others)';\ncosts\n//\n// Group rows after max count\n| extend inTopX = x_ResourceParentId in (topX)\n| extend x_ResourceParentId = iff(inTopX, x_ResourceParentId, otherId)\n| extend x_ResourceParentName = iff(inTopX, x_ResourceParentName, strcat('(', (count - maxGroupCount), ' others)'))\n//\n| summarize \n EffectiveCost = round(sum(EffectiveCost), 2),\n ResourceType = take_any(ResourceType),\n x_ResourceParentName = take_any(x_ResourceParentName),\n x_ResourceGroupName = take_any(x_ResourceGroupName),\n SubAccountName = take_any(SubAccountName)\n by\n ChargePeriodStart,\n x_ResourceParentId\n| project ChargePeriodStart, EffectiveCost, Hub = iff(x_ResourceParentId == otherId, x_ResourceParentName, strcat(x_ResourceParentName, ' (', x_ResourceGroupName, ' / ', SubAccountName, ')'))\n| order by EffectiveCost desc\n", + "id": "13813a00-e634-4428-9eac-ea255fd1eaca", + "usedVariables": [ + "CostsByDay", + "maxGroupCount" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet costs = CostsByMonth\n| extend x_ToolkitTool = tostring(Tags['ftk-tool'])\n| where x_ToolkitTool == 'FinOps hubs'\n| extend x_ToolkitVersion = tostring(Tags['ftk-version'])\n| extend x_ResourceParentId = tostring(Tags['cm-resource-parent'])\n| extend x_ResourceParentName = tostring(database('Ingestion').parse_resourceid(x_ResourceParentId).ResourceName)\n;\nlet data = (\n costs\n | summarize \n EffectiveCost = sum(EffectiveCost),\n x_ResourceParentName = take_any(x_ResourceParentName),\n RegionName = take_any(RegionName),\n x_ResourceGroupName = take_any(x_ResourceGroupName),\n SubAccountName = take_any(SubAccountName),\n x_ToolkitVersion = take_any(x_ToolkitVersion)\n by\n ChargePeriodStart,\n x_ResourceParentId\n | as per\n //\n // Append total\n | union (\n per\n | summarize \n EffectiveCost = sum(EffectiveCost)\n by\n x_ResourceParentId,\n x_ResourceParentName,\n x_ToolkitVersion,\n RegionName,\n x_ResourceGroupName,\n SubAccountName\n )\n | order by ChargePeriodStart asc\n | extend EffectiveCost = todouble(round(EffectiveCost, 2))\n | extend ChargePeriod = iff(isempty(ChargePeriodStart), strcat('Total'), strcat(format_datetime(ChargePeriodStart, 'yyyy-MM - '), monthname[monthofyear(ChargePeriodStart)]))\n);\ndata | evaluate pivot(ChargePeriod, sum(EffectiveCost), x_ResourceParentName, x_ToolkitVersion, RegionName, x_ResourceGroupName, SubAccountName)\n| project-rename Name = x_ResourceParentName, Version = x_ToolkitVersion\n| order by Total desc\n", + "id": "62bbceee-c089-45db-822c-0b4745358aa4", + "usedVariables": [ + "CostsByMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": ".show cluster extents\n| where TableName contains '_final_v'\n| extend Dataset = tostring(split(TableName, '_final_v')[0])\n| extend FocusVersion = replace_string(tostring(split(TableName, '_final_v')[1]), '_', '.')\n| extend ToolkitVersion = tostring(extract(@'drop-by:ftk-version-([^\\s]+)', 1, Tags))\n// | extend Table = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 1, Tags))\n| extend Date = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 2, Tags))\n| extend Scope = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 5, Tags))\n| extend Date = todatetime(strcat(replace_string(trim(@'/', Date), '/', '-'), '-01'))\n| extend Scope = replace_regex(Scope, @'/[^/]+$', '')\n| project Dataset, FocusVersion, ToolkitVersion, Date, Scope, LastUpdate = MaxCreatedOn, OriginalSize, RowCount\n| extend ScopeId = database('Ingestion').parse_resourceid(Scope).ResourceName\n// TODO: Look up the name from cost data\n| extend ScopeResourceType = database('Ingestion').parse_resourceid(Scope).x_ResourceType\n// TODO: Clean up scope resource type display names -- | extend ScopeType = resource_type(ScopeResourceType).SingularDisplayName\n| summarize Rows = sum(RowCount) by Date, Dataset", + "id": "b98c9ed7-47b3-41c1-a596-fd9d32725b33", + "usedVariables": [] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": ".show cluster extents\n| where TableName contains '_final_v'\n| extend Dataset = tostring(split(TableName, '_final_v')[0])\n| where Dataset == 'Costs'\n| extend FocusVersion = replace_string(tostring(split(TableName, '_final_v')[1]), '_', '.')\n| extend ToolkitVersion = tostring(extract(@'drop-by:ftk-version-([^\\s]+)', 1, Tags))\n// | extend Table = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 1, Tags))\n| extend Date = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 2, Tags))\n| extend Scope = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 5, Tags))\n| extend Date = todatetime(strcat(replace_string(trim(@'/', Date), '/', '-'), '-01'))\n| extend Scope = replace_regex(Scope, @'/[^/]+$', '')\n| project Dataset, FocusVersion, ToolkitVersion, Date, Scope, LastUpdate = MaxCreatedOn, OriginalSize, RowCount\n| extend ScopeId = tostring(database('Ingestion').parse_resourceid(Scope).ResourceName)\n| extend ScopeResourceType = database('Ingestion').parse_resourceid(Scope).x_ResourceType\n// TODO: Clean up scope resource type display names -- | extend ScopeType = resource_type(ScopeResourceType).SingularDisplayName\n| summarize Rows = sum(RowCount) by Date, ScopeId", + "id": "187e22fe-3d78-45a2-a2bf-090ece144fd1", + "usedVariables": [] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": ".show cluster extents\n| where TableName contains '_final_v'\n| extend Dataset = tostring(split(TableName, '_final_v')[0])\n| where Dataset == 'Costs'\n| extend FocusVersion = replace_string(tostring(split(TableName, '_final_v')[1]), '_', '.')\n| extend ToolkitVersion = tostring(extract(@'drop-by:ftk-version-([^\\s]+)', 1, Tags))\n// | extend Table = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 1, Tags))\n| extend Date = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 2, Tags))\n| extend Scope = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 5, Tags))\n| extend Date = todatetime(strcat(replace_string(trim(@'/', Date), '/', '-'), '-01'))\n| extend Scope = replace_regex(Scope, @'/[^/]+$', '')\n| project Dataset, FocusVersion, ToolkitVersion, Date, Scope, LastUpdate = MaxCreatedOn, OriginalSize, RowCount\n| extend ScopeId = tostring(database('Ingestion').parse_resourceid(Scope).ResourceName)\n// TODO: Look up the name from cost data\n| extend ScopeResourceType = database('Ingestion').parse_resourceid(Scope).x_ResourceType\n// TODO: Clean up scope resource type display names -- | extend ScopeType = resource_type(ScopeResourceType).SingularDisplayName\n| summarize dcount(Scope)", + "id": "d2c522dc-ef4b-4714-a473-09350e275557", + "usedVariables": [] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": ".show cluster extents\n| where TableName contains '_final_v'\n| extend Dataset = tostring(split(TableName, '_final_v')[0])\n| where Dataset == 'Costs'\n| extend FocusVersion = replace_string(tostring(split(TableName, '_final_v')[1]), '_', '.')\n| extend ToolkitVersion = tostring(extract(@'drop-by:ftk-version-([^\\s]+)', 1, Tags))\n// | extend Table = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 1, Tags))\n| extend Date = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 2, Tags))\n| extend Scope = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 5, Tags))\n| extend Date = todatetime(strcat(replace_string(trim(@'/', Date), '/', '-'), '-01'))\n| extend Scope = replace_regex(Scope, @'/[^/]+$', '')\n| project Dataset, FocusVersion, ToolkitVersion, Date, Scope, LastUpdate = MaxCreatedOn, OriginalSize, RowCount\n| extend ScopeId = tostring(database('Ingestion').parse_resourceid(Scope).ResourceName)\n// TODO: Look up the name from cost data\n| extend ScopeResourceType = database('Ingestion').parse_resourceid(Scope).x_ResourceType\n// TODO: Clean up scope resource type display names -- | extend ScopeType = resource_type(ScopeResourceType).SingularDisplayName\n| summarize dcount(startofmonth(Date))", + "id": "6fe11b78-82ef-4e9a-8411-a65e5b0d8bba", + "usedVariables": [] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": ".show cluster extents\n| where TableName contains '_final_v'\n| extend Dataset = tostring(split(TableName, '_final_v')[0])\n| where Dataset == 'Prices'\n| extend FocusVersion = replace_string(tostring(split(TableName, '_final_v')[1]), '_', '.')\n| extend ToolkitVersion = tostring(extract(@'drop-by:ftk-version-([^\\s]+)', 1, Tags))\n// | extend Table = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 1, Tags))\n| extend Date = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 2, Tags))\n| extend Scope = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 5, Tags))\n| extend Date = todatetime(strcat(replace_string(trim(@'/', Date), '/', '-'), '-01'))\n| extend Scope = replace_regex(Scope, @'/[^/]+$', '')\n| project Dataset, FocusVersion, ToolkitVersion, Date, Scope, LastUpdate = MaxCreatedOn, OriginalSize, RowCount\n| extend ScopeId = tostring(database('Ingestion').parse_resourceid(Scope).ResourceName)\n// TODO: Look up the name from cost data\n| extend ScopeResourceType = database('Ingestion').parse_resourceid(Scope).x_ResourceType\n// TODO: Clean up scope resource type display names -- | extend ScopeType = resource_type(ScopeResourceType).SingularDisplayName\n| summarize Rows = sum(RowCount) by Date, ScopeId", + "id": "b3621977-59be-4f98-a034-a94479612115", + "usedVariables": [] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": ".show cluster extents\n| where TableName contains '_final_v'\n| extend Dataset = tostring(split(TableName, '_final_v')[0])\n| where Dataset == 'Recommendations'\n| extend FocusVersion = replace_string(tostring(split(TableName, '_final_v')[1]), '_', '.')\n| extend ToolkitVersion = tostring(extract(@'drop-by:ftk-version-([^\\s]+)', 1, Tags))\n// | extend Table = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 1, Tags))\n| extend Date = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 2, Tags))\n| extend Scope = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 5, Tags))\n| extend Date = todatetime(strcat(replace_string(trim(@'/', Date), '/', '-'), '-01'))\n| extend Scope = replace_regex(Scope, @'/[^/]+$', '')\n| project Dataset, FocusVersion, ToolkitVersion, Date, Scope, LastUpdate = MaxCreatedOn, OriginalSize, RowCount\n| extend ScopeId = tostring(database('Ingestion').parse_resourceid(Scope).ResourceName)\n// TODO: Look up the name from cost data\n| extend ScopeResourceType = database('Ingestion').parse_resourceid(Scope).x_ResourceType\n// TODO: Clean up scope resource type display names -- | extend ScopeType = resource_type(ScopeResourceType).SingularDisplayName\n| summarize Rows = sum(RowCount) by Date, ScopeId", + "id": "4a2a77fe-e819-4059-b027-dab0a1770a0a", + "usedVariables": [] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": ".show cluster extents\n| where TableName contains '_final_v'\n| extend Dataset = tostring(split(TableName, '_final_v')[0])\n| where Dataset == 'Transactions'\n| extend FocusVersion = replace_string(tostring(split(TableName, '_final_v')[1]), '_', '.')\n| extend ToolkitVersion = tostring(extract(@'drop-by:ftk-version-([^\\s]+)', 1, Tags))\n// | extend Table = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 1, Tags))\n| extend Date = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 2, Tags))\n| extend Scope = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 5, Tags))\n| extend Date = todatetime(strcat(replace_string(trim(@'/', Date), '/', '-'), '-01'))\n| extend Scope = replace_regex(Scope, @'/[^/]+$', '')\n| project Dataset, FocusVersion, ToolkitVersion, Date, Scope, LastUpdate = MaxCreatedOn, OriginalSize, RowCount\n| extend ScopeId = tostring(database('Ingestion').parse_resourceid(Scope).ResourceName)\n// TODO: Look up the name from cost data\n| extend ScopeResourceType = database('Ingestion').parse_resourceid(Scope).x_ResourceType\n// TODO: Clean up scope resource type display names -- | extend ScopeType = resource_type(ScopeResourceType).SingularDisplayName\n| summarize Rows = sum(RowCount) by Date, ScopeId", + "id": "8f1ad1ef-6f5e-4f20-a1c4-3941e871d0cd", + "usedVariables": [] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": ".show cluster extents\n| where TableName contains '_final_v'\n| extend Dataset = tostring(split(TableName, '_final_v')[0])\n| where Dataset == 'CommitmentDiscountUsage'\n| extend FocusVersion = replace_string(tostring(split(TableName, '_final_v')[1]), '_', '.')\n| extend ToolkitVersion = tostring(extract(@'drop-by:ftk-version-([^\\s]+)', 1, Tags))\n// | extend Table = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 1, Tags))\n| extend Date = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 2, Tags))\n| extend Scope = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 5, Tags))\n| extend Date = todatetime(strcat(replace_string(trim(@'/', Date), '/', '-'), '-01'))\n| extend Scope = replace_regex(Scope, @'/[^/]+$', '')\n| project Dataset, FocusVersion, ToolkitVersion, Date, Scope, LastUpdate = MaxCreatedOn, OriginalSize, RowCount\n| extend ScopeId = tostring(database('Ingestion').parse_resourceid(Scope).ResourceName)\n// TODO: Look up the name from cost data\n| extend ScopeResourceType = database('Ingestion').parse_resourceid(Scope).x_ResourceType\n// TODO: Clean up scope resource type display names -- | extend ScopeType = resource_type(ScopeResourceType).SingularDisplayName\n| summarize Rows = sum(RowCount) by Date, ScopeId", + "id": "e9a9a135-6a74-4463-b69f-eac6930ff1f2", + "usedVariables": [] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let costs = CostsByDayAHB\n| where isnotempty(x_SkuLicenseStatus)\n//\n// Get the latest resource record first to guarrantee we have the latest status\n| summarize arg_max(ChargePeriodStart, *) by ResourceId\n| summarize\n x_ResourceCount = dcount(ResourceId),\n x_SkuLicenseUnusedQuantity = sum(x_SkuLicenseUnusedQuantity)\n by\n x_SkuLicenseStatus,\n x_SkuLicenseQuantity,\n x_SkuCoreCount\n| union (\n print json = dynamic([\n { \"x_SkuLicenseStatus\": \"Enabled\" },\n { \"x_SkuLicenseStatus\": \"Not Enabled\" }\n ])\n | mv-expand json\n | evaluate bag_unpack(json)\n);\n//\n// Coverage first\ncosts\n| summarize covered = sumif(x_ResourceCount, x_SkuLicenseStatus == 'Enabled'), all = sum(x_ResourceCount)\n| project Order = 1, Label = 'Coverage %', Value = percentstring(covered, all, 1)\n//\n// First column\n| union (costs | where x_SkuLicenseStatus == 'Enabled' | summarize Value = numberstring(sum(x_ResourceCount)) by Order = 3, Label = 'Covered resources')\n| union (costs | where x_SkuLicenseStatus == 'Not Enabled' | summarize Value = numberstring(sum(x_ResourceCount)) by Order = 5, Label = 'Eligible resources')\n//\n// Second column\n| union (costs | where x_SkuLicenseStatus == 'Enabled' | summarize Value = numberstring(sum(x_SkuLicenseUnusedQuantity)) by Order = 2, Label = 'Underutilized vCPU capacity')\n| union (costs | where x_SkuLicenseStatus == 'Enabled' | summarize Value = numberstring(sum(x_SkuLicenseQuantity)) by Order = 4, Label = 'Covered vCPU capacity')\n| union (costs | where x_SkuLicenseStatus == 'Not Enabled' | summarize Value = numberstring(sum(x_SkuLicenseQuantity)) by Order = 6, Label = 'Eligible vCPU capacity')\n| order by Order asc\n", + "id": "1a400b6c-9204-434d-88ff-2dc5260a95bf", + "usedVariables": [ + "CostsByDayAHB" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "// Underutilized capacity\n// Fully utilized capacity\n// Eligible resources\nCostsByDayAHB\n| where x_SkuLicenseStatus == 'Enabled'\n//\n// Get the latest resource record first to guarrantee we have the latest status\n| summarize\n arg_max(ChargePeriodStart, *),\n TotalConsumedQuantity = sum(ConsumedQuantity),\n TotalEffectiveCost = sum(EffectiveCost)\n by\n ResourceId\n| project \n [\"License type\"] = x_SkuLicenseType,\n ResourceName,\n ResourceType,\n SKU = x_SkuInstanceType,\n [\"SKU cores\"] = x_SkuCoreCount,\n [\"Required capacity\"] = x_SkuLicenseQuantity,\n [\"Unused capacity\"] = x_SkuLicenseUnusedQuantity,\n [\"Unused vCore hours\"] = x_SkuLicenseUnusedQuantity * ConsumedQuantity,\n EffectiveCost = TotalEffectiveCost,\n x_ResourceGroupName,\n SubAccountName\n| order by [\"Unused vCore hours\"] desc\n", + "id": "c2ad5370-f4df-4a13-ac31-2c095f88754d", + "usedVariables": [ + "CostsByDayAHB" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "CostsByDayAHB\n| where x_SkuLicenseStatus == 'Enabled'\n| where x_SkuLicenseUnusedQuantity == 0\n//\n// Get the latest resource record first to guarrantee we have the latest status\n| summarize\n arg_max(ChargePeriodStart, *),\n TotalConsumedQuantity = sum(ConsumedQuantity),\n TotalEffectiveCost = sum(EffectiveCost)\n by\n ResourceId\n| project \n [\"License type\"] = x_SkuLicenseType,\n ResourceName,\n ResourceType,\n SKU = x_SkuInstanceType,\n [\"SKU cores\"] = x_SkuCoreCount,\n [\"vCore hours\"] = x_SkuLicenseQuantity * ConsumedQuantity,\n EffectiveCost = TotalEffectiveCost,\n x_ResourceGroupName,\n SubAccountName\n| order by [\"vCore hours\"] desc\n", + "id": "4614acfb-2288-4a71-97f7-32845a9ddcb2", + "usedVariables": [ + "CostsByDayAHB" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "CostsByDayAHB\n| where x_SkuLicenseStatus == 'Not Enabled'\n//\n// Get the latest resource record first to guarrantee we have the latest status\n| summarize\n arg_max(ChargePeriodStart, *),\n TotalConsumedQuantity = sum(ConsumedQuantity),\n TotalEffectiveCost = sum(EffectiveCost)\n by\n ResourceId\n| project \n [\"License type\"] = x_SkuLicenseType,\n ResourceName,\n ResourceType,\n SKU = x_SkuInstanceType,\n [\"SKU cores\"] = x_SkuCoreCount,\n [\"Required capacity\"] = x_SkuLicenseQuantity,\n [\"Eligible vCore hours\"] = x_SkuLicenseQuantity * ConsumedQuantity,\n EffectiveCost = TotalEffectiveCost,\n x_ResourceGroupName,\n SubAccountName\n| order by [\"Eligible vCore hours\"] desc\n", + "id": "99d44f1f-c53a-40ee-b749-8e5557ec58c7", + "usedVariables": [ + "CostsByDayAHB" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let costs = CostsPlus | where ChargePeriodStart >= startofmonth(now(), -3) and ChargePeriodStart < startofday(ago(-1d));\nlet startOfPeriod = toscalar(costs | summarize min(startofday(ChargePeriodStart)));\nlet endOfPeriod = toscalar(costs | summarize max(startofday(ChargePeriodStart)));\nlet forecastHorizon = numberOfDays * 1d;\ncosts\n| make-series\n EffectiveCost = sum(EffectiveCost)\n on ChargePeriodStart\n from startOfPeriod to endOfPeriod + forecastHorizon step 1d\n // by SubAccountId\n| extend Forecast = series_decompose_forecast(EffectiveCost, numberOfDays)\n", + "id": "5fd3ea2a-8882-4438-9103-fa3945240604", + "usedVariables": [ + "CostsPlus", + "numberOfDays" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let data = materialize(\n CostsByMonth\n //\n // Don't double-count commitment discount purchases\n | where x_AmortizationClass != 'Principal'\n //\n | summarize \n ListCost = sum(ListCost),\n ContractedCost = sum(ContractedCost),\n EffectiveCost = sum(EffectiveCost)\n | extend TotalSavings = ListCost - EffectiveCost\n | extend EffectiveSavingsRate = TotalSavings / ListCost\n | project json = todynamic(strcat('[',\n '{ \"order\":11, \"type\":\"TotalSavings\", \"label\":\"Total savings\", \"value\":\"', numberstring(round(TotalSavings, 2)), '\" },',\n '{ \"order\":12, \"type\":\"\", \"label\":\"\", \"value\":\"\u2797\" },',\n '{ \"order\":13, \"type\":\"List\", \"label\":\"Cost without discounts\", \"value\":\"', numberstring(round(ListCost, 2)), '\" },',\n '{ \"order\":14, \"type\":\"\", \"label\":\"\", \"value\":\"\ud83d\udff0\" },',\n '{ \"order\":15, \"type\":\"EffectiveSavingsRate\", \"label\":\"Effective savings rate\", \"value\":\"', percentstring(EffectiveSavingsRate), '\" }',\n ']'))\n | mv-expand json\n | order by toint(json.order) asc\n | project Label = tostring(json.label), Value = tostring(json.value), Type = tostring(json.type)\n);\ndata", + "id": "e1bc2d51-44af-4dd9-8b0d-a71088b551f5", + "usedVariables": [ + "CostsByMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "CostsByMonth\n| extend x_AmortizationClass = case(\n ChargeCategory == 'Purchase' and isnotempty(CommitmentDiscountCategory), 'Principal',\n isnotempty(CommitmentDiscountCategory), 'Amortized Charge',\n ''\n)\n| extend x_CommitmentDiscountSavings = iff(ContractedCost == 0, real(0), ContractedCost - EffectiveCost)\n| extend x_NegotiatedDiscountSavings = iff(ListCost == 0, real(0), ListCost - ContractedCost)\n| extend x_TotalSavings = iff(ListCost == 0, real(0), ListCost - EffectiveCost)\n| summarize\n ['List cost'] = round(sumif(ListCost, x_AmortizationClass != 'Principal'), 2),\n ['Effective cost'] = round(sum(EffectiveCost), 2),\n Savings = round(sum(x_TotalSavings), 2)\n by\n Account = x_BillingProfileId,\n Month = substring(startofmonth(ChargePeriodStart), 0, 7)\n| extend ESR = percentstring(Savings/ ['List cost'])\n| order by Month desc", + "id": "3b3f0a58-2d84-4e3e-bebc-3e747a7d5ede", + "usedVariables": [ + "CostsByMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let data = materialize(\n CostsByDay\n //\n // Don't double-count commitment discount purchases\n | where x_AmortizationClass != 'Principal'\n //\n | summarize \n ListCost = sum(ListCost),\n ContractedCost = sum(ContractedCost),\n EffectiveCost = sum(EffectiveCost)\n | extend TotalSavings = ListCost - EffectiveCost\n | extend EffectiveSavingsRate = TotalSavings / ListCost\n | project json = todynamic(strcat('[',\n '{ \"order\":11, \"type\":\"TotalSavings\", \"label\":\"Total savings\", \"value\":\"', numberstring(round(TotalSavings, 2)), '\" },',\n '{ \"order\":12, \"type\":\"\", \"label\":\"\", \"value\":\"\u2797\" },',\n '{ \"order\":13, \"type\":\"List\", \"label\":\"Cost without discounts\", \"value\":\"', numberstring(round(ListCost, 2)), '\" },',\n '{ \"order\":14, \"type\":\"\", \"label\":\"\", \"value\":\"\ud83d\udff0\" },',\n '{ \"order\":15, \"type\":\"EffectiveSavingsRate\", \"label\":\"Effective savings rate\", \"value\":\"', percentstring(EffectiveSavingsRate), '\" }',\n ']'))\n | mv-expand json\n | order by toint(json.order) asc\n | project Label = tostring(json.label), Value = tostring(json.value), Type = tostring(json.type)\n);\ndata", + "id": "cb34f22b-9370-460c-9658-3e73d220bbc7", + "usedVariables": [ + "CostsByDay" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "database('Ingestion').HubSettings\n| extend temp = todynamic(strcat('[',\n '{\"Label\":\"Version\",\"Value\":', version, '},',\n '{\"Label\":\"Managed scopes\",\"Value\":', array_length(scopes), '},',\n '{\"Label\":\"Data retention\",\"Value\":\"', toint(retention.final.months), 'mo\"}',\n']'))\n| mvexpand temp\n| project Label = tostring(temp.Label), Value = tostring(temp.Value)\n", + "id": "8de47213-8327-44da-9d1b-8ba5de74c44a", + "usedVariables": [] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "CostsByMonth\n| extend x_AmortizationClass = case(\n ChargeCategory == 'Purchase' and isnotempty(CommitmentDiscountCategory), 'Principal',\n isnotempty(CommitmentDiscountCategory), 'Amortized Charge',\n ''\n)\n| extend x_CommitmentDiscountSavings = iff(ContractedCost == 0, real(0), ContractedCost - EffectiveCost)\n| extend x_NegotiatedDiscountSavings = iff(ListCost == 0, real(0), ListCost - ContractedCost)\n| extend x_TotalSavings = iff(ListCost == 0, real(0), ListCost - EffectiveCost)\n| summarize\n ['On-demand'] = round(sumif(EffectiveCost, PricingCategory == 'Standard'), 2),\n Spot = round(sumif(EffectiveCost, PricingCategory == 'Dynamic'), 2),\n Reservation = round(sumif(EffectiveCost, CommitmentDiscountType == 'Reservation'), 2),\n ['Savings plan'] = round(sumif(EffectiveCost, CommitmentDiscountType == 'Savings Plan'), 2),\n ['Other'] = round(sumif(EffectiveCost, PricingCategory !in ('Standard', 'Dynamic') and isempty(CommitmentDiscountType)), 2)\n by\n Account = x_BillingProfileId,\n Month = substring(startofmonth(ChargePeriodStart), 0, 7)\n| order by Month desc", + "id": "0341b3e4-eccc-4924-9555-9835b128c543", + "usedVariables": [ + "CostsByMonth" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "Costs\n| where isempty(selectedBillingCurrency) or BillingCurrency == selectedBillingCurrency\n| extend EffectiveOverContracted = iff(ContractedCost < EffectiveCost, ContractedCost - EffectiveCost, real(0))\n| extend ContractedOverList = iff(ListCost < ContractedCost, ListCost - ContractedCost, real(0))\n| extend EffectiveOverList = iff(ListCost < EffectiveCost, ListCost - EffectiveCost, real(0))\n| extend Scenario = case(\n ListCost == 0 and CommitmentDiscountCategory == 'Usage' and ChargeCategory == 'Usage', 'Reservation usage missing list',\n ListCost == 0 and CommitmentDiscountCategory == 'Usage' and ChargeCategory == 'Purchase', 'Reservation purchase missing list',\n ListCost == 0 and CommitmentDiscountCategory == 'Spend' and ChargeCategory == 'Usage', 'Savings plan usage missing list',\n ListCost == 0 and CommitmentDiscountCategory == 'Spend' and ChargeCategory == 'Purchase', 'Savings plan purchase missing list',\n ListCost == 0 and ChargeCategory == 'Purchase', 'Other purchase missing list',\n isnotempty(CommitmentDiscountStatus) and ContractedOverList == 0 and EffectiveOverContracted < 0, 'Commitment cost over contracted',\n ListCost == 0 and BilledCost == 0 and EffectiveCost == 0 and ContractedCost > 0 and x_SourceChanges !contains 'MissingContractedCost', 'ContractedCost should be 0',\n ListCost == 0 and ContractedCost == 0 and BilledCost > 0 and EffectiveCost > 0 and x_PublisherCategory == 'Vendor' and ChargeCategory == 'Usage', 'Marketplace usage missing list/contracted',\n ContractedOverList < 0 and EffectiveOverContracted == 0 and x_SourceChanges !contains 'MissingListCost', 'ListCost too low',\n ContractedUnitPrice == x_EffectiveUnitPrice and EffectiveOverContracted < 0 and x_SourceChanges !contains 'MissingContractedCost', 'ContractedCost doesn\\'t match price',\n EffectiveOverContracted != 0 and abs(EffectiveOverContracted) < 0.00000001, 'Rounding error',\n ContractedOverList != 0 and abs(ContractedOverList) < 0.00000001, 'Rounding error',\n EffectiveOverList != 0 and abs(EffectiveOverList) < 0.00000001, 'Rounding error',\n ContractedCost < EffectiveCost or ListCost < ContractedCost or ListCost < EffectiveCost, '',\n EffectiveCost <= ContractedCost and ContractedCost <= ListCost, 'Good',\n '')\n| project-reorder ListCost, ContractedCost, BilledCost, EffectiveCost, EffectiveOverList, EffectiveOverContracted, ContractedOverList, x_SourceChanges, ListUnitPrice, ContractedUnitPrice, x_BilledUnitPrice, x_EffectiveUnitPrice, CommitmentDiscountStatus, PricingQuantity, PricingUnit, x_PricingBlockSize, x_PricingUnitDescription\n// DEBUG -- | where isempty(scenario) | limit 1000\n// Summarize -- \n| summarize Rows = count(), EffectiveCost = round(sum(EffectiveCost), 2), EffectiveOverContracted = abs(sum(EffectiveOverContracted)), ContractedOverList = abs(sum(ContractedOverList)), EffectiveOverList = abs(sum(EffectiveOverList)), Agreement = arraystring(make_set(x_BillingAccountAgreement)) by Scenario | order by Rows desc\n", + "id": "56fd8707-bbb3-4f7e-8e37-8dd90ada3baa", + "usedVariables": [ + "selectedBillingCurrency" + ] + }, + { + "id": "43612ae4-c475-4f22-bb50-ce9d995abb8f", + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "CostsPlus\n| where startofmonth(BillingPeriodStart) == startofmonth(now())", + "usedVariables": [ + "CostsPlus" + ] + }, + { + "id": "cb1f5404-c0b1-42fd-99fb-3cff7b08daaa", + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "CostsPlus\n| where startofmonth(BillingPeriodStart) == startofmonth(startofmonth(now()) - 1d)", + "usedVariables": [ + "CostsPlus" + ] + }, + { + "id": "4ce0f587-2d45-436c-8f79-102c6b382439", + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "CostsPlus\r\n| where startofmonth(ChargePeriodStart) >= startofmonth(now(), -numberOfMonths)\r\n| extend ChargePeriodStart = startofmonth(ChargePeriodStart)\r\n| extend BillingPeriodStart = startofmonth(BillingPeriodStart)", + "usedVariables": [ + "CostsPlus", + "numberOfMonths" + ] + }, + { + "id": "6b598467-8c31-4693-b1eb-7ed683fcfc3a", + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "CostsPlus\r\n| where ChargePeriodStart >= ago(numberOfDays * 1d) - 1d and ChargePeriodStart < ago(1d)\r\n| extend ChargePeriodStart = startofday(ChargePeriodStart)", + "usedVariables": [ + "CostsPlus", + "numberOfDays" + ] + }, + { + "id": "4a1973bf-08e9-4e82-b8e6-6edff81cf0a5", + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "CostsPlus", + "usedVariables": [ + "CostsPlus" + ] + }, + { + "id": "eb9259cc-05b7-4441-a66d-a29026fe371b", + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "Costs_v1_2\n//\n// Apply summarization settings\n| where ChargePeriodStart >= monthsago(numberOfMonths)\n| where isempty(selectedBillingCurrency) or BillingCurrency == selectedBillingCurrency\n| as filteredCosts\n| extend x_ChargeMonth = startofmonth(ChargePeriodStart)\n// TODO: Should we add granularity? -- | extend x_ReportingDate = iff(#\"Default Granularity\" == 'Monthly'), x_ChargeMonth, startofday(ChargePeriodStart))\n//\n// SKU details\n| extend x_SkuUsageType = tostring(coalesce(SkuPriceDetails.x_UsageType, x_SkuDetails.UsageType))\n| extend x_SkuLicenseUnusedQuantity = x_SkuLicenseQuantity - x_SkuCoreCount\n//\n// Commitment discounts\n| extend x_CommitmentDiscountKey = iff(isempty(x_SkuInstanceType), '', strcat(x_SkuInstanceType, x_SkuMeterId))\n| extend x_SkuTermLabel = case(isempty(x_SkuTerm) or x_SkuTerm <= 0, '', x_SkuTerm < 12, strcat(x_SkuTerm, ' month', iff(x_SkuTerm != 1, 's', '')), strcat(x_SkuTerm / 12, ' year', iff(x_SkuTerm != 12, 's', '')))\n//\n// CSP partners\n// x_PartnerBilledCredit = iff(x_PartnerCreditApplied, BilledCost * x_PartnerCreditRate, todouble(0))\n// x_PartnerEffectiveCredit = iff(x_PartnerCreditApplied, EffectiveCost * x_PartnerCreditRate, todouble(0))\n//\n// Toolkit\n| extend x_ToolkitTool = tostring(Tags['ftk-tool'])\n| extend x_ToolkitVersion = tostring(Tags['ftk-version'])\n| extend tmp_ResourceParent = database('Ingestion').parse_resourceid(Tags['cm-resource-parent'])\n| extend x_ResourceParentId = tostring(tmp_ResourceParent.ResourceId)\n| extend x_ResourceParentName = tostring(tmp_ResourceParent.ResourceName)\n| extend x_ResourceParentType = tostring(tmp_ResourceParent.ResourceType)\n//\n// TODO: Only add differentiators when the name is not unique\n| extend CommitmentDiscountNameUnique = iff(isempty(CommitmentDiscountId), '', strcat(CommitmentDiscountName, ' (', CommitmentDiscountType, ')'))\n| extend ResourceNameUnique = iff(isempty(ResourceId), '', strcat(ResourceName, ' (', ResourceType, ')'))\n| extend x_ResourceGroupNameUnique = iff(isempty(x_ResourceGroupName), '', strcat(x_ResourceGroupName, ' (', SubAccountName, ')'))\n| extend SubAccountNameUnique = iff(isempty(SubAccountId), '', strcat(SubAccountName, ' (', split(SubAccountId, '/')[3], ')'))\n//\n// Explain why cost is 0\n| extend x_FreeReason = case(\n BilledCost != 0.0 or EffectiveCost != 0.0, '',\n PricingCategory == 'Committed', strcat('Unknown ', CommitmentDiscountStatus, ' Commitment'),\n x_BilledUnitPrice == 0.0 and x_EffectiveUnitPrice == 0.0 and ContractedUnitPrice == 0.0 and ListUnitPrice == 0.0 and isempty(CommitmentDiscountType), case(\n x_SkuDescription contains 'Trial', 'Trial',\n x_SkuDescription contains 'Preview', 'Preview',\n 'Other'\n ),\n x_BilledUnitPrice > 0.0 or x_EffectiveUnitPrice > 0.0, case(\n PricingQuantity > 0.0, 'Low Usage',\n PricingQuantity == 0.0, 'No Usage',\n 'Unknown Negative Quantity'\n ),\n 'Unknown'\n)\n//\n| extend x_ResourceTop1K = ChargeCategory != 'Usage' or isempty(ResourceId) or ResourceId in (\n filteredCosts\n | where isnotempty(ResourceId) and ChargeCategory == 'Usage'\n | summarize sum(EffectiveCost) by ResourceId\n | order by sum_EffectiveCost desc\n | limit 1000\n | distinct ResourceId\n)\n//\n| project-away tmp_ResourceParent", + "usedVariables": [ + "numberOfMonths", + "selectedBillingCurrency" + ] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let months = toscalar(database('Ingestion').HubSettings | project toint(retention.final.months));\nlet monthname = dynamic(['', 'Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']);\nrange Value from toint(1) to iff(isempty(months), 24, months) step 1\n| order by Value desc\n| extend MonthsAgo = monthsago(Value)\n| extend Label = strcat(Value, ' mo (', monthname[monthofyear(MonthsAgo)], format_datetime(MonthsAgo, ' yyyy'), ')')\n| project-away MonthsAgo\n", + "id": "c9039243-968d-4e75-9899-8d4ab51a9896", + "usedVariables": [] + }, + { + "dataSource": { + "kind": "inline", + "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" + }, + "text": "let months = toscalar(database('Ingestion').HubSettings | project toint(retention.final.months));\nlet currencies = Costs_v1_2\n | where ChargePeriodStart >= monthsago(iff(isempty(months), 24, months))\n | where isnotempty(BillingCurrency)\n | distinct BillingCurrency;\nlet currencyCount = toscalar(currencies | count);\ncurrencies\n| extend Label = BillingCurrency, _Order = 1\n| union (\n print BillingCurrency = '', Label = '\u26a0\ufe0f All', _Order = 0\n | where currencyCount > 1\n)\n| order by _Order asc, Label asc\n| project BillingCurrency, Label\n", + "id": "f2a8c4d6-3b5e-4a7f-9c2d-8e5b1f4a7d9c", + "usedVariables": [] + } + ] +} \ No newline at end of file diff --git a/src/templates/finops-hub-local/notes/dashboard.md b/src/templates/finops-hub-local/notes/dashboard.md new file mode 100644 index 000000000..2aa1aa6cb --- /dev/null +++ b/src/templates/finops-hub-local/notes/dashboard.md @@ -0,0 +1,133 @@ +# Using the local ADX dashboard + +This dashboard (`../dashboard.json`) is the upstream FinOps hub dashboard localized to the +ftklocal Kusto emulator. The only change from the upstream file is: + +| Field | Upstream value | Local value | +|---|---|---| +| `dataSources[0].clusterUri` | `https://..kusto.windows.net/` | `http://localhost:8082` | +| `dataSources[0].database` | `Hub` | `Hub` (unchanged) | + +All tiles and KQL queries are unmodified. The `Hub` database on the emulator exposes the same +view functions as a deployed hub: `Costs_v1_2`, `Prices_v1_2`, `Transactions_v1_2`, +`CommitmentDiscountUsage_v1_2`, and their aliases (`Costs`, `Prices`, …). + +--- + +## Prerequisites + +- ftklocal stack running: `docker compose up` from `src/templates/finops-hub-local/` +- Kusto emulator listening on `http://localhost:8082` +- Hub DB populated (run the ftklocal ingestion scripts or load sample data) + +--- + +## Proven path: query via the CLI + +The emulator's KQL REST API is plain HTTP with no authentication — you can query it directly +from the terminal today without a browser. Use `ftk query` (or `ftk run`) against the `Hub` +database: + +```bash +# Smoke-test: confirm view functions are populated +ftk query --db Hub "Costs_v1_2 | take 10" + +# Run any KQL query interactively +ftk run --db Hub +``` + +This is the correct first validation step on Mac. It does not depend on any browser +mixed-content policy and works regardless of HTTPS. + +Once the agent skill is built (see the skill roadmap), the same Hub view functions will be +available as structured tool calls through the skill surface — no manual KQL required. + +--- + +## Best-effort path: ADX web UI dashboard import + +> ⚠️ **Mixed-content limitation — this path is NOT proven on Mac.** +> +> `dataexplorer.azure.com` is served over HTTPS. Browsers (Chrome, Edge, Safari) apply strict +> mixed-content rules: an HTTPS page may not open active connections to a plain `http://` +> origin. Attempting to add `http://localhost:8082` as a cluster from the hosted web UI will +> likely be silently blocked or refused. **Do not assume this works until you verify it in +> your browser.** + +If you want to try the dashboard import anyway, follow the steps below. If the cluster +connection fails, see the [ngrok workaround](#workaround-expose-the-emulator-over-https-with-ngrok). + +### 1. Attempt to connect the ADX web UI + +1. Open **https://dataexplorer.azure.com** in your browser. +2. Click **+ Add cluster** (left sidebar). +3. Enter the connection URI: `http://localhost:8082` +4. Authentication: select **Anonymous** (the emulator runs no-auth HTTP). +5. Click **Add**. + +If the cluster appears in the sidebar with `Hub` and `Ingestion` databases, the connection +succeeded and you can proceed. If the connection hangs or errors, use the ngrok workaround +below. + +### 2. Import the dashboard + +1. In the ADX web UI, click **Dashboards** (left nav). +2. Click the **⋮ menu → Import dashboard from file**. +3. Select `src/templates/finops-hub-local/dashboard.json`. +4. After import, open **Dashboard settings → Data sources** and confirm: + - Cluster URI: `http://localhost:8082` + - Database: `Hub` +5. If the data source shows a different cluster URI, edit it to `http://localhost:8082` / + `Hub` and save. + +### 3. Validate tiles + +Open any tile (e.g., "Cost summary") and click **Refresh**. If tiles return empty results, +confirm the Hub DB has data by running `Costs_v1_2 | take 10` in the **Query** tab. + +--- + +## Workaround: expose the emulator over HTTPS with ngrok + +If the browser blocks the HTTP connection, tunnel the emulator behind HTTPS using +[ngrok](https://ngrok.com): + +```bash +# Install ngrok if needed: brew install ngrok/ngrok/ngrok +ngrok http 8082 +``` + +ngrok prints an HTTPS forwarding URL such as `https://xxxx-xxxx.ngrok-free.app`. Use that +URL as the cluster URI in the ADX web UI instead of `http://localhost:8082`. Update the +dashboard data source to match. + +> Note: ngrok free-tier URLs are ephemeral and change each session. You will need to update +> the data source in the dashboard each time. + +--- + +## What was proved vs what you must validate + +| Claim | Status | +|---|---| +| `dashboard.json` is valid JSON | ✅ Verified (Python `json.load` round-trip) | +| `dataSources[0].clusterUri` → `http://localhost:8082` | ✅ Verified in file | +| `dataSources[0].database` = `Hub` | ✅ Verified in file | +| Hub DB has `Costs_v1_2`, `Prices_v1_2`, etc. | ✅ Verified via `.show functions` on running emulator | +| No upstream cluster names, billing IDs, or absolute paths in dashboard | ✅ Verified | +| `ftk query` / `ftk run` against Hub DB | ✅ Works (HTTP REST, no browser) | +| Browser can add `http://localhost:8082` from `https://dataexplorer.azure.com` | ⚠️ **Not validated** — mixed-content blocked in most browsers | +| Dashboard tiles render data after import | ⚠️ **Not validated** — requires browser import + populated Hub data | + +--- + +## Ingestion vs Hub databases + +The emulator runs two databases: + +- **`Ingestion`** — raw, transform, and final staging tables used during ingestion. Not + intended as a query surface. +- **`Hub`** — view functions (`Costs_v1_2`, etc.) that join and normalize the ingested data. + This is the correct database for the dashboard. + +The dashboard's data source is already set to `Hub`. From bb422ad9d0ce55b22328e254b43d39ed690898da Mon Sep 17 00:00:00 2001 From: msbrett Date: Wed, 17 Jun 2026 16:38:18 -0700 Subject: [PATCH 06/51] feat(finops-hub-local): proactive backfill chunking + fail-fast double-ingest guard (T-5000.3) Add proactive per-extent chunked backfill for large final tables (default threshold 2,000,000 rows OR >4 source extents, both env-configurable via BACKFILL_CHUNK_ROW_THRESHOLD / BACKFILL_CHUNK_EXTENT_THRESHOLD), so the Prices transform never attempts a single-pass backfill that OOMs; Costs stays single-pass with the reactive chunked fallback intact. Add a fail-fast guard: re-staging a file under the same (scope,type,period,run-uuid) with a changed checksum now halts with a non-zero exit and a clear message instead of silently appending duplicate rows. Harden Invoke-KustoPost strict-mode catch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- src/templates/finops-hub-local/.env.example | 14 +++++ src/templates/finops-hub-local/Makefile | 4 +- .../finops-hub-local/scripts/ingest.ps1 | 52 +++++++++++++++++-- 3 files changed, 66 insertions(+), 4 deletions(-) diff --git a/src/templates/finops-hub-local/.env.example b/src/templates/finops-hub-local/.env.example index 366549e42..300991bb8 100644 --- a/src/templates/finops-hub-local/.env.example +++ b/src/templates/finops-hub-local/.env.example @@ -19,3 +19,17 @@ MEM_LIMIT=16g # Point to an absolute path to reuse an existing export without copying files: # EXPORT_DIR=/Users/you/src/ftklocal/export EXPORT_DIR=./export + +# Proactive chunked-backfill thresholds. +# ingest.ps1 measures raw-table row count AND extent count before each final-table backfill. +# Chunked backfill is used proactively when EITHER threshold is exceeded (avoiding OOM). +# If both are at or below their thresholds, single-pass is tried first with chunked fallback. +# +# Row threshold — default 2000000 is derived from measured headroom at MEM_LIMIT=16g, amd64-Rosetta: +# - Costs 1,350,000 rows → single-pass OK (safely below threshold) +# - Prices 12,735,587 rows → single-pass OOMs (safely above threshold) +# Extent threshold — default 4: >4 extents indicates a fragmented raw table that benefits +# from per-extent chunking regardless of total row count. +# Raise row threshold if you increase MEM_LIMIT and re-verify single-pass succeeds. +# BACKFILL_CHUNK_ROW_THRESHOLD=2000000 +# BACKFILL_CHUNK_EXTENT_THRESHOLD=4 diff --git a/src/templates/finops-hub-local/Makefile b/src/templates/finops-hub-local/Makefile index 2dd51ce4c..bb51632ed 100644 --- a/src/templates/finops-hub-local/Makefile +++ b/src/templates/finops-hub-local/Makefile @@ -57,7 +57,7 @@ help: @echo " Filter scope + period." @echo " ingest-status Show Ingest_Manifest summary." @echo " parity Run T-1.4 parity check suite." - @echo " chunked-prices-backfill Recovery path when Prices_transform OOMs." + @echo " chunked-prices-backfill Manual fallback: per-extent Prices backfill (auto-triggered by threshold)." up: $(COMPOSE) up -d @$(MAKE) --no-print-directory wait-healthy @@ -152,6 +152,8 @@ parity: # .set-or-append OOMs. Processes Prices_raw one source extent at a time # (bounded memory). Use after `make ingest` if Prices_final_v1_2 ends up # partially populated or empty. Wall-clock ~15 min on Apple Silicon. +# NOTE: ingest.ps1 now auto-chunks proactively when row count > BACKFILL_CHUNK_ROW_THRESHOLD +# (default 3,000,000). This target is the manual fallback / override path. chunked-prices-backfill: @pwsh scripts/chunked-prices-backfill.ps1 diff --git a/src/templates/finops-hub-local/scripts/ingest.ps1 b/src/templates/finops-hub-local/scripts/ingest.ps1 index 9276acaef..aa31d80ca 100755 --- a/src/templates/finops-hub-local/scripts/ingest.ps1 +++ b/src/templates/finops-hub-local/scripts/ingest.ps1 @@ -65,6 +65,17 @@ $script:RawToFinalPolicy = @{ 'Prices_raw' = [pscustomobject]@{ FinalTable = 'Prices_final_v1_2'; TransformFn = 'Prices_transform_v1_2' } } +# Proactive chunked-backfill thresholds. +# Measured baseline: MEM_LIMIT=16g, amd64-Rosetta; idle-loaded engine ~12.4 GiB working +# set (78% of 16g), ~3.5 GiB headroom. Costs 1.35M rows → single-pass OK; Prices 12.7M +# rows → single-pass OOMs (>16 GiB). Default row threshold 2,000,000 is safely above the +# largest known single-pass success (1.35M) and well below the smallest known OOM point (12.7M). +# Extent threshold: >4 extents also triggers chunked backfill regardless of row count +# (many small extents indicate a fragmented raw table that benefits from chunking). +# Override at runtime: BACKFILL_CHUNK_ROW_THRESHOLD=5000000 pwsh scripts/ingest.ps1 +$script:BackfillChunkRowThreshold = if ($env:BACKFILL_CHUNK_ROW_THRESHOLD) { [int64]$env:BACKFILL_CHUNK_ROW_THRESHOLD } else { [int64]2000000 } +$script:BackfillChunkExtentThreshold = if ($env:BACKFILL_CHUNK_EXTENT_THRESHOLD) { [int64]$env:BACKFILL_CHUNK_EXTENT_THRESHOLD } else { [int64]4 } + function Get-HostPort { if ($env:HOST_PORT) { return $env:HOST_PORT } $envFilePath = Join-Path $script:RepoRoot '.env' @@ -132,9 +143,12 @@ function Invoke-KustoPost { $bodyText = (New-Object IO.StreamReader($stream)).ReadToEnd() } catch { } } - if ($_.Exception.Response -and $bodyText) { - try { return (New-PostResult -StatusCode ([int]$_.Exception.Response.StatusCode) -BodyText $bodyText) } catch { } - } + # Guard against strict-mode PropertyNotFoundException when the exception has no .Response property. + try { + if ($_.Exception.Response -and $bodyText) { + try { return (New-PostResult -StatusCode ([int]$_.Exception.Response.StatusCode) -BodyText $bodyText) } catch { } + } + } catch { } $messageText = "Connection error: $($_.Exception.GetType().Name): $($_.Exception.Message)" if ($_.Exception.GetType().Name -match 'WebException|HttpRequestException|IOException') { return (New-PostResult -StatusCode 599 -BodyText $messageText) @@ -557,6 +571,27 @@ function Invoke-BackfillFinalsPerPeriod { $appended[$policyInfo.FinalTable] = 0 continue } + $extentCount = (Get-TableExtents -TableName $rawTableName).Count + # Proactive chunking: skip single-pass if row count OR extent count exceed the + # measured-headroom thresholds to avoid OOM on large/fragmented raw tables. + $overRowThreshold = [int64]$rawCount -gt $script:BackfillChunkRowThreshold + $overExtentThreshold = [int64]$extentCount -gt $script:BackfillChunkExtentThreshold + if ($overRowThreshold -or $overExtentThreshold) { + $reason = if ($overRowThreshold -and $overExtentThreshold) { + "{0:N0} rows > {1:N0} and {2} extents > {3}" -f [int64]$rawCount, $script:BackfillChunkRowThreshold, $extentCount, $script:BackfillChunkExtentThreshold + } elseif ($overRowThreshold) { + "{0:N0} rows > threshold {1:N0}" -f [int64]$rawCount, $script:BackfillChunkRowThreshold + } else { + "{0} extents > extent threshold {1}" -f $extentCount, $script:BackfillChunkExtentThreshold + } + Write-InfoLine " $($policyInfo.FinalTable): $reason — proactively using per-extent chunked backfill (skipping single-pass)" + $chunked = Invoke-ChunkedBackfillFinalTable -RawTableName $rawTableName -FinalTableName $policyInfo.FinalTable -TransformFunctionName $policyInfo.TransformFn -ExtentsPerBatch 1 + $appended[$policyInfo.FinalTable] = if ($chunked.Ok) { $chunked.Rows } else { -1 } + if (-not $chunked.Ok) { [Console]::Error.WriteLine(" ERROR: $($policyInfo.FinalTable) still incomplete after chunked backfill; manual intervention required.") } + continue + } + # Below both thresholds: try single-pass (fast), fall back to chunked as a safety net. + Write-InfoLine (" $($policyInfo.FinalTable): {0:N0} rows ≤ threshold {1:N0}, {2} extents ≤ {3} — using single-pass backfill" -f [int64]$rawCount, $script:BackfillChunkRowThreshold, $extentCount, $script:BackfillChunkExtentThreshold) $result = Invoke-BackfillFinalTable -FinalTableName $policyInfo.FinalTable -TransformFunctionName $policyInfo.TransformFn if ($result.Ok) { $appended[$policyInfo.FinalTable] = $result.Rows; continue } [Console]::Error.WriteLine(" $($policyInfo.FinalTable): single-pass backfill failed, falling back to per-extent chunked backfill...") @@ -747,6 +782,17 @@ function Invoke-RunIngest { $bucket.rows_ingested += [int64]($prior.rows_ingested ?? 0) continue } + elseif ($prior) { + # Same composite key (scope/type/period/run-uuid/filename) exists in + # Ingest_Manifest but with a DIFFERENT checksum. Proceeding would silently + # append a second copy of the data and corrupt cost totals (parity check 1's + # 5% tolerance would mask the duplication). + throw ("File '$($unit.HostPath.Name)' under run-uuid '$($plan.RunUuid)' for " + + "($($plan.Scope)/$($plan.ExportType)/$($plan.Period)) changed since last " + + "ingest (checksum mismatch: manifest=$($prior.checksum_sha256), file=$sha). " + + "Re-ingesting under the same run-uuid duplicates rows. " + + "Stage corrected data under a NEW run-uuid for the same (scope,type,period) to replace it (supersede).") + } $progressPrefix = ("[{0,3}/{1}] ingesting {2}/{3}/{4}/{5} ({6} KB, {7:N0} rows expected)" -f $displayIndex, $totalFiles, $plan.Scope, $plan.ExportType, $plan.Period, $unit.HostPath.Name, [Math]::Floor($sizeBytes / 1024), [int64]$unit.ExpectedRows) Write-Host -NoNewline ($progressPrefix + ' ... ') if ($DryRun) { Write-InfoLine 'dry-run skip'; continue } From 8be83460620ccbdf8ca260549f47bc6e6a19edc5 Mon Sep 17 00:00:00 2001 From: msbrett Date: Wed, 17 Jun 2026 16:38:37 -0700 Subject: [PATCH 07/51] chore(finops-hub-local): genericize EXPORT_DIR example path (avoid /Users/ literal) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- src/templates/finops-hub-local/.env.example | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/templates/finops-hub-local/.env.example b/src/templates/finops-hub-local/.env.example index 300991bb8..e3e1d92d6 100644 --- a/src/templates/finops-hub-local/.env.example +++ b/src/templates/finops-hub-local/.env.example @@ -17,7 +17,7 @@ MEM_LIMIT=16g # Mounted read-only at /data/export inside the container. # Default is ./export (relative to the Makefile / docker-compose.yml directory). # Point to an absolute path to reuse an existing export without copying files: -# EXPORT_DIR=/Users/you/src/ftklocal/export +# EXPORT_DIR=~/src/ftklocal/export EXPORT_DIR=./export # Proactive chunked-backfill thresholds. From e3b13eedf7c721df8b1066d898994f0386b9f763 Mon Sep 17 00:00:00 2001 From: msbrett Date: Wed, 17 Jun 2026 16:44:51 -0700 Subject: [PATCH 08/51] feat(agent-skills): add finops-hub-local skill + fix ftk.ps1 catalog resolution (T-5000.8) Add the finops-hub-local agent skill (SKILL.md thin router, README, local-vs-remote reference) that drives the local Kusto-emulator FinOps hub via the ftk.ps1 CLI and the finops-hub-local Makefile, reusing the published query catalog (no new KQL). Also fix ftk.ps1 catalog discovery for the in-repo layout: Get-FtkRepo assumed the old prototype's sibling-folder layout (/finops-toolkit) and failed to find src/queries/catalog when ftklocal lives inside the repo, breaking 'ftk list' and 'ftk run'. Resolve the toolkit root in-repo (three levels up from the script dir) with a git-toplevel fallback and the legacy sibling layout as last resort. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../agent-skills/finops-hub-local/README.md | 38 +++++ .../agent-skills/finops-hub-local/SKILL.md | 146 ++++++++++++++++++ .../references/local-vs-remote.md | 94 +++++++++++ .../finops-hub-local/scripts/ftk.ps1 | 12 ++ 4 files changed, 290 insertions(+) create mode 100644 src/templates/agent-skills/finops-hub-local/README.md create mode 100644 src/templates/agent-skills/finops-hub-local/SKILL.md create mode 100644 src/templates/agent-skills/finops-hub-local/references/local-vs-remote.md diff --git a/src/templates/agent-skills/finops-hub-local/README.md b/src/templates/agent-skills/finops-hub-local/README.md new file mode 100644 index 000000000..c9cbfa4a0 --- /dev/null +++ b/src/templates/agent-skills/finops-hub-local/README.md @@ -0,0 +1,38 @@ +# FinOps hub (local) skill + +A skill for running FinOps hub cost analytics **locally** — against a Kusto emulator +in Docker — without deploying anything to Azure. + +## What it is + +The FinOps Toolkit's KQL analytics (ingestion transforms, hub view functions, open +data, and the query catalog) loaded into a local [Kusto emulator] so an analyst can +query FOCUS cost exports on their own machine. It runs the **same** analytic IP a +deployed hub runs; only the cloud-orchestration glue (Data Factory, storage, +deployment) is replaced by local scripts in `src/templates/finops-hub-local/`. + +## When to use this skill vs the cloud `finops-toolkit` skill + +| Use **finops-hub-local** when… | Use **finops-toolkit** when… | +|--------------------------------|------------------------------| +| You have FOCUS Parquet exports and want to query them offline | You have (or want to deploy) a real hub on Azure Data Explorer / Fabric | +| You're developing/testing hub KQL without standing up Azure | You need scale, multi-user access, AAD auth, or Power BI | +| No Azure subscription / Azure MCP available | You're connecting to a live `*.kusto.windows.net` cluster | + +## Prerequisites + +- Docker +- PowerShell 7+ (`pwsh`) +- FOCUS cost exports as Parquet + +## Getting started + +See `SKILL.md` for the task-routing table and the quickstart. In short, from +`src/templates/finops-hub-local/`: + +```bash +make up && make load-ftk-kql && make ingest +pwsh scripts/ftk.ps1 query "Costs() | summarize sum(EffectiveCost) by ServiceName | top 10 by sum_EffectiveCost" +``` + +[Kusto emulator]: https://learn.microsoft.com/azure/data-explorer/kusto-emulator-overview diff --git a/src/templates/agent-skills/finops-hub-local/SKILL.md b/src/templates/agent-skills/finops-hub-local/SKILL.md new file mode 100644 index 000000000..ad18e9556 --- /dev/null +++ b/src/templates/agent-skills/finops-hub-local/SKILL.md @@ -0,0 +1,146 @@ +--- +name: finops-hub-local +description: This skill should be used when the user wants to run FinOps hub cost analytics LOCALLY without an Azure deployment — e.g. "local FinOps hub", "FinOps hub local", "Kustainer", "Kusto emulator", "ftklocal", "run FinOps queries locally", "analyze FOCUS cost exports offline", "query cost parquet locally", "local cost analysis", or "FinOps toolkit without Azure". It drives a local Kusto emulator that runs the FinOps Toolkit's KQL analytics (the same Costs/Prices/Transactions functions a deployed hub exposes) against FOCUS parquet exports on disk. DO NOT use it to deploy Azure resources, provision infrastructure, or query a live Azure Data Explorer / Fabric cluster — for those, use the `finops-toolkit` skill. +license: MIT +compatibility: Requires Docker and PowerShell 7+. Runs a local Kusto emulator container — no Azure subscription, Azure MCP Server, or cloud cluster required. FOCUS exports must already be Parquet. +metadata: + author: microsoft + version: "1.0" +--- + +# FinOps hub (local) + +Run the FinOps Toolkit's full KQL cost-analytics stack **locally**, against a Kusto +emulator (Docker) instead of a deployed Azure Data Explorer or Microsoft Fabric +cluster. The analytic IP is identical — the same `IngestionSetup` / `HubSetup` KQL, +the same hub view functions (`Costs`, `Prices`, `Transactions`, and their `*_v1_2` +variants), the same open data, and the same query catalog — so cost analysis works +offline on a developer machine. Only the cloud-orchestration glue (Data Factory, +storage, deployment) is replaced by local scripts. + +The tooling lives in `src/templates/finops-hub-local/`. Run the commands below from +that directory. + +## When to invoke + +- The user has FOCUS cost exports (Parquet) and wants to query them locally. +- The user wants to develop or test FinOps hub KQL without standing up Azure. +- The user asks for "the local FinOps hub", "ftklocal", "Kustainer", or offline cost analytics. + +Do **not** invoke for deploying or querying a real Azure hub — route those to the +`finops-toolkit` skill. + +## Topology (so queries are written correctly) + +The local stack mirrors a real hub's **two-database** layout: + +| Database | Holds | You query it for | +|----------|-------|------------------| +| `Ingestion` | Raw tables (`Costs_raw`, `Prices_raw`), the `*_transform_v1_2()` functions, and the final tables (`Costs_final_v1_2`, `Prices_final_v1_2`) | Ingestion/transform internals, parity checks | +| `Hub` | The hub **view functions** (`Costs`, `Prices`, `Transactions`, `Costs_v1_2`, `Prices_v1_2`, …) which read `database('Ingestion').*` | All analytics — this is the default query database | + +The cross-database references (`database('Ingestion').*`) inside the Hub functions +resolve in the emulator exactly as they do in ADX. The CLI defaults to the `Hub` +database. + +## Prerequisites + +- Docker (the emulator runs as a container via `docker compose`). +- PowerShell 7+ (`pwsh`) — all tooling is PowerShell; there are no Python dependencies. +- FOCUS cost exports as **Parquet**, staged under `export/` (see the staging contract + in `src/templates/finops-hub-local/notes/staging-contract.md`). + +## Quickstart + +```bash +# from src/templates/finops-hub-local/ +cp .env.example .env # 1. configure (HOST_PORT, MEM_LIMIT, EXPORT_DIR) +make up # 2. start the emulator; blocks until healthcheck passes +make load-ftk-kql # 3. create Ingestion + Hub, load FTK KQL + open data (idempotent) +make ingest # 4. bulk-ingest the Parquet exports under export/ +pwsh scripts/ftk.ps1 query "Costs() | summarize TotalEffectiveCost = sum(EffectiveCost) by ServiceName | top 10 by TotalEffectiveCost" +``` + +## Task routing + +| Task | Run this | +|------|----------| +| Start / stop the emulator | `make up` / `make down` (data preserved); `make nuke` (destructive reset) | +| Load the analytics schema + open data | `make load-ftk-kql` (idempotent; safe to re-run) | +| Ingest Parquet exports | `make ingest` (or `pwsh scripts/ingest.ps1`); scope/period subsets: `make ingest SCOPE= PERIOD=` | +| Check what's been ingested | `make ingest-status` | +| Run an ad-hoc KQL query | `pwsh scripts/ftk.ps1 query ""` (default database `Hub`) | +| List the reusable query catalog | `pwsh scripts/ftk.ps1 list` | +| Run a named catalog query | `pwsh scripts/ftk.ps1 run ` (add `--format csv`, `--start`/`--end`) | +| Verify transform parity | `make parity` (exit 0 iff all checks pass) | +| Recover a Prices backfill OOM | `make chunked-prices-backfill` (per-extent fallback) | +| See all targets | `make help` | + +The `ftk.ps1 run` command reuses the published FinOps Toolkit query catalog **in +place** (it never forks the `.kql` files). It applies a few small, inspectable +on-read adaptations so version-loose catalog queries run against the emulator — see +`references/local-vs-remote.md`. There is **no** new KQL in this skill; all analytic +logic is upstream FinOps Toolkit. + +## Common queries + +All examples use `Costs_v1_2()` — the Hub view function. Substitute `Costs_final_v1_2` +(in the `Ingestion` database) for the un-enriched final table. Results vary with the +data you ingested; if `Costs_v1_2() | count` returns 0, run `make ingest` first. + +```kql +// Top 10 services by effective cost +Costs_v1_2() +| summarize TotalEffectiveCost = sum(EffectiveCost) by ServiceName +| top 10 by TotalEffectiveCost +``` + +```kql +// Monthly cost trend +Costs_v1_2() +| summarize MonthlyCost = sum(EffectiveCost) by Month = monthstring(ChargePeriodStart) +| order by Month asc +``` + +```kql +// Top 20 most expensive resources +Costs_v1_2() +| summarize TotalCost = sum(EffectiveCost), BilledCost = sum(BilledCost) + by ResourceId, ResourceName, ServiceName, x_ResourceGroupName +| top 20 by TotalCost +``` + +```kql +// Subscription-level cost summary +Costs_v1_2() +| summarize EffectiveCost = sum(EffectiveCost), BilledCost = sum(BilledCost) + by SubAccountId, SubAccountName +| order by EffectiveCost desc +``` + +## Troubleshooting + +| Symptom | Cause / fix | +|---------|-------------| +| Emulator won't start; port in use | Another process holds `HOST_PORT` (default 8082). Change `HOST_PORT` in `.env`, then `make up`. | +| Container OOM during the Prices backfill | The single-pass `Prices_transform_v1_2()` exceeds memory on large datasets. Keep `MEM_LIMIT=16g` (the `.env.example` default); ingest now chunks large tables automatically, and `make chunked-prices-backfill` is the manual recovery. | +| `Costs_v1_2() \| count` returns 0 after load | The schema is loaded but no data is ingested yet. Run `make ingest`. | +| Database doesn't appear after `make up` | Emulator metadata can need a clean reset between image versions — `make nuke && make up && make load-ftk-kql`. | +| Ingest aborts: "checksum mismatch … stage under a NEW run-uuid" | A corrected file was re-staged under the same run-uuid. This guard prevents silently duplicating cost rows — stage the corrected data under a new run-uuid for the same scope/type/period (it supersedes the old run). | + +## Limits and non-goals + +- **Single user, local only.** No auth, no TLS, no multi-tenancy. +- **No Azure Data Factory.** Ingest orchestration (blob trigger → convert → ingest) is + replaced by `scripts/ingest.ps1`. Exports must already be FOCUS Parquet — there is no + CSV-to-Parquet convert step. +- **No deployment.** No Bicep/ARM; the stack is `docker compose`. +- **Not a production hub.** Use it for local analysis and KQL development. For a real + hub (scale, Power BI, multi-user, AAD), deploy with the `finops-toolkit` skill. + +## References + +- `references/local-vs-remote.md` — what is identical vs adapted between a deployed hub + and the local stack (connection, the on-read query adaptations, the IP-reuse map). +- `src/templates/finops-hub-local/notes/staging-contract.md` — the data-in contract for `export/`. +- `src/templates/finops-hub-local/notes/dashboard.md` — querying the local Hub from the ADX dashboard / CLI. diff --git a/src/templates/agent-skills/finops-hub-local/references/local-vs-remote.md b/src/templates/agent-skills/finops-hub-local/references/local-vs-remote.md new file mode 100644 index 000000000..e325e81cb --- /dev/null +++ b/src/templates/agent-skills/finops-hub-local/references/local-vs-remote.md @@ -0,0 +1,94 @@ +# Local vs remote: reusing the FinOps Toolkit analytics + +The local FinOps hub runs the **same FinOps Toolkit analytics** as a deployed hub, but +against a local Kusto emulator instead of an Azure Data Explorer (ADX) or Microsoft +Fabric cluster. The query IP is identical; only the **connection** and a couple of +small, on-read **query adaptations** differ. + +## What is the same: the analytic IP + +Both local and remote expose the FinOps hub analytic view functions, so the published +query catalog is portable across both: + +| Function | Purpose | +|----------|---------| +| `Costs()` / `Costs_v1_2()` | Cost & usage analytics (FOCUS-aligned) | +| `Prices()` / `Prices_v1_2()` | Price sheets (list / contracted / effective) | +| `Transactions()` | Commitment purchases, refunds, exchanges | + +These functions live in the `Hub` database and read the final tables in the +`Ingestion` database via `database('Ingestion').*` references — the same two-database +topology a deployed hub uses. The cross-database references resolve in the emulator +exactly as they do in ADX. + +## What is different: connecting + +| Aspect | Deployed hub | Local | +|--------|-------------|-------| +| Endpoint | `https://.kusto.windows.net` | `http://localhost:8082` (default `HOST_PORT`) | +| Auth | Azure AD (tenant) | none (local, single user) | +| Query database | `Hub` | `Hub` (same) | +| Client | MCP Kusto server / Power BI | `ftk.ps1` CLI | +| Settings | environment settings file | `.env` (`HOST_PORT`, `MEM_LIMIT`, `EXPORT_DIR`) | + +Run a catalog query locally: + +```bash +pwsh scripts/ftk.ps1 list # discover catalog queries +pwsh scripts/ftk.ps1 run savings-summary-report # adapt + run a named catalog query +pwsh scripts/ftk.ps1 query "Costs() | summarize sum(EffectiveCost) by ServiceName | top 10 by sum_EffectiveCost" +``` + +## On-read query adaptations (applied automatically by `ftk run`) + +The emulator is the same engine as ADX, so catalog queries run **almost** verbatim. +`ftk run` reads each upstream `.kql` **in place** (it never forks the file) and applies +a few small, inspectable transforms: + +1. **Date window** — catalog queries default to a trailing-30-day window, which returns + nothing against a static local export. `ftk` retargets the window to the latest + period present in the data. Override with `--start` / `--end`. +2. **`decimal` → `real`** — a few catalog queries use `decimal()` / `todecimal()` + literals. The hub's cost columns are typed `real`; ADX silently coerces the mix, but + the emulator (stricter) rejects it, so `ftk` normalizes `decimal(N)`→`real(N)` and + `todecimal(x)`→`toreal(x)`. +3. **Cross-database prefix** — the rare catalog query that explicitly names + `database('Ingestion').` is normalized for the local default-database context. +4. **`project-away` tolerance** — a few catalog queries `project-away` a column that + exists in a different hub schema version but not this one (for example `x_InvoiceId`, + which the v1.2 transform folds into the FOCUS-standard `InvoiceId`). `project-away` + errors on unknown columns, so `ftk` drops any such name from the list and the query + still runs. The local schema is correct; this only keeps version-loose catalog + queries portable. + +These adaptations are **local-only conveniences**. The upstream catalog files are +unchanged and run as-is against a real hub. + +## What the local stack does NOT have (vs a deployed hub) + +| Hub component | Local | +|---------------|-------| +| Azure Data Factory (blob trigger → convert → ingest) | Replaced by `scripts/ingest.ps1`. Exports must already be FOCUS Parquet — no CSV-to-Parquet convert step. | +| ADLS storage (msexports / ingestion containers) | Local `export/` folder + raw tables. | +| Bicep / ARM deployment | Not needed — `docker compose` (`make up`). | +| Power BI binding | Not wired locally; use `ftk.ps1 ... --format csv`, or see `notes/dashboard.md`. | +| Multi-user / TLS / auth | None. Single developer, local only. | + +There is **no** local Azure Data Factory emulator. Data Factory in a hub is only +orchestration glue and carries no analytic IP, so the local stack replaces it with the +ingest script rather than emulating it. + +## IP reuse, at a glance + +| FinOps Toolkit component | Local | +|--------------------------|-------| +| `IngestionSetup_v1_2.kql` (raw tables, update policies, `*_transform_v1_2()`) | Loaded verbatim into the `Ingestion` database | +| `HubSetup_v1_2.kql` (hub view functions + UDFs) | Loaded verbatim into the `Hub` database | +| Open-data reference CSVs | Loaded as-is | +| Query catalog (`src/queries/catalog/*.kql`, agent-skill subset) | Run as-is, with the four on-read adaptations above | +| Data Factory orchestration | Replaced by `scripts/ingest.ps1` | +| ADLS storage | Replaced by `export/` + raw tables | +| Bicep / ARM deploy | Not needed locally | + +**Net: essentially all of the analytic IP (transforms + hub functions + open data + +catalog queries) is reused; only cloud-orchestration glue is reimplemented locally.** diff --git a/src/templates/finops-hub-local/scripts/ftk.ps1 b/src/templates/finops-hub-local/scripts/ftk.ps1 index 8bcd3f5db..6825f92b0 100644 --- a/src/templates/finops-hub-local/scripts/ftk.ps1 +++ b/src/templates/finops-hub-local/scripts/ftk.ps1 @@ -215,6 +215,18 @@ $script:LibraryQueries = @('costs-enriched-base') function Get-FtkRepo { if ($env:FTK_REPO) { return $env:FTK_REPO } + # ftklocal lives in-repo at /src/templates/finops-hub-local, so the toolkit + # root (which holds src/queries/catalog) is three directories up from $RepoRoot. + $inRepoRoot = Split-Path -Parent (Split-Path -Parent (Split-Path -Parent $script:RepoRoot)) + if ($inRepoRoot -and (Test-Path (Join-Path $inRepoRoot 'src/queries/catalog') -PathType Container)) { + return $inRepoRoot + } + # Robust fallback for worktrees / unusual layouts: the git top-level. + $gitTop = (& git -C $script:RepoRoot rev-parse --show-toplevel 2>$null) + if ($gitTop -and (Test-Path (Join-Path $gitTop 'src/queries/catalog') -PathType Container)) { + return $gitTop + } + # Last resort: legacy prototype layout where ftklocal sat beside a finops-toolkit clone. return (Resolve-Path (Join-Path (Split-Path -Parent $script:RepoRoot) 'finops-toolkit') -ErrorAction SilentlyContinue) ` ?? (Join-Path (Split-Path -Parent $script:RepoRoot) 'finops-toolkit') } From e264089b918ae072f863eff59b8568021562e326 Mon Sep 17 00:00:00 2001 From: msbrett Date: Wed, 17 Jun 2026 16:47:21 -0700 Subject: [PATCH 09/51] fix(finops-hub-local): guard git fallback in ftk.ps1 Get-FtkRepo Under Set-StrictMode + $ErrorActionPreference='Stop', calling '& git' when git is absent throws CommandNotFoundException and skips the legacy sibling-layout fallback. Guard the git-toplevel probe with Get-Command so a missing git degrades gracefully. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- src/templates/finops-hub-local/scripts/ftk.ps1 | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/src/templates/finops-hub-local/scripts/ftk.ps1 b/src/templates/finops-hub-local/scripts/ftk.ps1 index 6825f92b0..4e1604c14 100644 --- a/src/templates/finops-hub-local/scripts/ftk.ps1 +++ b/src/templates/finops-hub-local/scripts/ftk.ps1 @@ -222,9 +222,13 @@ function Get-FtkRepo { return $inRepoRoot } # Robust fallback for worktrees / unusual layouts: the git top-level. - $gitTop = (& git -C $script:RepoRoot rev-parse --show-toplevel 2>$null) - if ($gitTop -and (Test-Path (Join-Path $gitTop 'src/queries/catalog') -PathType Container)) { - return $gitTop + # Guard on git being present so a missing git degrades to the legacy fallback + # instead of throwing under $ErrorActionPreference = 'Stop'. + if (Get-Command git -ErrorAction SilentlyContinue) { + $gitTop = (& git -C $script:RepoRoot rev-parse --show-toplevel 2>$null) + if ($gitTop -and (Test-Path (Join-Path $gitTop 'src/queries/catalog') -PathType Container)) { + return $gitTop + } } # Last resort: legacy prototype layout where ftklocal sat beside a finops-toolkit clone. return (Resolve-Path (Join-Path (Split-Path -Parent $script:RepoRoot) 'finops-toolkit') -ErrorAction SilentlyContinue) ` From 20fd3d3ee9af55a2e9347a9ce2be1fde6d42a0b0 Mon Sep 17 00:00:00 2001 From: msbrett Date: Wed, 17 Jun 2026 17:09:00 -0700 Subject: [PATCH 10/51] refactor(finops-hub-local): gate chunked backfill on row count alone + harden Prices value parity (T-5000.3) Per architect-duo ruling: drop the extents>4 proactive-chunk criterion (OOM is row-driven; >4 extents mis-fires on every multi-part export and would chunk Costs, which single-passes safely at 15 extents). Gate proactive chunking on rawCount > 2,000,000 alone; keep the reactive single-pass->chunked fallback. Extent count is now informational logging only. Remove the BACKFILL_CHUNK_EXTENT_THRESHOLD knob; note that adding a new RawToFinalPolicy table needs a fresh single-pass OOM measurement. Fix the Makefile threshold comment (3,000,000 -> 2,000,000). Harden parity CHECK 13 (Prices value parity): emit delta_pct so the runner enforces a 0.1% relative tolerance, with a sentinel that fails on zero rows or row-count mismatch so the check cannot pass vacuously. Verified: real data 9,804,381 rows / sums equal at $58.39B -> PASS; x1.01 corruption -> FAIL; zero rows -> FAIL. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- src/templates/finops-hub-local/.env.example | 25 +++++++------ src/templates/finops-hub-local/Makefile | 2 +- .../finops-hub-local/scripts/ingest.ps1 | 36 ++++++++----------- .../finops-hub-local/tests/parity-checks.kql | 26 ++++++++++++-- 4 files changed, 53 insertions(+), 36 deletions(-) diff --git a/src/templates/finops-hub-local/.env.example b/src/templates/finops-hub-local/.env.example index e3e1d92d6..5aa8cdf60 100644 --- a/src/templates/finops-hub-local/.env.example +++ b/src/templates/finops-hub-local/.env.example @@ -20,16 +20,19 @@ MEM_LIMIT=16g # EXPORT_DIR=~/src/ftklocal/export EXPORT_DIR=./export -# Proactive chunked-backfill thresholds. -# ingest.ps1 measures raw-table row count AND extent count before each final-table backfill. -# Chunked backfill is used proactively when EITHER threshold is exceeded (avoiding OOM). -# If both are at or below their thresholds, single-pass is tried first with chunked fallback. +# Proactive chunked-backfill threshold (rows). +# ingest.ps1 measures the raw-table row count before each final-table backfill. +# If the count exceeds this value, single-pass .set-or-append is skipped entirely +# and per-extent chunked backfill is used proactively (avoids OOM on large datasets). +# If at or below, single-pass is tried first with chunked backfill as the reactive +# safety-net fallback. OOM risk is row-driven, not extent-driven. # -# Row threshold — default 2000000 is derived from measured headroom at MEM_LIMIT=16g, amd64-Rosetta: -# - Costs 1,350,000 rows → single-pass OK (safely below threshold) -# - Prices 12,735,587 rows → single-pass OOMs (safely above threshold) -# Extent threshold — default 4: >4 extents indicates a fragmented raw table that benefits -# from per-extent chunking regardless of total row count. -# Raise row threshold if you increase MEM_LIMIT and re-verify single-pass succeeds. +# Default 2000000 is derived from measured headroom at MEM_LIMIT=16g, amd64-Rosetta: +# - Costs 1,350,000 rows → single-pass OK (safely below threshold) +# - Prices 12,735,587 rows → single-pass OOMs (safely above threshold) +# Raise this value if you increase MEM_LIMIT and re-verify single-pass succeeds. +# Note: if you add a 3rd table to RawToFinalPolicy, do NOT assume this threshold +# applies — do a fresh single-pass OOM measurement for that table's row count and +# MEM_LIMIT. The reactive fallback (single-pass fail → chunked) covers it in the +# meantime. # BACKFILL_CHUNK_ROW_THRESHOLD=2000000 -# BACKFILL_CHUNK_EXTENT_THRESHOLD=4 diff --git a/src/templates/finops-hub-local/Makefile b/src/templates/finops-hub-local/Makefile index bb51632ed..e36f339e8 100644 --- a/src/templates/finops-hub-local/Makefile +++ b/src/templates/finops-hub-local/Makefile @@ -153,7 +153,7 @@ parity: # (bounded memory). Use after `make ingest` if Prices_final_v1_2 ends up # partially populated or empty. Wall-clock ~15 min on Apple Silicon. # NOTE: ingest.ps1 now auto-chunks proactively when row count > BACKFILL_CHUNK_ROW_THRESHOLD -# (default 3,000,000). This target is the manual fallback / override path. +# (default 2,000,000). This target is the manual fallback / override path. chunked-prices-backfill: @pwsh scripts/chunked-prices-backfill.ps1 diff --git a/src/templates/finops-hub-local/scripts/ingest.ps1 b/src/templates/finops-hub-local/scripts/ingest.ps1 index aa31d80ca..9a72fb991 100755 --- a/src/templates/finops-hub-local/scripts/ingest.ps1 +++ b/src/templates/finops-hub-local/scripts/ingest.ps1 @@ -65,16 +65,15 @@ $script:RawToFinalPolicy = @{ 'Prices_raw' = [pscustomobject]@{ FinalTable = 'Prices_final_v1_2'; TransformFn = 'Prices_transform_v1_2' } } -# Proactive chunked-backfill thresholds. +# Proactive chunked-backfill threshold (rows). # Measured baseline: MEM_LIMIT=16g, amd64-Rosetta; idle-loaded engine ~12.4 GiB working # set (78% of 16g), ~3.5 GiB headroom. Costs 1.35M rows → single-pass OK; Prices 12.7M -# rows → single-pass OOMs (>16 GiB). Default row threshold 2,000,000 is safely above the -# largest known single-pass success (1.35M) and well below the smallest known OOM point (12.7M). -# Extent threshold: >4 extents also triggers chunked backfill regardless of row count -# (many small extents indicate a fragmented raw table that benefits from chunking). +# rows → single-pass OOMs (>16 GiB). Default 2,000,000 is safely above the largest known +# single-pass success (1.35M) and well below the smallest known OOM point (12.7M). +# OOM risk is row-driven, not extent-driven — use the reactive fallback below as the safety +# net for any table whose single-pass headroom is unknown. # Override at runtime: BACKFILL_CHUNK_ROW_THRESHOLD=5000000 pwsh scripts/ingest.ps1 -$script:BackfillChunkRowThreshold = if ($env:BACKFILL_CHUNK_ROW_THRESHOLD) { [int64]$env:BACKFILL_CHUNK_ROW_THRESHOLD } else { [int64]2000000 } -$script:BackfillChunkExtentThreshold = if ($env:BACKFILL_CHUNK_EXTENT_THRESHOLD) { [int64]$env:BACKFILL_CHUNK_EXTENT_THRESHOLD } else { [int64]4 } +$script:BackfillChunkRowThreshold = if ($env:BACKFILL_CHUNK_ROW_THRESHOLD) { [int64]$env:BACKFILL_CHUNK_ROW_THRESHOLD } else { [int64]2000000 } function Get-HostPort { if ($env:HOST_PORT) { return $env:HOST_PORT } @@ -572,26 +571,19 @@ function Invoke-BackfillFinalsPerPeriod { continue } $extentCount = (Get-TableExtents -TableName $rawTableName).Count - # Proactive chunking: skip single-pass if row count OR extent count exceed the - # measured-headroom thresholds to avoid OOM on large/fragmented raw tables. - $overRowThreshold = [int64]$rawCount -gt $script:BackfillChunkRowThreshold - $overExtentThreshold = [int64]$extentCount -gt $script:BackfillChunkExtentThreshold - if ($overRowThreshold -or $overExtentThreshold) { - $reason = if ($overRowThreshold -and $overExtentThreshold) { - "{0:N0} rows > {1:N0} and {2} extents > {3}" -f [int64]$rawCount, $script:BackfillChunkRowThreshold, $extentCount, $script:BackfillChunkExtentThreshold - } elseif ($overRowThreshold) { - "{0:N0} rows > threshold {1:N0}" -f [int64]$rawCount, $script:BackfillChunkRowThreshold - } else { - "{0} extents > extent threshold {1}" -f $extentCount, $script:BackfillChunkExtentThreshold - } - Write-InfoLine " $($policyInfo.FinalTable): $reason — proactively using per-extent chunked backfill (skipping single-pass)" + # Proactive chunking: skip single-pass entirely when row count exceeds the + # measured-headroom threshold. OOM is row-driven; extent count is informational only. + if ([int64]$rawCount -gt $script:BackfillChunkRowThreshold) { + Write-InfoLine (" $($policyInfo.FinalTable): {0:N0} rows > threshold {1:N0} ({2} extents) — proactively using per-extent chunked backfill (skipping single-pass)" -f [int64]$rawCount, $script:BackfillChunkRowThreshold, $extentCount) $chunked = Invoke-ChunkedBackfillFinalTable -RawTableName $rawTableName -FinalTableName $policyInfo.FinalTable -TransformFunctionName $policyInfo.TransformFn -ExtentsPerBatch 1 $appended[$policyInfo.FinalTable] = if ($chunked.Ok) { $chunked.Rows } else { -1 } if (-not $chunked.Ok) { [Console]::Error.WriteLine(" ERROR: $($policyInfo.FinalTable) still incomplete after chunked backfill; manual intervention required.") } continue } - # Below both thresholds: try single-pass (fast), fall back to chunked as a safety net. - Write-InfoLine (" $($policyInfo.FinalTable): {0:N0} rows ≤ threshold {1:N0}, {2} extents ≤ {3} — using single-pass backfill" -f [int64]$rawCount, $script:BackfillChunkRowThreshold, $extentCount, $script:BackfillChunkExtentThreshold) + # Below threshold: try single-pass (fast), fall back to chunked as a safety net. + # The reactive fallback also covers any future table added to RawToFinalPolicy whose + # single-pass headroom has not been measured yet. + Write-InfoLine (" $($policyInfo.FinalTable): {0:N0} rows ≤ threshold {1:N0} ({2} extents) — using single-pass backfill" -f [int64]$rawCount, $script:BackfillChunkRowThreshold, $extentCount) $result = Invoke-BackfillFinalTable -FinalTableName $policyInfo.FinalTable -TransformFunctionName $policyInfo.TransformFn if ($result.Ok) { $appended[$policyInfo.FinalTable] = $result.Rows; continue } [Console]::Error.WriteLine(" $($policyInfo.FinalTable): single-pass backfill failed, falling back to per-extent chunked backfill...") diff --git a/src/templates/finops-hub-local/tests/parity-checks.kql b/src/templates/finops-hub-local/tests/parity-checks.kql index 1704388f4..d3839c665 100644 --- a/src/templates/finops-hub-local/tests/parity-checks.kql +++ b/src/templates/finops-hub-local/tests/parity-checks.kql @@ -8,8 +8,8 @@ // EVAL rule, and produces a pass/fail report. // // ROUTING (enforced by run-parity-checks.ps1 per-check DB assignment): -// Checks 1–8, 11 → Ingestion (raw tables, final tables) -// Checks 9, 10 → Hub (view functions cross-referencing Ingestion) +// Checks 1–8, 11–13 → Ingestion (raw tables, final tables) +// Checks 9, 10 → Hub (view functions cross-referencing Ingestion) // // CONVENTIONS // - Each check is a block: // CHECK : , // EXPECT: , then the KQL. @@ -140,3 +140,25 @@ raw_by round(todouble(coalesce(final_n, 0) - coalesce(raw_n, 0)) / todouble(coalesce(raw_n, 1)) * 100.0, 4), 0.0) | order by raw_n desc + +// CHECK 13: Prices value parity — UnitPrice/ContractedUnitPrice (Consumption + RI) +// EXPECT: sum(UnitPrice) in Prices_raw equals sum(ContractedUnitPrice) in Prices_final_v1_2 +// for Consumption and ReservedInstance PriceTypes. These columns pass through the +// Prices transform unchanged. SavingsPlan is excluded — the transform enriches SP prices +// via cross-row lookups, so raw ≠ final for SP by design. +// Guards: (1) raw_n > 0 — rows for these types actually exist; (2) final_n == raw_n — +// no rows were lost or duplicated. If either guard fails, delta_pct is set to a large +// sentinel (9999) so the check FAILS rather than passing vacuously. +// EVAL: abs(raw_sum - final_sum) / abs(raw_sum) < 0.001 +let raw_n = toscalar(Prices_raw | where PriceType in ('Consumption', 'ReservedInstance') | count); +let final_n = toscalar(Prices_final_v1_2 | where x_SkuPriceType in ('Consumption', 'ReservedInstance') | count); +let raw_sum = toscalar(Prices_raw | where PriceType in ('Consumption', 'ReservedInstance') | summarize sum(UnitPrice)); +let final_sum = toscalar(Prices_final_v1_2 | where x_SkuPriceType in ('Consumption', 'ReservedInstance') | summarize sum(ContractedUnitPrice)); +print + raw_n = raw_n, + final_n = final_n, + raw_sum_UnitPrice = raw_sum, + final_sum_ContractedUnitPrice = final_sum, + delta_pct = iff(raw_n == 0 or final_n != raw_n, + 9999.0, + todouble(abs(raw_sum - final_sum)) / todouble(abs(raw_sum)) * 100.0) From f574559d56df414b9c33783f20d871de2a8ded79 Mon Sep 17 00:00:00 2001 From: msbrett Date: Wed, 17 Jun 2026 17:31:20 -0700 Subject: [PATCH 11/51] docs(finops-hub-local): README + adapter/parity notes; drop stale cross-db strip in ftk.ps1 (T-5000.7) Add user-facing docs under src/templates/finops-hub-local/: README.md (quickstart, two-database Ingestion+Hub architecture, ingest with the fail-fast double-ingest guard and automatic chunked backfill, querying via ftk.ps1 + the agent skill, dashboard best-effort, measured limits table, troubleshooting), notes/ftk-kql-adaptations.md (the load-time vs ftk.ps1 on-read adapter layers), notes/parity-gaps.md (the 13-check parity suite, grounded in a live run). Also remove the now-stale database('Ingestion'). strip from the ftk.ps1 on-read adapter: on the two-database topology cross-database references resolve correctly, so stripping them would break any catalog query that legitimately references an Ingestion table. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- src/templates/finops-hub-local/README.md | 358 ++++++++++++++++++ .../notes/ftk-kql-adaptations.md | 223 +++++++++++ .../finops-hub-local/notes/parity-gaps.md | 178 +++++++++ .../finops-hub-local/scripts/ftk.ps1 | 7 +- 4 files changed, 760 insertions(+), 6 deletions(-) create mode 100644 src/templates/finops-hub-local/README.md create mode 100644 src/templates/finops-hub-local/notes/ftk-kql-adaptations.md create mode 100644 src/templates/finops-hub-local/notes/parity-gaps.md diff --git a/src/templates/finops-hub-local/README.md b/src/templates/finops-hub-local/README.md new file mode 100644 index 000000000..fb4cfa7b0 --- /dev/null +++ b/src/templates/finops-hub-local/README.md @@ -0,0 +1,358 @@ +# FinOps hub (local) + +Run the FinOps Toolkit's full KQL cost-analytics stack **locally** against a Kusto +emulator (Docker) — no Azure subscription, no cloud cluster, no Data Factory required. +The analytic IP is identical to a deployed hub: the same `IngestionSetup` / `HubSetup` +KQL, the same hub view functions (`Costs`, `Prices`, `Transactions`, and their `*_v1_2` +variants), the same open data, and the same query catalog. + +--- + +## When to use this vs deploying a real hub + +| Situation | Use | +|-----------|-----| +| Offline analysis, air-gapped network, or no Azure subscription | **ftklocal** | +| Developing or testing KQL changes before deploying | **ftklocal** | +| Working with FOCUS parquet already on disk | **ftklocal** | +| Production analytics for an organization | [Deploy a FinOps hub](../finops-hub/README.md) | +| Querying a live Azure Data Explorer or Fabric cluster | [FinOps hub](../finops-hub/README.md) | + +--- + +## Prerequisites + +- **Docker Desktop ≥ 4.0** — Compose v2 is required (`docker compose`, not the legacy + `docker-compose`). +- **Apple Silicon (M-series) users:** enable **Rosetta** for x86/amd64 emulation. The + Kustainer image is `linux/amd64` only. Docker Desktop → Settings → General → "Use + Rosetta for x86_64/amd64 emulation on Apple Silicon". +- **PowerShell 7+** (`pwsh`) — all tooling is PowerShell; there are no Python + dependencies. +- **FOCUS cost exports as Parquet** — staged under `export/`. See + [notes/staging-contract.md](notes/staging-contract.md) for the folder layout. +- Roughly **16 GiB of RAM** available for the container (the default `MEM_LIMIT` in + `.env.example`). Costs-only datasets work at 8 GiB; 16 GiB is required for the full + Prices transform. See [Limits](#limits). +- Host port **8082** free (configurable via `HOST_PORT` in `.env`). + +--- + +## Quickstart + +```bash +# From src/templates/finops-hub-local/ + +cp .env.example .env # 1. review tunables (HOST_PORT, MEM_LIMIT, EXPORT_DIR) +make up # 2. start the emulator; blocks until healthcheck passes (~30 s) +make load-ftk-kql # 3. create Ingestion + Hub DBs, load FTK KQL + open data (idempotent) +make ingest # 4. bulk-ingest all Parquet exports under export/ + +# 5. query your FOCUS data +pwsh scripts/ftk.ps1 query "Costs() | summarize TotalCost = sum(EffectiveCost) by ServiceName | top 10 by TotalCost" +``` + +After step 4 completes, the Hub database holds the same view functions a deployed hub +exposes. `make ingest-status` shows a summary of what was ingested. + +--- + +## Configuration + +All tunables live in `.env` (gitignored). `.env.example` documents them: + +| Variable | Default | Meaning | +|----------|---------|---------| +| `HOST_PORT` | `8082` | Host port mapped to the Kustainer container's HTTP endpoint. | +| `MEM_LIMIT` | `16g` | Memory ceiling for the Kustainer container. | +| `EXPORT_DIR` | `./export` | Host directory with Parquet exports (read-only mount). | +| `BACKFILL_CHUNK_ROW_THRESHOLD` | `2000000` | Row count above which `ingest.ps1` uses proactive chunked backfill. See [Chunked backfill](#automatic-chunked-backfill). | + +--- + +## Architecture + +### Two-database topology + +The local stack mirrors the real hub's **two-database** layout: + +| Database | Holds | You query it for | +|----------|-------|------------------| +| **`Ingestion`** | Raw tables (`Costs_raw`, `Prices_raw`), `*_transform_v1_2()` functions, final tables (`Costs_final_v1_2`, `Prices_final_v1_2`), open-data lookups, and `Ingest_Manifest` | Ingestion internals, parity checks, transform debugging | +| **`Hub`** | Hub view functions (`Costs`, `Prices`, `Transactions`, `Costs_v1_2`, `Prices_v1_2`, …) that read `database('Ingestion').*` | All analytics — this is the default query database | + +The `Hub` database is the default for `make kql` and `ftk.ps1`. To inspect raw tables, +override: `make kql KQL_DB=Ingestion QUERY='Costs_raw | count'`. + +### How it maps to a real hub + +| Real hub component | Local equivalent | +|--------------------|-----------------| +| Azure Data Explorer (ADX) cluster | Kusto emulator (Docker, `http://localhost:8082`) | +| `Ingestion` ADX database | `Ingestion` database in the emulator | +| `Hub` ADX database | `Hub` database in the emulator | +| Data Factory pipeline (ADF) | `scripts/ingest.ps1` | +| Azure Storage (`msexports/` or `ingestion/` container) | `export/` directory on disk | +| Bicep/ARM deployment | `make load-ftk-kql` (idempotent KQL loader) | + +The KQL itself — `IngestionSetup_RawTables.kql`, `IngestionSetup_v1_2.kql`, +`HubSetup_v1_2.kql`, `HubSetup_OpenData.kql` — is loaded verbatim from the upstream +FTK source with [minimal on-load adaptations](notes/ftk-kql-adaptations.md). + +--- + +## Ingesting data + +### Staging your exports + +Parquet exports must be staged under `export/` in the layout described in +[notes/staging-contract.md](notes/staging-contract.md). The short version: + +``` +export/ +└── / + └── / # ms--focus-cost | ms--pricesheet + └── / + └── / + ├── manifest.json + └── *.parquet +``` + +A `manifest.json` produced directly by Azure Cost Management can be used without +modification. See [staging-contract.md](notes/staging-contract.md) for minimum +viable manifest fields and instructions for both `msexports/` and `ingestion/` +source containers. + +### Running ingest + +```bash +make ingest # ingest all scopes/periods +make ingest SCOPE=ea # one scope +make ingest SCOPE=ea PERIOD=20260501-20260531 # scope + period +make ingest-status # show Ingest_Manifest summary +``` + +`ingest.ps1` is idempotent: it tracks every ingested file in `Ingest_Manifest` by +`(scope, type, period, run-uuid, file-name)` and SHA-256 checksum. Re-running `make +ingest` on an unchanged export directory is safe — already-ingested files are skipped. + +### Overwrite semantics + +| Situation | Behavior | +|-----------|----------| +| Same run-uuid, same file checksum | Skip (already ingested) | +| Same `(scope, type, period)`, **new** run-uuid | Safe replace — old extents are dropped, new data is ingested | +| Same run-uuid, **changed** file checksum | **Fail-fast double-ingest guard** — the script detects the checksum mismatch and refuses. Stage the replacement files under a **new run-uuid** instead. | + +> **How to replace data:** stage replacement parquet under a new `` directory +> with a `manifest.json` whose `runInfo.submittedTime` is later than the existing run. +> `ingest.ps1` selects the latest run per `(scope, type, period)`, drops the old run's +> extents, and ingests the new files cleanly. + +### Automatic chunked backfill + +For large tables, `ingest.ps1` avoids the single-pass OOM by chunking the final-table +backfill automatically: + +- **Proactive:** if the raw table row count exceeds `BACKFILL_CHUNK_ROW_THRESHOLD` + (default 2,000,000), single-pass `.set-or-append` is skipped entirely and per-extent + chunked backfill is used from the start. +- **Reactive:** if the row count is at or below the threshold, single-pass is tried + first; if it OOMs the engine, `ingest.ps1` falls back to chunked automatically. + +The threshold is sized so that at `MEM_LIMIT=16g`: +- Costs (~1.35 M rows) → single-pass (safely below threshold). +- Prices (~12.7 M rows) → proactive chunked (safely above threshold). + +To trigger the chunked backfill manually (e.g., recovery after a crash): + +```bash +make chunked-prices-backfill +``` + +--- + +## Querying + +### `ftk.ps1` CLI + +```bash +# Ad-hoc query (Hub DB by default) +pwsh scripts/ftk.ps1 query "Costs() | summarize sum(EffectiveCost) by ServiceName | top 10 by sum_EffectiveCost" + +# List available named catalog queries +pwsh scripts/ftk.ps1 list + +# Run a named catalog query +pwsh scripts/ftk.ps1 run savings-summary-report +pwsh scripts/ftk.ps1 run savings-summary-report --format csv +pwsh scripts/ftk.ps1 run savings-summary-report --start 2026-01-01 --end 2026-06-30 + +# Inspect schema +pwsh scripts/ftk.ps1 schema +pwsh scripts/ftk.ps1 schema -Tables +pwsh scripts/ftk.ps1 schema -Functions +pwsh scripts/ftk.ps1 tables Costs_final_v1_2 + +# Override database (e.g., query Ingestion raw tables) +pwsh scripts/ftk.ps1 query "Costs_raw | count" -Database Ingestion +``` + +The CLI connects to `http://localhost:8082` by default. Override with the `HOST_PORT` +environment variable or the `-Endpoint` parameter. + +### Agent skill + +For agent-driven analytics, the `finops-hub-local` agent skill wraps the same CLI +into structured tool calls. See +[`../agent-skills/finops-hub-local/SKILL.md`](../agent-skills/finops-hub-local/SKILL.md). + +--- + +## Dashboard + +The included `dashboard.json` is the upstream FinOps hub dashboard with the cluster +URI changed to `http://localhost:8082` and the database set to `Hub`. All tiles and +KQL queries are unmodified. + +**Proven path (CLI):** `ftk.ps1 query` and `ftk.ps1 run` work today with no browser +dependency. + +**Best-effort path (ADX web UI):** Import `dashboard.json` via +**https://dataexplorer.azure.com → Dashboards → Import dashboard from file**. Note +that browsers apply strict mixed-content rules: connecting an HTTPS page to a plain +`http://localhost:8082` endpoint may be blocked. See +[notes/dashboard.md](notes/dashboard.md) for step-by-step instructions, the +mixed-content caveat, and the ngrok HTTPS workaround. + +--- + +## Limits + +These numbers were measured on `MEM_LIMIT=16g`, amd64-on-Rosetta (Apple Silicon), +with a dataset of approximately 1,350,561 Cost rows and 12,735,587 Price rows plus +open-data lookups. Your results vary by `MEM_LIMIT` and dataset size. + +| Metric | Measured value | +|--------|----------------| +| Full ingest wall-clock (~28 M total rows) | ~7.4 min (443 s) | +| Idle working set after full ingest | ~12.4 GiB (~78% of 16 GiB) | +| Chunked backfill peak (Prices, per-extent) | ~14.3–14.7 GiB (~89–92% of 16 GiB) | +| Single-pass Prices transform (~12.7 M rows) | OOMs at 16 GiB — this is why auto-chunking triggers above 2 M rows | +| Single-pass Costs transform (~1.35 M rows) | Completes comfortably within 16 GiB | + +A fuller row-count → memory curve will be published when T-5000.6 is complete. + +--- + +## Troubleshooting + +### Port 8082 already in use + +```bash +lsof -nP -iTCP:8082 -sTCP:LISTEN +``` + +Set `HOST_PORT=` in `.env` and re-run `make up`. Do not use port 8080. + +### Container does not become healthy in 90 s + +- **Apple Silicon:** ensure Rosetta is enabled (Docker Desktop → Settings → General). +- **Logs:** `make logs` — look for `Database NetDefaultDB has been created and is + answering queries`. +- **Memory:** `docker stats --no-stream kustainer` — raise `MEM_LIMIT` in `.env` if + near the ceiling. After editing `.env`, always use `make down && make up` (never + `--force-recreate`; see below). + +### `Ingestion` or `Hub` DB missing after `make up` + +After a `make down && make up` cycle the databases should reattach from `kustainer-data/`. +If they are missing: + +1. Run `make nuke` to remove the data volume. +2. Run `make up && make load-ftk-kql && make ingest` to rebuild from scratch. + +If you used `docker compose up --force-recreate` the metadata volume was torn down +and cannot be reattached. `make nuke` + full reload is the only recovery path. + +### Row count is 0 after `make ingest` + +Check whether the update policy was left disabled by a crashed run: + +```bash +make kql KQL_DB=Ingestion QUERY=".show table Costs_final_v1_2 policy update" +``` + +If `IsEnabled` is `false`, re-enable the policy and manually trigger the backfill. +`ingest.ps1` will refuse to run until the policy is restored; pass +`--force-policy-recapture` only if you have manually verified the disabled state is +safe. + +### Prices backfill OOM (container exits under load) + +Raise `MEM_LIMIT` to `16g` in `.env`: + +```bash +# Edit .env, then bounce: +make down && make up +make ingest +``` + +If the OOM persists at 16 GiB (rare, larger datasets), use the manual chunked path: + +```bash +make chunked-prices-backfill +``` + +`ingest.ps1` normally handles the reactive fallback automatically; this target is the +manual override. + +### Checksum mismatch guard + +If you modified files in an already-ingested run directory, `ingest.ps1` will refuse +with a checksum error. This is the **fail-fast double-ingest guard**: mutating an +ingested file is not a safe overwrite path. Stage the corrected files under a **new +`` directory** and re-run `make ingest`. + +### Rosetta SIGSEGV (rare crashes under heavy load) + +`restart: unless-stopped` in `docker-compose.yml` restarts the container automatically. +The `kustainer-data/` volume is preserved and the databases reattach. `ingest.ps1`'s +chunked-backfill retry loop catches the resulting `RemoteDisconnected` error, waits for +the engine to recover, drops any partial output, and retries. No operator action is +needed unless the same chunk fails repeatedly (lower `--extents-per-batch` or raise +`MEM_LIMIT`). + +--- + +## Parity status + +The parity check suite validates that the local transform produces output identical to +what the FTK pipeline would produce. Current status: **11 pass / 0 fail / 2 manual** +of 13 checks. See [notes/parity-gaps.md](notes/parity-gaps.md) for details. + +Run the suite: + +```bash +make parity +``` + +--- + +## Makefile reference + +| Target | What it does | +|--------|--------------| +| `make up` | Bring Kustainer up and block until the healthcheck passes. | +| `make down` | Stop the container (data volume preserved). | +| `make nuke` | Destructive full reset — removes the `kustainer-data/` volume. | +| `make logs` | Tail Kustainer container logs. | +| `make kql QUERY='...'` | One-shot KQL against Hub (override: `KQL_DB=Ingestion`). | +| `make load-ftk-kql` | Load FTK KQL + open data into Ingestion + Hub (idempotent). | +| `make ingest` | Bulk-ingest all Parquet exports under `export/`. | +| `make ingest SCOPE=` | Ingest one scope only. | +| `make ingest SCOPE= PERIOD=

` | Ingest one scope + period. | +| `make ingest-status` | Show `Ingest_Manifest` summary (files + rows per scope/type). | +| `make parity` | Run the parity check suite (exit 0 iff all checks pass). | +| `make chunked-prices-backfill` | Manual per-extent Prices backfill (auto-triggered by threshold). | +| `make help` | Print all targets. | diff --git a/src/templates/finops-hub-local/notes/ftk-kql-adaptations.md b/src/templates/finops-hub-local/notes/ftk-kql-adaptations.md new file mode 100644 index 000000000..d42323e14 --- /dev/null +++ b/src/templates/finops-hub-local/notes/ftk-kql-adaptations.md @@ -0,0 +1,223 @@ +# FTK KQL adaptations for the local stack + +There are **two distinct adaptation layers** in the local stack. This document covers +both. + +| Layer | Where it runs | Purpose | +|-------|--------------|---------| +| **Load-time** | `scripts/load-ftk-kql.ps1` | Loads upstream FTK KQL into the emulator at setup time (idempotent). | +| **On-read** | `scripts/ftk.ps1` `Convert-CatalogQuery` (lines 341–382) | Adapts catalog `.kql` files at query time before posting to the emulator. | + +The upstream KQL scripts live in +`src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/scripts/`. + +**Design constraint:** only two categories of adaptation are permitted in the load-time +layer: + +1. Replace ARM/Bicep deployment-time templating macros with concrete local values. +2. Adjust for the two-database topology that the local stack preserves (see below). + +No transform logic is modified. If an upstream FTK function has a bug, it is logged +as a gap — not patched locally. + +--- + +## Topology note + +FTK deploys to ADX as a **two-database** system: an `Ingestion` database (raw + transform ++ final tables) and a `Hub` database (view functions). The local stack preserves this +topology exactly — the emulator runs both `Ingestion` and `Hub` databases, and +`database('Ingestion').*` cross-references inside Hub view functions resolve in the +emulator the same way they do in ADX. + +Earlier prototype notes described collapsing these into a single local `FtkLocal` database. +That design was superseded. The current stack uses the two-database topology throughout. + +--- + +## Adaptation 1 — Macro substitution: `$$rawRetentionInDays$$` → `3650` + +**File:** `IngestionSetup_RawTables.kql` + +`$$rawRetentionInDays$$` is the FTK Bicep deployment-time macro substituted by the +ARM template at hub-deploy time. Kustainer rejects the literal macro syntax with a +`SyntaxException`. The loader substitutes `3650` (10 years — effectively "never +delete" for a local dev database). + +**Implementation:** `scripts/load-ftk-kql.ps1` applies the substitution via a +`subst_macros` transform pass on `IngestionSetup_RawTables.kql` only. + +--- + +## Adaptation 2 — Two-database cross-references pass through unmodified + +**Files:** `HubSetup_v1_2.kql`, `HubSetup_OpenData.kql` + +No rewrite is needed. The local emulator runs both `Ingestion` and `Hub` databases, +so `database('Ingestion').X` inside Hub view functions resolves correctly — the same +as in a real ADX deployment. The loader runs `HubSetup_v1_2.kql` and +`HubSetup_OpenData.kql` against the `Hub` database without modification. + +--- + +## Adaptation 3 — Empty v1_0 final-table stubs + +**Source:** synthesized in-memory by `scripts/load-ftk-kql.ps1` + +The v1_2 view functions in `HubSetup_v1_2.kql` union v1_2 final tables with v1_0 +final tables for backward compatibility. Kustainer eagerly resolves function bodies +at creation time, so without the v1_0 tables present, +`.create-or-alter function Costs_v1_2()` fails with `General_BadRequest`. + +The loader extracts only the five `.create-merge table X_final_v1_0 (...)` DDL blocks +from `IngestionSetup_v1_0.kql` — no transforms, no update policies, no ingestion +mappings. The resulting tables are empty and serve only to satisfy the eager-compilation +check. + +Tables created (empty stubs): +- `Costs_final_v1_0` +- `Prices_final_v1_0` +- `CommitmentDiscountUsage_final_v1_0` +- `Recommendations_final_v1_0` +- `Transactions_final_v1_0` + +Because the v1_0 tables are empty, union queries are semantically identical to +v1_2-only queries — no data is hidden or duplicated. + +--- + +## Adaptation 4 — Splitter: trailing inter-command comments stripped + +This is a feature of the loader's command splitter, not a modification to FTK content. + +FTK KQL scripts intersperse top-level commands with `//` comment lines between them. +The splitter accumulates lines into commands until the next line that starts a new +command (`^\.[a-z]`). Without special handling, the trailing `// ` line before +the next command gets glued onto the end of the preceding command body, causing +Kustainer's management endpoint to reject it with `General_BadRequest`. + +The splitter strips trailing blank lines and comment-only lines from each accumulated +command before posting. The commands themselves are submitted verbatim. This resolved +all 31 false failures observed across `Common.kql`, `IngestionSetup_HubInfra.kql`, +`IngestionSetup_v1_2.kql`, `HubSetup_v1_2.kql`, and `HubSetup_OpenData.kql`. + +--- + +## Adaptation 5 — Database creation prefix + +Not in any FTK script — the loader adds this as a prerequisite before loading any +FTK content. A clean checkout has no databases; the loader creates `Ingestion` and +`Hub` with their `persist` paths matching what `docker-compose.yml` mounts at +`/kustodata`. On re-runs, the `EntityNameAlreadyExistsException` response (HTTP 400) +is treated as "already exists — idempotent OK". + +--- + +## Adaptation 6 — Open-data CSV column mapping + +**Files:** `PricingUnits.csv`, `Regions.csv`, `ResourceTypes.csv`, `Services.csv` +(from `src/open-data/`) + +The open-data CSV files were generated against an older table schema than the one +declared in `IngestionSetup_HubInfra.kql`. The HubInfra DDL uses FOCUS-aligned column +names (`x_PricingUnitDescription`, `x_ResourceType`, etc.), but the CSVs still use +legacy raw names (`UnitOfMeasure`, `ResourceType`, etc.). In a production hub +deployment an Azure Function applies this rename; that function is not part of any KQL +script. + +The loader applies the same rename at load time. Key mappings: + +| Table | CSV column | Table column | +|-------|-----------|--------------| +| `PricingUnits` | `UnitOfMeasure` | `x_PricingUnitDescription` | +| `PricingUnits` | `PricingBlockSize` | `x_PricingBlockSize` | +| `PricingUnits` | `DistinctUnits` | `PricingUnit` | +| `Regions` | `OriginalValue` | `ResourceLocation` | +| `ResourceTypes` | `ResourceType` | `x_ResourceType` | +| `ResourceTypes` | `Icon` | `IconUri` | +| `Services` | `ConsumedService` | `x_ConsumedService` | +| `Services` | `ResourceType` | `x_ResourceType` | +| `Services` | `PublisherType` | `x_PublisherCategory` | +| `Services` | `Environment` | `x_Environment` | +| `Services` | `ServiceModel` | `x_ServiceModel` | + +The loader reads each CSV, projects to the target column order, clears the destination +table (idempotency), and ingests inline into the FTK-declared table schema. + +--- + +## Drops: none + +Nothing was dropped from the FTK scripts on grounds of "Kustainer doesn't support +this". Streaming ingestion policy commands (`alter table X policy streamingingestion +disable`) were empirically verified to be accepted by Kustainer (HTTP 200). No +`.alter cluster`, managed-identity, EventGrid, or other cluster-level commands appear +in the seven scripts the loader processes. + +--- + +## On-read adapter — `ftk.ps1 run` (`Convert-CatalogQuery`, lines 341–382) + +`ftk.ps1 run` adapts catalog `.kql` files at query time before posting them to the +emulator. This is a separate layer from the load-time adaptations above. The function +`Convert-CatalogQuery` applies the following rules in order: + +### Rule 1 — Date-window retarget (lines 352–359) + +Rewrites `let startDate = ...;` and `let endDate = ...;` bindings in catalog queries +to match the `--start` / `--end` CLI arguments (or the last complete month when no +range is supplied). This lets time-bounded catalog queries work locally without +editing the `.kql` files. + +### Rule 1b — `--param` typed-literal override (lines 361–371) + +Rewrites any top-level `let = ...;` binding when `--param name=value` is +supplied on the CLI. Values are serialised to KQL typed literals (datetime, timespan, +bool, int, real, or string). Throws if `name` does not match a top-level `let` in the +query (no silent no-ops). + +### Rule 2 — `decimal`→`real` normalisation (lines 373–376) + +``` +todecimal('') → real(null) +todecimal( → toreal( +decimal( → real( +``` + +Kustainer's free image does not support the `decimal` scalar type. The FTK query +catalog uses `decimal()` for monetary columns in some queries. This substitution is +lossless for the numeric range involved and is the verified compatibility fix. + +### Rule 3 — Project-away tolerance (lines 378–379) + +If a catalog query contains a `project-away` clause and calls a hub view function +(`Costs()`, `Prices()`, `Recommendations()`, or `Transactions()`), the adapter +inspects the live function schema and silently drops any column names from the +`project-away` list that do not exist in the local schema. This prevents errors when +an upstream catalog query projects away a column the local Hub does not yet expose. + +### Cross-database references — pass through unmodified + +Catalog queries that explicitly reference `database('Ingestion').X` are **not +rewritten**. The two-database topology means `database('Ingestion').*` resolves +correctly in the emulator (Hub functions hold live cross-DB references to Ingestion +final tables). Stripping the prefix would break any catalog query that legitimately +targets an Ingestion table directly. This is consistent with the load-time behaviour +(the Hub KQL bundle is also submitted verbatim). + +--- + +## Documented upstream FTK behavior (not adapted — logged only) + +### `.alter table Costs_raw policy update` collision + +`IngestionSetup_v1_2.kql` calls `.alter table Costs_raw policy update` twice (once for +the `ActualCosts_raw` chain, once for `AmortizedCosts_raw`). Because `.alter` replaces +rather than merges the policy, the second call overwrites the first. The result is that +only the `AmortizedCosts_raw → AmortizedCosts_transform_v1_2()` chain is active after +loading. + +This behavior is identical on ADX and on Kustainer. The local test dataset uses +FOCUS-format exports (not the legacy `ActualCosts`/`AmortizedCosts` format), so this +does not affect parity results. Documented here for awareness; not patched locally per +the "no transform modifications" rule. diff --git a/src/templates/finops-hub-local/notes/parity-gaps.md b/src/templates/finops-hub-local/notes/parity-gaps.md new file mode 100644 index 000000000..ace93a7b9 --- /dev/null +++ b/src/templates/finops-hub-local/notes/parity-gaps.md @@ -0,0 +1,178 @@ +# Parity gaps + +This document records known differences between the local Kusto emulator stack and a +deployed FinOps hub, as measured by the `tests/parity-checks.kql` suite. + +**Suite status: 11 pass / 0 fail / 2 manual** (of 13 total) + +Run the suite: + +```bash +make parity +``` + +--- + +## Actual suite run output (literal) + +Path in first line stripped to repo-relative; all other output verbatim. + +``` +Loaded 12 parity check(s) from tests/parity-checks.kql + +━━━ CHECK 1: row count parity [DB: Ingestion] ━━━ + EXPECT: final_count within 5% of raw_count (some rows may be filtered by transform). + ✓ PASS: delta = 0.0000% within 5% + +━━━ CHECK 2: BilledCost preservation [DB: Ingestion] ━━━ + EXPECT: sum(BilledCost) in final equals sum(BilledCost) in raw (no silent drops). + ✓ PASS: delta = $0.000000 + +━━━ CHECK 3: EffectiveCost zeroed for commitment-discount purchases [DB: Ingestion] ━━━ + EXPECT: sum(EffectiveCost) == 0 for ChargeCategory='Purchase' AND isnotempty(CommitmentDiscountId). + ✓ PASS: result = 0 + +━━━ CHECK 4: data-quality flag dictionary populates [DB: Ingestion] ━━━ + EXPECT: x_SourceChanges contains documented codes (MissingContractedCost, + ? MANUAL: cols=['first_code', 'Count'], rows[:5]=[['XEffectiveUnitPriceRoundingError', 1100011], ['LegacyFocusVersion', 117862], ['ContractedCostLessThanEffectiveCost', 110171], ['ListCostLessThanContractedCost', 19967], ['MissingContractedCost', 1887]] + rows[:3] = [['XEffectiveUnitPriceRoundingError', 1100011], ['LegacyFocusVersion', 117862], ['ContractedCostLessThanEffectiveCost', 110171]] + +━━━ CHECK 5: AmortizationClass labels amortized usage [DB: Ingestion] ━━━ + EXPECT: ZERO rows where ChargeCategory='Usage' AND isnotempty(CommitmentDiscountId) + ✓ PASS: result = 0 + +━━━ CHECK 6: TotalSavings populated for services with reservations [DB: Ingestion] ━━━ + EXPECT: sum(x_TotalSavings) > 0 for services that typically have RIs/SPs + ✓ PASS: top row sum_TotalSavings = 4694.06850131341 + +━━━ CHECK 7: parse_resourceid populates x_ResourceGroupName when ARM path has one [DB: Ingestion] ━━━ + EXPECT: ZERO rows where isempty(x_ResourceGroupName) AND isnotempty(ResourceId) + ✓ PASS: result = 0 + +━━━ CHECK 8: PublisherName normalized (no 'Microsoft Corporation') [DB: Ingestion] ━━━ + EXPECT: 'Microsoft Corporation' should be normalized to 'Microsoft' by the + ✓ PASS: result = 0 + +━━━ CHECK 9: Hub view function returns plausible totals [DB: Hub] ━━━ + EXPECT: Costs_v1_2 | summarize sum(EffectiveCost) by ServiceCategory returns + ✓ PASS: top row sum_EffectiveCost = 306236.631384515 + +━━━ CHECK 10: Costs_v1_2 vs Costs_final_v1_2 BilledCost reconciliation [DB: Hub] ━━━ + EXPECT: sum(BilledCost) from Costs_v1_2 (Hub view) == sum from Costs_final_v1_2 + ✓ PASS: delta = $0.000000 + +━━━ CHECK 11: Prices row count parity per PriceType [DB: Ingestion] ━━━ + EXPECT: Consumption + ReservedInstance counts in Prices_final match Prices_raw + ? MANUAL: cols=['PriceType', 'raw_n', 'final_n', 'delta', 'delta_pct'], rows[:5]=[['Consumption', 9591718, 9591718, 0, 0], ['SavingsPlan', 2931206, 2931206, 0, 0], ['ReservedInstance', 212663, 212663, 0, 0]] + rows[:3] = [['Consumption', 9591718, 9591718, 0, 0], ['SavingsPlan', 2931206, 2931206, 0, 0], ['ReservedInstance', 212663, 212663, 0, 0]] + +━━━ CHECK 13: Prices value parity — UnitPrice/ContractedUnitPrice (Consumption + RI) [DB: Ingestion] ━━━ + EXPECT: sum(UnitPrice) in Prices_raw equals sum(ContractedUnitPrice) in Prices_final_v1_2 + ✓ PASS: delta = 0.0000% within 0.1% + +━━━ CHECK 12: Name parity — Ingestion tables + Hub functions [DB: Ingestion+Hub] ━━━ + EXPECT: No FtkLocal-named objects; required core tables and functions present. + ✓ PASS: Ingestion: 23 table(s), Hub: 34 function(s); no FtkLocal objects; all core names present + +━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ +SUMMARY: 11 pass, 0 fail, 2 manual (of 13 total) + +Manual checks above require human inspection but did not block the run. +``` + +--- + +## Per-check results + +Checks 1–11 and 13 are defined in `tests/parity-checks.kql`. CHECK 12 is a synthetic +check generated by the runner itself (not in the `.kql` file) — see +[How "of 13 total" is counted](#how-of-13-total-is-counted) below. Checks run in file +order (1–11, 13) then CHECK 12 last. + +| # | Check | DB | Result | Detail | +|---|-------|----|--------|--------| +| 1 | Row count parity | Ingestion | ✅ PASS | `Costs_raw` = `Costs_final_v1_2`; delta = 0.0% | +| 2 | BilledCost preservation | Ingestion | ✅ PASS | delta = $0.000000 | +| 3 | EffectiveCost zeroed for commitment purchases | Ingestion | ✅ PASS | result = 0 | +| 4 | Data-quality flag dictionary populates | Ingestion | ⚠️ MANUAL | Print-only by design — see [Data-quality observations](#data-quality-observations-check-4) | +| 5 | Amortized usage labeled correctly | Ingestion | ✅ PASS | 0 mislabeled rows | +| 6 | TotalSavings populated for reservation services | Ingestion | ✅ PASS | top sum = 4,694.07 | +| 7 | `parse_resourceid` extracts `x_ResourceGroupName` | Ingestion | ✅ PASS | 0 missing extractions | +| 8 | `PublisherName` normalized | Ingestion | ✅ PASS | 0 unnormalized rows | +| 9 | Hub view function returns plausible totals | Hub | ✅ PASS | top sum_EffectiveCost = 306,236.63 | +| 10 | `Costs_v1_2` vs `Costs_final_v1_2` BilledCost reconciliation | Hub | ✅ PASS | delta = $0.000000 | +| 11 | Prices row count parity per PriceType | Ingestion | ⚠️ MANUAL | Runner always flags as MANUAL (print-only); all three PriceTypes show delta = 0 on current dataset | +| 12 | Name parity — Ingestion tables + Hub functions | Ingestion+Hub | ✅ PASS | 23 Ingestion tables, 34 Hub functions; no `FtkLocal`-named objects; all core names present. **Synthetic** — not in `.kql` file; generated by runner | +| 13 | Prices value parity — UnitPrice/ContractedUnitPrice (Consumption + RI) | Ingestion | ✅ PASS | delta = 0.0000% within 0.1% automated threshold | + +--- + +## How "of 13 total" is counted + +`run-parity-checks.ps1` loads **12 checks from `tests/parity-checks.kql`** (numbered +1–11 and 13; the file skips number 12). The runner then appends **one synthetic CHECK +12** — a name-parity assertion that verifies no `FtkLocal`-prefixed objects exist and +that all expected core tables and functions are present. The summary formula +(`run-parity-checks.ps1` line 499) is: + +```powershell +"SUMMARY: … (of $($checks.Count + ($runNameParity ? 1 : 0)) total)" +# 12 file-based + 1 synthetic = 13 total +``` + +CHECK 12 always runs last (after CHECK 13) because it is appended after the +file-based checks are processed. + +--- + +## Data-quality observations (check 4) + +Check 4 is informational by design — it verifies the `x_SourceChanges` dictionary is +populated, not that it is empty. The flags are FTK's data-quality findings on the +source export data, not defects in the local transform. Observed flags on the current +dataset: + +| Flag | Rows | +|------|-----:| +| `XEffectiveUnitPriceRoundingError` | 1,100,011 | +| `LegacyFocusVersion` | 117,862 | +| `ContractedCostLessThanEffectiveCost` | 110,171 | +| `ListCostLessThanContractedCost` | 19,967 | +| `MissingContractedCost` | 1,887 | + +Consumers of `Costs_final_v1_2` can filter on these flags: + +```kql +Costs_final_v1_2 | where x_SourceChanges has 'MissingContractedCost' +``` + +--- + +## Known upstream FTK behaviors (not local gaps) + +### Prices SavingsPlan lookup fanout + +`IngestionSetup_v1_2.kql` applies a `lookup kind=leftouter` to SavingsPlan rows, +joining on `tmp_SavingsPlanKey` against a `distinct` over four price columns. When a +single Consumption SKU key maps to multiple (List, Contracted, Base) price tuples +across billing months, the corresponding SavingsPlan row fans out one-to-many. On +multi-scope or multi-period datasets this produces a small percentage surplus in +`Prices_final_v1_2` SavingsPlan rows relative to `Prices_raw`. + +On the current single-billing-account dataset all three PriceTypes show delta = 0 +(confirmed by CHECK 11 above). + +**Impact for consumers if the surplus appears on your dataset:** deduplicate with: + +```kql +Prices_final_v1_2 +| summarize arg_max(x_IngestionTime, *) by SkuPriceIdv2 +``` + +### `.alter table Costs_raw policy update` collision + +`IngestionSetup_v1_2.kql` calls `.alter table Costs_raw policy update` twice. +Because `.alter` replaces rather than merges the policy, the second call +(`AmortizedCosts_raw → AmortizedCosts_transform_v1_2()`) overwrites the first. This +affects both ADX and Kustainer identically. The FOCUS-format test data used for +parity validation is not impacted. Documented for awareness; not patched locally. diff --git a/src/templates/finops-hub-local/scripts/ftk.ps1 b/src/templates/finops-hub-local/scripts/ftk.ps1 index 4e1604c14..54b6ec254 100644 --- a/src/templates/finops-hub-local/scripts/ftk.ps1 +++ b/src/templates/finops-hub-local/scripts/ftk.ps1 @@ -375,12 +375,7 @@ function Convert-CatalogQuery { $out = $out -replace '\btodecimal\s*\(', 'toreal(' $out = $out -replace '\bdecimal\s*\(', 'real(' - # 3. cross-db reference strip — catalog queries that explicitly name database('Ingestion'). - # are stripped when running against Hub, which already resolves these via its view - # functions. This is a no-op for standard catalog queries that call Costs_v1_2() etc. - $out = $out -replace "database\(\s*'Ingestion'\s*\)\.", '' - - # 4. project-away tolerance: drop columns the local schema doesn't expose + # 3. project-away tolerance: drop columns the local schema doesn't expose $out = Invoke-ProjectAwayTolerance -Kql $out -Endpoint $Endpoint -Database $Database return $out From 7a615f7f79bb097650a69d3577e8867da0e0de6f Mon Sep 17 00:00:00 2001 From: msbrett Date: Wed, 17 Jun 2026 18:19:20 -0700 Subject: [PATCH 12/51] docs(finops-hub-local): measured performance + limits (T-5000.6) Add notes/performance.md with measured memory characterization on MEM_LIMIT=16g (amd64-on-Rosetta), instrumented via cgroup v2: cold loaded floor ~1.12 GiB (7%), hot post-ingest working set 13.99 GiB (87.4%), Prices chunked-backfill peak 15.03 GiB (94%), and the single-pass Prices transform ceiling where Kusto self-terminates at 15.12 GiB (~94.5%). Includes the row-count to memory curve, the ingest wall-clock (raw 7.0 min, full 14.0 min), and an operating-states table that answers when the engine approaches the limit (only during ingest+transform of a 28M-row dataset). Reconcile the README limits table with these measured numbers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- src/templates/finops-hub-local/README.md | 17 +- .../finops-hub-local/notes/performance.md | 215 ++++++++++++++++++ 2 files changed, 225 insertions(+), 7 deletions(-) create mode 100644 src/templates/finops-hub-local/notes/performance.md diff --git a/src/templates/finops-hub-local/README.md b/src/templates/finops-hub-local/README.md index fb4cfa7b0..01e1fb8f3 100644 --- a/src/templates/finops-hub-local/README.md +++ b/src/templates/finops-hub-local/README.md @@ -235,13 +235,16 @@ open-data lookups. Your results vary by `MEM_LIMIT` and dataset size. | Metric | Measured value | |--------|----------------| -| Full ingest wall-clock (~28 M total rows) | ~7.4 min (443 s) | -| Idle working set after full ingest | ~12.4 GiB (~78% of 16 GiB) | -| Chunked backfill peak (Prices, per-extent) | ~14.3–14.7 GiB (~89–92% of 16 GiB) | -| Single-pass Prices transform (~12.7 M rows) | OOMs at 16 GiB — this is why auto-chunking triggers above 2 M rows | -| Single-pass Costs transform (~1.35 M rows) | Completes comfortably within 16 GiB | - -A fuller row-count → memory curve will be published when T-5000.6 is complete. +| Raw file ingest wall-clock (31 parts, ~991 MB) | 419.6 s (7.0 min) | +| Full `ingest.ps1` wall-clock (raw + all backfill) | 841.6 s (14.0 min) | +| **Cold loaded** — post-restart, data on disk, extents not yet materialized (anon) | **1.12 GiB (7.0% of 16 GiB)** — this is the floor | +| **Hot post-ingest** — immediately after transform pipeline; extents in memory (anon) | **13.99 GiB (≈87% of 16 GiB)** — sustained until restart | +| Chunked backfill memory.peak (cumulative HWM, Prices per-extent) | **15.03 GiB (93.9% of 16 GiB)** | +| Single-pass Prices transform (~12.7 M rows) | anon ~14.74 GiB / memory.peak **15.12 GiB** → engine crash | +| Single-pass Costs transform (~1.35 M rows) | Completes in 53 s; peak 6.89 GiB (43% of 16 GiB) | + +See [notes/performance.md](notes/performance.md) for the full row-count → memory curve, +per-stage cgroup v2 measurements, and OOM ceiling analysis. --- diff --git a/src/templates/finops-hub-local/notes/performance.md b/src/templates/finops-hub-local/notes/performance.md new file mode 100644 index 000000000..fa36de588 --- /dev/null +++ b/src/templates/finops-hub-local/notes/performance.md @@ -0,0 +1,215 @@ +# ftklocal performance and limits + +Measured on **MEM_LIMIT=16g**, **amd64 on Rosetta** (Apple Silicon / Docker Desktop), using +the Kustainer Linux image (`mcr.microsoft.com/azuredataexplorer/kustainer-linux:latest`, +build `1.0.9656.17219`). + +Dataset: 15 cost export parts (1,350,561 rows) + 16 price sheet parts (12,735,587 rows), +plus the four open-data lookup tables shipped with the FTK schema (3,293 rows). +Total physical rows at full load: **28,175,589** across Costs_raw, Costs_final_v1_2, +Prices_raw, Prices_final_v1_2, and open-data tables. + +--- + +## Measurement method + +All memory figures are read from **cgroup v2** files inside the container: + +- `memory.max` — hard limit (confirms MEM_LIMIT setting) +- `memory.current` — total container memory (anonymous + page cache) +- `memory.stat → anon` — **non-reclaimable anonymous working set** (the number that + matters for headroom: kernel page cache is reclaimable under pressure, but `anon` is not) +- `memory.peak` — **cumulative** high-water mark since container creation; the kernel + rejected `echo 0 > /sys/fs/cgroup/memory.peak` (read-only filesystem in this + environment), so memory.peak is cumulative across all stages +- `memory.events → oom / oom_kill` — OOM event counters; these reset when the container + restarts + +Intermediate stages (3a–3c) are read from a 3-second-interval poll loop (`mem_poll.log`) +started before the ingest and stopped afterwards. Stages 1, 2, 3d, and 4 are point-in-time +reads via `docker exec kustainer`. The OOM peak (stage 5) is from a separate 2-second poll +started just before the OOM attempt. + +--- + +## Measurement table + +| Stage | Total rows in engine | anon (GiB) | anon % of 16 g | memory.peak (GiB) | OOM events | +|-------|---------------------|-----------|----------------|------------------|------------| +| 1. Empty engine (just started, no data) | 0 | 0.49 | 3.1 % | 0.51 | — | +| 2. Schema + open-data loaded (`load-ftk-kql`) | 3,293 | 0.66 | 4.1 % | 0.68 | — | +| 3a. After Costs_raw ingested (15 parts) | 1,353,854 | 1.51 ¹ | 9.5 % | 2.13 ¹ | — | +| 3b. After all raw ingested, no finals yet | 14,089,441 | 1.99 ¹ | 12.4 % | 3.98 ¹ | — | +| 3c. After Costs_final_v1_2 backfill (single-pass) | 15,440,002 | 3.88 ¹ | 24.2 % | 6.89 ¹ | — | +| 3d. After full ingest — **hot** working set | 28,175,589 | **13.99** | **87.4 %** | **15.03** | 0 | +| 4. After representative query workload | 28,175,589 | 13.95 | 87.2 % | 15.03 | 0 | +| 5. Single-pass OOM attempt (Prices, 12.7 M rows) | 28,175,589 | 14.74 ¹ | 92.1 % | **15.12 ¹** | **crash** | +| 6. Cold loaded — post-restart, data on disk ² | 28,175,589 | **1.12** | **7.0 %** | 1.58 ² | 0 | + +**Notes:** +- Stages 1, 2, 3d, 4 are exact point-in-time measurements. Stages 3a–3c and 5 are read + from the 2–3 s poll loop and should be treated as approximate (±0.1 GiB). +- memory.peak is cumulative (not per-stage); the value in each row is the cumulative + high-water mark up to that stage. Stage 6's memory.peak is a **post-restart** reading + (cgroup reset on container restart, so it reflects only the cold-start HWM, not the + earlier ingest peak). +- After the OOM crash the container restarted automatically (`restart: unless-stopped`). + The cgroup counters (oom / oom_kill) reset on restart; they read 0 in both the pre-OOM + measurement and after recovery. The crash is confirmed by the Kustainer log entry + (`Killed`, SIGKILL) and the abrupt memory.current drop observed in the poll log. +- ² Stage 6 was read after the OOM-triggered restart. The full dataset (1,350,561 Costs + + 12,735,587 Prices rows) is intact on disk and returns correct counts; extents are not + yet materialized in memory (lazy-load). This is also the expected state after a planned + `make down && make up` cycle. + +--- + +## Row-count → memory curve + +The table above shows how the non-reclaimable working set grows as data is loaded: + +``` +Stage Total rows anon GiB +--------------------------------------------------------- +Empty engine 0 0.49 +Schema + open data 3,293 0.66 +After Costs_raw ingest 1,353,854 1.51 +After all raw (no finals) 14,089,441 1.99 +After Costs_final backfill 15,440,002 3.88 +After full ingest — hot 28,175,589 13.99 +Cold loaded (post-restart) 28,175,589 1.12 +``` + +Key observations: + +1. **Raw ingestion is memory-cheap.** Loading 14 M raw rows (Costs + Prices parquet) + raises anon only from 0.66 GiB to ~2.0 GiB. The engine writes extents to disk; the + ingest buffers are transient. + +2. **The transform backfill is memory-expensive.** Single-pass Costs_final (1.35 M rows, + 53.2 s) peaks at 6.89 GiB (43 % of 16 g) but settles at 3.88 GiB. The difference + (~3 GiB) is intermediate query-execution scratch space that is released after the + `.set-or-append` completes. + +3. **Prices_final dominates the hot working set.** Loading 12.7 M transformed Prices rows + raises anon from 3.88 GiB to 13.99 GiB — a 10 GiB jump. Immediately after the + ingest+transform pipeline the engine holds ≈87.4 % of MEM_LIMIT in non-reclaimable + memory. This is a **hot** state: extents materialized by the transform. After a restart + the same data idles at **1.12 GiB (7 %)** — extents remain on disk until queried. + +4. **Queries are nearly free in memory.** Five representative queries (Costs by service, + monthly trend, Prices count, Prices by service family) changed anon by < 0.1 GiB and + did not raise memory.peak above the ingest high-water mark. + +--- + +## Ingest wall-clock + +| Phase | Duration | +|-------|----------| +| Raw file ingest (31 parquet parts, 991 MB) | 419.6 s (7.0 min) | +| Costs_final_v1_2 single-pass backfill (1.35 M rows) | 53.2 s | +| Prices_final_v1_2 chunked backfill (12.7 M rows, 21 extents × 1) | ~340 s (~5.7 min) | +| **Full `ingest.ps1` wall-clock (raw + backfill)** | **841.6 s (14.0 min)** | + +The raw file ingest time is dominated by the 16 price sheet parts (247 s total, ~15 s each +for the large monthly files). The 15 cost parts take only 76.5 s. + +--- + +## Chunked-backfill peak and single-pass OOM + +### Why auto-chunking exists + +The 2,000,000-row threshold in `BACKFILL_CHUNK_ROW_THRESHOLD` separates two regimes: + +| Mode | Prices (12.7 M rows) | Peak anon | memory.peak | Outcome | +|------|---------------------|-----------|-------------|---------| +| **Chunked (default, ≤ 1 extent/batch)** | 21 chunks × ~600 K avg rows | **14.71 GiB** | **15.03 GiB** | ✅ completes | +| **Single-pass** (threshold raised to bypass) | all 12.7 M rows at once | **14.74 GiB at crash** | **15.12 GiB** | ❌ OOM crash | + +### Chunked-backfill peak detail + +The 2-second poll loop captured anon climbing from 3.88 GiB (start of Prices chunked) to +14.71 GiB during the later large-extent chunks (~1 M rows each). The cumulative +memory.peak reached 15.03 GiB (93.9 % of 16 g) during this phase. + +The individual chunks do NOT add up linearly in memory because the engine releases +completed-extent memory between chunks. Each chunk's peak is dominated by: +- the pre-existing working set (~13–14 GiB from previously loaded extents), plus +- the in-flight transform output for the current chunk (~0.5–1 GiB per ~1 M row chunk). + +### Single-pass OOM ceiling (stage 5) + +With `BACKFILL_CHUNK_ROW_THRESHOLD` bypassed, the engine attempted to materialise all +12.7 M transformed Prices rows at once: + +- Anon reached **14.74 GiB** in the 2 s before the crash. +- memory.peak rose from 15.03 GiB → **15.12 GiB** (94.5 % of 16 g) immediately before + the crash was captured by the poll loop. +- The engine process received **SIGKILL** (confirmed by Kustainer log entry) and exited. + Docker's `OOMKilled` flag was `false` — this is Kusto's internal memory-pressure + self-termination, not a kernel cgroup hard-limit breach. +- The container restarted automatically. cgroup counters (oom / oom_kill) reset to 0. +- Recovery: databases reattached from the persistent `kustainer-data/` volume; engine + became healthy within ~10 s. + +The gap between chunked peak (15.03 GiB) and single-pass crash (15.12 GiB) is only +~0.09 GiB — there is essentially **no safe headroom** for single-pass Prices at 16 g and +this dataset size. The auto-chunking threshold exists precisely because of this margin. + +--- + +## Operating states and limits + +These tiers answer the question **"when do we approach 16 GiB?"** + +A freshly-loaded or restarted engine idles at ~7 % of MEM_LIMIT (extents on disk, lazy +materialization). You only approach the ceiling **during** the ingest+transform pipeline +for a dataset of this size. The hot working set (~87 %) is a sustained post-ingest state +that persists until the engine restarts or memory is reclaimed. + +All tiers are calibrated to `MEM_LIMIT=16g` and this specific dataset. They scale with +both `MEM_LIMIT` and dataset size. Each cell identifies the metric and source stage. + +| State | Condition | Metric (source stage) | % of MEM_LIMIT | Notes | +|-------|-----------|----------------------|----------------|-------| +| **Cold loaded** | Post-restart or fresh `make up`, before heavy queries | anon: **1.12 GiB** (stage 6) | 7.0 % | Floor; extents on disk, not yet materialized. Data intact and queryable. | +| **Hot post-ingest** | Immediately after `make ingest` completes; extents fully materialized by transform | anon: **13.99 GiB** (stage 3d) | 87.4 % | Sustained until restart or OS reclaim; this is NOT the steady-state floor | +| **Degraded** | Prices chunked-backfill high-water (during `make ingest`) | memory.peak: **15.03 GiB** (stage 3d cumulative HWM) | 93.9 % | Within 0.09 GiB of single-pass crash; system completes but headroom is minimal | +| **Ceiling** | Single-pass Prices transform attempt | anon: ~14.74 GiB / memory.peak: **15.12 GiB** → crash (stage 5) | 92.1 % anon / 94.5 % peak → crash | Kusto self-terminates (SIGKILL); container restarts automatically | + +### When to raise MEM_LIMIT + +- If your Prices dataset significantly exceeds 12.7 M rows, raise `MEM_LIMIT` before + running `make ingest`. Each additional million Prices rows adds roughly 0.5–1 GiB to the + idle working set (extrapolated from the Prices_final contribution observed here). +- `MEM_LIMIT=8g` is sufficient for Costs-only workflows (Costs_final peaks at 6.89 GiB + during backfill, then settles to ~3.9 GiB idle). It is **not sufficient** for + Prices at these dataset sizes. +- After raising `MEM_LIMIT`, update `BACKFILL_CHUNK_ROW_THRESHOLD` accordingly and + re-run the single-pass OOM test before relying on single-pass for larger row counts. + +--- + +## Environment + +| Setting | Value | +|---------|-------| +| MEM_LIMIT | 16g (17,179,869,184 bytes; confirmed via `memory.max`) | +| Kustainer image | `mcr.microsoft.com/azuredataexplorer/kustainer-linux:latest` | +| Kustainer build | `1.0.9656.17219` (2026-06-09) | +| Platform | linux/amd64 on Rosetta (Apple Silicon, Docker Desktop) | +| Host OS | macOS (Apple Silicon, M-series) | +| cgroup version | v2 (confirmed) | +| memory.peak reset | Not possible — kernel rejected write (read-only filesystem) | +| memory.peak semantics | Cumulative since container start; NOT resettable between stages | +| Poll interval (ingest) | 3 s | +| Poll interval (OOM) | 2 s | + +--- + +## Cleanliness + +No absolute host paths, billing scope IDs, storage account names, or real cluster URIs +appear in this file. Dataset size is described in row counts only. From 5ec214c4b680f1715cb9b61f007cda0704f85b13 Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 18 Jun 2026 06:46:28 -0700 Subject: [PATCH 13/51] docs(finops-hub-local): correct ADX dashboard connectivity (UAT-measured) Replace the speculative 'mixed-content blocked / not proven' caveat with measured findings: the Kusto emulator returns CORS headers for https://dataexplorer.azure.com (OPTIONS 204 + POST 200 with Access-Control-Allow-Origin), real dashboard tile queries return renderable data over that path, and dashboard.json (schema v60, 135 tiles / 87 queries / 12 pages) has zero dangling references so it imports cleanly. localhost is exempt from mixed-content blocking; the real browser gate is the Edge 143+/Chrome 142+ Local Network Access permission prompt (click Allow), with browser-flag and ngrok fallbacks documented for locked-down environments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../finops-hub-local/notes/dashboard.md | 94 ++++++++++++++----- 1 file changed, 73 insertions(+), 21 deletions(-) diff --git a/src/templates/finops-hub-local/notes/dashboard.md b/src/templates/finops-hub-local/notes/dashboard.md index 2aa1aa6cb..ce4116567 100644 --- a/src/templates/finops-hub-local/notes/dashboard.md +++ b/src/templates/finops-hub-local/notes/dashboard.md @@ -44,29 +44,55 @@ available as structured tool calls through the skill surface — no manual KQL r --- -## Best-effort path: ADX web UI dashboard import +## ADX web UI dashboard import -> ⚠️ **Mixed-content limitation — this path is NOT proven on Mac.** +> ℹ️ **`localhost` is not blocked as mixed content, but Edge/Chrome will ask permission once.** > -> `dataexplorer.azure.com` is served over HTTPS. Browsers (Chrome, Edge, Safari) apply strict -> mixed-content rules: an HTTPS page may not open active connections to a plain `http://` -> origin. Attempting to add `http://localhost:8082` as a cluster from the hosted web UI will -> likely be silently blocked or refused. **Do not assume this works until you verify it in -> your browser.** +> `dataexplorer.azure.com` is served over HTTPS, and `http://localhost` (and `127.0.0.1`) is a +> **potentially trustworthy origin** under the W3C Secure Contexts spec, so Chromium-based +> browsers (**Microsoft Edge, Google Chrome**) do *not* apply mixed-content blocking to it. +> However, current Edge (143+) and Chrome (142+) enforce **Local Network Access (LNA)**: the +> first time the page reaches `http://localhost:8082` the browser shows a **"Local Network +> Access" permission prompt** — click **Allow** and the connection proceeds for the session. +> Use Edge or Chrome for this path. Safari is stricter; if it refuses the connection, use the +> [ngrok workaround](#if-the-browser-blocks-the-connection). +> +> The emulator also returns the CORS headers the web UI requires (measured — see +> [UAT findings](#uat-findings-measured) below), so once you allow local-network access the +> cross-origin queries from `dataexplorer.azure.com` to the emulator are permitted. -If you want to try the dashboard import anyway, follow the steps below. If the cluster -connection fails, see the [ngrok workaround](#workaround-expose-the-emulator-over-https-with-ngrok). +### UAT findings (measured) -### 1. Attempt to connect the ADX web UI +Measured against the running emulator (`http://localhost:8082`, Hub DB populated with the +sample dataset): -1. Open **https://dataexplorer.azure.com** in your browser. -2. Click **+ Add cluster** (left sidebar). -3. Enter the connection URI: `http://localhost:8082` -4. Authentication: select **Anonymous** (the emulator runs no-auth HTTP). -5. Click **Add**. +| Check | Result | +|---|---| +| CORS preflight (`OPTIONS`, `Origin: https://dataexplorer.azure.com`) | ✅ `204` with `Access-Control-Allow-Origin: https://dataexplorer.azure.com`, `Access-Control-Allow-Methods: POST`, `Access-Control-Allow-Headers: content-type,x-ms-client-version,authorization` | +| Actual query (`POST /v1/rest/query`, same `Origin`) | ✅ `200` with `Access-Control-Allow-Origin: https://dataexplorer.azure.com` | +| Real tile queries return renderable data via the CORS path | ✅ Monthly trend (15 months), top services (bar), Prices (table) all return rows | +| `dashboard.json` import schema | ✅ Official ADX dashboard schema **v60**, single `manual-kusto` data source → `http://localhost:8082` / `Hub` | +| `dashboard.json` referential integrity | ✅ 135 tiles / 87 queries / 6 base queries / 12 pages, **0 dangling references** → imports cleanly | +| Private Network Access (PNA) preflight (`Access-Control-Request-Private-Network: true`) | ⚠️ Emulator does **not** return `Access-Control-Allow-Private-Network: true`. Current Edge 143+/Chrome 142+ enforce **Local Network Access (LNA)** and show a one-time **permission prompt** for the public→localhost call — click **Allow** to proceed. Locked-down/enterprise-policy builds that suppress or deny the prompt need the [LNA fallback](#if-the-browser-blocks-the-connection). | + +The protocol path is proven server-side: standard CORS is allowed, `localhost` is exempt from +mixed-content blocking in Edge/Chrome, and the dashboard file imports cleanly. The two +remaining browser-side steps are (a) the interactive Azure (AAD) sign-in and (b) clicking +**Allow** on the one-time Local Network Access prompt — both handled in the steps below. + +### 1. Connect the ADX web UI to the emulator + +1. Open **https://dataexplorer.azure.com** in **Microsoft Edge or Google Chrome** (not Safari). +2. Sign in with your Azure account (the web UI itself requires AAD; the emulator cluster does not). +3. Click **+ Add cluster** (left sidebar). +4. Enter the connection URI: `http://localhost:8082` +5. Authentication: select **Anonymous** (the emulator runs no-auth HTTP). +6. Click **Add**. If Edge/Chrome shows a **"Local Network Access"** permission prompt + (a site wants to access your local network), click **Allow**. If the cluster appears in the sidebar with `Hub` and `Ingestion` databases, the connection -succeeded and you can proceed. If the connection hangs or errors, use the ngrok workaround +succeeded and you can proceed. If the connection is refused, see +[if the browser blocks the connection](#if-the-browser-blocks-the-connection) below. ### 2. Import the dashboard @@ -87,10 +113,36 @@ confirm the Hub DB has data by running `Costs_v1_2 | take 10` in the **Query** t --- -## Workaround: expose the emulator over HTTPS with ngrok +## If the browser blocks the connection + +Two browser-side mechanisms can refuse the `https://dataexplorer.azure.com → http://localhost:8082` +call. They are independent — try the lighter fix first. + +### Option A — Local Network Access (LNA) prompt, Chromium + +Current Edge (143+) and Chrome (142+) gate public→`localhost` requests behind a **Local +Network Access permission prompt**. In the normal case you simply click **Allow** when the +prompt appears (see step 1) and the connection works — no server header or flag needed. + +If your environment **suppresses or auto-denies** the prompt (enterprise policy, or a build +where you previously clicked *Block*), relax the check for local UAT by launching the browser +with LNA/PNA enforcement disabled: + +```bash +# Edge (quit Edge first), local UAT only: +open -na "Microsoft Edge" --args --disable-features=LocalNetworkAccessChecks,PrivateNetworkAccessRespectPreflightResults + +# Chrome equivalent: +open -na "Google Chrome" --args --disable-features=LocalNetworkAccessChecks,PrivateNetworkAccessRespectPreflightResults +``` + +You can also re-allow the site via the page-info (lock/info icon) → site permissions, or reset +its Local Network Access permission. Re-enable normal enforcement after testing. + +### Option B — expose the emulator over HTTPS with ngrok -If the browser blocks the HTTP connection, tunnel the emulator behind HTTPS using -[ngrok](https://ngrok.com): +If you'd rather not change browser flags (or you're on Safari), tunnel the emulator behind +HTTPS using [ngrok](https://ngrok.com): ```bash # Install ngrok if needed: brew install ngrok/ngrok/ngrok @@ -116,8 +168,8 @@ dashboard data source to match. | Hub DB has `Costs_v1_2`, `Prices_v1_2`, etc. | ✅ Verified via `.show functions` on running emulator | | No upstream cluster names, billing IDs, or absolute paths in dashboard | ✅ Verified | | `ftk query` / `ftk run` against Hub DB | ✅ Works (HTTP REST, no browser) | -| Browser can add `http://localhost:8082` from `https://dataexplorer.azure.com` | ⚠️ **Not validated** — mixed-content blocked in most browsers | -| Dashboard tiles render data after import | ⚠️ **Not validated** — requires browser import + populated Hub data | +| Browser can add `http://localhost:8082` from `https://dataexplorer.azure.com` | ✅ **CORS allowed** (emulator returns `Access-Control-Allow-Origin: https://dataexplorer.azure.com`); `localhost` is exempt from mixed-content blocking. Edge 143+/Chrome 142+ show a one-time **Local Network Access** prompt — click **Allow**. | +| Dashboard tiles render data after import | ⚠️ **Visual render pending** — protocol path proven (tile queries return data via CORS); requires the interactive browser import + AAD sign-in + the LNA **Allow** click to confirm on screen | --- From 801981e9eb2d85a072344d0b3b51e37181c544ed Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 18 Jun 2026 07:02:40 -0700 Subject: [PATCH 14/51] docs(finops-hub-local): Windows-primary + macOS install/deploy guides + cross-platform bring-up Add per-OS install guides (notes/install-windows.md primary, notes/install-mac.md secondary) and make the README quickstart cross-platform: 'docker compose up -d --wait' (honors the container healthcheck) plus the self-contained 'pwsh scripts/*.ps1' commands replace the Unix-only 'make' front door, which remains a macOS/Linux convenience. The scripts already self-resolve HOST_PORT from .env, so they run standalone on Windows. Windows guide covers Docker Desktop + WSL 2, PowerShell 7 (not Windows PowerShell 5.1), WSL 2 memory tuning, forward-slash EXPORT_DIR, and Windows-specific troubleshooting; notes that linux/amd64 runs natively on Windows x64 (no emulation) so it outperforms the Apple-Silicon numbers in performance.md. Supporting correctness: add .gitattributes (LF for .ps1/.sh/.kql/.json/Makefile) to prevent CRLF corruption on Windows checkout, and '#Requires -Version 7.0' to the entry scripts so PowerShell 5.1 fails with a clear message instead of cryptic parser errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- src/templates/finops-hub-local/.gitattributes | 17 ++ src/templates/finops-hub-local/README.md | 34 ++-- .../finops-hub-local/notes/install-mac.md | 104 ++++++++++++ .../finops-hub-local/notes/install-windows.md | 151 ++++++++++++++++++ .../scripts/chunked-prices-backfill.ps1 | 1 + .../finops-hub-local/scripts/ftk.ps1 | 1 + .../finops-hub-local/scripts/ingest.ps1 | 1 + .../finops-hub-local/scripts/load-ftk-kql.ps1 | 1 + .../scripts/run-parity-checks.ps1 | 1 + .../finops-hub-local/scripts/settings.ps1 | 1 + 10 files changed, 302 insertions(+), 10 deletions(-) create mode 100644 src/templates/finops-hub-local/.gitattributes create mode 100644 src/templates/finops-hub-local/notes/install-mac.md create mode 100644 src/templates/finops-hub-local/notes/install-windows.md mode change 100755 => 100644 src/templates/finops-hub-local/scripts/chunked-prices-backfill.ps1 mode change 100755 => 100644 src/templates/finops-hub-local/scripts/ingest.ps1 diff --git a/src/templates/finops-hub-local/.gitattributes b/src/templates/finops-hub-local/.gitattributes new file mode 100644 index 000000000..85097a122 --- /dev/null +++ b/src/templates/finops-hub-local/.gitattributes @@ -0,0 +1,17 @@ +# ftklocal line-ending policy. +# +# Keep LF on files that break or behave inconsistently under CRLF, so a Windows +# checkout (where git may otherwise convert to CRLF) stays correct: +# - the Makefile + shell recipes require LF (bash chokes on CRLF), +# - PowerShell tolerates either but LF keeps diffs clean across platforms, +# - KQL/JSON are submitted verbatim to the engine. +*.ps1 text eol=lf +*.psm1 text eol=lf +*.psd1 text eol=lf +*.sh text eol=lf +*.kql text eol=lf +*.json text eol=lf +Makefile text eol=lf +*.yml text eol=lf +*.yaml text eol=lf +.env.example text eol=lf diff --git a/src/templates/finops-hub-local/README.md b/src/templates/finops-hub-local/README.md index 01e1fb8f3..ed808cfd2 100644 --- a/src/templates/finops-hub-local/README.md +++ b/src/templates/finops-hub-local/README.md @@ -24,11 +24,9 @@ variants), the same open data, and the same query catalog. - **Docker Desktop ≥ 4.0** — Compose v2 is required (`docker compose`, not the legacy `docker-compose`). -- **Apple Silicon (M-series) users:** enable **Rosetta** for x86/amd64 emulation. The - Kustainer image is `linux/amd64` only. Docker Desktop → Settings → General → "Use - Rosetta for x86_64/amd64 emulation on Apple Silicon". - **PowerShell 7+** (`pwsh`) — all tooling is PowerShell; there are no Python - dependencies. + dependencies. On Windows this is **PowerShell 7**, not the built-in Windows PowerShell + 5.1 (`winget install Microsoft.PowerShell`). - **FOCUS cost exports as Parquet** — staged under `export/`. See [notes/staging-contract.md](notes/staging-contract.md) for the folder layout. - Roughly **16 GiB of RAM** available for the container (the default `MEM_LIMIT` in @@ -36,24 +34,40 @@ variants), the same open data, and the same query catalog. Prices transform. See [Limits](#limits). - Host port **8082** free (configurable via `HOST_PORT` in `.env`). +**Per-platform install guides** (prerequisites, memory tuning, troubleshooting): + +- 🪟 **[Windows](notes/install-windows.md)** — primary platform; the `linux/amd64` engine + runs natively (no emulation). Covers Docker Desktop + WSL 2, PowerShell 7, and WSL memory + tuning. +- 🍎 **[macOS](notes/install-mac.md)** — secondary; Apple Silicon runs the engine under + Rosetta emulation. Covers Rosetta and Docker Desktop memory. + --- ## Quickstart +The commands below work on **Windows, macOS, and Linux** (PowerShell 7 + Docker Compose v2). +`make` targets are an optional convenience on macOS/Linux — see the +[macOS guide](notes/install-mac.md). + ```bash # From src/templates/finops-hub-local/ -cp .env.example .env # 1. review tunables (HOST_PORT, MEM_LIMIT, EXPORT_DIR) -make up # 2. start the emulator; blocks until healthcheck passes (~30 s) -make load-ftk-kql # 3. create Ingestion + Hub DBs, load FTK KQL + open data (idempotent) -make ingest # 4. bulk-ingest all Parquet exports under export/ +cp .env.example .env # 1. review tunables (HOST_PORT, MEM_LIMIT, EXPORT_DIR) + # (Windows: Copy-Item .env.example .env) +docker compose up -d --wait # 2. start the emulator; blocks until the healthcheck passes +pwsh scripts/load-ftk-kql.ps1 # 3. create Ingestion + Hub DBs, load FTK KQL + open data (idempotent) +pwsh scripts/ingest.ps1 # 4. bulk-ingest all Parquet exports under export/ # 5. query your FOCUS data pwsh scripts/ftk.ps1 query "Costs() | summarize TotalCost = sum(EffectiveCost) by ServiceName | top 10 by TotalCost" ``` -After step 4 completes, the Hub database holds the same view functions a deployed hub -exposes. `make ingest-status` shows a summary of what was ingested. +`docker compose up -d --wait` is the cross-platform equivalent of `make up` — it uses the +container's healthcheck and returns only once the engine is answering. After step 4 +completes, the Hub database holds the same view functions a deployed hub exposes; +`pwsh scripts/ingest.ps1 -DryRun` previews ingest and `make ingest-status` (macOS/Linux) +summarizes what was loaded. --- diff --git a/src/templates/finops-hub-local/notes/install-mac.md b/src/templates/finops-hub-local/notes/install-mac.md new file mode 100644 index 000000000..b20208d43 --- /dev/null +++ b/src/templates/finops-hub-local/notes/install-mac.md @@ -0,0 +1,104 @@ +# Install and run ftklocal on macOS + +macOS is a supported secondary platform. On **Apple Silicon** the Kustainer image +(`linux/amd64`) runs under emulation, so ingest and transform are slower than on Windows +or Intel; the measured numbers in [performance.md](performance.md) were taken on Apple +Silicon and represent that emulated path. On **Intel Macs** the image runs natively. + +Everything runs from a normal terminal with PowerShell 7 (`pwsh`). The `make` targets are +available on macOS as a convenience, but the `docker compose` + `pwsh` commands below work +identically and are the cross-platform reference. + +--- + +## Prerequisites + +| Requirement | Notes | +|-------------|-------| +| **Docker Desktop ≥ 4.0** | Compose v2 is required (`docker compose`, not legacy `docker-compose`). | +| **Apple Silicon: enable Rosetta** | Docker Desktop → Settings → General → "Use Rosetta for x86_64/amd64 emulation on Apple Silicon". The Kustainer image is `linux/amd64` only. Intel Macs need nothing extra. | +| **PowerShell 7+** (`pwsh`) | `brew install --cask powershell`. All tooling is PowerShell; there are no Python dependencies. | +| **Git** | To clone the repository (`brew install git`, or use the Xcode command-line tools). | +| **~16 GiB RAM available to Docker** | Docker Desktop → Settings → Resources → Memory. The full Prices transform needs it; Costs-only datasets work at 8 GiB. See [Limits](performance.md). | +| **Host port 8082 free** | Configurable via `HOST_PORT` in `.env`. | +| **FOCUS cost exports as Parquet** | Staged under `export/`. See [staging-contract.md](staging-contract.md). | + +--- + +## Install + +```bash +# 1. Clone the toolkit and move into the template +git clone https://github.com/microsoft/finops-toolkit.git +cd finops-toolkit/src/templates/finops-hub-local + +# 2. Create your local config from the template +cp .env.example .env +``` + +Edit `.env` to point `EXPORT_DIR` at the folder holding your FOCUS Parquet exports: + +```ini +# .env +EXPORT_DIR=./export # or an absolute path, e.g. ~/cost-exports +HOST_PORT=8082 +MEM_LIMIT=16g +``` + +--- + +## Run + +Two equivalent paths — `make` (macOS/Linux convenience) or the cross-platform commands. + +### Using make + +```bash +make up # start the emulator; blocks until healthy +make load-ftk-kql # create Ingestion + Hub DBs, load FTK KQL + open data +make ingest # bulk-ingest the Parquet exports under export/ +pwsh scripts/ftk.ps1 query "Costs() | summarize TotalCost = sum(EffectiveCost) by ServiceName | top 10 by TotalCost" +``` + +### Cross-platform (no make) + +```bash +docker compose up -d --wait # start + block until the healthcheck passes +pwsh scripts/load-ftk-kql.ps1 +pwsh scripts/ingest.ps1 +pwsh scripts/ftk.ps1 query "Costs() | summarize TotalCost = sum(EffectiveCost) by ServiceName | top 10 by TotalCost" +``` + +Useful follow-ups: + +```bash +pwsh scripts/ftk.ps1 list # list the reusable query catalog +pwsh scripts/ftk.ps1 run # run a named catalog query +make ingest-status # summary of what was ingested +make parity # validate transform parity +``` + +--- + +## Stop, reset, and update + +```bash +docker compose down # stop, keep data (or: make down) +make nuke # full destructive reset (down -v + remove kustainer-data/) +git pull && docker compose up -d --wait && pwsh scripts/load-ftk-kql.ps1 # update +``` + +--- + +## Troubleshooting (macOS-specific) + +| Symptom | Fix | +|---------|-----| +| Very slow ingest / transform on Apple Silicon | Expected — `linux/amd64` runs under emulation. Confirm Rosetta is enabled (above). See [performance.md](performance.md) for the measured envelope. | +| Container exits under the Prices transform | Raise Docker Desktop's memory (Settings → Resources) to ≥ 16 GiB. The ingest already chunks large tables. | +| `pwsh: command not found` | `brew install --cask powershell`. | +| Rare SIGSEGV crashes under heavy load | A known Rosetta emulation issue; the container restarts automatically. See the main [README](../README.md) troubleshooting section. | +| Port 8082 already in use | Set a different `HOST_PORT` in `.env`, then bring the stack back up. | + +For the ADX web-UI dashboard, see [dashboard.md](dashboard.md). Use Edge or Chrome; Safari +is stricter about local connections. diff --git a/src/templates/finops-hub-local/notes/install-windows.md b/src/templates/finops-hub-local/notes/install-windows.md new file mode 100644 index 000000000..a8f6cf8a8 --- /dev/null +++ b/src/templates/finops-hub-local/notes/install-windows.md @@ -0,0 +1,151 @@ +# Install and run ftklocal on Windows + +This is the primary supported platform. The Kustainer image is `linux/amd64`, which runs +**natively** on Windows x64 (via the WSL 2 backend) with no emulation — so ingest and +transform are faster here than on Apple Silicon. The numbers in +[performance.md](performance.md) were measured under Apple-Silicon emulation and are a +conservative floor for Windows. + +Everything below runs from a normal **PowerShell 7** prompt. You do **not** need `make`, +WSL shell access, or any Unix tooling — `docker compose` and the `.ps1` scripts are the +whole surface. + +--- + +## Prerequisites + +| Requirement | Notes | +|-------------|-------| +| **Docker Desktop ≥ 4.0** with the **WSL 2 backend** | Settings → General → "Use the WSL 2 based engine". Compose v2 ships with it (`docker compose`, not legacy `docker-compose`). | +| **PowerShell 7+** (`pwsh`) | **Windows PowerShell 5.1 will not work** — the scripts use 7-only syntax. Install with `winget install Microsoft.PowerShell`, then use the `pwsh` command (not `powershell`). | +| **Git** | `winget install Git.Git`, to clone the repository. | +| **~16 GiB RAM available to Docker** | The full Prices transform needs it. Costs-only datasets work at 8 GiB. See [memory tuning](#tune-wsl-2-memory) below — on Windows the limit is governed by both the container `MEM_LIMIT` **and** WSL 2's own memory cap. | +| **Host port 8082 free** | Configurable via `HOST_PORT` in `.env`. | +| **FOCUS cost exports as Parquet** | Staged under `export/`. See [staging-contract.md](staging-contract.md). | + +> **Why PowerShell 7, not the built-in PowerShell?** Windows ships *Windows PowerShell 5.1* +> (`powershell.exe`). ftklocal's scripts use null-coalescing (`??`) and ternary operators +> that only exist in *PowerShell 7+* (`pwsh.exe`). Running them under 5.1 fails with parser +> errors. Always invoke them with `pwsh`. + +--- + +## Tune WSL 2 memory + +Docker Desktop on Windows runs containers inside a WSL 2 virtual machine. Two limits apply: + +1. The container limit — `MEM_LIMIT` in `.env` (default `16g`). +2. The WSL 2 VM limit — by default WSL 2 may cap memory below what the container needs. + +If the container is killed under load, raise the WSL 2 cap. Create or edit +`%UserProfile%\.wslconfig`: + +```ini +[wsl2] +memory=20GB +``` + +Then, from PowerShell, restart WSL and Docker Desktop: + +```powershell +wsl --shutdown +# then relaunch Docker Desktop +``` + +Give the VM a couple of GiB more than `MEM_LIMIT` so the engine has headroom. See +[performance.md](performance.md) for the measured working-set curve. + +--- + +## Install + +```powershell +# 1. Clone the toolkit and move into the template +git clone https://github.com/microsoft/finops-toolkit.git +cd finops-toolkit\src\templates\finops-hub-local + +# 2. Create your local config from the template +Copy-Item .env.example .env +``` + +Edit `.env` to point `EXPORT_DIR` at the folder holding your FOCUS Parquet exports. On +Windows, use a forward-slash path or an absolute drive path — both work with Docker +Desktop's bind mounts: + +```ini +# .env (examples — pick one) +EXPORT_DIR=./export +EXPORT_DIR=D:/cost-exports +HOST_PORT=8082 +MEM_LIMIT=16g +``` + +> Keep forward slashes in `EXPORT_DIR`. A backslash path like `D:\exports` can be +> mis-parsed by Compose variable expansion; `D:/exports` (or `C:/finops/exports`) is safe. + +--- + +## Run + +```powershell +# 3. Start the emulator and block until it reports healthy +docker compose up -d --wait + +# 4. Create the Ingestion + Hub databases, load the FTK KQL and open data (idempotent) +pwsh scripts\load-ftk-kql.ps1 + +# 5. Bulk-ingest every Parquet export under export\ (idempotent; chunks large tables) +pwsh scripts\ingest.ps1 + +# 6. Query your FOCUS data (Hub database, view functions) +pwsh scripts\ftk.ps1 query "Costs() | summarize TotalCost = sum(EffectiveCost) by ServiceName | top 10 by TotalCost" +``` + +`docker compose up -d --wait` uses the container's built-in healthcheck and returns only +once the engine answers queries — it is the cross-platform equivalent of `make up`. + +Useful follow-ups: + +```powershell +pwsh scripts\ftk.ps1 list # list the reusable query catalog +pwsh scripts\ftk.ps1 run # run a named catalog query +pwsh scripts\ingest.ps1 -DryRun # preview what would be ingested +pwsh scripts\run-parity-checks.ps1 # validate transform parity +``` + +--- + +## Stop, reset, and update + +```powershell +# Stop the emulator, keep the data +docker compose down + +# Full reset (destructive): remove the container and all ingested data +docker compose down -v +Remove-Item -Recurse -Force .\kustainer-data # if a bind-mounted data dir remains + +# Update to a newer toolkit version +git pull +docker compose up -d --wait +pwsh scripts\load-ftk-kql.ps1 # re-applies schema (idempotent) +``` + +--- + +## Troubleshooting (Windows-specific) + +| Symptom | Fix | +|---------|-----| +| `pwsh : The term 'pwsh' is not recognized` | PowerShell 7 isn't installed or isn't on `PATH`. `winget install Microsoft.PowerShell`, then open a new terminal. | +| Parser errors like `Unexpected token '??'` | You're running under Windows PowerShell 5.1. Use `pwsh`, not `powershell`. | +| `running scripts is disabled on this system` | Invoking as `pwsh scripts\x.ps1 ` normally bypasses this. If you dot-source instead, set `Set-ExecutionPolicy -Scope Process RemoteSigned` for the session. | +| `docker compose` not found | Update Docker Desktop (Compose v2 is bundled). Don't use the legacy `docker-compose`. | +| Container exits / killed under the Prices transform | WSL 2 memory cap too low — see [Tune WSL 2 memory](#tune-wsl-2-memory). The ingest already chunks large tables; raising the VM cap removes the ceiling. | +| Port 8082 already in use | Set a different `HOST_PORT` in `.env`, then `docker compose up -d --wait`. | +| `Ingestion`/`Hub` DB missing after start | Run `pwsh scripts\load-ftk-kql.ps1`. If it persists, full-reset (above) and reload. | +| Tiles/queries return 0 rows | You haven't ingested yet — run `pwsh scripts\ingest.ps1`. | + +For the ADX web-UI dashboard on Windows, see [dashboard.md](dashboard.md) — Edge and Chrome +are the supported browsers and will prompt once for **Local Network Access** (click +**Allow**). diff --git a/src/templates/finops-hub-local/scripts/chunked-prices-backfill.ps1 b/src/templates/finops-hub-local/scripts/chunked-prices-backfill.ps1 old mode 100755 new mode 100644 index 1bce6856b..23c7d42b8 --- a/src/templates/finops-hub-local/scripts/chunked-prices-backfill.ps1 +++ b/src/templates/finops-hub-local/scripts/chunked-prices-backfill.ps1 @@ -1,4 +1,5 @@ #!/usr/bin/env pwsh +#Requires -Version 7.0 <# .SYNOPSIS Standalone chunked backfill for Prices_final_v1_2. diff --git a/src/templates/finops-hub-local/scripts/ftk.ps1 b/src/templates/finops-hub-local/scripts/ftk.ps1 index 54b6ec254..2ad2b82c2 100644 --- a/src/templates/finops-hub-local/scripts/ftk.ps1 +++ b/src/templates/finops-hub-local/scripts/ftk.ps1 @@ -1,4 +1,5 @@ #!/usr/bin/env pwsh +#Requires -Version 7.0 <# .SYNOPSIS ftk - a tiny CLI for the local FinOps Toolkit (Kustainer) stack. diff --git a/src/templates/finops-hub-local/scripts/ingest.ps1 b/src/templates/finops-hub-local/scripts/ingest.ps1 old mode 100755 new mode 100644 index 9a72fb991..b17bb99cc --- a/src/templates/finops-hub-local/scripts/ingest.ps1 +++ b/src/templates/finops-hub-local/scripts/ingest.ps1 @@ -1,4 +1,5 @@ #!/usr/bin/env pwsh +#Requires -Version 7.0 <# .SYNOPSIS Bulk-ingest FOCUS Cost + Price Sheet parquet exports into local Kustainer raw tables. diff --git a/src/templates/finops-hub-local/scripts/load-ftk-kql.ps1 b/src/templates/finops-hub-local/scripts/load-ftk-kql.ps1 index b84611a47..1775adb04 100644 --- a/src/templates/finops-hub-local/scripts/load-ftk-kql.ps1 +++ b/src/templates/finops-hub-local/scripts/load-ftk-kql.ps1 @@ -1,4 +1,5 @@ #!/usr/bin/env pwsh +#Requires -Version 7.0 <# .SYNOPSIS Load the FinOps Toolkit analytics KQL into local Kustainer (two-DB topology). diff --git a/src/templates/finops-hub-local/scripts/run-parity-checks.ps1 b/src/templates/finops-hub-local/scripts/run-parity-checks.ps1 index 01fcf3689..3f6c3b329 100644 --- a/src/templates/finops-hub-local/scripts/run-parity-checks.ps1 +++ b/src/templates/finops-hub-local/scripts/run-parity-checks.ps1 @@ -1,4 +1,5 @@ #!/usr/bin/env pwsh +#Requires -Version 7.0 <# .SYNOPSIS Runs FTK-Local parity checks against the local Kustainer databases. diff --git a/src/templates/finops-hub-local/scripts/settings.ps1 b/src/templates/finops-hub-local/scripts/settings.ps1 index d929b47d6..3294accd6 100644 --- a/src/templates/finops-hub-local/scripts/settings.ps1 +++ b/src/templates/finops-hub-local/scripts/settings.ps1 @@ -1,4 +1,5 @@ #!/usr/bin/env pwsh +#Requires -Version 7.0 <# .SYNOPSIS Manage the local ftklocal settings.json (hub fidelity layer). From 69912ba54818d559cb5546b85722ea6451cd0f72 Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 18 Jun 2026 07:07:56 -0700 Subject: [PATCH 15/51] style(finops-hub-local): apply Prettier formatting to docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Run repo Prettier (.prettierrc) over the finops-hub-local markdown so the docs conform to the toolkit coding standard (docs-wiki/Coding-guidelines.md): aligned table columns and underscore emphasis. Formatting only — no content, measured numbers, or links changed. 'prettier --check' now passes for all finops-hub-local markdown. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- src/templates/finops-hub-local/README.md | 109 +++++++-------- .../notes/config-container.md | 44 +++--- .../finops-hub-local/notes/dashboard.md | 42 +++--- .../notes/ftk-kql-adaptations.md | 44 +++--- .../finops-hub-local/notes/install-mac.md | 30 ++--- .../finops-hub-local/notes/install-windows.md | 42 +++--- .../finops-hub-local/notes/parity-gaps.md | 44 +++--- .../finops-hub-local/notes/performance.md | 80 +++++------ .../notes/staging-contract.md | 126 +++++++++--------- 9 files changed, 284 insertions(+), 277 deletions(-) diff --git a/src/templates/finops-hub-local/README.md b/src/templates/finops-hub-local/README.md index ed808cfd2..f7c10a38e 100644 --- a/src/templates/finops-hub-local/README.md +++ b/src/templates/finops-hub-local/README.md @@ -10,13 +10,13 @@ variants), the same open data, and the same query catalog. ## When to use this vs deploying a real hub -| Situation | Use | -|-----------|-----| -| Offline analysis, air-gapped network, or no Azure subscription | **ftklocal** | -| Developing or testing KQL changes before deploying | **ftklocal** | -| Working with FOCUS parquet already on disk | **ftklocal** | -| Production analytics for an organization | [Deploy a FinOps hub](../finops-hub/README.md) | -| Querying a live Azure Data Explorer or Fabric cluster | [FinOps hub](../finops-hub/README.md) | +| Situation | Use | +| -------------------------------------------------------------- | ---------------------------------------------- | +| Offline analysis, air-gapped network, or no Azure subscription | **ftklocal** | +| Developing or testing KQL changes before deploying | **ftklocal** | +| Working with FOCUS parquet already on disk | **ftklocal** | +| Production analytics for an organization | [Deploy a FinOps hub](../finops-hub/README.md) | +| Querying a live Azure Data Explorer or Fabric cluster | [FinOps hub](../finops-hub/README.md) | --- @@ -75,12 +75,12 @@ summarizes what was loaded. All tunables live in `.env` (gitignored). `.env.example` documents them: -| Variable | Default | Meaning | -|----------|---------|---------| -| `HOST_PORT` | `8082` | Host port mapped to the Kustainer container's HTTP endpoint. | -| `MEM_LIMIT` | `16g` | Memory ceiling for the Kustainer container. | -| `EXPORT_DIR` | `./export` | Host directory with Parquet exports (read-only mount). | -| `BACKFILL_CHUNK_ROW_THRESHOLD` | `2000000` | Row count above which `ingest.ps1` uses proactive chunked backfill. See [Chunked backfill](#automatic-chunked-backfill). | +| Variable | Default | Meaning | +| ------------------------------ | ---------- | ------------------------------------------------------------------------------------------------------------------------ | +| `HOST_PORT` | `8082` | Host port mapped to the Kustainer container's HTTP endpoint. | +| `MEM_LIMIT` | `16g` | Memory ceiling for the Kustainer container. | +| `EXPORT_DIR` | `./export` | Host directory with Parquet exports (read-only mount). | +| `BACKFILL_CHUNK_ROW_THRESHOLD` | `2000000` | Row count above which `ingest.ps1` uses proactive chunked backfill. See [Chunked backfill](#automatic-chunked-backfill). | --- @@ -90,24 +90,24 @@ All tunables live in `.env` (gitignored). `.env.example` documents them: The local stack mirrors the real hub's **two-database** layout: -| Database | Holds | You query it for | -|----------|-------|------------------| +| Database | Holds | You query it for | +| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------- | | **`Ingestion`** | Raw tables (`Costs_raw`, `Prices_raw`), `*_transform_v1_2()` functions, final tables (`Costs_final_v1_2`, `Prices_final_v1_2`), open-data lookups, and `Ingest_Manifest` | Ingestion internals, parity checks, transform debugging | -| **`Hub`** | Hub view functions (`Costs`, `Prices`, `Transactions`, `Costs_v1_2`, `Prices_v1_2`, …) that read `database('Ingestion').*` | All analytics — this is the default query database | +| **`Hub`** | Hub view functions (`Costs`, `Prices`, `Transactions`, `Costs_v1_2`, `Prices_v1_2`, …) that read `database('Ingestion').*` | All analytics — this is the default query database | The `Hub` database is the default for `make kql` and `ftk.ps1`. To inspect raw tables, override: `make kql KQL_DB=Ingestion QUERY='Costs_raw | count'`. ### How it maps to a real hub -| Real hub component | Local equivalent | -|--------------------|-----------------| -| Azure Data Explorer (ADX) cluster | Kusto emulator (Docker, `http://localhost:8082`) | -| `Ingestion` ADX database | `Ingestion` database in the emulator | -| `Hub` ADX database | `Hub` database in the emulator | -| Data Factory pipeline (ADF) | `scripts/ingest.ps1` | -| Azure Storage (`msexports/` or `ingestion/` container) | `export/` directory on disk | -| Bicep/ARM deployment | `make load-ftk-kql` (idempotent KQL loader) | +| Real hub component | Local equivalent | +| ------------------------------------------------------ | ------------------------------------------------ | +| Azure Data Explorer (ADX) cluster | Kusto emulator (Docker, `http://localhost:8082`) | +| `Ingestion` ADX database | `Ingestion` database in the emulator | +| `Hub` ADX database | `Hub` database in the emulator | +| Data Factory pipeline (ADF) | `scripts/ingest.ps1` | +| Azure Storage (`msexports/` or `ingestion/` container) | `export/` directory on disk | +| Bicep/ARM deployment | `make load-ftk-kql` (idempotent KQL loader) | The KQL itself — `IngestionSetup_RawTables.kql`, `IngestionSetup_v1_2.kql`, `HubSetup_v1_2.kql`, `HubSetup_OpenData.kql` — is loaded verbatim from the upstream @@ -152,11 +152,11 @@ ingest` on an unchanged export directory is safe — already-ingested files are ### Overwrite semantics -| Situation | Behavior | -|-----------|----------| -| Same run-uuid, same file checksum | Skip (already ingested) | -| Same `(scope, type, period)`, **new** run-uuid | Safe replace — old extents are dropped, new data is ingested | -| Same run-uuid, **changed** file checksum | **Fail-fast double-ingest guard** — the script detects the checksum mismatch and refuses. Stage the replacement files under a **new run-uuid** instead. | +| Situation | Behavior | +| ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Same run-uuid, same file checksum | Skip (already ingested) | +| Same `(scope, type, period)`, **new** run-uuid | Safe replace — old extents are dropped, new data is ingested | +| Same run-uuid, **changed** file checksum | **Fail-fast double-ingest guard** — the script detects the checksum mismatch and refuses. Stage the replacement files under a **new run-uuid** instead. | > **How to replace data:** stage replacement parquet under a new `` directory > with a `manifest.json` whose `runInfo.submittedTime` is later than the existing run. @@ -175,6 +175,7 @@ backfill automatically: first; if it OOMs the engine, `ingest.ps1` falls back to chunked automatically. The threshold is sized so that at `MEM_LIMIT=16g`: + - Costs (~1.35 M rows) → single-pass (safely below threshold). - Prices (~12.7 M rows) → proactive chunked (safely above threshold). @@ -247,15 +248,15 @@ These numbers were measured on `MEM_LIMIT=16g`, amd64-on-Rosetta (Apple Silicon) with a dataset of approximately 1,350,561 Cost rows and 12,735,587 Price rows plus open-data lookups. Your results vary by `MEM_LIMIT` and dataset size. -| Metric | Measured value | -|--------|----------------| -| Raw file ingest wall-clock (31 parts, ~991 MB) | 419.6 s (7.0 min) | -| Full `ingest.ps1` wall-clock (raw + all backfill) | 841.6 s (14.0 min) | -| **Cold loaded** — post-restart, data on disk, extents not yet materialized (anon) | **1.12 GiB (7.0% of 16 GiB)** — this is the floor | -| **Hot post-ingest** — immediately after transform pipeline; extents in memory (anon) | **13.99 GiB (≈87% of 16 GiB)** — sustained until restart | -| Chunked backfill memory.peak (cumulative HWM, Prices per-extent) | **15.03 GiB (93.9% of 16 GiB)** | -| Single-pass Prices transform (~12.7 M rows) | anon ~14.74 GiB / memory.peak **15.12 GiB** → engine crash | -| Single-pass Costs transform (~1.35 M rows) | Completes in 53 s; peak 6.89 GiB (43% of 16 GiB) | +| Metric | Measured value | +| ------------------------------------------------------------------------------------ | ---------------------------------------------------------- | +| Raw file ingest wall-clock (31 parts, ~991 MB) | 419.6 s (7.0 min) | +| Full `ingest.ps1` wall-clock (raw + all backfill) | 841.6 s (14.0 min) | +| **Cold loaded** — post-restart, data on disk, extents not yet materialized (anon) | **1.12 GiB (7.0% of 16 GiB)** — this is the floor | +| **Hot post-ingest** — immediately after transform pipeline; extents in memory (anon) | **13.99 GiB (≈87% of 16 GiB)** — sustained until restart | +| Chunked backfill memory.peak (cumulative HWM, Prices per-extent) | **15.03 GiB (93.9% of 16 GiB)** | +| Single-pass Prices transform (~12.7 M rows) | anon ~14.74 GiB / memory.peak **15.12 GiB** → engine crash | +| Single-pass Costs transform (~1.35 M rows) | Completes in 53 s; peak 6.89 GiB (43% of 16 GiB) | See [notes/performance.md](notes/performance.md) for the full row-count → memory curve, per-stage cgroup v2 measurements, and OOM ceiling analysis. @@ -276,7 +277,7 @@ Set `HOST_PORT=` in `.env` and re-run `make up`. Do not use port 808 - **Apple Silicon:** ensure Rosetta is enabled (Docker Desktop → Settings → General). - **Logs:** `make logs` — look for `Database NetDefaultDB has been created and is - answering queries`. +answering queries`. - **Memory:** `docker stats --no-stream kustainer` — raise `MEM_LIMIT` in `.env` if near the ceiling. After editing `.env`, always use `make down && make up` (never `--force-recreate`; see below). @@ -358,18 +359,18 @@ make parity ## Makefile reference -| Target | What it does | -|--------|--------------| -| `make up` | Bring Kustainer up and block until the healthcheck passes. | -| `make down` | Stop the container (data volume preserved). | -| `make nuke` | Destructive full reset — removes the `kustainer-data/` volume. | -| `make logs` | Tail Kustainer container logs. | -| `make kql QUERY='...'` | One-shot KQL against Hub (override: `KQL_DB=Ingestion`). | -| `make load-ftk-kql` | Load FTK KQL + open data into Ingestion + Hub (idempotent). | -| `make ingest` | Bulk-ingest all Parquet exports under `export/`. | -| `make ingest SCOPE=` | Ingest one scope only. | -| `make ingest SCOPE= PERIOD=

` | Ingest one scope + period. | -| `make ingest-status` | Show `Ingest_Manifest` summary (files + rows per scope/type). | -| `make parity` | Run the parity check suite (exit 0 iff all checks pass). | -| `make chunked-prices-backfill` | Manual per-extent Prices backfill (auto-triggered by threshold). | -| `make help` | Print all targets. | +| Target | What it does | +| ---------------------------------- | ---------------------------------------------------------------- | +| `make up` | Bring Kustainer up and block until the healthcheck passes. | +| `make down` | Stop the container (data volume preserved). | +| `make nuke` | Destructive full reset — removes the `kustainer-data/` volume. | +| `make logs` | Tail Kustainer container logs. | +| `make kql QUERY='...'` | One-shot KQL against Hub (override: `KQL_DB=Ingestion`). | +| `make load-ftk-kql` | Load FTK KQL + open data into Ingestion + Hub (idempotent). | +| `make ingest` | Bulk-ingest all Parquet exports under `export/`. | +| `make ingest SCOPE=` | Ingest one scope only. | +| `make ingest SCOPE= PERIOD=

` | Ingest one scope + period. | +| `make ingest-status` | Show `Ingest_Manifest` summary (files + rows per scope/type). | +| `make parity` | Run the parity check suite (exit 0 iff all checks pass). | +| `make chunked-prices-backfill` | Manual per-extent Prices backfill (auto-triggered by threshold). | +| `make help` | Print all targets. | diff --git a/src/templates/finops-hub-local/notes/config-container.md b/src/templates/finops-hub-local/notes/config-container.md index 94db042e4..c79ef028a 100644 --- a/src/templates/finops-hub-local/notes/config-container.md +++ b/src/templates/finops-hub-local/notes/config-container.md @@ -5,18 +5,18 @@ A deployed FinOps Hub writes two artifacts to the Azure Storage `config` container: -| Artifact | Location in storage | Schema | -|---|---|---| -| `settings.json` | `config/settings.json` | `https://aka.ms/finops/hubs/settings-schema` | -| Export schemas | `config/schemas/*.json` | Internal column-typing contract | +| Artifact | Location in storage | Schema | +| --------------- | ----------------------- | -------------------------------------------- | +| `settings.json` | `config/settings.json` | `https://aka.ms/finops/hubs/settings-schema` | +| Export schemas | `config/schemas/*.json` | Internal column-typing contract | The `settings.json` file drives **scope registration** and **retention -configuration** for the Azure Data Factory (ADF) orchestration layer. The +configuration** for the Azure Data Factory (ADF) orchestration layer. The `config/schemas/*.json` files drive the ADF **msexports → ingestion conversion** step — column typing and normalization of the raw Cost Management CSV exports into Parquet. -ftklocal does not run ADF. The question this note answers is: which of these +ftklocal does not run ADF. The question this note answers is: which of these artifacts does ftklocal need, and when? --- @@ -26,19 +26,19 @@ artifacts does ftklocal need, and when? **What ftklocal does today.** ftklocal ingests the already-converted `ingestion/`-container Parquet directly -into Kusto (`Costs_raw`, `Prices_raw`). At this point the msexports → ingestion +into Kusto (`Costs_raw`, `Prices_raw`). At this point the msexports → ingestion conversion has already been performed by ADF in the deployed hub. **Config container requirement:** none. - `settings.json` — not required for the msexports → ingestion conversion path - (that is an ADF concern). However, `load-ftk-kql.ps1` does optionally read a + (that is an ADF concern). However, `load-ftk-kql.ps1` does optionally read a local `settings.json` for one value — `retention.raw.days` — to set the Kusto raw-table soft-delete policy via the `$$rawRetentionInDays$$` macro; it falls - back to 3650 days when the file is absent or when `raw.days` is 0. The + back to 3650 days when the file is absent or when `raw.days` is 0. The scopes array in `settings.json` is an ADF concern and is not read by the - ADX/KQL layer. See `settings.example.json` and `scripts/settings.ps1`. -- `config/schemas/*.json` — not required. The Parquet files in the ingestion + ADX/KQL layer. See `settings.example.json` and `scripts/settings.ps1`. +- `config/schemas/*.json` — not required. The Parquet files in the ingestion container are already normalized; the column-typing step that uses the schemas has already run. @@ -50,40 +50,40 @@ conversion has already been performed by ADF in the deployed hub. If ftklocal were extended to ingest directly from the raw Cost Management CSV exports (`msexports/` container), it would need to perform the same -msexports → ingestion conversion that ADF does. That conversion uses the schema +msexports → ingestion conversion that ADF does. That conversion uses the schema files to type, deduplicate, and normalize each export. **Config container requirement:** -- `config/schemas/*.json` — required for the conversion step. **These must +- `config/schemas/*.json` — required for the conversion step. **These must always be sourced from the in-repo copies** at `src/templates/finops-hub/modules/Microsoft.CostManagement/Exports/schemas/` (e.g. `focuscost_1.2.json`, `focuscost_1.2-preview.json`, - `pricesheet_2023-05-01_ea.json`). Never copy schemas from a customer's + `pricesheet_2023-05-01_ea.json`). Never copy schemas from a customer's storage; the in-repo versions are the canonical, version-controlled definitions. - `settings.json` — would be needed to resolve the scope list and retention parameters for the conversion orchestration. **Status:** the full local msexports converter is deferred to the bridge work -tracked in T-5000.9. Mode A remains the v1 happy path. +tracked in T-5000.9. Mode A remains the v1 happy path. --- ## ADX / KQL layer — `config/schemas/*.json` not consumed; `settings.json` read by `load-ftk-kql.ps1` for one value The `config/schemas/*.json` files (the msexports → ingestion column-typing -schemas) are not consumed by any component of the ADX/KQL layer. `settings.json` +schemas) are not consumed by any component of the ADX/KQL layer. `settings.json` is partially consumed: `load-ftk-kql.ps1` optionally reads it for -`retention.raw.days` only; everything else reads neither. Evidence: +`retention.raw.days` only; everything else reads neither. Evidence: - **`scripts/load-ftk-kql.ps1`** — optionally reads a local `settings.json` for `retention.raw.days` (with a 3650-day fallback) and passes it to the - `$$rawRetentionInDays$$` macro in the Ingestion bundle. It does not read - scopes, version, or any other `settings.json` field. It does not read + `$$rawRetentionInDays$$` macro in the Ingestion bundle. It does not read + scopes, version, or any other `settings.json` field. It does not read `config/schemas/*.json` at all. - **`scripts/ftk.ps1`** — reads neither `settings.json` nor - `config/schemas/*.json`. It does use the word "schema" extensively, but only + `config/schemas/*.json`. It does use the word "schema" extensively, but only for querying live Kusto table schemas via KQL (`.show table ... cslschema`, `getschema`) — not for any on-disk config file. - **Hub / Ingestion KQL queries** — the `.kql` files assembled by @@ -94,12 +94,12 @@ is partially consumed: `load-ftk-kql.ps1` optionally reads it for are consumed by the ADF pipeline, not by KQL. For v1, `config/schemas/*.json` is therefore **not required** (the -msexports → ingestion conversion is an ADF concern). `settings.json` is +msexports → ingestion conversion is an ADF concern). `settings.json` is **optional** — absent means 3650-day raw retention; present lets the operator set it explicitly. > **Local analysis note:** `retention.raw.days` in `settings.json` drives the > Kusto raw-table soft-delete window (the `$$rawRetentionInDays$$` macro in -> `load-ftk-kql.ps1`). Keep this value high (3650 days is the default) for +> `load-ftk-kql.ps1`). Keep this value high (3650 days is the default) for > local analysis — unlike Azure Storage, local raw tables ARE the source data; > early soft-deletion means re-running a full ingest to recover. diff --git a/src/templates/finops-hub-local/notes/dashboard.md b/src/templates/finops-hub-local/notes/dashboard.md index ce4116567..80689cd90 100644 --- a/src/templates/finops-hub-local/notes/dashboard.md +++ b/src/templates/finops-hub-local/notes/dashboard.md @@ -3,10 +3,10 @@ This dashboard (`../dashboard.json`) is the upstream FinOps hub dashboard localized to the ftklocal Kusto emulator. The only change from the upstream file is: -| Field | Upstream value | Local value | -|---|---|---| +| Field | Upstream value | Local value | +| --------------------------- | ----------------------------------------------------- | ----------------------- | | `dataSources[0].clusterUri` | `https://..kusto.windows.net/` | `http://localhost:8082` | -| `dataSources[0].database` | `Hub` | `Hub` (unchanged) | +| `dataSources[0].database` | `Hub` | `Hub` (unchanged) | All tiles and KQL queries are unmodified. The `Hub` database on the emulator exposes the same view functions as a deployed hub: `Costs_v1_2`, `Prices_v1_2`, `Transactions_v1_2`, @@ -50,7 +50,7 @@ available as structured tool calls through the skill surface — no manual KQL r > > `dataexplorer.azure.com` is served over HTTPS, and `http://localhost` (and `127.0.0.1`) is a > **potentially trustworthy origin** under the W3C Secure Contexts spec, so Chromium-based -> browsers (**Microsoft Edge, Google Chrome**) do *not* apply mixed-content blocking to it. +> browsers (**Microsoft Edge, Google Chrome**) do _not_ apply mixed-content blocking to it. > However, current Edge (143+) and Chrome (142+) enforce **Local Network Access (LNA)**: the > first time the page reaches `http://localhost:8082` the browser shows a **"Local Network > Access" permission prompt** — click **Allow** and the connection proceeds for the session. @@ -66,13 +66,13 @@ available as structured tool calls through the skill surface — no manual KQL r Measured against the running emulator (`http://localhost:8082`, Hub DB populated with the sample dataset): -| Check | Result | -|---|---| -| CORS preflight (`OPTIONS`, `Origin: https://dataexplorer.azure.com`) | ✅ `204` with `Access-Control-Allow-Origin: https://dataexplorer.azure.com`, `Access-Control-Allow-Methods: POST`, `Access-Control-Allow-Headers: content-type,x-ms-client-version,authorization` | -| Actual query (`POST /v1/rest/query`, same `Origin`) | ✅ `200` with `Access-Control-Allow-Origin: https://dataexplorer.azure.com` | -| Real tile queries return renderable data via the CORS path | ✅ Monthly trend (15 months), top services (bar), Prices (table) all return rows | -| `dashboard.json` import schema | ✅ Official ADX dashboard schema **v60**, single `manual-kusto` data source → `http://localhost:8082` / `Hub` | -| `dashboard.json` referential integrity | ✅ 135 tiles / 87 queries / 6 base queries / 12 pages, **0 dangling references** → imports cleanly | +| Check | Result | +| --------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| CORS preflight (`OPTIONS`, `Origin: https://dataexplorer.azure.com`) | ✅ `204` with `Access-Control-Allow-Origin: https://dataexplorer.azure.com`, `Access-Control-Allow-Methods: POST`, `Access-Control-Allow-Headers: content-type,x-ms-client-version,authorization` | +| Actual query (`POST /v1/rest/query`, same `Origin`) | ✅ `200` with `Access-Control-Allow-Origin: https://dataexplorer.azure.com` | +| Real tile queries return renderable data via the CORS path | ✅ Monthly trend (15 months), top services (bar), Prices (table) all return rows | +| `dashboard.json` import schema | ✅ Official ADX dashboard schema **v60**, single `manual-kusto` data source → `http://localhost:8082` / `Hub` | +| `dashboard.json` referential integrity | ✅ 135 tiles / 87 queries / 6 base queries / 12 pages, **0 dangling references** → imports cleanly | | Private Network Access (PNA) preflight (`Access-Control-Request-Private-Network: true`) | ⚠️ Emulator does **not** return `Access-Control-Allow-Private-Network: true`. Current Edge 143+/Chrome 142+ enforce **Local Network Access (LNA)** and show a one-time **permission prompt** for the public→localhost call — click **Allow** to proceed. Locked-down/enterprise-policy builds that suppress or deny the prompt need the [LNA fallback](#if-the-browser-blocks-the-connection). | The protocol path is proven server-side: standard CORS is allowed, `localhost` is exempt from @@ -125,7 +125,7 @@ Network Access permission prompt**. In the normal case you simply click **Allow* prompt appears (see step 1) and the connection works — no server header or flag needed. If your environment **suppresses or auto-denies** the prompt (enterprise policy, or a build -where you previously clicked *Block*), relax the check for local UAT by launching the browser +where you previously clicked _Block_), relax the check for local UAT by launching the browser with LNA/PNA enforcement disabled: ```bash @@ -160,16 +160,16 @@ dashboard data source to match. ## What was proved vs what you must validate -| Claim | Status | -|---|---| -| `dashboard.json` is valid JSON | ✅ Verified (Python `json.load` round-trip) | -| `dataSources[0].clusterUri` → `http://localhost:8082` | ✅ Verified in file | -| `dataSources[0].database` = `Hub` | ✅ Verified in file | -| Hub DB has `Costs_v1_2`, `Prices_v1_2`, etc. | ✅ Verified via `.show functions` on running emulator | -| No upstream cluster names, billing IDs, or absolute paths in dashboard | ✅ Verified | -| `ftk query` / `ftk run` against Hub DB | ✅ Works (HTTP REST, no browser) | +| Claim | Status | +| ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `dashboard.json` is valid JSON | ✅ Verified (Python `json.load` round-trip) | +| `dataSources[0].clusterUri` → `http://localhost:8082` | ✅ Verified in file | +| `dataSources[0].database` = `Hub` | ✅ Verified in file | +| Hub DB has `Costs_v1_2`, `Prices_v1_2`, etc. | ✅ Verified via `.show functions` on running emulator | +| No upstream cluster names, billing IDs, or absolute paths in dashboard | ✅ Verified | +| `ftk query` / `ftk run` against Hub DB | ✅ Works (HTTP REST, no browser) | | Browser can add `http://localhost:8082` from `https://dataexplorer.azure.com` | ✅ **CORS allowed** (emulator returns `Access-Control-Allow-Origin: https://dataexplorer.azure.com`); `localhost` is exempt from mixed-content blocking. Edge 143+/Chrome 142+ show a one-time **Local Network Access** prompt — click **Allow**. | -| Dashboard tiles render data after import | ⚠️ **Visual render pending** — protocol path proven (tile queries return data via CORS); requires the interactive browser import + AAD sign-in + the LNA **Allow** click to confirm on screen | +| Dashboard tiles render data after import | ⚠️ **Visual render pending** — protocol path proven (tile queries return data via CORS); requires the interactive browser import + AAD sign-in + the LNA **Allow** click to confirm on screen | --- diff --git a/src/templates/finops-hub-local/notes/ftk-kql-adaptations.md b/src/templates/finops-hub-local/notes/ftk-kql-adaptations.md index d42323e14..e79d0e8a4 100644 --- a/src/templates/finops-hub-local/notes/ftk-kql-adaptations.md +++ b/src/templates/finops-hub-local/notes/ftk-kql-adaptations.md @@ -3,10 +3,10 @@ There are **two distinct adaptation layers** in the local stack. This document covers both. -| Layer | Where it runs | Purpose | -|-------|--------------|---------| -| **Load-time** | `scripts/load-ftk-kql.ps1` | Loads upstream FTK KQL into the emulator at setup time (idempotent). | -| **On-read** | `scripts/ftk.ps1` `Convert-CatalogQuery` (lines 341–382) | Adapts catalog `.kql` files at query time before posting to the emulator. | +| Layer | Where it runs | Purpose | +| ------------- | -------------------------------------------------------- | ------------------------------------------------------------------------- | +| **Load-time** | `scripts/load-ftk-kql.ps1` | Loads upstream FTK KQL into the emulator at setup time (idempotent). | +| **On-read** | `scripts/ftk.ps1` `Convert-CatalogQuery` (lines 341–382) | Adapts catalog `.kql` files at query time before posting to the emulator. | The upstream KQL scripts live in `src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/scripts/`. @@ -25,10 +25,11 @@ as a gap — not patched locally. ## Topology note FTK deploys to ADX as a **two-database** system: an `Ingestion` database (raw + transform -+ final tables) and a `Hub` database (view functions). The local stack preserves this -topology exactly — the emulator runs both `Ingestion` and `Hub` databases, and -`database('Ingestion').*` cross-references inside Hub view functions resolve in the -emulator the same way they do in ADX. + +- final tables) and a `Hub` database (view functions). The local stack preserves this + topology exactly — the emulator runs both `Ingestion` and `Hub` databases, and + `database('Ingestion').*` cross-references inside Hub view functions resolve in the + emulator the same way they do in ADX. Earlier prototype notes described collapsing these into a single local `FtkLocal` database. That design was superseded. The current stack uses the two-database topology throughout. @@ -75,6 +76,7 @@ mappings. The resulting tables are empty and serve only to satisfy the eager-com check. Tables created (empty stubs): + - `Costs_final_v1_0` - `Prices_final_v1_0` - `CommitmentDiscountUsage_final_v1_0` @@ -127,19 +129,19 @@ script. The loader applies the same rename at load time. Key mappings: -| Table | CSV column | Table column | -|-------|-----------|--------------| -| `PricingUnits` | `UnitOfMeasure` | `x_PricingUnitDescription` | -| `PricingUnits` | `PricingBlockSize` | `x_PricingBlockSize` | -| `PricingUnits` | `DistinctUnits` | `PricingUnit` | -| `Regions` | `OriginalValue` | `ResourceLocation` | -| `ResourceTypes` | `ResourceType` | `x_ResourceType` | -| `ResourceTypes` | `Icon` | `IconUri` | -| `Services` | `ConsumedService` | `x_ConsumedService` | -| `Services` | `ResourceType` | `x_ResourceType` | -| `Services` | `PublisherType` | `x_PublisherCategory` | -| `Services` | `Environment` | `x_Environment` | -| `Services` | `ServiceModel` | `x_ServiceModel` | +| Table | CSV column | Table column | +| --------------- | ------------------ | -------------------------- | +| `PricingUnits` | `UnitOfMeasure` | `x_PricingUnitDescription` | +| `PricingUnits` | `PricingBlockSize` | `x_PricingBlockSize` | +| `PricingUnits` | `DistinctUnits` | `PricingUnit` | +| `Regions` | `OriginalValue` | `ResourceLocation` | +| `ResourceTypes` | `ResourceType` | `x_ResourceType` | +| `ResourceTypes` | `Icon` | `IconUri` | +| `Services` | `ConsumedService` | `x_ConsumedService` | +| `Services` | `ResourceType` | `x_ResourceType` | +| `Services` | `PublisherType` | `x_PublisherCategory` | +| `Services` | `Environment` | `x_Environment` | +| `Services` | `ServiceModel` | `x_ServiceModel` | The loader reads each CSV, projects to the target column order, clears the destination table (idempotency), and ingests inline into the FTK-declared table schema. diff --git a/src/templates/finops-hub-local/notes/install-mac.md b/src/templates/finops-hub-local/notes/install-mac.md index b20208d43..93b2f108b 100644 --- a/src/templates/finops-hub-local/notes/install-mac.md +++ b/src/templates/finops-hub-local/notes/install-mac.md @@ -13,15 +13,15 @@ identically and are the cross-platform reference. ## Prerequisites -| Requirement | Notes | -|-------------|-------| -| **Docker Desktop ≥ 4.0** | Compose v2 is required (`docker compose`, not legacy `docker-compose`). | -| **Apple Silicon: enable Rosetta** | Docker Desktop → Settings → General → "Use Rosetta for x86_64/amd64 emulation on Apple Silicon". The Kustainer image is `linux/amd64` only. Intel Macs need nothing extra. | -| **PowerShell 7+** (`pwsh`) | `brew install --cask powershell`. All tooling is PowerShell; there are no Python dependencies. | -| **Git** | To clone the repository (`brew install git`, or use the Xcode command-line tools). | -| **~16 GiB RAM available to Docker** | Docker Desktop → Settings → Resources → Memory. The full Prices transform needs it; Costs-only datasets work at 8 GiB. See [Limits](performance.md). | -| **Host port 8082 free** | Configurable via `HOST_PORT` in `.env`. | -| **FOCUS cost exports as Parquet** | Staged under `export/`. See [staging-contract.md](staging-contract.md). | +| Requirement | Notes | +| ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Docker Desktop ≥ 4.0** | Compose v2 is required (`docker compose`, not legacy `docker-compose`). | +| **Apple Silicon: enable Rosetta** | Docker Desktop → Settings → General → "Use Rosetta for x86_64/amd64 emulation on Apple Silicon". The Kustainer image is `linux/amd64` only. Intel Macs need nothing extra. | +| **PowerShell 7+** (`pwsh`) | `brew install --cask powershell`. All tooling is PowerShell; there are no Python dependencies. | +| **Git** | To clone the repository (`brew install git`, or use the Xcode command-line tools). | +| **~16 GiB RAM available to Docker** | Docker Desktop → Settings → Resources → Memory. The full Prices transform needs it; Costs-only datasets work at 8 GiB. See [Limits](performance.md). | +| **Host port 8082 free** | Configurable via `HOST_PORT` in `.env`. | +| **FOCUS cost exports as Parquet** | Staged under `export/`. See [staging-contract.md](staging-contract.md). | --- @@ -92,13 +92,13 @@ git pull && docker compose up -d --wait && pwsh scripts/load-ftk-kql.ps1 # upd ## Troubleshooting (macOS-specific) -| Symptom | Fix | -|---------|-----| +| Symptom | Fix | +| --------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | Very slow ingest / transform on Apple Silicon | Expected — `linux/amd64` runs under emulation. Confirm Rosetta is enabled (above). See [performance.md](performance.md) for the measured envelope. | -| Container exits under the Prices transform | Raise Docker Desktop's memory (Settings → Resources) to ≥ 16 GiB. The ingest already chunks large tables. | -| `pwsh: command not found` | `brew install --cask powershell`. | -| Rare SIGSEGV crashes under heavy load | A known Rosetta emulation issue; the container restarts automatically. See the main [README](../README.md) troubleshooting section. | -| Port 8082 already in use | Set a different `HOST_PORT` in `.env`, then bring the stack back up. | +| Container exits under the Prices transform | Raise Docker Desktop's memory (Settings → Resources) to ≥ 16 GiB. The ingest already chunks large tables. | +| `pwsh: command not found` | `brew install --cask powershell`. | +| Rare SIGSEGV crashes under heavy load | A known Rosetta emulation issue; the container restarts automatically. See the main [README](../README.md) troubleshooting section. | +| Port 8082 already in use | Set a different `HOST_PORT` in `.env`, then bring the stack back up. | For the ADX web-UI dashboard, see [dashboard.md](dashboard.md). Use Edge or Chrome; Safari is stricter about local connections. diff --git a/src/templates/finops-hub-local/notes/install-windows.md b/src/templates/finops-hub-local/notes/install-windows.md index a8f6cf8a8..cd937dc98 100644 --- a/src/templates/finops-hub-local/notes/install-windows.md +++ b/src/templates/finops-hub-local/notes/install-windows.md @@ -14,18 +14,18 @@ whole surface. ## Prerequisites -| Requirement | Notes | -|-------------|-------| -| **Docker Desktop ≥ 4.0** with the **WSL 2 backend** | Settings → General → "Use the WSL 2 based engine". Compose v2 ships with it (`docker compose`, not legacy `docker-compose`). | -| **PowerShell 7+** (`pwsh`) | **Windows PowerShell 5.1 will not work** — the scripts use 7-only syntax. Install with `winget install Microsoft.PowerShell`, then use the `pwsh` command (not `powershell`). | -| **Git** | `winget install Git.Git`, to clone the repository. | -| **~16 GiB RAM available to Docker** | The full Prices transform needs it. Costs-only datasets work at 8 GiB. See [memory tuning](#tune-wsl-2-memory) below — on Windows the limit is governed by both the container `MEM_LIMIT` **and** WSL 2's own memory cap. | -| **Host port 8082 free** | Configurable via `HOST_PORT` in `.env`. | -| **FOCUS cost exports as Parquet** | Staged under `export/`. See [staging-contract.md](staging-contract.md). | - -> **Why PowerShell 7, not the built-in PowerShell?** Windows ships *Windows PowerShell 5.1* +| Requirement | Notes | +| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Docker Desktop ≥ 4.0** with the **WSL 2 backend** | Settings → General → "Use the WSL 2 based engine". Compose v2 ships with it (`docker compose`, not legacy `docker-compose`). | +| **PowerShell 7+** (`pwsh`) | **Windows PowerShell 5.1 will not work** — the scripts use 7-only syntax. Install with `winget install Microsoft.PowerShell`, then use the `pwsh` command (not `powershell`). | +| **Git** | `winget install Git.Git`, to clone the repository. | +| **~16 GiB RAM available to Docker** | The full Prices transform needs it. Costs-only datasets work at 8 GiB. See [memory tuning](#tune-wsl-2-memory) below — on Windows the limit is governed by both the container `MEM_LIMIT` **and** WSL 2's own memory cap. | +| **Host port 8082 free** | Configurable via `HOST_PORT` in `.env`. | +| **FOCUS cost exports as Parquet** | Staged under `export/`. See [staging-contract.md](staging-contract.md). | + +> **Why PowerShell 7, not the built-in PowerShell?** Windows ships _Windows PowerShell 5.1_ > (`powershell.exe`). ftklocal's scripts use null-coalescing (`??`) and ternary operators -> that only exist in *PowerShell 7+* (`pwsh.exe`). Running them under 5.1 fails with parser +> that only exist in _PowerShell 7+_ (`pwsh.exe`). Running them under 5.1 fails with parser > errors. Always invoke them with `pwsh`. --- @@ -135,16 +135,16 @@ pwsh scripts\load-ftk-kql.ps1 # re-applies schema (idempotent ## Troubleshooting (Windows-specific) -| Symptom | Fix | -|---------|-----| -| `pwsh : The term 'pwsh' is not recognized` | PowerShell 7 isn't installed or isn't on `PATH`. `winget install Microsoft.PowerShell`, then open a new terminal. | -| Parser errors like `Unexpected token '??'` | You're running under Windows PowerShell 5.1. Use `pwsh`, not `powershell`. | -| `running scripts is disabled on this system` | Invoking as `pwsh scripts\x.ps1 ` normally bypasses this. If you dot-source instead, set `Set-ExecutionPolicy -Scope Process RemoteSigned` for the session. | -| `docker compose` not found | Update Docker Desktop (Compose v2 is bundled). Don't use the legacy `docker-compose`. | -| Container exits / killed under the Prices transform | WSL 2 memory cap too low — see [Tune WSL 2 memory](#tune-wsl-2-memory). The ingest already chunks large tables; raising the VM cap removes the ceiling. | -| Port 8082 already in use | Set a different `HOST_PORT` in `.env`, then `docker compose up -d --wait`. | -| `Ingestion`/`Hub` DB missing after start | Run `pwsh scripts\load-ftk-kql.ps1`. If it persists, full-reset (above) and reload. | -| Tiles/queries return 0 rows | You haven't ingested yet — run `pwsh scripts\ingest.ps1`. | +| Symptom | Fix | +| --------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `pwsh : The term 'pwsh' is not recognized` | PowerShell 7 isn't installed or isn't on `PATH`. `winget install Microsoft.PowerShell`, then open a new terminal. | +| Parser errors like `Unexpected token '??'` | You're running under Windows PowerShell 5.1. Use `pwsh`, not `powershell`. | +| `running scripts is disabled on this system` | Invoking as `pwsh scripts\x.ps1 ` normally bypasses this. If you dot-source instead, set `Set-ExecutionPolicy -Scope Process RemoteSigned` for the session. | +| `docker compose` not found | Update Docker Desktop (Compose v2 is bundled). Don't use the legacy `docker-compose`. | +| Container exits / killed under the Prices transform | WSL 2 memory cap too low — see [Tune WSL 2 memory](#tune-wsl-2-memory). The ingest already chunks large tables; raising the VM cap removes the ceiling. | +| Port 8082 already in use | Set a different `HOST_PORT` in `.env`, then `docker compose up -d --wait`. | +| `Ingestion`/`Hub` DB missing after start | Run `pwsh scripts\load-ftk-kql.ps1`. If it persists, full-reset (above) and reload. | +| Tiles/queries return 0 rows | You haven't ingested yet — run `pwsh scripts\ingest.ps1`. | For the ADX web-UI dashboard on Windows, see [dashboard.md](dashboard.md) — Edge and Chrome are the supported browsers and will prompt once for **Local Network Access** (click diff --git a/src/templates/finops-hub-local/notes/parity-gaps.md b/src/templates/finops-hub-local/notes/parity-gaps.md index ace93a7b9..6ca94fac5 100644 --- a/src/templates/finops-hub-local/notes/parity-gaps.md +++ b/src/templates/finops-hub-local/notes/parity-gaps.md @@ -89,21 +89,21 @@ check generated by the runner itself (not in the `.kql` file) — see [How "of 13 total" is counted](#how-of-13-total-is-counted) below. Checks run in file order (1–11, 13) then CHECK 12 last. -| # | Check | DB | Result | Detail | -|---|-------|----|--------|--------| -| 1 | Row count parity | Ingestion | ✅ PASS | `Costs_raw` = `Costs_final_v1_2`; delta = 0.0% | -| 2 | BilledCost preservation | Ingestion | ✅ PASS | delta = $0.000000 | -| 3 | EffectiveCost zeroed for commitment purchases | Ingestion | ✅ PASS | result = 0 | -| 4 | Data-quality flag dictionary populates | Ingestion | ⚠️ MANUAL | Print-only by design — see [Data-quality observations](#data-quality-observations-check-4) | -| 5 | Amortized usage labeled correctly | Ingestion | ✅ PASS | 0 mislabeled rows | -| 6 | TotalSavings populated for reservation services | Ingestion | ✅ PASS | top sum = 4,694.07 | -| 7 | `parse_resourceid` extracts `x_ResourceGroupName` | Ingestion | ✅ PASS | 0 missing extractions | -| 8 | `PublisherName` normalized | Ingestion | ✅ PASS | 0 unnormalized rows | -| 9 | Hub view function returns plausible totals | Hub | ✅ PASS | top sum_EffectiveCost = 306,236.63 | -| 10 | `Costs_v1_2` vs `Costs_final_v1_2` BilledCost reconciliation | Hub | ✅ PASS | delta = $0.000000 | -| 11 | Prices row count parity per PriceType | Ingestion | ⚠️ MANUAL | Runner always flags as MANUAL (print-only); all three PriceTypes show delta = 0 on current dataset | -| 12 | Name parity — Ingestion tables + Hub functions | Ingestion+Hub | ✅ PASS | 23 Ingestion tables, 34 Hub functions; no `FtkLocal`-named objects; all core names present. **Synthetic** — not in `.kql` file; generated by runner | -| 13 | Prices value parity — UnitPrice/ContractedUnitPrice (Consumption + RI) | Ingestion | ✅ PASS | delta = 0.0000% within 0.1% automated threshold | +| # | Check | DB | Result | Detail | +| --- | ---------------------------------------------------------------------- | ------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | +| 1 | Row count parity | Ingestion | ✅ PASS | `Costs_raw` = `Costs_final_v1_2`; delta = 0.0% | +| 2 | BilledCost preservation | Ingestion | ✅ PASS | delta = $0.000000 | +| 3 | EffectiveCost zeroed for commitment purchases | Ingestion | ✅ PASS | result = 0 | +| 4 | Data-quality flag dictionary populates | Ingestion | ⚠️ MANUAL | Print-only by design — see [Data-quality observations](#data-quality-observations-check-4) | +| 5 | Amortized usage labeled correctly | Ingestion | ✅ PASS | 0 mislabeled rows | +| 6 | TotalSavings populated for reservation services | Ingestion | ✅ PASS | top sum = 4,694.07 | +| 7 | `parse_resourceid` extracts `x_ResourceGroupName` | Ingestion | ✅ PASS | 0 missing extractions | +| 8 | `PublisherName` normalized | Ingestion | ✅ PASS | 0 unnormalized rows | +| 9 | Hub view function returns plausible totals | Hub | ✅ PASS | top sum_EffectiveCost = 306,236.63 | +| 10 | `Costs_v1_2` vs `Costs_final_v1_2` BilledCost reconciliation | Hub | ✅ PASS | delta = $0.000000 | +| 11 | Prices row count parity per PriceType | Ingestion | ⚠️ MANUAL | Runner always flags as MANUAL (print-only); all three PriceTypes show delta = 0 on current dataset | +| 12 | Name parity — Ingestion tables + Hub functions | Ingestion+Hub | ✅ PASS | 23 Ingestion tables, 34 Hub functions; no `FtkLocal`-named objects; all core names present. **Synthetic** — not in `.kql` file; generated by runner | +| 13 | Prices value parity — UnitPrice/ContractedUnitPrice (Consumption + RI) | Ingestion | ✅ PASS | delta = 0.0000% within 0.1% automated threshold | --- @@ -132,13 +132,13 @@ populated, not that it is empty. The flags are FTK's data-quality findings on th source export data, not defects in the local transform. Observed flags on the current dataset: -| Flag | Rows | -|------|-----:| -| `XEffectiveUnitPriceRoundingError` | 1,100,011 | -| `LegacyFocusVersion` | 117,862 | -| `ContractedCostLessThanEffectiveCost` | 110,171 | -| `ListCostLessThanContractedCost` | 19,967 | -| `MissingContractedCost` | 1,887 | +| Flag | Rows | +| ------------------------------------- | --------: | +| `XEffectiveUnitPriceRoundingError` | 1,100,011 | +| `LegacyFocusVersion` | 117,862 | +| `ContractedCostLessThanEffectiveCost` | 110,171 | +| `ListCostLessThanContractedCost` | 19,967 | +| `MissingContractedCost` | 1,887 | Consumers of `Costs_final_v1_2` can filter on these flags: diff --git a/src/templates/finops-hub-local/notes/performance.md b/src/templates/finops-hub-local/notes/performance.md index fa36de588..924b9d9e0 100644 --- a/src/templates/finops-hub-local/notes/performance.md +++ b/src/templates/finops-hub-local/notes/performance.md @@ -34,19 +34,20 @@ started just before the OOM attempt. ## Measurement table -| Stage | Total rows in engine | anon (GiB) | anon % of 16 g | memory.peak (GiB) | OOM events | -|-------|---------------------|-----------|----------------|------------------|------------| -| 1. Empty engine (just started, no data) | 0 | 0.49 | 3.1 % | 0.51 | — | -| 2. Schema + open-data loaded (`load-ftk-kql`) | 3,293 | 0.66 | 4.1 % | 0.68 | — | -| 3a. After Costs_raw ingested (15 parts) | 1,353,854 | 1.51 ¹ | 9.5 % | 2.13 ¹ | — | -| 3b. After all raw ingested, no finals yet | 14,089,441 | 1.99 ¹ | 12.4 % | 3.98 ¹ | — | -| 3c. After Costs_final_v1_2 backfill (single-pass) | 15,440,002 | 3.88 ¹ | 24.2 % | 6.89 ¹ | — | -| 3d. After full ingest — **hot** working set | 28,175,589 | **13.99** | **87.4 %** | **15.03** | 0 | -| 4. After representative query workload | 28,175,589 | 13.95 | 87.2 % | 15.03 | 0 | -| 5. Single-pass OOM attempt (Prices, 12.7 M rows) | 28,175,589 | 14.74 ¹ | 92.1 % | **15.12 ¹** | **crash** | -| 6. Cold loaded — post-restart, data on disk ² | 28,175,589 | **1.12** | **7.0 %** | 1.58 ² | 0 | +| Stage | Total rows in engine | anon (GiB) | anon % of 16 g | memory.peak (GiB) | OOM events | +| ------------------------------------------------- | -------------------- | ---------- | -------------- | ----------------- | ---------- | +| 1. Empty engine (just started, no data) | 0 | 0.49 | 3.1 % | 0.51 | — | +| 2. Schema + open-data loaded (`load-ftk-kql`) | 3,293 | 0.66 | 4.1 % | 0.68 | — | +| 3a. After Costs_raw ingested (15 parts) | 1,353,854 | 1.51 ¹ | 9.5 % | 2.13 ¹ | — | +| 3b. After all raw ingested, no finals yet | 14,089,441 | 1.99 ¹ | 12.4 % | 3.98 ¹ | — | +| 3c. After Costs_final_v1_2 backfill (single-pass) | 15,440,002 | 3.88 ¹ | 24.2 % | 6.89 ¹ | — | +| 3d. After full ingest — **hot** working set | 28,175,589 | **13.99** | **87.4 %** | **15.03** | 0 | +| 4. After representative query workload | 28,175,589 | 13.95 | 87.2 % | 15.03 | 0 | +| 5. Single-pass OOM attempt (Prices, 12.7 M rows) | 28,175,589 | 14.74 ¹ | 92.1 % | **15.12 ¹** | **crash** | +| 6. Cold loaded — post-restart, data on disk ² | 28,175,589 | **1.12** | **7.0 %** | 1.58 ² | 0 | **Notes:** + - Stages 1, 2, 3d, 4 are exact point-in-time measurements. Stages 3a–3c and 5 are read from the 2–3 s poll loop and should be treated as approximate (±0.1 GiB). - memory.peak is cumulative (not per-stage); the value in each row is the cumulative @@ -105,12 +106,12 @@ Key observations: ## Ingest wall-clock -| Phase | Duration | -|-------|----------| -| Raw file ingest (31 parquet parts, 991 MB) | 419.6 s (7.0 min) | -| Costs_final_v1_2 single-pass backfill (1.35 M rows) | 53.2 s | -| Prices_final_v1_2 chunked backfill (12.7 M rows, 21 extents × 1) | ~340 s (~5.7 min) | -| **Full `ingest.ps1` wall-clock (raw + backfill)** | **841.6 s (14.0 min)** | +| Phase | Duration | +| ---------------------------------------------------------------- | ---------------------- | +| Raw file ingest (31 parquet parts, 991 MB) | 419.6 s (7.0 min) | +| Costs_final_v1_2 single-pass backfill (1.35 M rows) | 53.2 s | +| Prices_final_v1_2 chunked backfill (12.7 M rows, 21 extents × 1) | ~340 s (~5.7 min) | +| **Full `ingest.ps1` wall-clock (raw + backfill)** | **841.6 s (14.0 min)** | The raw file ingest time is dominated by the 16 price sheet parts (247 s total, ~15 s each for the large monthly files). The 15 cost parts take only 76.5 s. @@ -123,10 +124,10 @@ for the large monthly files). The 15 cost parts take only 76.5 s. The 2,000,000-row threshold in `BACKFILL_CHUNK_ROW_THRESHOLD` separates two regimes: -| Mode | Prices (12.7 M rows) | Peak anon | memory.peak | Outcome | -|------|---------------------|-----------|-------------|---------| -| **Chunked (default, ≤ 1 extent/batch)** | 21 chunks × ~600 K avg rows | **14.71 GiB** | **15.03 GiB** | ✅ completes | -| **Single-pass** (threshold raised to bypass) | all 12.7 M rows at once | **14.74 GiB at crash** | **15.12 GiB** | ❌ OOM crash | +| Mode | Prices (12.7 M rows) | Peak anon | memory.peak | Outcome | +| -------------------------------------------- | --------------------------- | ---------------------- | ------------- | ------------ | +| **Chunked (default, ≤ 1 extent/batch)** | 21 chunks × ~600 K avg rows | **14.71 GiB** | **15.03 GiB** | ✅ completes | +| **Single-pass** (threshold raised to bypass) | all 12.7 M rows at once | **14.74 GiB at crash** | **15.12 GiB** | ❌ OOM crash | ### Chunked-backfill peak detail @@ -136,6 +137,7 @@ memory.peak reached 15.03 GiB (93.9 % of 16 g) during this phase. The individual chunks do NOT add up linearly in memory because the engine releases completed-extent memory between chunks. Each chunk's peak is dominated by: + - the pre-existing working set (~13–14 GiB from previously loaded extents), plus - the in-flight transform output for the current chunk (~0.5–1 GiB per ~1 M row chunk). @@ -172,12 +174,12 @@ that persists until the engine restarts or memory is reclaimed. All tiers are calibrated to `MEM_LIMIT=16g` and this specific dataset. They scale with both `MEM_LIMIT` and dataset size. Each cell identifies the metric and source stage. -| State | Condition | Metric (source stage) | % of MEM_LIMIT | Notes | -|-------|-----------|----------------------|----------------|-------| -| **Cold loaded** | Post-restart or fresh `make up`, before heavy queries | anon: **1.12 GiB** (stage 6) | 7.0 % | Floor; extents on disk, not yet materialized. Data intact and queryable. | -| **Hot post-ingest** | Immediately after `make ingest` completes; extents fully materialized by transform | anon: **13.99 GiB** (stage 3d) | 87.4 % | Sustained until restart or OS reclaim; this is NOT the steady-state floor | -| **Degraded** | Prices chunked-backfill high-water (during `make ingest`) | memory.peak: **15.03 GiB** (stage 3d cumulative HWM) | 93.9 % | Within 0.09 GiB of single-pass crash; system completes but headroom is minimal | -| **Ceiling** | Single-pass Prices transform attempt | anon: ~14.74 GiB / memory.peak: **15.12 GiB** → crash (stage 5) | 92.1 % anon / 94.5 % peak → crash | Kusto self-terminates (SIGKILL); container restarts automatically | +| State | Condition | Metric (source stage) | % of MEM_LIMIT | Notes | +| ------------------- | ---------------------------------------------------------------------------------- | --------------------------------------------------------------- | --------------------------------- | ------------------------------------------------------------------------------ | +| **Cold loaded** | Post-restart or fresh `make up`, before heavy queries | anon: **1.12 GiB** (stage 6) | 7.0 % | Floor; extents on disk, not yet materialized. Data intact and queryable. | +| **Hot post-ingest** | Immediately after `make ingest` completes; extents fully materialized by transform | anon: **13.99 GiB** (stage 3d) | 87.4 % | Sustained until restart or OS reclaim; this is NOT the steady-state floor | +| **Degraded** | Prices chunked-backfill high-water (during `make ingest`) | memory.peak: **15.03 GiB** (stage 3d cumulative HWM) | 93.9 % | Within 0.09 GiB of single-pass crash; system completes but headroom is minimal | +| **Ceiling** | Single-pass Prices transform attempt | anon: ~14.74 GiB / memory.peak: **15.12 GiB** → crash (stage 5) | 92.1 % anon / 94.5 % peak → crash | Kusto self-terminates (SIGKILL); container restarts automatically | ### When to raise MEM_LIMIT @@ -194,18 +196,18 @@ both `MEM_LIMIT` and dataset size. Each cell identifies the metric and source st ## Environment -| Setting | Value | -|---------|-------| -| MEM_LIMIT | 16g (17,179,869,184 bytes; confirmed via `memory.max`) | -| Kustainer image | `mcr.microsoft.com/azuredataexplorer/kustainer-linux:latest` | -| Kustainer build | `1.0.9656.17219` (2026-06-09) | -| Platform | linux/amd64 on Rosetta (Apple Silicon, Docker Desktop) | -| Host OS | macOS (Apple Silicon, M-series) | -| cgroup version | v2 (confirmed) | -| memory.peak reset | Not possible — kernel rejected write (read-only filesystem) | -| memory.peak semantics | Cumulative since container start; NOT resettable between stages | -| Poll interval (ingest) | 3 s | -| Poll interval (OOM) | 2 s | +| Setting | Value | +| ---------------------- | --------------------------------------------------------------- | +| MEM_LIMIT | 16g (17,179,869,184 bytes; confirmed via `memory.max`) | +| Kustainer image | `mcr.microsoft.com/azuredataexplorer/kustainer-linux:latest` | +| Kustainer build | `1.0.9656.17219` (2026-06-09) | +| Platform | linux/amd64 on Rosetta (Apple Silicon, Docker Desktop) | +| Host OS | macOS (Apple Silicon, M-series) | +| cgroup version | v2 (confirmed) | +| memory.peak reset | Not possible — kernel rejected write (read-only filesystem) | +| memory.peak semantics | Cumulative since container start; NOT resettable between stages | +| Poll interval (ingest) | 3 s | +| Poll interval (OOM) | 2 s | --- diff --git a/src/templates/finops-hub-local/notes/staging-contract.md b/src/templates/finops-hub-local/notes/staging-contract.md index a71a56a5a..36f8c7760 100644 --- a/src/templates/finops-hub-local/notes/staging-contract.md +++ b/src/templates/finops-hub-local/notes/staging-contract.md @@ -11,8 +11,8 @@ into the Kusto emulator (Part 2). ### Honest design constraint Cost Management exports are **Azure-only**. `New-FinOpsCostExport` writes parquet to an -Azure Storage account; there is no local export mode. Export *configuration* lives in Azure; -ftklocal *consumes* the parquet that configuration produces. No Azure export was created as +Azure Storage account; there is no local export mode. Export _configuration_ lives in Azure; +ftklocal _consumes_ the parquet that configuration produces. No Azure export was created as part of authoring this document. ### Relevant PowerShell cmdlets @@ -28,24 +28,24 @@ All three cmdlets are part of the FinOps Toolkit PowerShell module under Creates a Cost Management export that writes FOCUS parquet to Azure Storage. -| Parameter | Required | Notes | -|-----------|----------|-------| -| `-Name` | Yes | Unique name within the scope. | -| `-Scope` | Yes | Azure resource ID of the billing scope (subscription, billing account, etc.). | -| `-Dataset` | No | Default: `FocusCost`. Use `FocusCost` for cost data or `PriceSheet` for prices. | -| `-DatasetVersion` | No | Default: `1.2-preview` (FocusCost). Use `1.2-preview` to get the latest FOCUS schema. | -| `-Format` | No | Default: `Csv`. **Set to `Parquet`** for ftklocal (ingest.ps1 expects parquet). | -| `-CompressionMode` | No | Default: `None`. `Snappy` is also accepted but adds a decompression step not tested with ftklocal. | -| `-StorageAccountId` | Yes | Resource ID of the target Azure Storage account. | -| `-StorageContainer` | No | Default: `cost-management`. Use `msexports` to match hub conventions, or any container for ftklocal staging. | -| `-StoragePath` | No | Subfolder path prefix within the container. Defaults to the scope ID. | -| `-DoNotPartition` | No | Omit this flag (leave partitioning **enabled**). Partitioning is recommended for reliability. | -| `-DoNotOverwrite` | No | Default: overwrite enabled. For ftklocal, overwrite is acceptable; for FinOps hubs, use `-DoNotOverwrite` to simplify troubleshooting. | -| `-Monthly` | No | Monthly cadence (mutually exclusive with `-OneTime`). | -| `-OneTime` | No | Single run over an explicit date range. | -| `-StartDate` / `-EndDate` | No | Date range for the export. Defaults for scheduled and one-time differ; see cmdlet help. | -| `-Backfill` | No | Number of prior months to export immediately at creation time (scheduled exports only). | -| `-Execute` | No | Run the export once immediately after creation. | +| Parameter | Required | Notes | +| ------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------- | +| `-Name` | Yes | Unique name within the scope. | +| `-Scope` | Yes | Azure resource ID of the billing scope (subscription, billing account, etc.). | +| `-Dataset` | No | Default: `FocusCost`. Use `FocusCost` for cost data or `PriceSheet` for prices. | +| `-DatasetVersion` | No | Default: `1.2-preview` (FocusCost). Use `1.2-preview` to get the latest FOCUS schema. | +| `-Format` | No | Default: `Csv`. **Set to `Parquet`** for ftklocal (ingest.ps1 expects parquet). | +| `-CompressionMode` | No | Default: `None`. `Snappy` is also accepted but adds a decompression step not tested with ftklocal. | +| `-StorageAccountId` | Yes | Resource ID of the target Azure Storage account. | +| `-StorageContainer` | No | Default: `cost-management`. Use `msexports` to match hub conventions, or any container for ftklocal staging. | +| `-StoragePath` | No | Subfolder path prefix within the container. Defaults to the scope ID. | +| `-DoNotPartition` | No | Omit this flag (leave partitioning **enabled**). Partitioning is recommended for reliability. | +| `-DoNotOverwrite` | No | Default: overwrite enabled. For ftklocal, overwrite is acceptable; for FinOps hubs, use `-DoNotOverwrite` to simplify troubleshooting. | +| `-Monthly` | No | Monthly cadence (mutually exclusive with `-OneTime`). | +| `-OneTime` | No | Single run over an explicit date range. | +| `-StartDate` / `-EndDate` | No | Date range for the export. Defaults for scheduled and one-time differ; see cmdlet help. | +| `-Backfill` | No | Number of prior months to export immediately at creation time (scheduled exports only). | +| `-Execute` | No | Run the export once immediately after creation. | **Minimal FOCUS/Parquet example:** @@ -69,15 +69,15 @@ New-FinOpsCostExport ` Lists or filters existing Cost Management exports for a scope. -| Parameter | Notes | -|-----------|-------| -| `-Scope` | Azure resource ID of the scope. Defaults to current Az context subscription. | -| `-Name` | Filter by export name. Supports wildcards. | -| `-Dataset` | Filter by dataset type (e.g., `FocusCost`, `PriceSheet`). | -| `-DatasetVersion` | Filter by schema version. | -| `-StorageAccountId` | Filter by destination storage account. | -| `-StorageContainer` | Filter by container name. Supports wildcards. | -| `-RunHistory` | Switch. When present, the API response is expanded to include run history for each export. This is the enumeration primitive a future auto-download bridge (T-5000.9) would use to discover available runs and their storage paths. | +| Parameter | Notes | +| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `-Scope` | Azure resource ID of the scope. Defaults to current Az context subscription. | +| `-Name` | Filter by export name. Supports wildcards. | +| `-Dataset` | Filter by dataset type (e.g., `FocusCost`, `PriceSheet`). | +| `-DatasetVersion` | Filter by schema version. | +| `-StorageAccountId` | Filter by destination storage account. | +| `-StorageContainer` | Filter by container name. Supports wildcards. | +| `-RunHistory` | Switch. When present, the API response is expanded to include run history for each export. This is the enumeration primitive a future auto-download bridge (T-5000.9) would use to discover available runs and their storage paths. | **Enumerate run history (future auto-bridge use):** @@ -93,12 +93,12 @@ Get-FinOpsCostExport -Scope '/subscriptions/' -Dataset FocusCost Triggers an on-demand run of an existing export using the Cost Management Run API. -| Parameter | Notes | -|-----------|-------| -| `-Name` | Required. Name of the export to run. | -| `-Scope` | Azure resource ID. Defaults to current Az context subscription. | -| `-StartDate` / `-EndDate` | Optional date range override. | -| `-Backfill` | Number of prior months to run in sequence. | +| Parameter | Notes | +| ------------------------- | --------------------------------------------------------------- | +| `-Name` | Required. Name of the export to run. | +| `-Scope` | Azure resource ID. Defaults to current Az context subscription. | +| `-StartDate` / `-EndDate` | Optional date range override. | +| `-Backfill` | Number of prior months to run in sequence. | --- @@ -151,10 +151,10 @@ export/ Defined in `$script:DatasetTableMap` at `ingest.ps1` lines 59–62: -| Token | Raw table | Ingestion mapping | Cost Management dataset | -|-------|-----------|------------------|------------------------| -| `ms--focus-cost` | `Costs_raw` | `Costs_raw_mapping` | `FocusCost` | -| `ms--pricesheet` | `Prices_raw` | `Prices_raw_mapping` | `PriceSheet` | +| Token | Raw table | Ingestion mapping | Cost Management dataset | +| ---------------- | ------------ | -------------------- | ----------------------- | +| `ms--focus-cost` | `Costs_raw` | `Costs_raw_mapping` | `FocusCost` | +| `ms--pricesheet` | `Prices_raw` | `Prices_raw_mapping` | `PriceSheet` | Any directory name under `/` that is not one of these tokens is silently skipped with a warning (`ingest.ps1`, lines 248–252). @@ -165,13 +165,13 @@ Each `/` directory must contain a `manifest.json` file. The script rea this file in `Read-RunManifest` (`ingest.ps1`, lines 209–215) to obtain sort and row-count metadata. Fields consumed by `ingest.ps1`: -| JSON path | Required | Purpose | -|-----------|----------|---------| -| `runInfo.submittedTime` | Recommended | Primary sort key for latest-run selection. ISO 8601 string. | -| `runInfo.createdDate` | Fallback | Used when `submittedTime` is absent. | -| `dataRowCount` | Recommended | Total expected row count for the run (shown in the plan summary). | -| `blobs[].blobName` | Recommended | Matched by filename suffix to resolve per-file row counts. | -| `blobs[].dataRowCount` | Recommended | Expected rows for each parquet file (used in verify summary). | +| JSON path | Required | Purpose | +| ----------------------- | ----------- | ----------------------------------------------------------------- | +| `runInfo.submittedTime` | Recommended | Primary sort key for latest-run selection. ISO 8601 string. | +| `runInfo.createdDate` | Fallback | Used when `submittedTime` is absent. | +| `dataRowCount` | Recommended | Total expected row count for the run (shown in the plan summary). | +| `blobs[].blobName` | Recommended | Matched by filename suffix to resolve per-file row counts. | +| `blobs[].dataRowCount` | Recommended | Expected rows for each parquet file (used in verify summary). | **Minimum viable manifest for manual staging:** @@ -223,29 +223,29 @@ file, the file is skipped. (`ingest.ps1`, lines 741–748.) `Ingest_Manifest` schema: -| Column | Type | Notes | -|--------|------|-------| -| `scope` | string | Matches `` folder name. | -| `export_type` | string | Matches `` folder name. | -| `period` | string | Matches `` folder name. | -| `run_uuid` | string | Matches `` folder name. | -| `file_name` | string | Parquet file name (base name only). | -| `file_size` | long | File size in bytes at ingest time. | -| `rows_ingested` | long | Expected rows (from manifest, or 0 if not available). | -| `checksum_sha256` | string | SHA-256 hex of the parquet file. | -| `ingested_at` | datetime | UTC timestamp of ingest. | +| Column | Type | Notes | +| ----------------- | -------- | ----------------------------------------------------- | +| `scope` | string | Matches `` folder name. | +| `export_type` | string | Matches `` folder name. | +| `period` | string | Matches `` folder name. | +| `run_uuid` | string | Matches `` folder name. | +| `file_name` | string | Parquet file name (base name only). | +| `file_size` | long | File size in bytes at ingest time. | +| `rows_ingested` | long | Expected rows (from manifest, or 0 if not available). | +| `checksum_sha256` | string | SHA-256 hex of the parquet file. | +| `ingested_at` | datetime | UTC timestamp of ingest. | Defined at `ingest.ps1`, `Initialize-IngestManifestTable` / `Insert-ManifestRow` (lines 299–351). ### Overwrite semantics summary -| Condition | Behavior | -|-----------|----------| -| Same `(scope, type, period, run-uuid)`, same file checksum | **Skip.** File is already in `Ingest_Manifest` with a matching SHA-256; no ingest, no new row. (`ingest.ps1` lines 744–748.) | +| Condition | Behavior | +| ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Same `(scope, type, period, run-uuid)`, same file checksum | **Skip.** File is already in `Ingest_Manifest` with a matching SHA-256; no ingest, no new row. (`ingest.ps1` lines 744–748.) | | Same `(scope, type, period, run-uuid)`, **different** file checksum | **Double-ingest — do not do this.** The old `Ingest_Manifest` row is not deleted and the prior extents are not dropped; a second ingest call is issued and a second `Ingest_Manifest` row is appended for the same key. The raw table will contain duplicate data. (`ingest.ps1` lines 744–776.) | -| Same `(scope, type, period)`, **new** `run-uuid` | **Safe replace.** `Drop-SupersededExtents` drops extents tagged with older run-uuids from the raw table and deletes their `Ingest_Manifest` rows before the new run is ingested. (`ingest.ps1` lines 571–613.) | -| Different `(scope, type, period)` | **Independent.** No interaction with other tuples. | +| Same `(scope, type, period)`, **new** `run-uuid` | **Safe replace.** `Drop-SupersededExtents` drops extents tagged with older run-uuids from the raw table and deletes their `Ingest_Manifest` rows before the new run is ingested. (`ingest.ps1` lines 571–613.) | +| Different `(scope, type, period)` | **Independent.** No interaction with other tuples. | > **How to correct or replace data:** stage the replacement parquet under a **new ``** > directory for the same `(scope, type, period)`. Give it a `manifest.json` with a later @@ -279,6 +279,7 @@ All fields that `ingest.ps1` consumes are present. 4. Run `make ingest` (or `pwsh scripts/ingest.ps1`). **Notes:** + - If the export was configured with `-Format Csv` (the default), the data is CSV, not parquet, and cannot be ingested directly. **Always configure `-Format Parquet`.** - The dataset type token (`ms--focus-cost`) corresponds to the `FocusCost` export type. @@ -347,6 +348,7 @@ export/ ``` In this tree, `ingest.ps1` ingests four runs: + - `ea / ms--focus-cost / 20260501-20260531` → one run, ingested into `Costs_raw`. - `ea / ms--pricesheet / 20260501-20260531` → one run, ingested into `Prices_raw`. - `sub-prod / ms--focus-cost / 20260501-20260531` → two runs; the `_004` run is latest, From 91899bdba4ce2156bb53eb80b46e772035479370 Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 18 Jun 2026 07:08:44 -0700 Subject: [PATCH 16/51] chore(finops-hub-local): pin .md to LF in .gitattributes Matches the repo-wide markdown convention (all .md stored LF) and prevents CRLF drift for Windows contributors under core.autocrlf=true. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- src/templates/finops-hub-local/.gitattributes | 1 + 1 file changed, 1 insertion(+) diff --git a/src/templates/finops-hub-local/.gitattributes b/src/templates/finops-hub-local/.gitattributes index 85097a122..013b09e81 100644 --- a/src/templates/finops-hub-local/.gitattributes +++ b/src/templates/finops-hub-local/.gitattributes @@ -11,6 +11,7 @@ *.sh text eol=lf *.kql text eol=lf *.json text eol=lf +*.md text eol=lf Makefile text eol=lf *.yml text eol=lf *.yaml text eol=lf From c4a3908105cc429fa123ec235e0a59e3cdbee6f2 Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 18 Jun 2026 07:36:02 -0700 Subject: [PATCH 17/51] docs(finops-hub-local): add 'build by hand' runbook MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add notes/build-by-hand.md — the manual runbook for standing up a local FinOps hub: six steps, about a dozen raw commands (start emulator, create the two databases, load the schema, ingest Parquet, build the final tables, query). Each schema bundle is a single .execute database script command, so loading is one POST per database; the doc shows the exact curl/KQL for every step, verified against the emulator (118/118 statements load in one call). A closing table maps each manual step to the helper that automates it, so the scripts are no longer a black box. Link it from the README. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- src/templates/finops-hub-local/README.md | 5 + .../finops-hub-local/notes/build-by-hand.md | 179 ++++++++++++++++++ 2 files changed, 184 insertions(+) create mode 100644 src/templates/finops-hub-local/notes/build-by-hand.md diff --git a/src/templates/finops-hub-local/README.md b/src/templates/finops-hub-local/README.md index f7c10a38e..b43888b47 100644 --- a/src/templates/finops-hub-local/README.md +++ b/src/templates/finops-hub-local/README.md @@ -42,6 +42,11 @@ variants), the same open data, and the same query catalog. - 🍎 **[macOS](notes/install-mac.md)** — secondary; Apple Silicon runs the engine under Rosetta emulation. Covers Rosetta and Docker Desktop memory. +**Want to understand what the helpers do?** [Build a local FinOps hub by hand](notes/build-by-hand.md) +walks the whole setup as a dozen raw commands — start the emulator, create the two +databases, load the schema, ingest, build the final tables, query. The helper scripts +below automate exactly those steps. + --- ## Quickstart diff --git a/src/templates/finops-hub-local/notes/build-by-hand.md b/src/templates/finops-hub-local/notes/build-by-hand.md new file mode 100644 index 000000000..cec6b6583 --- /dev/null +++ b/src/templates/finops-hub-local/notes/build-by-hand.md @@ -0,0 +1,179 @@ +# Build a local FinOps hub by hand + +This is the manual runbook. It shows every command needed to stand up a local FinOps +hub on the Kusto emulator and load your FOCUS cost data — no helper scripts. The +[helper scripts](#automating-the-steps) automate exactly these steps; read this first so +you know what they do. + +The whole thing is six steps and about a dozen commands: + +1. [Start the emulator](#1-start-the-emulator) +2. [Create the two databases](#2-create-the-two-databases) +3. [Load the schema](#3-load-the-schema) +4. [Ingest your Parquet exports](#4-ingest-your-parquet-exports) +5. [Build the final tables](#5-build-the-final-tables) +6. [Query](#6-query) + +## Prerequisites + +- Docker and PowerShell 7 — see [install-windows.md](install-windows.md) or + [install-mac.md](install-mac.md). +- FOCUS cost exports as Parquet, staged under `export/` — see + [staging-contract.md](staging-contract.md). +- The two schema bundles, which the toolkit build produces from the FinOps hub KQL: + `release/finops-hub-fabric-setup-Ingestion.kql` and + `release/finops-hub-fabric-setup-Hub.kql`. Run `npm run build` (or + `pwsh ./src/scripts/Build-Toolkit.ps1 -Template finops-hub`) once to generate them. + +## How to run a command + +The emulator is a plain HTTP endpoint with no auth. Send any management command (a `.` +command) to `/v1/rest/mgmt` and any query to `/v1/rest/query`. Every command below can be +run with `curl`: + +```bash +curl -s "http://localhost:8082/v1/rest/mgmt" \ + -H "Content-Type: application/json" \ + -H "x-ms-client-version: Kusto.Python.Client:1.0.0" \ + -d '{"db":"","csl":""}' +``` + +You can also paste commands into the query window of the +[ADX web UI](dashboard.md) connected to `http://localhost:8082`. + +--- + +## 1. Start the emulator + +```bash +docker compose up -d --wait +``` + +`--wait` blocks until the container's healthcheck passes, so the engine is ready to +answer when the command returns. The emulator mounts your `export/` directory read-only +at `/data/export` inside the container. + +## 2. Create the two databases + +A FinOps hub uses two databases — `Ingestion` for the raw and transformed data, and `Hub` +for the view functions analysts query. Create both, persisted so they survive a restart: + +```kusto +// db: NetDefaultDB +.create database Ingestion persist (@'/kustodata/dbs/Ingestion/md', @'/kustodata/dbs/Ingestion/data') +.create database Hub persist (@'/kustodata/dbs/Hub/md', @'/kustodata/dbs/Hub/data') +``` + +## 3. Load the schema + +Each schema bundle is a single `.execute database script` command, so loading it is one +POST per database — the engine runs every statement in the file in order. + +The Ingestion bundle has one placeholder, `$$rawRetentionInDays$$`, the raw-data +retention in days. Replace it with a number (for example `90`) before loading. + +```bash +# Ingestion schema (raw tables, transform functions, final tables, open data) +sed 's/\$\$rawRetentionInDays\$\$/90/g' release/finops-hub-fabric-setup-Ingestion.kql \ + | python3 -c 'import sys,json; print(json.dumps({"db":"Ingestion","csl":sys.stdin.read()}))' \ + | curl -s "http://localhost:8082/v1/rest/mgmt" \ + -H "Content-Type: application/json" \ + -H "x-ms-client-version: Kusto.Python.Client:1.0.0" --data-binary @- + +# Hub schema (the Costs/Prices/Transactions view functions) +python3 -c 'import sys,json; print(json.dumps({"db":"Hub","csl":open("release/finops-hub-fabric-setup-Hub.kql").read()}))' \ + | curl -s "http://localhost:8082/v1/rest/mgmt" \ + -H "Content-Type: application/json" \ + -H "x-ms-client-version: Kusto.Python.Client:1.0.0" --data-binary @- +``` + +The bundle is loaded with `ContinueOnErrors=true`, so the response is a table with one row +per statement and its result — scan it for any `Failed` rows. The Hub functions reference +`database('Ingestion').*`; those resolve once both databases exist. + +## 4. Ingest your Parquet exports + +The schema created the raw tables (`Costs_raw`, `Prices_raw`) and their Parquet ingestion +mappings (`Costs_raw_mapping`, `Prices_raw_mapping`). Ingest the files mounted at +`/data/export`. List one `h@'...'` URI per file, comma-separated, to load many files in a +single command: + +```kusto +// db: Ingestion — cost exports +.ingest into table Costs_raw ( + h@'/data/export//ms--focus-cost///part_0.parquet', + h@'/data/export//ms--focus-cost///part_1.parquet' + ) with (format='parquet', ingestionMappingReference='Costs_raw_mapping') + +// db: Ingestion — price sheets +.ingest into table Prices_raw ( + h@'/data/export//ms--pricesheet///part_0.parquet' + ) with (format='parquet', ingestionMappingReference='Prices_raw_mapping') +``` + +Confirm the row counts: + +```kusto +// db: Ingestion +Costs_raw | count +Prices_raw | count +``` + +## 5. Build the final tables + +The raw tables hold the data as-exported. The FinOps transform functions +(`Costs_transform_v1_2`, `Prices_transform_v1_2`) apply the FOCUS normalization; run them +into the final tables: + +```kusto +// db: Ingestion +.set-or-append Costs_final_v1_2 <| Costs_transform_v1_2() +.set-or-append Prices_final_v1_2 <| Prices_transform_v1_2() +``` + +**Large datasets:** a single-pass transform materializes the whole table in memory and can +exceed the container's `MEM_LIMIT` on tens of millions of rows. When that happens, run the +transform one source extent at a time so each pass is bounded: + +```kusto +// db: Ingestion — repeat per extent_id() group +.set-or-append Prices_final_v1_2 <| + let Prices_raw = __table("Prices_raw", 'All', 'AllButRowStore') + | where extent_id() in (, ); + Prices_transform_v1_2() +``` + +The [`ingest.ps1`](../scripts/ingest.ps1) helper does this automatically (see +[performance.md](performance.md) for the memory measurements). Doing it by hand for a huge +table is impractical — this is the main reason to use the helper. + +## 6. Query + +The `Hub` database exposes the same view functions a deployed hub does. Query them +directly: + +```kusto +// db: Hub +Costs_v1_2() +| summarize EffectiveCost = round(sum(EffectiveCost), 2) by ServiceName +| top 10 by EffectiveCost +``` + +That is a working local FinOps hub. + +--- + +## Automating the steps + +The helper scripts in [`scripts/`](../scripts/) are this runbook, automated. They add +nothing the engine doesn't already do — they just save you the manual file-listing and, +for large data, the per-extent chunking: + +| Step | By hand | Helper | +| ----------------------------- | ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | +| 2–3. Create DBs + load schema | `.create database` ×2, POST each bundle | [`load-ftk-kql.ps1`](../scripts/load-ftk-kql.ps1) | +| 4–5. Ingest + build finals | `.ingest` per file, `.set-or-append` per table | [`ingest.ps1`](../scripts/ingest.ps1) — discovers your files, skips already-loaded ones, and chunks the transform for large tables | +| 6. Query | POST to `/v1/rest/query` | [`ftk.ps1`](../scripts/ftk.ps1) — also runs the published query catalog | + +On macOS and Linux the [`Makefile`](../Makefile) wraps these (`make up`, `make +load-ftk-kql`, `make ingest`). See the [README](../README.md) for the quickstart. From 41abdd45c290082a7889376cdc2c0df040040d68 Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 18 Jun 2026 07:41:26 -0700 Subject: [PATCH 18/51] refactor(finops-hub-local): load each schema bundle in one call The engine runs a whole bundle natively via '.execute database script with (ContinueOnErrors=true)', returning a per-statement result table. So wrap each built bundle and POST it once instead of splitting it into ~157 commands and posting each in a loop. Removes the command splitter and per-command loop (~48 lines); per-statement diagnostics are preserved by parsing the result table. Verified: 118/118 Ingestion and 39/39 Hub statements load in one call each, schema reload is non-destructive (data intact at the known total), and the dry-run path is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../finops-hub-local/scripts/load-ftk-kql.ps1 | 106 +++++------------- 1 file changed, 29 insertions(+), 77 deletions(-) diff --git a/src/templates/finops-hub-local/scripts/load-ftk-kql.ps1 b/src/templates/finops-hub-local/scripts/load-ftk-kql.ps1 index 1775adb04..38b14168a 100644 --- a/src/templates/finops-hub-local/scripts/load-ftk-kql.ps1 +++ b/src/templates/finops-hub-local/scripts/load-ftk-kql.ps1 @@ -76,7 +76,6 @@ if (Test-Path $settingsFile) { } catch { <# silently fall back to the 3650 default above #> } } -$script:TopLevelCommandRegex = [regex]'^\.[a-z]' $script:IdempotentOkTypes = @('Kusto.Common.Svc.Exceptions.EntityNameAlreadyExistsException') # --------------------------------------------------------------------------- # @@ -142,46 +141,6 @@ function Invoke-SubstMacros { return $Text.Replace('$$rawRetentionInDays$$', $script:RawRetentionDays) } -# --------------------------------------------------------------------------- # -# KQL command splitter -# --------------------------------------------------------------------------- # -function Split-KqlCommands { - param([Parameter(Mandatory)] [string] $ScriptText) - $lines = $ScriptText -split "(?<=`n)", 0 - if ($lines.Count -gt 0 -and $lines[-1] -eq '') { $lines = $lines[0..($lines.Count - 2)] } - - $blocks = [System.Collections.Generic.List[object]]::new() - $current = [System.Collections.Generic.List[string]]::new() - foreach ($line in $lines) { - if ($script:TopLevelCommandRegex.IsMatch($line)) { - if ($current.Count -gt 0) { [void]$blocks.Add([string[]]$current.ToArray()) } - $current = [System.Collections.Generic.List[string]]::new() - [void]$current.Add($line) - } - else { [void]$current.Add($line) } - } - if ($current.Count -gt 0) { [void]$blocks.Add([string[]]$current.ToArray()) } - - $cleaned = [System.Collections.Generic.List[string]]::new() - foreach ($blockObject in $blocks) { - $block = [System.Collections.Generic.List[string]]::new() - foreach ($entry in [string[]]$blockObject) { [void]$block.Add($entry) } - $hasCommand = $false - foreach ($entry in $block) { - if ($script:TopLevelCommandRegex.IsMatch($entry)) { $hasCommand = $true; break } - } - if (-not $hasCommand) { continue } - while ($block.Count -gt 0) { - $last = $block[$block.Count - 1] - if ($last.Trim() -eq '' -or $last.TrimStart().StartsWith('//')) { $block.RemoveAt($block.Count - 1) } - else { break } - } - $text = ([string]::Concat([string[]]$block.ToArray())).TrimEnd() - if ($text) { [void]$cleaned.Add($text) } - } - return [string[]]$cleaned.ToArray() -} - # --------------------------------------------------------------------------- # # Kustainer REST poster # --------------------------------------------------------------------------- # @@ -255,16 +214,6 @@ function New-StepReport { } } -function Get-FirstCommandLine { - param([Parameter(Mandatory)] [string] $CommandText) - foreach ($line in ($CommandText -split "`n")) { - if ($script:TopLevelCommandRegex.IsMatch($line)) { return $line.Trim() } - } - $lines = $CommandText -split "`n" - if ($lines.Count -gt 0) { return $lines[0] } - return '' -} - function Invoke-BundleLoad { param( [Parameter(Mandatory)] [string] $Label, @@ -272,38 +221,41 @@ function Invoke-BundleLoad { [Parameter(Mandatory)] [string] $DatabaseName, [bool] $IsDryRun = $false ) - $report = New-StepReport -Label $Label - $commands = @(Split-KqlCommands -ScriptText $BundleText) - $report.Total = $commands.Count + $report = New-StepReport -Label $Label Write-Host "" - Write-Host "=== $Label [$DatabaseName] ($($report.Total) commands) ===" - for ($i = 0; $i -lt $commands.Count; $i++) { - $commandText = $commands[$i] - $firstLine = Get-FirstCommandLine -CommandText $commandText - $preview = if ($firstLine.Length -gt 110) { $firstLine.Substring(0, 110) } else { $firstLine } - if ($IsDryRun) { - Write-Host (' [{0,3}] dry {1}' -f ($i + 1), $preview) - continue - } - $result = Post-Kql -Csl $commandText -DatabaseName $DatabaseName - if ($result.Status -ge 200 -and $result.Status -lt 300) { + Write-Host "=== $Label [$DatabaseName] ===" + if ($IsDryRun) { + Write-Host " dry would submit the bundle as one '.execute database script' command" + return $report + } + # The whole bundle is submitted as a single database script. ContinueOnErrors=true + # makes the engine attempt every statement and return a per-statement result table + # (OperationId, CommandType, CommandText, Result, Reason) -- so we keep per-statement + # diagnostics without splitting the bundle ourselves. + $databaseScript = ".execute database script with (ContinueOnErrors=true)`n<|`n$BundleText" + $result = Post-Kql -Csl $databaseScript -DatabaseName $DatabaseName + if ($result.Status -lt 200 -or $result.Status -ge 300) { + $parsed = Parse-KustoError -Body $result.Body + $report.Total = 1 + [void]$report.Failed.Add([pscustomobject]@{ Command = $Label; Error = "HTTP $($result.Status): [$($parsed[0])] $($parsed[1])" }) + Write-Host " FAIL batch submission failed: HTTP $($result.Status) [$($parsed[0])] $($parsed[1])" + return $report + } + $rows = @() + try { $rows = @((ConvertFrom-Json $result.Body).Tables[0].Rows) } catch { } + $report.Total = $rows.Count + foreach ($row in $rows) { + if ($row[3] -eq 'Completed') { $report.Ok++ - Write-Host (' [{0,3}] OK {1}' -f ($i + 1), $preview) - } - elseif (Test-IdempotentSuccess -Body $result.Body) { - $report.IdempotentOk++ - $parsed = Parse-KustoError -Body $result.Body - Write-Host (' [{0,3}] OK* {1} -- {2}' -f ($i + 1), $preview, $parsed[1]) } else { - $parsed = Parse-KustoError -Body $result.Body - $short = "[$($parsed[0])] $($parsed[1])" - if ($short.Length -gt 400) { $short = $short.Substring(0, 400) } - [void]$report.Failed.Add([pscustomobject]@{ Command = $preview; Error = "HTTP $($result.Status): $short" }) - Write-Host (' [{0,3}] FAIL {1}' -f ($i + 1), $preview) - Write-Host " HTTP $($result.Status): $short" + $commandText = if ($row[2]) { ($row[2] -split "`n" | Where-Object { $_.Trim() } | Select-Object -First 1) } else { $row[1] } + if ($commandText.Length -gt 110) { $commandText = $commandText.Substring(0, 110) } + [void]$report.Failed.Add([pscustomobject]@{ Command = $commandText; Error = $row[4] }) } } + Write-Host " loaded $($report.Total) statements: $($report.Ok) completed, $($report.Failed.Count) failed" + foreach ($failure in $report.Failed) { Write-Host " FAIL $($failure.Command) -- $($failure.Error)" } return $report } From ed1f4caeac43596a1fd5934d607f2244ff07d4e7 Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 18 Jun 2026 08:06:00 -0700 Subject: [PATCH 19/51] fix(finops-hub-local): correct the by-hand ingest steps (update policy auto-transforms) The schema ships the raw->final update policy enabled, so .ingest into Costs_raw already runs Costs_transform_v1_2() and populates Costs_final_v1_2 inline -- the same mechanism a deployed hub uses. The previous runbook then ALSO ran a manual .set-or-append, which appends a second copy and doubles the final tables (verified: ingest -> dst=1, manual backfill -> dst=2). Rewrite step 4 to note the finals auto-populate and warn against the double backfill, and recast step 5 as the large-dataset path the helper uses: disable the policy, bulk-ingest, run one chunked transform, re-enable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../finops-hub-local/notes/build-by-hand.md | 75 +++++++++++-------- 1 file changed, 44 insertions(+), 31 deletions(-) diff --git a/src/templates/finops-hub-local/notes/build-by-hand.md b/src/templates/finops-hub-local/notes/build-by-hand.md index cec6b6583..c7c2e6488 100644 --- a/src/templates/finops-hub-local/notes/build-by-hand.md +++ b/src/templates/finops-hub-local/notes/build-by-hand.md @@ -10,8 +10,8 @@ The whole thing is six steps and about a dozen commands: 1. [Start the emulator](#1-start-the-emulator) 2. [Create the two databases](#2-create-the-two-databases) 3. [Load the schema](#3-load-the-schema) -4. [Ingest your Parquet exports](#4-ingest-your-parquet-exports) -5. [Build the final tables](#5-build-the-final-tables) +4. [Ingest your Parquet exports](#4-ingest-your-parquet-exports) — the final tables populate themselves +5. [Large datasets: defer the transform](#5-large-datasets-defer-the-transform) 6. [Query](#6-query) ## Prerequisites @@ -94,9 +94,15 @@ per statement and its result — scan it for any `Failed` rows. The Hub function ## 4. Ingest your Parquet exports The schema created the raw tables (`Costs_raw`, `Prices_raw`) and their Parquet ingestion -mappings (`Costs_raw_mapping`, `Prices_raw_mapping`). Ingest the files mounted at -`/data/export`. List one `h@'...'` URI per file, comma-separated, to load many files in a -single command: +mappings (`Costs_raw_mapping`, `Prices_raw_mapping`). It also enabled an **update policy** +on each final table: as rows land in `Costs_raw`, the engine runs `Costs_transform_v1_2()` +automatically and appends the FOCUS-normalized result to `Costs_final_v1_2` (and the same +for prices). This is the exact mechanism a deployed FinOps hub uses — so on the local +emulator, **ingesting the raw data is all you need**; the final tables populate +themselves. + +Ingest the files mounted at `/data/export`. List one `h@'...'` URI per file, +comma-separated, to load many files in a single command: ```kusto // db: Ingestion — cost exports @@ -111,41 +117,48 @@ single command: ) with (format='parquet', ingestionMappingReference='Prices_raw_mapping') ``` -Confirm the row counts: +Confirm the raw counts and that the final tables filled in via the update policy: ```kusto // db: Ingestion Costs_raw | count +Costs_final_v1_2 | count // populated automatically by the update policy Prices_raw | count +Prices_final_v1_2 | count ``` -## 5. Build the final tables +> Do **not** also run `.set-or-append Costs_final_v1_2 <| Costs_transform_v1_2()` here — +> with the update policy enabled, the transform already ran during ingest, and a manual +> backfill would append a second copy and double the final tables. -The raw tables hold the data as-exported. The FinOps transform functions -(`Costs_transform_v1_2`, `Prices_transform_v1_2`) apply the FOCUS normalization; run them -into the final tables: +## 5. Large datasets: defer the transform -```kusto +The update policy is `IsTransactional`, so it transforms every ingested batch inline. That +is ideal for incremental exports, but a one-shot bulk load of tens of millions of rows +makes each batch's transform slow and memory-heavy, and a whole-table transform can exceed +the container's `MEM_LIMIT`. For a large historical load, turn the policy off, ingest, then +run the transform once — in chunks bounded by source extent: + +````kusto // db: Ingestion -.set-or-append Costs_final_v1_2 <| Costs_transform_v1_2() -.set-or-append Prices_final_v1_2 <| Prices_transform_v1_2() -``` +// 1. disable the update policy so ingest doesn't transform inline +.alter table Costs_final_v1_2 policy update ```[{"IsEnabled":false,"Source":"Costs_raw","Query":"Costs_transform_v1_2()","IsTransactional":true,"PropagateIngestionProperties":true}]``` -**Large datasets:** a single-pass transform materializes the whole table in memory and can -exceed the container's `MEM_LIMIT` on tens of millions of rows. When that happens, run the -transform one source extent at a time so each pass is bounded: +// 2. ingest all the raw data (as in step 4) -```kusto -// db: Ingestion — repeat per extent_id() group -.set-or-append Prices_final_v1_2 <| - let Prices_raw = __table("Prices_raw", 'All', 'AllButRowStore') +// 3. run the transform once, one extent group at a time so each pass is bounded +.set-or-append Costs_final_v1_2 <| + let Costs_raw = __table("Costs_raw", 'All', 'AllButRowStore') | where extent_id() in (, ); - Prices_transform_v1_2() -``` + Costs_transform_v1_2() + +// 4. re-enable the update policy (set IsEnabled back to true) +```` -The [`ingest.ps1`](../scripts/ingest.ps1) helper does this automatically (see -[performance.md](performance.md) for the memory measurements). Doing it by hand for a huge -table is impractical — this is the main reason to use the helper. +The [`ingest.ps1`](../scripts/ingest.ps1) helper does exactly this — disable, bulk-ingest, +chunked backfill, re-enable — so you don't have to manage the policy or the extent GUIDs by +hand. See [performance.md](performance.md) for the memory measurements that set the chunk +threshold. **For large data, use the helper; this section is what it does under the hood.** ## 6. Query @@ -169,11 +182,11 @@ The helper scripts in [`scripts/`](../scripts/) are this runbook, automated. The nothing the engine doesn't already do — they just save you the manual file-listing and, for large data, the per-extent chunking: -| Step | By hand | Helper | -| ----------------------------- | ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | -| 2–3. Create DBs + load schema | `.create database` ×2, POST each bundle | [`load-ftk-kql.ps1`](../scripts/load-ftk-kql.ps1) | -| 4–5. Ingest + build finals | `.ingest` per file, `.set-or-append` per table | [`ingest.ps1`](../scripts/ingest.ps1) — discovers your files, skips already-loaded ones, and chunks the transform for large tables | -| 6. Query | POST to `/v1/rest/query` | [`ftk.ps1`](../scripts/ftk.ps1) — also runs the published query catalog | +| Step | By hand | Helper | +| ----------------------------- | ------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 2–3. Create DBs + load schema | `.create database` ×2, POST each bundle | [`load-ftk-kql.ps1`](../scripts/load-ftk-kql.ps1) | +| 4–5. Ingest | `.ingest` per file; for large data, disable policy + backfill | [`ingest.ps1`](../scripts/ingest.ps1) — discovers your files, skips already-loaded ones, and for large tables disables the update policy and chunks the transform | +| 6. Query | POST to `/v1/rest/query` | [`ftk.ps1`](../scripts/ftk.ps1) — also runs the published query catalog | On macOS and Linux the [`Makefile`](../Makefile) wraps these (`make up`, `make load-ftk-kql`, `make ingest`). See the [README](../README.md) for the quickstart. From a961d73137ee6cd1b377c4fa884eedfffc8e1262 Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 18 Jun 2026 08:07:30 -0700 Subject: [PATCH 20/51] docs(finops-hub-local): align README by-hand summary with the update-policy flow Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- src/templates/finops-hub-local/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/templates/finops-hub-local/README.md b/src/templates/finops-hub-local/README.md index b43888b47..bed5db483 100644 --- a/src/templates/finops-hub-local/README.md +++ b/src/templates/finops-hub-local/README.md @@ -44,8 +44,8 @@ variants), the same open data, and the same query catalog. **Want to understand what the helpers do?** [Build a local FinOps hub by hand](notes/build-by-hand.md) walks the whole setup as a dozen raw commands — start the emulator, create the two -databases, load the schema, ingest, build the final tables, query. The helper scripts -below automate exactly those steps. +databases, load the schema, ingest (the final tables transform themselves via an update +policy), and query. The helper scripts below automate exactly those steps. --- From 421b9a71ef08941805c7e48709f1ccf04f578697 Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 18 Jun 2026 08:17:52 -0700 Subject: [PATCH 21/51] fix(finops-hub-local): add .build.config to keep runtime data out of release Build-Toolkit copies a template directory wholesale into release/. Without a build config, a local build would bundle the user's export/ (customer-confidential FOCUS data), .env (secrets), settings.json, and kustainer-data/ runtime state into the released artifact. Add a .build.config that ignores those runtime/secret paths. Verified: a build with planted customer data + secrets strips all of them from release/ while keeping the template files (.env.example, settings.example.json, scripts, docs, dashboard). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- src/templates/finops-hub-local/.build.config | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 src/templates/finops-hub-local/.build.config diff --git a/src/templates/finops-hub-local/.build.config b/src/templates/finops-hub-local/.build.config new file mode 100644 index 000000000..db3c9f780 --- /dev/null +++ b/src/templates/finops-hub-local/.build.config @@ -0,0 +1,3 @@ +{ + "ignore": ["export", "kustainer-data", ".env", "settings.json", "*.log"] +} From 5278e2fb178a31ca293a682702699d4b4ef05366 Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 18 Jun 2026 08:37:50 -0700 Subject: [PATCH 22/51] Revert "refactor(finops-hub-local): load each schema bundle in one call" This reverts commit 41abdd45c290082a7889376cdc2c0df040040d68. --- .../finops-hub-local/scripts/load-ftk-kql.ps1 | 106 +++++++++++++----- 1 file changed, 77 insertions(+), 29 deletions(-) diff --git a/src/templates/finops-hub-local/scripts/load-ftk-kql.ps1 b/src/templates/finops-hub-local/scripts/load-ftk-kql.ps1 index 38b14168a..1775adb04 100644 --- a/src/templates/finops-hub-local/scripts/load-ftk-kql.ps1 +++ b/src/templates/finops-hub-local/scripts/load-ftk-kql.ps1 @@ -76,6 +76,7 @@ if (Test-Path $settingsFile) { } catch { <# silently fall back to the 3650 default above #> } } +$script:TopLevelCommandRegex = [regex]'^\.[a-z]' $script:IdempotentOkTypes = @('Kusto.Common.Svc.Exceptions.EntityNameAlreadyExistsException') # --------------------------------------------------------------------------- # @@ -141,6 +142,46 @@ function Invoke-SubstMacros { return $Text.Replace('$$rawRetentionInDays$$', $script:RawRetentionDays) } +# --------------------------------------------------------------------------- # +# KQL command splitter +# --------------------------------------------------------------------------- # +function Split-KqlCommands { + param([Parameter(Mandatory)] [string] $ScriptText) + $lines = $ScriptText -split "(?<=`n)", 0 + if ($lines.Count -gt 0 -and $lines[-1] -eq '') { $lines = $lines[0..($lines.Count - 2)] } + + $blocks = [System.Collections.Generic.List[object]]::new() + $current = [System.Collections.Generic.List[string]]::new() + foreach ($line in $lines) { + if ($script:TopLevelCommandRegex.IsMatch($line)) { + if ($current.Count -gt 0) { [void]$blocks.Add([string[]]$current.ToArray()) } + $current = [System.Collections.Generic.List[string]]::new() + [void]$current.Add($line) + } + else { [void]$current.Add($line) } + } + if ($current.Count -gt 0) { [void]$blocks.Add([string[]]$current.ToArray()) } + + $cleaned = [System.Collections.Generic.List[string]]::new() + foreach ($blockObject in $blocks) { + $block = [System.Collections.Generic.List[string]]::new() + foreach ($entry in [string[]]$blockObject) { [void]$block.Add($entry) } + $hasCommand = $false + foreach ($entry in $block) { + if ($script:TopLevelCommandRegex.IsMatch($entry)) { $hasCommand = $true; break } + } + if (-not $hasCommand) { continue } + while ($block.Count -gt 0) { + $last = $block[$block.Count - 1] + if ($last.Trim() -eq '' -or $last.TrimStart().StartsWith('//')) { $block.RemoveAt($block.Count - 1) } + else { break } + } + $text = ([string]::Concat([string[]]$block.ToArray())).TrimEnd() + if ($text) { [void]$cleaned.Add($text) } + } + return [string[]]$cleaned.ToArray() +} + # --------------------------------------------------------------------------- # # Kustainer REST poster # --------------------------------------------------------------------------- # @@ -214,6 +255,16 @@ function New-StepReport { } } +function Get-FirstCommandLine { + param([Parameter(Mandatory)] [string] $CommandText) + foreach ($line in ($CommandText -split "`n")) { + if ($script:TopLevelCommandRegex.IsMatch($line)) { return $line.Trim() } + } + $lines = $CommandText -split "`n" + if ($lines.Count -gt 0) { return $lines[0] } + return '' +} + function Invoke-BundleLoad { param( [Parameter(Mandatory)] [string] $Label, @@ -221,41 +272,38 @@ function Invoke-BundleLoad { [Parameter(Mandatory)] [string] $DatabaseName, [bool] $IsDryRun = $false ) - $report = New-StepReport -Label $Label + $report = New-StepReport -Label $Label + $commands = @(Split-KqlCommands -ScriptText $BundleText) + $report.Total = $commands.Count Write-Host "" - Write-Host "=== $Label [$DatabaseName] ===" - if ($IsDryRun) { - Write-Host " dry would submit the bundle as one '.execute database script' command" - return $report - } - # The whole bundle is submitted as a single database script. ContinueOnErrors=true - # makes the engine attempt every statement and return a per-statement result table - # (OperationId, CommandType, CommandText, Result, Reason) -- so we keep per-statement - # diagnostics without splitting the bundle ourselves. - $databaseScript = ".execute database script with (ContinueOnErrors=true)`n<|`n$BundleText" - $result = Post-Kql -Csl $databaseScript -DatabaseName $DatabaseName - if ($result.Status -lt 200 -or $result.Status -ge 300) { - $parsed = Parse-KustoError -Body $result.Body - $report.Total = 1 - [void]$report.Failed.Add([pscustomobject]@{ Command = $Label; Error = "HTTP $($result.Status): [$($parsed[0])] $($parsed[1])" }) - Write-Host " FAIL batch submission failed: HTTP $($result.Status) [$($parsed[0])] $($parsed[1])" - return $report - } - $rows = @() - try { $rows = @((ConvertFrom-Json $result.Body).Tables[0].Rows) } catch { } - $report.Total = $rows.Count - foreach ($row in $rows) { - if ($row[3] -eq 'Completed') { + Write-Host "=== $Label [$DatabaseName] ($($report.Total) commands) ===" + for ($i = 0; $i -lt $commands.Count; $i++) { + $commandText = $commands[$i] + $firstLine = Get-FirstCommandLine -CommandText $commandText + $preview = if ($firstLine.Length -gt 110) { $firstLine.Substring(0, 110) } else { $firstLine } + if ($IsDryRun) { + Write-Host (' [{0,3}] dry {1}' -f ($i + 1), $preview) + continue + } + $result = Post-Kql -Csl $commandText -DatabaseName $DatabaseName + if ($result.Status -ge 200 -and $result.Status -lt 300) { $report.Ok++ + Write-Host (' [{0,3}] OK {1}' -f ($i + 1), $preview) + } + elseif (Test-IdempotentSuccess -Body $result.Body) { + $report.IdempotentOk++ + $parsed = Parse-KustoError -Body $result.Body + Write-Host (' [{0,3}] OK* {1} -- {2}' -f ($i + 1), $preview, $parsed[1]) } else { - $commandText = if ($row[2]) { ($row[2] -split "`n" | Where-Object { $_.Trim() } | Select-Object -First 1) } else { $row[1] } - if ($commandText.Length -gt 110) { $commandText = $commandText.Substring(0, 110) } - [void]$report.Failed.Add([pscustomobject]@{ Command = $commandText; Error = $row[4] }) + $parsed = Parse-KustoError -Body $result.Body + $short = "[$($parsed[0])] $($parsed[1])" + if ($short.Length -gt 400) { $short = $short.Substring(0, 400) } + [void]$report.Failed.Add([pscustomobject]@{ Command = $preview; Error = "HTTP $($result.Status): $short" }) + Write-Host (' [{0,3}] FAIL {1}' -f ($i + 1), $preview) + Write-Host " HTTP $($result.Status): $short" } } - Write-Host " loaded $($report.Total) statements: $($report.Ok) completed, $($report.Failed.Count) failed" - foreach ($failure in $report.Failed) { Write-Host " FAIL $($failure.Command) -- $($failure.Error)" } return $report } From e016dc307d31707584fad1986772380f586c61d1 Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 18 Jun 2026 09:50:38 -0700 Subject: [PATCH 23/51] fix(finops-hub-local): fix single-element-array unwrap in ingest backfill + correct notes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ingest.ps1: Get-TableExtents returned `@(...)`, which pipeline-unwraps a single-element result to a scalar under `Set-StrictMode -Version Latest`, so `(Get-TableExtents).Count` threw "property 'Count' cannot be found" whenever a raw table had exactly one extent — leaving final tables empty and failing parity (CHECK 2 BilledCost, CHECK 13 Prices). The chunked backfill (>2M rows) also mis-handled the array, producing space-joined GUIDs and a Kusto SYN0002 syntax error. Fix: keep `return @(...)` and wrap all four call sites with `@(...)`, which correctly handles empty/single/ multi-extent results. Apply the same `@(...)` wrap to the Build-Plan call site so the zero-plan "nothing to do" path no longer throws under StrictMode. Verified end-to-end: full ingest → 21/21 chunked extents → Prices_final 12,735,587 rows; parity 11 pass / 0 fail / 2 manual. notes: address duo-review findings — replace brittle ingest.ps1/ftk.ps1 line-number citations with function/step anchors; correct "four runs" to "three"; describe checksum-mismatch as fail-fast throw (not double-ingest); fix dashboard CLI examples to `pwsh scripts/ftk.ps1 ... --database`; soften dashboard "only change" wording; make Prices the large-data OOM worked example in build-by-hand; fix performance 9.4% rounding; correct .env.example measured row counts (1.35M / 12.7M). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- src/templates/finops-hub-local/.env.example | 4 +- .../finops-hub-local/notes/build-by-hand.md | 25 ++++++----- .../finops-hub-local/notes/dashboard.md | 14 ++++--- .../notes/ftk-kql-adaptations.md | 18 ++++---- .../finops-hub-local/notes/performance.md | 2 +- .../notes/staging-contract.md | 41 ++++++++++--------- .../finops-hub-local/scripts/ingest.ps1 | 4 +- 7 files changed, 58 insertions(+), 50 deletions(-) diff --git a/src/templates/finops-hub-local/.env.example b/src/templates/finops-hub-local/.env.example index 5aa8cdf60..cb0aa126e 100644 --- a/src/templates/finops-hub-local/.env.example +++ b/src/templates/finops-hub-local/.env.example @@ -8,9 +8,9 @@ HOST_PORT=8082 # Memory ceiling for the Kustainer container. # # 16g is the recommended default. The full Prices_transform_v1_2() backfill -# over 32M rows OOM'd at 8g on amd64-emulation (Apple Silicon under Rosetta). +# over ~12.7M rows OOM'd at 8g on amd64-emulation (Apple Silicon under Rosetta). # Costs-only workflows or smaller datasets work at 8g, but reliability across -# both Costs (13M rows) and Prices (32M rows) requires 16g. +# both Costs (~1.35M rows) and Prices (~12.7M rows) requires 16g. MEM_LIMIT=16g # Directory on the host that holds the FOCUS parquet exports. diff --git a/src/templates/finops-hub-local/notes/build-by-hand.md b/src/templates/finops-hub-local/notes/build-by-hand.md index c7c2e6488..479a4da97 100644 --- a/src/templates/finops-hub-local/notes/build-by-hand.md +++ b/src/templates/finops-hub-local/notes/build-by-hand.md @@ -136,29 +136,34 @@ Prices_final_v1_2 | count The update policy is `IsTransactional`, so it transforms every ingested batch inline. That is ideal for incremental exports, but a one-shot bulk load of tens of millions of rows makes each batch's transform slow and memory-heavy, and a whole-table transform can exceed -the container's `MEM_LIMIT`. For a large historical load, turn the policy off, ingest, then -run the transform once — in chunks bounded by source extent: +the container's `MEM_LIMIT`. In practice `Prices_final_v1_2` (~12.7M rows) is the table that +OOMs at 16g, while `Costs_final_v1_2` (~1.35M rows) transforms single-pass without trouble — +so the table below uses `Prices` as the worked example. For a large historical load, turn the +policy off, ingest, then run the transform once — in chunks bounded by source extent: ````kusto // db: Ingestion // 1. disable the update policy so ingest doesn't transform inline -.alter table Costs_final_v1_2 policy update ```[{"IsEnabled":false,"Source":"Costs_raw","Query":"Costs_transform_v1_2()","IsTransactional":true,"PropagateIngestionProperties":true}]``` +.alter table Prices_final_v1_2 policy update ```[{"IsEnabled":false,"Source":"Prices_raw","Query":"Prices_transform_v1_2()","IsTransactional":true,"PropagateIngestionProperties":true}]``` // 2. ingest all the raw data (as in step 4) // 3. run the transform once, one extent group at a time so each pass is bounded -.set-or-append Costs_final_v1_2 <| - let Costs_raw = __table("Costs_raw", 'All', 'AllButRowStore') +.set-or-append Prices_final_v1_2 <| + let Prices_raw = __table("Prices_raw", 'All', 'AllButRowStore') | where extent_id() in (, ); - Costs_transform_v1_2() + Prices_transform_v1_2() // 4. re-enable the update policy (set IsEnabled back to true) ```` -The [`ingest.ps1`](../scripts/ingest.ps1) helper does exactly this — disable, bulk-ingest, -chunked backfill, re-enable — so you don't have to manage the policy or the extent GUIDs by -hand. See [performance.md](performance.md) for the memory measurements that set the chunk -threshold. **For large data, use the helper; this section is what it does under the hood.** +`Costs_final_v1_2` needs the same treatment only if its row count grows past the single-pass +headroom — swap `Costs_raw` / `Costs_transform_v1_2()` / `Costs_final_v1_2` into the commands +above. The [`ingest.ps1`](../scripts/ingest.ps1) helper does exactly this for both tables — +disable, bulk-ingest, chunked backfill, re-enable — so you don't have to manage the policy or +the extent GUIDs by hand. See [performance.md](performance.md) for the memory measurements +that set the chunk threshold. **For large data, use the helper; this section is what it does +under the hood.** ## 6. Query diff --git a/src/templates/finops-hub-local/notes/dashboard.md b/src/templates/finops-hub-local/notes/dashboard.md index 80689cd90..651bf23c3 100644 --- a/src/templates/finops-hub-local/notes/dashboard.md +++ b/src/templates/finops-hub-local/notes/dashboard.md @@ -1,7 +1,9 @@ # Using the local ADX dashboard This dashboard (`../dashboard.json`) is the upstream FinOps hub dashboard localized to the -ftklocal Kusto emulator. The only change from the upstream file is: +ftklocal Kusto emulator. The only behaviorally significant change from the upstream file is +`clusterUri` (the `title`, `id`, and `eTag` metadata fields also differ, but they do not +affect tiles or queries): | Field | Upstream value | Local value | | --------------------------- | ----------------------------------------------------- | ----------------------- | @@ -25,15 +27,15 @@ view functions as a deployed hub: `Costs_v1_2`, `Prices_v1_2`, `Transactions_v1_ ## Proven path: query via the CLI The emulator's KQL REST API is plain HTTP with no authentication — you can query it directly -from the terminal today without a browser. Use `ftk query` (or `ftk run`) against the `Hub` -database: +from the terminal today without a browser. Use `ftk.ps1 query` (or `ftk.ps1 run`) against the +`Hub` database: ```bash # Smoke-test: confirm view functions are populated -ftk query --db Hub "Costs_v1_2 | take 10" +pwsh scripts/ftk.ps1 query --database Hub "Costs_v1_2 | take 10" # Run any KQL query interactively -ftk run --db Hub +pwsh scripts/ftk.ps1 run --database Hub ``` This is the correct first validation step on Mac. It does not depend on any browser @@ -167,7 +169,7 @@ dashboard data source to match. | `dataSources[0].database` = `Hub` | ✅ Verified in file | | Hub DB has `Costs_v1_2`, `Prices_v1_2`, etc. | ✅ Verified via `.show functions` on running emulator | | No upstream cluster names, billing IDs, or absolute paths in dashboard | ✅ Verified | -| `ftk query` / `ftk run` against Hub DB | ✅ Works (HTTP REST, no browser) | +| `ftk.ps1 query` / `ftk.ps1 run` against Hub DB | ✅ Works (HTTP REST, no browser) | | Browser can add `http://localhost:8082` from `https://dataexplorer.azure.com` | ✅ **CORS allowed** (emulator returns `Access-Control-Allow-Origin: https://dataexplorer.azure.com`); `localhost` is exempt from mixed-content blocking. Edge 143+/Chrome 142+ show a one-time **Local Network Access** prompt — click **Allow**. | | Dashboard tiles render data after import | ⚠️ **Visual render pending** — protocol path proven (tile queries return data via CORS); requires the interactive browser import + AAD sign-in + the LNA **Allow** click to confirm on screen | diff --git a/src/templates/finops-hub-local/notes/ftk-kql-adaptations.md b/src/templates/finops-hub-local/notes/ftk-kql-adaptations.md index e79d0e8a4..7205b13cd 100644 --- a/src/templates/finops-hub-local/notes/ftk-kql-adaptations.md +++ b/src/templates/finops-hub-local/notes/ftk-kql-adaptations.md @@ -3,10 +3,10 @@ There are **two distinct adaptation layers** in the local stack. This document covers both. -| Layer | Where it runs | Purpose | -| ------------- | -------------------------------------------------------- | ------------------------------------------------------------------------- | -| **Load-time** | `scripts/load-ftk-kql.ps1` | Loads upstream FTK KQL into the emulator at setup time (idempotent). | -| **On-read** | `scripts/ftk.ps1` `Convert-CatalogQuery` (lines 341–382) | Adapts catalog `.kql` files at query time before posting to the emulator. | +| Layer | Where it runs | Purpose | +| ------------- | ---------------------------------------- | ------------------------------------------------------------------------- | +| **Load-time** | `scripts/load-ftk-kql.ps1` | Loads upstream FTK KQL into the emulator at setup time (idempotent). | +| **On-read** | `scripts/ftk.ps1` `Convert-CatalogQuery` | Adapts catalog `.kql` files at query time before posting to the emulator. | The upstream KQL scripts live in `src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/scripts/`. @@ -158,27 +158,27 @@ in the seven scripts the loader processes. --- -## On-read adapter — `ftk.ps1 run` (`Convert-CatalogQuery`, lines 341–382) +## On-read adapter — `ftk.ps1 run` (`Convert-CatalogQuery`) `ftk.ps1 run` adapts catalog `.kql` files at query time before posting them to the emulator. This is a separate layer from the load-time adaptations above. The function `Convert-CatalogQuery` applies the following rules in order: -### Rule 1 — Date-window retarget (lines 352–359) +### Rule 1 — Date-window retarget (step `# 1. Date window`) Rewrites `let startDate = ...;` and `let endDate = ...;` bindings in catalog queries to match the `--start` / `--end` CLI arguments (or the last complete month when no range is supplied). This lets time-bounded catalog queries work locally without editing the `.kql` files. -### Rule 1b — `--param` typed-literal override (lines 361–371) +### Rule 1b — `--param` typed-literal override (step `# 1b.`) Rewrites any top-level `let = ...;` binding when `--param name=value` is supplied on the CLI. Values are serialised to KQL typed literals (datetime, timespan, bool, int, real, or string). Throws if `name` does not match a top-level `let` in the query (no silent no-ops). -### Rule 2 — `decimal`→`real` normalisation (lines 373–376) +### Rule 2 — `decimal`→`real` normalisation (step `# 2.`) ``` todecimal('') → real(null) @@ -190,7 +190,7 @@ Kustainer's free image does not support the `decimal` scalar type. The FTK query catalog uses `decimal()` for monetary columns in some queries. This substitution is lossless for the numeric range involved and is the verified compatibility fix. -### Rule 3 — Project-away tolerance (lines 378–379) +### Rule 3 — Project-away tolerance (step `# 3.`) If a catalog query contains a `project-away` clause and calls a hub view function (`Costs()`, `Prices()`, `Recommendations()`, or `Transactions()`), the adapter diff --git a/src/templates/finops-hub-local/notes/performance.md b/src/templates/finops-hub-local/notes/performance.md index 924b9d9e0..170e8f410 100644 --- a/src/templates/finops-hub-local/notes/performance.md +++ b/src/templates/finops-hub-local/notes/performance.md @@ -38,7 +38,7 @@ started just before the OOM attempt. | ------------------------------------------------- | -------------------- | ---------- | -------------- | ----------------- | ---------- | | 1. Empty engine (just started, no data) | 0 | 0.49 | 3.1 % | 0.51 | — | | 2. Schema + open-data loaded (`load-ftk-kql`) | 3,293 | 0.66 | 4.1 % | 0.68 | — | -| 3a. After Costs_raw ingested (15 parts) | 1,353,854 | 1.51 ¹ | 9.5 % | 2.13 ¹ | — | +| 3a. After Costs_raw ingested (15 parts) | 1,353,854 | 1.51 ¹ | 9.4 % | 2.13 ¹ | — | | 3b. After all raw ingested, no finals yet | 14,089,441 | 1.99 ¹ | 12.4 % | 3.98 ¹ | — | | 3c. After Costs_final_v1_2 backfill (single-pass) | 15,440,002 | 3.88 ¹ | 24.2 % | 6.89 ¹ | — | | 3d. After full ingest — **hot** working set | 28,175,589 | **13.99** | **87.4 %** | **15.03** | 0 | diff --git a/src/templates/finops-hub-local/notes/staging-contract.md b/src/templates/finops-hub-local/notes/staging-contract.md index 36f8c7760..5f3a57f29 100644 --- a/src/templates/finops-hub-local/notes/staging-contract.md +++ b/src/templates/finops-hub-local/notes/staging-contract.md @@ -135,12 +135,12 @@ export/ - **`export/`** — root staging directory. Default: `export/` relative to the ftklocal checkout root. Override via the `EXPORT_DIR` environment variable or `.env` file. - (`ingest.ps1`, `Get-ExportDir`, lines 41–51.) + (`ingest.ps1`, `Get-ExportDir`.) - **``** — a short, user-defined label for the billing scope (e.g., `ea`, `sub-prod`, `mca-`). This becomes the `scope` column in `Ingest_Manifest`. - **``** — dataset type token. Must be a key in `$script:DatasetTableMap` - (`ingest.ps1`, lines 59–62). See [Dataset type tokens](#dataset-type-tokens) below. + (`ingest.ps1`). See [Dataset type tokens](#dataset-type-tokens) below. - **``** — export billing period in `YYYYMMDD-YYYYMMDD` format (e.g., `20260501-20260531`). Must match this format; it becomes the `period` column. - **``** — a UUID that identifies one export run. Multiple run directories @@ -149,7 +149,7 @@ export/ ### Dataset type tokens -Defined in `$script:DatasetTableMap` at `ingest.ps1` lines 59–62: +Defined in `$script:DatasetTableMap` in `ingest.ps1`: | Token | Raw table | Ingestion mapping | Cost Management dataset | | ---------------- | ------------ | -------------------- | ----------------------- | @@ -157,12 +157,12 @@ Defined in `$script:DatasetTableMap` at `ingest.ps1` lines 59–62: | `ms--pricesheet` | `Prices_raw` | `Prices_raw_mapping` | `PriceSheet` | Any directory name under `/` that is not one of these tokens is silently skipped -with a warning (`ingest.ps1`, lines 248–252). +with a warning (`ingest.ps1`, `Build-Plan`). ### `manifest.json` per run Each `/` directory must contain a `manifest.json` file. The script reads -this file in `Read-RunManifest` (`ingest.ps1`, lines 209–215) to obtain sort and +this file in `Read-RunManifest` (`ingest.ps1`) to obtain sort and row-count metadata. Fields consumed by `ingest.ps1`: | JSON path | Required | Purpose | @@ -200,26 +200,26 @@ One or more `.parquet` files in the run directory. File names must match the `bl suffix pattern in `manifest.json` for per-file row counts to resolve correctly, though mismatches are tolerated (the per-file count defaults to 0; the run is still ingested). Files are sorted lexicographically by name before ingest (`ingest.ps1`, -`Sort-ByNameOrdinal`, lines 102–107 and 273). +`Sort-ByNameOrdinal`). ### Latest-run selection For each `(scope, type, period)` triple, `ingest.ps1` collects all run directories that contain a valid `manifest.json`, then picks the **one latest run** using a three-key sort (ascending): `submittedTime` → `LastWriteTimeUtc` → directory name ordinal. The last entry -after sorting is the winner. (`ingest.ps1`, `Build-Plan`, lines 269–272.) +after sorting is the winner. (`ingest.ps1`, `Build-Plan`.) All other runs in the same `(scope, type, period)` are **superseded**. Their previously ingested extents are dropped from the raw table and their rows are removed from `Ingest_Manifest` before the winning run is ingested. (`ingest.ps1`, -`Drop-SupersededExtents`, lines 571–613.) +`Drop-SupersededExtents`.) ### Idempotency via `Ingest_Manifest` Ingest is idempotent at the file level. Before ingesting a parquet file, `ingest.ps1` looks up the composite key `(scope, export_type, period, run_uuid, file_name)` in the `Ingest_Manifest` table. If the key exists and the stored SHA-256 checksum matches the -file, the file is skipped. (`ingest.ps1`, lines 741–748.) +file, the file is skipped. (`ingest.ps1`, `Invoke-RunIngest`.) `Ingest_Manifest` schema: @@ -235,24 +235,24 @@ file, the file is skipped. (`ingest.ps1`, lines 741–748.) | `checksum_sha256` | string | SHA-256 hex of the parquet file. | | `ingested_at` | datetime | UTC timestamp of ingest. | -Defined at `ingest.ps1`, `Initialize-IngestManifestTable` / `Insert-ManifestRow` -(lines 299–351). +Defined at `ingest.ps1`, `Initialize-IngestManifestTable` / `Insert-ManifestRow`. ### Overwrite semantics summary -| Condition | Behavior | -| ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| Same `(scope, type, period, run-uuid)`, same file checksum | **Skip.** File is already in `Ingest_Manifest` with a matching SHA-256; no ingest, no new row. (`ingest.ps1` lines 744–748.) | -| Same `(scope, type, period, run-uuid)`, **different** file checksum | **Double-ingest — do not do this.** The old `Ingest_Manifest` row is not deleted and the prior extents are not dropped; a second ingest call is issued and a second `Ingest_Manifest` row is appended for the same key. The raw table will contain duplicate data. (`ingest.ps1` lines 744–776.) | -| Same `(scope, type, period)`, **new** `run-uuid` | **Safe replace.** `Drop-SupersededExtents` drops extents tagged with older run-uuids from the raw table and deletes their `Ingest_Manifest` rows before the new run is ingested. (`ingest.ps1` lines 571–613.) | -| Different `(scope, type, period)` | **Independent.** No interaction with other tuples. | +| Condition | Behavior | +| ------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Same `(scope, type, period, run-uuid)`, same file checksum | **Skip.** File is already in `Ingest_Manifest` with a matching SHA-256; no ingest, no new row. (`ingest.ps1`, `Invoke-RunIngest`.) | +| Same `(scope, type, period, run-uuid)`, **different** file checksum | **Hard error (fail-fast).** `Invoke-RunIngest` detects the prior `Ingest_Manifest` row for the same composite key with a different SHA-256 and **throws** rather than ingest — re-ingesting under the same run-uuid would duplicate rows and silently corrupt cost totals. Stage corrected data under a **new** run-uuid for the same `(scope, type, period)` to replace it (supersede). (`ingest.ps1`, `Invoke-RunIngest`.) | +| Same `(scope, type, period)`, **new** `run-uuid` | **Safe replace.** `Drop-SupersededExtents` drops extents tagged with older run-uuids from the raw table and deletes their `Ingest_Manifest` rows before the new run is ingested. (`ingest.ps1`, `Drop-SupersededExtents`.) | +| Different `(scope, type, period)` | **Independent.** No interaction with other tuples. | > **How to correct or replace data:** stage the replacement parquet under a **new ``** > directory for the same `(scope, type, period)`. Give it a `manifest.json` with a later > `runInfo.submittedTime` than the run being replaced. `ingest.ps1` will then select it as > the latest run, drop the old run's extents and manifest rows, and ingest the new files -> cleanly. Never mutate files in an already-ingested run directory — that path has no safe -> overwrite; it only appends a duplicate. +> cleanly. Never mutate files in an already-ingested run directory — `ingest.ps1` detects the +> checksum change and **fails fast with a hard error** rather than silently duplicating data; +> the safe overwrite path is always a new run-uuid. --- @@ -347,7 +347,8 @@ export/ └── focus_v2.parquet ``` -In this tree, `ingest.ps1` ingests four runs: +In this tree, `ingest.ps1` ingests three runs (the latest of each +`(scope, type, period)` tuple): - `ea / ms--focus-cost / 20260501-20260531` → one run, ingested into `Costs_raw`. - `ea / ms--pricesheet / 20260501-20260531` → one run, ingested into `Prices_raw`. diff --git a/src/templates/finops-hub-local/scripts/ingest.ps1 b/src/templates/finops-hub-local/scripts/ingest.ps1 index b17bb99cc..35f53df7f 100644 --- a/src/templates/finops-hub-local/scripts/ingest.ps1 +++ b/src/templates/finops-hub-local/scripts/ingest.ps1 @@ -571,7 +571,7 @@ function Invoke-BackfillFinalsPerPeriod { $appended[$policyInfo.FinalTable] = 0 continue } - $extentCount = (Get-TableExtents -TableName $rawTableName).Count + $extentCount = @(Get-TableExtents -TableName $rawTableName).Count # Proactive chunking: skip single-pass entirely when row count exceeds the # measured-headroom threshold. OOM is row-driven; extent count is informational only. if ([int64]$rawCount -gt $script:BackfillChunkRowThreshold) { @@ -919,7 +919,7 @@ function Invoke-IngestMain { catch { [Console]::Error.WriteLine("error: $($_.Exception.Message)"); return 2 } if ($opts.ContainsKey('Help') -and $opts.Help) { Show-IngestUsage; return 0 } try { - $plans = Build-Plan -ExportRootPath $script:ExportDirHost -ScopeFilter $opts.Scope -PeriodFilter $opts.Period + $plans = @(Build-Plan -ExportRootPath $script:ExportDirHost -ScopeFilter $opts.Scope -PeriodFilter $opts.Period) if ($plans.Count -eq 0) { Write-InfoLine 'no (scope,type,period) tuples matched filters; nothing to do'; return 0 } $stats = Invoke-RunIngest -Plans $plans -DryRun ([bool]$opts.DryRun) -ForcePolicyRecapture ([bool]$opts.ForcePolicyRecapture) if ($opts.DryRun) { Write-InfoLine "`n(dry-run; no actual ingest occurred)"; return 0 } From 4ef64579b3645ec863daae436d74682d83fdbdf0 Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 18 Jun 2026 13:01:01 -0700 Subject: [PATCH 24/51] docs(hubs): add Run hubs locally guide; remove finops-hub-local template Add a docs-mslearn guide for running a FinOps hub in a local Kusto emulator container using PowerShell, sourcing schema, transforms, and open data from the toolkit release. Remove the finops-hub-local template and agent skill in favor of the doc-only approach. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs-mslearn/TOC.yml | 2 + docs-mslearn/toolkit/changelog.md | 4 +- docs-mslearn/toolkit/hubs/run-hubs-locally.md | 222 + .../agent-skills/finops-hub-local/README.md | 38 - .../agent-skills/finops-hub-local/SKILL.md | 146 - .../references/local-vs-remote.md | 94 - src/templates/finops-hub-local/.build.config | 3 - src/templates/finops-hub-local/.env.example | 38 - src/templates/finops-hub-local/.gitattributes | 18 - src/templates/finops-hub-local/.gitignore | 20 - src/templates/finops-hub-local/Makefile | 160 - src/templates/finops-hub-local/README.md | 381 -- src/templates/finops-hub-local/dashboard.json | 5559 ----------------- .../finops-hub-local/docker-compose.yml | 32 - src/templates/finops-hub-local/ftkver.txt | 1 - .../finops-hub-local/notes/build-by-hand.md | 197 - .../notes/config-container.md | 105 - .../finops-hub-local/notes/dashboard.md | 187 - .../notes/ftk-kql-adaptations.md | 225 - .../finops-hub-local/notes/install-mac.md | 104 - .../finops-hub-local/notes/install-windows.md | 151 - .../finops-hub-local/notes/parity-gaps.md | 178 - .../finops-hub-local/notes/performance.md | 217 - .../notes/staging-contract.md | 364 -- .../scripts/chunked-prices-backfill.ps1 | 49 - .../finops-hub-local/scripts/ftk.ps1 | 495 -- .../finops-hub-local/scripts/ingest.ps1 | 935 --- .../finops-hub-local/scripts/load-ftk-kql.ps1 | 508 -- .../scripts/run-parity-checks.ps1 | 515 -- .../finops-hub-local/scripts/settings.ps1 | 100 - .../finops-hub-local/settings.example.json | 15 - .../finops-hub-local/tests/parity-checks.kql | 164 - 32 files changed, 227 insertions(+), 11000 deletions(-) create mode 100644 docs-mslearn/toolkit/hubs/run-hubs-locally.md delete mode 100644 src/templates/agent-skills/finops-hub-local/README.md delete mode 100644 src/templates/agent-skills/finops-hub-local/SKILL.md delete mode 100644 src/templates/agent-skills/finops-hub-local/references/local-vs-remote.md delete mode 100644 src/templates/finops-hub-local/.build.config delete mode 100644 src/templates/finops-hub-local/.env.example delete mode 100644 src/templates/finops-hub-local/.gitattributes delete mode 100644 src/templates/finops-hub-local/.gitignore delete mode 100644 src/templates/finops-hub-local/Makefile delete mode 100644 src/templates/finops-hub-local/README.md delete mode 100644 src/templates/finops-hub-local/dashboard.json delete mode 100644 src/templates/finops-hub-local/docker-compose.yml delete mode 100644 src/templates/finops-hub-local/ftkver.txt delete mode 100644 src/templates/finops-hub-local/notes/build-by-hand.md delete mode 100644 src/templates/finops-hub-local/notes/config-container.md delete mode 100644 src/templates/finops-hub-local/notes/dashboard.md delete mode 100644 src/templates/finops-hub-local/notes/ftk-kql-adaptations.md delete mode 100644 src/templates/finops-hub-local/notes/install-mac.md delete mode 100644 src/templates/finops-hub-local/notes/install-windows.md delete mode 100644 src/templates/finops-hub-local/notes/parity-gaps.md delete mode 100644 src/templates/finops-hub-local/notes/performance.md delete mode 100644 src/templates/finops-hub-local/notes/staging-contract.md delete mode 100644 src/templates/finops-hub-local/scripts/chunked-prices-backfill.ps1 delete mode 100644 src/templates/finops-hub-local/scripts/ftk.ps1 delete mode 100644 src/templates/finops-hub-local/scripts/ingest.ps1 delete mode 100644 src/templates/finops-hub-local/scripts/load-ftk-kql.ps1 delete mode 100644 src/templates/finops-hub-local/scripts/run-parity-checks.ps1 delete mode 100644 src/templates/finops-hub-local/scripts/settings.ps1 delete mode 100644 src/templates/finops-hub-local/settings.example.json delete mode 100644 src/templates/finops-hub-local/tests/parity-checks.kql diff --git a/docs-mslearn/TOC.yml b/docs-mslearn/TOC.yml index 4d48e0aff..c11a155fb 100644 --- a/docs-mslearn/TOC.yml +++ b/docs-mslearn/TOC.yml @@ -128,6 +128,8 @@ href: toolkit/hubs/finops-hubs-overview.md - name: Create or upgrade hubs href: toolkit/hubs/deploy.md + - name: Run hubs locally + href: toolkit/hubs/run-hubs-locally.md - name: Configure private networking href: toolkit/hubs/private-networking.md - name: Configure scopes diff --git a/docs-mslearn/toolkit/changelog.md b/docs-mslearn/toolkit/changelog.md index 1de8c58e3..a916e9894 100644 --- a/docs-mslearn/toolkit/changelog.md +++ b/docs-mslearn/toolkit/changelog.md @@ -3,7 +3,7 @@ title: FinOps toolkit changelog description: Review the latest features and enhancements in the FinOps toolkit, including updates to FinOps hubs, Power BI reports, and more. author: MSBrett ms.author: brettwil -ms.date: 05/26/2026 +ms.date: 06/18/2026 ms.topic: reference ms.service: finops ms.subservice: finops-toolkit @@ -47,6 +47,8 @@ _Released June 2026_ ### [FinOps hubs](hubs/finops-hubs-overview.md) v15 +- **Added** + - Added a [Run hubs locally](hubs/run-hubs-locally.md) guide to stand up a FinOps hub in a local Kusto emulator container and ingest cost data using the same KQL, transforms, and open data as a deployed hub. - **Changed** - Added a callout to the `config_RunBackfillJob` backfill option clarifying that it isn't supported on Microsoft Customer Agreement (MCA) billing accounts or billing profiles ([#2113](https://github.com/microsoft/finops-toolkit/issues/2113)). - **Fixed** diff --git a/docs-mslearn/toolkit/hubs/run-hubs-locally.md b/docs-mslearn/toolkit/hubs/run-hubs-locally.md new file mode 100644 index 000000000..4461bea9f --- /dev/null +++ b/docs-mslearn/toolkit/hubs/run-hubs-locally.md @@ -0,0 +1,222 @@ +--- +title: Run FinOps hubs locally +description: Stand up a FinOps hub on your machine in a local container and ingest cost data, using the same KQL and open data as a deployed hub. +author: flanakin +ms.author: micflan +ms.date: 06/18/2026 +ms.topic: how-to +ms.service: finops +ms.subservice: finops-toolkit +ms.reviewer: micflan +#customer intent: As a FinOps user, I want to run a FinOps hub locally so I can explore my cost data without deploying Azure resources. +--- + + + +# Run FinOps hubs locally + +This article shows how to run a FinOps hub on your own machine using the [Kusto emulator](/azure/data-explorer/kusto-emulator-overview), a free local container. It uses the same analytics KQL, transforms, and open data as a deployed [FinOps hub](finops-hubs-overview.md) — only the host changes from Azure Data Explorer to a local container. Run the PowerShell blocks below in order; each is a step you (or an agent) can run one after another. + +A local hub is useful to explore the FinOps hub data model, validate cost data, or work offline. It isn't a replacement for a deployed hub — there's no scheduled ingestion, networking, or sharing. + +
+ +## Prerequisites + +- [Docker](https://docs.docker.com/get-docker/), to run the Kusto emulator container, with at least 16 GB of memory available to the container. +- [PowerShell 7](/powershell/scripting/install/installing-powershell) — every command in this article is PowerShell. +- [Azure PowerShell](/powershell/azure/install-azure-powershell) (the `Az` modules) — only needed to download cost data from a storage account. +- FOCUS cost exports as Parquet, either from a [FinOps hub storage account](configure-scopes.md) or [Cost Management exports](/cost-management-billing/costs/tutorial-improved-exports). + +For background on the emulator and its platform requirements, see the [Kusto emulator overview](/azure/data-explorer/kusto-emulator-overview) and [installation guide](/azure/data-explorer/kusto-emulator-install). + +> [!NOTE] +> This article assumes you're already signed in to Azure with `Connect-AzAccount` and have selected the subscription that holds your cost data. It never runs a sign-in command. If you aren't downloading data from a storage account, you can skip Azure sign-in entirely. + +
+ +## Start the emulator + +Create a working folder, then start the emulator. The `export` folder is mounted read-only into the container at `/data/export`; you'll download cost data into it later. For other ways to run the container, see [Install the Kusto emulator](/azure/data-explorer/kusto-emulator-install). + +```powershell +New-Item -ItemType Directory -Force -Path export | Out-Null +docker run -d --name finops-hub-local --platform linux/amd64 ` + -p 8082:8080 -m 16g -e ACCEPT_EULA=Y ` + -v "$($PWD.Path)/export:/data/export:ro" ` + mcr.microsoft.com/azuredataexplorer/kustainer-linux:latest +``` + +All later steps talk to the emulator's HTTP endpoint, so define a small helper to send a command. Management commands (those starting with `.`) go to `/v1/rest/mgmt`; queries go to `/v1/rest/query`. The endpoint has no authentication. + +```powershell +$hub = 'http://localhost:8082/v1/rest' +function Invoke-Kusto { + param([string]$Database, [string]$Command, [ValidateSet('mgmt','query')][string]$Endpoint = 'mgmt') + Invoke-RestMethod -Uri "$hub/$Endpoint" -Method Post -ContentType 'application/json' ` + -Body (@{ db = $Database; csl = $Command } | ConvertTo-Json) +} +``` + +Wait for the engine to answer before continuing: + +```powershell +do { + Start-Sleep -Seconds 3 + $ready = try { Invoke-Kusto NetDefaultDB '.show version'; $true } catch { Write-Host 'waiting for emulator...'; $false } +} until ($ready) +Write-Host 'emulator ready' +``` + +
+ +## Create the two databases + +A FinOps hub uses two databases: `Ingestion` for raw and transformed data, and `Hub` for the view functions you query. + +```powershell +foreach ($db in 'Ingestion', 'Hub') { + Invoke-Kusto NetDefaultDB ".create database $db persist (@`"/kustodata/dbs/$db/md`", @`"/kustodata/dbs/$db/data`")" | Out-Null +} +``` + +
+ +## Load the schema + +Download the hub's setup scripts from the latest toolkit release and load each into its database. These are the same `finops-hub-fabric-setup-*.kql` files used to set up a [Microsoft Fabric hub](deploy.md#set-up-microsoft-fabric) — each is a single `.execute database script` that creates every table, mapping, transform function, and update policy. + +The Ingestion script has one placeholder, `$$rawRetentionInDays$$` (how long to keep raw data). Replace it with a number of days before loading. + +```powershell +$release = 'https://github.com/microsoft/finops-toolkit/releases/latest/download' +Invoke-WebRequest "$release/finops-hub-fabric-setup-Ingestion.kql" -OutFile ./setup-Ingestion.kql +Invoke-WebRequest "$release/finops-hub-fabric-setup-Hub.kql" -OutFile ./setup-Hub.kql + +# Ingestion: raw tables, transforms, final tables, update policies +$ingestion = (Get-Content ./setup-Ingestion.kql -Raw) -replace '\$\$rawRetentionInDays\$\$', '90' +Invoke-Kusto Ingestion $ingestion + +# Hub: the Costs/Prices/Transactions view functions +Invoke-Kusto Hub (Get-Content ./setup-Hub.kql -Raw) +``` + +Each response is a table with one row per statement. The Hub functions reference `database('Ingestion').*`, which resolves because both databases exist. + +
+ +## Load the open data + +FinOps hubs enrich cost data with open-data reference tables (regions, services, resource types, and pricing units). Download the reference CSVs from the toolkit release, then load them with the same commands a deployed hub uses. + +```powershell +New-Item -ItemType Directory -Force -Path export/open-data | Out-Null +foreach ($f in 'PricingUnits', 'Regions', 'ResourceTypes', 'Services') { + Invoke-WebRequest "$release/$f.csv" -OutFile "export/open-data/$f.csv" +} + +$openData = @' +.execute database script with (ContinueOnErrors=true) <| +.set-or-replace PricingUnits <| externaldata(x_PricingUnitDescription: string, AccountTypes: string, x_PricingBlockSize: real, PricingUnit: string)[@'/data/export/open-data/PricingUnits.csv'] with (format='csv', ignoreFirstRecord=true) | project-away AccountTypes +.set-or-replace Regions <| externaldata(ResourceLocation: string, RegionId: string, RegionName: string)[@'/data/export/open-data/Regions.csv'] with (format='csv', ignoreFirstRecord=true) +.set-or-replace ResourceTypes <| externaldata(x_ResourceType: string, SingularDisplayName: string, PluralDisplayName: string, LowerSingularDisplayName: string, LowerPluralDisplayName: string, IsPreview: bool, Description: string, IconUri: string, Links: string)[@'/data/export/open-data/ResourceTypes.csv'] with (format='csv', ignoreFirstRecord=true) | project-away Links +.set-or-replace Services <| externaldata(x_ConsumedService: string, x_ResourceType: string, ServiceName: string, ServiceCategory: string, ServiceSubcategory: string, PublisherName: string, x_PublisherCategory: string, x_Environment: string, x_ServiceModel: string)[@'/data/export/open-data/Services.csv'] with (format='csv', ignoreFirstRecord=true) +'@ +Invoke-Kusto Ingestion $openData +``` + +
+ +## Download your cost data + +Download FOCUS cost and price exports from your storage account into the `export` folder, using your existing Azure sign-in. This works against either the **msexports** or **ingestion** container of a FinOps hub, or any container holding Cost Management exports. Only the data files (`manifest.json` and `*.parquet`) are downloaded. + +Set these to match your storage account, then run the download: + +```powershell +$account = '' +$container = 'ingestion' # or: msexports +$prefix = '' # optional, e.g. 'billingAccounts/00000000' for msexports + +$ctx = New-AzStorageContext -StorageAccountName $account -UseConnectedAccount +Get-AzStorageBlob -Container $container -Context $ctx -Prefix $prefix | + Where-Object { $_.Name -match '(manifest\.json|\.parquet)$' } | + ForEach-Object { + Get-AzStorageBlobContent -Container $container -Context $ctx -Blob $_.Name ` + -Destination ./export -Force | Out-Null + } +``` + +> [!NOTE] +> FinOps hub storage accounts use a hierarchical namespace, where each folder is also a zero-byte marker blob. Downloading only `manifest.json` and `*.parquet` skips those markers, which otherwise collide with same-named folders on disk. `-UseConnectedAccount` reuses your existing Azure sign-in. + +
+ +## Ingest the data + +Ingest each Parquet file into the matching raw table. As rows land, the update policy created by the schema runs the FOCUS transform automatically and appends the result to the final table — the same mechanism a deployed hub uses. Cost exports go to `Costs_raw`; price sheets go to `Prices_raw`. + +```powershell +function Invoke-Ingest { + param([string]$Table, [string]$Mapping, [string]$File) + $rel = [IO.Path]::GetRelativePath((Resolve-Path ./export), $File) -replace '\\', '/' + $path = "/data/export/$rel" + Invoke-Kusto Ingestion ".ingest into table $Table (h@'$path') with (format='parquet', ingestionMappingReference='$Mapping')" | Out-Null + Write-Host " ingested $(Split-Path $File -Leaf)" +} + +# Group exports by manifest type, then ingest each dataset's Parquet files +Get-ChildItem ./export -Recurse -Filter manifest.json | ForEach-Object { + $type = if ((Get-Content $_.FullName -Raw) -match '"type"\s*:\s*"(FocusCost|PriceSheet)"') { $Matches[1] } else { return } + $target = if ($type -eq 'FocusCost') { 'Costs_raw', 'Costs_raw_mapping' } else { 'Prices_raw', 'Prices_raw_mapping' } + Get-ChildItem $_.Directory -Filter *.parquet | ForEach-Object { + Invoke-Ingest $target[0] $target[1] $_.FullName + } +} +``` + +Confirm the data landed and the final tables filled in: + +```powershell +foreach ($t in 'Costs_raw', 'Costs_final_v1_2', 'Prices_raw', 'Prices_final_v1_2') { + $count = (Invoke-Kusto Ingestion "$t | count" -Endpoint query).Tables[0].Rows[0][0] + Write-Host "$t`: $count" +} +``` + +> [!TIP] +> Ingesting raw data is all you need — the final tables populate themselves through the update policy. Keep the policy enabled so each file's transform stays small; it scales to tens of millions of rows without running out of memory. + +
+ +## Query your data + +The `Hub` database exposes the same view functions a deployed hub does. Query them directly: + +```powershell +$result = Invoke-Kusto Hub 'Costs_v1_2 | summarize EffectiveCost = round(sum(EffectiveCost), 2) by ServiceCategory | top 10 by EffectiveCost' -Endpoint query +$result.Tables[0].Rows +``` + +You now have a working local FinOps hub. To explore it visually, connect the [Azure Data Explorer web UI](https://dataexplorer.azure.com) to `http://localhost:8082`, or point Power BI and other tools at the same endpoint. + +
+ +## Clean up + +Stop and remove the container when you're done. Your downloaded data stays in the `export` folder. + +```powershell +docker rm -f finops-hub-local +``` + +
+ +## Related content + +- [FinOps hubs overview](finops-hubs-overview.md) +- [Create and update FinOps hubs](deploy.md) +- [FinOps hub data model](data-model.md) +- [How data is processed](data-processing.md) + +
diff --git a/src/templates/agent-skills/finops-hub-local/README.md b/src/templates/agent-skills/finops-hub-local/README.md deleted file mode 100644 index c9cbfa4a0..000000000 --- a/src/templates/agent-skills/finops-hub-local/README.md +++ /dev/null @@ -1,38 +0,0 @@ -# FinOps hub (local) skill - -A skill for running FinOps hub cost analytics **locally** — against a Kusto emulator -in Docker — without deploying anything to Azure. - -## What it is - -The FinOps Toolkit's KQL analytics (ingestion transforms, hub view functions, open -data, and the query catalog) loaded into a local [Kusto emulator] so an analyst can -query FOCUS cost exports on their own machine. It runs the **same** analytic IP a -deployed hub runs; only the cloud-orchestration glue (Data Factory, storage, -deployment) is replaced by local scripts in `src/templates/finops-hub-local/`. - -## When to use this skill vs the cloud `finops-toolkit` skill - -| Use **finops-hub-local** when… | Use **finops-toolkit** when… | -|--------------------------------|------------------------------| -| You have FOCUS Parquet exports and want to query them offline | You have (or want to deploy) a real hub on Azure Data Explorer / Fabric | -| You're developing/testing hub KQL without standing up Azure | You need scale, multi-user access, AAD auth, or Power BI | -| No Azure subscription / Azure MCP available | You're connecting to a live `*.kusto.windows.net` cluster | - -## Prerequisites - -- Docker -- PowerShell 7+ (`pwsh`) -- FOCUS cost exports as Parquet - -## Getting started - -See `SKILL.md` for the task-routing table and the quickstart. In short, from -`src/templates/finops-hub-local/`: - -```bash -make up && make load-ftk-kql && make ingest -pwsh scripts/ftk.ps1 query "Costs() | summarize sum(EffectiveCost) by ServiceName | top 10 by sum_EffectiveCost" -``` - -[Kusto emulator]: https://learn.microsoft.com/azure/data-explorer/kusto-emulator-overview diff --git a/src/templates/agent-skills/finops-hub-local/SKILL.md b/src/templates/agent-skills/finops-hub-local/SKILL.md deleted file mode 100644 index ad18e9556..000000000 --- a/src/templates/agent-skills/finops-hub-local/SKILL.md +++ /dev/null @@ -1,146 +0,0 @@ ---- -name: finops-hub-local -description: This skill should be used when the user wants to run FinOps hub cost analytics LOCALLY without an Azure deployment — e.g. "local FinOps hub", "FinOps hub local", "Kustainer", "Kusto emulator", "ftklocal", "run FinOps queries locally", "analyze FOCUS cost exports offline", "query cost parquet locally", "local cost analysis", or "FinOps toolkit without Azure". It drives a local Kusto emulator that runs the FinOps Toolkit's KQL analytics (the same Costs/Prices/Transactions functions a deployed hub exposes) against FOCUS parquet exports on disk. DO NOT use it to deploy Azure resources, provision infrastructure, or query a live Azure Data Explorer / Fabric cluster — for those, use the `finops-toolkit` skill. -license: MIT -compatibility: Requires Docker and PowerShell 7+. Runs a local Kusto emulator container — no Azure subscription, Azure MCP Server, or cloud cluster required. FOCUS exports must already be Parquet. -metadata: - author: microsoft - version: "1.0" ---- - -# FinOps hub (local) - -Run the FinOps Toolkit's full KQL cost-analytics stack **locally**, against a Kusto -emulator (Docker) instead of a deployed Azure Data Explorer or Microsoft Fabric -cluster. The analytic IP is identical — the same `IngestionSetup` / `HubSetup` KQL, -the same hub view functions (`Costs`, `Prices`, `Transactions`, and their `*_v1_2` -variants), the same open data, and the same query catalog — so cost analysis works -offline on a developer machine. Only the cloud-orchestration glue (Data Factory, -storage, deployment) is replaced by local scripts. - -The tooling lives in `src/templates/finops-hub-local/`. Run the commands below from -that directory. - -## When to invoke - -- The user has FOCUS cost exports (Parquet) and wants to query them locally. -- The user wants to develop or test FinOps hub KQL without standing up Azure. -- The user asks for "the local FinOps hub", "ftklocal", "Kustainer", or offline cost analytics. - -Do **not** invoke for deploying or querying a real Azure hub — route those to the -`finops-toolkit` skill. - -## Topology (so queries are written correctly) - -The local stack mirrors a real hub's **two-database** layout: - -| Database | Holds | You query it for | -|----------|-------|------------------| -| `Ingestion` | Raw tables (`Costs_raw`, `Prices_raw`), the `*_transform_v1_2()` functions, and the final tables (`Costs_final_v1_2`, `Prices_final_v1_2`) | Ingestion/transform internals, parity checks | -| `Hub` | The hub **view functions** (`Costs`, `Prices`, `Transactions`, `Costs_v1_2`, `Prices_v1_2`, …) which read `database('Ingestion').*` | All analytics — this is the default query database | - -The cross-database references (`database('Ingestion').*`) inside the Hub functions -resolve in the emulator exactly as they do in ADX. The CLI defaults to the `Hub` -database. - -## Prerequisites - -- Docker (the emulator runs as a container via `docker compose`). -- PowerShell 7+ (`pwsh`) — all tooling is PowerShell; there are no Python dependencies. -- FOCUS cost exports as **Parquet**, staged under `export/` (see the staging contract - in `src/templates/finops-hub-local/notes/staging-contract.md`). - -## Quickstart - -```bash -# from src/templates/finops-hub-local/ -cp .env.example .env # 1. configure (HOST_PORT, MEM_LIMIT, EXPORT_DIR) -make up # 2. start the emulator; blocks until healthcheck passes -make load-ftk-kql # 3. create Ingestion + Hub, load FTK KQL + open data (idempotent) -make ingest # 4. bulk-ingest the Parquet exports under export/ -pwsh scripts/ftk.ps1 query "Costs() | summarize TotalEffectiveCost = sum(EffectiveCost) by ServiceName | top 10 by TotalEffectiveCost" -``` - -## Task routing - -| Task | Run this | -|------|----------| -| Start / stop the emulator | `make up` / `make down` (data preserved); `make nuke` (destructive reset) | -| Load the analytics schema + open data | `make load-ftk-kql` (idempotent; safe to re-run) | -| Ingest Parquet exports | `make ingest` (or `pwsh scripts/ingest.ps1`); scope/period subsets: `make ingest SCOPE= PERIOD=` | -| Check what's been ingested | `make ingest-status` | -| Run an ad-hoc KQL query | `pwsh scripts/ftk.ps1 query ""` (default database `Hub`) | -| List the reusable query catalog | `pwsh scripts/ftk.ps1 list` | -| Run a named catalog query | `pwsh scripts/ftk.ps1 run ` (add `--format csv`, `--start`/`--end`) | -| Verify transform parity | `make parity` (exit 0 iff all checks pass) | -| Recover a Prices backfill OOM | `make chunked-prices-backfill` (per-extent fallback) | -| See all targets | `make help` | - -The `ftk.ps1 run` command reuses the published FinOps Toolkit query catalog **in -place** (it never forks the `.kql` files). It applies a few small, inspectable -on-read adaptations so version-loose catalog queries run against the emulator — see -`references/local-vs-remote.md`. There is **no** new KQL in this skill; all analytic -logic is upstream FinOps Toolkit. - -## Common queries - -All examples use `Costs_v1_2()` — the Hub view function. Substitute `Costs_final_v1_2` -(in the `Ingestion` database) for the un-enriched final table. Results vary with the -data you ingested; if `Costs_v1_2() | count` returns 0, run `make ingest` first. - -```kql -// Top 10 services by effective cost -Costs_v1_2() -| summarize TotalEffectiveCost = sum(EffectiveCost) by ServiceName -| top 10 by TotalEffectiveCost -``` - -```kql -// Monthly cost trend -Costs_v1_2() -| summarize MonthlyCost = sum(EffectiveCost) by Month = monthstring(ChargePeriodStart) -| order by Month asc -``` - -```kql -// Top 20 most expensive resources -Costs_v1_2() -| summarize TotalCost = sum(EffectiveCost), BilledCost = sum(BilledCost) - by ResourceId, ResourceName, ServiceName, x_ResourceGroupName -| top 20 by TotalCost -``` - -```kql -// Subscription-level cost summary -Costs_v1_2() -| summarize EffectiveCost = sum(EffectiveCost), BilledCost = sum(BilledCost) - by SubAccountId, SubAccountName -| order by EffectiveCost desc -``` - -## Troubleshooting - -| Symptom | Cause / fix | -|---------|-------------| -| Emulator won't start; port in use | Another process holds `HOST_PORT` (default 8082). Change `HOST_PORT` in `.env`, then `make up`. | -| Container OOM during the Prices backfill | The single-pass `Prices_transform_v1_2()` exceeds memory on large datasets. Keep `MEM_LIMIT=16g` (the `.env.example` default); ingest now chunks large tables automatically, and `make chunked-prices-backfill` is the manual recovery. | -| `Costs_v1_2() \| count` returns 0 after load | The schema is loaded but no data is ingested yet. Run `make ingest`. | -| Database doesn't appear after `make up` | Emulator metadata can need a clean reset between image versions — `make nuke && make up && make load-ftk-kql`. | -| Ingest aborts: "checksum mismatch … stage under a NEW run-uuid" | A corrected file was re-staged under the same run-uuid. This guard prevents silently duplicating cost rows — stage the corrected data under a new run-uuid for the same scope/type/period (it supersedes the old run). | - -## Limits and non-goals - -- **Single user, local only.** No auth, no TLS, no multi-tenancy. -- **No Azure Data Factory.** Ingest orchestration (blob trigger → convert → ingest) is - replaced by `scripts/ingest.ps1`. Exports must already be FOCUS Parquet — there is no - CSV-to-Parquet convert step. -- **No deployment.** No Bicep/ARM; the stack is `docker compose`. -- **Not a production hub.** Use it for local analysis and KQL development. For a real - hub (scale, Power BI, multi-user, AAD), deploy with the `finops-toolkit` skill. - -## References - -- `references/local-vs-remote.md` — what is identical vs adapted between a deployed hub - and the local stack (connection, the on-read query adaptations, the IP-reuse map). -- `src/templates/finops-hub-local/notes/staging-contract.md` — the data-in contract for `export/`. -- `src/templates/finops-hub-local/notes/dashboard.md` — querying the local Hub from the ADX dashboard / CLI. diff --git a/src/templates/agent-skills/finops-hub-local/references/local-vs-remote.md b/src/templates/agent-skills/finops-hub-local/references/local-vs-remote.md deleted file mode 100644 index e325e81cb..000000000 --- a/src/templates/agent-skills/finops-hub-local/references/local-vs-remote.md +++ /dev/null @@ -1,94 +0,0 @@ -# Local vs remote: reusing the FinOps Toolkit analytics - -The local FinOps hub runs the **same FinOps Toolkit analytics** as a deployed hub, but -against a local Kusto emulator instead of an Azure Data Explorer (ADX) or Microsoft -Fabric cluster. The query IP is identical; only the **connection** and a couple of -small, on-read **query adaptations** differ. - -## What is the same: the analytic IP - -Both local and remote expose the FinOps hub analytic view functions, so the published -query catalog is portable across both: - -| Function | Purpose | -|----------|---------| -| `Costs()` / `Costs_v1_2()` | Cost & usage analytics (FOCUS-aligned) | -| `Prices()` / `Prices_v1_2()` | Price sheets (list / contracted / effective) | -| `Transactions()` | Commitment purchases, refunds, exchanges | - -These functions live in the `Hub` database and read the final tables in the -`Ingestion` database via `database('Ingestion').*` references — the same two-database -topology a deployed hub uses. The cross-database references resolve in the emulator -exactly as they do in ADX. - -## What is different: connecting - -| Aspect | Deployed hub | Local | -|--------|-------------|-------| -| Endpoint | `https://.kusto.windows.net` | `http://localhost:8082` (default `HOST_PORT`) | -| Auth | Azure AD (tenant) | none (local, single user) | -| Query database | `Hub` | `Hub` (same) | -| Client | MCP Kusto server / Power BI | `ftk.ps1` CLI | -| Settings | environment settings file | `.env` (`HOST_PORT`, `MEM_LIMIT`, `EXPORT_DIR`) | - -Run a catalog query locally: - -```bash -pwsh scripts/ftk.ps1 list # discover catalog queries -pwsh scripts/ftk.ps1 run savings-summary-report # adapt + run a named catalog query -pwsh scripts/ftk.ps1 query "Costs() | summarize sum(EffectiveCost) by ServiceName | top 10 by sum_EffectiveCost" -``` - -## On-read query adaptations (applied automatically by `ftk run`) - -The emulator is the same engine as ADX, so catalog queries run **almost** verbatim. -`ftk run` reads each upstream `.kql` **in place** (it never forks the file) and applies -a few small, inspectable transforms: - -1. **Date window** — catalog queries default to a trailing-30-day window, which returns - nothing against a static local export. `ftk` retargets the window to the latest - period present in the data. Override with `--start` / `--end`. -2. **`decimal` → `real`** — a few catalog queries use `decimal()` / `todecimal()` - literals. The hub's cost columns are typed `real`; ADX silently coerces the mix, but - the emulator (stricter) rejects it, so `ftk` normalizes `decimal(N)`→`real(N)` and - `todecimal(x)`→`toreal(x)`. -3. **Cross-database prefix** — the rare catalog query that explicitly names - `database('Ingestion').` is normalized for the local default-database context. -4. **`project-away` tolerance** — a few catalog queries `project-away` a column that - exists in a different hub schema version but not this one (for example `x_InvoiceId`, - which the v1.2 transform folds into the FOCUS-standard `InvoiceId`). `project-away` - errors on unknown columns, so `ftk` drops any such name from the list and the query - still runs. The local schema is correct; this only keeps version-loose catalog - queries portable. - -These adaptations are **local-only conveniences**. The upstream catalog files are -unchanged and run as-is against a real hub. - -## What the local stack does NOT have (vs a deployed hub) - -| Hub component | Local | -|---------------|-------| -| Azure Data Factory (blob trigger → convert → ingest) | Replaced by `scripts/ingest.ps1`. Exports must already be FOCUS Parquet — no CSV-to-Parquet convert step. | -| ADLS storage (msexports / ingestion containers) | Local `export/` folder + raw tables. | -| Bicep / ARM deployment | Not needed — `docker compose` (`make up`). | -| Power BI binding | Not wired locally; use `ftk.ps1 ... --format csv`, or see `notes/dashboard.md`. | -| Multi-user / TLS / auth | None. Single developer, local only. | - -There is **no** local Azure Data Factory emulator. Data Factory in a hub is only -orchestration glue and carries no analytic IP, so the local stack replaces it with the -ingest script rather than emulating it. - -## IP reuse, at a glance - -| FinOps Toolkit component | Local | -|--------------------------|-------| -| `IngestionSetup_v1_2.kql` (raw tables, update policies, `*_transform_v1_2()`) | Loaded verbatim into the `Ingestion` database | -| `HubSetup_v1_2.kql` (hub view functions + UDFs) | Loaded verbatim into the `Hub` database | -| Open-data reference CSVs | Loaded as-is | -| Query catalog (`src/queries/catalog/*.kql`, agent-skill subset) | Run as-is, with the four on-read adaptations above | -| Data Factory orchestration | Replaced by `scripts/ingest.ps1` | -| ADLS storage | Replaced by `export/` + raw tables | -| Bicep / ARM deploy | Not needed locally | - -**Net: essentially all of the analytic IP (transforms + hub functions + open data + -catalog queries) is reused; only cloud-orchestration glue is reimplemented locally.** diff --git a/src/templates/finops-hub-local/.build.config b/src/templates/finops-hub-local/.build.config deleted file mode 100644 index db3c9f780..000000000 --- a/src/templates/finops-hub-local/.build.config +++ /dev/null @@ -1,3 +0,0 @@ -{ - "ignore": ["export", "kustainer-data", ".env", "settings.json", "*.log"] -} diff --git a/src/templates/finops-hub-local/.env.example b/src/templates/finops-hub-local/.env.example deleted file mode 100644 index cb0aa126e..000000000 --- a/src/templates/finops-hub-local/.env.example +++ /dev/null @@ -1,38 +0,0 @@ -# ftklocal tunables — copy to `.env` and edit as needed. -# `.env` is gitignored; `.env.example` is the documented template. - -# Host port to expose the Kustainer HTTP endpoint on. -# IMPORTANT: do NOT use 8080 — that port is reserved for maenifold (the MCP daemon). -HOST_PORT=8082 - -# Memory ceiling for the Kustainer container. -# -# 16g is the recommended default. The full Prices_transform_v1_2() backfill -# over ~12.7M rows OOM'd at 8g on amd64-emulation (Apple Silicon under Rosetta). -# Costs-only workflows or smaller datasets work at 8g, but reliability across -# both Costs (~1.35M rows) and Prices (~12.7M rows) requires 16g. -MEM_LIMIT=16g - -# Directory on the host that holds the FOCUS parquet exports. -# Mounted read-only at /data/export inside the container. -# Default is ./export (relative to the Makefile / docker-compose.yml directory). -# Point to an absolute path to reuse an existing export without copying files: -# EXPORT_DIR=~/src/ftklocal/export -EXPORT_DIR=./export - -# Proactive chunked-backfill threshold (rows). -# ingest.ps1 measures the raw-table row count before each final-table backfill. -# If the count exceeds this value, single-pass .set-or-append is skipped entirely -# and per-extent chunked backfill is used proactively (avoids OOM on large datasets). -# If at or below, single-pass is tried first with chunked backfill as the reactive -# safety-net fallback. OOM risk is row-driven, not extent-driven. -# -# Default 2000000 is derived from measured headroom at MEM_LIMIT=16g, amd64-Rosetta: -# - Costs 1,350,000 rows → single-pass OK (safely below threshold) -# - Prices 12,735,587 rows → single-pass OOMs (safely above threshold) -# Raise this value if you increase MEM_LIMIT and re-verify single-pass succeeds. -# Note: if you add a 3rd table to RawToFinalPolicy, do NOT assume this threshold -# applies — do a fresh single-pass OOM measurement for that table's row count and -# MEM_LIMIT. The reactive fallback (single-pass fail → chunked) covers it in the -# meantime. -# BACKFILL_CHUNK_ROW_THRESHOLD=2000000 diff --git a/src/templates/finops-hub-local/.gitattributes b/src/templates/finops-hub-local/.gitattributes deleted file mode 100644 index 013b09e81..000000000 --- a/src/templates/finops-hub-local/.gitattributes +++ /dev/null @@ -1,18 +0,0 @@ -# ftklocal line-ending policy. -# -# Keep LF on files that break or behave inconsistently under CRLF, so a Windows -# checkout (where git may otherwise convert to CRLF) stays correct: -# - the Makefile + shell recipes require LF (bash chokes on CRLF), -# - PowerShell tolerates either but LF keeps diffs clean across platforms, -# - KQL/JSON are submitted verbatim to the engine. -*.ps1 text eol=lf -*.psm1 text eol=lf -*.psd1 text eol=lf -*.sh text eol=lf -*.kql text eol=lf -*.json text eol=lf -*.md text eol=lf -Makefile text eol=lf -*.yml text eol=lf -*.yaml text eol=lf -.env.example text eol=lf diff --git a/src/templates/finops-hub-local/.gitignore b/src/templates/finops-hub-local/.gitignore deleted file mode 100644 index 129589d3d..000000000 --- a/src/templates/finops-hub-local/.gitignore +++ /dev/null @@ -1,20 +0,0 @@ -# Ephemeral state and secrets — never commit. -.gate/ -kustainer-data/ -*.duckdb -.env -settings.json -.DS_Store -*.log -__pycache__/ -*.pyc - -# Customer data — never commit. The raw FOCUS exports are customer-confidential, -# and so is any analysis derived from them. -export/ -notes/figures/ -notes/*refund* -notes/*sp-* -notes/*savings* -notes/*misalloc* -notes/*unused* diff --git a/src/templates/finops-hub-local/Makefile b/src/templates/finops-hub-local/Makefile deleted file mode 100644 index e36f339e8..000000000 --- a/src/templates/finops-hub-local/Makefile +++ /dev/null @@ -1,160 +0,0 @@ -# ftklocal — Makefile -# -# Targets: -# make up Bring Kustainer up and block until healthcheck passes. -# make down Stop and remove the container (keeps the data volume). -# make nuke Down + remove kustainer-data/ (destructive; full reset). -# make logs Tail Kustainer logs. -# make kql QUERY='...' Run a one-shot KQL query (default DB: Hub; override e.g. KQL_DB=Ingestion). -# -# DB topology: -# Ingestion raw tables (Costs_raw, Prices_raw), transform functions, final tables, -# open-data lookups, and Ingest_Manifest. -# Used by: ingest, ingest-status, checks 1–8 + 11, parity (base DB). -# Hub view functions (Costs_v1_2, Prices_v1_2, Costs, Prices, etc.) that -# cross-reference database('Ingestion').*. -# Used by: ftk query/run/schema, parity checks 9–10, kql (default). -# -# Notes: -# - The Kustainer HTTP endpoint is on host port $(HOST_PORT) (default 8082). -# Port 8080 is intentionally avoided — that belongs to maenifold. -# - `kql` talks directly to Kustainer's REST API. Kustainer is NOT an MCP -# server, so a plain curl is the right tool here. - -SHELL := /bin/bash - -# Load .env if present so HOST_PORT / MEM_LIMIT / EXPORT_DIR are visible to -# both `docker compose` and the recipes below. -ifneq (,$(wildcard ./.env)) -include .env -export -endif - -HOST_PORT ?= 8082 -# KQL_DB: default database for ad-hoc `make kql` queries. -# Defaults to Hub so view functions (Costs_v1_2, etc.) work without an override. -# Override for raw-table inspection: make kql KQL_DB=Ingestion QUERY='Costs_raw | count' -KQL_DB ?= Hub -# PARITY_DB: base database passed to run-parity-checks.ps1 --db. -# Checks 1-8/11 run against PARITY_DB (Ingestion); the script internally forces -# checks 9/10 to Hub regardless of this value. -PARITY_DB ?= Ingestion -COMPOSE ?= docker compose - -.PHONY: up down nuke logs kql ftk ftk-list wait-healthy load-ftk-kql ingest ingest-status chunked-prices-backfill parity help - -help: - @echo "Targets:" - @echo " up Bring Kustainer up and block until healthy." - @echo " down Stop the container (keeps data volume)." - @echo " nuke Destructive full reset (drops kustainer-data/)." - @echo " logs Tail Kustainer logs." - @echo " kql QUERY='...' One-shot KQL against Hub (override: KQL_DB=Ingestion)." - @echo " load-ftk-kql Load FTK analytics KQL + open-data lookups." - @echo " ingest Bulk-ingest ALL parquet under export/." - @echo " ingest SCOPE= Ingest only one scope." - @echo " ingest SCOPE= PERIOD=20260501-20260531" - @echo " Filter scope + period." - @echo " ingest-status Show Ingest_Manifest summary." - @echo " parity Run T-1.4 parity check suite." - @echo " chunked-prices-backfill Manual fallback: per-extent Prices backfill (auto-triggered by threshold)." -up: - $(COMPOSE) up -d - @$(MAKE) --no-print-directory wait-healthy - -wait-healthy: - @echo "Waiting for kustainer to become healthy (up to 90s)..." - @for i in $$(seq 1 90); do \ - status=$$(docker inspect -f '{{.State.Health.Status}}' kustainer 2>/dev/null || echo "missing"); \ - if [ "$$status" = "healthy" ]; then \ - echo "kustainer is healthy."; \ - exit 0; \ - fi; \ - if [ "$$status" = "missing" ]; then \ - echo "kustainer container not found yet..."; \ - fi; \ - sleep 1; \ - done; \ - echo "ERROR: kustainer did not become healthy in 90s. Last status: $$status" >&2; \ - docker inspect -f '{{json .State.Health}}' kustainer 2>/dev/null | head -c 2000 >&2; \ - exit 1 - -down: - $(COMPOSE) down - -nuke: - $(COMPOSE) down -v - rm -rf kustainer-data - -logs: - $(COMPOSE) logs -f kustainer - -# Run a one-shot KQL query against $(KQL_DB) (default: Hub). Override on the command line: -# make kql QUERY='.show tables' -# make kql QUERY='Costs_v1_2 | count' (Hub — default) -# make kql KQL_DB=Ingestion QUERY='Costs_raw | count' -kql: - @if [ -z "$(QUERY)" ]; then \ - echo "usage: make kql QUERY=''" >&2; exit 2; \ - fi - @QUERY="$(QUERY)" DB="$(KQL_DB)" pwsh -NoProfile -Command '@{ db = $$env:DB; csl = $$env:QUERY } | ConvertTo-Json -Compress' | \ - curl -sS --fail-with-body --max-time 60 \ - -H 'Content-Type: application/json' \ - -d @- \ - "http://localhost:$(HOST_PORT)/v1/rest/$$(echo '$(QUERY)' | grep -q '^\.' && echo mgmt || echo query)" - @echo - -# ftk CLI passthrough: `make ftk ARGS='run savings-summary-report'`. -# For everyday use call `pwsh scripts/ftk.ps1 ...` directly. -ftk: - @pwsh scripts/ftk.ps1 $(ARGS) - -# List the FinOps Toolkit catalog queries discoverable locally. -ftk-list: - @pwsh scripts/ftk.ps1 list - -# Load the FTK analytics KQL into the two-DB topology (Ingestion + Hub). -# Reads src/templates/finops-hub/.build.config combineKql bundles and -# concatenates the listed .kql files in dependency order. Idempotent. -load-ftk-kql: - @pwsh scripts/load-ftk-kql.ps1 - -# Bulk-ingest parquet under ./export/ into Costs_raw / Prices_raw. Picks the -# latest run-uuid per (scope, type, period) by manifest.runInfo.submittedTime, -# enforces OverwritePreviousReport semantics (drops superseded extents by -# tag), and is idempotent via the Ingest_Manifest table. -# make ingest -# make ingest SCOPE= -# make ingest SCOPE= PERIOD=20260501-20260531 -INGEST_FILTERS := -ifneq ($(SCOPE),) -INGEST_FILTERS += --scope $(SCOPE) -endif -ifneq ($(PERIOD),) -INGEST_FILTERS += --period $(PERIOD) -endif -ingest: - @pwsh scripts/ingest.ps1 $(INGEST_FILTERS) - -# Quick summary of Ingest_Manifest: files + rows per (scope, export_type). -# Targets Ingestion DB explicitly (Ingest_Manifest lives there, not in Hub). -ingest-status: - @$(MAKE) --no-print-directory kql KQL_DB=Ingestion \ - QUERY="Ingest_Manifest | summarize files=count(), rows=sum(rows_ingested) by scope, export_type | order by scope asc, export_type asc" - -# Run the T-1.4 parity check suite. Reads tests/parity-checks.kql, executes -# each numbered check against Kustainer's query endpoint, evaluates the EVAL -# rule, and prints a pass/fail report. Exit code 0 only if all checks pass. -parity: - @pwsh scripts/run-parity-checks.ps1 --endpoint http://localhost:$(HOST_PORT) --db $(PARITY_DB) - -# Recovery path for the Prices_transform_v1_2() backfill when the single-pass -# .set-or-append OOMs. Processes Prices_raw one source extent at a time -# (bounded memory). Use after `make ingest` if Prices_final_v1_2 ends up -# partially populated or empty. Wall-clock ~15 min on Apple Silicon. -# NOTE: ingest.ps1 now auto-chunks proactively when row count > BACKFILL_CHUNK_ROW_THRESHOLD -# (default 2,000,000). This target is the manual fallback / override path. -chunked-prices-backfill: - @pwsh scripts/chunked-prices-backfill.ps1 - -# (Customer-specific savings-plan analysis targets removed — see T-3.5.) diff --git a/src/templates/finops-hub-local/README.md b/src/templates/finops-hub-local/README.md deleted file mode 100644 index bed5db483..000000000 --- a/src/templates/finops-hub-local/README.md +++ /dev/null @@ -1,381 +0,0 @@ -# FinOps hub (local) - -Run the FinOps Toolkit's full KQL cost-analytics stack **locally** against a Kusto -emulator (Docker) — no Azure subscription, no cloud cluster, no Data Factory required. -The analytic IP is identical to a deployed hub: the same `IngestionSetup` / `HubSetup` -KQL, the same hub view functions (`Costs`, `Prices`, `Transactions`, and their `*_v1_2` -variants), the same open data, and the same query catalog. - ---- - -## When to use this vs deploying a real hub - -| Situation | Use | -| -------------------------------------------------------------- | ---------------------------------------------- | -| Offline analysis, air-gapped network, or no Azure subscription | **ftklocal** | -| Developing or testing KQL changes before deploying | **ftklocal** | -| Working with FOCUS parquet already on disk | **ftklocal** | -| Production analytics for an organization | [Deploy a FinOps hub](../finops-hub/README.md) | -| Querying a live Azure Data Explorer or Fabric cluster | [FinOps hub](../finops-hub/README.md) | - ---- - -## Prerequisites - -- **Docker Desktop ≥ 4.0** — Compose v2 is required (`docker compose`, not the legacy - `docker-compose`). -- **PowerShell 7+** (`pwsh`) — all tooling is PowerShell; there are no Python - dependencies. On Windows this is **PowerShell 7**, not the built-in Windows PowerShell - 5.1 (`winget install Microsoft.PowerShell`). -- **FOCUS cost exports as Parquet** — staged under `export/`. See - [notes/staging-contract.md](notes/staging-contract.md) for the folder layout. -- Roughly **16 GiB of RAM** available for the container (the default `MEM_LIMIT` in - `.env.example`). Costs-only datasets work at 8 GiB; 16 GiB is required for the full - Prices transform. See [Limits](#limits). -- Host port **8082** free (configurable via `HOST_PORT` in `.env`). - -**Per-platform install guides** (prerequisites, memory tuning, troubleshooting): - -- 🪟 **[Windows](notes/install-windows.md)** — primary platform; the `linux/amd64` engine - runs natively (no emulation). Covers Docker Desktop + WSL 2, PowerShell 7, and WSL memory - tuning. -- 🍎 **[macOS](notes/install-mac.md)** — secondary; Apple Silicon runs the engine under - Rosetta emulation. Covers Rosetta and Docker Desktop memory. - -**Want to understand what the helpers do?** [Build a local FinOps hub by hand](notes/build-by-hand.md) -walks the whole setup as a dozen raw commands — start the emulator, create the two -databases, load the schema, ingest (the final tables transform themselves via an update -policy), and query. The helper scripts below automate exactly those steps. - ---- - -## Quickstart - -The commands below work on **Windows, macOS, and Linux** (PowerShell 7 + Docker Compose v2). -`make` targets are an optional convenience on macOS/Linux — see the -[macOS guide](notes/install-mac.md). - -```bash -# From src/templates/finops-hub-local/ - -cp .env.example .env # 1. review tunables (HOST_PORT, MEM_LIMIT, EXPORT_DIR) - # (Windows: Copy-Item .env.example .env) -docker compose up -d --wait # 2. start the emulator; blocks until the healthcheck passes -pwsh scripts/load-ftk-kql.ps1 # 3. create Ingestion + Hub DBs, load FTK KQL + open data (idempotent) -pwsh scripts/ingest.ps1 # 4. bulk-ingest all Parquet exports under export/ - -# 5. query your FOCUS data -pwsh scripts/ftk.ps1 query "Costs() | summarize TotalCost = sum(EffectiveCost) by ServiceName | top 10 by TotalCost" -``` - -`docker compose up -d --wait` is the cross-platform equivalent of `make up` — it uses the -container's healthcheck and returns only once the engine is answering. After step 4 -completes, the Hub database holds the same view functions a deployed hub exposes; -`pwsh scripts/ingest.ps1 -DryRun` previews ingest and `make ingest-status` (macOS/Linux) -summarizes what was loaded. - ---- - -## Configuration - -All tunables live in `.env` (gitignored). `.env.example` documents them: - -| Variable | Default | Meaning | -| ------------------------------ | ---------- | ------------------------------------------------------------------------------------------------------------------------ | -| `HOST_PORT` | `8082` | Host port mapped to the Kustainer container's HTTP endpoint. | -| `MEM_LIMIT` | `16g` | Memory ceiling for the Kustainer container. | -| `EXPORT_DIR` | `./export` | Host directory with Parquet exports (read-only mount). | -| `BACKFILL_CHUNK_ROW_THRESHOLD` | `2000000` | Row count above which `ingest.ps1` uses proactive chunked backfill. See [Chunked backfill](#automatic-chunked-backfill). | - ---- - -## Architecture - -### Two-database topology - -The local stack mirrors the real hub's **two-database** layout: - -| Database | Holds | You query it for | -| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------- | -| **`Ingestion`** | Raw tables (`Costs_raw`, `Prices_raw`), `*_transform_v1_2()` functions, final tables (`Costs_final_v1_2`, `Prices_final_v1_2`), open-data lookups, and `Ingest_Manifest` | Ingestion internals, parity checks, transform debugging | -| **`Hub`** | Hub view functions (`Costs`, `Prices`, `Transactions`, `Costs_v1_2`, `Prices_v1_2`, …) that read `database('Ingestion').*` | All analytics — this is the default query database | - -The `Hub` database is the default for `make kql` and `ftk.ps1`. To inspect raw tables, -override: `make kql KQL_DB=Ingestion QUERY='Costs_raw | count'`. - -### How it maps to a real hub - -| Real hub component | Local equivalent | -| ------------------------------------------------------ | ------------------------------------------------ | -| Azure Data Explorer (ADX) cluster | Kusto emulator (Docker, `http://localhost:8082`) | -| `Ingestion` ADX database | `Ingestion` database in the emulator | -| `Hub` ADX database | `Hub` database in the emulator | -| Data Factory pipeline (ADF) | `scripts/ingest.ps1` | -| Azure Storage (`msexports/` or `ingestion/` container) | `export/` directory on disk | -| Bicep/ARM deployment | `make load-ftk-kql` (idempotent KQL loader) | - -The KQL itself — `IngestionSetup_RawTables.kql`, `IngestionSetup_v1_2.kql`, -`HubSetup_v1_2.kql`, `HubSetup_OpenData.kql` — is loaded verbatim from the upstream -FTK source with [minimal on-load adaptations](notes/ftk-kql-adaptations.md). - ---- - -## Ingesting data - -### Staging your exports - -Parquet exports must be staged under `export/` in the layout described in -[notes/staging-contract.md](notes/staging-contract.md). The short version: - -``` -export/ -└── / - └── / # ms--focus-cost | ms--pricesheet - └── / - └── / - ├── manifest.json - └── *.parquet -``` - -A `manifest.json` produced directly by Azure Cost Management can be used without -modification. See [staging-contract.md](notes/staging-contract.md) for minimum -viable manifest fields and instructions for both `msexports/` and `ingestion/` -source containers. - -### Running ingest - -```bash -make ingest # ingest all scopes/periods -make ingest SCOPE=ea # one scope -make ingest SCOPE=ea PERIOD=20260501-20260531 # scope + period -make ingest-status # show Ingest_Manifest summary -``` - -`ingest.ps1` is idempotent: it tracks every ingested file in `Ingest_Manifest` by -`(scope, type, period, run-uuid, file-name)` and SHA-256 checksum. Re-running `make -ingest` on an unchanged export directory is safe — already-ingested files are skipped. - -### Overwrite semantics - -| Situation | Behavior | -| ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Same run-uuid, same file checksum | Skip (already ingested) | -| Same `(scope, type, period)`, **new** run-uuid | Safe replace — old extents are dropped, new data is ingested | -| Same run-uuid, **changed** file checksum | **Fail-fast double-ingest guard** — the script detects the checksum mismatch and refuses. Stage the replacement files under a **new run-uuid** instead. | - -> **How to replace data:** stage replacement parquet under a new `` directory -> with a `manifest.json` whose `runInfo.submittedTime` is later than the existing run. -> `ingest.ps1` selects the latest run per `(scope, type, period)`, drops the old run's -> extents, and ingests the new files cleanly. - -### Automatic chunked backfill - -For large tables, `ingest.ps1` avoids the single-pass OOM by chunking the final-table -backfill automatically: - -- **Proactive:** if the raw table row count exceeds `BACKFILL_CHUNK_ROW_THRESHOLD` - (default 2,000,000), single-pass `.set-or-append` is skipped entirely and per-extent - chunked backfill is used from the start. -- **Reactive:** if the row count is at or below the threshold, single-pass is tried - first; if it OOMs the engine, `ingest.ps1` falls back to chunked automatically. - -The threshold is sized so that at `MEM_LIMIT=16g`: - -- Costs (~1.35 M rows) → single-pass (safely below threshold). -- Prices (~12.7 M rows) → proactive chunked (safely above threshold). - -To trigger the chunked backfill manually (e.g., recovery after a crash): - -```bash -make chunked-prices-backfill -``` - ---- - -## Querying - -### `ftk.ps1` CLI - -```bash -# Ad-hoc query (Hub DB by default) -pwsh scripts/ftk.ps1 query "Costs() | summarize sum(EffectiveCost) by ServiceName | top 10 by sum_EffectiveCost" - -# List available named catalog queries -pwsh scripts/ftk.ps1 list - -# Run a named catalog query -pwsh scripts/ftk.ps1 run savings-summary-report -pwsh scripts/ftk.ps1 run savings-summary-report --format csv -pwsh scripts/ftk.ps1 run savings-summary-report --start 2026-01-01 --end 2026-06-30 - -# Inspect schema -pwsh scripts/ftk.ps1 schema -pwsh scripts/ftk.ps1 schema -Tables -pwsh scripts/ftk.ps1 schema -Functions -pwsh scripts/ftk.ps1 tables Costs_final_v1_2 - -# Override database (e.g., query Ingestion raw tables) -pwsh scripts/ftk.ps1 query "Costs_raw | count" -Database Ingestion -``` - -The CLI connects to `http://localhost:8082` by default. Override with the `HOST_PORT` -environment variable or the `-Endpoint` parameter. - -### Agent skill - -For agent-driven analytics, the `finops-hub-local` agent skill wraps the same CLI -into structured tool calls. See -[`../agent-skills/finops-hub-local/SKILL.md`](../agent-skills/finops-hub-local/SKILL.md). - ---- - -## Dashboard - -The included `dashboard.json` is the upstream FinOps hub dashboard with the cluster -URI changed to `http://localhost:8082` and the database set to `Hub`. All tiles and -KQL queries are unmodified. - -**Proven path (CLI):** `ftk.ps1 query` and `ftk.ps1 run` work today with no browser -dependency. - -**Best-effort path (ADX web UI):** Import `dashboard.json` via -**https://dataexplorer.azure.com → Dashboards → Import dashboard from file**. Note -that browsers apply strict mixed-content rules: connecting an HTTPS page to a plain -`http://localhost:8082` endpoint may be blocked. See -[notes/dashboard.md](notes/dashboard.md) for step-by-step instructions, the -mixed-content caveat, and the ngrok HTTPS workaround. - ---- - -## Limits - -These numbers were measured on `MEM_LIMIT=16g`, amd64-on-Rosetta (Apple Silicon), -with a dataset of approximately 1,350,561 Cost rows and 12,735,587 Price rows plus -open-data lookups. Your results vary by `MEM_LIMIT` and dataset size. - -| Metric | Measured value | -| ------------------------------------------------------------------------------------ | ---------------------------------------------------------- | -| Raw file ingest wall-clock (31 parts, ~991 MB) | 419.6 s (7.0 min) | -| Full `ingest.ps1` wall-clock (raw + all backfill) | 841.6 s (14.0 min) | -| **Cold loaded** — post-restart, data on disk, extents not yet materialized (anon) | **1.12 GiB (7.0% of 16 GiB)** — this is the floor | -| **Hot post-ingest** — immediately after transform pipeline; extents in memory (anon) | **13.99 GiB (≈87% of 16 GiB)** — sustained until restart | -| Chunked backfill memory.peak (cumulative HWM, Prices per-extent) | **15.03 GiB (93.9% of 16 GiB)** | -| Single-pass Prices transform (~12.7 M rows) | anon ~14.74 GiB / memory.peak **15.12 GiB** → engine crash | -| Single-pass Costs transform (~1.35 M rows) | Completes in 53 s; peak 6.89 GiB (43% of 16 GiB) | - -See [notes/performance.md](notes/performance.md) for the full row-count → memory curve, -per-stage cgroup v2 measurements, and OOM ceiling analysis. - ---- - -## Troubleshooting - -### Port 8082 already in use - -```bash -lsof -nP -iTCP:8082 -sTCP:LISTEN -``` - -Set `HOST_PORT=` in `.env` and re-run `make up`. Do not use port 8080. - -### Container does not become healthy in 90 s - -- **Apple Silicon:** ensure Rosetta is enabled (Docker Desktop → Settings → General). -- **Logs:** `make logs` — look for `Database NetDefaultDB has been created and is -answering queries`. -- **Memory:** `docker stats --no-stream kustainer` — raise `MEM_LIMIT` in `.env` if - near the ceiling. After editing `.env`, always use `make down && make up` (never - `--force-recreate`; see below). - -### `Ingestion` or `Hub` DB missing after `make up` - -After a `make down && make up` cycle the databases should reattach from `kustainer-data/`. -If they are missing: - -1. Run `make nuke` to remove the data volume. -2. Run `make up && make load-ftk-kql && make ingest` to rebuild from scratch. - -If you used `docker compose up --force-recreate` the metadata volume was torn down -and cannot be reattached. `make nuke` + full reload is the only recovery path. - -### Row count is 0 after `make ingest` - -Check whether the update policy was left disabled by a crashed run: - -```bash -make kql KQL_DB=Ingestion QUERY=".show table Costs_final_v1_2 policy update" -``` - -If `IsEnabled` is `false`, re-enable the policy and manually trigger the backfill. -`ingest.ps1` will refuse to run until the policy is restored; pass -`--force-policy-recapture` only if you have manually verified the disabled state is -safe. - -### Prices backfill OOM (container exits under load) - -Raise `MEM_LIMIT` to `16g` in `.env`: - -```bash -# Edit .env, then bounce: -make down && make up -make ingest -``` - -If the OOM persists at 16 GiB (rare, larger datasets), use the manual chunked path: - -```bash -make chunked-prices-backfill -``` - -`ingest.ps1` normally handles the reactive fallback automatically; this target is the -manual override. - -### Checksum mismatch guard - -If you modified files in an already-ingested run directory, `ingest.ps1` will refuse -with a checksum error. This is the **fail-fast double-ingest guard**: mutating an -ingested file is not a safe overwrite path. Stage the corrected files under a **new -`` directory** and re-run `make ingest`. - -### Rosetta SIGSEGV (rare crashes under heavy load) - -`restart: unless-stopped` in `docker-compose.yml` restarts the container automatically. -The `kustainer-data/` volume is preserved and the databases reattach. `ingest.ps1`'s -chunked-backfill retry loop catches the resulting `RemoteDisconnected` error, waits for -the engine to recover, drops any partial output, and retries. No operator action is -needed unless the same chunk fails repeatedly (lower `--extents-per-batch` or raise -`MEM_LIMIT`). - ---- - -## Parity status - -The parity check suite validates that the local transform produces output identical to -what the FTK pipeline would produce. Current status: **11 pass / 0 fail / 2 manual** -of 13 checks. See [notes/parity-gaps.md](notes/parity-gaps.md) for details. - -Run the suite: - -```bash -make parity -``` - ---- - -## Makefile reference - -| Target | What it does | -| ---------------------------------- | ---------------------------------------------------------------- | -| `make up` | Bring Kustainer up and block until the healthcheck passes. | -| `make down` | Stop the container (data volume preserved). | -| `make nuke` | Destructive full reset — removes the `kustainer-data/` volume. | -| `make logs` | Tail Kustainer container logs. | -| `make kql QUERY='...'` | One-shot KQL against Hub (override: `KQL_DB=Ingestion`). | -| `make load-ftk-kql` | Load FTK KQL + open data into Ingestion + Hub (idempotent). | -| `make ingest` | Bulk-ingest all Parquet exports under `export/`. | -| `make ingest SCOPE=` | Ingest one scope only. | -| `make ingest SCOPE= PERIOD=

` | Ingest one scope + period. | -| `make ingest-status` | Show `Ingest_Manifest` summary (files + rows per scope/type). | -| `make parity` | Run the parity check suite (exit 0 iff all checks pass). | -| `make chunked-prices-backfill` | Manual per-extent Prices backfill (auto-triggered by threshold). | -| `make help` | Print all targets. | diff --git a/src/templates/finops-hub-local/dashboard.json b/src/templates/finops-hub-local/dashboard.json deleted file mode 100644 index 521dec787..000000000 --- a/src/templates/finops-hub-local/dashboard.json +++ /dev/null @@ -1,5559 +0,0 @@ -{ - "$schema": "https://dataexplorer.azure.com/static/d/schema/60/dashboard.json", - "id": "ftklocal-adx-dashboard", - "eTag": "", - "title": "FinOps hub (local)", - "schema_version": "60", - "tiles": [ - { - "id": "acf2ca3d-9d8f-477f-b259-f3937bf938d4", - "title": "Cost last month", - "visualType": "multistat", - "pageId": "f416685d-f559-4514-8e45-5e0e09aec286", - "layout": { - "x": 0, - "y": 3, - "width": 4, - "height": 8 - }, - "queryRef": { - "kind": "query", - "queryId": "152f2041-bbc1-41e4-b155-271b2e0cf6e9" - }, - "visualOptions": { - "multiStat__textSize": "large", - "multiStat__valueColumn": null, - "colorRulesDisabled": true, - "colorStyle": "light", - "multiStat__displayOrientation": "horizontal", - "multiStat__labelColumn": null, - "multiStat__slot": { - "width": 1, - "height": 2 - }, - "colorRules": [] - } - }, - { - "id": "595e39e5-02ff-4ee7-99f9-68694ffd0495", - "title": "Cost over time", - "visualType": "column", - "pageId": "f416685d-f559-4514-8e45-5e0e09aec286", - "layout": { - "x": 4, - "y": 3, - "width": 17, - "height": 4 - }, - "queryRef": { - "kind": "query", - "queryId": "bc24e050-f2b9-4b4a-a08d-69fc4a4bb95e" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": false, - "legendLocation": "left", - "xColumnTitle": "", - "xColumn": null, - "yColumns": null, - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "efd2b03d-c80d-4d78-b77a-efa3f932efdb", - "title": "Change over time", - "visualType": "column", - "pageId": "f416685d-f559-4514-8e45-5e0e09aec286", - "layout": { - "x": 4, - "y": 7, - "width": 17, - "height": 4 - }, - "queryRef": { - "kind": "query", - "queryId": "0d91ea4a-c81d-4a21-b708-b6af37be1eec" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": false, - "legendLocation": "left", - "xColumnTitle": "", - "xColumn": null, - "yColumns": null, - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "9c5e208a-f0ce-4e2e-9969-35130eef22c1", - "title": "Counts", - "visualType": "multistat", - "pageId": "f416685d-f559-4514-8e45-5e0e09aec286", - "layout": { - "x": 0, - "y": 11, - "width": 12, - "height": 4 - }, - "queryRef": { - "kind": "query", - "queryId": "f2cecbb0-13f8-4642-afa4-bbcc0558f777" - }, - "visualOptions": { - "multiStat__textSize": "auto", - "multiStat__valueColumn": null, - "colorRulesDisabled": false, - "colorStyle": "light", - "multiStat__displayOrientation": "horizontal", - "multiStat__labelColumn": null, - "multiStat__slot": { - "width": 4, - "height": 1 - }, - "colorRules": [] - } - }, - { - "id": "2a1ee947-f1e3-447e-9388-91e318068608", - "title": "Counts (last n days)", - "visualType": "multistat", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 15, - "width": 6, - "height": 8 - }, - "queryRef": { - "kind": "query", - "queryId": "5d63e04d-1a11-4307-96f1-ebbf68e09be0" - }, - "visualOptions": { - "multiStat__textSize": "auto", - "multiStat__valueColumn": null, - "colorRulesDisabled": true, - "colorStyle": "light", - "multiStat__displayOrientation": "vertical", - "multiStat__labelColumn": null, - "multiStat__slot": { - "width": 2, - "height": 2 - }, - "colorRules": [ - { - "id": "414e8053-b30c-40dd-9f3f-510910d54d83", - "ruleType": "colorByCondition", - "applyToColumn": null, - "hideText": false, - "applyTo": "cells", - "conditions": [ - { - "column": null, - "operator": ">", - "values": [ - "" - ] - } - ], - "chainingOperator": "and", - "colorStyle": "bold", - "color": null, - "tag": "", - "icon": null, - "ruleName": "", - "visualType": "multistat" - } - ] - } - }, - { - "id": "49534f2e-97ea-4b41-87c2-5e0f4bc26db8", - "title": "Effective cost", - "visualType": "column", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 6, - "y": 2, - "width": 16, - "height": 9 - }, - "queryRef": { - "kind": "query", - "queryId": "290e7eab-8159-4338-8531-85e2718cedb1" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "Effective cost", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [ - { - "id": "e7c54e29-4843-465c-ab63-75e816da8fe0", - "columns": [ - "Change" - ], - "label": "% change", - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - } - ], - "showMultiplePanels": true - }, - "hideLegend": true, - "legendLocation": "left", - "xColumnTitle": "", - "xColumn": null, - "yColumns": null, - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "8e17c9f7-c513-4b93-bab8-44d3b0e48e9c", - "title": "", - "visualType": "markdownCard", - "pageId": "969ddf4c-8f2a-4ec4-9588-bb2f39473c9f", - "layout": { - "x": 0, - "y": 0, - "width": 18, - "height": 6 - }, - "markdownText": "# FinOps hubs $$ftkver$$ ($$build-month$$)\nThe FinOps hub dashboard provides several summaries of your cost and usage in aligment with the [FinOps Framework](https://aka.ms/finops/fx). All data formatted based on the [FinOps Open Cost and Usage Specification (FOCUS)](https://aka.ms/finops/focus).\n\nMost of this dashboard is designed to work with effective (amortized) cost rather than billed (actual) cost. Amortization breaks down reservation and savings plan purchases and allocates costs to the resources that received the benefit to facilitate chargeback reporting. Effective costs will not match your invoice. For invoice reconciliation, refer to the **Invoicing** page.\n\n[Learn more](https://aka.ms/finops/hubs)     [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Summary)\n", - "visualOptions": {} - }, - { - "id": "f5fc42ae-fb77-4df7-9407-5c9c3588dd84", - "title": "About the FinOps toolkit", - "visualType": "markdownCard", - "pageId": "969ddf4c-8f2a-4ec4-9588-bb2f39473c9f", - "layout": { - "x": 18, - "y": 0, - "width": 4, - "height": 6 - }, - "markdownText": "FinOps hubs are part of the [FinOps toolkit](https://aka.ms/finops/toolkit), an open-source collection of FinOps solutions that help you manage and optimize your cost, usage, and carbon.\n\nTo contribute to the FinOps toolkit, [join us on GitHub](https://aka.ms/ftk).\n", - "visualOptions": {} - }, - { - "id": "570e40a5-f67a-4c4b-84f6-c207ed021812", - "title": "3-month running total trend", - "visualType": "column", - "pageId": "5838c918-4541-44fe-90d2-77306ef1e241", - "layout": { - "x": 0, - "y": 14, - "width": 16, - "height": 7 - }, - "queryRef": { - "kind": "query", - "queryId": "d5ed469a-45ea-49a2-b305-b841050213cf" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": false, - "legendLocation": "bottom", - "xColumnTitle": "", - "xColumn": null, - "yColumns": null, - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "1ac02a9f-3c12-4828-a4dd-60a0d411f60c", - "title": "Running total - This month and last", - "visualType": "area", - "pageId": "5c57f940-6f0f-4e39-930b-b6e89bb758ad", - "layout": { - "x": 0, - "y": 3, - "width": 16, - "height": 8 - }, - "queryRef": { - "kind": "query", - "queryId": "10300876-866a-40d7-836d-b3a8783ecece" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": false, - "legendLocation": "bottom", - "xColumnTitle": "", - "xColumn": null, - "yColumns": null, - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "27ff6f92-2e6f-431f-bf3b-f36aad59b5f5", - "title": "3-month daily trend", - "visualType": "column", - "pageId": "5838c918-4541-44fe-90d2-77306ef1e241", - "layout": { - "x": 0, - "y": 21, - "width": 16, - "height": 5 - }, - "queryRef": { - "kind": "query", - "queryId": "9e41a624-d5f9-40c6-b47f-fef4b50ec3dd" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": false, - "legendLocation": "bottom", - "xColumnTitle": "", - "xColumn": null, - "yColumns": null, - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "97df5060-929c-4457-a7e9-47005c845f16", - "title": "3-month daily trend by subscription", - "visualType": "stackedcolumn", - "pageId": "5838c918-4541-44fe-90d2-77306ef1e241", - "layout": { - "x": 0, - "y": 26, - "width": 16, - "height": 7 - }, - "queryRef": { - "kind": "query", - "queryId": "f26e2204-270e-4219-8f68-5acef1c9393f" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": true, - "legendLocation": "right", - "xColumnTitle": "", - "xColumn": null, - "yColumns": null, - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "9", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "dad96fbb-750c-47a3-bd2e-e2bee74ebd22", - "title": "Cost + savings running total (this month and last)", - "visualType": "stackedarea", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 6, - "y": 11, - "width": 16, - "height": 7 - }, - "queryRef": { - "kind": "query", - "queryId": "dcc1f533-e5f9-4855-9e4c-21e21fcdf943" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "Cost + savings", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": false, - "legendLocation": "bottom", - "xColumnTitle": "", - "xColumn": null, - "yColumns": [ - "EffectiveCostRunningTotal", - "NegotiatedDiscountSavingsRunningTotal", - "CommitmentDiscountSavingsRunningTotal" - ], - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "66dcd731-a141-47b2-b36a-53a138cb75f2", - "title": "Summary", - "visualType": "multistat", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 2, - "width": 6, - "height": 13 - }, - "queryRef": { - "kind": "query", - "queryId": "e8b343dc-7430-4487-8d44-ef48ac454f2d" - }, - "visualOptions": { - "multiStat__textSize": "auto", - "multiStat__valueColumn": null, - "colorRulesDisabled": false, - "colorStyle": "light", - "multiStat__displayOrientation": "vertical", - "multiStat__labelColumn": null, - "multiStat__slot": { - "width": 2, - "height": 4 - }, - "colorRules": [ - { - "id": "a45d5307-a815-4baa-9743-d142253dc699", - "ruleType": "colorByCondition", - "applyToColumn": null, - "hideText": false, - "applyTo": "cells", - "conditions": [ - { - "operator": "==", - "column": "IsThisMonth", - "values": [ - "true" - ] - } - ], - "chainingOperator": "and", - "colorStyle": "light", - "color": "blue", - "tag": "This month", - "icon": null, - "ruleName": "This month", - "visualType": "multistat" - }, - { - "id": "a1307a22-3b48-48a9-9c1b-ef6b6a228796", - "ruleType": "colorByCondition", - "applyToColumn": null, - "hideText": false, - "applyTo": "cells", - "conditions": [ - { - "operator": "==", - "column": "IsThisMonth", - "values": [ - "false" - ] - } - ], - "chainingOperator": "and", - "colorStyle": "light", - "color": "green", - "tag": "Last month", - "icon": null, - "ruleName": "Last month", - "visualType": "multistat" - } - ] - } - }, - { - "id": "04cb63d9-0a92-488f-b863-815466d6725c", - "title": "Summary", - "visualType": "table", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 23, - "width": 22, - "height": 5 - }, - "queryRef": { - "kind": "query", - "queryId": "c077a6d4-719f-42fe-b39b-36f77dc68976" - }, - "visualOptions": { - "table__enableRenderLinks": true, - "colorRulesDisabled": false, - "colorStyle": "light", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "table__renderLinks": [], - "colorRules": [] - } - }, - { - "id": "4faba5ce-4190-45f9-b41e-5d9c6a9c8098", - "title": "Summary", - "visualType": "table", - "pageId": "5c57f940-6f0f-4e39-930b-b6e89bb758ad", - "layout": { - "x": 0, - "y": 11, - "width": 22, - "height": 5 - }, - "queryRef": { - "kind": "query", - "queryId": "3d947a7c-edca-46b5-a862-de77a725c85f" - }, - "visualOptions": { - "table__enableRenderLinks": true, - "colorRulesDisabled": false, - "colorStyle": "light", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "table__renderLinks": [], - "colorRules": [] - } - }, - { - "id": "890e0366-a9a5-4001-bc6e-f1b7f23fa8a7", - "title": "Effective cost by subscription", - "visualType": "table", - "pageId": "5c57f940-6f0f-4e39-930b-b6e89bb758ad", - "layout": { - "x": 0, - "y": 16, - "width": 22, - "height": 5 - }, - "queryRef": { - "kind": "query", - "queryId": "3886b5cd-34a8-42d7-9e16-33ea4d236953" - }, - "visualOptions": { - "table__enableRenderLinks": true, - "colorRulesDisabled": false, - "colorStyle": "light", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "table__renderLinks": [], - "colorRules": [] - } - }, - { - "id": "e47767ba-1884-4ca7-8458-24025b92464a", - "title": "", - "visualType": "markdownCard", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 58, - "width": 22, - "height": 4 - }, - "markdownText": "# Regions\r\nEffective cost breakdown by region\r\n\r\n\u2b06\ufe0f [Top](?tile=e3ac3bbc-a9ef-44dd-828f-226afbcfda33)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.Summary.Regions)", - "visualOptions": {} - }, - { - "id": "d3e265ff-2c5a-4a03-a8b8-e0b4e10cf362", - "title": "Monthly trend by region", - "visualType": "stackedcolumn", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 62, - "width": 22, - "height": 5 - }, - "queryRef": { - "kind": "query", - "queryId": "d5224d06-c8e7-4dd9-afec-595b39712f5a" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": true, - "legendLocation": "right", - "xColumnTitle": "", - "xColumn": null, - "yColumns": null, - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "9", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "c54de4ff-e793-49b7-b33d-db22fccd0290", - "title": "Monthly trend by region", - "visualType": "table", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 67, - "width": 22, - "height": 8 - }, - "queryRef": { - "kind": "query", - "queryId": "0dcf2c54-7f1d-45e9-a53b-d598a24493a4" - }, - "visualOptions": { - "table__enableRenderLinks": true, - "colorRulesDisabled": false, - "colorStyle": "light", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "table__renderLinks": [], - "colorRules": [] - } - }, - { - "id": "b2916ba0-cb54-433d-89a1-8905b8d8618a", - "title": "Monthly trend by service name", - "visualType": "stackedcolumn", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 45, - "width": 22, - "height": 5 - }, - "queryRef": { - "kind": "query", - "queryId": "30644718-defd-4c6c-9ffa-a9c2cd1f871f" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": true, - "legendLocation": "right", - "xColumnTitle": "", - "xColumn": null, - "yColumns": null, - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "9", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "6208961b-6249-4fd4-a2d6-16b5e58b9baa", - "title": "Monthly trend by service name", - "visualType": "table", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 50, - "width": 22, - "height": 8 - }, - "queryRef": { - "kind": "query", - "queryId": "6259f773-593c-4953-898c-15aa5ff6e53a" - }, - "visualOptions": { - "table__enableRenderLinks": true, - "colorRulesDisabled": false, - "colorStyle": "light", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "table__renderLinks": [], - "colorRules": [] - } - }, - { - "id": "4956ec68-b338-4926-bd72-ae6ca9e886d6", - "title": "", - "visualType": "markdownCard", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 75, - "width": 22, - "height": 4 - }, - "markdownText": "# Subscriptions\r\nEffective cost breakdown by subscription (subaccount)\r\n\r\n\u2b06\ufe0f [Top](?tile=e3ac3bbc-a9ef-44dd-828f-226afbcfda33)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.Summary.Subscriptions)", - "visualOptions": {} - }, - { - "id": "b28897e5-0606-4112-a13a-0cd814055ea7", - "title": "Monthly trend by subscription", - "visualType": "stackedcolumn", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 79, - "width": 22, - "height": 5 - }, - "queryRef": { - "kind": "query", - "queryId": "21a87abe-19e2-44ec-8298-ba872e66c162" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": true, - "legendLocation": "right", - "xColumnTitle": "", - "xColumn": null, - "yColumns": null, - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "9", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "7b24b810-3cd3-4b7c-92cc-6b40f071b24d", - "title": "Monthly trend by subscription", - "visualType": "table", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 84, - "width": 22, - "height": 8 - }, - "queryRef": { - "kind": "query", - "queryId": "90502e9a-2d0d-4ae4-8d9d-cc21f9b72d5d" - }, - "visualOptions": { - "table__enableRenderLinks": true, - "colorRulesDisabled": false, - "colorStyle": "light", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "table__renderLinks": [], - "colorRules": [] - } - }, - { - "id": "f56a2d38-7d69-4dfd-8cc8-6b7a292758b1", - "title": "Monthly trend by resource group", - "visualType": "table", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 101, - "width": 22, - "height": 8 - }, - "queryRef": { - "kind": "query", - "queryId": "c2c65ec0-e57d-4834-8a6b-b5975afeb9a0" - }, - "visualOptions": { - "table__enableRenderLinks": true, - "colorRulesDisabled": false, - "colorStyle": "light", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "table__renderLinks": [], - "colorRules": [] - } - }, - { - "id": "d2672640-0e48-47e5-a253-de927a77a637", - "title": "Monthly trend by resource group", - "visualType": "stackedcolumn", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 96, - "width": 22, - "height": 5 - }, - "queryRef": { - "kind": "query", - "queryId": "61b26784-6a73-4e80-85b2-9c5cfbd2dd06" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": true, - "legendLocation": "right", - "xColumnTitle": "", - "xColumn": null, - "yColumns": null, - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "9", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "6750ce15-ec4e-4c60-8079-b00a4b462d63", - "title": "", - "visualType": "markdownCard", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 92, - "width": 22, - "height": 4 - }, - "markdownText": "# Resource groups\r\nEffective cost breakdown by resource group\r\n\r\n\u2b06\ufe0f [Top](?tile=e3ac3bbc-a9ef-44dd-828f-226afbcfda33)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.Summary.ResourceGroups)", - "visualOptions": {} - }, - { - "id": "52055726-2788-4216-b813-9d91714c5953", - "title": "", - "visualType": "markdownCard", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 109, - "width": 22, - "height": 4 - }, - "markdownText": "# Resources\r\nEffective cost breakdown by resource\r\n\r\n\u2b06\ufe0f [Top](?tile=e3ac3bbc-a9ef-44dd-828f-226afbcfda33)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.Summary.Resources)", - "visualOptions": {} - }, - { - "id": "d84a014f-0b68-490b-a522-fba66fee1e0f", - "title": "Monthly trend by resource", - "visualType": "stackedcolumn", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 113, - "width": 22, - "height": 5 - }, - "queryRef": { - "kind": "query", - "queryId": "3924981c-23e4-464d-a872-045df1752750" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": true, - "legendLocation": "right", - "xColumnTitle": "", - "xColumn": null, - "yColumns": null, - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "9", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "586ebe0b-ff5c-4794-91a1-a8ec845cad3a", - "title": "Monthly trend by resource", - "visualType": "table", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 123, - "width": 22, - "height": 8 - }, - "queryRef": { - "kind": "query", - "queryId": "c7be613e-deb4-4779-ad75-4445e8d5e01f" - }, - "visualOptions": { - "table__enableRenderLinks": true, - "colorRulesDisabled": false, - "colorStyle": "light", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "table__renderLinks": [], - "colorRules": [] - } - }, - { - "id": "c03c1e57-5e66-4eb1-a20b-10c887eb2140", - "title": "Daily trend by resource", - "visualType": "stackedcolumn", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 118, - "width": 22, - "height": 5 - }, - "queryRef": { - "kind": "query", - "queryId": "49e24ee0-91de-4b1c-973f-036a3c060aca" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": true, - "legendLocation": "right", - "xColumnTitle": "", - "xColumn": null, - "yColumns": null, - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "9", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "d9960334-1cff-4686-9d4a-e6afc5574aa7", - "title": "", - "visualType": "markdownCard", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 28, - "width": 22, - "height": 4 - }, - "markdownText": "# Services\r\nEffective cost breakdown by service category and service name\r\n\r\n\u2b06\ufe0f [Top](?tile=e3ac3bbc-a9ef-44dd-828f-226afbcfda33)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.Summary.Services)", - "visualOptions": {} - }, - { - "id": "700f736d-56b1-4194-8282-891683f63134", - "title": "Monthly trend by service category", - "visualType": "stackedcolumn", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 32, - "width": 22, - "height": 5 - }, - "queryRef": { - "kind": "query", - "queryId": "4c7a7614-9b8c-415b-a4e1-d2d53c023d31" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": true, - "legendLocation": "right", - "xColumnTitle": "", - "xColumn": null, - "yColumns": null, - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "9", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "97872267-01c6-44f9-ad0d-34940d4b58fa", - "title": "Monthly trend by service category", - "visualType": "table", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 37, - "width": 22, - "height": 8 - }, - "queryRef": { - "kind": "query", - "queryId": "1d273b55-d2ea-427c-8a5f-01f6c240e98a" - }, - "visualOptions": { - "table__enableRenderLinks": true, - "colorRulesDisabled": false, - "colorStyle": "light", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "table__renderLinks": [], - "colorRules": [] - } - }, - { - "id": "a8edc317-353b-4648-9d32-e73e0d0a2894", - "title": "", - "visualType": "markdownCard", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 131, - "width": 22, - "height": 4 - }, - "markdownText": "# Resource inventory\r\nSummary of resources and their costs\r\n\r\n\u2b06\ufe0f [Top](?tile=e3ac3bbc-a9ef-44dd-828f-226afbcfda33)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.Summary.ResourceInventory)", - "visualOptions": {} - }, - { - "id": "e26eb29d-8021-406c-a84e-5568ddd234f4", - "title": "Monthly trend by resource type", - "visualType": "stackedcolumn", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 154, - "width": 22, - "height": 5 - }, - "queryRef": { - "kind": "query", - "queryId": "90e30901-a931-4a1d-b81e-0a1821032c3c" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": true, - "legendLocation": "right", - "xColumnTitle": "", - "xColumn": null, - "yColumns": null, - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "9", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "8d63f5d3-b1d8-4117-a358-bce7939a13ae", - "title": "Daily trend by resource type (last n days)", - "visualType": "stackedcolumn", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 141, - "width": 22, - "height": 5 - }, - "queryRef": { - "kind": "query", - "queryId": "ebe41e27-e9f9-478e-ab90-fd1f87906766" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": true, - "legendLocation": "right", - "xColumnTitle": "", - "xColumn": null, - "yColumns": null, - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "9", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "8caa438b-10a8-418c-ab3b-5c5aee53d3e6", - "title": "Monthly trend by resource", - "visualType": "table", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 159, - "width": 22, - "height": 8 - }, - "queryRef": { - "kind": "query", - "queryId": "7f7b08a9-ae15-46f8-8b0f-767280375add" - }, - "visualOptions": { - "table__enableRenderLinks": true, - "colorRulesDisabled": false, - "colorStyle": "light", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "table__renderLinks": [], - "colorRules": [] - } - }, - { - "id": "011c2b2c-a232-4459-acf4-8db4055f4f70", - "title": "Most cost (last n days)", - "visualType": "bar", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 17, - "y": 146, - "width": 5, - "height": 8 - }, - "queryRef": { - "kind": "query", - "queryId": "a35b4c05-8ecb-4c51-9aaf-980af3d7923e" - }, - "description": "Uses the Daily trend parameter for the lookback period", - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": true, - "legendLocation": "right", - "xColumnTitle": "", - "xColumn": null, - "yColumns": [ - "EffectiveCost", - "ResourceCount" - ], - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "9", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "4495e71e-1005-4224-91f4-8103f06981da", - "title": "Most used (last n days)", - "visualType": "bar", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 12, - "y": 146, - "width": 5, - "height": 8 - }, - "queryRef": { - "kind": "query", - "queryId": "9c5d3cf0-b6cb-45a2-acfd-b19df425bddf" - }, - "description": "Uses the Daily trend parameter for the lookback period", - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": true, - "legendLocation": "right", - "xColumnTitle": "", - "xColumn": null, - "yColumns": [ - "EffectiveCost", - "ResourceCount" - ], - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "9", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "38ffb57b-4646-4944-a92b-6966a3a6a807", - "title": "Resource type summary (last n days)", - "visualType": "table", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 146, - "width": 12, - "height": 8 - }, - "queryRef": { - "kind": "query", - "queryId": "2d7b6447-2769-40b8-958a-f252dab68b1e" - }, - "visualOptions": { - "table__enableRenderLinks": true, - "colorRulesDisabled": false, - "colorStyle": "light", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "table__renderLinks": [], - "colorRules": [] - } - }, - { - "id": "a83872cb-44ef-402a-8512-b9d5ba04744e", - "title": "Resource inventory summary (last n days)", - "visualType": "multistat", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 135, - "width": 22, - "height": 6 - }, - "queryRef": { - "kind": "query", - "queryId": "f062533b-8c94-412f-bf83-0eb5cd06063d" - }, - "hideTitle": true, - "visualOptions": { - "multiStat__textSize": "auto", - "multiStat__valueColumn": "Value", - "colorRulesDisabled": false, - "colorStyle": "light", - "multiStat__displayOrientation": "horizontal", - "multiStat__labelColumn": null, - "multiStat__slot": { - "width": 4, - "height": 2 - }, - "colorRules": [] - } - }, - { - "id": "5bd3258c-034c-4e23-b543-7cdf87bd153d", - "title": "", - "visualType": "markdownCard", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 0, - "y": 114, - "width": 22, - "height": 4 - }, - "markdownText": "# SKU prices\r\nSummary of the SKUs used and their applicable discounts\r\n\r\n\u2b06\ufe0f [Top](?tile=c0bd4bc4-a910-4a74-89a9-d4ae0f995563)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.RateOptimization.SkuPrices)", - "visualOptions": {} - }, - { - "id": "9d9838ca-63b2-4239-b2a7-346273bc2b86", - "title": "", - "visualType": "markdownCard", - "pageId": "ea47329e-0bc9-4c11-b110-534878dbb3ad", - "layout": { - "x": 0, - "y": 0, - "width": 22, - "height": 5 - }, - "markdownText": "# Understand uage and cost\nThe **Understand usage and cost** domain is focused on data acquisition, reporting, analysis, and alerting on top of your cost, usage, and carbon consumption. This domain focuses on observability and business intelligence. It involves gathering data (ingestion), organizing it for the organization (allocation), generating reports (reporting), and monitoring to proactively identify and address issues (anomalies).\n\n[Learn more](https://learn.microsoft.com/cloud-computing/finops/framework/understand/understand-cloud-usage-cost)     [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand)\n", - "visualOptions": {} - }, - { - "id": "679acf08-65b5-434c-a0fc-29e178c958c3", - "title": "", - "visualType": "markdownCard", - "pageId": "ea47329e-0bc9-4c11-b110-534878dbb3ad", - "layout": { - "x": 0, - "y": 5, - "width": 5, - "height": 7 - }, - "markdownText": "### Data ingestion\nData ingestion refers to the process of collecting, transforming, and organizing data from various sources into a single, easily accessible repository. With this capability, you identify data sources needed to support your FinOps practice and facilitate FinOps tasks.\n\n \n\n \n\n\ud83d\udcca [View report](#9e099251-9658-48da-b416-80422a2a47c7)\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/ingestion)", - "visualOptions": {} - }, - { - "id": "0d15a5a6-6e0e-4825-8cad-77f660b000da", - "title": "", - "visualType": "markdownCard", - "pageId": "ea47329e-0bc9-4c11-b110-534878dbb3ad", - "layout": { - "x": 5, - "y": 5, - "width": 6, - "height": 7 - }, - "markdownText": "### Allocation\nAllocation is the process of attributing, assigning, and redistributing shared costs and usage by using accounts, tags, and other metadata. It helps establish accountability among teams and projects within an organization. With this capability, you identify the metadata needed to map cost, usage, and carbon data to the teams responsible. This capability is critical to support organizational reporting and driving the right level of accountability.\n\n \n\n\ud83d\udcca Coming soon\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/allocation)\n", - "visualOptions": {} - }, - { - "id": "c249daec-c935-4669-b6fd-00d1b3fc8818", - "title": "", - "visualType": "markdownCard", - "pageId": "ea47329e-0bc9-4c11-b110-534878dbb3ad", - "layout": { - "x": 11, - "y": 5, - "width": 6, - "height": 7 - }, - "markdownText": "### Reporting + analytics\nReporting and analytics involve analyzing cloud data and creating reports to understand usage and spending patterns. It helps identify opportunities for improvement and supports informed decision-making about cloud resources. With this capability, you identify and build the reports and reporting infrastructure needed to facilitate other FinOps capabilities. This capability is critical to ensuring proper visibility and transparency and unblocks monitoring and optimization at scale across the organization.\n\n \n\n\ud83d\udcca [View report](#f8ee3008-df7e-442f-825e-2e3b46e4c185)\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/reporting)\n", - "visualOptions": {} - }, - { - "id": "652a4347-742c-42d2-aa43-39459b1e71c6", - "title": "", - "visualType": "markdownCard", - "pageId": "ea47329e-0bc9-4c11-b110-534878dbb3ad", - "layout": { - "x": 17, - "y": 5, - "width": 5, - "height": 7 - }, - "markdownText": "### Anomaly management\nAnomaly management refers to the practice of detecting and addressing abnormal or unexpected cost and usage patterns in a timely manner. With this capability, you identify, triage, and address unexpected changes in cost, usage, and carbon. This capability is especially important with usage-based billing and sustainability models, where small fluctuations in usage can result in large fiscal and environmental impact.\n\n\ud83d\udcca [View report](#5838c918-4541-44fe-90d2-77306ef1e241)\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/anomalies)\n", - "visualOptions": {} - }, - { - "id": "6ea8b37d-617b-47dd-add0-8e3c0f708540", - "title": "", - "visualType": "markdownCard", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 0, - "y": 103, - "width": 22, - "height": 4 - }, - "markdownText": "# Purchases\r\nCosts from commitment discount, Marketplace, and other purchases\r\n\r\n\u2b06\ufe0f [Top](?tile=c0bd4bc4-a910-4a74-89a9-d4ae0f995563)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.RateOptimization.Purchases)", - "visualOptions": {} - }, - { - "id": "e3ac3bbc-a9ef-44dd-828f-226afbcfda33", - "title": "On this page", - "visualType": "markdownCard", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 0, - "width": 22, - "height": 2 - }, - "markdownText": "  [Services](?tile=d9960334-1cff-4686-9d4a-e6afc5574aa7)\r\n\u2022 [Regions](?tile=e47767ba-1884-4ca7-8458-24025b92464a)\r\n\u2022 [Subscriptions](?tile=4956ec68-b338-4926-bd72-ae6ca9e886d6)\r\n\u2022 [Resource groups](?tile=6750ce15-ec4e-4c60-8079-b00a4b462d63)\r\n\u2022 [Resources](?tile=52055726-2788-4216-b813-9d91714c5953)\r\n\u2022 [Inventory](?tile=a8edc317-353b-4648-9d32-e73e0d0a2894)\r\n\u2022 [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.Summary.Nav)", - "visualOptions": {} - }, - { - "id": "00b9ea3a-3902-4234-806f-1a7bf87b3992", - "title": "Daily trend", - "visualType": "column", - "pageId": "5838c918-4541-44fe-90d2-77306ef1e241", - "layout": { - "x": 0, - "y": 4, - "width": 16, - "height": 10 - }, - "queryRef": { - "kind": "query", - "queryId": "1d9c166d-22b6-48fd-9a90-9f983083ecc7" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "Effective cost", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [ - { - "id": "e7c54e29-4843-465c-ab63-75e816da8fe0", - "columns": [ - "Change" - ], - "label": "% change", - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - } - ], - "showMultiplePanels": true - }, - "hideLegend": true, - "legendLocation": "left", - "xColumnTitle": "", - "xColumn": null, - "yColumns": null, - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "c0bd4bc4-a910-4a74-89a9-d4ae0f995563", - "title": "On this page", - "visualType": "markdownCard", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 0, - "y": 0, - "width": 22, - "height": 2 - }, - "markdownText": "  [Savings](?tile=fa177e0f-9a74-4475-b64d-a399224e96e0)\r\n\u2022 [Commitment discounts](?tile=cad5d666-02e8-4319-bbad-1bdf46ca2961)\r\n\u2022 [Purchases](?tile=6ea8b37d-617b-47dd-add0-8e3c0f708540)\r\n\u2022 [SKU prices](?tile=5bd3258c-034c-4e23-b543-7cdf87bd153d)\r\n\u2022 [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.RateOptimization.Nav)\r\n", - "visualOptions": {} - }, - { - "id": "72aa5a62-e129-4ffa-be62-82c833282d27", - "title": "SKUs with negotiated discounts (last n days)", - "visualType": "table", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 0, - "y": 118, - "width": 14, - "height": 6 - }, - "queryRef": { - "kind": "query", - "queryId": "af6424d4-8f73-4b81-bfdd-f0287ebcaaca" - }, - "visualOptions": { - "table__enableRenderLinks": true, - "colorRulesDisabled": false, - "colorStyle": "light", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "table__renderLinks": [], - "colorRules": [] - } - }, - { - "id": "3d7688f7-0a1c-4564-b851-f380a22817bf", - "title": "SKUs with commitment discounts (last n days)", - "visualType": "table", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 0, - "y": 124, - "width": 14, - "height": 6 - }, - "queryRef": { - "kind": "query", - "queryId": "c4f6542d-a9cb-4284-bd4e-b9f94ad02192" - }, - "visualOptions": { - "table__enableRenderLinks": true, - "colorRulesDisabled": false, - "colorStyle": "light", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "table__renderLinks": [], - "colorRules": [] - } - }, - { - "id": "b2c9dcef-b40d-4dad-a0a3-cb5fe2c77156", - "title": "SKUs with no discounts (last n days)", - "visualType": "table", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 0, - "y": 130, - "width": 14, - "height": 6 - }, - "queryRef": { - "kind": "query", - "queryId": "00ae3917-c783-45f6-a04e-9113e4c5d445" - }, - "visualOptions": { - "table__enableRenderLinks": true, - "colorRulesDisabled": false, - "colorStyle": "light", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "table__renderLinks": [], - "colorRules": [] - } - }, - { - "id": "a3d814b8-ff20-4919-ae2a-f1caec997edf", - "title": "Most used SKUs (last n days)", - "visualType": "bar", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 14, - "y": 118, - "width": 8, - "height": 18 - }, - "queryRef": { - "kind": "query", - "queryId": "d8a4634e-7ebe-47ce-83b5-ff7f50f6bee6" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": true, - "legendLocation": "bottom", - "xColumnTitle": "", - "xColumn": "SKU", - "yColumns": [ - "EffectiveCost" - ], - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "18517864-8a43-4f0c-80e3-863554f1b189", - "title": "", - "visualType": "markdownCard", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 167, - "width": 22, - "height": 4 - }, - "markdownText": "# Purchases\r\nCosts from commitment discount, Marketplace, and other purchases\r\n\r\n\u2b06\ufe0f [Top](?tile=c0bd4bc4-a910-4a74-89a9-d4ae0f995563)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.Summary.Purchases)", - "visualOptions": {} - }, - { - "id": "4383c893-8839-4678-88c6-9a6393595d0a", - "title": "Purchases", - "visualType": "table", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 0, - "y": 171, - "width": 15, - "height": 7 - }, - "queryRef": { - "kind": "query", - "queryId": "98c9b9b5-bebf-41f8-8319-5fa2523f9dd0" - }, - "visualOptions": { - "table__enableRenderLinks": true, - "colorRulesDisabled": false, - "colorStyle": "light", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "table__renderLinks": [], - "colorRules": [] - } - }, - { - "id": "84da605b-6a0d-4a2a-aac3-a4160f9de10d", - "title": "Purchases", - "visualType": "table", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 0, - "y": 107, - "width": 22, - "height": 7 - }, - "queryRef": { - "kind": "query", - "queryId": "35a3a2b3-4ab0-4449-96f7-c78db789089e" - }, - "visualOptions": { - "table__enableRenderLinks": true, - "colorRulesDisabled": false, - "colorStyle": "light", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "table__renderLinks": [], - "colorRules": [] - } - }, - { - "id": "ea97e18e-5253-4a9d-9eed-0f2fc45b3c5b", - "title": "Cost summary", - "description": "List cost is based on retail prices without any discounts, contracted cost includes negotiated discounts, and effective cost includes commitment and negotiated discounts.", - "visualType": "bar", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 6, - "y": 18, - "width": 8, - "height": 5 - }, - "queryRef": { - "kind": "query", - "queryId": "5ff29428-de83-4a2c-8f86-d8beebe68750" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": false, - "legendLocation": "bottom", - "xColumnTitle": "", - "xColumn": null, - "yColumns": null, - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "d9959281-dc89-44fc-a2b3-b4f5227e0179", - "title": "Savings summary", - "description": "Negotiated discounts are based on your contractual agreement and represented in the \"contracted cost\". Commitment discounts are based on reservations and savings plans and represented in the \"effective cost\".", - "visualType": "bar", - "pageId": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "layout": { - "x": 14, - "y": 18, - "width": 8, - "height": 5 - }, - "queryRef": { - "kind": "query", - "queryId": "f5f240a8-a818-4f27-bdfb-e96fcfe433bd" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": false, - "legendLocation": "bottom", - "xColumnTitle": "", - "xColumn": null, - "yColumns": null, - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "cad5d666-02e8-4319-bbad-1bdf46ca2961", - "title": "", - "visualType": "markdownCard", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 0, - "y": 63, - "width": 22, - "height": 4 - }, - "markdownText": "# Commitment discounts\r\nCommitment discount usage and costs\r\n\r\n\u2b06\ufe0f [Top](?tile=c0bd4bc4-a910-4a74-89a9-d4ae0f995563)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.RateOptimization.CommitmentDiscounts)", - "visualOptions": {} - }, - { - "id": "36c1dede-49c9-4ae3-a19a-a230f87e9197", - "title": "Commitment discount usage (last n days)", - "visualType": "table", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 0, - "y": 67, - "width": 13, - "height": 6 - }, - "queryRef": { - "kind": "query", - "queryId": "d7f31381-ba44-46a1-ad3e-dc6b8826706d" - }, - "visualOptions": { - "table__enableRenderLinks": true, - "colorRulesDisabled": false, - "colorStyle": "light", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "table__renderLinks": [], - "colorRules": [] - } - }, - { - "id": "9513d9d7-be4a-468c-aed0-f0076c9bf3c4", - "title": "Savings summary (last n months)", - "visualType": "multistat", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 0, - "y": 8, - "width": 15, - "height": 9 - }, - "queryRef": { - "kind": "query", - "queryId": "e17346d1-0227-4aa4-8765-f9b4bf43bed5" - }, - "visualOptions": { - "multiStat__textSize": "auto", - "multiStat__valueColumn": "Value", - "colorRulesDisabled": false, - "colorStyle": "light", - "multiStat__displayOrientation": "horizontal", - "multiStat__labelColumn": "Label", - "multiStat__slot": { - "width": 5, - "height": 3 - }, - "colorRules": [ - { - "id": "a45d5307-a815-4baa-9743-d142253dc699", - "ruleType": "colorByCondition", - "applyToColumn": null, - "hideText": false, - "applyTo": "cells", - "conditions": [ - { - "operator": "==", - "column": "Type", - "values": [ - "List" - ] - } - ], - "chainingOperator": "and", - "colorStyle": "light", - "color": "red", - "tag": "List cost", - "icon": null, - "ruleName": "List cost", - "visualType": "multistat" - }, - { - "id": "a1307a22-3b48-48a9-9c1b-ef6b6a228796", - "ruleType": "colorByCondition", - "applyToColumn": null, - "hideText": false, - "applyTo": "cells", - "conditions": [ - { - "operator": "==", - "column": "Type", - "values": [ - "Contracted" - ] - } - ], - "chainingOperator": "and", - "colorStyle": "light", - "color": "yellow", - "tag": "Contracted cost", - "icon": null, - "ruleName": "Contracted cost", - "visualType": "multistat" - }, - { - "id": "65c6cac8-64f6-4029-bca0-b09dbfef9516", - "ruleType": "colorByCondition", - "applyToColumn": null, - "hideText": false, - "applyTo": "cells", - "conditions": [ - { - "operator": "==", - "column": "Type", - "values": [ - "Effective" - ] - } - ], - "chainingOperator": "and", - "colorStyle": "light", - "color": "green", - "tag": "Effective cost", - "icon": null, - "ruleName": "Effective cost", - "visualType": "multistat" - }, - { - "id": "33f99591-c8a6-4aae-be3f-a613bbc85c28", - "ruleType": "colorByCondition", - "applyToColumn": null, - "hideText": false, - "applyTo": "cells", - "conditions": [ - { - "operator": "==", - "column": "Type", - "values": [ - "PartialSavings" - ] - } - ], - "chainingOperator": "and", - "colorStyle": "bold", - "color": "blue", - "tag": "Partial savings", - "icon": null, - "ruleName": "Partial savings", - "visualType": "multistat" - }, - { - "id": "55430568-c3ef-4ff7-84b9-f3f82fe1a05a", - "ruleType": "colorByCondition", - "applyToColumn": null, - "hideText": false, - "applyTo": "cells", - "conditions": [ - { - "operator": "==", - "column": "Type", - "values": [ - "TotalSavings" - ] - } - ], - "chainingOperator": "and", - "colorStyle": "bold", - "color": "green", - "tag": "Total savings", - "icon": null, - "ruleName": "Total savings", - "visualType": "multistat" - } - ] - } - }, - { - "id": "9aec9f5c-f7cc-48da-8f0d-303137623945", - "title": "Savings summary (last n days)", - "visualType": "multistat", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 0, - "y": 44, - "width": 15, - "height": 9 - }, - "queryRef": { - "kind": "query", - "queryId": "f1ef29df-7a1d-4dd0-8619-0c0164707b31" - }, - "visualOptions": { - "multiStat__textSize": "auto", - "multiStat__valueColumn": "Value", - "colorRulesDisabled": false, - "colorStyle": "light", - "multiStat__displayOrientation": "horizontal", - "multiStat__labelColumn": "Label", - "multiStat__slot": { - "width": 5, - "height": 3 - }, - "colorRules": [ - { - "id": "a45d5307-a815-4baa-9743-d142253dc699", - "ruleType": "colorByCondition", - "applyToColumn": null, - "hideText": false, - "applyTo": "cells", - "conditions": [ - { - "operator": "==", - "column": "Type", - "values": [ - "List" - ] - } - ], - "chainingOperator": "and", - "colorStyle": "light", - "color": "red", - "tag": "List cost", - "icon": null, - "ruleName": "List cost", - "visualType": "multistat" - }, - { - "id": "a1307a22-3b48-48a9-9c1b-ef6b6a228796", - "ruleType": "colorByCondition", - "applyToColumn": null, - "hideText": false, - "applyTo": "cells", - "conditions": [ - { - "operator": "==", - "column": "Type", - "values": [ - "Contracted" - ] - } - ], - "chainingOperator": "and", - "colorStyle": "light", - "color": "yellow", - "tag": "Contracted cost", - "icon": null, - "ruleName": "Contracted cost", - "visualType": "multistat" - }, - { - "id": "65c6cac8-64f6-4029-bca0-b09dbfef9516", - "ruleType": "colorByCondition", - "applyToColumn": null, - "hideText": false, - "applyTo": "cells", - "conditions": [ - { - "operator": "==", - "column": "Type", - "values": [ - "Effective" - ] - } - ], - "chainingOperator": "and", - "colorStyle": "light", - "color": "green", - "tag": "Effective cost", - "icon": null, - "ruleName": "Effective cost", - "visualType": "multistat" - }, - { - "id": "33f99591-c8a6-4aae-be3f-a613bbc85c28", - "ruleType": "colorByCondition", - "applyToColumn": null, - "hideText": false, - "applyTo": "cells", - "conditions": [ - { - "operator": "==", - "column": "Type", - "values": [ - "PartialSavings" - ] - } - ], - "chainingOperator": "and", - "colorStyle": "bold", - "color": "blue", - "tag": "Partial savings", - "icon": null, - "ruleName": "Partial savings", - "visualType": "multistat" - }, - { - "id": "55430568-c3ef-4ff7-84b9-f3f82fe1a05a", - "ruleType": "colorByCondition", - "applyToColumn": null, - "hideText": false, - "applyTo": "cells", - "conditions": [ - { - "operator": "==", - "column": "Type", - "values": [ - "TotalSavings" - ] - } - ], - "chainingOperator": "and", - "colorStyle": "bold", - "color": "green", - "tag": "Total savings", - "icon": null, - "ruleName": "Total savings", - "visualType": "multistat" - } - ] - } - }, - { - "id": "e8018da4-7269-4e91-9f32-4bcf390746af", - "title": "Commitment discount breakdown (last n months)", - "visualType": "multistat", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 15, - "y": 8, - "width": 7, - "height": 9 - }, - "queryRef": { - "kind": "query", - "queryId": "7d0c2c1f-338b-4534-b173-36b284779131" - }, - "visualOptions": { - "multiStat__textSize": "auto", - "multiStat__valueColumn": "Value", - "colorRulesDisabled": true, - "colorStyle": "light", - "multiStat__displayOrientation": "horizontal", - "multiStat__labelColumn": "Label", - "multiStat__slot": { - "width": 2, - "height": 2 - }, - "colorRules": [] - } - }, - { - "id": "8dd83ee9-6692-4b94-a8c3-68caa7adbb64", - "title": "Commitment discount breakdown (last n days)", - "visualType": "multistat", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 15, - "y": 44, - "width": 7, - "height": 9 - }, - "queryRef": { - "kind": "query", - "queryId": "84ecad69-79ac-45b9-a8af-60be28dcc748" - }, - "visualOptions": { - "multiStat__textSize": "auto", - "multiStat__valueColumn": "Value", - "colorRulesDisabled": true, - "colorStyle": "light", - "multiStat__displayOrientation": "horizontal", - "multiStat__labelColumn": "Label", - "multiStat__slot": { - "width": 2, - "height": 2 - }, - "colorRules": [] - } - }, - { - "id": "e69fe03b-606c-4b93-b9b5-6f1048b24deb", - "title": "Monthly trend by service category", - "visualType": "table", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 0, - "y": 34, - "width": 22, - "height": 8 - }, - "queryRef": { - "kind": "query", - "queryId": "85320475-730b-411f-98d5-fbc345bf5e68" - }, - "visualOptions": { - "table__enableRenderLinks": true, - "colorRulesDisabled": false, - "colorStyle": "light", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "table__renderLinks": [], - "colorRules": [] - } - }, - { - "id": "df06246e-5feb-4fd1-bcb0-c04448ac4c80", - "title": "Monthly savings trend", - "visualType": "stackedcolumn", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 0, - "y": 21, - "width": 15, - "height": 6 - }, - "queryRef": { - "kind": "query", - "queryId": "5bbb5369-ac95-45fc-853c-a6a2ce6a9e7b" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": false, - "legendLocation": "right", - "xColumnTitle": "", - "xColumn": null, - "yColumns": null, - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "9", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "fa177e0f-9a74-4475-b64d-a399224e96e0", - "title": "", - "visualType": "markdownCard", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 0, - "y": 2, - "width": 22, - "height": 4 - }, - "markdownText": "# Savings\r\nSummary of negotiated and commitment discount savings\r\n\r\n\u2b06\ufe0f [Top](?tile=c0bd4bc4-a910-4a74-89a9-d4ae0f995563)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.RateOptimization.Savings)", - "visualOptions": {} - }, - { - "id": "8f92a18b-9000-48f8-952a-5f21c535105c", - "title": "Total savings (last n months)", - "visualType": "pie", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 15, - "y": 17, - "width": 7, - "height": 10 - }, - "queryRef": { - "kind": "query", - "queryId": "93f0eb9f-fa4d-4c54-96c2-8c1d3387d13a" - }, - "visualOptions": { - "hideLegend": false, - "legendLocation": "bottom", - "xColumn": "Type", - "yColumns": null, - "seriesColumns": null, - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "labelDisabled": false, - "pie__label": [ - "name", - "percentage" - ], - "tooltipDisabled": false, - "pie__tooltip": [ - "name", - "percentage", - "value" - ], - "pie__orderBy": "size", - "pie__kind": "pie", - "pie__topNSlices": null, - "crossFilter": [], - "drillthrough": [] - } - }, - { - "id": "b28fdb79-8d7a-414a-9da4-75ab9bffe134", - "title": "Daily savings trend", - "visualType": "stackedcolumn", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 0, - "y": 57, - "width": 15, - "height": 6 - }, - "queryRef": { - "kind": "query", - "queryId": "5c442903-65b0-4b53-9e7c-3ea9c1af7be5" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": false, - "legendLocation": "right", - "xColumnTitle": "", - "xColumn": null, - "yColumns": null, - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "9", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "088944a4-6cc6-4320-a8eb-82d35ee5b5d6", - "title": "Total savings (last n days)", - "visualType": "pie", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 15, - "y": 53, - "width": 7, - "height": 10 - }, - "queryRef": { - "kind": "query", - "queryId": "9a9ee4b0-a37d-475f-bf1d-f51573243491" - }, - "visualOptions": { - "hideLegend": false, - "legendLocation": "bottom", - "xColumn": "Type", - "yColumns": null, - "seriesColumns": null, - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "labelDisabled": false, - "pie__label": [ - "name", - "percentage" - ], - "tooltipDisabled": false, - "pie__tooltip": [ - "name", - "percentage", - "value" - ], - "pie__orderBy": "size", - "pie__kind": "pie", - "pie__topNSlices": null, - "crossFilter": [], - "drillthrough": [] - } - }, - { - "id": "bd461109-e42e-4329-a4b9-16a83c1ad9b3", - "title": "", - "visualType": "markdownCard", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 0, - "y": 6, - "width": 22, - "height": 2 - }, - "markdownText": "## Monthly trend (last n months)", - "visualOptions": {} - }, - { - "id": "40a4e89e-e521-4fb1-bdca-95166b7c9267", - "title": "", - "visualType": "markdownCard", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 0, - "y": 42, - "width": 22, - "height": 2 - }, - "markdownText": "## Daily trend (last n days)", - "visualOptions": {} - }, - { - "id": "cee8e33c-bb08-4f48-9c8e-c785985d16e5", - "title": "", - "visualType": "markdownCard", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 0, - "y": 73, - "width": 22, - "height": 2 - }, - "markdownText": "## Subscription chargeback (last n months)", - "visualOptions": {} - }, - { - "id": "fad3f76a-1c81-4c4f-8787-99da226100d1", - "title": "", - "visualType": "markdownCard", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 0, - "y": 83, - "width": 22, - "height": 2 - }, - "markdownText": "## Resource group chargeback (last n months)", - "visualOptions": {} - }, - { - "id": "b2c319c7-0ef0-477b-a80f-9b6e022b7e5e", - "title": "Commitment discount chargeback by resource group", - "visualType": "table", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 0, - "y": 85, - "width": 22, - "height": 8 - }, - "queryRef": { - "kind": "query", - "queryId": "377e3693-0738-45d7-97d9-4b6e71ec5b36" - }, - "visualOptions": { - "table__enableRenderLinks": true, - "colorRulesDisabled": false, - "colorStyle": "light", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "table__renderLinks": [], - "colorRules": [] - } - }, - { - "id": "e22f49e5-766c-4b35-8b4c-b99bef142d42", - "title": "Commitment discount chargeback by subscription", - "visualType": "table", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 0, - "y": 75, - "width": 22, - "height": 8 - }, - "queryRef": { - "kind": "query", - "queryId": "13cad52d-91e7-4ab4-aad5-4aae21c1019a" - }, - "visualOptions": { - "table__enableRenderLinks": true, - "colorRulesDisabled": false, - "colorStyle": "light", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "table__renderLinks": [], - "colorRules": [] - } - }, - { - "id": "c7e80308-bbb1-4aa8-a3d7-ce9eb0145f92", - "title": "Commitment discount usage by resource (last n days)", - "visualType": "table", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 0, - "y": 95, - "width": 22, - "height": 8 - }, - "queryRef": { - "kind": "query", - "queryId": "dcf69b47-233e-4bc4-b914-3f6f09f4cb82" - }, - "visualOptions": { - "table__enableRenderLinks": true, - "colorRulesDisabled": false, - "colorStyle": "light", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "table__renderLinks": [], - "colorRules": [] - } - }, - { - "id": "6e4a3743-e023-4fbc-a34d-43c08f201c6e", - "title": "", - "visualType": "markdownCard", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 0, - "y": 93, - "width": 22, - "height": 2 - }, - "markdownText": "## Resources usage (last n days)", - "visualOptions": {} - }, - { - "id": "ab300a6a-3bbd-450b-942c-7f30113cba15", - "title": "On this page", - "visualType": "markdownCard", - "pageId": "9e099251-9658-48da-b416-80422a2a47c7", - "layout": { - "x": 0, - "y": 0, - "width": 22, - "height": 2 - }, - "markdownText": "  [FinOps hubs](?tile=1e089e2c-d94a-4bfa-95e7-5f8b1249c982)\r\n\u2022 [Ingested data](?tile=10d7d4d3-0201-4cdf-977b-4814152fda24)\r\n\u2022 [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.DataIngestion.Nav)", - "visualOptions": {} - }, - { - "id": "1e089e2c-d94a-4bfa-95e7-5f8b1249c982", - "title": "", - "visualType": "markdownCard", - "pageId": "9e099251-9658-48da-b416-80422a2a47c7", - "layout": { - "x": 0, - "y": 5, - "width": 22, - "height": 4 - }, - "markdownText": "# FinOps hubs\r\nSummary of the cost and usage of FinOps hubs infrastructure\r\n\r\n\u2b06\ufe0f [Top](?tile=ab300a6a-3bbd-450b-942c-7f30113cba15)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.DataIngestion.FinOpsHubs)", - "visualOptions": {} - }, - { - "id": "05764729-51e4-4048-8cf8-2ac56eb3e643", - "title": "Monthly trend by hub instance", - "visualType": "stackedcolumn", - "pageId": "9e099251-9658-48da-b416-80422a2a47c7", - "layout": { - "x": 0, - "y": 13, - "width": 22, - "height": 5 - }, - "queryRef": { - "kind": "query", - "queryId": "d7a9ff96-da17-4826-9fb5-b23f4c7b938d" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": true, - "legendLocation": "right", - "xColumnTitle": "", - "xColumn": null, - "yColumns": null, - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "9", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "37a8bef0-7dfb-42b9-aeab-55f8c9c178b8", - "title": "Daily trend by hub instance", - "visualType": "stackedcolumn", - "pageId": "9e099251-9658-48da-b416-80422a2a47c7", - "layout": { - "x": 0, - "y": 18, - "width": 22, - "height": 5 - }, - "queryRef": { - "kind": "query", - "queryId": "13813a00-e634-4428-9eac-ea255fd1eaca" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": true, - "legendLocation": "right", - "xColumnTitle": "", - "xColumn": null, - "yColumns": null, - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "9", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "c48e07c5-f0a5-43e2-8a13-ee43ed1b92f9", - "title": "Monthly trend by hub instance", - "visualType": "table", - "pageId": "9e099251-9658-48da-b416-80422a2a47c7", - "layout": { - "x": 0, - "y": 23, - "width": 22, - "height": 8 - }, - "queryRef": { - "kind": "query", - "queryId": "62bbceee-c089-45db-822c-0b4745358aa4" - }, - "visualOptions": { - "table__enableRenderLinks": true, - "colorRulesDisabled": false, - "colorStyle": "light", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "table__renderLinks": [], - "colorRules": [] - } - }, - { - "id": "10d7d4d3-0201-4cdf-977b-4814152fda24", - "title": "", - "visualType": "markdownCard", - "pageId": "9e099251-9658-48da-b416-80422a2a47c7", - "layout": { - "x": 0, - "y": 31, - "width": 22, - "height": 4 - }, - "markdownText": "# Ingested data\r\nSummary of the data that's been ingested into FinOps hubs\r\n\r\n\u2b06\ufe0f [Top](?tile=ab300a6a-3bbd-450b-942c-7f30113cba15)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.DataIngestion.IngestedData)", - "visualOptions": {} - }, - { - "id": "0b192258-9639-4bd1-bffb-f0fbdcd14d1f", - "title": "Ingested data by table", - "visualType": "stackedcolumn", - "pageId": "9e099251-9658-48da-b416-80422a2a47c7", - "layout": { - "x": 0, - "y": 38, - "width": 22, - "height": 5 - }, - "queryRef": { - "kind": "query", - "queryId": "b98c9ed7-47b3-41c1-a596-fd9d32725b33" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": false, - "legendLocation": "right", - "xColumnTitle": "", - "xColumn": null, - "yColumns": [ - "Rows" - ], - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "9", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "db78922e-5215-4b93-9762-212eb0e32bf3", - "title": "Ingested cost data by scope", - "visualType": "stackedcolumn", - "pageId": "9e099251-9658-48da-b416-80422a2a47c7", - "layout": { - "x": 0, - "y": 43, - "width": 22, - "height": 5 - }, - "queryRef": { - "kind": "query", - "queryId": "187e22fe-3d78-45a2-a2bf-090ece144fd1" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": false, - "legendLocation": "right", - "xColumnTitle": "", - "xColumn": null, - "yColumns": [ - "Rows" - ], - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "9", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "c636ea6c-8356-4a69-a4d6-61e6596a3c4e", - "title": "Ingested scopes", - "visualType": "card", - "pageId": "9e099251-9658-48da-b416-80422a2a47c7", - "layout": { - "x": 0, - "y": 35, - "width": 4, - "height": 3 - }, - "queryRef": { - "kind": "query", - "queryId": "d2c522dc-ef4b-4714-a473-09350e275557" - }, - "visualOptions": { - "multiStat__textSize": "auto", - "multiStat__valueColumn": null, - "colorRulesDisabled": false, - "colorStyle": "light", - "colorRules": [] - } - }, - { - "id": "379b25b8-7188-4657-b82f-dee14b344bec", - "title": "Ingested months", - "visualType": "card", - "pageId": "9e099251-9658-48da-b416-80422a2a47c7", - "layout": { - "x": 4, - "y": 35, - "width": 4, - "height": 3 - }, - "queryRef": { - "kind": "query", - "queryId": "6fe11b78-82ef-4e9a-8411-a65e5b0d8bba" - }, - "visualOptions": { - "multiStat__textSize": "auto", - "multiStat__valueColumn": null, - "colorRulesDisabled": false, - "colorStyle": "light", - "colorRules": [] - } - }, - { - "id": "71b29fd4-2999-45dc-9e4b-14152c48a4f4", - "title": "Ingested price data by scope", - "visualType": "stackedcolumn", - "pageId": "9e099251-9658-48da-b416-80422a2a47c7", - "layout": { - "x": 0, - "y": 48, - "width": 22, - "height": 5 - }, - "queryRef": { - "kind": "query", - "queryId": "b3621977-59be-4f98-a034-a94479612115" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": false, - "legendLocation": "right", - "xColumnTitle": "", - "xColumn": null, - "yColumns": [ - "Rows" - ], - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "9", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "29f1c705-131d-44b9-9bd6-368c93aedcdf", - "title": "Ingested recommendation data by scope", - "visualType": "stackedcolumn", - "pageId": "9e099251-9658-48da-b416-80422a2a47c7", - "layout": { - "x": 0, - "y": 53, - "width": 22, - "height": 5 - }, - "queryRef": { - "kind": "query", - "queryId": "4a2a77fe-e819-4059-b027-dab0a1770a0a" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": false, - "legendLocation": "right", - "xColumnTitle": "", - "xColumn": null, - "yColumns": [ - "Rows" - ], - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "9", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "47471816-9569-4ff3-816b-98fbc47b5c17", - "title": "Ingested transaction data by scope", - "visualType": "stackedcolumn", - "pageId": "9e099251-9658-48da-b416-80422a2a47c7", - "layout": { - "x": 0, - "y": 58, - "width": 22, - "height": 5 - }, - "queryRef": { - "kind": "query", - "queryId": "8f1ad1ef-6f5e-4f20-a1c4-3941e871d0cd" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": false, - "legendLocation": "right", - "xColumnTitle": "", - "xColumn": null, - "yColumns": [ - "Rows" - ], - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "9", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "fcedcd3b-cd11-42ab-8363-de9619fc2018", - "title": "Ingested commitment discount usage data by scope", - "visualType": "stackedcolumn", - "pageId": "9e099251-9658-48da-b416-80422a2a47c7", - "layout": { - "x": 0, - "y": 63, - "width": 22, - "height": 5 - }, - "queryRef": { - "kind": "query", - "queryId": "e9a9a135-6a74-4463-b69f-eac6930ff1f2" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": false, - "legendLocation": "right", - "xColumnTitle": "", - "xColumn": null, - "yColumns": [ - "Rows" - ], - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "9", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "e29123b0-320a-4d36-a5bd-14c1757af8b8", - "title": "About this domain", - "visualType": "markdownCard", - "pageId": "9e099251-9658-48da-b416-80422a2a47c7", - "layout": { - "x": 0, - "y": 2, - "width": 22, - "height": 3 - }, - "markdownText": "Data ingestion refers to the process of collecting, transforming, and organizing data from various sources into a single, easily accessible repository. With this capability, you identify data sources needed to support your FinOps practice and facilitate FinOps tasks. [Learn more](http://aka.ms/ftk/fx/ingestion)", - "visualOptions": {} - }, - { - "id": "fe3644dd-bb95-4fee-b212-5e75374b3100", - "title": "", - "visualType": "markdownCard", - "pageId": "8beab65c-f5ec-4661-bc67-37b10baffb16", - "layout": { - "x": 0, - "y": 0, - "width": 22, - "height": 5 - }, - "markdownText": "# Quantify business value\nThe **Quantify business value** domain focuses on analyzing cost, usage, and sustainability to align with organizational plans and measure the return on investment from cloud computing efforts. This domain is all about measuring and maximizing the business value each team and workload gets from the cloud to maximize future potential.\n\n[Learn more](https://learn.microsoft.com/cloud-computing/finops/framework/quantify/quantify-business-value)     [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Quantify)\n", - "visualOptions": {} - }, - { - "id": "353712a3-e590-4d4e-a041-54c4b3105c99", - "title": "", - "visualType": "markdownCard", - "pageId": "8beab65c-f5ec-4661-bc67-37b10baffb16", - "layout": { - "x": 0, - "y": 5, - "width": 4, - "height": 8 - }, - "markdownText": "### Planning and estimating\nPlanning and estimating involve predicting the cost and usage of new and existing workloads based on potential architectural changes and shifting business priorities. With this capability, you establish baseline expectations for new cloud workloads and prepare for changes to existing workloads. This capability is an important part of establishing and maintaining organizational budgets.\n\n \n\n\ud83d\udcca Coming soon\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/planning)", - "visualOptions": {} - }, - { - "id": "cbc699f4-0197-420a-a439-3c869bf48227", - "title": "", - "visualType": "markdownCard", - "pageId": "8beab65c-f5ec-4661-bc67-37b10baffb16", - "layout": { - "x": 4, - "y": 5, - "width": 4, - "height": 8 - }, - "markdownText": "### Forecasting\nForecasting involves analyzing historical trends and future plans to predict costs, understand the impact on current budgets, and influence future budgets. With this capability, you'll project future cost, usage, and carbon emissions based on historical trends. This capability is a major part of establishing accurate budgets for each team.\n\n \n\n\ud83d\udcca Coming soon\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/forecasting)", - "visualOptions": {} - }, - { - "id": "5206bf80-0c56-4d07-bedb-ab0dc15bffb5", - "title": "", - "visualType": "markdownCard", - "pageId": "8beab65c-f5ec-4661-bc67-37b10baffb16", - "layout": { - "x": 8, - "y": 5, - "width": 5, - "height": 8 - }, - "markdownText": "### Budgeting\nBudgeting is the process of monitoring and managing financial plans and limits over a specific period to control spending effectively. This capability allows you to use established estimates and forecasts to allocate funds to each team for fiscal year planning and set up alerts to ensure budget accountability. This capability plays a key role in staying within established fiscal constraints on an ongoing basis.\n\n \n\n\ud83d\udcca [View report](#5c57f940-6f0f-4e39-930b-b6e89bb758ad)\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/budgeting)", - "visualOptions": {} - }, - { - "id": "566ad60b-4c09-45e4-aa2f-e6817c181a61", - "title": "", - "visualType": "markdownCard", - "pageId": "8beab65c-f5ec-4661-bc67-37b10baffb16", - "layout": { - "x": 13, - "y": 5, - "width": 4, - "height": 8 - }, - "markdownText": "### Benchmarking\nBenchmarking is a systematic process of evaluating the performance and value of cloud services using efficiency metrics, either within an organization or against industry peers. This capability allows you to measure and track key performance indicators to identify and accelerate the adoption of successful initiatives, maximizing cloud return on investment (ROI) across teams and external organizations when applicable.\n\n \n\n\ud83d\udcca Coming soon\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/benchmarking)", - "visualOptions": {} - }, - { - "id": "6282be37-d78a-4555-91e4-9c71dee8fcb7", - "title": "", - "visualType": "markdownCard", - "pageId": "8beab65c-f5ec-4661-bc67-37b10baffb16", - "layout": { - "x": 17, - "y": 5, - "width": 5, - "height": 8 - }, - "markdownText": "### Unit economics\nUnit economics refers to the process of calculating the cost and carbon emissions of a single unit of a business that can show the business value of the cloud. With this capability, you break down cost, usage, and carbon emissions and compare that with the business value per unit to identify high- and low-value workloads. This capability helps you identify areas that can be scaled out to accelerate business value through specific technology decisions.\n\n \n\n\ud83d\udcca Coming soon\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/uniteconomics)", - "visualOptions": {} - }, - { - "id": "c19ca143-e39b-4407-9592-cbaf87a37529", - "title": "", - "visualType": "markdownCard", - "pageId": "d01a8154-8a60-4d22-96ea-54b45b1417fe", - "layout": { - "x": 0, - "y": 0, - "width": 22, - "height": 5 - }, - "markdownText": "# Optimize usage and cost\nThe **Optimize usage and cost** domain focused on designing and optimizing solutions for efficiency to ensure you get the most out of your cloud investments.\n\n[Learn more](https://learn.microsoft.com/cloud-computing/finops/framework/optimize/optimize-cloud-usage-cost)     [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Quantify)\n", - "visualOptions": {} - }, - { - "id": "4ce93188-d865-432f-8488-c5d5758d0f14", - "title": "", - "visualType": "markdownCard", - "pageId": "d01a8154-8a60-4d22-96ea-54b45b1417fe", - "layout": { - "x": 0, - "y": 5, - "width": 5, - "height": 8 - }, - "markdownText": "### Architecting for the cloud\nArchitecting for the cloud involves designing and implementing cloud infrastructure and applications to optimize cost, performance, scalability, and reliability. Simultaneously, you align with business objectives.\n\nWith this capability, you ensure solutions are designed for efficiency in early development and migration stages to ensure they follow proven practices to maximize value and reduce waste. This capability is an important step towards reducing post-deployment optimization efforts, which often require much time and effort.\n\n\ud83d\udcca Coming soon\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/architecting)", - "visualOptions": {} - }, - { - "id": "87e197cb-12de-4eca-86e6-ee1d469c8990", - "title": "", - "visualType": "markdownCard", - "pageId": "d01a8154-8a60-4d22-96ea-54b45b1417fe", - "layout": { - "x": 5, - "y": 5, - "width": 5, - "height": 8 - }, - "markdownText": "### Workload optimization\nWorkload optimization refers to the process of ensuring cloud services are utilized and tuned to maximize business value and minimize wasteful usage and spending. With this capability, you analyze cost, usage, and carbon emissions for cloud workloads to identify opportunities to maximize efficiency. This capability usually starts with recommendations and expands into more nuanced optimization efforts based on detailed resource utilization analysis. This capability can be time and effort intensive as each cloud service has its different optimization opportunities.\n\n\ud83d\udcca Coming soon\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/workloads)", - "visualOptions": {} - }, - { - "id": "a60dda7f-8f14-429d-b7ff-a2db94feb809", - "title": "", - "visualType": "markdownCard", - "pageId": "d01a8154-8a60-4d22-96ea-54b45b1417fe", - "layout": { - "x": 10, - "y": 5, - "width": 4, - "height": 8 - }, - "markdownText": "### Rate optimization\nRate optimization is the practice of obtaining reduced rates on cloud services, often by committing to a certain level of usage or spend over a specific period. With this capability, you analyze SKU usage patterns to inform rate negotiation efforts and commitment discount planning. This capability can lead to impressive discounts based on consistent usage patterns.\n\n\ud83d\udcca [View report](#306fef9a-c760-4559-a326-7c25d196b616)\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/rates)", - "visualOptions": {} - }, - { - "id": "aa339c66-412d-4619-9737-06d5e4c282fe", - "title": "", - "visualType": "markdownCard", - "pageId": "d01a8154-8a60-4d22-96ea-54b45b1417fe", - "layout": { - "x": 14, - "y": 5, - "width": 4, - "height": 8 - }, - "markdownText": "### Licensing and SaaS\nLicensing & SaaS involves managing and optimizing the financial aspects of software licensing agreements and Software as a Service (SaaS) investments within an organization. With this capability, you track and manage software licenses and prepaid SaaS products to ensure they're being fully utilized. This capability also covers bring your own license offers, like Azure Hybrid Benefit.\n\n\ud83d\udcca [View report](#a4ec1d55-5b6e-49af-bb9e-4f3d136cdf05)\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/licensing)", - "visualOptions": {} - }, - { - "id": "2ba70b12-ac64-41b9-b13f-d3c91dab6df4", - "title": "On this page", - "visualType": "markdownCard", - "pageId": "a4ec1d55-5b6e-49af-bb9e-4f3d136cdf05", - "layout": { - "x": 0, - "y": 0, - "width": 22, - "height": 2 - }, - "markdownText": "  [Hybrid Benefit](?tile=805cac8c-a189-45cb-8ba3-fbb1a06dfba0)\r\n\u2022 [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.Licensing.Nav)", - "visualOptions": {} - }, - { - "id": "805cac8c-a189-45cb-8ba3-fbb1a06dfba0", - "title": "", - "visualType": "markdownCard", - "pageId": "a4ec1d55-5b6e-49af-bb9e-4f3d136cdf05", - "layout": { - "x": 0, - "y": 5, - "width": 22, - "height": 4 - }, - "markdownText": "# Azure Hybrid Benefit\r\nSummary of the Hybrid Benefit coverage and utilization\r\n\r\n\u2b06\ufe0f [Top](?tile=2ba70b12-ac64-41b9-b13f-d3c91dab6df4)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.Licensing.HybridBenefit)", - "visualOptions": {} - }, - { - "id": "d2e4a321-9c2c-411b-8a20-19fd86dab3fe", - "title": "Hybrid Benefit summary (last n days)", - "visualType": "multistat", - "pageId": "a4ec1d55-5b6e-49af-bb9e-4f3d136cdf05", - "layout": { - "x": 0, - "y": 9, - "width": 6, - "height": 15 - }, - "queryRef": { - "kind": "query", - "queryId": "1a400b6c-9204-434d-88ff-2dc5260a95bf" - }, - "visualOptions": { - "multiStat__textSize": "auto", - "multiStat__valueColumn": "Value", - "colorRulesDisabled": true, - "colorStyle": "light", - "multiStat__displayOrientation": "vertical", - "multiStat__labelColumn": "Label", - "multiStat__slot": { - "width": 2, - "height": 3 - }, - "colorRules": [] - } - }, - { - "id": "80e1ecb4-69f6-451d-9e1c-f90d17b7f865", - "title": "Underutilized vCPU capacity (last n days)", - "visualType": "table", - "pageId": "a4ec1d55-5b6e-49af-bb9e-4f3d136cdf05", - "layout": { - "x": 6, - "y": 9, - "width": 16, - "height": 5 - }, - "queryRef": { - "kind": "query", - "queryId": "c2ad5370-f4df-4a13-ac31-2c095f88754d" - }, - "visualOptions": { - "table__enableRenderLinks": true, - "colorRulesDisabled": true, - "colorStyle": "light", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "table__renderLinks": [], - "colorRules": [] - } - }, - { - "id": "59a1756c-7586-4364-8047-604aa7d6f09a", - "title": "Fully utilized vCPU capacity (last n days)", - "visualType": "table", - "pageId": "a4ec1d55-5b6e-49af-bb9e-4f3d136cdf05", - "layout": { - "x": 6, - "y": 14, - "width": 16, - "height": 5 - }, - "queryRef": { - "kind": "query", - "queryId": "4614acfb-2288-4a71-97f7-32845a9ddcb2" - }, - "visualOptions": { - "table__enableRenderLinks": true, - "colorRulesDisabled": true, - "colorStyle": "light", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "table__renderLinks": [], - "colorRules": [] - } - }, - { - "id": "9197f822-bdf3-42af-8b43-5609c9b91f00", - "title": "Eligible resources (last n days)", - "visualType": "table", - "pageId": "a4ec1d55-5b6e-49af-bb9e-4f3d136cdf05", - "layout": { - "x": 6, - "y": 19, - "width": 16, - "height": 5 - }, - "queryRef": { - "kind": "query", - "queryId": "99d44f1f-c53a-40ee-b749-8e5557ec58c7" - }, - "visualOptions": { - "table__enableRenderLinks": true, - "colorRulesDisabled": true, - "colorStyle": "light", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "table__renderLinks": [], - "colorRules": [] - } - }, - { - "id": "d9d6d2e0-6403-418f-8aa0-1e2eb4ce2822", - "title": "", - "visualType": "markdownCard", - "pageId": "d01a8154-8a60-4d22-96ea-54b45b1417fe", - "layout": { - "x": 18, - "y": 5, - "width": 4, - "height": 8 - }, - "markdownText": "### Cloud sustainability\nCloud sustainability balances environmental and financial efficiency in cloud optimization, ensuring alignment with strategic objectives. With this capability, you focus explicitly on optimizing carbon emissions through usage and workload analysis. This capability relies heavily on understanding the carbon impact of different services across different regions to identify the right cost/carbon balance for your organizational goals.\n\n\ud83d\udcca Coming soon\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/sustainability)", - "visualOptions": {} - }, - { - "id": "43e7bf02-fa40-4a35-9b41-e45496d0404b", - "title": "", - "visualType": "markdownCard", - "pageId": "11f019cd-e0cc-4aa7-a125-4dc852eb3f10", - "layout": { - "x": 0, - "y": 0, - "width": 22, - "height": 5 - }, - "markdownText": "# Manage the FinOps practice\nThe **Manage the FinOps practice** domain focuses on establishing a clear and consistent vision of FinOps and driving cultural adoption across your organization. Other domains are focused on the FinOps tasks you perform to drive efficiency and maximize value. This domain is focused more on how you run your FinOps practice and supporting those efforts.\n\n[Learn more](https://learn.microsoft.com/cloud-computing/finops/framework/manage/manage-finops)     [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Manage)\n", - "visualOptions": {} - }, - { - "id": "9c05ec17-5d60-4ed3-bc8c-6214e8955312", - "title": "", - "visualType": "markdownCard", - "pageId": "11f019cd-e0cc-4aa7-a125-4dc852eb3f10", - "layout": { - "x": 0, - "y": 5, - "width": 6, - "height": 7 - }, - "markdownText": "### FinOps education and enablement\nFinOps education and enablement involve refers to the process of providing training, resources, and support to help individuals and teams within an organization adopt FinOps practices. This capability ensures that stakeholders across the organization have the resources they need to:\n\n- Understand FinOps\n- Understand how it can help them achieve their goals\n- Understand how to perform FinOps tasks using the available tools and services.\n\n\ud83e\udded [Open FinOps guide](https://aka.ms/finops/guide)\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/education)", - "visualOptions": {} - }, - { - "id": "abf6cf66-48b8-4d86-826d-64ebc9ddcc90", - "title": "", - "visualType": "markdownCard", - "pageId": "11f019cd-e0cc-4aa7-a125-4dc852eb3f10", - "layout": { - "x": 6, - "y": 5, - "width": 6, - "height": 7 - }, - "markdownText": "### FinOps practice operations\nFinOps practice operations refer to the process of:\n\n- Building and managing a robust FinOps team\n- Defining clear cross-functional responsibilities\n- Integrating FinOps practices into organizational processes to manage cloud cost and usage effectively.\n\nWith this capability, you establish and manage your FinOps practice, focusing on supporting the needs of stakeholders across the organization. This capability is pivotal to driving organizational adoption.\n\n\ud83d\udcca Coming soon\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/operations)", - "visualOptions": {} - }, - { - "id": "20c47c65-228a-4c70-9d7c-418ad1c8c5cf", - "title": "", - "visualType": "markdownCard", - "pageId": "11f019cd-e0cc-4aa7-a125-4dc852eb3f10", - "layout": { - "x": 12, - "y": 5, - "width": 5, - "height": 7 - }, - "markdownText": "### Onboarding workloads\nOnboarding workloads refers to the process of bringing new and existing applications into the cloud based on their financial and technical feasibility. With this capability, you:\n\n- Understand requirements for new workloads\n- Identify if there are any architectural requirements or changes that should be considered\n- Validate that the budget is available to support their needs\n\n\ud83d\udcca Coming soon\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/onboarding)", - "visualOptions": {} - }, - { - "id": "31c7508b-6298-4257-9c35-a7d9cd7c84e2", - "title": "", - "visualType": "markdownCard", - "pageId": "11f019cd-e0cc-4aa7-a125-4dc852eb3f10", - "layout": { - "x": 17, - "y": 5, - "width": 5, - "height": 7 - }, - "markdownText": "### Cloud policy and governance\nCloud policy and governance refer to the process of defining, implementing, and monitoring a framework of rules that guide an organization's FinOps efforts. With this capability, you identify and implement policies to support organizational goals by promoting or limiting the use of:\n\n- Specific SKUs\n- Resource configurations\n- Other practices that might affect cost, usage, and carbon growth\n\n\ud83d\udcca Coming soon\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/policy)", - "visualOptions": {} - }, - { - "id": "a000abed-3a39-4675-806c-9423ff344d2e", - "title": "", - "visualType": "markdownCard", - "pageId": "11f019cd-e0cc-4aa7-a125-4dc852eb3f10", - "layout": { - "x": 0, - "y": 12, - "width": 6, - "height": 7 - }, - "markdownText": "### Invoicing and chargeback\nInvoicing and chargeback refer to the process of receiving, reconciling, and paying provider invoices, and then billing internal teams for their respective cloud costs using existing internal finance tools and processes. With this capability, you break down invoices and cross-charge portions of these invoices to the teams responsible for the charges.\n\n\ud83d\udcca [View report](#f416685d-f559-4514-8e45-5e0e09aec286)\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/chargeback)", - "visualOptions": {} - }, - { - "id": "fc3beac9-e524-4262-bdc5-39f7168e11ff", - "title": "", - "visualType": "markdownCard", - "pageId": "11f019cd-e0cc-4aa7-a125-4dc852eb3f10", - "layout": { - "x": 6, - "y": 12, - "width": 6, - "height": 7 - }, - "markdownText": "### FinOps assessment\nFinOps assessment refers to the process of measuring the effectiveness of a FinOps practice, aligning activities with organizational goals, and identifying areas for improvement to drive FinOps maturity. With this capability, you assess and benchmark the maturity of different teams to identify focus areas for key milestones. This capability is important to set clear, quantitative expectations with teams that are backed up with recommendations to drive targeted improvements and prioritize organizational goals.\n\n\ud83d\udccb [Start a FinOps review](https://aka.ms/finops/review)\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/assessment)", - "visualOptions": {} - }, - { - "id": "d525fe75-1afb-43f3-89c3-b8ccfbc8f570", - "title": "", - "visualType": "markdownCard", - "pageId": "11f019cd-e0cc-4aa7-a125-4dc852eb3f10", - "layout": { - "x": 12, - "y": 12, - "width": 5, - "height": 7 - }, - "markdownText": "### FinOps tools and services\nFinOps tools and services encapsulate identifying, configuring, and integrating tools and services that meet the needs of FinOps capabilities. They also enable the FinOps practice at scale throughout the organization. With this capability, you identify the tools and services needed to accelerate FinOps tasks and support your FinOps practice.\n\n\ud83d\udcca Coming soon\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/tools)", - "visualOptions": {} - }, - { - "id": "c2452fca-3ba3-40f6-9ccb-94ea6d075a12", - "title": "", - "visualType": "markdownCard", - "pageId": "11f019cd-e0cc-4aa7-a125-4dc852eb3f10", - "layout": { - "x": 17, - "y": 12, - "width": 5, - "height": 7 - }, - "markdownText": "### Intersecting frameworks\nFinOps and intersecting frameworks refers to integrating FinOps practices with other frameworks and methodologies used by an organization. With this capability, you'll partner with stakeholders in other disciplines to understand each other's others priorities to achieve an acceptable balance compared to cost, usage, and carbon efficiency goals.\n\n\ud83d\udcca Coming soon\n   \n\ud83d\udcd7 [Learn more](http://aka.ms/ftk/fx/frameworks)", - "visualOptions": {} - }, - { - "id": "876b81e5-4f71-4c69-bd62-8d470f50bf31", - "title": "About this capability", - "visualType": "markdownCard", - "pageId": "f416685d-f559-4514-8e45-5e0e09aec286", - "layout": { - "x": 0, - "y": 0, - "width": 21, - "height": 3 - }, - "markdownText": "Invoicing and chargeback refer to the process of receiving, reconciling, and paying provider invoices, and then billing internal teams for their respective cloud costs using existing internal finance tools and processes. With this capability, you break down invoices and cross-charge portions of these invoices to the teams responsible for the charges.\r\n\r\n[Learn more](http://aka.ms/ftk/fx/chargeback)     [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Manage.Invoicing)", - "visualOptions": {} - }, - { - "id": "a66f0b2b-6bce-4bfb-ad3e-13f7eabcb586", - "title": "About this capability", - "visualType": "markdownCard", - "pageId": "5c57f940-6f0f-4e39-930b-b6e89bb758ad", - "layout": { - "x": 0, - "y": 0, - "width": 22, - "height": 3 - }, - "markdownText": "Budgeting is the process of monitoring and managing financial plans and limits over a specific period to control spending effectively. This capability allows you to use established estimates and forecasts to allocate funds to each team for fiscal year planning and set up alerts to ensure budget accountability. This capability plays a key role in staying within established fiscal constraints on an ongoing basis.\r\n\r\n[Learn more](http://aka.ms/ftk/fx/budgeting)     [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.Quantify.Budgeting.Nav)", - "visualOptions": {} - }, - { - "id": "08c92e20-6563-464b-893f-81d80dd3caa1", - "title": "About this capability", - "visualType": "markdownCard", - "pageId": "5838c918-4541-44fe-90d2-77306ef1e241", - "layout": { - "x": 0, - "y": 0, - "width": 16, - "height": 4 - }, - "markdownText": "Anomaly management refers to the practice of detecting and addressing abnormal or unexpected cost and usage patterns in a timely manner. With this capability, you identify, triage, and address unexpected changes in cost, usage, and carbon. This capability is especially important with usage-based billing and sustainability models, where small fluctuations in usage can result in large fiscal and environmental impact.\n\n[Learn more](http://aka.ms/ftk/fx/anomalies)     [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.Anomalies.Nav)\n", - "visualOptions": {} - }, - { - "id": "6fb4c25e-ae84-495b-88ae-6dd2035500d1", - "title": "Forecast (next n days)", - "visualType": "timechart", - "pageId": "5838c918-4541-44fe-90d2-77306ef1e241", - "layout": { - "x": 0, - "y": 33, - "width": 9, - "height": 7 - }, - "queryRef": { - "kind": "query", - "queryId": "5fd3ea2a-8882-4438-9103-fa3945240604" - }, - "visualOptions": { - "multipleYAxes": { - "base": { - "id": "-1", - "label": "", - "columns": [], - "yAxisMaximumValue": null, - "yAxisMinimumValue": null, - "yAxisScale": "linear", - "horizontalLines": [] - }, - "additional": [], - "showMultiplePanels": false - }, - "hideLegend": false, - "legendLocation": "bottom", - "xColumnTitle": "", - "xColumn": null, - "yColumns": null, - "seriesColumns": null, - "xAxisScale": "linear", - "verticalLine": "", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "selectedDataOnLoad": { - "all": true, - "limit": 10 - }, - "dataPointsTooltip": { - "all": false, - "limit": 1 - } - } - }, - { - "id": "961b8980-213f-4bc7-9512-8a08614229e0", - "title": "Effective Savings Rate (last n months)", - "visualType": "multistat", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 0, - "y": 17, - "width": 15, - "height": 4 - }, - "queryRef": { - "kind": "query", - "queryId": "e1bc2d51-44af-4dd9-8b0d-a71088b551f5" - }, - "visualOptions": { - "multiStat__textSize": "auto", - "multiStat__valueColumn": "Value", - "colorRulesDisabled": false, - "colorStyle": "light", - "multiStat__displayOrientation": "horizontal", - "multiStat__labelColumn": "Label", - "multiStat__slot": { - "width": 5, - "height": 1 - }, - "colorRules": [ - { - "id": "a45d5307-a815-4baa-9743-d142253dc699", - "ruleType": "colorByCondition", - "applyToColumn": null, - "hideText": false, - "applyTo": "cells", - "conditions": [ - { - "operator": "==", - "column": "Type", - "values": [ - "List" - ] - } - ], - "chainingOperator": "and", - "colorStyle": "light", - "color": "red", - "tag": "List cost", - "icon": null, - "ruleName": "List cost", - "visualType": "multistat" - }, - { - "id": "a1307a22-3b48-48a9-9c1b-ef6b6a228796", - "ruleType": "colorByCondition", - "applyToColumn": null, - "hideText": false, - "applyTo": "cells", - "conditions": [ - { - "operator": "==", - "column": "Type", - "values": [ - "Contracted" - ] - } - ], - "chainingOperator": "and", - "colorStyle": "light", - "color": "yellow", - "tag": "Contracted cost", - "icon": null, - "ruleName": "Contracted cost", - "visualType": "multistat" - }, - { - "id": "65c6cac8-64f6-4029-bca0-b09dbfef9516", - "ruleType": "colorByCondition", - "applyToColumn": null, - "hideText": false, - "applyTo": "cells", - "conditions": [ - { - "operator": "==", - "column": "Type", - "values": [ - "Effective" - ] - } - ], - "chainingOperator": "and", - "colorStyle": "light", - "color": "green", - "tag": "Effective cost", - "icon": null, - "ruleName": "Effective cost", - "visualType": "multistat" - }, - { - "id": "33f99591-c8a6-4aae-be3f-a613bbc85c28", - "ruleType": "colorByCondition", - "applyToColumn": null, - "hideText": false, - "applyTo": "cells", - "conditions": [ - { - "operator": "==", - "column": "Type", - "values": [ - "PartialSavings" - ] - } - ], - "chainingOperator": "and", - "colorStyle": "bold", - "color": "blue", - "tag": "Partial savings", - "icon": null, - "ruleName": "Partial savings", - "visualType": "multistat" - }, - { - "id": "55430568-c3ef-4ff7-84b9-f3f82fe1a05a", - "ruleType": "colorByCondition", - "applyToColumn": null, - "hideText": false, - "applyTo": "cells", - "conditions": [ - { - "operator": "==", - "column": "Type", - "values": [ - "TotalSavings" - ] - } - ], - "chainingOperator": "and", - "colorStyle": "bold", - "color": "green", - "tag": "Total savings", - "icon": null, - "ruleName": "Total savings", - "visualType": "multistat" - }, - { - "id": "852c731e-90ec-4438-8bb7-00be0127b802", - "ruleType": "colorByCondition", - "applyToColumn": null, - "hideText": false, - "applyTo": "cells", - "conditions": [ - { - "operator": "==", - "column": "Type", - "values": [ - "EffectiveSavingsRate" - ] - } - ], - "chainingOperator": "and", - "colorStyle": "bold", - "color": "yellow", - "tag": "ESR", - "icon": null, - "ruleName": "ESR", - "visualType": "multistat" - } - ] - } - }, - { - "id": "6367a5d7-99ca-4afc-b9d4-4c77ec259a07", - "title": "Savings breakdown by month", - "visualType": "table", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 0, - "y": 27, - "width": 10, - "height": 7 - }, - "queryRef": { - "kind": "query", - "queryId": "3b3f0a58-2d84-4e3e-bebc-3e747a7d5ede" - }, - "visualOptions": { - "table__enableRenderLinks": true, - "colorRulesDisabled": false, - "colorStyle": "light", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "table__renderLinks": [], - "colorRules": [] - } - }, - { - "id": "69e3a7e2-9477-4b63-8c86-9d66a782501b", - "title": "Effective Savings Rate (last n days)", - "visualType": "multistat", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 0, - "y": 53, - "width": 15, - "height": 4 - }, - "queryRef": { - "kind": "query", - "queryId": "cb34f22b-9370-460c-9658-3e73d220bbc7" - }, - "visualOptions": { - "multiStat__textSize": "auto", - "multiStat__valueColumn": "Value", - "colorRulesDisabled": false, - "colorStyle": "light", - "multiStat__displayOrientation": "horizontal", - "multiStat__labelColumn": "Label", - "multiStat__slot": { - "width": 5, - "height": 1 - }, - "colorRules": [ - { - "id": "a45d5307-a815-4baa-9743-d142253dc699", - "ruleType": "colorByCondition", - "applyToColumn": null, - "hideText": false, - "applyTo": "cells", - "conditions": [ - { - "operator": "==", - "column": "Type", - "values": [ - "List" - ] - } - ], - "chainingOperator": "and", - "colorStyle": "light", - "color": "red", - "tag": "List cost", - "icon": null, - "ruleName": "List cost", - "visualType": "multistat" - }, - { - "id": "a1307a22-3b48-48a9-9c1b-ef6b6a228796", - "ruleType": "colorByCondition", - "applyToColumn": null, - "hideText": false, - "applyTo": "cells", - "conditions": [ - { - "operator": "==", - "column": "Type", - "values": [ - "Contracted" - ] - } - ], - "chainingOperator": "and", - "colorStyle": "light", - "color": "yellow", - "tag": "Contracted cost", - "icon": null, - "ruleName": "Contracted cost", - "visualType": "multistat" - }, - { - "id": "65c6cac8-64f6-4029-bca0-b09dbfef9516", - "ruleType": "colorByCondition", - "applyToColumn": null, - "hideText": false, - "applyTo": "cells", - "conditions": [ - { - "operator": "==", - "column": "Type", - "values": [ - "Effective" - ] - } - ], - "chainingOperator": "and", - "colorStyle": "light", - "color": "green", - "tag": "Effective cost", - "icon": null, - "ruleName": "Effective cost", - "visualType": "multistat" - }, - { - "id": "33f99591-c8a6-4aae-be3f-a613bbc85c28", - "ruleType": "colorByCondition", - "applyToColumn": null, - "hideText": false, - "applyTo": "cells", - "conditions": [ - { - "operator": "==", - "column": "Type", - "values": [ - "PartialSavings" - ] - } - ], - "chainingOperator": "and", - "colorStyle": "bold", - "color": "blue", - "tag": "Partial savings", - "icon": null, - "ruleName": "Partial savings", - "visualType": "multistat" - }, - { - "id": "55430568-c3ef-4ff7-84b9-f3f82fe1a05a", - "ruleType": "colorByCondition", - "applyToColumn": null, - "hideText": false, - "applyTo": "cells", - "conditions": [ - { - "operator": "==", - "column": "Type", - "values": [ - "TotalSavings" - ] - } - ], - "chainingOperator": "and", - "colorStyle": "bold", - "color": "green", - "tag": "Total savings", - "icon": null, - "ruleName": "Total savings", - "visualType": "multistat" - }, - { - "id": "852c731e-90ec-4438-8bb7-00be0127b802", - "ruleType": "colorByCondition", - "applyToColumn": null, - "hideText": false, - "applyTo": "cells", - "conditions": [ - { - "operator": "==", - "column": "Type", - "values": [ - "EffectiveSavingsRate" - ] - } - ], - "chainingOperator": "and", - "colorStyle": "bold", - "color": "yellow", - "tag": "ESR", - "icon": null, - "ruleName": "ESR", - "visualType": "multistat" - } - ] - } - }, - { - "id": "fc2e27cf-57be-486f-afa5-6a93d6e9c6a7", - "title": "Ingested months", - "visualType": "multistat", - "pageId": "9e099251-9658-48da-b416-80422a2a47c7", - "layout": { - "x": 0, - "y": 9, - "width": 9, - "height": 4 - }, - "queryRef": { - "kind": "query", - "queryId": "8de47213-8327-44da-9d1b-8ba5de74c44a" - }, - "visualOptions": { - "multiStat__textSize": "auto", - "multiStat__valueColumn": "Value", - "colorRulesDisabled": false, - "colorStyle": "light", - "multiStat__displayOrientation": "horizontal", - "multiStat__labelColumn": "Label", - "multiStat__slot": { - "width": 3, - "height": 1 - }, - "colorRules": [] - } - }, - { - "id": "2e9d6a7f-6393-41ad-83df-51faddbe3b8e", - "title": "Effective cost breakdown by month", - "visualType": "table", - "pageId": "306fef9a-c760-4559-a326-7c25d196b616", - "layout": { - "x": 10, - "y": 27, - "width": 12, - "height": 7 - }, - "queryRef": { - "kind": "query", - "queryId": "0341b3e4-eccc-4924-9555-9835b128c543" - }, - "visualOptions": { - "table__enableRenderLinks": true, - "colorRulesDisabled": false, - "colorStyle": "light", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "table__renderLinks": [], - "colorRules": [] - } - }, - { - "id": "daa1b2bd-0ccd-43ff-ad2c-1337d247805a", - "title": "", - "visualType": "markdownCard", - "pageId": "9e099251-9658-48da-b416-80422a2a47c7", - "layout": { - "x": 0, - "y": 68, - "width": 22, - "height": 4 - }, - "markdownText": "# Data quality\r\nData quality checks that may impact your reports.\r\n\r\n\u2b06\ufe0f [Top](?tile=ab300a6a-3bbd-450b-942c-7f30113cba15)     \ud83d\udc9c [Give feedback](https://portal.azure.com/#view/HubsExtension/InProductFeedbackBlade/extensionName/FinOpsToolkit/cesQuestion/How%20easy%20or%20hard%20is%20it%20to%20use%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/cvaQuestion/How%20valuable%20the%20FinOps%20hubs%20Data%20Explorer%20dashboard%3F/surveyId/FTK$$ftkver$$/bladeName/Hubs.Dashboard/featureName/Understand.DataIngestion.DataQuality)", - "visualOptions": {} - }, - { - "id": "d5ea61ef-d4f5-423e-bfa4-d1ab473867c2", - "title": "Summary of list, contracted, and effective cost alignment", - "visualType": "table", - "pageId": "9e099251-9658-48da-b416-80422a2a47c7", - "layout": { - "x": 0, - "y": 72, - "width": 22, - "height": 8 - }, - "queryRef": { - "kind": "query", - "queryId": "56fd8707-bbb3-4f7e-8e37-8dd90ada3baa" - }, - "description": "This table summarizes rows where cost columns are not set or may be incorrect in the data provided by Cost Management.", - "visualOptions": { - "table__enableRenderLinks": true, - "colorRulesDisabled": true, - "colorStyle": "light", - "crossFilterDisabled": false, - "drillthroughDisabled": false, - "crossFilter": [], - "drillthrough": [], - "table__renderLinks": [], - "colorRules": [] - } - } - ], - "baseQueries": [ - { - "id": "58764bcb-2ba0-4c7f-b018-e5d7f6cff688", - "queryId": "43612ae4-c475-4f22-bb50-ce9d995abb8f", - "variableName": "CostsThisMonth" - }, - { - "id": "21512220-154c-4646-a188-72e8c437d8ed", - "queryId": "cb1f5404-c0b1-42fd-99fb-3cff7b08daaa", - "variableName": "CostsLastMonth" - }, - { - "id": "5fa73857-4e92-4a16-8179-f94563e5f605", - "queryId": "4ce0f587-2d45-436c-8f79-102c6b382439", - "variableName": "CostsByMonth" - }, - { - "id": "4880346a-0a24-48f9-bf25-b7427df29d69", - "queryId": "6b598467-8c31-4693-b1eb-7ed683fcfc3a", - "variableName": "CostsByDay" - }, - { - "id": "8ca40660-3fbc-4a08-b12e-92e7382d9449", - "queryId": "4a1973bf-08e9-4e82-b8e6-6edff81cf0a5", - "variableName": "CostsByDayAHB" - }, - { - "id": "48ebd897-d085-490f-8b4b-9b43d4fb2efc", - "queryId": "eb9259cc-05b7-4441-a66d-a29026fe371b", - "variableName": "CostsPlus" - } - ], - "parameters": [ - { - "kind": "int", - "id": "50e8acdc-5d90-4937-96cb-53a9b0ad047f", - "displayName": "Monthly trend", - "description": "How many months should be shown in monthly trend charts and tables?", - "variableName": "numberOfMonths", - "selectionType": "scalar", - "includeAllOption": false, - "defaultValue": { - "kind": "query-result" - }, - "dataSource": { - "kind": "query", - "columns": { - "value": "Value", - "label": "Label" - }, - "queryRef": { - "kind": "query", - "queryId": "c9039243-968d-4e75-9899-8d4ab51a9896" - } - }, - "showOnPages": { - "kind": "all" - } - }, - { - "kind": "int", - "selectionType": "freetext", - "id": "c016c954-1d37-41fb-8c4e-7607a9be2e96", - "displayName": "Daily trend", - "variableName": "numberOfDays", - "description": "How many days should be shown in daily trend charts and tables? Does not apply to \"this month and last\" queries.", - "defaultValue": { - "kind": "value", - "value": 28 - }, - "showOnPages": { - "kind": "all" - } - }, - { - "kind": "int", - "selectionType": "freetext", - "id": "55463317-2576-4d98-875e-85a1fa032c12", - "displayName": "Max group count", - "variableName": "maxGroupCount", - "description": "How many groups should be shown in charts? Remaining will be in an \"others\" group.", - "defaultValue": { - "kind": "value", - "value": 9 - }, - "showOnPages": { - "kind": "all" - } - }, - { - "kind": "string", - "id": "7c3e4a2b-1d8f-4e9a-b5c7-a9d6f3e2b4c1", - "displayName": "Currency", - "description": "Filter costs to a single billing currency. Aggregating values across currencies without conversion produces incorrect totals and savings rates.", - "variableName": "selectedBillingCurrency", - "selectionType": "scalar", - "includeAllOption": false, - "defaultValue": { - "kind": "query-result" - }, - "dataSource": { - "kind": "query", - "columns": { - "value": "BillingCurrency", - "label": "Label" - }, - "queryRef": { - "kind": "query", - "queryId": "f2a8c4d6-3b5e-4a7f-9c2d-8e5b1f4a7d9c" - } - }, - "showOnPages": { - "kind": "all" - } - } - ], - "dataSources": [ - { - "id": "23540be2-ffc9-4b61-8c4c-05e493e682a6", - "kind": "manual-kusto", - "scopeId": "kusto", - "name": "Hub", - "clusterUri": "http://localhost:8082", - "database": "Hub" - } - ], - "pages": [ - { - "name": "About", - "id": "969ddf4c-8f2a-4ec4-9588-bb2f39473c9f" - }, - { - "name": "UNDERSTAND", - "id": "ea47329e-0bc9-4c11-b110-534878dbb3ad" - }, - { - "id": "f8ee3008-df7e-442f-825e-2e3b46e4c185", - "name": "- Summary" - }, - { - "name": "- Anomaly management", - "id": "5838c918-4541-44fe-90d2-77306ef1e241" - }, - { - "name": "- Data ingestion", - "id": "9e099251-9658-48da-b416-80422a2a47c7" - }, - { - "name": "OPTIMIZE", - "id": "d01a8154-8a60-4d22-96ea-54b45b1417fe" - }, - { - "name": "- Rate optimization", - "id": "306fef9a-c760-4559-a326-7c25d196b616" - }, - { - "id": "a4ec1d55-5b6e-49af-bb9e-4f3d136cdf05", - "name": "- Licensing + SaaS" - }, - { - "name": "QUANTIFY", - "id": "8beab65c-f5ec-4661-bc67-37b10baffb16" - }, - { - "name": "- Budgeting", - "id": "5c57f940-6f0f-4e39-930b-b6e89bb758ad" - }, - { - "name": "MANAGE", - "id": "11f019cd-e0cc-4aa7-a125-4dc852eb3f10" - }, - { - "name": "- Invoicing + chargeback", - "id": "f416685d-f559-4514-8e45-5e0e09aec286" - } - ], - "queries": [ - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let monthname = dynamic(['(ignore)', 'Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']);\nlet costs = materialize(\n CostsLastMonth\n | summarize BilledCost = round(sum(BilledCost), 2), EffectiveCost = round(sum(EffectiveCost), 2) by BillingPeriodStart = startofmonth(BillingPeriodStart)\n | extend json = todynamic(strcat('[{\"type\":\"Billed cost\", \"Cost\":', BilledCost, '}, {\"type\":\"Effective cost\", \"Cost\":', EffectiveCost, '}]'))\n | mv-expand json\n | project Type = strcat(json.type, ' (', monthname[monthofyear(BillingPeriodStart)], ' ', format_datetime(BillingPeriodStart, 'yyyy'), ')'), Cost = todouble(json.Cost)\n);\ncosts", - "id": "152f2041-bbc1-41e4-b155-271b2e0cf6e9", - "usedVariables": [ - "CostsLastMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "CostsByMonth\n| summarize BilledCost = round(sum(BilledCost), 2), EffectiveCost = round(sum(EffectiveCost), 2) by BillingPeriodStart = startofmonth(BillingPeriodStart)\n| render timechart", - "id": "bc24e050-f2b9-4b4a-a08d-69fc4a4bb95e", - "usedVariables": [ - "CostsByMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "CostsByMonth\n// | summarize BilledCost = round(sum(BilledCost), 2), EffectiveCost = round(sum(EffectiveCost), 2) by BillingPeriodStart = startofmonth(BillingPeriodStart)\n// | render timechart\n| summarize BilledCost = sum(BilledCost), EffectiveCost = sum(EffectiveCost) by BillingPeriodStart = startofmonth(BillingPeriodStart)\n| order by BillingPeriodStart asc\n| extend PreviousBilledCost = prev(BilledCost)\n| extend PreviousEffectiveCost = prev(EffectiveCost)\n| project BillingPeriodStart\n , BilledCost = iif(isempty(PreviousBilledCost), todouble(0), todouble((BilledCost - PreviousBilledCost) * 100.0 / PreviousBilledCost))\n , EffectiveCost = iif(isempty(PreviousEffectiveCost), todouble(0), todouble((EffectiveCost - PreviousEffectiveCost) * 100.0 / PreviousEffectiveCost))\n", - "id": "0d91ea4a-c81d-4a21-b708-b6af37be1eec", - "usedVariables": [ - "CostsByMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let data = materialize(\n CostsLastMonth\n | summarize\n Subscriptions = dcount(SubAccountId),\n ResourceGroups = dcount(strcat(SubAccountId, x_ResourceGroupName)),\n Resources = dcount(ResourceId),\n Services = dcount(ServiceName)\n | project json = todynamic(strcat('[{ \"Type\":\"Subscriptions\", \"Count\":', Subscriptions, ' }, { \"Type\":\"Resource groups\", \"Count\":', ResourceGroups, ' }, { \"Type\":\"Resources\", \"Count\":', Resources, ' }, { \"Type\":\"Services\", \"Count\":', Services, ' }]'))\n | mv-expand json\n | project Label = tostring(json.Type), Count = tolong(json.Count)\n);\ndata", - "id": "f2cecbb0-13f8-4642-afa4-bbcc0558f777", - "usedVariables": [ - "CostsLastMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let data = materialize(\n CostsByDay\n | summarize\n Subscriptions = dcount(SubAccountId),\n ResourceGroups = dcount(strcat(SubAccountId, x_ResourceGroupName)),\n Resources = dcount(ResourceId),\n Services = dcount(ServiceName)\n | project json = todynamic(strcat('[{ \"Type\":\"Subscriptions\", \"Count\":', Subscriptions, ' }, { \"Type\":\"Resource groups\", \"Count\":', ResourceGroups, ' }, { \"Type\":\"Resources\", \"Count\":', Resources, ' }, { \"Type\":\"Services\", \"Count\":', Services, ' }]'))\n | mv-expand json\n | project Label = tostring(json.Type), Count = tolong(json.Count)\n);\ndata", - "id": "5d63e04d-1a11-4307-96f1-ebbf68e09be0", - "usedVariables": [ - "CostsByDay" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "CostsByMonth\n| summarize EffectiveCost = sum(EffectiveCost) by BillingPeriodStart\n| order by BillingPeriodStart asc\n| extend PreviousEffectiveCost = prev(EffectiveCost)\n| project BillingPeriodStart, EffectiveCost, Change = iif(isempty(PreviousEffectiveCost), todouble(0), todouble((EffectiveCost - PreviousEffectiveCost) / PreviousEffectiveCost)) * 100\n", - "id": "290e7eab-8159-4338-8531-85e2718cedb1", - "usedVariables": [ - "CostsByMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nCostsPlus\n| where ChargePeriodStart >= startofmonth(ago(90d))\n| summarize EffectiveCost = sum(EffectiveCost) by ChargePeriodStart, Day = dayofmonth(ChargePeriodStart), Month = strcat(format_datetime(ChargePeriodStart, 'MM '), monthname[monthofyear(ChargePeriodStart)])\n| order by ChargePeriodStart asc\n| extend EffectiveCostRunningTotal = row_cumsum(EffectiveCost, prev(Month) != Month)\n| project Day, EffectiveCostRunningTotal, Month\n| render areachart ", - "id": "d5ed469a-45ea-49a2-b305-b841050213cf", - "usedVariables": [ - "CostsPlus" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nCostsPlus\n| where ChargePeriodStart >= startofmonth(startofmonth(now()) - 1d)\n| summarize EffectiveCost = sum(EffectiveCost) by ChargePeriodStart, Month = strcat(format_datetime(ChargePeriodStart, 'MM '), monthname[monthofyear(ChargePeriodStart)])\n| order by ChargePeriodStart asc\n| extend EffectiveCostRunningTotal = row_cumsum(EffectiveCost, prev(Month) != Month)\n| project ChargePeriodStart, EffectiveCostRunningTotal, Month\n| render areachart ", - "id": "10300876-866a-40d7-836d-b3a8783ecece", - "usedVariables": [ - "CostsPlus" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nCostsPlus\n| where ChargePeriodStart >= startofmonth(ago(90d))\n| summarize EffectiveCost = sum(EffectiveCost) by ChargePeriodStart, Day = dayofmonth(ChargePeriodStart), Month = strcat(format_datetime(ChargePeriodStart, 'MM '), monthname[monthofyear(ChargePeriodStart)])\n| order by ChargePeriodStart asc\n| project Day, EffectiveCost, Month\n| render columnchart", - "id": "9e41a624-d5f9-40c6-b47f-fef4b50ec3dd", - "usedVariables": [ - "CostsPlus" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let costs = CostsPlus | where ChargePeriodStart >= startofmonth(ago(90d));\nlet all = costs | summarize sum(EffectiveCost) by SubAccountId;\nlet count = toscalar(all | order by sum_EffectiveCost desc | count);\nlet topX = all | order by sum_EffectiveCost desc | limit maxGroupCount;\nlet otherId = '(others)';\ncosts\n//\n// Group rows after max count\n| extend inTopX = SubAccountId in (topX)\n| extend SubAccountId = iff(inTopX, SubAccountId, otherId)\n| extend SubAccountName = iff(inTopX, SubAccountName, strcat('(', (count - maxGroupCount), ' others)'))\n//\n| summarize \n EffectiveCost = round(sum(EffectiveCost), 2),\n SubAccountName = take_any(SubAccountName)\n by\n ChargePeriodStart,\n SubAccountId\n| project ChargePeriodStart, EffectiveCost, Sub = iff(SubAccountId == otherId, SubAccountName, strcat(SubAccountName, ' (', split(SubAccountId, '/')[2], ')'))\n| order by EffectiveCost desc\n| render columnchart", - "id": "f26e2204-270e-4219-8f68-5acef1c9393f", - "usedVariables": [ - "CostsPlus", - "maxGroupCount" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nCostsPlus\n| where startofmonth(ChargePeriodStart) >= startofmonth(now(), -1)\n| summarize \n EffectiveCost = sum(EffectiveCost),\n ContractedCost = sum(ContractedCost),\n ListCost = sum(ListCost)\n by\n ChargePeriodStart,\n Month = strcat(format_datetime(ChargePeriodStart, 'MM '), monthname[monthofyear(ChargePeriodStart)])\n| extend CommitmentDiscountSavings = ContractedCost - EffectiveCost\n| extend NegotiatedDiscountSavings = ListCost - ContractedCost\n| order by ChargePeriodStart asc\n| extend EffectiveCostRunningTotal = row_cumsum(EffectiveCost, prev(Month) != Month)\n| extend CommitmentDiscountSavingsRunningTotal = row_cumsum(CommitmentDiscountSavings, prev(Month) != Month)\n| extend NegotiatedDiscountSavingsRunningTotal = row_cumsum(NegotiatedDiscountSavings, prev(Month) != Month)\n| project ChargePeriodStart, CommitmentDiscountSavingsRunningTotal, NegotiatedDiscountSavingsRunningTotal, EffectiveCostRunningTotal, Month\n| render areachart ", - "id": "dcc1f533-e5f9-4855-9e4c-21e21fcdf943", - "usedVariables": [ - "CostsPlus" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet data = materialize(\n CostsByMonth\n | where ChargePeriodStart >= startofmonth(now(), -1)\n | summarize \n BilledCost = sum(BilledCost),\n EffectiveCost = sum(EffectiveCost),\n ContractedCost = sum(ContractedCost),\n ListCost = sum(ListCost)\n by\n ChargePeriodStart\n | order by ChargePeriodStart asc\n | extend CommitmentDiscountSavings = ContractedCost - EffectiveCost\n | extend NegotiatedDiscountSavings = ListCost - ContractedCost\n | extend Month = monthname[monthofyear(ChargePeriodStart)]\n | project json = todynamic(strcat('[{ \"Type\":\"Billed cost\", \"Count\":', BilledCost, ' }, { \"Type\":\"Effective cost\", \"Count\":', EffectiveCost, ' }, { \"Type\":\"Commitment savings\", \"Count\":', CommitmentDiscountSavings, ' }, { \"Type\":\"Negotiated savings\", \"Count\":', NegotiatedDiscountSavings, ' }]')), Month, IsThisMonth = ChargePeriodStart >= startofmonth(now())\n | mv-expand json\n | project Label = strcat(json.Type, ' (', Month, ')'), Count = tolong(json.Count), IsThisMonth\n);\ndata", - "id": "e8b343dc-7430-4487-8d44-ef48ac454f2d", - "usedVariables": [ - "CostsByMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet data = (\n CostsByMonth\n | summarize \n BilledCost = todouble(round(sum(BilledCost), 2)),\n EffectiveCost = todouble(round(sum(EffectiveCost), 2)),\n ContractedCost = todouble(round(sum(ContractedCost), 2)),\n ListCost = todouble(round(sum(ListCost), 2))\n by\n ChargePeriodStart\n | order by ChargePeriodStart asc\n | extend CommitmentDiscountSavings = todouble(round(ContractedCost - EffectiveCost, 2))\n | extend NegotiatedDiscountSavings = todouble(round(ListCost - ContractedCost, 2))\n | extend ChargePeriod = strcat(format_datetime(ChargePeriodStart, 'yyyy-MM - '), monthname[monthofyear(ChargePeriodStart)])\n);\ndata | extend Data = 'Billed cost' | evaluate pivot(ChargePeriod, sum(BilledCost), Data)\n| union (data | extend Data = 'Effective cost' | evaluate pivot(ChargePeriod, sum(EffectiveCost), Data))\n| union (data | extend Data = 'Negotiated savings' | evaluate pivot(ChargePeriod, sum(NegotiatedDiscountSavings), Data))\n| union (data | extend Data = 'Commitment savings' | evaluate pivot(ChargePeriod, sum(CommitmentDiscountSavings), Data))\n", - "id": "c077a6d4-719f-42fe-b39b-36f77dc68976", - "usedVariables": [ - "CostsByMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet data = (\n CostsByMonth\n | summarize \n BilledCost = todouble(round(sum(BilledCost), 2)),\n EffectiveCost = todouble(round(sum(EffectiveCost), 2)),\n ContractedCost = todouble(round(sum(ContractedCost), 2)),\n ListCost = todouble(round(sum(ListCost), 2))\n by\n ChargePeriodStart = startofmonth(ChargePeriodStart)\n | order by ChargePeriodStart asc\n | extend CommitmentDiscountSavings = todouble(round(ContractedCost - EffectiveCost, 2))\n | extend NegotiatedDiscountSavings = todouble(round(ListCost - ContractedCost, 2))\n | extend ChargePeriod = strcat(format_datetime(ChargePeriodStart, 'yyyy-MM - '), monthname[monthofyear(ChargePeriodStart)])\n);\ndata | extend Data = 'Billed cost' | evaluate pivot(ChargePeriod, sum(BilledCost), Data)\n| union (data | extend Data = 'Effective cost' | evaluate pivot(ChargePeriod, sum(EffectiveCost), Data))\n| union (data | extend Data = 'Negotiated savings' | evaluate pivot(ChargePeriod, sum(NegotiatedDiscountSavings), Data))\n| union (data | extend Data = 'Commitment savings' | evaluate pivot(ChargePeriod, sum(CommitmentDiscountSavings), Data))\n//, BilledCost, NegotiatedDiscountSavings, CommitmentDiscountSavings)\n//| project ChargePeriod, BilledCost = round(BilledCost, 2), EffectiveCost = round(EffectiveCost, 2), NegotiatedDiscountSavings, CommitmentDiscountSavings\n", - "id": "3d947a7c-edca-46b5-a862-de77a725c85f", - "usedVariables": [ - "CostsByMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet data = (\n CostsByMonth\n | summarize \n BilledCost = sum(BilledCost),\n EffectiveCost = sum(EffectiveCost),\n ContractedCost = sum(ContractedCost),\n ListCost = sum(ListCost),\n SubAccountName = take_any(SubAccountName)\n by\n ChargePeriodStart = startofmonth(ChargePeriodStart),\n SubAccountId\n | as per\n | union (\n per\n | summarize \n BilledCost = sum(BilledCost),\n EffectiveCost = sum(EffectiveCost),\n ContractedCost = sum(ContractedCost),\n ListCost = sum(ListCost),\n SubAccountName = take_any(SubAccountName)\n by\n SubAccountId\n )\n | order by ChargePeriodStart asc\n | extend BilledCost = todouble(round(BilledCost, 2))\n | extend EffectiveCost = todouble(round(EffectiveCost, 2))\n | extend ContractedCost = todouble(round(ContractedCost, 2))\n | extend ListCost = todouble(round(ListCost, 2))\n | extend CommitmentDiscountSavings = todouble(round(ContractedCost - EffectiveCost, 2))\n | extend NegotiatedDiscountSavings = todouble(round(ListCost - ContractedCost, 2))\n | extend ChargePeriod = iff(isempty(ChargePeriodStart), strcat('Total'), strcat(format_datetime(ChargePeriodStart, 'yyyy-MM - '), monthname[monthofyear(ChargePeriodStart)]))\n);\ndata | evaluate pivot(ChargePeriod, sum(EffectiveCost), SubAccountName)\n| order by Total desc", - "id": "3886b5cd-34a8-42d7-9e16-33ea4d236953", - "usedVariables": [ - "CostsByMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let costs = CostsByMonth;\nlet all = costs | where isnotempty(RegionName) | summarize sum(EffectiveCost) by RegionName;\nlet count = toscalar(all | order by sum_EffectiveCost desc | count);\nlet topX = all | order by sum_EffectiveCost desc | limit iff(count - maxGroupCount > 1, maxGroupCount, count);\nlet otherId = '(others)';\ncosts\n//\n// Group rows after max count\n| extend inTopX = RegionName in (topX)\n| extend RegionName = iff(inTopX, RegionName, strcat('(', count - maxGroupCount, ' others)'))\n//\n| summarize \n EffectiveCost = round(sum(EffectiveCost), 2)\n by\n ChargePeriodStart = startofmonth(ChargePeriodStart),\n RegionName\n| project ChargePeriodStart, EffectiveCost, Region = RegionName\n| order by EffectiveCost desc", - "id": "d5224d06-c8e7-4dd9-afec-595b39712f5a", - "usedVariables": [ - "CostsByMonth", - "maxGroupCount" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet data = (\n CostsByMonth\n | summarize \n EffectiveCost = sum(EffectiveCost)\n by\n ChargePeriodStart = startofmonth(ChargePeriodStart),\n RegionName\n | as per\n | union (\n per\n | summarize \n EffectiveCost = sum(EffectiveCost)\n by\n RegionName\n )\n | order by ChargePeriodStart asc\n | extend EffectiveCost = todouble(round(EffectiveCost, 2))\n | extend ChargePeriod = iff(isempty(ChargePeriodStart), strcat('Total'), strcat(format_datetime(ChargePeriodStart, 'yyyy-MM - '), monthname[monthofyear(ChargePeriodStart)]))\n);\ndata | evaluate pivot(ChargePeriod, sum(EffectiveCost), RegionName)\n| order by Total desc", - "id": "0dcf2c54-7f1d-45e9-a53b-d598a24493a4", - "usedVariables": [ - "CostsByMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let costs = CostsByMonth;\nlet all = costs | where isnotempty(ServiceName) | summarize sum(EffectiveCost) by ServiceName;\nlet count = toscalar(all | order by sum_EffectiveCost desc | count);\nlet topX = all | order by sum_EffectiveCost desc | limit maxGroupCount;\nlet otherId = '(others)';\ncosts\n//\n// Group rows after max count\n| extend inTopX = ServiceName in (topX)\n| extend ServiceName = iff(inTopX, ServiceName, otherId)\n//\n| summarize \n EffectiveCost = round(sum(EffectiveCost), 2)\n by\n ChargePeriodStart,\n ServiceName\n| project ChargePeriodStart, EffectiveCost, Category = ServiceName\n| order by EffectiveCost desc\n| render columnchart", - "id": "30644718-defd-4c6c-9ffa-a9c2cd1f871f", - "usedVariables": [ - "CostsByMonth", - "maxGroupCount" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet data = (\n CostsByMonth\n | summarize \n EffectiveCost = sum(EffectiveCost)\n by\n ChargePeriodStart,\n ServiceCategory,\n ServiceName\n | as per\n | union (\n per\n | summarize \n EffectiveCost = sum(EffectiveCost)\n by\n ServiceCategory,\n ServiceName\n )\n | order by ChargePeriodStart asc\n | extend EffectiveCost = todouble(round(EffectiveCost, 2))\n | extend ChargePeriod = iff(isempty(ChargePeriodStart), strcat('Total'), strcat(format_datetime(ChargePeriodStart, 'yyyy-MM - '), monthname[monthofyear(ChargePeriodStart)]))\n);\ndata | evaluate pivot(ChargePeriod, sum(EffectiveCost), ServiceName, ServiceCategory)\n| order by Total desc", - "id": "6259f773-593c-4953-898c-15aa5ff6e53a", - "usedVariables": [ - "CostsByMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let costs = CostsByMonth;\nlet all = costs | where isnotempty(SubAccountId) | summarize sum(EffectiveCost) by SubAccountId;\nlet count = toscalar(all | order by sum_EffectiveCost desc | count);\nlet topX = all | order by sum_EffectiveCost desc | limit maxGroupCount;\nlet otherId = '(others)';\ncosts\n//\n// Group rows after max count\n| extend inTopX = SubAccountId in (topX)\n| extend SubAccountId = iff(inTopX, SubAccountId, otherId)\n| extend SubAccountName = iff(inTopX, SubAccountName, strcat('(', (count - maxGroupCount), ' others)'))\n//\n| summarize \n EffectiveCost = round(sum(EffectiveCost), 2),\n SubAccountName = take_any(SubAccountName)\n by\n ChargePeriodStart = startofmonth(ChargePeriodStart),\n SubAccountId\n| project ChargePeriodStart, EffectiveCost, Sub = iff(SubAccountId == otherId, SubAccountName, strcat(SubAccountName, ' (', split(SubAccountId, '/')[2], ')'))\n| order by EffectiveCost desc\n| render columnchart", - "id": "21a87abe-19e2-44ec-8298-ba872e66c162", - "usedVariables": [ - "CostsByMonth", - "maxGroupCount" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet data = (\n CostsByMonth\n | summarize \n BilledCost = sum(BilledCost),\n EffectiveCost = sum(EffectiveCost),\n ContractedCost = sum(ContractedCost),\n ListCost = sum(ListCost),\n SubAccountName = take_any(SubAccountName)\n by\n ChargePeriodStart,\n SubAccountId\n | as per\n | union (\n per\n | summarize \n BilledCost = sum(BilledCost),\n EffectiveCost = sum(EffectiveCost),\n ContractedCost = sum(ContractedCost),\n ListCost = sum(ListCost),\n SubAccountName = take_any(SubAccountName)\n by\n SubAccountId\n )\n | order by ChargePeriodStart asc\n | extend BilledCost = todouble(round(BilledCost, 2))\n | extend EffectiveCost = todouble(round(EffectiveCost, 2))\n | extend ContractedCost = todouble(round(ContractedCost, 2))\n | extend ListCost = todouble(round(ListCost, 2))\n | extend CommitmentDiscountSavings = todouble(round(ContractedCost - EffectiveCost, 2))\n | extend NegotiatedDiscountSavings = todouble(round(ListCost - ContractedCost, 2))\n | extend ChargePeriod = iff(isempty(ChargePeriodStart), strcat('Total'), strcat(format_datetime(ChargePeriodStart, 'yyyy-MM - '), monthname[monthofyear(ChargePeriodStart)]))\n);\ndata | evaluate pivot(ChargePeriod, sum(EffectiveCost), SubAccountName)\n| order by Total desc", - "id": "90502e9a-2d0d-4ae4-8d9d-cc21f9b72d5d", - "usedVariables": [ - "CostsByMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet data = (\n CostsByMonth\n | extend x_ResourceGroupId = strcat(SubAccountId, '/resourcegroups/', x_ResourceGroupName)\n | summarize \n EffectiveCost = sum(EffectiveCost),\n SubAccountName = take_any(SubAccountName),\n x_ResourceGroupName = take_any(x_ResourceGroupName)\n by\n ChargePeriodStart,\n x_ResourceGroupId\n | as per\n | union (\n per\n | summarize \n EffectiveCost = sum(EffectiveCost),\n SubAccountName = take_any(SubAccountName),\n x_ResourceGroupName = take_any(x_ResourceGroupName)\n by\n x_ResourceGroupId\n )\n | order by ChargePeriodStart asc\n | extend EffectiveCost = todouble(round(EffectiveCost, 2))\n | extend ChargePeriod = iff(isempty(ChargePeriodStart), strcat('Total'), strcat(format_datetime(ChargePeriodStart, 'yyyy-MM - '), monthname[monthofyear(ChargePeriodStart)]))\n);\ndata | evaluate pivot(ChargePeriod, sum(EffectiveCost), x_ResourceGroupName, SubAccountName)\n| order by Total desc", - "id": "c2c65ec0-e57d-4834-8a6b-b5975afeb9a0", - "usedVariables": [ - "CostsByMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let costs = CostsByMonth | extend x_ResourceGroupId = strcat(SubAccountId, '/resourcegroups/', x_ResourceGroupName);\nlet all = costs | where isnotempty(x_ResourceGroupId) | summarize sum(EffectiveCost) by x_ResourceGroupId;\nlet count = toscalar(all | order by sum_EffectiveCost desc | count);\nlet topX = all | order by sum_EffectiveCost desc | limit maxGroupCount;\nlet otherId = '(others)';\ncosts\n//\n// Group rows after max count\n| extend inTopX = x_ResourceGroupId in (topX)\n| extend x_ResourceGroupId = iff(inTopX, x_ResourceGroupId, otherId)\n| extend x_ResourceGroupName = iff(inTopX, x_ResourceGroupName, strcat('(', (count - maxGroupCount), ' others)'))\n//\n| summarize \n EffectiveCost = round(sum(EffectiveCost), 2),\n SubAccountName = take_any(SubAccountName),\n x_ResourceGroupName = take_any(x_ResourceGroupName)\n by\n ChargePeriodStart,\n x_ResourceGroupId\n| project ChargePeriodStart, EffectiveCost, RG = iff(x_ResourceGroupId == otherId, x_ResourceGroupName, strcat(x_ResourceGroupName, ' (', SubAccountName, ')'))\n| order by EffectiveCost desc\n| render columnchart", - "id": "61b26784-6a73-4e80-85b2-9c5cfbd2dd06", - "usedVariables": [ - "CostsByMonth", - "maxGroupCount" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let costs = CostsByMonth;\nlet all = costs | where isnotempty(ResourceId) | summarize sum(EffectiveCost) by ResourceId;\nlet count = toscalar(all | order by sum_EffectiveCost desc | count);\nlet topX = all | order by sum_EffectiveCost desc | limit maxGroupCount;\nlet otherId = '(others)';\ncosts\n//\n// Group rows after max count\n| extend inTopX = ResourceId in (topX)\n| extend ResourceId = iff(inTopX, ResourceId, otherId)\n| extend ResourceName = iff(inTopX, ResourceName, strcat('(', (count - maxGroupCount), ' others)'))\n//\n| summarize \n EffectiveCost = round(sum(EffectiveCost), 2),\n ResourceType = take_any(ResourceType),\n ResourceName = take_any(ResourceName)\n by\n ChargePeriodStart,\n ResourceId\n| project ChargePeriodStart, EffectiveCost, RG = iff(ResourceId == otherId, ResourceName, strcat(ResourceName, ' (', ResourceType, ')'))\n| order by EffectiveCost desc\n| render columnchart", - "id": "3924981c-23e4-464d-a872-045df1752750", - "usedVariables": [ - "CostsByMonth", - "maxGroupCount" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet data = (\n CostsByMonth\n | summarize \n EffectiveCost = sum(EffectiveCost),\n ResourceName = take_any(ResourceName),\n ResourceType = take_any(ResourceType),\n RegionName = take_any(RegionName),\n x_ResourceGroupName = take_any(x_ResourceGroupName),\n SubAccountName = take_any(SubAccountName)\n by\n ChargePeriodStart,\n ResourceId\n | as per\n | union (\n per\n | summarize \n EffectiveCost = sum(EffectiveCost)\n by\n ResourceId,\n ResourceName,\n ResourceType,\n RegionName,\n x_ResourceGroupName,\n SubAccountName\n )\n | order by ChargePeriodStart asc\n | extend EffectiveCost = todouble(round(EffectiveCost, 2))\n | extend ChargePeriod = iff(isempty(ChargePeriodStart), strcat('Total'), strcat(format_datetime(ChargePeriodStart, 'yyyy-MM - '), monthname[monthofyear(ChargePeriodStart)]))\n);\ndata | evaluate pivot(ChargePeriod, sum(EffectiveCost), ResourceName, ResourceType, RegionName, x_ResourceGroupName, SubAccountName)\n| order by Total desc", - "id": "c7be613e-deb4-4779-ad75-4445e8d5e01f", - "usedVariables": [ - "CostsByMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let costs = CostsByDay;\nlet all = costs | summarize sum(EffectiveCost) by ResourceId;\nlet count = toscalar(all | order by sum_EffectiveCost desc | count);\nlet topX = all | order by sum_EffectiveCost desc | limit maxGroupCount;\nlet otherId = '(others)';\ncosts\n//\n// Group rows after max count\n| extend inTopX = ResourceId in (topX)\n| extend ResourceId = iff(inTopX, ResourceId, otherId)\n| extend ResourceName = iff(inTopX, ResourceName, strcat('(', (count - maxGroupCount), ' others)'))\n//\n| summarize \n EffectiveCost = round(sum(EffectiveCost), 2),\n ResourceType = take_any(ResourceType),\n ResourceName = take_any(ResourceName)\n by\n ChargePeriodStart,\n ResourceId\n| project ChargePeriodStart, EffectiveCost, RG = iff(ResourceId == otherId, ResourceName, strcat(ResourceName, ' (', ResourceType, ')'))\n| order by EffectiveCost desc\n| render columnchart", - "id": "49e24ee0-91de-4b1c-973f-036a3c060aca", - "usedVariables": [ - "CostsByDay", - "maxGroupCount" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let costs = CostsByMonth;\nlet all = costs | where isnotempty(ServiceCategory) | summarize sum(EffectiveCost) by ServiceCategory;\nlet count = toscalar(all | order by sum_EffectiveCost desc | count);\nlet topX = all | order by sum_EffectiveCost desc | limit maxGroupCount;\nlet otherId = '(others)';\ncosts\n//\n// Group rows after max count\n| extend inTopX = ServiceCategory in (topX)\n| extend ServiceCategory = iff(inTopX, ServiceCategory, otherId)\n//\n| summarize \n EffectiveCost = round(sum(EffectiveCost), 2)\n by\n ChargePeriodStart,\n ServiceCategory\n| project ChargePeriodStart, EffectiveCost, Category = ServiceCategory\n| order by EffectiveCost desc\n| render columnchart", - "id": "4c7a7614-9b8c-415b-a4e1-d2d53c023d31", - "usedVariables": [ - "CostsByMonth", - "maxGroupCount" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet data = (\n CostsByMonth\n | summarize \n EffectiveCost = sum(EffectiveCost)\n by\n ChargePeriodStart,\n ServiceCategory,\n ServiceName\n | as per\n | union (\n per\n | summarize \n EffectiveCost = sum(EffectiveCost)\n by\n ServiceCategory,\n ServiceName\n )\n | order by ChargePeriodStart asc\n | extend EffectiveCost = todouble(round(EffectiveCost, 2))\n | extend ChargePeriod = iff(isempty(ChargePeriodStart), strcat('Total'), strcat(format_datetime(ChargePeriodStart, 'yyyy-MM - '), monthname[monthofyear(ChargePeriodStart)]))\n);\ndata | evaluate pivot(ChargePeriod, sum(EffectiveCost), ServiceCategory)\n| order by Total desc", - "id": "1d273b55-d2ea-427c-8a5f-01f6c240e98a", - "usedVariables": [ - "CostsByMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let costs = CostsByMonth;\nlet all = costs | where isnotempty(ResourceType) | summarize sum(EffectiveCost) by ResourceType;\nlet count = toscalar(all | order by sum_EffectiveCost desc | count);\nlet topX = all | order by sum_EffectiveCost desc | limit maxGroupCount;\nlet otherId = '(others)';\ncosts\n//\n// Group rows after max count\n| extend inTopX = ResourceType in (topX)\n| extend ResourceType = iff(inTopX, ResourceType, strcat('(', (count - maxGroupCount), ' others)'))\n//\n| summarize \n EffectiveCost = round(sum(EffectiveCost), 2)\n by\n ChargePeriodStart,\n ResourceType\n| project ChargePeriodStart, EffectiveCost, Type = ResourceType\n| order by EffectiveCost desc\n| render columnchart", - "id": "90e30901-a931-4a1d-b81e-0a1821032c3c", - "usedVariables": [ - "CostsByMonth", - "maxGroupCount" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let costs = CostsByDay;\nlet all = costs | where isnotempty(ResourceType) | summarize sum(EffectiveCost) by ResourceType;\nlet count = toscalar(all | order by sum_EffectiveCost desc | count);\nlet topX = all | order by sum_EffectiveCost desc | limit maxGroupCount;\nlet otherId = '(others)';\ncosts\n| where isnotempty(ResourceType)\n//\n// Group rows after max count\n| extend inTopX = ResourceType in (topX)\n| extend ResourceType = iff(inTopX, ResourceType, strcat('(', (count - maxGroupCount), ' others)'))\n//\n| summarize \n EffectiveCost = round(sum(EffectiveCost), 2)\n by\n ChargePeriodStart = startofday(ChargePeriodStart),\n ResourceType\n| project ChargePeriodStart, EffectiveCost, Type = ResourceType\n| order by EffectiveCost desc\n| render columnchart", - "id": "ebe41e27-e9f9-478e-ab90-fd1f87906766", - "usedVariables": [ - "CostsByDay", - "maxGroupCount" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet data = (\n CostsByMonth\n | summarize \n EffectiveCost = sum(EffectiveCost),\n ResourceName = take_any(ResourceName),\n ResourceType = take_any(ResourceType),\n RegionName = take_any(RegionName),\n x_ResourceGroupName = take_any(x_ResourceGroupName),\n SubAccountName = take_any(SubAccountName)\n by\n ChargePeriodStart,\n ResourceId\n | as per\n | union (\n per\n | summarize \n EffectiveCost = sum(EffectiveCost)\n by\n ResourceId,\n ResourceName,\n ResourceType,\n RegionName,\n x_ResourceGroupName,\n SubAccountName\n )\n | order by ChargePeriodStart asc\n | extend EffectiveCost = todouble(round(EffectiveCost, 2))\n | extend ChargePeriod = iff(isempty(ChargePeriodStart), strcat('Total'), strcat(format_datetime(ChargePeriodStart, 'yyyy-MM - '), monthname[monthofyear(ChargePeriodStart)]))\n);\ndata | evaluate pivot(ChargePeriod, sum(EffectiveCost), ResourceName, ResourceType, RegionName, x_ResourceGroupName, SubAccountName)\n| order by Total desc", - "id": "7f7b08a9-ae15-46f8-8b0f-767280375add", - "usedVariables": [ - "CostsByMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let costs = CostsByDay | where isnotempty(ResourceType);\nlet all = costs | summarize sum(EffectiveCost) by ResourceType;\nlet count = toscalar(all | order by sum_EffectiveCost desc | count);\nlet topX = all | order by sum_EffectiveCost desc | limit maxGroupCount;\nlet otherId = '(others)';\ncosts\n| where isnotempty(ResourceType)\n//\n// Group rows after max count\n| extend inTopX = ResourceType in (topX)\n| extend ResourceType = iff(inTopX, ResourceType, strcat('(', (count - maxGroupCount), ' others)'))\n//\n| summarize \n EffectiveCost = round(sum(EffectiveCost), 2)\n by\n ResourceType\n| order by EffectiveCost desc\n| render columnchart", - "id": "a35b4c05-8ecb-4c51-9aaf-980af3d7923e", - "usedVariables": [ - "CostsByDay", - "maxGroupCount" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let costs = CostsByDay | where isnotempty(ResourceType);\nlet all = costs | summarize ResourceCount = dcount(ResourceId) by ResourceType;\nlet count = toscalar(all | order by ResourceCount desc | count);\nlet topX = all | order by ResourceCount desc | limit maxGroupCount;\nlet otherId = '(others)';\ncosts\n| where isnotempty(ResourceType)\n//\n// Group rows after max count\n| extend inTopX = ResourceType in (topX)\n| extend ResourceType = iff(inTopX, ResourceType, strcat('(', (count - maxGroupCount), ' others)'))\n//\n| summarize \n ResourceCount = dcount(ResourceId)\n by\n ResourceType\n| order by ResourceCount desc\n| render columnchart", - "id": "9c5d3cf0-b6cb-45a2-acfd-b19df425bddf", - "usedVariables": [ - "CostsByDay", - "maxGroupCount" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nCostsByDay\n| where isnotempty(ResourceType)\n| summarize \n Count = dcount(ResourceId),\n EffectiveCost = sum(EffectiveCost),\n ListCost = sum(ListCost),\n ContractedCost = sum(ContractedCost)\n by\n ResourceType\n| order by Count desc\n| project \n Type = ResourceType,\n Count,\n Cost = round(EffectiveCost, 2),\n // NegotiatedSavings = round(ListCost - ContractedCost, 2),\n // CommitmentSavings = round(ContractedCost - EffectiveCost, 2),\n Savings = round(ListCost - EffectiveCost, 2),\n [\"Cost / Resource\"] = round(EffectiveCost / Count, 2)\n", - "id": "2d7b6447-2769-40b8-958a-f252dab68b1e", - "usedVariables": [ - "CostsByDay" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let costs = CostsByDay\n| where isnotempty(ResourceType);\ncosts | summarize Value = tostring(dcount(ResourceId)) by Label = \"Resources\", Order = 10\n| union (costs | summarize Value = tostring(dcount(ResourceType)) by Label = \"Resource types\", Order = 11)\n| union (costs | summarize Count = dcount(ResourceId) by Value = ResourceType | order by Count desc | limit 1 | extend Label = \"Most used\", Order = 21)\n| union (costs | summarize sum(EffectiveCost) by Value = ResourceType | order by sum_EffectiveCost desc | limit 1 | extend Label = \"Most cost\", Order = 22)\n| union (costs | summarize AllTypes = dcount(ResourceType), CommittedTypes = dcountif(ResourceType, isnotempty(CommitmentDiscountType)) by Label = \"Covered by commitment discounts\", Order = 31 | extend Value = strcat(round(1.0 * CommittedTypes / AllTypes * 100, 1), '%'))\n| union (costs | summarize Savings = sum(ListCost - EffectiveCost) by Value = ResourceType | order by Savings desc | limit 1 | extend Label = \"Most savings\", Order = 32)\n| union (costs | summarize CostPerResource = sum(EffectiveCost) / dcount(ResourceId) by Value = ResourceType | order by CostPerResource desc | limit 1 | extend Label = \"Most expensive (cost / resource)\", Order = 41)\n| union (costs | where ResourceType !in (CostsPlus | where ChargePeriodStart < ago(numberOfDays * 1d) - 1d | distinct ResourceType) | summarize Count = dcount(ResourceId) by ResourceType | order by Count desc | as d | count | extend Label = \"New in last n days\", Value = case(Count == 0, '(none)', Count == 1, toscalar(d | project ResourceType), strcat(toscalar(d | limit 1 | project ResourceType), ' and ', (Count - 1), ' more')), Order = 42)\n| order by Order asc\n", - "id": "f062533b-8c94-412f-bf83-0eb5cd06063d", - "usedVariables": [ - "CostsByDay", - "CostsPlus", - "numberOfDays" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "CostsByDay\n| summarize EffectiveCost = sum(EffectiveCost) by ChargePeriodStart = startofday(ChargePeriodStart)\n| order by ChargePeriodStart asc\n| extend PreviousEffectiveCost = prev(EffectiveCost)\n| project ChargePeriodStart, EffectiveCost, Change = iif(isempty(PreviousEffectiveCost), todouble(0), todouble((EffectiveCost - PreviousEffectiveCost) / PreviousEffectiveCost)) * 100\n", - "id": "1d9c166d-22b6-48fd-9a90-9f983083ecc7", - "usedVariables": [ - "CostsByDay" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "CostsByDay\n| where isnotempty(x_SkuDescription)\n//\n// Only include SKUs with effective prices\n| where x_EffectiveUnitPrice != 0 and isnotempty(x_EffectiveUnitPrice)\n//\n// Only include SKUs with negotiated discounts and not commitment discounts\n| where ListUnitPrice > x_EffectiveUnitPrice\n| where isempty(CommitmentDiscountStatus)\n//\n| summarize \n x_ResourceCount = dcount(ResourceId),\n EffectiveCost = round(sum(EffectiveCost), 2),\n BilledCost = round(sum(BilledCost), 2),\n ListUnitPrice = round(take_any(ListUnitPrice), 4),\n ContractedUnitPrice = round(take_any(ContractedUnitPrice), 4),\n x_EffectiveUnitPrice = round(take_any(x_EffectiveUnitPrice), 4),\n PricingQuantity = sum(PricingQuantity)\n by\n x_SkuDescription,\n PricingUnit,\n CommitmentDiscountType,\n x_SkuTerm\n| order by EffectiveCost desc\n| project \n x_SkuDescription,\n // CommitmentDiscountType,\n // x_SkuTerm,\n Quantity = round(PricingQuantity, 4),\n Unit = PricingUnit,\n // Resources = x_ResourceCount,\n List = ListUnitPrice,\n // ContractedUnitPrice,\n // x_EffectiveUnitPrice,\n Discount = round(ListUnitPrice - x_EffectiveUnitPrice, 4),\n // BilledCost,\n Cost = EffectiveCost\n| where Discount > 0\n", - "id": "af6424d4-8f73-4b81-bfdd-f0287ebcaaca", - "usedVariables": [ - "CostsByDay" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "CostsByDay\n| where isnotempty(x_SkuDescription)\n//\n// Only include SKUs with effective prices\n| where x_EffectiveUnitPrice != 0 and isnotempty(x_EffectiveUnitPrice)\n//\n// Only include SKUs with commitment discounts\n| where isnotempty(CommitmentDiscountStatus)\n//\n| summarize \n x_ResourceCount = dcount(ResourceId),\n EffectiveCost = round(sum(EffectiveCost), 2),\n BilledCost = round(sum(BilledCost), 2),\n ListUnitPrice = round(take_any(ListUnitPrice), 4),\n ContractedUnitPrice = round(take_any(ContractedUnitPrice), 4),\n x_EffectiveUnitPrice = round(take_any(x_EffectiveUnitPrice), 4),\n PricingQuantity = sum(PricingQuantity)\n by\n x_SkuDescription,\n PricingUnit,\n CommitmentDiscountType,\n x_SkuTerm\n| order by EffectiveCost desc\n| project \n x_SkuDescription,\n CommitmentDiscountType,\n Term = case(\n isempty(x_SkuTerm) or x_SkuTerm <= 0, '',\n x_SkuTerm < 12, strcat(x_SkuTerm, ' month', iff(x_SkuTerm != 1, 's', '')),\n strcat(x_SkuTerm / 12, ' year', iff(x_SkuTerm != 12, 's', ''))\n ),\n Quantity = round(PricingQuantity, 4),\n Unit = PricingUnit,\n // Resources = x_ResourceCount,\n List = ListUnitPrice,\n // ContractedUnitPrice,\n // x_EffectiveUnitPrice,\n Discount = round(ListUnitPrice - x_EffectiveUnitPrice, 4),\n // BilledCost,\n Cost = EffectiveCost\n//| where Discount > 0\n", - "id": "c4f6542d-a9cb-4284-bd4e-b9f94ad02192", - "usedVariables": [ - "CostsByDay" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "CostsByDay\n| where isnotempty(x_SkuDescription)\n//\n// Only include SKUs with effective prices\n| where x_EffectiveUnitPrice != 0 and isnotempty(x_EffectiveUnitPrice)\n//\n// Only include SKUs without any discounts\n| where ListUnitPrice == x_EffectiveUnitPrice\n| where isempty(CommitmentDiscountStatus)\n//\n| summarize \n x_ResourceCount = dcount(ResourceId),\n EffectiveCost = round(sum(EffectiveCost), 2),\n BilledCost = round(sum(BilledCost), 2),\n ListUnitPrice = round(take_any(ListUnitPrice), 4),\n ContractedUnitPrice = round(take_any(ContractedUnitPrice), 4),\n x_EffectiveUnitPrice = round(take_any(x_EffectiveUnitPrice), 4),\n PricingQuantity = sum(PricingQuantity)\n by\n x_SkuDescription,\n PricingUnit,\n CommitmentDiscountType,\n x_SkuTerm\n| order by EffectiveCost desc\n| project \n x_SkuDescription,\n // CommitmentDiscountType,\n // x_SkuTerm,\n Quantity = round(PricingQuantity, 4),\n Unit = PricingUnit,\n // Resources = x_ResourceCount,\n List = ListUnitPrice,\n // ContractedUnitPrice,\n // x_EffectiveUnitPrice,\n Discount = round(ListUnitPrice - x_EffectiveUnitPrice, 4),\n // BilledCost,\n Cost = EffectiveCost\n| where Discount == 0\n", - "id": "00ae3917-c783-45f6-a04e-9113e4c5d445", - "usedVariables": [ - "CostsByDay" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let doubleGroupCount = maxGroupCount * 2;\nlet costs = CostsByDay\n| where isnotempty(x_SkuDescription)\n;\nlet all = costs | summarize EffectiveCost = round(sum(EffectiveCost), 2) by x_SkuDescription;\nlet count = toscalar(all | count);\nlet topX = all | order by EffectiveCost desc | limit doubleGroupCount;\ncosts\n//\n// Group rows after max count\n| extend inTopX = x_SkuDescription in (topX)\n| extend x_SkuDescription = iff(inTopX, x_SkuDescription, strcat('(', (count - doubleGroupCount), ' others)'))\n//\n| summarize EffectiveCost = round(sum(EffectiveCost), 2) by SKU = x_SkuDescription\n| order by EffectiveCost desc\n", - "id": "d8a4634e-7ebe-47ce-83b5-ff7f50f6bee6", - "usedVariables": [ - "CostsByDay", - "maxGroupCount" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nCostsByMonth\n| where ChargeCategory == 'Purchase'\n| project\n ChargePeriodStart = substring(ChargePeriodStart, 0, 10),\n x_SkuDescription,\n CommitmentDiscountType,\n Term = case(isempty(x_SkuTerm) or x_SkuTerm <= 0, '', x_SkuTerm < 12, strcat(x_SkuTerm, ' month', iff(x_SkuTerm != 1, 's', '')), strcat(x_SkuTerm / 12, ' year', iff(x_SkuTerm != 12, 's', ''))),\n PricingQuantity,\n BilledCost,\n BillingCurrency\n| order by ChargePeriodStart desc\n", - "id": "98c9b9b5-bebf-41f8-8319-5fa2523f9dd0", - "usedVariables": [ - "CostsByMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nCostsByMonth\n| as costs\n| where ChargeCategory == 'Purchase'\n| where isnotempty(CommitmentDiscountType)\n| project\n ChargePeriodStart = substring(ChargePeriodStart, 0, 10),\n x_SkuDescription,\n CommitmentDiscountType,\n Term = case(isempty(x_SkuTerm) or x_SkuTerm <= 0, '', x_SkuTerm < 12, strcat(x_SkuTerm, ' month', iff(x_SkuTerm != 1, 's', '')), strcat(x_SkuTerm / 12, ' year', iff(x_SkuTerm != 12, 's', ''))),\n ChargeFrequency,\n ChargeClass,\n PricingQuantity,\n Utilization = real(0),\n BilledCost,\n x_SkuOrderId\n| join kind=leftouter (\n costs\n | where ChargeCategory == 'Usage'\n | where isnotempty(CommitmentDiscountId)\n //\n | extend x_CommitmentDiscountUtilizationPotential = case(\n ProviderName == 'Microsoft', EffectiveCost,\n CommitmentDiscountCategory == 'Usage', ConsumedQuantity,\n CommitmentDiscountCategory == 'Spend', EffectiveCost,\n real(0)\n )\n | extend x_CommitmentDiscountUtilizationAmount = iff(CommitmentDiscountStatus == 'Used', x_CommitmentDiscountUtilizationPotential, real(0))\n //\n | summarize\n x_CommitmentDiscountUtilizationAmount = sum(x_CommitmentDiscountUtilizationAmount),\n x_CommitmentDiscountUtilizationPotential = sum(x_CommitmentDiscountUtilizationPotential)\n by\n x_SkuOrderId\n | project\n x_SkuOrderId,\n Utilization = round(x_CommitmentDiscountUtilizationAmount / x_CommitmentDiscountUtilizationPotential * 100, 1)\n) on x_SkuOrderId\n| extend Utilization = coalesce(Utilization1, real(0))\n| project-away x_SkuOrderId, x_SkuOrderId1, Utilization1\n| order by ChargePeriodStart desc\n\n\n", - "id": "35a3a2b3-4ab0-4449-96f7-c78db789089e", - "usedVariables": [ - "CostsByMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let data = materialize(\n CostsByMonth | extend Period = 'Last n months'\n | union (CostsByDay | extend Period = 'Last n days')\n | summarize \n ListCost = round(sum(ListCost), 2),\n ContractedCost = round(sum(ContractedCost), 2),\n EffectiveCost = round(sum(EffectiveCost), 2)\n by\n Period\n | project Period, json = todynamic(strcat('[{ \"Label\":\"List\", \"Value\":', ListCost, ' }, { \"Label\":\"Contracted\", \"Value\":', ContractedCost, ' }, { \"Label\":\"Effective\", \"Value\":', EffectiveCost, ' }]'))\n | mv-expand json\n | project Label = tostring(json.Label), Value = tolong(json.Value), Period\n);\ndata", - "id": "5ff29428-de83-4a2c-8f86-d8beebe68750", - "usedVariables": [ - "CostsByDay", - "CostsByMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let data = materialize(\n CostsByMonth | extend Period = 'Last n months'\n | union (CostsByDay | extend Period = 'Last n days')\n | summarize\n ListCost = round(sum(ListCost), 2),\n ContractedCost = round(sum(ContractedCost), 2),\n EffectiveCost = round(sum(EffectiveCost), 2)\n by\n Period\n | project Period, json = todynamic(strcat('[{ \"Label\":\"Total\", \"Value\":', ListCost - EffectiveCost, ' }, { \"Label\":\"Negotiated\", \"Value\":', ListCost - ContractedCost, ' }, { \"Label\":\"Commitment\", \"Value\":', ContractedCost - EffectiveCost, ' }]'))\n | mv-expand json\n | project Label = tostring(json.Label), Value = tolong(json.Value), Period\n);\ndata\n", - "id": "f5f240a8-a818-4f27-bdfb-e96fcfe433bd", - "usedVariables": [ - "CostsByDay", - "CostsByMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nCostsByDay\n| where ChargeCategory == 'Usage'\n| where isnotempty(CommitmentDiscountId)\n//\n| extend x_CommitmentDiscountUtilizationPotential = case(\n ProviderName == 'Microsoft', EffectiveCost,\n CommitmentDiscountCategory == 'Usage', ConsumedQuantity,\n CommitmentDiscountCategory == 'Spend', EffectiveCost,\n real(0)\n)\n| extend x_CommitmentDiscountUtilizationAmount = iff(CommitmentDiscountStatus == 'Used', x_CommitmentDiscountUtilizationPotential, real(0))\n//\n| summarize\n CommitmentDiscountName = take_any(CommitmentDiscountName),\n CommitmentDiscountType = take_any(CommitmentDiscountType),\n x_SkuTerm = take_any(x_SkuTerm),\n ListCost = sum(ListCost),\n ContractedCost = sum(ContractedCost),\n EffectiveCost = sum(EffectiveCost),\n x_CommitmentDiscountUtilizationAmount = sum(x_CommitmentDiscountUtilizationAmount),\n x_CommitmentDiscountUtilizationPotential = sum(x_CommitmentDiscountUtilizationPotential)\n by\n CommitmentDiscountId\n| order by EffectiveCost desc\n| project\n CommitmentDiscountName,\n CommitmentDiscountType,\n Term = case(isempty(x_SkuTerm) or x_SkuTerm <= 0, '', x_SkuTerm < 12, strcat(x_SkuTerm, ' month', iff(x_SkuTerm != 1, 's', '')), strcat(x_SkuTerm / 12, ' year', iff(x_SkuTerm != 12, 's', ''))),\n Utilization = round(x_CommitmentDiscountUtilizationAmount / x_CommitmentDiscountUtilizationPotential * 100, 1),\n Cost = round(EffectiveCost, 2),\n Savings = round(ListCost - EffectiveCost, 2)\n", - "id": "d7f31381-ba44-46a1-ad3e-dc6b8826706d", - "usedVariables": [ - "CostsByDay" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let data = materialize(\n CostsByMonth\n //\n // Don't double-count commitment discount purchases\n | where x_AmortizationClass != 'Principal'\n //\n | summarize \n ListCost = sum(ListCost),\n ContractedCost = sum(ContractedCost),\n EffectiveCost = sum(EffectiveCost)\n | extend CommitmentDiscountSavings = ContractedCost - EffectiveCost\n | extend NegotiatedDiscountSavings = ListCost - ContractedCost\n | extend TotalSavings = ListCost - EffectiveCost\n | project json = todynamic(strcat('[',\n '{ \"order\":11, \"type\":\"List\", \"label\":\"Cost without discounts\", \"value\":\"', numberstring(round(ListCost, 2)), '\" },',\n '{ \"order\":12, \"type\":\"\", \"label\":\"\", \"value\":\"\u2796\" },',\n '{ \"order\":13, \"type\":\"Contracted\", \"label\":\"After negotiated discounts\", \"value\":\"', numberstring(round(ContractedCost, 2)), '\" },',\n '{ \"order\":14, \"type\":\"\", \"label\":\"\", \"value\":\"\ud83d\udff0\" },',\n '{ \"order\":15, \"type\":\"PartialSavings\", \"label\":\"Negotiated savings\", \"value\":\"', numberstring(round(NegotiatedDiscountSavings, 2)), '\" },',\n //\n '{ \"order\":21, \"type\":\"Contracted\", \"label\":\"After negotiated discounts\", \"value\":\"', numberstring(round(ContractedCost, 2)), '\" },',\n '{ \"order\":22, \"type\":\"\", \"label\":\"\", \"value\":\"\u2796\" },',\n '{ \"order\":23, \"type\":\"Effective\", \"label\":\"After commitment discounts\", \"value\":\"', numberstring(round(EffectiveCost, 2)), '\" },',\n '{ \"order\":24, \"type\":\"\", \"label\":\"\", \"value\":\"\ud83d\udff0\" },',\n '{ \"order\":25, \"type\":\"PartialSavings\", \"label\":\"Commitment savings\", \"value\":\"', numberstring(round(CommitmentDiscountSavings, 2)), '\" },',\n //\n '{ \"order\":31, \"type\":\"List\", \"label\":\"Cost without discounts\", \"value\":\"', numberstring(round(ListCost, 2)), '\" },',\n '{ \"order\":32, \"type\":\"\", \"label\":\"\", \"value\":\"\u2796\" },',\n '{ \"order\":33, \"type\":\"Effective\", \"label\":\"After commitment discounts\", \"value\":\"', numberstring(round(EffectiveCost, 2)), '\" },',\n '{ \"order\":34, \"type\":\"\", \"label\":\"\", \"value\":\"\ud83d\udff0\" },',\n '{ \"order\":35, \"type\":\"TotalSavings\", \"label\":\"Total savings\", \"value\":\"', numberstring(round(TotalSavings, 2)), '\" }',\n ']'))\n | mv-expand json\n | order by toint(json.order) asc\n | project Label = tostring(json.label), Value = tostring(json.value), Type = tostring(json.type)\n);\ndata", - "id": "e17346d1-0227-4aa4-8765-f9b4bf43bed5", - "usedVariables": [ - "CostsByMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let data = materialize(\n CostsByDay\n //\n // Don't double-count commitment discount purchases\n | where ChargeCategory == 'Usage' or isempty(CommitmentDiscountId)\n //\n | summarize \n ListCost = sum(ListCost),\n ContractedCost = sum(ContractedCost),\n EffectiveCost = sum(EffectiveCost)\n | extend CommitmentDiscountSavings = ContractedCost - EffectiveCost\n | extend NegotiatedDiscountSavings = ListCost - ContractedCost\n | extend TotalSavings = ListCost - EffectiveCost\n | project json = todynamic(strcat('[',\n '{ \"order\":11, \"type\":\"List\", \"label\":\"Cost without discounts\", \"value\":\"', numberstring(round(ListCost, 2)), '\" },',\n '{ \"order\":12, \"type\":\"\", \"label\":\"\", \"value\":\"\u2796\" },',\n '{ \"order\":13, \"type\":\"Contracted\", \"label\":\"After negotiated discounts\", \"value\":\"', numberstring(round(ContractedCost, 2)), '\" },',\n '{ \"order\":14, \"type\":\"\", \"label\":\"\", \"value\":\"\ud83d\udff0\" },',\n '{ \"order\":15, \"type\":\"PartialSavings\", \"label\":\"Negotiated savings\", \"value\":\"', numberstring(round(NegotiatedDiscountSavings, 2)), '\" },',\n //\n '{ \"order\":21, \"type\":\"Contracted\", \"label\":\"After negotiated discounts\", \"value\":\"', numberstring(round(ContractedCost, 2)), '\" },',\n '{ \"order\":22, \"type\":\"\", \"label\":\"\", \"value\":\"\u2796\" },',\n '{ \"order\":23, \"type\":\"Effective\", \"label\":\"After commitment discounts\", \"value\":\"', numberstring(round(EffectiveCost, 2)), '\" },',\n '{ \"order\":24, \"type\":\"\", \"label\":\"\", \"value\":\"\ud83d\udff0\" },',\n '{ \"order\":25, \"type\":\"PartialSavings\", \"label\":\"Commitment savings\", \"value\":\"', numberstring(round(CommitmentDiscountSavings, 2)), '\" },',\n //\n '{ \"order\":31, \"type\":\"List\", \"label\":\"Cost without discounts\", \"value\":\"', numberstring(round(ListCost, 2)), '\" },',\n '{ \"order\":32, \"type\":\"\", \"label\":\"\", \"value\":\"\u2796\" },',\n '{ \"order\":33, \"type\":\"Effective\", \"label\":\"After commitment discounts\", \"value\":\"', numberstring(round(EffectiveCost, 2)), '\" },',\n '{ \"order\":34, \"type\":\"\", \"label\":\"\", \"value\":\"\ud83d\udff0\" },',\n '{ \"order\":35, \"type\":\"TotalSavings\", \"label\":\"Total savings\", \"value\":\"', numberstring(round(TotalSavings, 2)), '\" }',\n ']'))\n | mv-expand json\n | order by toint(json.order) asc\n | project Label = tostring(json.label), Value = tostring(json.value), Type = tostring(json.type)\n);\ndata", - "id": "f1ef29df-7a1d-4dd0-8619-0c0164707b31", - "usedVariables": [ - "CostsByDay" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let data = materialize(\n CostsByMonth\n //\n | where isnotempty(CommitmentDiscountStatus)\n //\n // Guarantee there's a row for every combination\n | union (\n print json = dynamic([\n {\"order\": 11, \"CommitmentDiscountType\": \"Reservation\", \"CommitmentDiscountStatus\": \"Used\"},\n {\"order\": 12, \"CommitmentDiscountType\": \"Reservation\", \"CommitmentDiscountStatus\": \"Unused\"},\n {\"order\": 21, \"CommitmentDiscountType\": \"Savings Plan\", \"CommitmentDiscountStatus\": \"Used\"},\n {\"order\": 22, \"CommitmentDiscountType\": \"Savings Plan\", \"CommitmentDiscountStatus\": \"Unused\"}\n ])\n | mv-expand json\n | evaluate bag_unpack(json)\n | extend EffectiveCost = toreal(0)\n )\n //\n | summarize Value = sum(EffectiveCost), order = sum(order) by CommitmentDiscountStatus, CommitmentDiscountType\n | order by order asc\n | project Label = strcat(CommitmentDiscountStatus, ' ', tolower(CommitmentDiscountType), 's'), Value\n);\ndata", - "id": "7d0c2c1f-338b-4534-b173-36b284779131", - "usedVariables": [ - "CostsByMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let data = materialize(\n CostsByDay\n //\n // Don't double-count commitment discount purchases\n | where isnotempty(CommitmentDiscountStatus)\n //\n // Guarantee there's a row for every combination\n | union (\n print json = dynamic([\n {\"order\": 11, \"CommitmentDiscountType\": \"Reservation\", \"CommitmentDiscountStatus\": \"Used\"},\n {\"order\": 12, \"CommitmentDiscountType\": \"Reservation\", \"CommitmentDiscountStatus\": \"Unused\"},\n {\"order\": 21, \"CommitmentDiscountType\": \"Savings Plan\", \"CommitmentDiscountStatus\": \"Used\"},\n {\"order\": 22, \"CommitmentDiscountType\": \"Savings Plan\", \"CommitmentDiscountStatus\": \"Unused\"}\n ])\n | mv-expand json\n | evaluate bag_unpack(json)\n | extend EffectiveCost = toreal(0)\n )\n //\n | summarize Value = sum(EffectiveCost), order = sum(order) by CommitmentDiscountStatus, CommitmentDiscountType\n | order by order asc\n | project Label = strcat(CommitmentDiscountStatus, ' ', tolower(CommitmentDiscountType), 's'), Value\n);\ndata", - "id": "84ecad69-79ac-45b9-a8af-60be28dcc748", - "usedVariables": [ - "CostsByDay" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet data = (\n CostsByMonth\n | summarize \n EffectiveCost = sum(EffectiveCost)\n by\n ChargePeriodStart = startofmonth(ChargePeriodStart),\n ServiceCategory,\n ServiceName\n | as per\n | union (\n per\n | summarize \n EffectiveCost = sum(EffectiveCost)\n by\n ServiceCategory,\n ServiceName\n )\n | order by ChargePeriodStart asc\n | extend EffectiveCost = todouble(round(EffectiveCost, 2))\n | extend ChargePeriod = iff(isempty(ChargePeriodStart), strcat('Total'), strcat(format_datetime(ChargePeriodStart, 'yyyy-MM - '), monthname[monthofyear(ChargePeriodStart)]))\n);\ndata | evaluate pivot(ChargePeriod, sum(EffectiveCost), ServiceCategory)\n| order by Total desc", - "id": "85320475-730b-411f-98d5-fbc345bf5e68", - "usedVariables": [ - "CostsByMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "CostsByMonth\n//\n// Filter out commitment discount purchases\n| where ChargeCategory == 'Usage' or isempty(CommitmentDiscountId)\n//\n| summarize \n ListCost = sum(ListCost),\n ContractedCost = sum(ContractedCost),\n EffectiveCost = sum(EffectiveCost)\n by\n ChargePeriodStart = startofmonth(ChargePeriodStart),\n CommitmentDiscountType\n| project\n ChargePeriodStart,\n Savings = round(ListCost - EffectiveCost, 2),\n Type = case(\n isnotempty(CommitmentDiscountType), CommitmentDiscountType,\n 'Negotiated'\n )", - "id": "5bbb5369-ac95-45fc-853c-a6a2ce6a9e7b", - "usedVariables": [ - "CostsByMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "CostsByMonth\n//\n// Filter out commitment discount purchases\n| where ChargeCategory == 'Usage' or isempty(CommitmentDiscountId)\n//\n| summarize \n Savings = round(sum(ListCost - EffectiveCost), 2)\n by\n Type = case(\n isnotempty(CommitmentDiscountType), CommitmentDiscountType,\n 'Negotiated'\n )", - "id": "93f0eb9f-fa4d-4c54-96c2-8c1d3387d13a", - "usedVariables": [ - "CostsByMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "CostsByDay\n//\n// Filter out commitment discount purchases\n| where ChargeCategory == 'Usage' or isempty(CommitmentDiscountId)\n//\n| summarize \n ListCost = sum(ListCost),\n ContractedCost = sum(ContractedCost),\n EffectiveCost = sum(EffectiveCost)\n by\n ChargePeriodStart = startofday(ChargePeriodStart),\n CommitmentDiscountType\n| project\n ChargePeriodStart,\n Savings = round(ListCost - EffectiveCost, 2),\n Type = case(\n isnotempty(CommitmentDiscountType), CommitmentDiscountType,\n 'Negotiated'\n )", - "id": "5c442903-65b0-4b53-9e7c-3ea9c1af7be5", - "usedVariables": [ - "CostsByDay" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "CostsByDay\n//\n// Filter out commitment discount purchases\n| where ChargeCategory == 'Usage' or isempty(CommitmentDiscountId)\n//\n| summarize \n Savings = round(sum(ListCost - EffectiveCost), 2)\n by\n Type = case(\n isnotempty(CommitmentDiscountType), CommitmentDiscountType,\n 'Negotiated'\n )", - "id": "9a9ee4b0-a37d-475f-bf1d-f51573243491", - "usedVariables": [ - "CostsByDay" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet data = (\n CostsByMonth\n | where ChargeCategory == 'Usage'\n | where isnotempty(CommitmentDiscountId)\n | where EffectiveCost != 0\n | extend x_ResourceGroupId = strcat(SubAccountId, '/resourcegroups/', x_ResourceGroupName)\n | summarize \n EffectiveCost = sum(EffectiveCost),\n SubAccountName = take_anyif(SubAccountName, isnotempty(SubAccountId)),\n x_ResourceGroupName = take_any(x_ResourceGroupName),\n CommitmentDiscountName = take_any(CommitmentDiscountName),\n CommitmentDiscountType = take_any(CommitmentDiscountType)\n by\n ChargePeriodStart,\n SubAccountId,\n x_ResourceGroupId,\n CommitmentDiscountId,\n CommitmentDiscountStatus\n | as per\n | union (\n per\n | summarize \n EffectiveCost = sum(EffectiveCost),\n SubAccountName = take_anyif(SubAccountName, isnotempty(SubAccountId)),\n x_ResourceGroupName = take_any(x_ResourceGroupName),\n CommitmentDiscountName = take_any(CommitmentDiscountName),\n CommitmentDiscountType = take_any(CommitmentDiscountType)\n by\n x_ResourceGroupId,\n CommitmentDiscountId,\n CommitmentDiscountStatus\n )\n | order by ChargePeriodStart asc\n | extend EffectiveCost = todouble(round(EffectiveCost, 2))\n | extend ChargePeriod = iff(isempty(ChargePeriodStart), strcat('Total'), strcat(format_datetime(ChargePeriodStart, 'yyyy-MM - '), monthname[monthofyear(ChargePeriodStart)]))\n | extend x_ResourceGroupName = iff(isempty(x_ResourceGroupName) and CommitmentDiscountStatus == 'Unused', '(Unused)', x_ResourceGroupName)\n | extend SubAccountName = iff(isempty(SubAccountName) and CommitmentDiscountStatus == 'Unused', '(Unused)', SubAccountName)\n);\npercent((\n data | evaluate pivot(ChargePeriod, sum(EffectiveCost), x_ResourceGroupName, SubAccountName, CommitmentDiscountName, CommitmentDiscountType)\n | extend Count = tolong(Total * 1000)\n))\n| project-away Count\n| order by x_ResourceGroupName asc, SubAccountName asc, Total desc", - "id": "377e3693-0738-45d7-97d9-4b6e71ec5b36", - "usedVariables": [ - "CostsByMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet data = (\n CostsByMonth\n | where ChargeCategory == 'Usage'\n | where isnotempty(CommitmentDiscountId)\n | where EffectiveCost != 0\n | summarize \n BilledCost = sum(BilledCost),\n EffectiveCost = sum(EffectiveCost),\n ContractedCost = sum(ContractedCost),\n ListCost = sum(ListCost),\n SubAccountName = take_anyif(SubAccountName, isnotempty(SubAccountId)),\n CommitmentDiscountName = take_any(CommitmentDiscountName),\n CommitmentDiscountType = take_any(CommitmentDiscountType)\n by\n ChargePeriodStart,\n SubAccountId,\n CommitmentDiscountId,\n CommitmentDiscountStatus\n | as per\n | union (\n per\n | summarize \n BilledCost = sum(BilledCost),\n EffectiveCost = sum(EffectiveCost),\n ContractedCost = sum(ContractedCost),\n ListCost = sum(ListCost),\n SubAccountName = take_anyif(SubAccountName, isnotempty(SubAccountId)),\n CommitmentDiscountName = take_any(CommitmentDiscountName),\n CommitmentDiscountType = take_any(CommitmentDiscountType)\n by\n SubAccountId,\n CommitmentDiscountId,\n CommitmentDiscountStatus\n )\n | order by ChargePeriodStart asc\n | extend BilledCost = todouble(round(BilledCost, 2))\n | extend EffectiveCost = todouble(round(EffectiveCost, 2))\n | extend ContractedCost = todouble(round(ContractedCost, 2))\n | extend ListCost = todouble(round(ListCost, 2))\n | extend CommitmentDiscountSavings = todouble(round(ContractedCost - EffectiveCost, 2))\n | extend NegotiatedDiscountSavings = todouble(round(ListCost - ContractedCost, 2))\n | extend ChargePeriod = iff(isempty(ChargePeriodStart), strcat('Total'), strcat(format_datetime(ChargePeriodStart, 'yyyy-MM - '), monthname[monthofyear(ChargePeriodStart)]))\n | extend SubAccountName = iff(isempty(SubAccountName) and CommitmentDiscountStatus == 'Unused', '(Unused)', SubAccountName)\n);\npercent((\n data | evaluate pivot(ChargePeriod, sum(EffectiveCost), SubAccountName, CommitmentDiscountName, CommitmentDiscountType)\n | extend Count = tolong(Total * 1000)\n))\n| project-away Count\n| order by SubAccountName asc, Total desc", - "id": "13cad52d-91e7-4ab4-aad5-4aae21c1019a", - "usedVariables": [ - "CostsByMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "CostsByDay\n| where ChargeCategory == 'Usage'\n| where isnotempty(CommitmentDiscountId)\n| where EffectiveCost != 0\n| summarize \n EffectiveCost = sum(EffectiveCost),\n ResourceName = take_any(ResourceName),\n ResourceType = take_any(ResourceType),\n RegionName = take_any(RegionName),\n x_ResourceGroupName = take_any(x_ResourceGroupName),\n SubAccountName = take_any(SubAccountName),\n CommitmentDiscountName = take_any(CommitmentDiscountName),\n CommitmentDiscountType = take_any(CommitmentDiscountType)\n by\n ResourceId,\n CommitmentDiscountId\n| project \n CommitmentDiscountType,\n CommitmentDiscountName,\n ResourceName,\n ResourceType,\n RegionName,\n x_ResourceGroupName,\n SubAccountName,\n EffectiveCost = round(EffectiveCost, 2)\n| order by CommitmentDiscountType asc, CommitmentDiscountName asc, ResourceName asc, ResourceType asc, x_ResourceGroupName asc, SubAccountName asc, EffectiveCost desc", - "id": "dcf69b47-233e-4bc4-b914-3f6f09f4cb82", - "usedVariables": [ - "CostsByDay" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let costs = CostsByMonth\n| extend x_ToolkitTool = tostring(Tags['ftk-tool'])\n| where x_ToolkitTool == 'FinOps hubs'\n| extend x_ToolkitVersion = tostring(Tags['ftk-version'])\n| extend x_ResourceParentId = tostring(Tags['cm-resource-parent'])\n| extend x_ResourceParentName = database('Ingestion').parse_resourceid(x_ResourceParentId).ResourceName\n;\nlet all = costs | summarize sum(EffectiveCost) by x_ResourceParentId;\nlet count = toscalar(all | order by sum_EffectiveCost desc | count);\nlet topX = all | order by sum_EffectiveCost desc | limit maxGroupCount;\nlet otherId = '(others)';\ncosts\n//\n// Group rows after max count\n| extend inTopX = x_ResourceParentId in (topX)\n| extend x_ResourceParentId = iff(inTopX, x_ResourceParentId, otherId)\n| extend x_ResourceParentName = iff(inTopX, x_ResourceParentName, strcat('(', (count - maxGroupCount), ' others)'))\n//\n| summarize \n EffectiveCost = round(sum(EffectiveCost), 2),\n ResourceType = take_any(ResourceType),\n x_ResourceParentName = take_any(x_ResourceParentName),\n x_ResourceGroupName = take_any(x_ResourceGroupName),\n SubAccountName = take_any(SubAccountName)\n by\n ChargePeriodStart,\n x_ResourceParentId\n| project ChargePeriodStart, EffectiveCost, Hub = iff(x_ResourceParentId == otherId, x_ResourceParentName, strcat(x_ResourceParentName, ' (', x_ResourceGroupName, ' / ', SubAccountName, ')'))\n| order by EffectiveCost desc\n", - "id": "d7a9ff96-da17-4826-9fb5-b23f4c7b938d", - "usedVariables": [ - "CostsByMonth", - "maxGroupCount" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let costs = CostsByDay\n| extend x_ToolkitTool = tostring(Tags['ftk-tool'])\n| where x_ToolkitTool == 'FinOps hubs'\n| extend x_ToolkitVersion = tostring(Tags['ftk-version'])\n| extend x_ResourceParentId = tostring(Tags['cm-resource-parent'])\n| extend x_ResourceParentName = database('Ingestion').parse_resourceid(x_ResourceParentId).ResourceName\n;\nlet all = costs | summarize sum(EffectiveCost) by x_ResourceParentId;\nlet count = toscalar(all | order by sum_EffectiveCost desc | count);\nlet topX = all | order by sum_EffectiveCost desc | limit maxGroupCount;\nlet otherId = '(others)';\ncosts\n//\n// Group rows after max count\n| extend inTopX = x_ResourceParentId in (topX)\n| extend x_ResourceParentId = iff(inTopX, x_ResourceParentId, otherId)\n| extend x_ResourceParentName = iff(inTopX, x_ResourceParentName, strcat('(', (count - maxGroupCount), ' others)'))\n//\n| summarize \n EffectiveCost = round(sum(EffectiveCost), 2),\n ResourceType = take_any(ResourceType),\n x_ResourceParentName = take_any(x_ResourceParentName),\n x_ResourceGroupName = take_any(x_ResourceGroupName),\n SubAccountName = take_any(SubAccountName)\n by\n ChargePeriodStart,\n x_ResourceParentId\n| project ChargePeriodStart, EffectiveCost, Hub = iff(x_ResourceParentId == otherId, x_ResourceParentName, strcat(x_ResourceParentName, ' (', x_ResourceGroupName, ' / ', SubAccountName, ')'))\n| order by EffectiveCost desc\n", - "id": "13813a00-e634-4428-9eac-ea255fd1eaca", - "usedVariables": [ - "CostsByDay", - "maxGroupCount" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let monthname = dynamic(['', 'January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']);\nlet costs = CostsByMonth\n| extend x_ToolkitTool = tostring(Tags['ftk-tool'])\n| where x_ToolkitTool == 'FinOps hubs'\n| extend x_ToolkitVersion = tostring(Tags['ftk-version'])\n| extend x_ResourceParentId = tostring(Tags['cm-resource-parent'])\n| extend x_ResourceParentName = tostring(database('Ingestion').parse_resourceid(x_ResourceParentId).ResourceName)\n;\nlet data = (\n costs\n | summarize \n EffectiveCost = sum(EffectiveCost),\n x_ResourceParentName = take_any(x_ResourceParentName),\n RegionName = take_any(RegionName),\n x_ResourceGroupName = take_any(x_ResourceGroupName),\n SubAccountName = take_any(SubAccountName),\n x_ToolkitVersion = take_any(x_ToolkitVersion)\n by\n ChargePeriodStart,\n x_ResourceParentId\n | as per\n //\n // Append total\n | union (\n per\n | summarize \n EffectiveCost = sum(EffectiveCost)\n by\n x_ResourceParentId,\n x_ResourceParentName,\n x_ToolkitVersion,\n RegionName,\n x_ResourceGroupName,\n SubAccountName\n )\n | order by ChargePeriodStart asc\n | extend EffectiveCost = todouble(round(EffectiveCost, 2))\n | extend ChargePeriod = iff(isempty(ChargePeriodStart), strcat('Total'), strcat(format_datetime(ChargePeriodStart, 'yyyy-MM - '), monthname[monthofyear(ChargePeriodStart)]))\n);\ndata | evaluate pivot(ChargePeriod, sum(EffectiveCost), x_ResourceParentName, x_ToolkitVersion, RegionName, x_ResourceGroupName, SubAccountName)\n| project-rename Name = x_ResourceParentName, Version = x_ToolkitVersion\n| order by Total desc\n", - "id": "62bbceee-c089-45db-822c-0b4745358aa4", - "usedVariables": [ - "CostsByMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": ".show cluster extents\n| where TableName contains '_final_v'\n| extend Dataset = tostring(split(TableName, '_final_v')[0])\n| extend FocusVersion = replace_string(tostring(split(TableName, '_final_v')[1]), '_', '.')\n| extend ToolkitVersion = tostring(extract(@'drop-by:ftk-version-([^\\s]+)', 1, Tags))\n// | extend Table = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 1, Tags))\n| extend Date = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 2, Tags))\n| extend Scope = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 5, Tags))\n| extend Date = todatetime(strcat(replace_string(trim(@'/', Date), '/', '-'), '-01'))\n| extend Scope = replace_regex(Scope, @'/[^/]+$', '')\n| project Dataset, FocusVersion, ToolkitVersion, Date, Scope, LastUpdate = MaxCreatedOn, OriginalSize, RowCount\n| extend ScopeId = database('Ingestion').parse_resourceid(Scope).ResourceName\n// TODO: Look up the name from cost data\n| extend ScopeResourceType = database('Ingestion').parse_resourceid(Scope).x_ResourceType\n// TODO: Clean up scope resource type display names -- | extend ScopeType = resource_type(ScopeResourceType).SingularDisplayName\n| summarize Rows = sum(RowCount) by Date, Dataset", - "id": "b98c9ed7-47b3-41c1-a596-fd9d32725b33", - "usedVariables": [] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": ".show cluster extents\n| where TableName contains '_final_v'\n| extend Dataset = tostring(split(TableName, '_final_v')[0])\n| where Dataset == 'Costs'\n| extend FocusVersion = replace_string(tostring(split(TableName, '_final_v')[1]), '_', '.')\n| extend ToolkitVersion = tostring(extract(@'drop-by:ftk-version-([^\\s]+)', 1, Tags))\n// | extend Table = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 1, Tags))\n| extend Date = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 2, Tags))\n| extend Scope = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 5, Tags))\n| extend Date = todatetime(strcat(replace_string(trim(@'/', Date), '/', '-'), '-01'))\n| extend Scope = replace_regex(Scope, @'/[^/]+$', '')\n| project Dataset, FocusVersion, ToolkitVersion, Date, Scope, LastUpdate = MaxCreatedOn, OriginalSize, RowCount\n| extend ScopeId = tostring(database('Ingestion').parse_resourceid(Scope).ResourceName)\n| extend ScopeResourceType = database('Ingestion').parse_resourceid(Scope).x_ResourceType\n// TODO: Clean up scope resource type display names -- | extend ScopeType = resource_type(ScopeResourceType).SingularDisplayName\n| summarize Rows = sum(RowCount) by Date, ScopeId", - "id": "187e22fe-3d78-45a2-a2bf-090ece144fd1", - "usedVariables": [] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": ".show cluster extents\n| where TableName contains '_final_v'\n| extend Dataset = tostring(split(TableName, '_final_v')[0])\n| where Dataset == 'Costs'\n| extend FocusVersion = replace_string(tostring(split(TableName, '_final_v')[1]), '_', '.')\n| extend ToolkitVersion = tostring(extract(@'drop-by:ftk-version-([^\\s]+)', 1, Tags))\n// | extend Table = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 1, Tags))\n| extend Date = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 2, Tags))\n| extend Scope = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 5, Tags))\n| extend Date = todatetime(strcat(replace_string(trim(@'/', Date), '/', '-'), '-01'))\n| extend Scope = replace_regex(Scope, @'/[^/]+$', '')\n| project Dataset, FocusVersion, ToolkitVersion, Date, Scope, LastUpdate = MaxCreatedOn, OriginalSize, RowCount\n| extend ScopeId = tostring(database('Ingestion').parse_resourceid(Scope).ResourceName)\n// TODO: Look up the name from cost data\n| extend ScopeResourceType = database('Ingestion').parse_resourceid(Scope).x_ResourceType\n// TODO: Clean up scope resource type display names -- | extend ScopeType = resource_type(ScopeResourceType).SingularDisplayName\n| summarize dcount(Scope)", - "id": "d2c522dc-ef4b-4714-a473-09350e275557", - "usedVariables": [] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": ".show cluster extents\n| where TableName contains '_final_v'\n| extend Dataset = tostring(split(TableName, '_final_v')[0])\n| where Dataset == 'Costs'\n| extend FocusVersion = replace_string(tostring(split(TableName, '_final_v')[1]), '_', '.')\n| extend ToolkitVersion = tostring(extract(@'drop-by:ftk-version-([^\\s]+)', 1, Tags))\n// | extend Table = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 1, Tags))\n| extend Date = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 2, Tags))\n| extend Scope = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 5, Tags))\n| extend Date = todatetime(strcat(replace_string(trim(@'/', Date), '/', '-'), '-01'))\n| extend Scope = replace_regex(Scope, @'/[^/]+$', '')\n| project Dataset, FocusVersion, ToolkitVersion, Date, Scope, LastUpdate = MaxCreatedOn, OriginalSize, RowCount\n| extend ScopeId = tostring(database('Ingestion').parse_resourceid(Scope).ResourceName)\n// TODO: Look up the name from cost data\n| extend ScopeResourceType = database('Ingestion').parse_resourceid(Scope).x_ResourceType\n// TODO: Clean up scope resource type display names -- | extend ScopeType = resource_type(ScopeResourceType).SingularDisplayName\n| summarize dcount(startofmonth(Date))", - "id": "6fe11b78-82ef-4e9a-8411-a65e5b0d8bba", - "usedVariables": [] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": ".show cluster extents\n| where TableName contains '_final_v'\n| extend Dataset = tostring(split(TableName, '_final_v')[0])\n| where Dataset == 'Prices'\n| extend FocusVersion = replace_string(tostring(split(TableName, '_final_v')[1]), '_', '.')\n| extend ToolkitVersion = tostring(extract(@'drop-by:ftk-version-([^\\s]+)', 1, Tags))\n// | extend Table = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 1, Tags))\n| extend Date = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 2, Tags))\n| extend Scope = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 5, Tags))\n| extend Date = todatetime(strcat(replace_string(trim(@'/', Date), '/', '-'), '-01'))\n| extend Scope = replace_regex(Scope, @'/[^/]+$', '')\n| project Dataset, FocusVersion, ToolkitVersion, Date, Scope, LastUpdate = MaxCreatedOn, OriginalSize, RowCount\n| extend ScopeId = tostring(database('Ingestion').parse_resourceid(Scope).ResourceName)\n// TODO: Look up the name from cost data\n| extend ScopeResourceType = database('Ingestion').parse_resourceid(Scope).x_ResourceType\n// TODO: Clean up scope resource type display names -- | extend ScopeType = resource_type(ScopeResourceType).SingularDisplayName\n| summarize Rows = sum(RowCount) by Date, ScopeId", - "id": "b3621977-59be-4f98-a034-a94479612115", - "usedVariables": [] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": ".show cluster extents\n| where TableName contains '_final_v'\n| extend Dataset = tostring(split(TableName, '_final_v')[0])\n| where Dataset == 'Recommendations'\n| extend FocusVersion = replace_string(tostring(split(TableName, '_final_v')[1]), '_', '.')\n| extend ToolkitVersion = tostring(extract(@'drop-by:ftk-version-([^\\s]+)', 1, Tags))\n// | extend Table = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 1, Tags))\n| extend Date = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 2, Tags))\n| extend Scope = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 5, Tags))\n| extend Date = todatetime(strcat(replace_string(trim(@'/', Date), '/', '-'), '-01'))\n| extend Scope = replace_regex(Scope, @'/[^/]+$', '')\n| project Dataset, FocusVersion, ToolkitVersion, Date, Scope, LastUpdate = MaxCreatedOn, OriginalSize, RowCount\n| extend ScopeId = tostring(database('Ingestion').parse_resourceid(Scope).ResourceName)\n// TODO: Look up the name from cost data\n| extend ScopeResourceType = database('Ingestion').parse_resourceid(Scope).x_ResourceType\n// TODO: Clean up scope resource type display names -- | extend ScopeType = resource_type(ScopeResourceType).SingularDisplayName\n| summarize Rows = sum(RowCount) by Date, ScopeId", - "id": "4a2a77fe-e819-4059-b027-dab0a1770a0a", - "usedVariables": [] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": ".show cluster extents\n| where TableName contains '_final_v'\n| extend Dataset = tostring(split(TableName, '_final_v')[0])\n| where Dataset == 'Transactions'\n| extend FocusVersion = replace_string(tostring(split(TableName, '_final_v')[1]), '_', '.')\n| extend ToolkitVersion = tostring(extract(@'drop-by:ftk-version-([^\\s]+)', 1, Tags))\n// | extend Table = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 1, Tags))\n| extend Date = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 2, Tags))\n| extend Scope = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 5, Tags))\n| extend Date = todatetime(strcat(replace_string(trim(@'/', Date), '/', '-'), '-01'))\n| extend Scope = replace_regex(Scope, @'/[^/]+$', '')\n| project Dataset, FocusVersion, ToolkitVersion, Date, Scope, LastUpdate = MaxCreatedOn, OriginalSize, RowCount\n| extend ScopeId = tostring(database('Ingestion').parse_resourceid(Scope).ResourceName)\n// TODO: Look up the name from cost data\n| extend ScopeResourceType = database('Ingestion').parse_resourceid(Scope).x_ResourceType\n// TODO: Clean up scope resource type display names -- | extend ScopeType = resource_type(ScopeResourceType).SingularDisplayName\n| summarize Rows = sum(RowCount) by Date, ScopeId", - "id": "8f1ad1ef-6f5e-4f20-a1c4-3941e871d0cd", - "usedVariables": [] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": ".show cluster extents\n| where TableName contains '_final_v'\n| extend Dataset = tostring(split(TableName, '_final_v')[0])\n| where Dataset == 'CommitmentDiscountUsage'\n| extend FocusVersion = replace_string(tostring(split(TableName, '_final_v')[1]), '_', '.')\n| extend ToolkitVersion = tostring(extract(@'drop-by:ftk-version-([^\\s]+)', 1, Tags))\n// | extend Table = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 1, Tags))\n| extend Date = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 2, Tags))\n| extend Scope = tostring(extract(@'drop-by:([^/\\s]+)(/[0-9]{4}(/[0-9]{2}(/[0-9]{2})?)?)?(/[^\\s]+)', 5, Tags))\n| extend Date = todatetime(strcat(replace_string(trim(@'/', Date), '/', '-'), '-01'))\n| extend Scope = replace_regex(Scope, @'/[^/]+$', '')\n| project Dataset, FocusVersion, ToolkitVersion, Date, Scope, LastUpdate = MaxCreatedOn, OriginalSize, RowCount\n| extend ScopeId = tostring(database('Ingestion').parse_resourceid(Scope).ResourceName)\n// TODO: Look up the name from cost data\n| extend ScopeResourceType = database('Ingestion').parse_resourceid(Scope).x_ResourceType\n// TODO: Clean up scope resource type display names -- | extend ScopeType = resource_type(ScopeResourceType).SingularDisplayName\n| summarize Rows = sum(RowCount) by Date, ScopeId", - "id": "e9a9a135-6a74-4463-b69f-eac6930ff1f2", - "usedVariables": [] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let costs = CostsByDayAHB\n| where isnotempty(x_SkuLicenseStatus)\n//\n// Get the latest resource record first to guarrantee we have the latest status\n| summarize arg_max(ChargePeriodStart, *) by ResourceId\n| summarize\n x_ResourceCount = dcount(ResourceId),\n x_SkuLicenseUnusedQuantity = sum(x_SkuLicenseUnusedQuantity)\n by\n x_SkuLicenseStatus,\n x_SkuLicenseQuantity,\n x_SkuCoreCount\n| union (\n print json = dynamic([\n { \"x_SkuLicenseStatus\": \"Enabled\" },\n { \"x_SkuLicenseStatus\": \"Not Enabled\" }\n ])\n | mv-expand json\n | evaluate bag_unpack(json)\n);\n//\n// Coverage first\ncosts\n| summarize covered = sumif(x_ResourceCount, x_SkuLicenseStatus == 'Enabled'), all = sum(x_ResourceCount)\n| project Order = 1, Label = 'Coverage %', Value = percentstring(covered, all, 1)\n//\n// First column\n| union (costs | where x_SkuLicenseStatus == 'Enabled' | summarize Value = numberstring(sum(x_ResourceCount)) by Order = 3, Label = 'Covered resources')\n| union (costs | where x_SkuLicenseStatus == 'Not Enabled' | summarize Value = numberstring(sum(x_ResourceCount)) by Order = 5, Label = 'Eligible resources')\n//\n// Second column\n| union (costs | where x_SkuLicenseStatus == 'Enabled' | summarize Value = numberstring(sum(x_SkuLicenseUnusedQuantity)) by Order = 2, Label = 'Underutilized vCPU capacity')\n| union (costs | where x_SkuLicenseStatus == 'Enabled' | summarize Value = numberstring(sum(x_SkuLicenseQuantity)) by Order = 4, Label = 'Covered vCPU capacity')\n| union (costs | where x_SkuLicenseStatus == 'Not Enabled' | summarize Value = numberstring(sum(x_SkuLicenseQuantity)) by Order = 6, Label = 'Eligible vCPU capacity')\n| order by Order asc\n", - "id": "1a400b6c-9204-434d-88ff-2dc5260a95bf", - "usedVariables": [ - "CostsByDayAHB" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "// Underutilized capacity\n// Fully utilized capacity\n// Eligible resources\nCostsByDayAHB\n| where x_SkuLicenseStatus == 'Enabled'\n//\n// Get the latest resource record first to guarrantee we have the latest status\n| summarize\n arg_max(ChargePeriodStart, *),\n TotalConsumedQuantity = sum(ConsumedQuantity),\n TotalEffectiveCost = sum(EffectiveCost)\n by\n ResourceId\n| project \n [\"License type\"] = x_SkuLicenseType,\n ResourceName,\n ResourceType,\n SKU = x_SkuInstanceType,\n [\"SKU cores\"] = x_SkuCoreCount,\n [\"Required capacity\"] = x_SkuLicenseQuantity,\n [\"Unused capacity\"] = x_SkuLicenseUnusedQuantity,\n [\"Unused vCore hours\"] = x_SkuLicenseUnusedQuantity * ConsumedQuantity,\n EffectiveCost = TotalEffectiveCost,\n x_ResourceGroupName,\n SubAccountName\n| order by [\"Unused vCore hours\"] desc\n", - "id": "c2ad5370-f4df-4a13-ac31-2c095f88754d", - "usedVariables": [ - "CostsByDayAHB" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "CostsByDayAHB\n| where x_SkuLicenseStatus == 'Enabled'\n| where x_SkuLicenseUnusedQuantity == 0\n//\n// Get the latest resource record first to guarrantee we have the latest status\n| summarize\n arg_max(ChargePeriodStart, *),\n TotalConsumedQuantity = sum(ConsumedQuantity),\n TotalEffectiveCost = sum(EffectiveCost)\n by\n ResourceId\n| project \n [\"License type\"] = x_SkuLicenseType,\n ResourceName,\n ResourceType,\n SKU = x_SkuInstanceType,\n [\"SKU cores\"] = x_SkuCoreCount,\n [\"vCore hours\"] = x_SkuLicenseQuantity * ConsumedQuantity,\n EffectiveCost = TotalEffectiveCost,\n x_ResourceGroupName,\n SubAccountName\n| order by [\"vCore hours\"] desc\n", - "id": "4614acfb-2288-4a71-97f7-32845a9ddcb2", - "usedVariables": [ - "CostsByDayAHB" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "CostsByDayAHB\n| where x_SkuLicenseStatus == 'Not Enabled'\n//\n// Get the latest resource record first to guarrantee we have the latest status\n| summarize\n arg_max(ChargePeriodStart, *),\n TotalConsumedQuantity = sum(ConsumedQuantity),\n TotalEffectiveCost = sum(EffectiveCost)\n by\n ResourceId\n| project \n [\"License type\"] = x_SkuLicenseType,\n ResourceName,\n ResourceType,\n SKU = x_SkuInstanceType,\n [\"SKU cores\"] = x_SkuCoreCount,\n [\"Required capacity\"] = x_SkuLicenseQuantity,\n [\"Eligible vCore hours\"] = x_SkuLicenseQuantity * ConsumedQuantity,\n EffectiveCost = TotalEffectiveCost,\n x_ResourceGroupName,\n SubAccountName\n| order by [\"Eligible vCore hours\"] desc\n", - "id": "99d44f1f-c53a-40ee-b749-8e5557ec58c7", - "usedVariables": [ - "CostsByDayAHB" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let costs = CostsPlus | where ChargePeriodStart >= startofmonth(now(), -3) and ChargePeriodStart < startofday(ago(-1d));\nlet startOfPeriod = toscalar(costs | summarize min(startofday(ChargePeriodStart)));\nlet endOfPeriod = toscalar(costs | summarize max(startofday(ChargePeriodStart)));\nlet forecastHorizon = numberOfDays * 1d;\ncosts\n| make-series\n EffectiveCost = sum(EffectiveCost)\n on ChargePeriodStart\n from startOfPeriod to endOfPeriod + forecastHorizon step 1d\n // by SubAccountId\n| extend Forecast = series_decompose_forecast(EffectiveCost, numberOfDays)\n", - "id": "5fd3ea2a-8882-4438-9103-fa3945240604", - "usedVariables": [ - "CostsPlus", - "numberOfDays" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let data = materialize(\n CostsByMonth\n //\n // Don't double-count commitment discount purchases\n | where x_AmortizationClass != 'Principal'\n //\n | summarize \n ListCost = sum(ListCost),\n ContractedCost = sum(ContractedCost),\n EffectiveCost = sum(EffectiveCost)\n | extend TotalSavings = ListCost - EffectiveCost\n | extend EffectiveSavingsRate = TotalSavings / ListCost\n | project json = todynamic(strcat('[',\n '{ \"order\":11, \"type\":\"TotalSavings\", \"label\":\"Total savings\", \"value\":\"', numberstring(round(TotalSavings, 2)), '\" },',\n '{ \"order\":12, \"type\":\"\", \"label\":\"\", \"value\":\"\u2797\" },',\n '{ \"order\":13, \"type\":\"List\", \"label\":\"Cost without discounts\", \"value\":\"', numberstring(round(ListCost, 2)), '\" },',\n '{ \"order\":14, \"type\":\"\", \"label\":\"\", \"value\":\"\ud83d\udff0\" },',\n '{ \"order\":15, \"type\":\"EffectiveSavingsRate\", \"label\":\"Effective savings rate\", \"value\":\"', percentstring(EffectiveSavingsRate), '\" }',\n ']'))\n | mv-expand json\n | order by toint(json.order) asc\n | project Label = tostring(json.label), Value = tostring(json.value), Type = tostring(json.type)\n);\ndata", - "id": "e1bc2d51-44af-4dd9-8b0d-a71088b551f5", - "usedVariables": [ - "CostsByMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "CostsByMonth\n| extend x_AmortizationClass = case(\n ChargeCategory == 'Purchase' and isnotempty(CommitmentDiscountCategory), 'Principal',\n isnotempty(CommitmentDiscountCategory), 'Amortized Charge',\n ''\n)\n| extend x_CommitmentDiscountSavings = iff(ContractedCost == 0, real(0), ContractedCost - EffectiveCost)\n| extend x_NegotiatedDiscountSavings = iff(ListCost == 0, real(0), ListCost - ContractedCost)\n| extend x_TotalSavings = iff(ListCost == 0, real(0), ListCost - EffectiveCost)\n| summarize\n ['List cost'] = round(sumif(ListCost, x_AmortizationClass != 'Principal'), 2),\n ['Effective cost'] = round(sum(EffectiveCost), 2),\n Savings = round(sum(x_TotalSavings), 2)\n by\n Account = x_BillingProfileId,\n Month = substring(startofmonth(ChargePeriodStart), 0, 7)\n| extend ESR = percentstring(Savings/ ['List cost'])\n| order by Month desc", - "id": "3b3f0a58-2d84-4e3e-bebc-3e747a7d5ede", - "usedVariables": [ - "CostsByMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let data = materialize(\n CostsByDay\n //\n // Don't double-count commitment discount purchases\n | where x_AmortizationClass != 'Principal'\n //\n | summarize \n ListCost = sum(ListCost),\n ContractedCost = sum(ContractedCost),\n EffectiveCost = sum(EffectiveCost)\n | extend TotalSavings = ListCost - EffectiveCost\n | extend EffectiveSavingsRate = TotalSavings / ListCost\n | project json = todynamic(strcat('[',\n '{ \"order\":11, \"type\":\"TotalSavings\", \"label\":\"Total savings\", \"value\":\"', numberstring(round(TotalSavings, 2)), '\" },',\n '{ \"order\":12, \"type\":\"\", \"label\":\"\", \"value\":\"\u2797\" },',\n '{ \"order\":13, \"type\":\"List\", \"label\":\"Cost without discounts\", \"value\":\"', numberstring(round(ListCost, 2)), '\" },',\n '{ \"order\":14, \"type\":\"\", \"label\":\"\", \"value\":\"\ud83d\udff0\" },',\n '{ \"order\":15, \"type\":\"EffectiveSavingsRate\", \"label\":\"Effective savings rate\", \"value\":\"', percentstring(EffectiveSavingsRate), '\" }',\n ']'))\n | mv-expand json\n | order by toint(json.order) asc\n | project Label = tostring(json.label), Value = tostring(json.value), Type = tostring(json.type)\n);\ndata", - "id": "cb34f22b-9370-460c-9658-3e73d220bbc7", - "usedVariables": [ - "CostsByDay" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "database('Ingestion').HubSettings\n| extend temp = todynamic(strcat('[',\n '{\"Label\":\"Version\",\"Value\":', version, '},',\n '{\"Label\":\"Managed scopes\",\"Value\":', array_length(scopes), '},',\n '{\"Label\":\"Data retention\",\"Value\":\"', toint(retention.final.months), 'mo\"}',\n']'))\n| mvexpand temp\n| project Label = tostring(temp.Label), Value = tostring(temp.Value)\n", - "id": "8de47213-8327-44da-9d1b-8ba5de74c44a", - "usedVariables": [] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "CostsByMonth\n| extend x_AmortizationClass = case(\n ChargeCategory == 'Purchase' and isnotempty(CommitmentDiscountCategory), 'Principal',\n isnotempty(CommitmentDiscountCategory), 'Amortized Charge',\n ''\n)\n| extend x_CommitmentDiscountSavings = iff(ContractedCost == 0, real(0), ContractedCost - EffectiveCost)\n| extend x_NegotiatedDiscountSavings = iff(ListCost == 0, real(0), ListCost - ContractedCost)\n| extend x_TotalSavings = iff(ListCost == 0, real(0), ListCost - EffectiveCost)\n| summarize\n ['On-demand'] = round(sumif(EffectiveCost, PricingCategory == 'Standard'), 2),\n Spot = round(sumif(EffectiveCost, PricingCategory == 'Dynamic'), 2),\n Reservation = round(sumif(EffectiveCost, CommitmentDiscountType == 'Reservation'), 2),\n ['Savings plan'] = round(sumif(EffectiveCost, CommitmentDiscountType == 'Savings Plan'), 2),\n ['Other'] = round(sumif(EffectiveCost, PricingCategory !in ('Standard', 'Dynamic') and isempty(CommitmentDiscountType)), 2)\n by\n Account = x_BillingProfileId,\n Month = substring(startofmonth(ChargePeriodStart), 0, 7)\n| order by Month desc", - "id": "0341b3e4-eccc-4924-9555-9835b128c543", - "usedVariables": [ - "CostsByMonth" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "Costs\n| where isempty(selectedBillingCurrency) or BillingCurrency == selectedBillingCurrency\n| extend EffectiveOverContracted = iff(ContractedCost < EffectiveCost, ContractedCost - EffectiveCost, real(0))\n| extend ContractedOverList = iff(ListCost < ContractedCost, ListCost - ContractedCost, real(0))\n| extend EffectiveOverList = iff(ListCost < EffectiveCost, ListCost - EffectiveCost, real(0))\n| extend Scenario = case(\n ListCost == 0 and CommitmentDiscountCategory == 'Usage' and ChargeCategory == 'Usage', 'Reservation usage missing list',\n ListCost == 0 and CommitmentDiscountCategory == 'Usage' and ChargeCategory == 'Purchase', 'Reservation purchase missing list',\n ListCost == 0 and CommitmentDiscountCategory == 'Spend' and ChargeCategory == 'Usage', 'Savings plan usage missing list',\n ListCost == 0 and CommitmentDiscountCategory == 'Spend' and ChargeCategory == 'Purchase', 'Savings plan purchase missing list',\n ListCost == 0 and ChargeCategory == 'Purchase', 'Other purchase missing list',\n isnotempty(CommitmentDiscountStatus) and ContractedOverList == 0 and EffectiveOverContracted < 0, 'Commitment cost over contracted',\n ListCost == 0 and BilledCost == 0 and EffectiveCost == 0 and ContractedCost > 0 and x_SourceChanges !contains 'MissingContractedCost', 'ContractedCost should be 0',\n ListCost == 0 and ContractedCost == 0 and BilledCost > 0 and EffectiveCost > 0 and x_PublisherCategory == 'Vendor' and ChargeCategory == 'Usage', 'Marketplace usage missing list/contracted',\n ContractedOverList < 0 and EffectiveOverContracted == 0 and x_SourceChanges !contains 'MissingListCost', 'ListCost too low',\n ContractedUnitPrice == x_EffectiveUnitPrice and EffectiveOverContracted < 0 and x_SourceChanges !contains 'MissingContractedCost', 'ContractedCost doesn\\'t match price',\n EffectiveOverContracted != 0 and abs(EffectiveOverContracted) < 0.00000001, 'Rounding error',\n ContractedOverList != 0 and abs(ContractedOverList) < 0.00000001, 'Rounding error',\n EffectiveOverList != 0 and abs(EffectiveOverList) < 0.00000001, 'Rounding error',\n ContractedCost < EffectiveCost or ListCost < ContractedCost or ListCost < EffectiveCost, '',\n EffectiveCost <= ContractedCost and ContractedCost <= ListCost, 'Good',\n '')\n| project-reorder ListCost, ContractedCost, BilledCost, EffectiveCost, EffectiveOverList, EffectiveOverContracted, ContractedOverList, x_SourceChanges, ListUnitPrice, ContractedUnitPrice, x_BilledUnitPrice, x_EffectiveUnitPrice, CommitmentDiscountStatus, PricingQuantity, PricingUnit, x_PricingBlockSize, x_PricingUnitDescription\n// DEBUG -- | where isempty(scenario) | limit 1000\n// Summarize -- \n| summarize Rows = count(), EffectiveCost = round(sum(EffectiveCost), 2), EffectiveOverContracted = abs(sum(EffectiveOverContracted)), ContractedOverList = abs(sum(ContractedOverList)), EffectiveOverList = abs(sum(EffectiveOverList)), Agreement = arraystring(make_set(x_BillingAccountAgreement)) by Scenario | order by Rows desc\n", - "id": "56fd8707-bbb3-4f7e-8e37-8dd90ada3baa", - "usedVariables": [ - "selectedBillingCurrency" - ] - }, - { - "id": "43612ae4-c475-4f22-bb50-ce9d995abb8f", - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "CostsPlus\n| where startofmonth(BillingPeriodStart) == startofmonth(now())", - "usedVariables": [ - "CostsPlus" - ] - }, - { - "id": "cb1f5404-c0b1-42fd-99fb-3cff7b08daaa", - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "CostsPlus\n| where startofmonth(BillingPeriodStart) == startofmonth(startofmonth(now()) - 1d)", - "usedVariables": [ - "CostsPlus" - ] - }, - { - "id": "4ce0f587-2d45-436c-8f79-102c6b382439", - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "CostsPlus\r\n| where startofmonth(ChargePeriodStart) >= startofmonth(now(), -numberOfMonths)\r\n| extend ChargePeriodStart = startofmonth(ChargePeriodStart)\r\n| extend BillingPeriodStart = startofmonth(BillingPeriodStart)", - "usedVariables": [ - "CostsPlus", - "numberOfMonths" - ] - }, - { - "id": "6b598467-8c31-4693-b1eb-7ed683fcfc3a", - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "CostsPlus\r\n| where ChargePeriodStart >= ago(numberOfDays * 1d) - 1d and ChargePeriodStart < ago(1d)\r\n| extend ChargePeriodStart = startofday(ChargePeriodStart)", - "usedVariables": [ - "CostsPlus", - "numberOfDays" - ] - }, - { - "id": "4a1973bf-08e9-4e82-b8e6-6edff81cf0a5", - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "CostsPlus", - "usedVariables": [ - "CostsPlus" - ] - }, - { - "id": "eb9259cc-05b7-4441-a66d-a29026fe371b", - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "Costs_v1_2\n//\n// Apply summarization settings\n| where ChargePeriodStart >= monthsago(numberOfMonths)\n| where isempty(selectedBillingCurrency) or BillingCurrency == selectedBillingCurrency\n| as filteredCosts\n| extend x_ChargeMonth = startofmonth(ChargePeriodStart)\n// TODO: Should we add granularity? -- | extend x_ReportingDate = iff(#\"Default Granularity\" == 'Monthly'), x_ChargeMonth, startofday(ChargePeriodStart))\n//\n// SKU details\n| extend x_SkuUsageType = tostring(coalesce(SkuPriceDetails.x_UsageType, x_SkuDetails.UsageType))\n| extend x_SkuLicenseUnusedQuantity = x_SkuLicenseQuantity - x_SkuCoreCount\n//\n// Commitment discounts\n| extend x_CommitmentDiscountKey = iff(isempty(x_SkuInstanceType), '', strcat(x_SkuInstanceType, x_SkuMeterId))\n| extend x_SkuTermLabel = case(isempty(x_SkuTerm) or x_SkuTerm <= 0, '', x_SkuTerm < 12, strcat(x_SkuTerm, ' month', iff(x_SkuTerm != 1, 's', '')), strcat(x_SkuTerm / 12, ' year', iff(x_SkuTerm != 12, 's', '')))\n//\n// CSP partners\n// x_PartnerBilledCredit = iff(x_PartnerCreditApplied, BilledCost * x_PartnerCreditRate, todouble(0))\n// x_PartnerEffectiveCredit = iff(x_PartnerCreditApplied, EffectiveCost * x_PartnerCreditRate, todouble(0))\n//\n// Toolkit\n| extend x_ToolkitTool = tostring(Tags['ftk-tool'])\n| extend x_ToolkitVersion = tostring(Tags['ftk-version'])\n| extend tmp_ResourceParent = database('Ingestion').parse_resourceid(Tags['cm-resource-parent'])\n| extend x_ResourceParentId = tostring(tmp_ResourceParent.ResourceId)\n| extend x_ResourceParentName = tostring(tmp_ResourceParent.ResourceName)\n| extend x_ResourceParentType = tostring(tmp_ResourceParent.ResourceType)\n//\n// TODO: Only add differentiators when the name is not unique\n| extend CommitmentDiscountNameUnique = iff(isempty(CommitmentDiscountId), '', strcat(CommitmentDiscountName, ' (', CommitmentDiscountType, ')'))\n| extend ResourceNameUnique = iff(isempty(ResourceId), '', strcat(ResourceName, ' (', ResourceType, ')'))\n| extend x_ResourceGroupNameUnique = iff(isempty(x_ResourceGroupName), '', strcat(x_ResourceGroupName, ' (', SubAccountName, ')'))\n| extend SubAccountNameUnique = iff(isempty(SubAccountId), '', strcat(SubAccountName, ' (', split(SubAccountId, '/')[3], ')'))\n//\n// Explain why cost is 0\n| extend x_FreeReason = case(\n BilledCost != 0.0 or EffectiveCost != 0.0, '',\n PricingCategory == 'Committed', strcat('Unknown ', CommitmentDiscountStatus, ' Commitment'),\n x_BilledUnitPrice == 0.0 and x_EffectiveUnitPrice == 0.0 and ContractedUnitPrice == 0.0 and ListUnitPrice == 0.0 and isempty(CommitmentDiscountType), case(\n x_SkuDescription contains 'Trial', 'Trial',\n x_SkuDescription contains 'Preview', 'Preview',\n 'Other'\n ),\n x_BilledUnitPrice > 0.0 or x_EffectiveUnitPrice > 0.0, case(\n PricingQuantity > 0.0, 'Low Usage',\n PricingQuantity == 0.0, 'No Usage',\n 'Unknown Negative Quantity'\n ),\n 'Unknown'\n)\n//\n| extend x_ResourceTop1K = ChargeCategory != 'Usage' or isempty(ResourceId) or ResourceId in (\n filteredCosts\n | where isnotempty(ResourceId) and ChargeCategory == 'Usage'\n | summarize sum(EffectiveCost) by ResourceId\n | order by sum_EffectiveCost desc\n | limit 1000\n | distinct ResourceId\n)\n//\n| project-away tmp_ResourceParent", - "usedVariables": [ - "numberOfMonths", - "selectedBillingCurrency" - ] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let months = toscalar(database('Ingestion').HubSettings | project toint(retention.final.months));\nlet monthname = dynamic(['', 'Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']);\nrange Value from toint(1) to iff(isempty(months), 24, months) step 1\n| order by Value desc\n| extend MonthsAgo = monthsago(Value)\n| extend Label = strcat(Value, ' mo (', monthname[monthofyear(MonthsAgo)], format_datetime(MonthsAgo, ' yyyy'), ')')\n| project-away MonthsAgo\n", - "id": "c9039243-968d-4e75-9899-8d4ab51a9896", - "usedVariables": [] - }, - { - "dataSource": { - "kind": "inline", - "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" - }, - "text": "let months = toscalar(database('Ingestion').HubSettings | project toint(retention.final.months));\nlet currencies = Costs_v1_2\n | where ChargePeriodStart >= monthsago(iff(isempty(months), 24, months))\n | where isnotempty(BillingCurrency)\n | distinct BillingCurrency;\nlet currencyCount = toscalar(currencies | count);\ncurrencies\n| extend Label = BillingCurrency, _Order = 1\n| union (\n print BillingCurrency = '', Label = '\u26a0\ufe0f All', _Order = 0\n | where currencyCount > 1\n)\n| order by _Order asc, Label asc\n| project BillingCurrency, Label\n", - "id": "f2a8c4d6-3b5e-4a7f-9c2d-8e5b1f4a7d9c", - "usedVariables": [] - } - ] -} \ No newline at end of file diff --git a/src/templates/finops-hub-local/docker-compose.yml b/src/templates/finops-hub-local/docker-compose.yml deleted file mode 100644 index f5401e9d5..000000000 --- a/src/templates/finops-hub-local/docker-compose.yml +++ /dev/null @@ -1,32 +0,0 @@ -services: - kustainer: - image: mcr.microsoft.com/azuredataexplorer/kustainer-linux:latest - platform: linux/amd64 - container_name: kustainer - ports: - # Host 8082 -> container 8080. Host port 8080 is reserved for maenifold. - - "${HOST_PORT:-8082}:8080" - environment: - ACCEPT_EULA: "Y" - volumes: - # Parquet exports — read-only, the engine only ingests from here. - - "${EXPORT_DIR:-./export}:/data/export:ro" - # Persistent engine state (databases, metadata, extents). - - ./kustainer-data:/kustodata - mem_limit: ${MEM_LIMIT:-8g} - restart: unless-stopped - healthcheck: - # Use the in-container curl to hit the internal mgmt endpoint. - # `.show version` is the cheapest mgmt command and proves the engine - # is actually answering, not just listening. - test: - - CMD-SHELL - - >- - curl -fsS --max-time 10 - -H 'Content-Type: application/json' - -d '{"db":"NetDefaultDB","csl":".show version"}' - http://localhost:8080/v1/rest/mgmt > /dev/null || exit 1 - interval: 10s - timeout: 15s - retries: 12 - start_period: 30s diff --git a/src/templates/finops-hub-local/ftkver.txt b/src/templates/finops-hub-local/ftkver.txt deleted file mode 100644 index b9dc6d407..000000000 --- a/src/templates/finops-hub-local/ftkver.txt +++ /dev/null @@ -1 +0,0 @@ -15.0-dev diff --git a/src/templates/finops-hub-local/notes/build-by-hand.md b/src/templates/finops-hub-local/notes/build-by-hand.md deleted file mode 100644 index 479a4da97..000000000 --- a/src/templates/finops-hub-local/notes/build-by-hand.md +++ /dev/null @@ -1,197 +0,0 @@ -# Build a local FinOps hub by hand - -This is the manual runbook. It shows every command needed to stand up a local FinOps -hub on the Kusto emulator and load your FOCUS cost data — no helper scripts. The -[helper scripts](#automating-the-steps) automate exactly these steps; read this first so -you know what they do. - -The whole thing is six steps and about a dozen commands: - -1. [Start the emulator](#1-start-the-emulator) -2. [Create the two databases](#2-create-the-two-databases) -3. [Load the schema](#3-load-the-schema) -4. [Ingest your Parquet exports](#4-ingest-your-parquet-exports) — the final tables populate themselves -5. [Large datasets: defer the transform](#5-large-datasets-defer-the-transform) -6. [Query](#6-query) - -## Prerequisites - -- Docker and PowerShell 7 — see [install-windows.md](install-windows.md) or - [install-mac.md](install-mac.md). -- FOCUS cost exports as Parquet, staged under `export/` — see - [staging-contract.md](staging-contract.md). -- The two schema bundles, which the toolkit build produces from the FinOps hub KQL: - `release/finops-hub-fabric-setup-Ingestion.kql` and - `release/finops-hub-fabric-setup-Hub.kql`. Run `npm run build` (or - `pwsh ./src/scripts/Build-Toolkit.ps1 -Template finops-hub`) once to generate them. - -## How to run a command - -The emulator is a plain HTTP endpoint with no auth. Send any management command (a `.` -command) to `/v1/rest/mgmt` and any query to `/v1/rest/query`. Every command below can be -run with `curl`: - -```bash -curl -s "http://localhost:8082/v1/rest/mgmt" \ - -H "Content-Type: application/json" \ - -H "x-ms-client-version: Kusto.Python.Client:1.0.0" \ - -d '{"db":"","csl":""}' -``` - -You can also paste commands into the query window of the -[ADX web UI](dashboard.md) connected to `http://localhost:8082`. - ---- - -## 1. Start the emulator - -```bash -docker compose up -d --wait -``` - -`--wait` blocks until the container's healthcheck passes, so the engine is ready to -answer when the command returns. The emulator mounts your `export/` directory read-only -at `/data/export` inside the container. - -## 2. Create the two databases - -A FinOps hub uses two databases — `Ingestion` for the raw and transformed data, and `Hub` -for the view functions analysts query. Create both, persisted so they survive a restart: - -```kusto -// db: NetDefaultDB -.create database Ingestion persist (@'/kustodata/dbs/Ingestion/md', @'/kustodata/dbs/Ingestion/data') -.create database Hub persist (@'/kustodata/dbs/Hub/md', @'/kustodata/dbs/Hub/data') -``` - -## 3. Load the schema - -Each schema bundle is a single `.execute database script` command, so loading it is one -POST per database — the engine runs every statement in the file in order. - -The Ingestion bundle has one placeholder, `$$rawRetentionInDays$$`, the raw-data -retention in days. Replace it with a number (for example `90`) before loading. - -```bash -# Ingestion schema (raw tables, transform functions, final tables, open data) -sed 's/\$\$rawRetentionInDays\$\$/90/g' release/finops-hub-fabric-setup-Ingestion.kql \ - | python3 -c 'import sys,json; print(json.dumps({"db":"Ingestion","csl":sys.stdin.read()}))' \ - | curl -s "http://localhost:8082/v1/rest/mgmt" \ - -H "Content-Type: application/json" \ - -H "x-ms-client-version: Kusto.Python.Client:1.0.0" --data-binary @- - -# Hub schema (the Costs/Prices/Transactions view functions) -python3 -c 'import sys,json; print(json.dumps({"db":"Hub","csl":open("release/finops-hub-fabric-setup-Hub.kql").read()}))' \ - | curl -s "http://localhost:8082/v1/rest/mgmt" \ - -H "Content-Type: application/json" \ - -H "x-ms-client-version: Kusto.Python.Client:1.0.0" --data-binary @- -``` - -The bundle is loaded with `ContinueOnErrors=true`, so the response is a table with one row -per statement and its result — scan it for any `Failed` rows. The Hub functions reference -`database('Ingestion').*`; those resolve once both databases exist. - -## 4. Ingest your Parquet exports - -The schema created the raw tables (`Costs_raw`, `Prices_raw`) and their Parquet ingestion -mappings (`Costs_raw_mapping`, `Prices_raw_mapping`). It also enabled an **update policy** -on each final table: as rows land in `Costs_raw`, the engine runs `Costs_transform_v1_2()` -automatically and appends the FOCUS-normalized result to `Costs_final_v1_2` (and the same -for prices). This is the exact mechanism a deployed FinOps hub uses — so on the local -emulator, **ingesting the raw data is all you need**; the final tables populate -themselves. - -Ingest the files mounted at `/data/export`. List one `h@'...'` URI per file, -comma-separated, to load many files in a single command: - -```kusto -// db: Ingestion — cost exports -.ingest into table Costs_raw ( - h@'/data/export//ms--focus-cost///part_0.parquet', - h@'/data/export//ms--focus-cost///part_1.parquet' - ) with (format='parquet', ingestionMappingReference='Costs_raw_mapping') - -// db: Ingestion — price sheets -.ingest into table Prices_raw ( - h@'/data/export//ms--pricesheet///part_0.parquet' - ) with (format='parquet', ingestionMappingReference='Prices_raw_mapping') -``` - -Confirm the raw counts and that the final tables filled in via the update policy: - -```kusto -// db: Ingestion -Costs_raw | count -Costs_final_v1_2 | count // populated automatically by the update policy -Prices_raw | count -Prices_final_v1_2 | count -``` - -> Do **not** also run `.set-or-append Costs_final_v1_2 <| Costs_transform_v1_2()` here — -> with the update policy enabled, the transform already ran during ingest, and a manual -> backfill would append a second copy and double the final tables. - -## 5. Large datasets: defer the transform - -The update policy is `IsTransactional`, so it transforms every ingested batch inline. That -is ideal for incremental exports, but a one-shot bulk load of tens of millions of rows -makes each batch's transform slow and memory-heavy, and a whole-table transform can exceed -the container's `MEM_LIMIT`. In practice `Prices_final_v1_2` (~12.7M rows) is the table that -OOMs at 16g, while `Costs_final_v1_2` (~1.35M rows) transforms single-pass without trouble — -so the table below uses `Prices` as the worked example. For a large historical load, turn the -policy off, ingest, then run the transform once — in chunks bounded by source extent: - -````kusto -// db: Ingestion -// 1. disable the update policy so ingest doesn't transform inline -.alter table Prices_final_v1_2 policy update ```[{"IsEnabled":false,"Source":"Prices_raw","Query":"Prices_transform_v1_2()","IsTransactional":true,"PropagateIngestionProperties":true}]``` - -// 2. ingest all the raw data (as in step 4) - -// 3. run the transform once, one extent group at a time so each pass is bounded -.set-or-append Prices_final_v1_2 <| - let Prices_raw = __table("Prices_raw", 'All', 'AllButRowStore') - | where extent_id() in (, ); - Prices_transform_v1_2() - -// 4. re-enable the update policy (set IsEnabled back to true) -```` - -`Costs_final_v1_2` needs the same treatment only if its row count grows past the single-pass -headroom — swap `Costs_raw` / `Costs_transform_v1_2()` / `Costs_final_v1_2` into the commands -above. The [`ingest.ps1`](../scripts/ingest.ps1) helper does exactly this for both tables — -disable, bulk-ingest, chunked backfill, re-enable — so you don't have to manage the policy or -the extent GUIDs by hand. See [performance.md](performance.md) for the memory measurements -that set the chunk threshold. **For large data, use the helper; this section is what it does -under the hood.** - -## 6. Query - -The `Hub` database exposes the same view functions a deployed hub does. Query them -directly: - -```kusto -// db: Hub -Costs_v1_2() -| summarize EffectiveCost = round(sum(EffectiveCost), 2) by ServiceName -| top 10 by EffectiveCost -``` - -That is a working local FinOps hub. - ---- - -## Automating the steps - -The helper scripts in [`scripts/`](../scripts/) are this runbook, automated. They add -nothing the engine doesn't already do — they just save you the manual file-listing and, -for large data, the per-extent chunking: - -| Step | By hand | Helper | -| ----------------------------- | ------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| 2–3. Create DBs + load schema | `.create database` ×2, POST each bundle | [`load-ftk-kql.ps1`](../scripts/load-ftk-kql.ps1) | -| 4–5. Ingest | `.ingest` per file; for large data, disable policy + backfill | [`ingest.ps1`](../scripts/ingest.ps1) — discovers your files, skips already-loaded ones, and for large tables disables the update policy and chunks the transform | -| 6. Query | POST to `/v1/rest/query` | [`ftk.ps1`](../scripts/ftk.ps1) — also runs the published query catalog | - -On macOS and Linux the [`Makefile`](../Makefile) wraps these (`make up`, `make -load-ftk-kql`, `make ingest`). See the [README](../README.md) for the quickstart. diff --git a/src/templates/finops-hub-local/notes/config-container.md b/src/templates/finops-hub-local/notes/config-container.md deleted file mode 100644 index c79ef028a..000000000 --- a/src/templates/finops-hub-local/notes/config-container.md +++ /dev/null @@ -1,105 +0,0 @@ -# Config container — decision record - -## Context - -A deployed FinOps Hub writes two artifacts to the Azure Storage `config` -container: - -| Artifact | Location in storage | Schema | -| --------------- | ----------------------- | -------------------------------------------- | -| `settings.json` | `config/settings.json` | `https://aka.ms/finops/hubs/settings-schema` | -| Export schemas | `config/schemas/*.json` | Internal column-typing contract | - -The `settings.json` file drives **scope registration** and **retention -configuration** for the Azure Data Factory (ADF) orchestration layer. The -`config/schemas/*.json` files drive the ADF **msexports → ingestion -conversion** step — column typing and normalization of the raw Cost Management -CSV exports into Parquet. - -ftklocal does not run ADF. The question this note answers is: which of these -artifacts does ftklocal need, and when? - ---- - -## Mode A — ingestion container (already-converted Parquet) ✅ v1 default - -**What ftklocal does today.** - -ftklocal ingests the already-converted `ingestion/`-container Parquet directly -into Kusto (`Costs_raw`, `Prices_raw`). At this point the msexports → ingestion -conversion has already been performed by ADF in the deployed hub. - -**Config container requirement:** none. - -- `settings.json` — not required for the msexports → ingestion conversion path - (that is an ADF concern). However, `load-ftk-kql.ps1` does optionally read a - local `settings.json` for one value — `retention.raw.days` — to set the Kusto - raw-table soft-delete policy via the `$$rawRetentionInDays$$` macro; it falls - back to 3650 days when the file is absent or when `raw.days` is 0. The - scopes array in `settings.json` is an ADF concern and is not read by the - ADX/KQL layer. See `settings.example.json` and `scripts/settings.ps1`. -- `config/schemas/*.json` — not required. The Parquet files in the ingestion - container are already normalized; the column-typing step that uses the schemas - has already run. - ---- - -## Mode B — msexports container (raw Cost Management exports) 🔮 future / deferred - -**Not implemented in v1.** - -If ftklocal were extended to ingest directly from the raw Cost Management CSV -exports (`msexports/` container), it would need to perform the same -msexports → ingestion conversion that ADF does. That conversion uses the schema -files to type, deduplicate, and normalize each export. - -**Config container requirement:** - -- `config/schemas/*.json` — required for the conversion step. **These must - always be sourced from the in-repo copies** at - `src/templates/finops-hub/modules/Microsoft.CostManagement/Exports/schemas/` - (e.g. `focuscost_1.2.json`, `focuscost_1.2-preview.json`, - `pricesheet_2023-05-01_ea.json`). Never copy schemas from a customer's - storage; the in-repo versions are the canonical, version-controlled - definitions. -- `settings.json` — would be needed to resolve the scope list and retention - parameters for the conversion orchestration. - -**Status:** the full local msexports converter is deferred to the bridge work -tracked in T-5000.9. Mode A remains the v1 happy path. - ---- - -## ADX / KQL layer — `config/schemas/*.json` not consumed; `settings.json` read by `load-ftk-kql.ps1` for one value - -The `config/schemas/*.json` files (the msexports → ingestion column-typing -schemas) are not consumed by any component of the ADX/KQL layer. `settings.json` -is partially consumed: `load-ftk-kql.ps1` optionally reads it for -`retention.raw.days` only; everything else reads neither. Evidence: - -- **`scripts/load-ftk-kql.ps1`** — optionally reads a local `settings.json` for - `retention.raw.days` (with a 3650-day fallback) and passes it to the - `$$rawRetentionInDays$$` macro in the Ingestion bundle. It does not read - scopes, version, or any other `settings.json` field. It does not read - `config/schemas/*.json` at all. -- **`scripts/ftk.ps1`** — reads neither `settings.json` nor - `config/schemas/*.json`. It does use the word "schema" extensively, but only - for querying live Kusto table schemas via KQL (`.show table ... cslschema`, - `getschema`) — not for any on-disk config file. -- **Hub / Ingestion KQL queries** — the `.kql` files assembled by - `src/templates/finops-hub/.build.config` `combineKql` bundles contain no - reference to `settings.json` or `config/schemas/*.json`. -- The schema files in - `src/templates/finops-hub/modules/Microsoft.CostManagement/Exports/schemas/` - are consumed by the ADF pipeline, not by KQL. - -For v1, `config/schemas/*.json` is therefore **not required** (the -msexports → ingestion conversion is an ADF concern). `settings.json` is -**optional** — absent means 3650-day raw retention; present lets the operator -set it explicitly. - -> **Local analysis note:** `retention.raw.days` in `settings.json` drives the -> Kusto raw-table soft-delete window (the `$$rawRetentionInDays$$` macro in -> `load-ftk-kql.ps1`). Keep this value high (3650 days is the default) for -> local analysis — unlike Azure Storage, local raw tables ARE the source data; -> early soft-deletion means re-running a full ingest to recover. diff --git a/src/templates/finops-hub-local/notes/dashboard.md b/src/templates/finops-hub-local/notes/dashboard.md deleted file mode 100644 index 651bf23c3..000000000 --- a/src/templates/finops-hub-local/notes/dashboard.md +++ /dev/null @@ -1,187 +0,0 @@ -# Using the local ADX dashboard - -This dashboard (`../dashboard.json`) is the upstream FinOps hub dashboard localized to the -ftklocal Kusto emulator. The only behaviorally significant change from the upstream file is -`clusterUri` (the `title`, `id`, and `eTag` metadata fields also differ, but they do not -affect tiles or queries): - -| Field | Upstream value | Local value | -| --------------------------- | ----------------------------------------------------- | ----------------------- | -| `dataSources[0].clusterUri` | `https://..kusto.windows.net/` | `http://localhost:8082` | -| `dataSources[0].database` | `Hub` | `Hub` (unchanged) | - -All tiles and KQL queries are unmodified. The `Hub` database on the emulator exposes the same -view functions as a deployed hub: `Costs_v1_2`, `Prices_v1_2`, `Transactions_v1_2`, -`CommitmentDiscountUsage_v1_2`, and their aliases (`Costs`, `Prices`, …). - ---- - -## Prerequisites - -- ftklocal stack running: `docker compose up` from `src/templates/finops-hub-local/` -- Kusto emulator listening on `http://localhost:8082` -- Hub DB populated (run the ftklocal ingestion scripts or load sample data) - ---- - -## Proven path: query via the CLI - -The emulator's KQL REST API is plain HTTP with no authentication — you can query it directly -from the terminal today without a browser. Use `ftk.ps1 query` (or `ftk.ps1 run`) against the -`Hub` database: - -```bash -# Smoke-test: confirm view functions are populated -pwsh scripts/ftk.ps1 query --database Hub "Costs_v1_2 | take 10" - -# Run any KQL query interactively -pwsh scripts/ftk.ps1 run --database Hub -``` - -This is the correct first validation step on Mac. It does not depend on any browser -mixed-content policy and works regardless of HTTPS. - -Once the agent skill is built (see the skill roadmap), the same Hub view functions will be -available as structured tool calls through the skill surface — no manual KQL required. - ---- - -## ADX web UI dashboard import - -> ℹ️ **`localhost` is not blocked as mixed content, but Edge/Chrome will ask permission once.** -> -> `dataexplorer.azure.com` is served over HTTPS, and `http://localhost` (and `127.0.0.1`) is a -> **potentially trustworthy origin** under the W3C Secure Contexts spec, so Chromium-based -> browsers (**Microsoft Edge, Google Chrome**) do _not_ apply mixed-content blocking to it. -> However, current Edge (143+) and Chrome (142+) enforce **Local Network Access (LNA)**: the -> first time the page reaches `http://localhost:8082` the browser shows a **"Local Network -> Access" permission prompt** — click **Allow** and the connection proceeds for the session. -> Use Edge or Chrome for this path. Safari is stricter; if it refuses the connection, use the -> [ngrok workaround](#if-the-browser-blocks-the-connection). -> -> The emulator also returns the CORS headers the web UI requires (measured — see -> [UAT findings](#uat-findings-measured) below), so once you allow local-network access the -> cross-origin queries from `dataexplorer.azure.com` to the emulator are permitted. - -### UAT findings (measured) - -Measured against the running emulator (`http://localhost:8082`, Hub DB populated with the -sample dataset): - -| Check | Result | -| --------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| CORS preflight (`OPTIONS`, `Origin: https://dataexplorer.azure.com`) | ✅ `204` with `Access-Control-Allow-Origin: https://dataexplorer.azure.com`, `Access-Control-Allow-Methods: POST`, `Access-Control-Allow-Headers: content-type,x-ms-client-version,authorization` | -| Actual query (`POST /v1/rest/query`, same `Origin`) | ✅ `200` with `Access-Control-Allow-Origin: https://dataexplorer.azure.com` | -| Real tile queries return renderable data via the CORS path | ✅ Monthly trend (15 months), top services (bar), Prices (table) all return rows | -| `dashboard.json` import schema | ✅ Official ADX dashboard schema **v60**, single `manual-kusto` data source → `http://localhost:8082` / `Hub` | -| `dashboard.json` referential integrity | ✅ 135 tiles / 87 queries / 6 base queries / 12 pages, **0 dangling references** → imports cleanly | -| Private Network Access (PNA) preflight (`Access-Control-Request-Private-Network: true`) | ⚠️ Emulator does **not** return `Access-Control-Allow-Private-Network: true`. Current Edge 143+/Chrome 142+ enforce **Local Network Access (LNA)** and show a one-time **permission prompt** for the public→localhost call — click **Allow** to proceed. Locked-down/enterprise-policy builds that suppress or deny the prompt need the [LNA fallback](#if-the-browser-blocks-the-connection). | - -The protocol path is proven server-side: standard CORS is allowed, `localhost` is exempt from -mixed-content blocking in Edge/Chrome, and the dashboard file imports cleanly. The two -remaining browser-side steps are (a) the interactive Azure (AAD) sign-in and (b) clicking -**Allow** on the one-time Local Network Access prompt — both handled in the steps below. - -### 1. Connect the ADX web UI to the emulator - -1. Open **https://dataexplorer.azure.com** in **Microsoft Edge or Google Chrome** (not Safari). -2. Sign in with your Azure account (the web UI itself requires AAD; the emulator cluster does not). -3. Click **+ Add cluster** (left sidebar). -4. Enter the connection URI: `http://localhost:8082` -5. Authentication: select **Anonymous** (the emulator runs no-auth HTTP). -6. Click **Add**. If Edge/Chrome shows a **"Local Network Access"** permission prompt - (a site wants to access your local network), click **Allow**. - -If the cluster appears in the sidebar with `Hub` and `Ingestion` databases, the connection -succeeded and you can proceed. If the connection is refused, see -[if the browser blocks the connection](#if-the-browser-blocks-the-connection) -below. - -### 2. Import the dashboard - -1. In the ADX web UI, click **Dashboards** (left nav). -2. Click the **⋮ menu → Import dashboard from file**. -3. Select `src/templates/finops-hub-local/dashboard.json`. -4. After import, open **Dashboard settings → Data sources** and confirm: - - Cluster URI: `http://localhost:8082` - - Database: `Hub` -5. If the data source shows a different cluster URI, edit it to `http://localhost:8082` / - `Hub` and save. - -### 3. Validate tiles - -Open any tile (e.g., "Cost summary") and click **Refresh**. If tiles return empty results, -confirm the Hub DB has data by running `Costs_v1_2 | take 10` in the **Query** tab. - ---- - -## If the browser blocks the connection - -Two browser-side mechanisms can refuse the `https://dataexplorer.azure.com → http://localhost:8082` -call. They are independent — try the lighter fix first. - -### Option A — Local Network Access (LNA) prompt, Chromium - -Current Edge (143+) and Chrome (142+) gate public→`localhost` requests behind a **Local -Network Access permission prompt**. In the normal case you simply click **Allow** when the -prompt appears (see step 1) and the connection works — no server header or flag needed. - -If your environment **suppresses or auto-denies** the prompt (enterprise policy, or a build -where you previously clicked _Block_), relax the check for local UAT by launching the browser -with LNA/PNA enforcement disabled: - -```bash -# Edge (quit Edge first), local UAT only: -open -na "Microsoft Edge" --args --disable-features=LocalNetworkAccessChecks,PrivateNetworkAccessRespectPreflightResults - -# Chrome equivalent: -open -na "Google Chrome" --args --disable-features=LocalNetworkAccessChecks,PrivateNetworkAccessRespectPreflightResults -``` - -You can also re-allow the site via the page-info (lock/info icon) → site permissions, or reset -its Local Network Access permission. Re-enable normal enforcement after testing. - -### Option B — expose the emulator over HTTPS with ngrok - -If you'd rather not change browser flags (or you're on Safari), tunnel the emulator behind -HTTPS using [ngrok](https://ngrok.com): - -```bash -# Install ngrok if needed: brew install ngrok/ngrok/ngrok -ngrok http 8082 -``` - -ngrok prints an HTTPS forwarding URL such as `https://xxxx-xxxx.ngrok-free.app`. Use that -URL as the cluster URI in the ADX web UI instead of `http://localhost:8082`. Update the -dashboard data source to match. - -> Note: ngrok free-tier URLs are ephemeral and change each session. You will need to update -> the data source in the dashboard each time. - ---- - -## What was proved vs what you must validate - -| Claim | Status | -| ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `dashboard.json` is valid JSON | ✅ Verified (Python `json.load` round-trip) | -| `dataSources[0].clusterUri` → `http://localhost:8082` | ✅ Verified in file | -| `dataSources[0].database` = `Hub` | ✅ Verified in file | -| Hub DB has `Costs_v1_2`, `Prices_v1_2`, etc. | ✅ Verified via `.show functions` on running emulator | -| No upstream cluster names, billing IDs, or absolute paths in dashboard | ✅ Verified | -| `ftk.ps1 query` / `ftk.ps1 run` against Hub DB | ✅ Works (HTTP REST, no browser) | -| Browser can add `http://localhost:8082` from `https://dataexplorer.azure.com` | ✅ **CORS allowed** (emulator returns `Access-Control-Allow-Origin: https://dataexplorer.azure.com`); `localhost` is exempt from mixed-content blocking. Edge 143+/Chrome 142+ show a one-time **Local Network Access** prompt — click **Allow**. | -| Dashboard tiles render data after import | ⚠️ **Visual render pending** — protocol path proven (tile queries return data via CORS); requires the interactive browser import + AAD sign-in + the LNA **Allow** click to confirm on screen | - ---- - -## Ingestion vs Hub databases - -The emulator runs two databases: - -- **`Ingestion`** — raw, transform, and final staging tables used during ingestion. Not - intended as a query surface. -- **`Hub`** — view functions (`Costs_v1_2`, etc.) that join and normalize the ingested data. - This is the correct database for the dashboard. - -The dashboard's data source is already set to `Hub`. diff --git a/src/templates/finops-hub-local/notes/ftk-kql-adaptations.md b/src/templates/finops-hub-local/notes/ftk-kql-adaptations.md deleted file mode 100644 index 7205b13cd..000000000 --- a/src/templates/finops-hub-local/notes/ftk-kql-adaptations.md +++ /dev/null @@ -1,225 +0,0 @@ -# FTK KQL adaptations for the local stack - -There are **two distinct adaptation layers** in the local stack. This document covers -both. - -| Layer | Where it runs | Purpose | -| ------------- | ---------------------------------------- | ------------------------------------------------------------------------- | -| **Load-time** | `scripts/load-ftk-kql.ps1` | Loads upstream FTK KQL into the emulator at setup time (idempotent). | -| **On-read** | `scripts/ftk.ps1` `Convert-CatalogQuery` | Adapts catalog `.kql` files at query time before posting to the emulator. | - -The upstream KQL scripts live in -`src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/scripts/`. - -**Design constraint:** only two categories of adaptation are permitted in the load-time -layer: - -1. Replace ARM/Bicep deployment-time templating macros with concrete local values. -2. Adjust for the two-database topology that the local stack preserves (see below). - -No transform logic is modified. If an upstream FTK function has a bug, it is logged -as a gap — not patched locally. - ---- - -## Topology note - -FTK deploys to ADX as a **two-database** system: an `Ingestion` database (raw + transform - -- final tables) and a `Hub` database (view functions). The local stack preserves this - topology exactly — the emulator runs both `Ingestion` and `Hub` databases, and - `database('Ingestion').*` cross-references inside Hub view functions resolve in the - emulator the same way they do in ADX. - -Earlier prototype notes described collapsing these into a single local `FtkLocal` database. -That design was superseded. The current stack uses the two-database topology throughout. - ---- - -## Adaptation 1 — Macro substitution: `$$rawRetentionInDays$$` → `3650` - -**File:** `IngestionSetup_RawTables.kql` - -`$$rawRetentionInDays$$` is the FTK Bicep deployment-time macro substituted by the -ARM template at hub-deploy time. Kustainer rejects the literal macro syntax with a -`SyntaxException`. The loader substitutes `3650` (10 years — effectively "never -delete" for a local dev database). - -**Implementation:** `scripts/load-ftk-kql.ps1` applies the substitution via a -`subst_macros` transform pass on `IngestionSetup_RawTables.kql` only. - ---- - -## Adaptation 2 — Two-database cross-references pass through unmodified - -**Files:** `HubSetup_v1_2.kql`, `HubSetup_OpenData.kql` - -No rewrite is needed. The local emulator runs both `Ingestion` and `Hub` databases, -so `database('Ingestion').X` inside Hub view functions resolves correctly — the same -as in a real ADX deployment. The loader runs `HubSetup_v1_2.kql` and -`HubSetup_OpenData.kql` against the `Hub` database without modification. - ---- - -## Adaptation 3 — Empty v1_0 final-table stubs - -**Source:** synthesized in-memory by `scripts/load-ftk-kql.ps1` - -The v1_2 view functions in `HubSetup_v1_2.kql` union v1_2 final tables with v1_0 -final tables for backward compatibility. Kustainer eagerly resolves function bodies -at creation time, so without the v1_0 tables present, -`.create-or-alter function Costs_v1_2()` fails with `General_BadRequest`. - -The loader extracts only the five `.create-merge table X_final_v1_0 (...)` DDL blocks -from `IngestionSetup_v1_0.kql` — no transforms, no update policies, no ingestion -mappings. The resulting tables are empty and serve only to satisfy the eager-compilation -check. - -Tables created (empty stubs): - -- `Costs_final_v1_0` -- `Prices_final_v1_0` -- `CommitmentDiscountUsage_final_v1_0` -- `Recommendations_final_v1_0` -- `Transactions_final_v1_0` - -Because the v1_0 tables are empty, union queries are semantically identical to -v1_2-only queries — no data is hidden or duplicated. - ---- - -## Adaptation 4 — Splitter: trailing inter-command comments stripped - -This is a feature of the loader's command splitter, not a modification to FTK content. - -FTK KQL scripts intersperse top-level commands with `//` comment lines between them. -The splitter accumulates lines into commands until the next line that starts a new -command (`^\.[a-z]`). Without special handling, the trailing `// ` line before -the next command gets glued onto the end of the preceding command body, causing -Kustainer's management endpoint to reject it with `General_BadRequest`. - -The splitter strips trailing blank lines and comment-only lines from each accumulated -command before posting. The commands themselves are submitted verbatim. This resolved -all 31 false failures observed across `Common.kql`, `IngestionSetup_HubInfra.kql`, -`IngestionSetup_v1_2.kql`, `HubSetup_v1_2.kql`, and `HubSetup_OpenData.kql`. - ---- - -## Adaptation 5 — Database creation prefix - -Not in any FTK script — the loader adds this as a prerequisite before loading any -FTK content. A clean checkout has no databases; the loader creates `Ingestion` and -`Hub` with their `persist` paths matching what `docker-compose.yml` mounts at -`/kustodata`. On re-runs, the `EntityNameAlreadyExistsException` response (HTTP 400) -is treated as "already exists — idempotent OK". - ---- - -## Adaptation 6 — Open-data CSV column mapping - -**Files:** `PricingUnits.csv`, `Regions.csv`, `ResourceTypes.csv`, `Services.csv` -(from `src/open-data/`) - -The open-data CSV files were generated against an older table schema than the one -declared in `IngestionSetup_HubInfra.kql`. The HubInfra DDL uses FOCUS-aligned column -names (`x_PricingUnitDescription`, `x_ResourceType`, etc.), but the CSVs still use -legacy raw names (`UnitOfMeasure`, `ResourceType`, etc.). In a production hub -deployment an Azure Function applies this rename; that function is not part of any KQL -script. - -The loader applies the same rename at load time. Key mappings: - -| Table | CSV column | Table column | -| --------------- | ------------------ | -------------------------- | -| `PricingUnits` | `UnitOfMeasure` | `x_PricingUnitDescription` | -| `PricingUnits` | `PricingBlockSize` | `x_PricingBlockSize` | -| `PricingUnits` | `DistinctUnits` | `PricingUnit` | -| `Regions` | `OriginalValue` | `ResourceLocation` | -| `ResourceTypes` | `ResourceType` | `x_ResourceType` | -| `ResourceTypes` | `Icon` | `IconUri` | -| `Services` | `ConsumedService` | `x_ConsumedService` | -| `Services` | `ResourceType` | `x_ResourceType` | -| `Services` | `PublisherType` | `x_PublisherCategory` | -| `Services` | `Environment` | `x_Environment` | -| `Services` | `ServiceModel` | `x_ServiceModel` | - -The loader reads each CSV, projects to the target column order, clears the destination -table (idempotency), and ingests inline into the FTK-declared table schema. - ---- - -## Drops: none - -Nothing was dropped from the FTK scripts on grounds of "Kustainer doesn't support -this". Streaming ingestion policy commands (`alter table X policy streamingingestion -disable`) were empirically verified to be accepted by Kustainer (HTTP 200). No -`.alter cluster`, managed-identity, EventGrid, or other cluster-level commands appear -in the seven scripts the loader processes. - ---- - -## On-read adapter — `ftk.ps1 run` (`Convert-CatalogQuery`) - -`ftk.ps1 run` adapts catalog `.kql` files at query time before posting them to the -emulator. This is a separate layer from the load-time adaptations above. The function -`Convert-CatalogQuery` applies the following rules in order: - -### Rule 1 — Date-window retarget (step `# 1. Date window`) - -Rewrites `let startDate = ...;` and `let endDate = ...;` bindings in catalog queries -to match the `--start` / `--end` CLI arguments (or the last complete month when no -range is supplied). This lets time-bounded catalog queries work locally without -editing the `.kql` files. - -### Rule 1b — `--param` typed-literal override (step `# 1b.`) - -Rewrites any top-level `let = ...;` binding when `--param name=value` is -supplied on the CLI. Values are serialised to KQL typed literals (datetime, timespan, -bool, int, real, or string). Throws if `name` does not match a top-level `let` in the -query (no silent no-ops). - -### Rule 2 — `decimal`→`real` normalisation (step `# 2.`) - -``` -todecimal('') → real(null) -todecimal( → toreal( -decimal( → real( -``` - -Kustainer's free image does not support the `decimal` scalar type. The FTK query -catalog uses `decimal()` for monetary columns in some queries. This substitution is -lossless for the numeric range involved and is the verified compatibility fix. - -### Rule 3 — Project-away tolerance (step `# 3.`) - -If a catalog query contains a `project-away` clause and calls a hub view function -(`Costs()`, `Prices()`, `Recommendations()`, or `Transactions()`), the adapter -inspects the live function schema and silently drops any column names from the -`project-away` list that do not exist in the local schema. This prevents errors when -an upstream catalog query projects away a column the local Hub does not yet expose. - -### Cross-database references — pass through unmodified - -Catalog queries that explicitly reference `database('Ingestion').X` are **not -rewritten**. The two-database topology means `database('Ingestion').*` resolves -correctly in the emulator (Hub functions hold live cross-DB references to Ingestion -final tables). Stripping the prefix would break any catalog query that legitimately -targets an Ingestion table directly. This is consistent with the load-time behaviour -(the Hub KQL bundle is also submitted verbatim). - ---- - -## Documented upstream FTK behavior (not adapted — logged only) - -### `.alter table Costs_raw policy update` collision - -`IngestionSetup_v1_2.kql` calls `.alter table Costs_raw policy update` twice (once for -the `ActualCosts_raw` chain, once for `AmortizedCosts_raw`). Because `.alter` replaces -rather than merges the policy, the second call overwrites the first. The result is that -only the `AmortizedCosts_raw → AmortizedCosts_transform_v1_2()` chain is active after -loading. - -This behavior is identical on ADX and on Kustainer. The local test dataset uses -FOCUS-format exports (not the legacy `ActualCosts`/`AmortizedCosts` format), so this -does not affect parity results. Documented here for awareness; not patched locally per -the "no transform modifications" rule. diff --git a/src/templates/finops-hub-local/notes/install-mac.md b/src/templates/finops-hub-local/notes/install-mac.md deleted file mode 100644 index 93b2f108b..000000000 --- a/src/templates/finops-hub-local/notes/install-mac.md +++ /dev/null @@ -1,104 +0,0 @@ -# Install and run ftklocal on macOS - -macOS is a supported secondary platform. On **Apple Silicon** the Kustainer image -(`linux/amd64`) runs under emulation, so ingest and transform are slower than on Windows -or Intel; the measured numbers in [performance.md](performance.md) were taken on Apple -Silicon and represent that emulated path. On **Intel Macs** the image runs natively. - -Everything runs from a normal terminal with PowerShell 7 (`pwsh`). The `make` targets are -available on macOS as a convenience, but the `docker compose` + `pwsh` commands below work -identically and are the cross-platform reference. - ---- - -## Prerequisites - -| Requirement | Notes | -| ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Docker Desktop ≥ 4.0** | Compose v2 is required (`docker compose`, not legacy `docker-compose`). | -| **Apple Silicon: enable Rosetta** | Docker Desktop → Settings → General → "Use Rosetta for x86_64/amd64 emulation on Apple Silicon". The Kustainer image is `linux/amd64` only. Intel Macs need nothing extra. | -| **PowerShell 7+** (`pwsh`) | `brew install --cask powershell`. All tooling is PowerShell; there are no Python dependencies. | -| **Git** | To clone the repository (`brew install git`, or use the Xcode command-line tools). | -| **~16 GiB RAM available to Docker** | Docker Desktop → Settings → Resources → Memory. The full Prices transform needs it; Costs-only datasets work at 8 GiB. See [Limits](performance.md). | -| **Host port 8082 free** | Configurable via `HOST_PORT` in `.env`. | -| **FOCUS cost exports as Parquet** | Staged under `export/`. See [staging-contract.md](staging-contract.md). | - ---- - -## Install - -```bash -# 1. Clone the toolkit and move into the template -git clone https://github.com/microsoft/finops-toolkit.git -cd finops-toolkit/src/templates/finops-hub-local - -# 2. Create your local config from the template -cp .env.example .env -``` - -Edit `.env` to point `EXPORT_DIR` at the folder holding your FOCUS Parquet exports: - -```ini -# .env -EXPORT_DIR=./export # or an absolute path, e.g. ~/cost-exports -HOST_PORT=8082 -MEM_LIMIT=16g -``` - ---- - -## Run - -Two equivalent paths — `make` (macOS/Linux convenience) or the cross-platform commands. - -### Using make - -```bash -make up # start the emulator; blocks until healthy -make load-ftk-kql # create Ingestion + Hub DBs, load FTK KQL + open data -make ingest # bulk-ingest the Parquet exports under export/ -pwsh scripts/ftk.ps1 query "Costs() | summarize TotalCost = sum(EffectiveCost) by ServiceName | top 10 by TotalCost" -``` - -### Cross-platform (no make) - -```bash -docker compose up -d --wait # start + block until the healthcheck passes -pwsh scripts/load-ftk-kql.ps1 -pwsh scripts/ingest.ps1 -pwsh scripts/ftk.ps1 query "Costs() | summarize TotalCost = sum(EffectiveCost) by ServiceName | top 10 by TotalCost" -``` - -Useful follow-ups: - -```bash -pwsh scripts/ftk.ps1 list # list the reusable query catalog -pwsh scripts/ftk.ps1 run # run a named catalog query -make ingest-status # summary of what was ingested -make parity # validate transform parity -``` - ---- - -## Stop, reset, and update - -```bash -docker compose down # stop, keep data (or: make down) -make nuke # full destructive reset (down -v + remove kustainer-data/) -git pull && docker compose up -d --wait && pwsh scripts/load-ftk-kql.ps1 # update -``` - ---- - -## Troubleshooting (macOS-specific) - -| Symptom | Fix | -| --------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | -| Very slow ingest / transform on Apple Silicon | Expected — `linux/amd64` runs under emulation. Confirm Rosetta is enabled (above). See [performance.md](performance.md) for the measured envelope. | -| Container exits under the Prices transform | Raise Docker Desktop's memory (Settings → Resources) to ≥ 16 GiB. The ingest already chunks large tables. | -| `pwsh: command not found` | `brew install --cask powershell`. | -| Rare SIGSEGV crashes under heavy load | A known Rosetta emulation issue; the container restarts automatically. See the main [README](../README.md) troubleshooting section. | -| Port 8082 already in use | Set a different `HOST_PORT` in `.env`, then bring the stack back up. | - -For the ADX web-UI dashboard, see [dashboard.md](dashboard.md). Use Edge or Chrome; Safari -is stricter about local connections. diff --git a/src/templates/finops-hub-local/notes/install-windows.md b/src/templates/finops-hub-local/notes/install-windows.md deleted file mode 100644 index cd937dc98..000000000 --- a/src/templates/finops-hub-local/notes/install-windows.md +++ /dev/null @@ -1,151 +0,0 @@ -# Install and run ftklocal on Windows - -This is the primary supported platform. The Kustainer image is `linux/amd64`, which runs -**natively** on Windows x64 (via the WSL 2 backend) with no emulation — so ingest and -transform are faster here than on Apple Silicon. The numbers in -[performance.md](performance.md) were measured under Apple-Silicon emulation and are a -conservative floor for Windows. - -Everything below runs from a normal **PowerShell 7** prompt. You do **not** need `make`, -WSL shell access, or any Unix tooling — `docker compose` and the `.ps1` scripts are the -whole surface. - ---- - -## Prerequisites - -| Requirement | Notes | -| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Docker Desktop ≥ 4.0** with the **WSL 2 backend** | Settings → General → "Use the WSL 2 based engine". Compose v2 ships with it (`docker compose`, not legacy `docker-compose`). | -| **PowerShell 7+** (`pwsh`) | **Windows PowerShell 5.1 will not work** — the scripts use 7-only syntax. Install with `winget install Microsoft.PowerShell`, then use the `pwsh` command (not `powershell`). | -| **Git** | `winget install Git.Git`, to clone the repository. | -| **~16 GiB RAM available to Docker** | The full Prices transform needs it. Costs-only datasets work at 8 GiB. See [memory tuning](#tune-wsl-2-memory) below — on Windows the limit is governed by both the container `MEM_LIMIT` **and** WSL 2's own memory cap. | -| **Host port 8082 free** | Configurable via `HOST_PORT` in `.env`. | -| **FOCUS cost exports as Parquet** | Staged under `export/`. See [staging-contract.md](staging-contract.md). | - -> **Why PowerShell 7, not the built-in PowerShell?** Windows ships _Windows PowerShell 5.1_ -> (`powershell.exe`). ftklocal's scripts use null-coalescing (`??`) and ternary operators -> that only exist in _PowerShell 7+_ (`pwsh.exe`). Running them under 5.1 fails with parser -> errors. Always invoke them with `pwsh`. - ---- - -## Tune WSL 2 memory - -Docker Desktop on Windows runs containers inside a WSL 2 virtual machine. Two limits apply: - -1. The container limit — `MEM_LIMIT` in `.env` (default `16g`). -2. The WSL 2 VM limit — by default WSL 2 may cap memory below what the container needs. - -If the container is killed under load, raise the WSL 2 cap. Create or edit -`%UserProfile%\.wslconfig`: - -```ini -[wsl2] -memory=20GB -``` - -Then, from PowerShell, restart WSL and Docker Desktop: - -```powershell -wsl --shutdown -# then relaunch Docker Desktop -``` - -Give the VM a couple of GiB more than `MEM_LIMIT` so the engine has headroom. See -[performance.md](performance.md) for the measured working-set curve. - ---- - -## Install - -```powershell -# 1. Clone the toolkit and move into the template -git clone https://github.com/microsoft/finops-toolkit.git -cd finops-toolkit\src\templates\finops-hub-local - -# 2. Create your local config from the template -Copy-Item .env.example .env -``` - -Edit `.env` to point `EXPORT_DIR` at the folder holding your FOCUS Parquet exports. On -Windows, use a forward-slash path or an absolute drive path — both work with Docker -Desktop's bind mounts: - -```ini -# .env (examples — pick one) -EXPORT_DIR=./export -EXPORT_DIR=D:/cost-exports -HOST_PORT=8082 -MEM_LIMIT=16g -``` - -> Keep forward slashes in `EXPORT_DIR`. A backslash path like `D:\exports` can be -> mis-parsed by Compose variable expansion; `D:/exports` (or `C:/finops/exports`) is safe. - ---- - -## Run - -```powershell -# 3. Start the emulator and block until it reports healthy -docker compose up -d --wait - -# 4. Create the Ingestion + Hub databases, load the FTK KQL and open data (idempotent) -pwsh scripts\load-ftk-kql.ps1 - -# 5. Bulk-ingest every Parquet export under export\ (idempotent; chunks large tables) -pwsh scripts\ingest.ps1 - -# 6. Query your FOCUS data (Hub database, view functions) -pwsh scripts\ftk.ps1 query "Costs() | summarize TotalCost = sum(EffectiveCost) by ServiceName | top 10 by TotalCost" -``` - -`docker compose up -d --wait` uses the container's built-in healthcheck and returns only -once the engine answers queries — it is the cross-platform equivalent of `make up`. - -Useful follow-ups: - -```powershell -pwsh scripts\ftk.ps1 list # list the reusable query catalog -pwsh scripts\ftk.ps1 run # run a named catalog query -pwsh scripts\ingest.ps1 -DryRun # preview what would be ingested -pwsh scripts\run-parity-checks.ps1 # validate transform parity -``` - ---- - -## Stop, reset, and update - -```powershell -# Stop the emulator, keep the data -docker compose down - -# Full reset (destructive): remove the container and all ingested data -docker compose down -v -Remove-Item -Recurse -Force .\kustainer-data # if a bind-mounted data dir remains - -# Update to a newer toolkit version -git pull -docker compose up -d --wait -pwsh scripts\load-ftk-kql.ps1 # re-applies schema (idempotent) -``` - ---- - -## Troubleshooting (Windows-specific) - -| Symptom | Fix | -| --------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `pwsh : The term 'pwsh' is not recognized` | PowerShell 7 isn't installed or isn't on `PATH`. `winget install Microsoft.PowerShell`, then open a new terminal. | -| Parser errors like `Unexpected token '??'` | You're running under Windows PowerShell 5.1. Use `pwsh`, not `powershell`. | -| `running scripts is disabled on this system` | Invoking as `pwsh scripts\x.ps1 ` normally bypasses this. If you dot-source instead, set `Set-ExecutionPolicy -Scope Process RemoteSigned` for the session. | -| `docker compose` not found | Update Docker Desktop (Compose v2 is bundled). Don't use the legacy `docker-compose`. | -| Container exits / killed under the Prices transform | WSL 2 memory cap too low — see [Tune WSL 2 memory](#tune-wsl-2-memory). The ingest already chunks large tables; raising the VM cap removes the ceiling. | -| Port 8082 already in use | Set a different `HOST_PORT` in `.env`, then `docker compose up -d --wait`. | -| `Ingestion`/`Hub` DB missing after start | Run `pwsh scripts\load-ftk-kql.ps1`. If it persists, full-reset (above) and reload. | -| Tiles/queries return 0 rows | You haven't ingested yet — run `pwsh scripts\ingest.ps1`. | - -For the ADX web-UI dashboard on Windows, see [dashboard.md](dashboard.md) — Edge and Chrome -are the supported browsers and will prompt once for **Local Network Access** (click -**Allow**). diff --git a/src/templates/finops-hub-local/notes/parity-gaps.md b/src/templates/finops-hub-local/notes/parity-gaps.md deleted file mode 100644 index 6ca94fac5..000000000 --- a/src/templates/finops-hub-local/notes/parity-gaps.md +++ /dev/null @@ -1,178 +0,0 @@ -# Parity gaps - -This document records known differences between the local Kusto emulator stack and a -deployed FinOps hub, as measured by the `tests/parity-checks.kql` suite. - -**Suite status: 11 pass / 0 fail / 2 manual** (of 13 total) - -Run the suite: - -```bash -make parity -``` - ---- - -## Actual suite run output (literal) - -Path in first line stripped to repo-relative; all other output verbatim. - -``` -Loaded 12 parity check(s) from tests/parity-checks.kql - -━━━ CHECK 1: row count parity [DB: Ingestion] ━━━ - EXPECT: final_count within 5% of raw_count (some rows may be filtered by transform). - ✓ PASS: delta = 0.0000% within 5% - -━━━ CHECK 2: BilledCost preservation [DB: Ingestion] ━━━ - EXPECT: sum(BilledCost) in final equals sum(BilledCost) in raw (no silent drops). - ✓ PASS: delta = $0.000000 - -━━━ CHECK 3: EffectiveCost zeroed for commitment-discount purchases [DB: Ingestion] ━━━ - EXPECT: sum(EffectiveCost) == 0 for ChargeCategory='Purchase' AND isnotempty(CommitmentDiscountId). - ✓ PASS: result = 0 - -━━━ CHECK 4: data-quality flag dictionary populates [DB: Ingestion] ━━━ - EXPECT: x_SourceChanges contains documented codes (MissingContractedCost, - ? MANUAL: cols=['first_code', 'Count'], rows[:5]=[['XEffectiveUnitPriceRoundingError', 1100011], ['LegacyFocusVersion', 117862], ['ContractedCostLessThanEffectiveCost', 110171], ['ListCostLessThanContractedCost', 19967], ['MissingContractedCost', 1887]] - rows[:3] = [['XEffectiveUnitPriceRoundingError', 1100011], ['LegacyFocusVersion', 117862], ['ContractedCostLessThanEffectiveCost', 110171]] - -━━━ CHECK 5: AmortizationClass labels amortized usage [DB: Ingestion] ━━━ - EXPECT: ZERO rows where ChargeCategory='Usage' AND isnotempty(CommitmentDiscountId) - ✓ PASS: result = 0 - -━━━ CHECK 6: TotalSavings populated for services with reservations [DB: Ingestion] ━━━ - EXPECT: sum(x_TotalSavings) > 0 for services that typically have RIs/SPs - ✓ PASS: top row sum_TotalSavings = 4694.06850131341 - -━━━ CHECK 7: parse_resourceid populates x_ResourceGroupName when ARM path has one [DB: Ingestion] ━━━ - EXPECT: ZERO rows where isempty(x_ResourceGroupName) AND isnotempty(ResourceId) - ✓ PASS: result = 0 - -━━━ CHECK 8: PublisherName normalized (no 'Microsoft Corporation') [DB: Ingestion] ━━━ - EXPECT: 'Microsoft Corporation' should be normalized to 'Microsoft' by the - ✓ PASS: result = 0 - -━━━ CHECK 9: Hub view function returns plausible totals [DB: Hub] ━━━ - EXPECT: Costs_v1_2 | summarize sum(EffectiveCost) by ServiceCategory returns - ✓ PASS: top row sum_EffectiveCost = 306236.631384515 - -━━━ CHECK 10: Costs_v1_2 vs Costs_final_v1_2 BilledCost reconciliation [DB: Hub] ━━━ - EXPECT: sum(BilledCost) from Costs_v1_2 (Hub view) == sum from Costs_final_v1_2 - ✓ PASS: delta = $0.000000 - -━━━ CHECK 11: Prices row count parity per PriceType [DB: Ingestion] ━━━ - EXPECT: Consumption + ReservedInstance counts in Prices_final match Prices_raw - ? MANUAL: cols=['PriceType', 'raw_n', 'final_n', 'delta', 'delta_pct'], rows[:5]=[['Consumption', 9591718, 9591718, 0, 0], ['SavingsPlan', 2931206, 2931206, 0, 0], ['ReservedInstance', 212663, 212663, 0, 0]] - rows[:3] = [['Consumption', 9591718, 9591718, 0, 0], ['SavingsPlan', 2931206, 2931206, 0, 0], ['ReservedInstance', 212663, 212663, 0, 0]] - -━━━ CHECK 13: Prices value parity — UnitPrice/ContractedUnitPrice (Consumption + RI) [DB: Ingestion] ━━━ - EXPECT: sum(UnitPrice) in Prices_raw equals sum(ContractedUnitPrice) in Prices_final_v1_2 - ✓ PASS: delta = 0.0000% within 0.1% - -━━━ CHECK 12: Name parity — Ingestion tables + Hub functions [DB: Ingestion+Hub] ━━━ - EXPECT: No FtkLocal-named objects; required core tables and functions present. - ✓ PASS: Ingestion: 23 table(s), Hub: 34 function(s); no FtkLocal objects; all core names present - -━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -SUMMARY: 11 pass, 0 fail, 2 manual (of 13 total) - -Manual checks above require human inspection but did not block the run. -``` - ---- - -## Per-check results - -Checks 1–11 and 13 are defined in `tests/parity-checks.kql`. CHECK 12 is a synthetic -check generated by the runner itself (not in the `.kql` file) — see -[How "of 13 total" is counted](#how-of-13-total-is-counted) below. Checks run in file -order (1–11, 13) then CHECK 12 last. - -| # | Check | DB | Result | Detail | -| --- | ---------------------------------------------------------------------- | ------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | -| 1 | Row count parity | Ingestion | ✅ PASS | `Costs_raw` = `Costs_final_v1_2`; delta = 0.0% | -| 2 | BilledCost preservation | Ingestion | ✅ PASS | delta = $0.000000 | -| 3 | EffectiveCost zeroed for commitment purchases | Ingestion | ✅ PASS | result = 0 | -| 4 | Data-quality flag dictionary populates | Ingestion | ⚠️ MANUAL | Print-only by design — see [Data-quality observations](#data-quality-observations-check-4) | -| 5 | Amortized usage labeled correctly | Ingestion | ✅ PASS | 0 mislabeled rows | -| 6 | TotalSavings populated for reservation services | Ingestion | ✅ PASS | top sum = 4,694.07 | -| 7 | `parse_resourceid` extracts `x_ResourceGroupName` | Ingestion | ✅ PASS | 0 missing extractions | -| 8 | `PublisherName` normalized | Ingestion | ✅ PASS | 0 unnormalized rows | -| 9 | Hub view function returns plausible totals | Hub | ✅ PASS | top sum_EffectiveCost = 306,236.63 | -| 10 | `Costs_v1_2` vs `Costs_final_v1_2` BilledCost reconciliation | Hub | ✅ PASS | delta = $0.000000 | -| 11 | Prices row count parity per PriceType | Ingestion | ⚠️ MANUAL | Runner always flags as MANUAL (print-only); all three PriceTypes show delta = 0 on current dataset | -| 12 | Name parity — Ingestion tables + Hub functions | Ingestion+Hub | ✅ PASS | 23 Ingestion tables, 34 Hub functions; no `FtkLocal`-named objects; all core names present. **Synthetic** — not in `.kql` file; generated by runner | -| 13 | Prices value parity — UnitPrice/ContractedUnitPrice (Consumption + RI) | Ingestion | ✅ PASS | delta = 0.0000% within 0.1% automated threshold | - ---- - -## How "of 13 total" is counted - -`run-parity-checks.ps1` loads **12 checks from `tests/parity-checks.kql`** (numbered -1–11 and 13; the file skips number 12). The runner then appends **one synthetic CHECK -12** — a name-parity assertion that verifies no `FtkLocal`-prefixed objects exist and -that all expected core tables and functions are present. The summary formula -(`run-parity-checks.ps1` line 499) is: - -```powershell -"SUMMARY: … (of $($checks.Count + ($runNameParity ? 1 : 0)) total)" -# 12 file-based + 1 synthetic = 13 total -``` - -CHECK 12 always runs last (after CHECK 13) because it is appended after the -file-based checks are processed. - ---- - -## Data-quality observations (check 4) - -Check 4 is informational by design — it verifies the `x_SourceChanges` dictionary is -populated, not that it is empty. The flags are FTK's data-quality findings on the -source export data, not defects in the local transform. Observed flags on the current -dataset: - -| Flag | Rows | -| ------------------------------------- | --------: | -| `XEffectiveUnitPriceRoundingError` | 1,100,011 | -| `LegacyFocusVersion` | 117,862 | -| `ContractedCostLessThanEffectiveCost` | 110,171 | -| `ListCostLessThanContractedCost` | 19,967 | -| `MissingContractedCost` | 1,887 | - -Consumers of `Costs_final_v1_2` can filter on these flags: - -```kql -Costs_final_v1_2 | where x_SourceChanges has 'MissingContractedCost' -``` - ---- - -## Known upstream FTK behaviors (not local gaps) - -### Prices SavingsPlan lookup fanout - -`IngestionSetup_v1_2.kql` applies a `lookup kind=leftouter` to SavingsPlan rows, -joining on `tmp_SavingsPlanKey` against a `distinct` over four price columns. When a -single Consumption SKU key maps to multiple (List, Contracted, Base) price tuples -across billing months, the corresponding SavingsPlan row fans out one-to-many. On -multi-scope or multi-period datasets this produces a small percentage surplus in -`Prices_final_v1_2` SavingsPlan rows relative to `Prices_raw`. - -On the current single-billing-account dataset all three PriceTypes show delta = 0 -(confirmed by CHECK 11 above). - -**Impact for consumers if the surplus appears on your dataset:** deduplicate with: - -```kql -Prices_final_v1_2 -| summarize arg_max(x_IngestionTime, *) by SkuPriceIdv2 -``` - -### `.alter table Costs_raw policy update` collision - -`IngestionSetup_v1_2.kql` calls `.alter table Costs_raw policy update` twice. -Because `.alter` replaces rather than merges the policy, the second call -(`AmortizedCosts_raw → AmortizedCosts_transform_v1_2()`) overwrites the first. This -affects both ADX and Kustainer identically. The FOCUS-format test data used for -parity validation is not impacted. Documented for awareness; not patched locally. diff --git a/src/templates/finops-hub-local/notes/performance.md b/src/templates/finops-hub-local/notes/performance.md deleted file mode 100644 index 170e8f410..000000000 --- a/src/templates/finops-hub-local/notes/performance.md +++ /dev/null @@ -1,217 +0,0 @@ -# ftklocal performance and limits - -Measured on **MEM_LIMIT=16g**, **amd64 on Rosetta** (Apple Silicon / Docker Desktop), using -the Kustainer Linux image (`mcr.microsoft.com/azuredataexplorer/kustainer-linux:latest`, -build `1.0.9656.17219`). - -Dataset: 15 cost export parts (1,350,561 rows) + 16 price sheet parts (12,735,587 rows), -plus the four open-data lookup tables shipped with the FTK schema (3,293 rows). -Total physical rows at full load: **28,175,589** across Costs_raw, Costs_final_v1_2, -Prices_raw, Prices_final_v1_2, and open-data tables. - ---- - -## Measurement method - -All memory figures are read from **cgroup v2** files inside the container: - -- `memory.max` — hard limit (confirms MEM_LIMIT setting) -- `memory.current` — total container memory (anonymous + page cache) -- `memory.stat → anon` — **non-reclaimable anonymous working set** (the number that - matters for headroom: kernel page cache is reclaimable under pressure, but `anon` is not) -- `memory.peak` — **cumulative** high-water mark since container creation; the kernel - rejected `echo 0 > /sys/fs/cgroup/memory.peak` (read-only filesystem in this - environment), so memory.peak is cumulative across all stages -- `memory.events → oom / oom_kill` — OOM event counters; these reset when the container - restarts - -Intermediate stages (3a–3c) are read from a 3-second-interval poll loop (`mem_poll.log`) -started before the ingest and stopped afterwards. Stages 1, 2, 3d, and 4 are point-in-time -reads via `docker exec kustainer`. The OOM peak (stage 5) is from a separate 2-second poll -started just before the OOM attempt. - ---- - -## Measurement table - -| Stage | Total rows in engine | anon (GiB) | anon % of 16 g | memory.peak (GiB) | OOM events | -| ------------------------------------------------- | -------------------- | ---------- | -------------- | ----------------- | ---------- | -| 1. Empty engine (just started, no data) | 0 | 0.49 | 3.1 % | 0.51 | — | -| 2. Schema + open-data loaded (`load-ftk-kql`) | 3,293 | 0.66 | 4.1 % | 0.68 | — | -| 3a. After Costs_raw ingested (15 parts) | 1,353,854 | 1.51 ¹ | 9.4 % | 2.13 ¹ | — | -| 3b. After all raw ingested, no finals yet | 14,089,441 | 1.99 ¹ | 12.4 % | 3.98 ¹ | — | -| 3c. After Costs_final_v1_2 backfill (single-pass) | 15,440,002 | 3.88 ¹ | 24.2 % | 6.89 ¹ | — | -| 3d. After full ingest — **hot** working set | 28,175,589 | **13.99** | **87.4 %** | **15.03** | 0 | -| 4. After representative query workload | 28,175,589 | 13.95 | 87.2 % | 15.03 | 0 | -| 5. Single-pass OOM attempt (Prices, 12.7 M rows) | 28,175,589 | 14.74 ¹ | 92.1 % | **15.12 ¹** | **crash** | -| 6. Cold loaded — post-restart, data on disk ² | 28,175,589 | **1.12** | **7.0 %** | 1.58 ² | 0 | - -**Notes:** - -- Stages 1, 2, 3d, 4 are exact point-in-time measurements. Stages 3a–3c and 5 are read - from the 2–3 s poll loop and should be treated as approximate (±0.1 GiB). -- memory.peak is cumulative (not per-stage); the value in each row is the cumulative - high-water mark up to that stage. Stage 6's memory.peak is a **post-restart** reading - (cgroup reset on container restart, so it reflects only the cold-start HWM, not the - earlier ingest peak). -- After the OOM crash the container restarted automatically (`restart: unless-stopped`). - The cgroup counters (oom / oom_kill) reset on restart; they read 0 in both the pre-OOM - measurement and after recovery. The crash is confirmed by the Kustainer log entry - (`Killed`, SIGKILL) and the abrupt memory.current drop observed in the poll log. -- ² Stage 6 was read after the OOM-triggered restart. The full dataset (1,350,561 Costs + - 12,735,587 Prices rows) is intact on disk and returns correct counts; extents are not - yet materialized in memory (lazy-load). This is also the expected state after a planned - `make down && make up` cycle. - ---- - -## Row-count → memory curve - -The table above shows how the non-reclaimable working set grows as data is loaded: - -``` -Stage Total rows anon GiB ---------------------------------------------------------- -Empty engine 0 0.49 -Schema + open data 3,293 0.66 -After Costs_raw ingest 1,353,854 1.51 -After all raw (no finals) 14,089,441 1.99 -After Costs_final backfill 15,440,002 3.88 -After full ingest — hot 28,175,589 13.99 -Cold loaded (post-restart) 28,175,589 1.12 -``` - -Key observations: - -1. **Raw ingestion is memory-cheap.** Loading 14 M raw rows (Costs + Prices parquet) - raises anon only from 0.66 GiB to ~2.0 GiB. The engine writes extents to disk; the - ingest buffers are transient. - -2. **The transform backfill is memory-expensive.** Single-pass Costs_final (1.35 M rows, - 53.2 s) peaks at 6.89 GiB (43 % of 16 g) but settles at 3.88 GiB. The difference - (~3 GiB) is intermediate query-execution scratch space that is released after the - `.set-or-append` completes. - -3. **Prices_final dominates the hot working set.** Loading 12.7 M transformed Prices rows - raises anon from 3.88 GiB to 13.99 GiB — a 10 GiB jump. Immediately after the - ingest+transform pipeline the engine holds ≈87.4 % of MEM_LIMIT in non-reclaimable - memory. This is a **hot** state: extents materialized by the transform. After a restart - the same data idles at **1.12 GiB (7 %)** — extents remain on disk until queried. - -4. **Queries are nearly free in memory.** Five representative queries (Costs by service, - monthly trend, Prices count, Prices by service family) changed anon by < 0.1 GiB and - did not raise memory.peak above the ingest high-water mark. - ---- - -## Ingest wall-clock - -| Phase | Duration | -| ---------------------------------------------------------------- | ---------------------- | -| Raw file ingest (31 parquet parts, 991 MB) | 419.6 s (7.0 min) | -| Costs_final_v1_2 single-pass backfill (1.35 M rows) | 53.2 s | -| Prices_final_v1_2 chunked backfill (12.7 M rows, 21 extents × 1) | ~340 s (~5.7 min) | -| **Full `ingest.ps1` wall-clock (raw + backfill)** | **841.6 s (14.0 min)** | - -The raw file ingest time is dominated by the 16 price sheet parts (247 s total, ~15 s each -for the large monthly files). The 15 cost parts take only 76.5 s. - ---- - -## Chunked-backfill peak and single-pass OOM - -### Why auto-chunking exists - -The 2,000,000-row threshold in `BACKFILL_CHUNK_ROW_THRESHOLD` separates two regimes: - -| Mode | Prices (12.7 M rows) | Peak anon | memory.peak | Outcome | -| -------------------------------------------- | --------------------------- | ---------------------- | ------------- | ------------ | -| **Chunked (default, ≤ 1 extent/batch)** | 21 chunks × ~600 K avg rows | **14.71 GiB** | **15.03 GiB** | ✅ completes | -| **Single-pass** (threshold raised to bypass) | all 12.7 M rows at once | **14.74 GiB at crash** | **15.12 GiB** | ❌ OOM crash | - -### Chunked-backfill peak detail - -The 2-second poll loop captured anon climbing from 3.88 GiB (start of Prices chunked) to -14.71 GiB during the later large-extent chunks (~1 M rows each). The cumulative -memory.peak reached 15.03 GiB (93.9 % of 16 g) during this phase. - -The individual chunks do NOT add up linearly in memory because the engine releases -completed-extent memory between chunks. Each chunk's peak is dominated by: - -- the pre-existing working set (~13–14 GiB from previously loaded extents), plus -- the in-flight transform output for the current chunk (~0.5–1 GiB per ~1 M row chunk). - -### Single-pass OOM ceiling (stage 5) - -With `BACKFILL_CHUNK_ROW_THRESHOLD` bypassed, the engine attempted to materialise all -12.7 M transformed Prices rows at once: - -- Anon reached **14.74 GiB** in the 2 s before the crash. -- memory.peak rose from 15.03 GiB → **15.12 GiB** (94.5 % of 16 g) immediately before - the crash was captured by the poll loop. -- The engine process received **SIGKILL** (confirmed by Kustainer log entry) and exited. - Docker's `OOMKilled` flag was `false` — this is Kusto's internal memory-pressure - self-termination, not a kernel cgroup hard-limit breach. -- The container restarted automatically. cgroup counters (oom / oom_kill) reset to 0. -- Recovery: databases reattached from the persistent `kustainer-data/` volume; engine - became healthy within ~10 s. - -The gap between chunked peak (15.03 GiB) and single-pass crash (15.12 GiB) is only -~0.09 GiB — there is essentially **no safe headroom** for single-pass Prices at 16 g and -this dataset size. The auto-chunking threshold exists precisely because of this margin. - ---- - -## Operating states and limits - -These tiers answer the question **"when do we approach 16 GiB?"** - -A freshly-loaded or restarted engine idles at ~7 % of MEM_LIMIT (extents on disk, lazy -materialization). You only approach the ceiling **during** the ingest+transform pipeline -for a dataset of this size. The hot working set (~87 %) is a sustained post-ingest state -that persists until the engine restarts or memory is reclaimed. - -All tiers are calibrated to `MEM_LIMIT=16g` and this specific dataset. They scale with -both `MEM_LIMIT` and dataset size. Each cell identifies the metric and source stage. - -| State | Condition | Metric (source stage) | % of MEM_LIMIT | Notes | -| ------------------- | ---------------------------------------------------------------------------------- | --------------------------------------------------------------- | --------------------------------- | ------------------------------------------------------------------------------ | -| **Cold loaded** | Post-restart or fresh `make up`, before heavy queries | anon: **1.12 GiB** (stage 6) | 7.0 % | Floor; extents on disk, not yet materialized. Data intact and queryable. | -| **Hot post-ingest** | Immediately after `make ingest` completes; extents fully materialized by transform | anon: **13.99 GiB** (stage 3d) | 87.4 % | Sustained until restart or OS reclaim; this is NOT the steady-state floor | -| **Degraded** | Prices chunked-backfill high-water (during `make ingest`) | memory.peak: **15.03 GiB** (stage 3d cumulative HWM) | 93.9 % | Within 0.09 GiB of single-pass crash; system completes but headroom is minimal | -| **Ceiling** | Single-pass Prices transform attempt | anon: ~14.74 GiB / memory.peak: **15.12 GiB** → crash (stage 5) | 92.1 % anon / 94.5 % peak → crash | Kusto self-terminates (SIGKILL); container restarts automatically | - -### When to raise MEM_LIMIT - -- If your Prices dataset significantly exceeds 12.7 M rows, raise `MEM_LIMIT` before - running `make ingest`. Each additional million Prices rows adds roughly 0.5–1 GiB to the - idle working set (extrapolated from the Prices_final contribution observed here). -- `MEM_LIMIT=8g` is sufficient for Costs-only workflows (Costs_final peaks at 6.89 GiB - during backfill, then settles to ~3.9 GiB idle). It is **not sufficient** for - Prices at these dataset sizes. -- After raising `MEM_LIMIT`, update `BACKFILL_CHUNK_ROW_THRESHOLD` accordingly and - re-run the single-pass OOM test before relying on single-pass for larger row counts. - ---- - -## Environment - -| Setting | Value | -| ---------------------- | --------------------------------------------------------------- | -| MEM_LIMIT | 16g (17,179,869,184 bytes; confirmed via `memory.max`) | -| Kustainer image | `mcr.microsoft.com/azuredataexplorer/kustainer-linux:latest` | -| Kustainer build | `1.0.9656.17219` (2026-06-09) | -| Platform | linux/amd64 on Rosetta (Apple Silicon, Docker Desktop) | -| Host OS | macOS (Apple Silicon, M-series) | -| cgroup version | v2 (confirmed) | -| memory.peak reset | Not possible — kernel rejected write (read-only filesystem) | -| memory.peak semantics | Cumulative since container start; NOT resettable between stages | -| Poll interval (ingest) | 3 s | -| Poll interval (OOM) | 2 s | - ---- - -## Cleanliness - -No absolute host paths, billing scope IDs, storage account names, or real cluster URIs -appear in this file. Dataset size is described in row counts only. diff --git a/src/templates/finops-hub-local/notes/staging-contract.md b/src/templates/finops-hub-local/notes/staging-contract.md deleted file mode 100644 index 5f3a57f29..000000000 --- a/src/templates/finops-hub-local/notes/staging-contract.md +++ /dev/null @@ -1,364 +0,0 @@ -# ftklocal staging contract - -This document defines the data-in story for ftklocal: how Cost Management exports are -configured (Part 1) and how the resulting parquet files are staged locally for ingestion -into the Kusto emulator (Part 2). - ---- - -## Part 1 — Export configuration (Azure-only) - -### Honest design constraint - -Cost Management exports are **Azure-only**. `New-FinOpsCostExport` writes parquet to an -Azure Storage account; there is no local export mode. Export _configuration_ lives in Azure; -ftklocal _consumes_ the parquet that configuration produces. No Azure export was created as -part of authoring this document. - -### Relevant PowerShell cmdlets - -All three cmdlets are part of the FinOps Toolkit PowerShell module under -`src/powershell/Public/`. - ---- - -#### `New-FinOpsCostExport` - -**File:** `src/powershell/Public/New-FinOpsCostExport.ps1` - -Creates a Cost Management export that writes FOCUS parquet to Azure Storage. - -| Parameter | Required | Notes | -| ------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------- | -| `-Name` | Yes | Unique name within the scope. | -| `-Scope` | Yes | Azure resource ID of the billing scope (subscription, billing account, etc.). | -| `-Dataset` | No | Default: `FocusCost`. Use `FocusCost` for cost data or `PriceSheet` for prices. | -| `-DatasetVersion` | No | Default: `1.2-preview` (FocusCost). Use `1.2-preview` to get the latest FOCUS schema. | -| `-Format` | No | Default: `Csv`. **Set to `Parquet`** for ftklocal (ingest.ps1 expects parquet). | -| `-CompressionMode` | No | Default: `None`. `Snappy` is also accepted but adds a decompression step not tested with ftklocal. | -| `-StorageAccountId` | Yes | Resource ID of the target Azure Storage account. | -| `-StorageContainer` | No | Default: `cost-management`. Use `msexports` to match hub conventions, or any container for ftklocal staging. | -| `-StoragePath` | No | Subfolder path prefix within the container. Defaults to the scope ID. | -| `-DoNotPartition` | No | Omit this flag (leave partitioning **enabled**). Partitioning is recommended for reliability. | -| `-DoNotOverwrite` | No | Default: overwrite enabled. For ftklocal, overwrite is acceptable; for FinOps hubs, use `-DoNotOverwrite` to simplify troubleshooting. | -| `-Monthly` | No | Monthly cadence (mutually exclusive with `-OneTime`). | -| `-OneTime` | No | Single run over an explicit date range. | -| `-StartDate` / `-EndDate` | No | Date range for the export. Defaults for scheduled and one-time differ; see cmdlet help. | -| `-Backfill` | No | Number of prior months to export immediately at creation time (scheduled exports only). | -| `-Execute` | No | Run the export once immediately after creation. | - -**Minimal FOCUS/Parquet example:** - -```powershell -New-FinOpsCostExport ` - -Name 'focus-parquet-monthly' ` - -Scope '/subscriptions/' ` - -Dataset FocusCost ` - -DatasetVersion '1.2-preview' ` - -Format Parquet ` - -StorageAccountId '/subscriptions//resourceGroups//providers/Microsoft.Storage/storageAccounts/' ` - -StorageContainer 'msexports' ` - -Monthly -``` - ---- - -#### `Get-FinOpsCostExport` - -**File:** `src/powershell/Public/Get-FinOpsCostExport.ps1` - -Lists or filters existing Cost Management exports for a scope. - -| Parameter | Notes | -| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `-Scope` | Azure resource ID of the scope. Defaults to current Az context subscription. | -| `-Name` | Filter by export name. Supports wildcards. | -| `-Dataset` | Filter by dataset type (e.g., `FocusCost`, `PriceSheet`). | -| `-DatasetVersion` | Filter by schema version. | -| `-StorageAccountId` | Filter by destination storage account. | -| `-StorageContainer` | Filter by container name. Supports wildcards. | -| `-RunHistory` | Switch. When present, the API response is expanded to include run history for each export. This is the enumeration primitive a future auto-download bridge (T-5000.9) would use to discover available runs and their storage paths. | - -**Enumerate run history (future auto-bridge use):** - -```powershell -Get-FinOpsCostExport -Scope '/subscriptions/' -Dataset FocusCost -RunHistory -``` - ---- - -#### `Start-FinOpsCostExport` - -**File:** `src/powershell/Public/Start-FinOpsCostExport.ps1` - -Triggers an on-demand run of an existing export using the Cost Management Run API. - -| Parameter | Notes | -| ------------------------- | --------------------------------------------------------------- | -| `-Name` | Required. Name of the export to run. | -| `-Scope` | Azure resource ID. Defaults to current Az context subscription. | -| `-StartDate` / `-EndDate` | Optional date range override. | -| `-Backfill` | Number of prior months to run in sequence. | - ---- - -### Auto-download bridge (deferred) - -The automatic storage-to-local-folder sync is **deferred** and tracked in **T-5000.9**. In -v1, parquet files are staged manually (see Part 2). `Get-FinOpsCostExport -RunHistory` is -the intended enumeration primitive for that future bridge. - ---- - -## Part 2 — Local staging contract - -### Overview - -`ingest.ps1` walks a local directory tree and ingests parquet files into the Kusto emulator. -The contract below is the exact interface that script expects. It applies regardless of where -the parquet originated. - -### Folder layout - -``` -export/ -└── / - └── / - └── / - ├── / - │ ├── manifest.json - │ └── *.parquet - └── / ← older run; superseded if present - ├── manifest.json - └── *.parquet -``` - -- **`export/`** — root staging directory. Default: `export/` relative to the ftklocal - checkout root. Override via the `EXPORT_DIR` environment variable or `.env` file. - (`ingest.ps1`, `Get-ExportDir`.) -- **``** — a short, user-defined label for the billing scope (e.g., `ea`, - `sub-prod`, `mca-`). This becomes the `scope` column in - `Ingest_Manifest`. -- **``** — dataset type token. Must be a key in `$script:DatasetTableMap` - (`ingest.ps1`). See [Dataset type tokens](#dataset-type-tokens) below. -- **``** — export billing period in `YYYYMMDD-YYYYMMDD` format - (e.g., `20260501-20260531`). Must match this format; it becomes the `period` column. -- **``** — a UUID that identifies one export run. Multiple run directories - under the same `(scope, type, period)` are allowed; `ingest.ps1` selects only the - latest. Becomes `run_uuid` in `Ingest_Manifest`. - -### Dataset type tokens - -Defined in `$script:DatasetTableMap` in `ingest.ps1`: - -| Token | Raw table | Ingestion mapping | Cost Management dataset | -| ---------------- | ------------ | -------------------- | ----------------------- | -| `ms--focus-cost` | `Costs_raw` | `Costs_raw_mapping` | `FocusCost` | -| `ms--pricesheet` | `Prices_raw` | `Prices_raw_mapping` | `PriceSheet` | - -Any directory name under `/` that is not one of these tokens is silently skipped -with a warning (`ingest.ps1`, `Build-Plan`). - -### `manifest.json` per run - -Each `/` directory must contain a `manifest.json` file. The script reads -this file in `Read-RunManifest` (`ingest.ps1`) to obtain sort and -row-count metadata. Fields consumed by `ingest.ps1`: - -| JSON path | Required | Purpose | -| ----------------------- | ----------- | ----------------------------------------------------------------- | -| `runInfo.submittedTime` | Recommended | Primary sort key for latest-run selection. ISO 8601 string. | -| `runInfo.createdDate` | Fallback | Used when `submittedTime` is absent. | -| `dataRowCount` | Recommended | Total expected row count for the run (shown in the plan summary). | -| `blobs[].blobName` | Recommended | Matched by filename suffix to resolve per-file row counts. | -| `blobs[].dataRowCount` | Recommended | Expected rows for each parquet file (used in verify summary). | - -**Minimum viable manifest for manual staging:** - -```json -{ - "runInfo": { - "submittedTime": "", - "runId": "" - }, - "dataRowCount": , - "blobs": [ - { - "blobName": "/.parquet", - "dataRowCount": - } - ] -} -``` - -A manifest.json produced directly by Cost Management (from an `msexports/` export) already -contains all these fields and can be used without modification. - -### Parquet files - -One or more `.parquet` files in the run directory. File names must match the `blobName` -suffix pattern in `manifest.json` for per-file row counts to resolve correctly, though -mismatches are tolerated (the per-file count defaults to 0; the run is still ingested). -Files are sorted lexicographically by name before ingest (`ingest.ps1`, -`Sort-ByNameOrdinal`). - -### Latest-run selection - -For each `(scope, type, period)` triple, `ingest.ps1` collects all run directories that -contain a valid `manifest.json`, then picks the **one latest run** using a three-key sort -(ascending): `submittedTime` → `LastWriteTimeUtc` → directory name ordinal. The last entry -after sorting is the winner. (`ingest.ps1`, `Build-Plan`.) - -All other runs in the same `(scope, type, period)` are **superseded**. Their previously -ingested extents are dropped from the raw table and their rows are removed from -`Ingest_Manifest` before the winning run is ingested. (`ingest.ps1`, -`Drop-SupersededExtents`.) - -### Idempotency via `Ingest_Manifest` - -Ingest is idempotent at the file level. Before ingesting a parquet file, `ingest.ps1` -looks up the composite key `(scope, export_type, period, run_uuid, file_name)` in the -`Ingest_Manifest` table. If the key exists and the stored SHA-256 checksum matches the -file, the file is skipped. (`ingest.ps1`, `Invoke-RunIngest`.) - -`Ingest_Manifest` schema: - -| Column | Type | Notes | -| ----------------- | -------- | ----------------------------------------------------- | -| `scope` | string | Matches `` folder name. | -| `export_type` | string | Matches `` folder name. | -| `period` | string | Matches `` folder name. | -| `run_uuid` | string | Matches `` folder name. | -| `file_name` | string | Parquet file name (base name only). | -| `file_size` | long | File size in bytes at ingest time. | -| `rows_ingested` | long | Expected rows (from manifest, or 0 if not available). | -| `checksum_sha256` | string | SHA-256 hex of the parquet file. | -| `ingested_at` | datetime | UTC timestamp of ingest. | - -Defined at `ingest.ps1`, `Initialize-IngestManifestTable` / `Insert-ManifestRow`. - -### Overwrite semantics summary - -| Condition | Behavior | -| ------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Same `(scope, type, period, run-uuid)`, same file checksum | **Skip.** File is already in `Ingest_Manifest` with a matching SHA-256; no ingest, no new row. (`ingest.ps1`, `Invoke-RunIngest`.) | -| Same `(scope, type, period, run-uuid)`, **different** file checksum | **Hard error (fail-fast).** `Invoke-RunIngest` detects the prior `Ingest_Manifest` row for the same composite key with a different SHA-256 and **throws** rather than ingest — re-ingesting under the same run-uuid would duplicate rows and silently corrupt cost totals. Stage corrected data under a **new** run-uuid for the same `(scope, type, period)` to replace it (supersede). (`ingest.ps1`, `Invoke-RunIngest`.) | -| Same `(scope, type, period)`, **new** `run-uuid` | **Safe replace.** `Drop-SupersededExtents` drops extents tagged with older run-uuids from the raw table and deletes their `Ingest_Manifest` rows before the new run is ingested. (`ingest.ps1`, `Drop-SupersededExtents`.) | -| Different `(scope, type, period)` | **Independent.** No interaction with other tuples. | - -> **How to correct or replace data:** stage the replacement parquet under a **new ``** -> directory for the same `(scope, type, period)`. Give it a `manifest.json` with a later -> `runInfo.submittedTime` than the run being replaced. `ingest.ps1` will then select it as -> the latest run, drop the old run's extents and manifest rows, and ingest the new files -> cleanly. Never mutate files in an already-ingested run directory — `ingest.ps1` detects the -> checksum change and **fails fast with a hard error** rather than silently duplicating data; -> the safe overwrite path is always a new run-uuid. - ---- - -## Source modes — mapping to the local contract - -### Mode A: `msexports/` container (no hub deployed) - -When FinOps hubs is **not** deployed, Cost Management exports write directly to the -`msexports` Azure Storage container. The raw container path is: - -``` -msexports/{path}/{date-range}/{export-name}/{export-time}/{guid}/{file} -``` - -The `manifest.json` produced here contains `exportConfig.type` (dataset), -`runInfo.submittedTime`, `runInfo.runId`, and `blobs[].blobName` / `blobs[].dataRowCount`. -All fields that `ingest.ps1` consumes are present. - -**To use with ftklocal (v1 — manual staging):** - -1. Create the staging directory: `export//ms--focus-cost///` -2. Copy the parquet file(s) from the `msexports` run folder into this directory. -3. Copy the `manifest.json` from the same run folder. No field changes are needed. -4. Run `make ingest` (or `pwsh scripts/ingest.ps1`). - -**Notes:** - -- If the export was configured with `-Format Csv` (the default), the data is CSV, not - parquet, and cannot be ingested directly. **Always configure `-Format Parquet`.** -- The dataset type token (`ms--focus-cost`) corresponds to the `FocusCost` export type. - Cost Management uses a different name in `exportConfig.type` (e.g., `FocusCost`); the - `ms--focus-cost` token is ftklocal's local convention for the staging directory name. - -### Mode B: `ingestion/` container (hub already deployed) - -When FinOps hubs **is** deployed, the hub ETL pipeline converts raw `msexports/` data into -normalized FOCUS parquet and writes it to the `ingestion/` container: - -``` -ingestion/{dataset}/{date-folder-path}/{scope-id-path}/{ingestion-id}__{original-file-name}.parquet -``` - -The `ingestion/` parquet has already been normalized by the hub's FTK transforms (including -`x_SourceValues` and other derived columns). **This mode maps most cleanly to ftklocal** -because: - -- The parquet schema matches what the FTK KQL transform functions loaded via - `load-ftk-kql.ps1` expect. -- Row counts and manifest metadata are already reconciled by the hub pipeline. - -**To use with ftklocal (v1 — manual staging):** - -1. Identify the relevant `ingestion/` run folder(s) in Azure Storage. -2. Stage files: `export//ms--focus-cost///` -3. Construct a `manifest.json` (minimum viable form above) referencing the parquet files. -4. Run `make ingest`. - -### Auto-download (deferred — T-5000.9) - -In v1, **all storage-to-local staging is manual**. The automatic bridge that would use -`Get-FinOpsCostExport -RunHistory` to enumerate runs, download parquet from the storage -container, and arrange files into the `export/{scope}/{type}/{period}/{run}/` layout is -tracked in **T-5000.9** and is not implemented in v1. - ---- - -## Worked example directory tree - -The following illustrates a fully staged `export/` tree for two scopes and two dataset types: - -``` -export/ -├── ea/ -│ ├── ms--focus-cost/ -│ │ └── 20260501-20260531/ -│ │ └── a1b2c3d4-0000-0000-0000-000000000001/ -│ │ ├── manifest.json -│ │ └── focus_2026_05.parquet -│ └── ms--pricesheet/ -│ └── 20260501-20260531/ -│ └── a1b2c3d4-0000-0000-0000-000000000002/ -│ ├── manifest.json -│ └── pricesheet_2026_05.parquet -└── sub-prod/ - └── ms--focus-cost/ - └── 20260501-20260531/ - ├── a1b2c3d4-0000-0000-0000-000000000003/ ← superseded - │ ├── manifest.json - │ └── focus_v1.parquet - └── a1b2c3d4-0000-0000-0000-000000000004/ ← latest (ingested) - ├── manifest.json - └── focus_v2.parquet -``` - -In this tree, `ingest.ps1` ingests three runs (the latest of each -`(scope, type, period)` tuple): - -- `ea / ms--focus-cost / 20260501-20260531` → one run, ingested into `Costs_raw`. -- `ea / ms--pricesheet / 20260501-20260531` → one run, ingested into `Prices_raw`. -- `sub-prod / ms--focus-cost / 20260501-20260531` → two runs; the `_004` run is latest, - `_003` is superseded and its extents are dropped. - ---- - -## gitignore note - -The `export/` directory is already listed in -`src/templates/finops-hub-local/.gitignore` and is never committed to the repository. -All staged parquet and manifest files remain local only. diff --git a/src/templates/finops-hub-local/scripts/chunked-prices-backfill.ps1 b/src/templates/finops-hub-local/scripts/chunked-prices-backfill.ps1 deleted file mode 100644 index 23c7d42b8..000000000 --- a/src/templates/finops-hub-local/scripts/chunked-prices-backfill.ps1 +++ /dev/null @@ -1,49 +0,0 @@ -#!/usr/bin/env pwsh -#Requires -Version 7.0 -<# -.SYNOPSIS - Standalone chunked backfill for Prices_final_v1_2. - -.DESCRIPTION - Recovery backfill for Prices_transform_v1_2. Dot-sources ingest.ps1 and - backfills Prices_final_v1_2 from Prices_raw one extent at a time, with the same - update-policy disable/restore and Rosetta crash retry behavior as ingest.ps1. -#> -[CmdletBinding()] -param() - -Set-StrictMode -Version Latest -$ErrorActionPreference = 'Stop' - -. (Join-Path $PSScriptRoot 'ingest.ps1') - -function Get-TableCount { - param([Parameter(Mandatory)] [string] $TableName) - $postResult = Invoke-KustoPost -CslText "$TableName | count" -EndpointUrl $script:KustainerQuery -TimeoutSec 60 - if ($postResult.Status -ge 200 -and $postResult.Status -lt 300) { - $jsonObject = Get-KustoJson -BodyText $postResult.Body - return [int64](Get-PrimaryRows $jsonObject)[0][0] - } - return -1 -} - -function Invoke-ChunkedPricesBackfillMain { - Write-InfoLine ("Before: Prices_final_v1_2 = {0:N0}" -f (Get-TableCount -TableName 'Prices_final_v1_2')) - $savedPolicies = Disable-UpdatePolicies - Write-InfoLine "Disabled update policies on $($savedPolicies.Count) table(s)" - $stopwatch = [Diagnostics.Stopwatch]::StartNew() - try { - $result = Invoke-ChunkedBackfillFinalTable -RawTableName 'Prices_raw' -FinalTableName 'Prices_final_v1_2' -TransformFunctionName 'Prices_transform_v1_2' -ExtentsPerBatch 1 - } - finally { - Restore-UpdatePolicies -SavedPolicies $savedPolicies - } - $stopwatch.Stop() - $afterCount = Get-TableCount -TableName 'Prices_final_v1_2' - Write-InfoLine ("After: Prices_final_v1_2 = {0:N0}" -f $afterCount) - Write-InfoLine ("Wall-clock: {0:N1}s ({1:N1} min). OK={2}, reported_rows={3:N0}" -f $stopwatch.Elapsed.TotalSeconds, ($stopwatch.Elapsed.TotalSeconds / 60.0), $result.Ok, $result.Rows) - if ($result.Ok) { return 0 } - return 1 -} - -exit (Invoke-ChunkedPricesBackfillMain) diff --git a/src/templates/finops-hub-local/scripts/ftk.ps1 b/src/templates/finops-hub-local/scripts/ftk.ps1 deleted file mode 100644 index 2ad2b82c2..000000000 --- a/src/templates/finops-hub-local/scripts/ftk.ps1 +++ /dev/null @@ -1,495 +0,0 @@ -#!/usr/bin/env pwsh -#Requires -Version 7.0 -<# -.SYNOPSIS - ftk - a tiny CLI for the local FinOps Toolkit (Kustainer) stack. - -.DESCRIPTION - Runs KQL against the local Kusto Emulator (Kustainer) and runs the FinOps - Toolkit published query catalog locally by adapting the upstream .kql files - on read (no forked copies). - - Subcommands: - ftk query "" Run ad-hoc KQL (.-prefixed = mgmt command). - ftk schema [-Tables|-Functions] - List tables and/or functions in the database. - ftk tables Show one table's column schema (cslschema). - ftk list List the catalog queries discovered on disk. - ftk run [opts] Adapt + run a named catalog query locally. - - Connection (overridable with -Endpoint / -Database): - KUSTAINER_QUERY full query URL (default http://localhost:/v1/rest/query) - HOST_PORT port (default 8082; read from .env if present) - FTK_DB database name (default Hub) - Never point this at port 8080 (that is the maenifold MCP gateway). - - Catalog (FTK_CATALOG_PATH overrides; else FTK_REPO / ../finops-toolkit): - 1. $FTK_CATALOG_PATH - 2. /release/agent-skills/finops-toolkit/references/queries/catalog - 3. /src/queries/catalog - -.EXAMPLE - pwsh scripts/ftk.ps1 query "Costs() | summarize sum(EffectiveCost) by ServiceName | top 10 by sum_EffectiveCost" - -.EXAMPLE - pwsh scripts/ftk.ps1 run savings-summary-report --format csv -#> -[CmdletBinding()] -param( - [Parameter(Position = 0)] - [string] $Command, - - [Parameter(Position = 1)] - [string] $Arg1, - - [Parameter(ValueFromRemainingArguments = $true)] - [string[]] $Rest -) - -Set-StrictMode -Version Latest -$ErrorActionPreference = 'Stop' - -$script:RepoRoot = Split-Path -Parent (Split-Path -Parent $PSCommandPath) - -# --------------------------------------------------------------------------- # -# Connection resolution -# --------------------------------------------------------------------------- # -function Get-HostPort { - if ($env:HOST_PORT) { return $env:HOST_PORT } - $envFile = Join-Path $script:RepoRoot '.env' - if (Test-Path $envFile) { - foreach ($line in Get-Content $envFile) { - $t = $line.Trim() - if ($t -like 'HOST_PORT=*') { return ($t -split '=', 2)[1].Trim() } - } - } - return '8082' -} - -function Get-QueryUrl { - param([string] $Override) - if ($Override) { return $Override } - if ($env:KUSTAINER_QUERY) { return $env:KUSTAINER_QUERY } - return "http://localhost:$(Get-HostPort)/v1/rest/query" -} - -function Get-MgmtUrl { - param([string] $Override) - return (Get-QueryUrl $Override) -replace '/v1/rest/query', '/v1/rest/mgmt' -} - -function Get-Database { - param([string] $Override) - if ($Override) { return $Override } - if ($env:FTK_DB) { return $env:FTK_DB } - return 'Hub' -} - -# --------------------------------------------------------------------------- # -# REST client (same REST pattern used across the ftklocal tooling) -# --------------------------------------------------------------------------- # -function Invoke-Kusto { - <# - Run a query (or .mgmt command) and return a PSCustomObject with - .Columns (string[]) and .Rows (object[][]). - #> - param( - [Parameter(Mandatory)] [string] $Csl, - [string] $Endpoint, - [string] $Database, - [int] $TimeoutSec = 120 - ) - $db = Get-Database $Database - $isMgmt = $Csl.TrimStart().StartsWith('.') - $url = if ($isMgmt) { Get-MgmtUrl $Endpoint } else { Get-QueryUrl $Endpoint } - $body = @{ db = $db; csl = $Csl } | ConvertTo-Json -Compress - $headers = @{ - 'Content-Type' = 'application/json' - 'x-ms-client-version' = 'Kusto.Python.Client:1.0.0' # unlocks full JSON error bodies - } - try { - $resp = Invoke-RestMethod -Method Post -Uri $url -Headers $headers -Body $body ` - -TimeoutSec $TimeoutSec - } - catch { - throw (ConvertTo-KustoError $_ $url) - } - if (-not $resp.Tables) { - return [pscustomobject]@{ Columns = @(); Rows = @() } - } - $table = $resp.Tables[0] - $cols = @($table.Columns | ForEach-Object { $_.ColumnName }) - return [pscustomobject]@{ Columns = $cols; Rows = $table.Rows } -} - -function ConvertTo-KustoError { - param($ErrorRecord, [string] $Url) - $raw = $null - try { $raw = $ErrorRecord.ErrorDetails.Message } catch { } - if (-not $raw) { - try { - $stream = $ErrorRecord.Exception.Response.GetResponseStream() - $raw = (New-Object IO.StreamReader($stream)).ReadToEnd() - } catch { } - } - if (-not $raw) { - return "Cannot reach Kustainer at $Url ($($ErrorRecord.Exception.Message)). Is the container up? Try ``make up``." - } - try { - $obj = $raw | ConvertFrom-Json - foreach ($key in 'error', 'Error') { - if ($obj.PSObject.Properties.Name -contains $key -and $obj.$key) { - $err = $obj.$key - $msg = $err.'@message'; if (-not $msg) { $msg = $err.message }; if (-not $msg) { $msg = $err.'@type' } - $extra = $null - if ($err.PSObject.Properties.Name -contains '@context' -and $err.'@context') { - $extra = $err.'@context'.'@message' - } - if ($extra) { return "Kusto error: $msg | $extra" } - return "Kusto error: $msg" - } - } - } catch { } - return "Kusto error: $($raw.Substring(0, [Math]::Min(400, $raw.Length)))" -} - -function Get-Scalar { - param([string] $Csl, [string] $Endpoint, [string] $Database) - $r = Invoke-Kusto -Csl $Csl -Endpoint $Endpoint -Database $Database - if ($r.Rows -and $r.Rows.Count -gt 0 -and $r.Rows[0].Count -gt 0) { return $r.Rows[0][0] } - return $null -} - -# --------------------------------------------------------------------------- # -# Output formatting -# --------------------------------------------------------------------------- # -function Format-Result { - param([string[]] $Columns, $Rows, [string] $Format) - - if ($Format -eq 'json') { - $list = foreach ($row in $Rows) { - $o = [ordered]@{} - for ($i = 0; $i -lt $Columns.Count; $i++) { $o[$Columns[$i]] = $row[$i] } - [pscustomobject]$o - } - return ($list | ConvertTo-Json -Depth 10) - } - if ($Format -eq 'csv') { - $sb = [Text.StringBuilder]::new() - [void]$sb.AppendLine(($Columns -join ',')) - foreach ($row in $Rows) { - $cells = for ($i = 0; $i -lt $Columns.Count; $i++) { - $v = if ($null -eq $row[$i]) { '' } else { [string]$row[$i] } - if ($v -match '[",\n]') { '"' + ($v -replace '"', '""') + '"' } else { $v } - } - [void]$sb.AppendLine(($cells -join ',')) - } - return $sb.ToString().TrimEnd("`r", "`n") - } - # table (default) - if (-not $Columns -or $Columns.Count -eq 0) { return '(no columns)' } - $widths = @($Columns | ForEach-Object { $_.Length }) - $strRows = @() - foreach ($row in $Rows) { - $sr = for ($i = 0; $i -lt $Columns.Count; $i++) { if ($null -eq $row[$i]) { '' } else { [string]$row[$i] } } - $sr = @($sr) - $strRows += , $sr - for ($i = 0; $i -lt $Columns.Count; $i++) { if ($sr[$i].Length -gt $widths[$i]) { $widths[$i] = $sr[$i].Length } } - } - $sep = ' ' - $lines = @() - $lines += (0..($Columns.Count - 1) | ForEach-Object { $Columns[$_].PadRight($widths[$_]) }) -join $sep - $lines += (0..($Columns.Count - 1) | ForEach-Object { '-' * $widths[$_] }) -join $sep - foreach ($sr in $strRows) { - $lines += (0..($Columns.Count - 1) | ForEach-Object { $sr[$_].PadRight($widths[$_]) }) -join $sep - } - $n = @($Rows).Count - $lines += '' - $lines += "($n row$(if ($n -ne 1) { 's' }))" - return ($lines -join "`n") -} - -# --------------------------------------------------------------------------- # -# Catalog discovery -# --------------------------------------------------------------------------- # -$script:LibraryQueries = @('costs-enriched-base') - -function Get-FtkRepo { - if ($env:FTK_REPO) { return $env:FTK_REPO } - # ftklocal lives in-repo at /src/templates/finops-hub-local, so the toolkit - # root (which holds src/queries/catalog) is three directories up from $RepoRoot. - $inRepoRoot = Split-Path -Parent (Split-Path -Parent (Split-Path -Parent $script:RepoRoot)) - if ($inRepoRoot -and (Test-Path (Join-Path $inRepoRoot 'src/queries/catalog') -PathType Container)) { - return $inRepoRoot - } - # Robust fallback for worktrees / unusual layouts: the git top-level. - # Guard on git being present so a missing git degrades to the legacy fallback - # instead of throwing under $ErrorActionPreference = 'Stop'. - if (Get-Command git -ErrorAction SilentlyContinue) { - $gitTop = (& git -C $script:RepoRoot rev-parse --show-toplevel 2>$null) - if ($gitTop -and (Test-Path (Join-Path $gitTop 'src/queries/catalog') -PathType Container)) { - return $gitTop - } - } - # Last resort: legacy prototype layout where ftklocal sat beside a finops-toolkit clone. - return (Resolve-Path (Join-Path (Split-Path -Parent $script:RepoRoot) 'finops-toolkit') -ErrorAction SilentlyContinue) ` - ?? (Join-Path (Split-Path -Parent $script:RepoRoot) 'finops-toolkit') -} - -function Get-CatalogDir { - $candidates = @() - if ($env:FTK_CATALOG_PATH) { $candidates += $env:FTK_CATALOG_PATH } - $repo = Get-FtkRepo - $candidates += (Join-Path $repo 'release/agent-skills/finops-toolkit/references/queries/catalog') - $candidates += (Join-Path $repo 'src/queries/catalog') - foreach ($c in $candidates) { if (Test-Path $c -PathType Container) { return $c } } - return $null -} - -function Get-CatalogQueries { - $d = Get-CatalogDir - if (-not $d) { return @() } - return @(Get-ChildItem -Path $d -Filter '*.kql' | Sort-Object Name) -} - -function Get-QueryDescription { - param([string] $Path) - $lines = Get-Content -Path $Path - for ($i = 0; $i -lt $lines.Count; $i++) { - if ($lines[$i] -match '//\s*Description:') { - for ($j = $i + 1; $j -lt [Math]::Min($i + 4, $lines.Count); $j++) { - if ($lines[$j] -match '^\s*//\s*(.+)') { $t = $Matches[1].Trim(); if ($t) { return $t } } - } - } - } - foreach ($line in $lines) { - if ($line -match '^\s*//\s*Query:\s*(.+)') { return $Matches[1].Trim() } - } - return '' -} - -# --------------------------------------------------------------------------- # -# Query adapter (the only local-compat logic) -# --------------------------------------------------------------------------- # -$script:SchemaCache = @{} - -function Get-FunctionSchema { - param([string] $Fn, [string] $Endpoint, [string] $Database) - $key = "$(Get-Database $Database)::$Fn" - if ($script:SchemaCache.ContainsKey($key)) { return $script:SchemaCache[$key] } - $cols = @{} - try { - $r = Invoke-Kusto -Csl "$Fn() | getschema | project ColumnName" -Endpoint $Endpoint -Database $Database - foreach ($row in $r.Rows) { $cols[$row[0]] = $true } - } catch { } - $script:SchemaCache[$key] = $cols - return $cols -} - -function ConvertTo-KqlLiteral { - param([string] $Value) - $v = $Value.Trim() - if ($v -match '^-?\d+$') { return $v } - if ($v -match '^-?\d*\.\d+$') { return $v } - if ($v -in 'true', 'false') { return $v.ToLower() } - if ($v -match '^\d{4}-\d{2}-\d{2}([ T].*)?$') { return "datetime($($v -replace ' ', 'T'))" } - return "'" + ($v -replace "'", "\'") + "'" -} - -function Get-InferredSpanMonths { - param([string] $Rhs) - if ($Rhs -match 'ago\((\d+)d\)') { return [Math]::Max(1, [Math]::Round([int]$Matches[1] / 30.0)) } - if ($Rhs -match 'monthsago\((\d+)\)') { return [Math]::Max(1, [int]$Matches[1]) } - return 1 -} - -function Resolve-Window { - param([string] $Kql, [string] $Start, [string] $End, [string] $Endpoint, [string] $Database) - if ($Start -or $End) { - $s = if ($Start) { ConvertTo-KqlLiteral $Start } else { $null } - $e = if ($End) { ConvertTo-KqlLiteral $End } else { $null } - return @($s, $e) - } - $span = 1 - if ($Kql -match 'let\s+startDate\s*=\s*([^;]+);') { $span = Get-InferredSpanMonths $Matches[1] } - $maxRaw = $null - try { $maxRaw = Get-Scalar -Csl 'Costs() | summarize max(ChargePeriodStart)' -Endpoint $Endpoint -Database $Database } catch { $maxRaw = $null } - if (-not $maxRaw) { return @($null, $null) } - $maxd = ([datetime]$maxRaw).ToUniversalTime() - $endDt = (Get-Date -Year $maxd.Year -Month $maxd.Month -Day 1 -Hour 0 -Minute 0 -Second 0).AddMonths(1) - $startDt = $endDt.AddMonths(-$span) - $fmt = { param($d) "datetime($($d.ToString('yyyy-MM-ddTHH:mm:ssZ')))" } - return @((& $fmt $startDt), (& $fmt $endDt)) -} - -function Invoke-ProjectAwayTolerance { - param([string] $Kql, [string] $Endpoint, [string] $Database) - if ($Kql -notmatch 'project-away') { return $Kql } - if ($Kql -notmatch '\b(Costs|Prices|Recommendations|Transactions)\(\)') { return $Kql } - $fn = $Matches[1] - $schema = Get-FunctionSchema -Fn $fn -Endpoint $Endpoint -Database $Database - if ($schema.Count -eq 0) { return $Kql } - return [regex]::Replace($Kql, '(\|\s*project-away\s+)([^\n|]+)', { - param($m) - $prefix = $m.Groups[1].Value - $names = $m.Groups[2].Value -split ',' | ForEach-Object { $_.Trim() } | Where-Object { $_ } - $kept = @($names | Where-Object { $schema.ContainsKey($_) }) - if ($kept.Count -eq 0) { return '' } - return $prefix + ($kept -join ', ') - }) -} - -function Convert-CatalogQuery { - param( - [string] $Kql, - [hashtable] $Params, - [string] $Start, - [string] $End, - [string] $Endpoint, - [string] $Database - ) - $out = $Kql - - # 1. Date window - $win = Resolve-Window -Kql $out -Start $Start -End $End -Endpoint $Endpoint -Database $Database - if ($null -ne $win[0]) { - $out = [regex]::Replace($out, '(let\s+startDate\s*=\s*)([^;]+)(;)', { param($m) $m.Groups[1].Value + $win[0] + $m.Groups[3].Value }, 1) - } - if ($null -ne $win[1]) { - $out = [regex]::Replace($out, '(let\s+endDate\s*=\s*)([^;]+)(;)', { param($m) $m.Groups[1].Value + $win[1] + $m.Groups[3].Value }, 1) - } - - # 1b. Arbitrary --param overrides on top-level let bindings (typed literals). - if ($Params) { - foreach ($k in $Params.Keys) { - $lit = ConvertTo-KqlLiteral $Params[$k] - $pat = "(let\s+$([regex]::Escape($k))\s*=\s*)([^;]+)(;)" - if ($out -notmatch $pat) { - throw "--param '$k' does not match a top-level ``let $k = ...;`` in the query" - } - $out = [regex]::Replace($out, $pat, { param($m) $m.Groups[1].Value + $lit + $m.Groups[3].Value }, 1) - } - } - - # 2. decimal -> real normalization (the verified Kustainer compatibility fix) - $out = $out -replace "todecimal\(\s*''\s*\)", 'real(null)' - $out = $out -replace '\btodecimal\s*\(', 'toreal(' - $out = $out -replace '\bdecimal\s*\(', 'real(' - - # 3. project-away tolerance: drop columns the local schema doesn't expose - $out = Invoke-ProjectAwayTolerance -Kql $out -Endpoint $Endpoint -Database $Database - - return $out -} - -# --------------------------------------------------------------------------- # -# Option parsing for the remaining args (mirrors the CLI flag surface) -# --------------------------------------------------------------------------- # -function Read-Options { - param([string[]] $Tokens) - $opts = @{ format = 'table'; params = @{}; positional = @() } - $i = 0 - while ($i -lt $Tokens.Count) { - switch -Regex ($Tokens[$i]) { - '^--format$|^-Format$' { $opts.format = $Tokens[++$i] } - '^--param$|^-Param$' { $kv = $Tokens[++$i]; $p = $kv -split '=', 2; if ($p.Count -ne 2) { throw "--param must be key=value, got '$kv'" }; $opts.params[$p[0].Trim()] = $p[1] } - '^--start$|^-Start$' { $opts.start = $Tokens[++$i] } - '^--end$|^-End$' { $opts.end = $Tokens[++$i] } - '^--show$|^-Show$' { $opts.show = $true } - '^--tables$|^-Tables$' { $opts.tables = $true } - '^--functions$|^-Functions$' { $opts.functions = $true } - '^--endpoint$|^-Endpoint$' { $opts.endpoint = $Tokens[++$i] } - '^--database$|^-Database$' { $opts.database = $Tokens[++$i] } - default { $opts.positional += $Tokens[$i] } - } - $i++ - } - return $opts -} - -# --------------------------------------------------------------------------- # -# Subcommands -# --------------------------------------------------------------------------- # -function Show-Usage { - @' -ftk - local FinOps Toolkit query CLI - - ftk query "" [--format table|json|csv] - ftk schema [--tables] [--functions] - ftk tables - ftk list - ftk run [--param k=v] [--start ] [--end ] [--show] [--format ...] - - Global: [--endpoint ] [--database ] -'@ -} - -$argv = @() -if ($Arg1) { $argv += $Arg1 } -if ($Rest) { $argv += $Rest } -$opt = Read-Options $argv - -$endpoint = if ($opt.ContainsKey('endpoint')) { $opt.endpoint } else { $null } -$db = if ($opt.ContainsKey('database')) { $opt.database } else { $null } - -try { - switch ($Command) { - 'query' { - $kql = $opt.positional[0] - if (-not $kql) { throw 'usage: ftk query ""' } - $r = Invoke-Kusto -Csl $kql -Endpoint $endpoint -Database $db - Write-Output (Format-Result -Columns $r.Columns -Rows $r.Rows -Format $opt.format) - } - 'schema' { - $wantTables = $opt.ContainsKey('tables') -or -not $opt.ContainsKey('functions') - $wantFuncs = $opt.ContainsKey('functions') -or -not $opt.ContainsKey('tables') - if ($wantTables) { - $r = Invoke-Kusto -Csl '.show tables | project TableName | order by TableName asc' -Endpoint $endpoint -Database $db - Write-Output 'Tables:' - foreach ($row in $r.Rows) { Write-Output " $($row[0])" } - } - if ($wantFuncs) { - $r = Invoke-Kusto -Csl '.show functions | project Name | order by Name asc' -Endpoint $endpoint -Database $db - Write-Output 'Functions:' - foreach ($row in $r.Rows) { Write-Output " $($row[0])" } - } - } - 'tables' { - $name = $opt.positional[0] - if (-not $name) { throw 'usage: ftk tables ' } - $r = Invoke-Kusto -Csl ".show table $name cslschema | project Schema" -Endpoint $endpoint -Database $db - if ($r.Rows) { Write-Output $r.Rows[0][0] } - } - 'list' { - $d = Get-CatalogDir - if (-not $d) { Write-Error 'No catalog found. Set FTK_CATALOG_PATH or FTK_REPO (looked under ../finops-toolkit).'; exit 1 } - Write-Output "Catalog: $d`n" - foreach ($p in Get-CatalogQueries) { - $nm = [IO.Path]::GetFileNameWithoutExtension($p.Name) - $tag = if ($script:LibraryQueries -contains $nm) { ' [library]' } else { '' } - $desc = Get-QueryDescription $p.FullName - Write-Output (' {0,-42}{1} {2}' -f $nm, $tag, $desc) - } - } - 'run' { - $name = $opt.positional[0] - if (-not $name) { throw 'usage: ftk run ' } - if ($script:LibraryQueries -contains $name) { throw "'$name' is a library building block, not a runnable query." } - $path = $null - foreach ($p in Get-CatalogQueries) { if ([IO.Path]::GetFileNameWithoutExtension($p.Name) -eq $name) { $path = $p.FullName; break } } - if (-not $path) { throw "Catalog query '$name' not found. Try ``ftk list``." } - $raw = Get-Content -Raw -Path $path - $start = if ($opt.ContainsKey('start')) { $opt.start } else { $null } - $end = if ($opt.ContainsKey('end')) { $opt.end } else { $null } - $adapted = Convert-CatalogQuery -Kql $raw -Params $opt.params -Start $start -End $end -Endpoint $endpoint -Database $db - if ($opt.ContainsKey('show')) { Write-Output $adapted; break } - $r = Invoke-Kusto -Csl $adapted -Endpoint $endpoint -Database $db - Write-Output (Format-Result -Columns $r.Columns -Rows $r.Rows -Format $opt.format) - } - default { Show-Usage; if ($Command) { exit 2 } } - } -} -catch { - Write-Error "error: $($_.Exception.Message)" - exit 2 -} diff --git a/src/templates/finops-hub-local/scripts/ingest.ps1 b/src/templates/finops-hub-local/scripts/ingest.ps1 deleted file mode 100644 index 35f53df7f..000000000 --- a/src/templates/finops-hub-local/scripts/ingest.ps1 +++ /dev/null @@ -1,935 +0,0 @@ -#!/usr/bin/env pwsh -#Requires -Version 7.0 -<# -.SYNOPSIS - Bulk-ingest FOCUS Cost + Price Sheet parquet exports into local Kustainer raw tables. - -.DESCRIPTION - Bulk-ingests FOCUS parquet exports. Walks export/{scope}/{type}/{period}/{run-uuid}, - selects the latest manifest run per (scope,type,period), tracks idempotency in - Ingest_Manifest by checksum, enforces overwrite semantics for superseded runs, disables - transactional final-table update policies during bulk ingest, and explicitly backfills finals. - -.PARAMETER Scope - Filter to one scope subfolder under export/. - -.PARAMETER Period - Filter to one period (YYYYMMDD-YYYYMMDD). - -.PARAMETER DryRun - Plan and print work, including manifest skip decisions, without mutating Kustainer. - -.PARAMETER ForcePolicyRecapture - Allow capturing update policies that are already disabled. Dangerous; mirrors the ingest policy-capture guard. - -.EXAMPLE - pwsh scripts/ingest.ps1 --dry-run - -.EXAMPLE - pwsh scripts/ingest.ps1 --scope ea --period 20260501-20260531 -#> -[CmdletBinding()] -param( - [Parameter(ValueFromRemainingArguments = $true)] - [string[]] $CliArgs -) - -Set-StrictMode -Version Latest -$ErrorActionPreference = 'Stop' - -$script:RepoRoot = Split-Path -Parent (Split-Path -Parent $PSCommandPath) - -function Get-ExportDir { - if ($env:EXPORT_DIR) { return $env:EXPORT_DIR } - $envFilePath = Join-Path $script:RepoRoot '.env' - if (Test-Path -LiteralPath $envFilePath) { - foreach ($lineText in Get-Content -LiteralPath $envFilePath) { - $trimmed = $lineText.Trim() - if ($trimmed -like 'EXPORT_DIR=*') { return ($trimmed -split '=', 2)[1].Trim() } - } - } - return (Join-Path $script:RepoRoot 'export') -} - -$script:ExportDirHost = Get-ExportDir -$script:ExportDirContainer = '/data/export' -$script:DbName = if ($env:FTK_DB) { $env:FTK_DB } else { 'Ingestion' } -$script:IngestManifestTable = 'Ingest_Manifest' -$script:RowCountTolerance = 0.001 -$script:IngestTimeoutSec = 1800 -$script:DatasetTableMap = @{ - 'ms--focus-cost' = [pscustomobject]@{ Table = 'Costs_raw'; Mapping = 'Costs_raw_mapping' } - 'ms--pricesheet' = [pscustomobject]@{ Table = 'Prices_raw'; Mapping = 'Prices_raw_mapping' } -} -$script:RawToFinalPolicy = @{ - 'Costs_raw' = [pscustomobject]@{ FinalTable = 'Costs_final_v1_2'; TransformFn = 'Costs_transform_v1_2' } - 'Prices_raw' = [pscustomobject]@{ FinalTable = 'Prices_final_v1_2'; TransformFn = 'Prices_transform_v1_2' } -} - -# Proactive chunked-backfill threshold (rows). -# Measured baseline: MEM_LIMIT=16g, amd64-Rosetta; idle-loaded engine ~12.4 GiB working -# set (78% of 16g), ~3.5 GiB headroom. Costs 1.35M rows → single-pass OK; Prices 12.7M -# rows → single-pass OOMs (>16 GiB). Default 2,000,000 is safely above the largest known -# single-pass success (1.35M) and well below the smallest known OOM point (12.7M). -# OOM risk is row-driven, not extent-driven — use the reactive fallback below as the safety -# net for any table whose single-pass headroom is unknown. -# Override at runtime: BACKFILL_CHUNK_ROW_THRESHOLD=5000000 pwsh scripts/ingest.ps1 -$script:BackfillChunkRowThreshold = if ($env:BACKFILL_CHUNK_ROW_THRESHOLD) { [int64]$env:BACKFILL_CHUNK_ROW_THRESHOLD } else { [int64]2000000 } - -function Get-HostPort { - if ($env:HOST_PORT) { return $env:HOST_PORT } - $envFilePath = Join-Path $script:RepoRoot '.env' - if (Test-Path -LiteralPath $envFilePath) { - foreach ($lineText in Get-Content -LiteralPath $envFilePath) { - $trimmed = $lineText.Trim() - if ($trimmed -like 'HOST_PORT=*') { return ($trimmed -split '=', 2)[1].Trim() } - } - } - return '8082' -} - -function Get-KustainerMgmtUrl { - if ($env:KUSTAINER_MGMT) { return $env:KUSTAINER_MGMT } - return "http://localhost:$(Get-HostPort)/v1/rest/mgmt" -} - -function Get-KustainerQueryUrl { - return (Get-KustainerMgmtUrl) -replace '/mgmt$', '/query' -} - -$script:KustainerMgmt = Get-KustainerMgmtUrl -$script:KustainerQuery = Get-KustainerQueryUrl - -function New-PostResult { - param([int] $StatusCode, [string] $BodyText) - [pscustomobject]@{ Status = $StatusCode; Body = $BodyText } -} - -function Write-InfoLine { - param([string] $Message = '') - [Console]::Out.WriteLine($Message) -} - -function Sort-ByNameOrdinal { - param([object[]] $Items) - $arr = @($Items) - [Array]::Sort($arr, [Comparison[object]] { param($left, $right) [StringComparer]::Ordinal.Compare($left.Name, $right.Name) }) - return $arr -} - -function Invoke-KustoPost { - param( - [Parameter(Mandatory)] [string] $CslText, - [string] $EndpointUrl = $script:KustainerMgmt, - [int] $TimeoutSec = 60 - ) - $payload = @{ db = $script:DbName; csl = $CslText } | ConvertTo-Json -Compress - $headers = @{ - 'Content-Type' = 'application/json' - 'x-ms-client-version' = 'Kusto.Python.Client:1.0.0' - 'Accept' = 'application/json' - } - try { - $response = Invoke-WebRequest -Method Post -Uri $EndpointUrl -Headers $headers -Body $payload ` - -TimeoutSec $TimeoutSec -SkipHttpErrorCheck - return (New-PostResult -StatusCode ([int]$response.StatusCode) -BodyText ([string]$response.Content)) - } - catch { - $bodyText = $null - try { $bodyText = $_.ErrorDetails.Message } catch { } - if (-not $bodyText) { - try { - $stream = $_.Exception.Response.GetResponseStream() - $bodyText = (New-Object IO.StreamReader($stream)).ReadToEnd() - } catch { } - } - # Guard against strict-mode PropertyNotFoundException when the exception has no .Response property. - try { - if ($_.Exception.Response -and $bodyText) { - try { return (New-PostResult -StatusCode ([int]$_.Exception.Response.StatusCode) -BodyText $bodyText) } catch { } - } - } catch { } - $messageText = "Connection error: $($_.Exception.GetType().Name): $($_.Exception.Message)" - if ($_.Exception.GetType().Name -match 'WebException|HttpRequestException|IOException') { - return (New-PostResult -StatusCode 599 -BodyText $messageText) - } - return (New-PostResult -StatusCode 599 -BodyText $messageText) - } -} - -function Get-KustoJson { - param([Parameter(Mandatory)] [string] $BodyText) - return ($BodyText | ConvertFrom-Json) -} - -function Get-PrimaryRows { - param($JsonObject) - if (-not $JsonObject.Tables -or @($JsonObject.Tables).Count -eq 0) { return ,@() } - return ,@($JsonObject.Tables[0].Rows) -} - -function Get-PrimaryColumnNames { - param($JsonObject) - if (-not $JsonObject.Tables -or @($JsonObject.Tables).Count -eq 0) { return ,@() } - return ,@($JsonObject.Tables[0].Columns | ForEach-Object { $_.ColumnName }) -} - -function Get-QueryScalar { - param([Parameter(Mandatory)] [string] $CslText) - $postResult = Invoke-KustoPost -CslText $CslText -EndpointUrl $script:KustainerQuery -TimeoutSec 60 - if ($postResult.Status -lt 200 -or $postResult.Status -ge 300) { return $null } - try { - $jsonObject = Get-KustoJson -BodyText $postResult.Body - $rows = Get-PrimaryRows $jsonObject - if ($rows.Count -eq 0) { return $null } - return $rows[0][0] - } - catch { return $null } -} - -function Parse-KustoError { - param([string] $BodyText) - try { - $jsonObject = $BodyText | ConvertFrom-Json - $err = $null - if ($jsonObject.PSObject.Properties.Name -contains 'error') { $err = $jsonObject.error } - if ($err) { - $errType = $err.'@type'; if (-not $errType) { $errType = $err.code }; if (-not $errType) { $errType = 'Unknown' } - $errMsg = $err.'@message'; if (-not $errMsg) { $errMsg = $err.message }; if (-not $errMsg) { $errMsg = $BodyText.Substring(0, [Math]::Min(200, $BodyText.Length)) } - return @([string]$errType, [string]$errMsg) - } - } - catch { } - return @('RawText', $BodyText.Substring(0, [Math]::Min(300, $BodyText.Length))) -} - -function Get-JsonPropertyValue { - param($ObjectValue, [string] $Name, $DefaultValue = $null) - if ($null -eq $ObjectValue) { return $DefaultValue } - if ($ObjectValue.PSObject.Properties.Name -contains $Name) { return $ObjectValue.$Name } - return $DefaultValue -} - -function Get-SubmittedTime { - param($ManifestObject) - $runInfo = Get-JsonPropertyValue -ObjectValue $ManifestObject -Name 'runInfo' - $submitted = Get-JsonPropertyValue -ObjectValue $runInfo -Name 'submittedTime' - if ($submitted) { return [string]$submitted } - $created = Get-JsonPropertyValue -ObjectValue $runInfo -Name 'createdDate' - if ($created) { return [string]$created } - return $null -} - -function Read-RunManifest { - param([Parameter(Mandatory)] [string] $RunDirPath) - $manifestPath = Join-Path $RunDirPath 'manifest.json' - if (-not (Test-Path -LiteralPath $manifestPath -PathType Leaf)) { return $null } - try { return (Get-Content -LiteralPath $manifestPath -Raw -Encoding UTF8 | ConvertFrom-Json) } - catch { [Console]::Error.WriteLine(" WARN: cannot read $manifestPath`: $($_.Exception.Message)"); return $null } -} - -function New-RunPlan { - param( - [string] $ScopeName, - [string] $ExportTypeName, - [string] $PeriodName, - [string] $RunUuidValue, - [string] $RunDirPath, - [string] $SubmittedTimeValue, - [int64] $ExpectedRowsValue, - [object[]] $ParquetFilesValue, - [string[]] $SupersededRunUuidsValue - ) - [pscustomobject]@{ - Scope = $ScopeName - ExportType = $ExportTypeName - Period = $PeriodName - RunUuid = $RunUuidValue - RunDir = $RunDirPath - SubmittedTime = $SubmittedTimeValue - ExpectedRows = $ExpectedRowsValue - ParquetFiles = @($ParquetFilesValue) - SupersededRunUuids = @($SupersededRunUuidsValue) - } -} - -function Build-Plan { - param([string] $ExportRootPath, [string] $ScopeFilter, [string] $PeriodFilter) - $plans = @() - if (-not (Test-Path -LiteralPath $ExportRootPath -PathType Container)) { throw "export root not found: $ExportRootPath" } - foreach ($scopeDir in @(Sort-ByNameOrdinal @(Get-ChildItem -LiteralPath $ExportRootPath -Directory))) { - if ($ScopeFilter -and $scopeDir.Name -ne $ScopeFilter) { continue } - foreach ($typeDir in @(Sort-ByNameOrdinal @(Get-ChildItem -LiteralPath $scopeDir.FullName -Directory))) { - if (-not $script:DatasetTableMap.ContainsKey($typeDir.Name)) { - [Console]::Error.WriteLine(" skip: unknown dataset $($scopeDir.Name)/$($typeDir.Name)") - continue - } - foreach ($periodDir in @(Sort-ByNameOrdinal @(Get-ChildItem -LiteralPath $typeDir.FullName -Directory))) { - if ($PeriodFilter -and $periodDir.Name -ne $PeriodFilter) { continue } - $runs = @() - foreach ($runDir in @(Sort-ByNameOrdinal @(Get-ChildItem -LiteralPath $periodDir.FullName -Directory))) { - $manifestObject = Read-RunManifest -RunDirPath $runDir.FullName - if ($null -eq $manifestObject) { continue } - $runs += [pscustomobject]@{ - RunDir = $runDir - Submitted = Get-SubmittedTime -ManifestObject $manifestObject - Manifest = $manifestObject - SubmittedSort = if (Get-SubmittedTime -ManifestObject $manifestObject) { Get-SubmittedTime -ManifestObject $manifestObject } else { '' } - Mtime = $runDir.LastWriteTimeUtc.Ticks - RunName = $runDir.Name - } - } - if ($runs.Count -eq 0) { continue } - $sortedRuns = @($runs | Sort-Object SubmittedSort, Mtime, RunName) - $latest = $sortedRuns[$sortedRuns.Count - 1] - $superseded = @() - if ($sortedRuns.Count -gt 1) { $superseded = @($sortedRuns[0..($sortedRuns.Count - 2)] | ForEach-Object { $_.RunDir.Name }) } - $parts = @(Sort-ByNameOrdinal @(Get-ChildItem -LiteralPath $latest.RunDir.FullName -Filter '*.parquet' -File)) - if ($parts.Count -eq 0) { - [Console]::Error.WriteLine(" WARN: latest run has no parquet parts: $($latest.RunDir.FullName)") - continue - } - $expectedRaw = Get-JsonPropertyValue -ObjectValue $latest.Manifest -Name 'dataRowCount' -DefaultValue 0 - $plans += New-RunPlan -ScopeName $scopeDir.Name -ExportTypeName $typeDir.Name -PeriodName $periodDir.Name ` - -RunUuidValue $latest.RunDir.Name -RunDirPath $latest.RunDir.FullName -SubmittedTimeValue $latest.Submitted ` - -ExpectedRowsValue ([int64]$expectedRaw) -ParquetFilesValue $parts -SupersededRunUuidsValue $superseded - } - } - } - return @($plans) -} - -function Get-FileSha256Lower { - param([Parameter(Mandatory)] [string] $PathValue) - return (Get-FileHash -LiteralPath $PathValue -Algorithm SHA256).Hash.ToLowerInvariant() -} - -function Join-ManifestKey { - param([string] $ScopeName, [string] $ExportTypeName, [string] $PeriodName, [string] $RunUuidValue, [string] $FileNameValue) - $sep = [string][char]31 - return @($ScopeName, $ExportTypeName, $PeriodName, $RunUuidValue, $FileNameValue) -join $sep -} - -function Initialize-IngestManifestTable { - $schema = 'scope:string, export_type:string, period:string, run_uuid:string, file_name:string, file_size:long, rows_ingested:long, checksum_sha256:string, ingested_at:datetime' - $cslText = ".create-merge table $script:IngestManifestTable ($schema)" - $postResult = Invoke-KustoPost -CslText $cslText -TimeoutSec 60 - if ($postResult.Status -lt 200 -or $postResult.Status -ge 300) { - $errParts = Parse-KustoError -BodyText $postResult.Body - throw "failed to ensure $script:IngestManifestTable table: $($errParts[0]): $($errParts[1])" - } -} - -function Fetch-ManifestIndex { - $cslText = "$script:IngestManifestTable | project scope, export_type, period, run_uuid, file_name, file_size, rows_ingested, checksum_sha256, ingested_at" - $postResult = Invoke-KustoPost -CslText $cslText -EndpointUrl $script:KustainerQuery -TimeoutSec 60 - if ($postResult.Status -lt 200 -or $postResult.Status -ge 300) { - $errParts = Parse-KustoError -BodyText $postResult.Body - $msgLower = $errParts[1].ToLowerInvariant() - if ($msgLower.Contains('not found') -or $msgLower.Contains('failed to resolve') -or $msgLower.Contains('could not be resolved') -or $errParts[0].Contains('EntityNotFound') -or $errParts[1].Contains('SEM0100')) { return @{} } - throw "failed to read $script:IngestManifestTable`: $($errParts[0]): $($errParts[1])" - } - $jsonObject = Get-KustoJson -BodyText $postResult.Body - $rows = Get-PrimaryRows $jsonObject - $cols = Get-PrimaryColumnNames $jsonObject - $index = @{} - foreach ($row in $rows) { - $rec = [ordered]@{} - for ($idx = 0; $idx -lt $cols.Count; $idx++) { $rec[$cols[$idx]] = $row[$idx] } - $key = Join-ManifestKey -ScopeName $rec['scope'] -ExportTypeName $rec['export_type'] -PeriodName $rec['period'] -RunUuidValue $rec['run_uuid'] -FileNameValue $rec['file_name'] - $index[$key] = [pscustomobject]$rec - } - return $index -} - -function ConvertTo-CsvQuotedField { - param($Value) - $textValue = if ($null -eq $Value) { '' } else { [string]$Value } - return '"' + ($textValue -replace '"', '""') + '"' -} - -function Insert-ManifestRow { - param([Parameter(Mandatory)] $RecordObject) - $fields = @( - $RecordObject.scope, $RecordObject.export_type, $RecordObject.period, $RecordObject.run_uuid, - $RecordObject.file_name, [string]$RecordObject.file_size, [string]$RecordObject.rows_ingested, - $RecordObject.checksum_sha256, $RecordObject.ingested_at - ) | ForEach-Object { ConvertTo-CsvQuotedField $_ } - $csvLine = $fields -join ',' - $cslText = ".ingest inline into table $script:IngestManifestTable with (format='csv') <|`n$csvLine" - $postResult = Invoke-KustoPost -CslText $cslText -TimeoutSec 60 - if ($postResult.Status -lt 200 -or $postResult.Status -ge 300) { - $errParts = Parse-KustoError -BodyText $postResult.Body - throw "failed to insert $script:IngestManifestTable row: $($errParts[0]): $($errParts[1])" - } -} - -function Get-UpdatePolicy { - param([Parameter(Mandatory)] [string] $FinalTableName) - $postResult = Invoke-KustoPost -CslText ".show table $FinalTableName policy update" -TimeoutSec 30 - if ($postResult.Status -lt 200 -or $postResult.Status -ge 300) { return $null } - try { - $jsonObject = Get-KustoJson -BodyText $postResult.Body - $rows = Get-PrimaryRows $jsonObject - if ($rows.Count -eq 0) { return $null } - $bodyText = $rows[0][2] - if (-not $bodyText) { return $null } - return @($bodyText | ConvertFrom-Json) - } - catch { return $null } -} - -function Set-UpdatePolicy { - param([Parameter(Mandatory)] [string] $FinalTableName, [Parameter(Mandatory)] [object[]] $PolicyObject) - # Kusto expects a JSON array of policy objects. Serialize each element and - # join, so a single-element collection isn't emitted as a bare JSON object. - $elements = @($PolicyObject | ForEach-Object { $_ | ConvertTo-Json -Depth 20 -Compress }) - $bodyText = '[' + ($elements -join ',') + ']' - $cslText = '.alter table ' + $FinalTableName + ' policy update ```' + $bodyText + '```' - $postResult = Invoke-KustoPost -CslText $cslText -TimeoutSec 60 - if ($postResult.Status -lt 200 -or $postResult.Status -ge 300) { - $errParts = Parse-KustoError -BodyText $postResult.Body - throw "failed to set update policy on ${FinalTableName}: $($errParts[0]): $($errParts[1])" - } -} - -function Disable-UpdatePolicies { - param([bool] $ForceRecapture = $false) - $saved = @{} - foreach ($rawTableName in @('Costs_raw', 'Prices_raw')) { - $finalTableName = $script:RawToFinalPolicy[$rawTableName].FinalTable - $policy = @(Get-UpdatePolicy -FinalTableName $finalTableName) - if ($policy.Count -eq 0) { - [Console]::Error.WriteLine(" NOTE: no update policy on $finalTableName; skipping disable") - continue - } - $alreadyDisabled = $false - foreach ($polItem in $policy) { - $enabledValue = Get-JsonPropertyValue -ObjectValue $polItem -Name 'IsEnabled' -DefaultValue $true - if (-not [bool]$enabledValue) { $alreadyDisabled = $true; break } - } - if ($alreadyDisabled -and -not $ForceRecapture) { - throw "$finalTableName update policy is ALREADY disabled (IsEnabled=false). A previous ingest run likely crashed between disable/restore. Refusing to capture the disabled state as 'original' — that would silently bake the broken state into future runs.`n`nResolution: manually re-enable the policy via`n make kql QUERY=`".alter table $finalTableName policy update ''`"`nThen re-run this script. Or override with --force-policy-recapture if you know what you're doing." - } - $saved[$finalTableName] = @($policy | ConvertTo-Json -Depth 20 | ConvertFrom-Json) - $disabled = @($policy | ConvertTo-Json -Depth 20 | ConvertFrom-Json) - foreach ($polItem in $disabled) { $polItem.IsEnabled = $false } - Set-UpdatePolicy -FinalTableName $finalTableName -PolicyObject $disabled - $firstPolicy = $policy[0] - $wasEnabled = Get-JsonPropertyValue -ObjectValue $firstPolicy -Name 'IsEnabled' - $wasTransactional = Get-JsonPropertyValue -ObjectValue $firstPolicy -Name 'IsTransactional' - Write-InfoLine " PRE-INGEST: disabled update policy on $finalTableName (was: IsEnabled=$wasEnabled, IsTransactional=$wasTransactional)" - } - return $saved -} - -function Restore-UpdatePolicies { - param([System.Collections.IDictionary] $SavedPolicies) - foreach ($finalTableName in $SavedPolicies.Keys) { - try { - Set-UpdatePolicy -FinalTableName $finalTableName -PolicyObject @($SavedPolicies[$finalTableName]) - Write-InfoLine " POST-INGEST: restored update policy on $finalTableName" - } - catch { [Console]::Error.WriteLine(" ERROR: could not restore update policy on $finalTableName`: $($_.Exception.Message)") } - } -} - -function Get-RowsFromSetOrAppendResponse { - param([string] $BodyText) - $rowsAdded = 0L - try { - $jsonObject = Get-KustoJson -BodyText $BodyText - $cols = Get-PrimaryColumnNames $jsonObject - $rcIdx = [Array]::IndexOf([object[]]$cols, 'RowCount') - if ($rcIdx -ge 0) { - foreach ($row in (Get-PrimaryRows $jsonObject)) { $rowsAdded += [int64]($row[$rcIdx] ?? 0) } - } - } - catch { } - return $rowsAdded -} - -function Invoke-BackfillFinalTable { - param([string] $FinalTableName, [string] $TransformFunctionName) - $cslText = ".set-or-append $FinalTableName <| $TransformFunctionName()" - Write-InfoLine " BACKFILL: running $cslText ..." - $startTime = [Diagnostics.Stopwatch]::StartNew() - $postResult = Invoke-KustoPost -CslText $cslText -TimeoutSec $script:IngestTimeoutSec - $startTime.Stop() - if ($postResult.Status -lt 200 -or $postResult.Status -ge 300) { - $errParts = Parse-KustoError -BodyText $postResult.Body - [Console]::Error.WriteLine(" BACKFILL FAIL: $($errParts[0]): $($errParts[1].Substring(0, [Math]::Min(300, $errParts[1].Length)))") - return [pscustomobject]@{ Ok = $false; Rows = 0L } - } - $rowsAdded = Get-RowsFromSetOrAppendResponse -BodyText $postResult.Body - Write-InfoLine (" BACKFILL OK: {0:N0} rows in {1:N1}s" -f $rowsAdded, $startTime.Elapsed.TotalSeconds) - return [pscustomobject]@{ Ok = $true; Rows = $rowsAdded } -} - -function Get-TableExtents { - param([Parameter(Mandatory)] [string] $TableName) - $postResult = Invoke-KustoPost -CslText ".show table $TableName extents | project ExtentId" -TimeoutSec 120 - if ($postResult.Status -lt 200 -or $postResult.Status -ge 300) { return @() } - try { - $jsonObject = Get-KustoJson -BodyText $postResult.Body - return @((Get-PrimaryRows $jsonObject) | ForEach-Object { [string]$_[0] }) - } - catch { return @() } -} - -function Wait-ForHealth { - param([int] $TimeoutSec = 60) - $deadline = [DateTimeOffset]::UtcNow.AddSeconds($TimeoutSec) - while ([DateTimeOffset]::UtcNow -lt $deadline) { - $postResult = Invoke-KustoPost -CslText '.show version' -TimeoutSec 5 - if ($postResult.Status -ge 200 -and $postResult.Status -lt 300) { return $true } - Start-Sleep -Seconds 2 - } - return $false -} - -function Invoke-ChunkedBackfillFinalTable { - param( - [Parameter(Mandatory)] [string] $RawTableName, - [Parameter(Mandatory)] [string] $FinalTableName, - [Parameter(Mandatory)] [string] $TransformFunctionName, - [int] $ExtentsPerBatch = 2 - ) - $extents = @(Get-TableExtents -TableName $RawTableName) - if ($extents.Count -eq 0) { - Write-InfoLine " CHUNKED BACKFILL: no extents in $RawTableName, nothing to do" - return [pscustomobject]@{ Ok = $true; Rows = 0L } - } - $totalRows = 0L - $batchCount = [int][Math]::Ceiling($extents.Count / [double]$ExtentsPerBatch) - Write-InfoLine " CHUNKED BACKFILL: $($extents.Count) extents → $batchCount batches of up to $ExtentsPerBatch extents each" - for ($batchIndex = 0; $batchIndex -lt $batchCount; $batchIndex++) { - $startIdx = $batchIndex * $ExtentsPerBatch - $endIdx = [Math]::Min($startIdx + $ExtentsPerBatch - 1, $extents.Count - 1) - $chunk = @($extents[$startIdx..$endIdx]) - $extentGuids = ($chunk | ForEach-Object { "guid($_)" }) -join ', ' - $cslText = ".set-or-append $FinalTableName <|`nlet $RawTableName = __table(`"$RawTableName`", 'All', 'AllButRowStore')`n | where extent_id() in ($extentGuids);`n$TransformFunctionName()" - if ($batchIndex -gt 0) { Start-Sleep -Seconds 5 } - $maxAttempts = 4 - $attempt = 0 - $rowsAdded = 0L - $postResult = $null - $preExtentSet = @{} - foreach ($extentId in @(Get-TableExtents -TableName $FinalTableName)) { $preExtentSet[$extentId] = $true } - while ($attempt -lt $maxAttempts) { - $attempt++ - $stopwatch = [Diagnostics.Stopwatch]::StartNew() - $postResult = Invoke-KustoPost -CslText $cslText -TimeoutSec $script:IngestTimeoutSec - $stopwatch.Stop() - if ($postResult.Status -ge 200 -and $postResult.Status -lt 300) { break } - $errParts = Parse-KustoError -BodyText $postResult.Body - $errType = [string]$errParts[0] - $errMsg = [string]$errParts[1] - $isTransient = ($errMsg.Contains('Connection') -or $errMsg.Contains('RemoteDisconnected') -or $errType.Contains('Internal service') -or $errType.Contains('LowMemoryCondition') -or $errMsg.Contains('LowMemoryCondition') -or $errMsg.ToLowerInvariant().Contains('memory') -or $errType -eq 'Unknown' -or $postResult.Status -eq 599) - if ($isTransient -and $attempt -lt $maxAttempts) { - $waitSeconds = 15 * $attempt - [Console]::Error.WriteLine(" CHUNK $($batchIndex + 1)/$batchCount attempt $attempt/$maxAttempts`: transient error ($($errType.Substring(0, [Math]::Min(40, $errType.Length))): $($errMsg.Substring(0, [Math]::Min(80, $errMsg.Length)))), waiting ${waitSeconds}s for engine recovery...") - Start-Sleep -Seconds $waitSeconds - [void](Wait-ForHealth -TimeoutSec 120) - $postExtentSet = @{} - foreach ($extentId in @(Get-TableExtents -TableName $FinalTableName)) { $postExtentSet[$extentId] = $true } - $partial = @($postExtentSet.Keys | Where-Object { -not $preExtentSet.ContainsKey($_) } | Sort-Object) - if ($partial.Count -gt 0) { - $preview = ($partial | Select-Object -First 3) -join ', ' - if ($partial.Count -gt 3) { $preview += '...' } - [Console]::Error.WriteLine(" CHUNK $($batchIndex + 1)/$batchCount`: dropping $($partial.Count) partial extent(s) from failed attempt before retry: $preview") - $partialGuids = ($partial | ForEach-Object { "guid($_)" }) -join ', ' - $dropCsl = ".drop extents <| .show table $FinalTableName extents where ExtentId in ($partialGuids)" - $dropResult = Invoke-KustoPost -CslText $dropCsl -TimeoutSec 60 - if ($dropResult.Status -lt 200 -or $dropResult.Status -ge 300) { - $dropErr = Parse-KustoError -BodyText $dropResult.Body - [Console]::Error.WriteLine(" CHUNK $($batchIndex + 1)/$batchCount ABORT: could not drop partial extents ($($dropErr[0]): $($dropErr[1].Substring(0, [Math]::Min(120, $dropErr[1].Length))))") - return [pscustomobject]@{ Ok = $false; Rows = $totalRows } - } - } - continue - } - [Console]::Error.WriteLine(" CHUNK $($batchIndex + 1)/$batchCount FAIL after $attempt attempt(s): $errType`: $($errMsg.Substring(0, [Math]::Min(200, $errMsg.Length)))") - return [pscustomobject]@{ Ok = $false; Rows = $totalRows } - } - if ($null -eq $postResult -or $postResult.Status -lt 200 -or $postResult.Status -ge 300) { return [pscustomobject]@{ Ok = $false; Rows = $totalRows } } - $rowsAdded = Get-RowsFromSetOrAppendResponse -BodyText $postResult.Body - $totalRows += $rowsAdded - Write-InfoLine (" CHUNK {0}/{1}: +{2:N0} rows in {3:N1}s" -f ($batchIndex + 1), $batchCount, $rowsAdded, $stopwatch.Elapsed.TotalSeconds) - } - return [pscustomobject]@{ Ok = $true; Rows = $totalRows } -} - -function Invoke-BackfillFinalsPerPeriod { - param([object[]] $Plans) - $appended = @{} - foreach ($rawTableName in @('Costs_raw', 'Prices_raw')) { - $policyInfo = $script:RawToFinalPolicy[$rawTableName] - $rawCount = Get-QueryScalar -CslText "$rawTableName | count" - if (-not $rawCount) { - Write-InfoLine " $($policyInfo.FinalTable): $rawTableName is empty; skipping" - $appended[$policyInfo.FinalTable] = 0 - continue - } - $extentCount = @(Get-TableExtents -TableName $rawTableName).Count - # Proactive chunking: skip single-pass entirely when row count exceeds the - # measured-headroom threshold. OOM is row-driven; extent count is informational only. - if ([int64]$rawCount -gt $script:BackfillChunkRowThreshold) { - Write-InfoLine (" $($policyInfo.FinalTable): {0:N0} rows > threshold {1:N0} ({2} extents) — proactively using per-extent chunked backfill (skipping single-pass)" -f [int64]$rawCount, $script:BackfillChunkRowThreshold, $extentCount) - $chunked = Invoke-ChunkedBackfillFinalTable -RawTableName $rawTableName -FinalTableName $policyInfo.FinalTable -TransformFunctionName $policyInfo.TransformFn -ExtentsPerBatch 1 - $appended[$policyInfo.FinalTable] = if ($chunked.Ok) { $chunked.Rows } else { -1 } - if (-not $chunked.Ok) { [Console]::Error.WriteLine(" ERROR: $($policyInfo.FinalTable) still incomplete after chunked backfill; manual intervention required.") } - continue - } - # Below threshold: try single-pass (fast), fall back to chunked as a safety net. - # The reactive fallback also covers any future table added to RawToFinalPolicy whose - # single-pass headroom has not been measured yet. - Write-InfoLine (" $($policyInfo.FinalTable): {0:N0} rows ≤ threshold {1:N0} ({2} extents) — using single-pass backfill" -f [int64]$rawCount, $script:BackfillChunkRowThreshold, $extentCount) - $result = Invoke-BackfillFinalTable -FinalTableName $policyInfo.FinalTable -TransformFunctionName $policyInfo.TransformFn - if ($result.Ok) { $appended[$policyInfo.FinalTable] = $result.Rows; continue } - [Console]::Error.WriteLine(" $($policyInfo.FinalTable): single-pass backfill failed, falling back to per-extent chunked backfill...") - [void](Wait-ForHealth -TimeoutSec 60) - $chunked = Invoke-ChunkedBackfillFinalTable -RawTableName $rawTableName -FinalTableName $policyInfo.FinalTable -TransformFunctionName $policyInfo.TransformFn -ExtentsPerBatch 1 - $appended[$policyInfo.FinalTable] = if ($chunked.Ok) { $chunked.Rows } else { -1 } - if (-not $chunked.Ok) { [Console]::Error.WriteLine(" ERROR: $($policyInfo.FinalTable) still incomplete; manual intervention required (raise Kustainer mem_limit or shrink extents).") } - } - return $appended -} - -function Drop-SupersededExtents { - param( - [string] $TableName, - [string] $ScopeName, - [string] $ExportTypeName, - [string] $PeriodName, - [string] $KeepRunUuid, - [hashtable] $ManifestIndex - ) - $supersededRuns = @{} - foreach ($entry in $ManifestIndex.GetEnumerator()) { - $rec = $entry.Value - if ($rec.scope -eq $ScopeName -and $rec.export_type -eq $ExportTypeName -and $rec.period -eq $PeriodName -and $rec.run_uuid -ne $KeepRunUuid) { - if (-not $supersededRuns.ContainsKey($rec.run_uuid)) { $supersededRuns[$rec.run_uuid] = @() } - $supersededRuns[$rec.run_uuid] = @($supersededRuns[$rec.run_uuid]) + $rec - } - } - if ($supersededRuns.Count -eq 0) { return 0L } - $totalRows = 0L - foreach ($runUuidValue in @($supersededRuns.Keys)) { - $records = @($supersededRuns[$runUuidValue]) - foreach ($rec in $records) { $totalRows += [int64]($rec.rows_ingested ?? 0) } - $dropCsl = ".drop extents <| .show table $TableName extents where tags has 'run:$runUuidValue'" - $dropResult = Invoke-KustoPost -CslText $dropCsl -TimeoutSec 120 - if ($dropResult.Status -lt 200 -or $dropResult.Status -ge 300) { - $errParts = Parse-KustoError -BodyText $dropResult.Body - [Console]::Error.WriteLine(" WARN: drop superseded extents for $TableName/$runUuidValue failed: $($errParts[0]): $($errParts[1])") - continue - } - $deleteCsl = ".delete table $script:IngestManifestTable records <| $script:IngestManifestTable | where scope == '$ScopeName' and export_type == '$ExportTypeName' and period == '$PeriodName' and run_uuid == '$runUuidValue'" - $deleteResult = Invoke-KustoPost -CslText $deleteCsl -TimeoutSec 120 - if ($deleteResult.Status -lt 200 -or $deleteResult.Status -ge 300) { - $errParts = Parse-KustoError -BodyText $deleteResult.Body - [Console]::Error.WriteLine(" WARN: delete superseded manifest rows for $runUuidValue failed: $($errParts[0]): $($errParts[1])") - } - $keysToDrop = @($ManifestIndex.Keys | Where-Object { - $rec = $ManifestIndex[$_] - $rec.scope -eq $ScopeName -and $rec.export_type -eq $ExportTypeName -and $rec.period -eq $PeriodName -and $rec.run_uuid -eq $runUuidValue - }) - foreach ($keyValue in $keysToDrop) { $ManifestIndex.Remove($keyValue) } - } - return $totalRows -} - -function Get-ContainerPath { - param([Parameter(Mandatory)] [string] $HostPathValue) - $rootFull = [IO.Path]::GetFullPath($script:ExportDirHost).TrimEnd([IO.Path]::DirectorySeparatorChar, [IO.Path]::AltDirectorySeparatorChar) - $pathFull = [IO.Path]::GetFullPath($HostPathValue) - if (-not $pathFull.StartsWith($rootFull + [IO.Path]::DirectorySeparatorChar, [StringComparison]::Ordinal)) { - throw "parquet file $HostPathValue is outside $script:ExportDirHost" - } - $relPath = $pathFull.Substring($rootFull.Length + 1).Replace([IO.Path]::DirectorySeparatorChar, '/') - return "$script:ExportDirContainer/$relPath" -} - -function Escape-KqlHUri { - param([string] $UriText) - return ($UriText -replace '"', '\"') -} - -function Invoke-DoIngest { - param( - [string] $TableName, - [string] $MappingReference, - [string] $HostPathValue, - [string] $RunUuidValue, - [string] $ScopeName, - [string] $ExportTypeName, - [string] $PeriodName - ) - $uriText = Escape-KqlHUri -UriText (Get-ContainerPath -HostPathValue $HostPathValue) - $tagItems = @("run:$RunUuidValue", "scope:$ScopeName", "type:$ExportTypeName", "period:$PeriodName") - $tagsLiteral = '[' + (($tagItems | ForEach-Object { "'$_'" }) -join ',') + ']' - $cslText = ".ingest into table $TableName (h@`"$uriText`") with (format='parquet', ingestionMappingReference='$MappingReference', tags=`"$tagsLiteral`")" - $stopwatch = [Diagnostics.Stopwatch]::StartNew() - $postResult = Invoke-KustoPost -CslText $cslText -TimeoutSec $script:IngestTimeoutSec - $stopwatch.Stop() - if ($postResult.Status -lt 200 -or $postResult.Status -ge 300) { - $errParts = Parse-KustoError -BodyText $postResult.Body - throw "ingest failed for $([IO.Path]::GetFileName($HostPathValue)): HTTP $($postResult.Status) [$($errParts[0])] $($errParts[1])" - } - try { - $jsonObject = Get-KustoJson -BodyText $postResult.Body - $rows = Get-PrimaryRows $jsonObject - if ($rows.Count -gt 0) { - foreach ($row in $rows) { if ($row[3]) { throw "ingest reported HasErrors=true for $([IO.Path]::GetFileName($HostPathValue)): $($rows | ConvertTo-Json -Compress)" } } - } - } - catch { throw "could not parse ingest response: $($_.Exception.Message); body=$($postResult.Body.Substring(0, [Math]::Min(300, $postResult.Body.Length)))" } - return [pscustomobject]@{ Rows = 0; Seconds = $stopwatch.Elapsed.TotalSeconds } -} - -function New-IngestStats { [pscustomobject]@{ files_ingested = 0; files_skipped = 0; files_failed = 0; bytes_ingested = 0L; rows_ingested = 0L; expected_rows = 0L; seconds = 0.0 } } - -function Get-ManifestRowsForBlob { - param($ManifestObject, [string] $BlobFileName) - $blobs = Get-JsonPropertyValue -ObjectValue $ManifestObject -Name 'blobs' - foreach ($blob in @($blobs)) { - $blobName = [string](Get-JsonPropertyValue -ObjectValue $blob -Name 'blobName' -DefaultValue '') - if ($blobName.EndsWith('/' + $BlobFileName, [StringComparison]::Ordinal) -or $blobName -eq $BlobFileName) { - return [int64](Get-JsonPropertyValue -ObjectValue $blob -Name 'dataRowCount' -DefaultValue 0) - } - } - return $null -} - -function Invoke-RunIngest { - param([object[]] $Plans, [bool] $DryRun = $false, [bool] $ForcePolicyRecapture = $false) - if (-not $DryRun) { Initialize-IngestManifestTable } - $manifestIndex = Fetch-ManifestIndex - if ($manifestIndex.Count -gt 0) { [Console]::Error.WriteLine("INFO: Ingest_Manifest has $($manifestIndex.Count) existing rows") } - if ($manifestIndex.Count -eq 0) { - $rawTotal = 0L - foreach ($rawTableName in @('Costs_raw', 'Prices_raw')) { - try { - $postResult = Invoke-KustoPost -CslText "$rawTableName | count" -EndpointUrl $script:KustainerQuery -TimeoutSec 30 - if ($postResult.Status -ge 200 -and $postResult.Status -lt 300) { - $jsonObject = Get-KustoJson -BodyText $postResult.Body - $rawTotal += [int64](Get-PrimaryRows $jsonObject)[0][0] - } - } catch { } - } - if ($rawTotal -gt 0) { - throw "Raw tables have $($rawTotal.ToString('N0')) rows but Ingest_Manifest is empty. The container was likely recreated and lost the manifest while leaving raw data orphaned.`n`nResolution: either drop all raw extents to start fresh`n make kql QUERY=`".clear table Costs_raw data`"`n make kql QUERY=`".clear table Prices_raw data`"`n…then re-run, OR re-populate Ingest_Manifest from a backup. Refusing to proceed — silently treating every file as new would double-ingest the orphaned data." - } - } - - $fileUnits = @() - foreach ($plan in $Plans) { - $manifestObject = Read-RunManifest -RunDirPath $plan.RunDir - foreach ($part in @($plan.ParquetFiles)) { - $perBlob = Get-ManifestRowsForBlob -ManifestObject $manifestObject -BlobFileName $part.Name - if ($null -eq $perBlob) { $perBlob = 0L } - $fileUnits += [pscustomobject]@{ Plan = $plan; HostPath = $part; ExpectedRows = [int64]$perBlob } - } - } - $totalFiles = $fileUnits.Count - if ($totalFiles -eq 0) { Write-InfoLine 'no parquet files matched filters; nothing to do'; return @{} } - Write-InfoLine "PLAN: $($Plans.Count) (scope,type,period) tuples → $totalFiles parquet parts" - foreach ($plan in $Plans) { - $sup = if ($plan.SupersededRunUuids.Count -gt 0) { " (supersedes $($plan.SupersededRunUuids.Count) prior run-uuids)" } else { '' } - Write-InfoLine (" - {0}/{1}/{2} → run {3}… {4} part(s) · {5:N0} rows expected{6}" -f $plan.Scope, $plan.ExportType, $plan.Period, $plan.RunUuid.Substring(0, [Math]::Min(8, $plan.RunUuid.Length)), $plan.ParquetFiles.Count, $plan.ExpectedRows, $sup) - } - - if (-not $DryRun) { - foreach ($plan in $Plans) { - $tableName = $script:DatasetTableMap[$plan.ExportType].Table - $droppedRows = Drop-SupersededExtents -TableName $tableName -ScopeName $plan.Scope -ExportTypeName $plan.ExportType -PeriodName $plan.Period -KeepRunUuid $plan.RunUuid -ManifestIndex $manifestIndex - if ($droppedRows) { Write-InfoLine (" OVERWRITE: dropped {0:N0} rows from {1} for superseded run(s) of {2}/{3}/{4}" -f $droppedRows, $tableName, $plan.Scope, $plan.ExportType, $plan.Period) } - } - } - - $statsByDataset = @{} - $overall = [Diagnostics.Stopwatch]::StartNew() - $savedPolicies = @{} - if (-not $DryRun) { $savedPolicies = Disable-UpdatePolicies -ForceRecapture $ForcePolicyRecapture } - $paceFloorSec = if ($env:INGEST_PACE_FLOOR_S) { [double]$env:INGEST_PACE_FLOOR_S } else { 1.5 } - $pacePerMbSec = if ($env:INGEST_PACE_PER_MB_S) { [double]$env:INGEST_PACE_PER_MB_S } else { 0.05 } - try { - for ($unitIndex = 0; $unitIndex -lt $fileUnits.Count; $unitIndex++) { - $unit = $fileUnits[$unitIndex] - $plan = $unit.Plan - $mapInfo = $script:DatasetTableMap[$plan.ExportType] - $bucketKey = "$($plan.Scope)$([char]31)$($plan.ExportType)" - if (-not $statsByDataset.ContainsKey($bucketKey)) { $statsByDataset[$bucketKey] = New-IngestStats } - $bucket = $statsByDataset[$bucketKey] - $bucket.expected_rows += [int64]$unit.ExpectedRows - $sizeBytes = [int64]$unit.HostPath.Length - $bucket.bytes_ingested += $sizeBytes - $sha = Get-FileSha256Lower -PathValue $unit.HostPath.FullName - $manifestKey = Join-ManifestKey -ScopeName $plan.Scope -ExportTypeName $plan.ExportType -PeriodName $plan.Period -RunUuidValue $plan.RunUuid -FileNameValue $unit.HostPath.Name - $prior = if ($manifestIndex.ContainsKey($manifestKey)) { $manifestIndex[$manifestKey] } else { $null } - $displayIndex = $unitIndex + 1 - if ($prior -and $prior.checksum_sha256 -eq $sha) { - $bucket.files_skipped++ - Write-InfoLine ("[{0,3}/{1}] SKIP {2}/{3}/{4}/{5} ({6} KB, {7:N0} rows) already in Ingest_Manifest with matching checksum" -f $displayIndex, $totalFiles, $plan.Scope, $plan.ExportType, $plan.Period, $unit.HostPath.Name, [Math]::Floor($sizeBytes / 1024), [int64]$unit.ExpectedRows) - $bucket.rows_ingested += [int64]($prior.rows_ingested ?? 0) - continue - } - elseif ($prior) { - # Same composite key (scope/type/period/run-uuid/filename) exists in - # Ingest_Manifest but with a DIFFERENT checksum. Proceeding would silently - # append a second copy of the data and corrupt cost totals (parity check 1's - # 5% tolerance would mask the duplication). - throw ("File '$($unit.HostPath.Name)' under run-uuid '$($plan.RunUuid)' for " + - "($($plan.Scope)/$($plan.ExportType)/$($plan.Period)) changed since last " + - "ingest (checksum mismatch: manifest=$($prior.checksum_sha256), file=$sha). " + - "Re-ingesting under the same run-uuid duplicates rows. " + - "Stage corrected data under a NEW run-uuid for the same (scope,type,period) to replace it (supersede).") - } - $progressPrefix = ("[{0,3}/{1}] ingesting {2}/{3}/{4}/{5} ({6} KB, {7:N0} rows expected)" -f $displayIndex, $totalFiles, $plan.Scope, $plan.ExportType, $plan.Period, $unit.HostPath.Name, [Math]::Floor($sizeBytes / 1024), [int64]$unit.ExpectedRows) - Write-Host -NoNewline ($progressPrefix + ' ... ') - if ($DryRun) { Write-InfoLine 'dry-run skip'; continue } - $paceSec = $paceFloorSec + ($sizeBytes / (1024.0 * 1024.0)) * $pacePerMbSec - if ($displayIndex -gt 1 -and $paceSec -gt 0) { Start-Sleep -Seconds $paceSec } - try { - $ingestResult = Invoke-DoIngest -TableName $mapInfo.Table -MappingReference $mapInfo.Mapping -HostPathValue $unit.HostPath.FullName -RunUuidValue $plan.RunUuid -ScopeName $plan.Scope -ExportTypeName $plan.ExportType -PeriodName $plan.Period - } - catch { - $bucket.files_failed++ - Write-InfoLine "FAIL ($($_.Exception.Message))" - if ($_.Exception.Message.Contains('Connection reset') -or $_.Exception.Message.Contains('RemoteDisconnected')) { - Write-InfoLine ' (engine may have crashed; waiting 30s for recovery)' - Start-Sleep -Seconds 30 - } - continue - } - $bucket.files_ingested++ - $bucket.rows_ingested += [int64]$unit.ExpectedRows - $bucket.seconds += [double]$ingestResult.Seconds - Write-InfoLine ("{0:N1}s OK" -f $ingestResult.Seconds) - $recordObject = [pscustomobject]@{ - scope = $plan.Scope; export_type = $plan.ExportType; period = $plan.Period; run_uuid = $plan.RunUuid - file_name = $unit.HostPath.Name; file_size = $sizeBytes; rows_ingested = [int64]$unit.ExpectedRows - checksum_sha256 = $sha; ingested_at = (Get-Date).ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ssZ') - } - try { Insert-ManifestRow -RecordObject $recordObject; $manifestIndex[$manifestKey] = $recordObject } - catch { [Console]::Error.WriteLine(" WARN: could not record Ingest_Manifest row: $($_.Exception.Message)") } - } - } - finally { - if ($savedPolicies.Count -gt 0) { Restore-UpdatePolicies -SavedPolicies $savedPolicies } - } - $overall.Stop() - Write-InfoLine ("`nINGEST WALL-CLOCK: {0:N1}s ({1:N1} min)" -f $overall.Elapsed.TotalSeconds, ($overall.Elapsed.TotalSeconds / 60.0)) - $anyIngested = $false - foreach ($statsItem in $statsByDataset.Values) { if ($statsItem.files_ingested -gt 0) { $anyIngested = $true; break } } - if (-not $DryRun -and $anyIngested) { - Write-InfoLine "`n=== Backfilling final tables via FTK transforms ===" - [void](Invoke-BackfillFinalsPerPeriod -Plans $Plans) - } - return $statsByDataset -} - -function Invoke-VerifyAndSummarize { - param([object[]] $Plans, [hashtable] $StatsByDataset) - Write-InfoLine ('') - Write-InfoLine ('=' * 78) - Write-InfoLine 'INGEST SUMMARY' - Write-InfoLine ('=' * 78) - $anyMismatch = $false - foreach ($bucketKey in @($StatsByDataset.Keys | Sort-Object)) { - $parts = $bucketKey -split [string][char]31, 2 - $scopeName = $parts[0] - $exportTypeName = $parts[1] - $stats = $StatsByDataset[$bucketKey] - $tableName = $script:DatasetTableMap[$exportTypeName].Table - Write-InfoLine (" {0,10} / {1,-16} → {2,-14} files ok={3} skip={4} fail={5} manifest-rows={6,11:N0} size={7,7:N1} MB ingest-time={8:N1}s" -f $scopeName, $exportTypeName, $tableName, $stats.files_ingested, $stats.files_skipped, $stats.files_failed, $stats.expected_rows, ($stats.bytes_ingested / 1024.0 / 1024.0), $stats.seconds) - if ($stats.files_failed) { $anyMismatch = $true } - } - $declared = @{} - $bucketFilters = @{} - foreach ($bucketKey in $StatsByDataset.Keys) { - $parts = $bucketKey -split [string][char]31, 2 - $scopeName = $parts[0] - $exportTypeName = $parts[1] - $tableName = $script:DatasetTableMap[$exportTypeName].Table - if (-not $declared.ContainsKey($tableName)) { $declared[$tableName] = 0L; $bucketFilters[$tableName] = @() } - $declared[$tableName] += [int64]$StatsByDataset[$bucketKey].expected_rows - $bucketFilters[$tableName] = @($bucketFilters[$tableName]) + [pscustomobject]@{ Scope = $scopeName; ExportType = $exportTypeName } - } - Write-InfoLine '' - foreach ($tableName in @($declared.Keys | Sort-Object)) { - $bucketPairs = (@($bucketFilters[$tableName]) | ForEach-Object { "(scope=='$($_.Scope)' and export_type=='$($_.ExportType)')" }) -join ', ' - $actualCsl = "$script:IngestManifestTable | where $bucketPairs | summarize sum(rows_ingested)" - $got = Get-QueryScalar -CslText $actualCsl - if ($null -eq $got) { Write-InfoLine " $tableName`: count query failed (cannot verify)"; $anyMismatch = $true; continue } - $want = [int64]$declared[$tableName] - $diff = [Math]::Abs([int64]$got - $want) - $pct = if ($want -gt 0) { $diff / [double]$want } else { 0.0 } - $status = if ($pct -le $script:RowCountTolerance) { 'OK' } else { 'MISMATCH' } - if ($status -eq 'MISMATCH') { $anyMismatch = $true } - Write-InfoLine (" {0}: ingested={1,11:N0} manifest-declared={2,11:N0} diff={3,7:N0} ({4:N4}%) {5}" -f $tableName, [int64]$got, $want, $diff, ($pct * 100.0), $status) - $full = Get-QueryScalar -CslText "$tableName | count" - if ($null -ne $full) { Write-InfoLine (" (table total across all runs: {0:N0})" -f [int64]$full) } - } - Write-InfoLine '' - $summaryCsl = "$script:IngestManifestTable | summarize files=count(), rows=sum(rows_ingested) by scope, export_type | order by scope asc, export_type asc" - $postResult = Invoke-KustoPost -CslText $summaryCsl -EndpointUrl $script:KustainerQuery -TimeoutSec 60 - if ($postResult.Status -ge 200 -and $postResult.Status -lt 300) { - $jsonObject = Get-KustoJson -BodyText $postResult.Body - $cols = Get-PrimaryColumnNames $jsonObject - $rows = Get-PrimaryRows $jsonObject - Write-InfoLine 'Ingest_Manifest summary:' - Write-InfoLine (' ' + (($cols | ForEach-Object { '{0,14}' -f $_ }) -join ' | ')) - foreach ($row in $rows) { Write-InfoLine (' ' + (($row | ForEach-Object { '{0,14}' -f ([string]$_) }) -join ' | ')) } - } - return (-not $anyMismatch) -} - -function Read-IngestOptions { - param([string[]] $Tokens) - $Tokens = @($Tokens | Where-Object { $null -ne $_ -and $_ -ne '' }) - $opts = @{ Scope = $null; Period = $null; DryRun = $false; ForcePolicyRecapture = $false } - $idx = 0 - while ($idx -lt $Tokens.Count) { - switch -Regex ($Tokens[$idx]) { - '^--scope$|^-Scope$' { $idx++; if ($idx -ge $Tokens.Count) { throw 'argument --scope: expected one argument' }; $opts.Scope = $Tokens[$idx] } - '^--period$|^-Period$' { $idx++; if ($idx -ge $Tokens.Count) { throw 'argument --period: expected one argument' }; $opts.Period = $Tokens[$idx] } - '^--dry-run$|^-DryRun$' { $opts.DryRun = $true } - '^--force-policy-recapture$|^-ForcePolicyRecapture$' { $opts.ForcePolicyRecapture = $true } - '^--help$|^-h$|^-\?$' { $opts.Help = $true } - default { throw "unrecognized arguments: $($Tokens[$idx])" } - } - $idx++ - } - return $opts -} - -function Show-IngestUsage { - @' -usage: ingest.ps1 [--scope SCOPE] [--period PERIOD] [--dry-run] [--force-policy-recapture] - -Bulk-ingest FOCUS Cost + Price Sheet parquet exports under ./export/ into Kustainer raw tables. -'@ -} - -function Invoke-IngestMain { - param([string[]] $Arguments) - try { $opts = Read-IngestOptions -Tokens @($Arguments) } - catch { [Console]::Error.WriteLine("error: $($_.Exception.Message)"); return 2 } - if ($opts.ContainsKey('Help') -and $opts.Help) { Show-IngestUsage; return 0 } - try { - $plans = @(Build-Plan -ExportRootPath $script:ExportDirHost -ScopeFilter $opts.Scope -PeriodFilter $opts.Period) - if ($plans.Count -eq 0) { Write-InfoLine 'no (scope,type,period) tuples matched filters; nothing to do'; return 0 } - $stats = Invoke-RunIngest -Plans $plans -DryRun ([bool]$opts.DryRun) -ForcePolicyRecapture ([bool]$opts.ForcePolicyRecapture) - if ($opts.DryRun) { Write-InfoLine "`n(dry-run; no actual ingest occurred)"; return 0 } - $ok = Invoke-VerifyAndSummarize -Plans $plans -StatsByDataset $stats - if ($ok) { return 0 } - return 2 - } - catch { [Console]::Error.WriteLine("error: $($_.Exception.Message)"); return 1 } -} - -if ($MyInvocation.InvocationName -ne '.') { - exit (Invoke-IngestMain -Arguments @($CliArgs)) -} diff --git a/src/templates/finops-hub-local/scripts/load-ftk-kql.ps1 b/src/templates/finops-hub-local/scripts/load-ftk-kql.ps1 deleted file mode 100644 index 1775adb04..000000000 --- a/src/templates/finops-hub-local/scripts/load-ftk-kql.ps1 +++ /dev/null @@ -1,508 +0,0 @@ -#!/usr/bin/env pwsh -#Requires -Version 7.0 -<# -.SYNOPSIS - Load the FinOps Toolkit analytics KQL into local Kustainer (two-DB topology). - -.DESCRIPTION - Reads the combineKql bundles from src/templates/finops-hub/.build.config, - concatenates the listed .kql files in dependency order, and submits each - bundle to its target database. - - Two persisted databases are created idempotently: - Ingestion -- raw tables, transforms, final tables, open-data lookups - Hub -- view functions referencing database('Ingestion').* - - The $$rawRetentionInDays$$ macro in the Ingestion bundle is replaced with - RawRetentionDays (default 3650). The Hub bundle is submitted verbatim -- - database('Ingestion'). cross-DB references are preserved and resolve once - both databases exist. - - Open-data CSVs (PricingUnits/Regions/ResourceTypes/Services) are ingested - into the Ingestion database. - - Connection: - KUSTAINER_MGMT full mgmt URL (default http://localhost:/v1/rest/mgmt) - HOST_PORT port (default 8082; read from .env if present) - - FTK source paths (all derived from repo root by default): - FTK_BUILD_CONFIG path to finops-hub .build.config - FTK_OPEN_DATA overrides src/open-data path - FTK_REPO repo root override - -.EXAMPLE - pwsh scripts/load-ftk-kql.ps1 - -.EXAMPLE - pwsh scripts/load-ftk-kql.ps1 --dry-run -#> -[CmdletBinding()] -param( - [switch] $DryRun, - [switch] $SkipOpenData, - [switch] $SkipKql, - [Parameter(ValueFromRemainingArguments = $true)] - [string[]] $RemainingArgs -) - -Set-StrictMode -Version Latest -$ErrorActionPreference = 'Stop' - -if ($RemainingArgs) { - for ($argIndex = 0; $argIndex -lt $RemainingArgs.Count; $argIndex++) { - switch ($RemainingArgs[$argIndex]) { - '--dry-run' { $DryRun = $true } - '--skip-open-data' { $SkipOpenData = $true } - '--skip-kql' { $SkipKql = $true } - default { throw "unrecognized argument: $($RemainingArgs[$argIndex])" } - } - } -} - -# Script lives at /src/templates/finops-hub-local/scripts/ -$script:FtkLocalHome = Split-Path -Parent (Split-Path -Parent $PSCommandPath) -$script:RepoRoot = Split-Path -Parent (Split-Path -Parent (Split-Path -Parent $script:FtkLocalHome)) -$script:RawRetentionDays = '3650' # default; overridden by settings.json retention.raw.days when > 0 -$settingsFile = Join-Path $script:FtkLocalHome 'settings.json' -if (Test-Path $settingsFile) { - try { - $rawDays = (Get-Content -Raw -Path $settingsFile | ConvertFrom-Json).retention.raw.days - if ($null -ne $rawDays -and [int]$rawDays -gt 0) { - $script:RawRetentionDays = [string][int]$rawDays - if ([int]$rawDays -lt 90) { - Write-Warning "retention.raw.days=$rawDays (settings.json): local raw tables will soft-delete data older than $rawDays days. Re-ingest restores it. Consider setting raw.days >= 3650 for local analysis." - } - } - } - catch { <# silently fall back to the 3650 default above #> } -} -$script:TopLevelCommandRegex = [regex]'^\.[a-z]' -$script:IdempotentOkTypes = @('Kusto.Common.Svc.Exceptions.EntityNameAlreadyExistsException') - -# --------------------------------------------------------------------------- # -# Configuration / path resolution -# --------------------------------------------------------------------------- # -function Get-HostPort { - if ($env:HOST_PORT) { return $env:HOST_PORT } - $envFile = Join-Path $script:FtkLocalHome '.env' - if (Test-Path $envFile) { - foreach ($line in Get-Content -Path $envFile) { - $trimmed = $line.Trim() - if ($trimmed -like 'HOST_PORT=*') { return ($trimmed -split '=', 2)[1].Trim() } - } - } - return '8082' -} - -function Get-RepoRoot { - if ($env:FTK_REPO) { return $env:FTK_REPO } - return $script:RepoRoot -} - -function Get-BuildConfigPath { - if ($env:FTK_BUILD_CONFIG) { return $env:FTK_BUILD_CONFIG } - return Join-Path (Get-RepoRoot) 'src/templates/finops-hub/.build.config' -} - -function Get-OpenDataPath { - if ($env:FTK_OPEN_DATA) { return $env:FTK_OPEN_DATA } - return Join-Path (Get-RepoRoot) 'src/open-data' -} - -function Get-MgmtUrl { - if ($env:KUSTAINER_MGMT) { return $env:KUSTAINER_MGMT } - return "http://localhost:$(Get-HostPort)/v1/rest/mgmt" -} - -function Get-QueryUrl { - return ((Get-MgmtUrl) -replace '/mgmt$', '/query') -} - -$script:OpenData = Get-OpenDataPath -$script:MgmtUrl = Get-MgmtUrl - -# --------------------------------------------------------------------------- # -# Bundle builder -- reads .build.config and concatenates .kql files in order -# --------------------------------------------------------------------------- # -function Build-KqlBundle { - param([Parameter(Mandatory)] [string] $BundleName) - $configPath = Get-BuildConfigPath - $config = Get-Content -Raw -Path $configPath | ConvertFrom-Json - $entry = $config.combineKql | Where-Object { $_.name -eq $BundleName } | Select-Object -First 1 - if (-not $entry) { throw "combineKql entry '$BundleName' not found in $(Split-Path -Leaf $configPath)" } - $templateDir = Split-Path -Parent $configPath - $parts = foreach ($relPath in $entry.files) { - Get-Content -Raw -Path (Join-Path $templateDir $relPath) - } - return ($parts -join "`n") -} - -function Invoke-SubstMacros { - param([Parameter(Mandatory)] [string] $Text) - return $Text.Replace('$$rawRetentionInDays$$', $script:RawRetentionDays) -} - -# --------------------------------------------------------------------------- # -# KQL command splitter -# --------------------------------------------------------------------------- # -function Split-KqlCommands { - param([Parameter(Mandatory)] [string] $ScriptText) - $lines = $ScriptText -split "(?<=`n)", 0 - if ($lines.Count -gt 0 -and $lines[-1] -eq '') { $lines = $lines[0..($lines.Count - 2)] } - - $blocks = [System.Collections.Generic.List[object]]::new() - $current = [System.Collections.Generic.List[string]]::new() - foreach ($line in $lines) { - if ($script:TopLevelCommandRegex.IsMatch($line)) { - if ($current.Count -gt 0) { [void]$blocks.Add([string[]]$current.ToArray()) } - $current = [System.Collections.Generic.List[string]]::new() - [void]$current.Add($line) - } - else { [void]$current.Add($line) } - } - if ($current.Count -gt 0) { [void]$blocks.Add([string[]]$current.ToArray()) } - - $cleaned = [System.Collections.Generic.List[string]]::new() - foreach ($blockObject in $blocks) { - $block = [System.Collections.Generic.List[string]]::new() - foreach ($entry in [string[]]$blockObject) { [void]$block.Add($entry) } - $hasCommand = $false - foreach ($entry in $block) { - if ($script:TopLevelCommandRegex.IsMatch($entry)) { $hasCommand = $true; break } - } - if (-not $hasCommand) { continue } - while ($block.Count -gt 0) { - $last = $block[$block.Count - 1] - if ($last.Trim() -eq '' -or $last.TrimStart().StartsWith('//')) { $block.RemoveAt($block.Count - 1) } - else { break } - } - $text = ([string]::Concat([string[]]$block.ToArray())).TrimEnd() - if ($text) { [void]$cleaned.Add($text) } - } - return [string[]]$cleaned.ToArray() -} - -# --------------------------------------------------------------------------- # -# Kustainer REST poster -# --------------------------------------------------------------------------- # -function ConvertTo-BodyString { - param($Value) - if ($null -eq $Value) { return '' } - if ($Value -is [string]) { return $Value } - return ($Value | ConvertTo-Json -Depth 100 -Compress) -} - -function Post-Kql { - param( - [Parameter(Mandatory)] [string] $Csl, - [Parameter(Mandatory)] [string] $DatabaseName, - [int] $TimeoutSec = 60 - ) - $payload = @{ db = $DatabaseName; csl = $Csl } | ConvertTo-Json -Compress - $headers = @{ - 'Content-Type' = 'application/json' - 'x-ms-client-version' = 'Kusto.Python.Client:1.0.0' - 'Accept' = 'application/json' - } - try { - $statusCodeValue = $null - $responseBody = Invoke-RestMethod -Method Post -Uri $script:MgmtUrl -Headers $headers -Body $payload ` - -TimeoutSec $TimeoutSec -SkipHttpErrorCheck -StatusCodeVariable statusCodeValue - return [pscustomobject]@{ Status = [int]$statusCodeValue; Body = (ConvertTo-BodyString $responseBody) } - } - catch { - return [pscustomobject]@{ Status = 599; Body = "URL error: $($_.Exception.GetType().FullName): $($_.Exception.Message)" } - } -} - -function Parse-KustoError { - param([Parameter(Mandatory)] [string] $Body) - try { - $json = $Body | ConvertFrom-Json - $err = $null - if ($json.PSObject.Properties.Name -contains 'error') { $err = $json.error } - if ($null -ne $err) { - $typeValue = if ($err.PSObject.Properties.Name -contains '@type' -and $err.'@type') { $err.'@type' } - elseif ($err.PSObject.Properties.Name -contains 'code' -and $err.code) { $err.code } - else { 'Unknown' } - $messageValue = if ($err.PSObject.Properties.Name -contains '@message' -and $err.'@message') { $err.'@message' } - elseif ($err.PSObject.Properties.Name -contains 'message' -and $err.message) { $err.message } - else { $Body.Substring(0, [Math]::Min(200, $Body.Length)) } - return @([string]$typeValue, [string]$messageValue) - } - return @('Unknown', $Body.Substring(0, [Math]::Min(200, $Body.Length))) - } - catch { return @('RawText', $Body.Substring(0, [Math]::Min(300, $Body.Length))) } -} - -function Test-IdempotentSuccess { - param([Parameter(Mandatory)] [string] $Body) - return ($script:IdempotentOkTypes -contains (Parse-KustoError -Body $Body)[0]) -} - -# --------------------------------------------------------------------------- # -# Step reporting and execution -# --------------------------------------------------------------------------- # -function New-StepReport { - param([Parameter(Mandatory)] [string] $Label) - return [pscustomobject]@{ - Label = $Label - Total = 0 - Ok = 0 - IdempotentOk = 0 - Dropped = [System.Collections.Generic.List[object]]::new() - Failed = [System.Collections.Generic.List[object]]::new() - } -} - -function Get-FirstCommandLine { - param([Parameter(Mandatory)] [string] $CommandText) - foreach ($line in ($CommandText -split "`n")) { - if ($script:TopLevelCommandRegex.IsMatch($line)) { return $line.Trim() } - } - $lines = $CommandText -split "`n" - if ($lines.Count -gt 0) { return $lines[0] } - return '' -} - -function Invoke-BundleLoad { - param( - [Parameter(Mandatory)] [string] $Label, - [Parameter(Mandatory)] [string] $BundleText, - [Parameter(Mandatory)] [string] $DatabaseName, - [bool] $IsDryRun = $false - ) - $report = New-StepReport -Label $Label - $commands = @(Split-KqlCommands -ScriptText $BundleText) - $report.Total = $commands.Count - Write-Host "" - Write-Host "=== $Label [$DatabaseName] ($($report.Total) commands) ===" - for ($i = 0; $i -lt $commands.Count; $i++) { - $commandText = $commands[$i] - $firstLine = Get-FirstCommandLine -CommandText $commandText - $preview = if ($firstLine.Length -gt 110) { $firstLine.Substring(0, 110) } else { $firstLine } - if ($IsDryRun) { - Write-Host (' [{0,3}] dry {1}' -f ($i + 1), $preview) - continue - } - $result = Post-Kql -Csl $commandText -DatabaseName $DatabaseName - if ($result.Status -ge 200 -and $result.Status -lt 300) { - $report.Ok++ - Write-Host (' [{0,3}] OK {1}' -f ($i + 1), $preview) - } - elseif (Test-IdempotentSuccess -Body $result.Body) { - $report.IdempotentOk++ - $parsed = Parse-KustoError -Body $result.Body - Write-Host (' [{0,3}] OK* {1} -- {2}' -f ($i + 1), $preview, $parsed[1]) - } - else { - $parsed = Parse-KustoError -Body $result.Body - $short = "[$($parsed[0])] $($parsed[1])" - if ($short.Length -gt 400) { $short = $short.Substring(0, 400) } - [void]$report.Failed.Add([pscustomobject]@{ Command = $preview; Error = "HTTP $($result.Status): $short" }) - Write-Host (' [{0,3}] FAIL {1}' -f ($i + 1), $preview) - Write-Host " HTTP $($result.Status): $short" - } - } - return $report -} - -# --------------------------------------------------------------------------- # -# Open-data CSV ingestion (into Ingestion database) -# --------------------------------------------------------------------------- # -function ConvertTo-InlineCsvBlock { - param([Parameter(Mandatory)] [object[]] $Rows) - $builder = [System.Text.StringBuilder]::new() - foreach ($row in $Rows) { - $cells = foreach ($cell in $row) { - $value = if ($null -eq $cell) { '' } else { [string]$cell } - '"' + ($value -replace '"', '""') + '"' - } - [void]$builder.AppendLine(($cells -join ',')) - } - return $builder.ToString() -} - -function Read-CsvRows { - param([Parameter(Mandatory)] [string] $Path) - $csvRows = @(Import-Csv -Path $Path) - if ($csvRows.Count -gt 0) { - $headers = [string[]]@($csvRows[0].PSObject.Properties.Name) - } - else { - $headerLine = Get-Content -Path $Path -TotalCount 1 - $headers = if ($headerLine) { [string[]]($headerLine -split ',' | ForEach-Object { $_.Trim().Trim('"') }) } else { [string[]]@() } - } - return [pscustomobject]@{ Headers = $headers; Rows = $csvRows } -} - -function Invoke-OpenDataIngestion { - $db = 'Ingestion' - $report = New-StepReport -Label 'Open-data CSV ingestion (PricingUnits/Regions/ResourceTypes/Services)' - $plans = @( - [pscustomobject]@{ Table = 'PricingUnits'; Path = (Join-Path $script:OpenData 'PricingUnits.csv'); Columns = @( - [pscustomobject]@{ Target = 'x_PricingUnitDescription'; Source = 'UnitOfMeasure'; Type = 'string' }, - [pscustomobject]@{ Target = 'x_PricingBlockSize'; Source = 'PricingBlockSize'; Type = 'real' }, - [pscustomobject]@{ Target = 'PricingUnit'; Source = 'DistinctUnits'; Type = 'string' } - ) }, - [pscustomobject]@{ Table = 'Regions'; Path = (Join-Path $script:OpenData 'Regions.csv'); Columns = @( - [pscustomobject]@{ Target = 'ResourceLocation'; Source = 'OriginalValue'; Type = 'string' }, - [pscustomobject]@{ Target = 'RegionId'; Source = 'RegionId'; Type = 'string' }, - [pscustomobject]@{ Target = 'RegionName'; Source = 'RegionName'; Type = 'string' } - ) }, - [pscustomobject]@{ Table = 'ResourceTypes'; Path = (Join-Path $script:OpenData 'ResourceTypes.csv'); Columns = @( - [pscustomobject]@{ Target = 'x_ResourceType'; Source = 'ResourceType'; Type = 'string' }, - [pscustomobject]@{ Target = 'SingularDisplayName'; Source = 'SingularDisplayName'; Type = 'string' }, - [pscustomobject]@{ Target = 'PluralDisplayName'; Source = 'PluralDisplayName'; Type = 'string' }, - [pscustomobject]@{ Target = 'LowerSingularDisplayName'; Source = 'LowerSingularDisplayName'; Type = 'string' }, - [pscustomobject]@{ Target = 'LowerPluralDisplayName'; Source = 'LowerPluralDisplayName'; Type = 'string' }, - [pscustomobject]@{ Target = 'IsPreview'; Source = 'IsPreview'; Type = 'bool' }, - [pscustomobject]@{ Target = 'Description'; Source = 'Description'; Type = 'string' }, - [pscustomobject]@{ Target = 'IconUri'; Source = 'Icon'; Type = 'string' } - ) }, - [pscustomobject]@{ Table = 'Services'; Path = (Join-Path $script:OpenData 'Services.csv'); Columns = @( - [pscustomobject]@{ Target = 'x_ConsumedService'; Source = 'ConsumedService'; Type = 'string' }, - [pscustomobject]@{ Target = 'x_ResourceType'; Source = 'ResourceType'; Type = 'string' }, - [pscustomobject]@{ Target = 'ServiceName'; Source = 'ServiceName'; Type = 'string' }, - [pscustomobject]@{ Target = 'ServiceCategory'; Source = 'ServiceCategory'; Type = 'string' }, - [pscustomobject]@{ Target = 'ServiceSubcategory'; Source = 'ServiceSubcategory'; Type = 'string' }, - [pscustomobject]@{ Target = 'PublisherName'; Source = 'PublisherName'; Type = 'string' }, - [pscustomobject]@{ Target = 'x_PublisherCategory'; Source = 'PublisherType'; Type = 'string' }, - [pscustomobject]@{ Target = 'x_Environment'; Source = 'Environment'; Type = 'string' }, - [pscustomobject]@{ Target = 'x_ServiceModel'; Source = 'ServiceModel'; Type = 'string' } - ) } - ) - - Write-Host "" - Write-Host "=== $($report.Label) [$db] ===" - foreach ($plan in $plans) { - $report.Total++ - try { - $csvData = Read-CsvRows -Path $plan.Path - foreach ($col in $plan.Columns) { - if ($csvData.Headers -notcontains $col.Source) { throw "'$($col.Source)' not found in CSV headers" } - } - } - catch { - [void]$report.Failed.Add([pscustomobject]@{ Command = $plan.Table; Error = "CSV header mismatch: $($_.Exception.Message)" }) - Write-Host " FAIL $($plan.Table): header mismatch $($_.Exception.Message)" - continue - } - - $projectedRows = foreach ($csvRow in $csvData.Rows) { - $rowValues = foreach ($col in $plan.Columns) { - $prop = $csvRow.PSObject.Properties[$col.Source] - if ($null -eq $prop -or $null -eq $prop.Value) { '' } else { [string]$prop.Value } - } - ,([object[]]$rowValues) - } - $csvBody = ConvertTo-InlineCsvBlock -Rows @($projectedRows) - - $clearResult = Post-Kql -Csl ".clear table $($plan.Table) data" -DatabaseName $db - if (-not ($clearResult.Status -ge 200 -and $clearResult.Status -lt 300)) { - [void]$report.Failed.Add([pscustomobject]@{ Command = $plan.Table; Error = "clear failed: HTTP $($clearResult.Status): $($clearResult.Body.Substring(0, [Math]::Min(200, $clearResult.Body.Length)))" }) - Write-Host " FAIL $($plan.Table): clear failed $($clearResult.Status)" - continue - } - - $ingestCommand = ".ingest inline into table $($plan.Table) with (format='csv') <|`n$($csvBody.TrimEnd("`r", "`n"))" - $ingestResult = Post-Kql -Csl $ingestCommand -DatabaseName $db -TimeoutSec 180 - if (-not ($ingestResult.Status -ge 200 -and $ingestResult.Status -lt 300)) { - $shortBody = $ingestResult.Body.Substring(0, [Math]::Min(200, $ingestResult.Body.Length)) - [void]$report.Failed.Add([pscustomobject]@{ Command = $plan.Table; Error = "ingest failed: HTTP $($ingestResult.Status): $shortBody" }) - Write-Host " FAIL $($plan.Table): ingest failed $($ingestResult.Status): $shortBody" - continue - } - - try { - $countPayload = @{ db = $db; csl = "$($plan.Table) | count" } | ConvertTo-Json -Compress - $queryResponse = Invoke-RestMethod -Method Post -Uri (Get-QueryUrl) -Headers @{ 'Content-Type' = 'application/json' } -Body $countPayload -TimeoutSec 30 - $count = [int]$queryResponse.Tables[0].Rows[0][0] - $expected = @($csvData.Rows).Count - if ($count -ne $expected) { - [void]$report.Failed.Add([pscustomobject]@{ Command = $plan.Table; Error = "row count $count != expected $expected" }) - Write-Host " FAIL $($plan.Table): row count $count != expected $expected" - continue - } - $report.Ok++ - Write-Host " OK $($plan.Table): $count rows" - } - catch { - [void]$report.Failed.Add([pscustomobject]@{ Command = $plan.Table; Error = "count verification failed: $($_.Exception.Message)" }) - Write-Host " FAIL $($plan.Table): count verification $($_.Exception.Message)" - } - } - return $report -} - -# --------------------------------------------------------------------------- # -# Summary / main -# --------------------------------------------------------------------------- # -function Write-Summary { - param([Parameter(Mandatory)] [object[]] $Reports) - Write-Host "" - Write-Host ('=' * 78) - Write-Host 'SUMMARY' - Write-Host ('=' * 78) - $anyFailed = $false - foreach ($rpt in $Reports) { - Write-Host (" $($rpt.Label): total=$($rpt.Total) ok=$($rpt.Ok) idempotent_ok=$($rpt.IdempotentOk) dropped=$($rpt.Dropped.Count) failed=$($rpt.Failed.Count)") - if ($rpt.Failed.Count -gt 0) { - $anyFailed = $true - foreach ($failure in $rpt.Failed) { - Write-Host " FAIL: $($failure.Command)" - Write-Host " $($failure.Error)" - } - } - } - return (-not $anyFailed) -} - -try { - $reports = [System.Collections.Generic.List[object]]::new() - - if (-not $SkipKql) { - # 1. Bootstrap: create Ingestion and Hub databases (idempotent) - foreach ($dbName in @('Ingestion', 'Hub')) { - $csl = ".create database $dbName persist (@'/kustodata/dbs/$dbName/md', @'/kustodata/dbs/$dbName/data')" - if ($DryRun) { - Write-Host " BOOTSTRAP dry .create database $dbName persist ..." - } - else { - $result = Post-Kql -Csl $csl -DatabaseName 'NetDefaultDB' - if ($result.Status -ge 200 -and $result.Status -lt 300) { - Write-Host " BOOTSTRAP Created database '$dbName'." - } - elseif (Test-IdempotentSuccess -Body $result.Body) { - Write-Host " BOOTSTRAP Database '$dbName' already exists (idempotent)." - } - else { - $parsed = Parse-KustoError -Body $result.Body - throw "Failed to create database '$dbName': [$($parsed[0])] $($parsed[1])" - } - } - } - - # 2. Ingestion bundle -- apply $$rawRetentionInDays$$ macro substitution - $ingestionText = Invoke-SubstMacros -Text (Build-KqlBundle -BundleName 'finops-hub-fabric-setup-Ingestion.kql') - [void]$reports.Add((Invoke-BundleLoad -Label 'finops-hub-fabric-setup-Ingestion.kql' ` - -BundleText $ingestionText -DatabaseName 'Ingestion' -IsDryRun:$DryRun.IsPresent)) - - # 3. Hub bundle -- submitted verbatim; database('Ingestion'). refs preserved - $hubText = Build-KqlBundle -BundleName 'finops-hub-fabric-setup-Hub.kql' - [void]$reports.Add((Invoke-BundleLoad -Label 'finops-hub-fabric-setup-Hub.kql' ` - -BundleText $hubText -DatabaseName 'Hub' -IsDryRun:$DryRun.IsPresent)) - } - - if (-not $SkipOpenData -and -not $DryRun) { - [void]$reports.Add((Invoke-OpenDataIngestion)) - } - - $ok = Write-Summary -Reports ([object[]]$reports.ToArray()) - if ($ok) { exit 0 } - exit 1 -} -catch { - Write-Error "error: $($_.Exception.Message)" - exit 2 -} diff --git a/src/templates/finops-hub-local/scripts/run-parity-checks.ps1 b/src/templates/finops-hub-local/scripts/run-parity-checks.ps1 deleted file mode 100644 index 3f6c3b329..000000000 --- a/src/templates/finops-hub-local/scripts/run-parity-checks.ps1 +++ /dev/null @@ -1,515 +0,0 @@ -#!/usr/bin/env pwsh -#Requires -Version 7.0 -<# -.SYNOPSIS - Runs FTK-Local parity checks against the local Kustainer databases. - -.DESCRIPTION - Runs the ftklocal parity-check suite against the two-DB topology - (Ingestion + Hub). Reads tests/parity-checks.kql, executes each numbered - CHECK block against the appropriate database, evaluates the EVAL rule from - the block's comment header, and prints a pass/fail report. - - Database routing (enforced per-check): - Checks 1–8, 11 → Ingestion (raw tables, final tables) - Checks 9, 10 → Hub (Costs_v1_2 and other Hub view functions) - - After the file-based checks, a synthetic NAME-PARITY check (CHECK 12) - verifies that no object in either database has a FtkLocal-prefixed name and - that all expected core tables / functions are present. - - Exit code 0 only if ALL checks pass. Non-zero means at least one parity - violation. - - Connection: - KUSTAINER_QUERY full query URL (default http://localhost:/v1/rest/query) - HOST_PORT port (default 8082; read from .env if present) - FTK_DB default database for Ingestion-targeted checks - (default Ingestion; Hub checks are always routed to Hub) - -.EXAMPLE - pwsh scripts/run-parity-checks.ps1 - -.EXAMPLE - pwsh scripts/run-parity-checks.ps1 --check 3 --check 5 -#> - -Set-StrictMode -Version Latest -$ErrorActionPreference = 'Stop' - -$script:RepoRoot = Split-Path -Parent (Split-Path -Parent $PSCommandPath) -$script:ParityFile = Join-Path $script:RepoRoot 'tests/parity-checks.kql' - -function Get-HostPort { - if ($env:HOST_PORT) { return $env:HOST_PORT } - $envFile = Join-Path $script:RepoRoot '.env' - if (Test-Path $envFile) { - foreach ($line in Get-Content -Path $envFile) { - $trimmed = $line.Trim() - if ($trimmed -like 'HOST_PORT=*') { return ($trimmed -split '=', 2)[1].Trim() } - } - } - return '8082' -} - -function Get-QueryUrl { - param([string] $Override) - if ($Override) { return $Override } - if ($env:KUSTAINER_QUERY) { - if ($env:KUSTAINER_QUERY -match '/v1/rest/(query|mgmt)$') { return $env:KUSTAINER_QUERY } - return "$($env:KUSTAINER_QUERY.TrimEnd('/'))/v1/rest/query" - } - return "http://localhost:$(Get-HostPort)/v1/rest/query" -} - -function Get-Database { - param([string] $Override) - if ($Override) { return $Override } - if ($env:FTK_DB) { return $env:FTK_DB } - return 'Ingestion' -} - -class ParityCheck { - [int] $Number - [string] $Name - [string] $Expect - [string] $Evaluator - [string] $Kql - [string] $EvalKind = 'manual' - [Nullable[double]] $EvalArg - [string] $EvalCol - # Which Kustainer database to run this check against. Defaults to the - # caller-supplied baseline (Ingestion). Hub checks (9, 10) are overridden - # to 'Hub' after parsing so they resolve Costs_v1_2 and other Hub functions. - [string] $Database = '' - - ParityCheck([int] $number, [string] $name, [string] $expect, [string] $evaluator, [string] $kql) { - $this.Number = $number - $this.Name = $name - $this.Expect = $expect - $this.Evaluator = $evaluator - $this.Kql = $kql - } -} - -function Set-Evaluator { - param([Parameter(Mandatory)] [ParityCheck] $CheckItem) - - $evaluatorText = $CheckItem.Evaluator.ToLowerInvariant() - if ($evaluatorText.Contains('result == 0') -or $evaluatorText.Contains('should be 0')) { - $CheckItem.EvalKind = 'zero' - } - elseif ($evaluatorText.Contains('abs(') -and ($evaluatorText.Contains('<=') -or $evaluatorText.Contains('<'))) { - $match = [regex]::Match($evaluatorText, '<=?\s*([0-9.]+)') - if ($match.Success) { - $CheckItem.EvalKind = 'within_pct' - $value = [double]$match.Groups[1].Value - if ($evaluatorText.Contains('pct') -or $value -lt 1.0) { - $CheckItem.EvalArg = $value * 100 - } - else { - $CheckItem.EvalArg = $value - } - } - } - elseif ($evaluatorText.Contains('at least one row') -or $evaluatorText.Contains('non-empty')) { - $CheckItem.EvalKind = 'nonzero' - } - else { - $CheckItem.EvalKind = 'manual' - } -} - -function Get-ParityChecks { - param([Parameter(Mandatory)] [string] $Text) - - $parts = [regex]::Split($Text, '(?m)^// CHECK ') - if ($parts.Count -lt 2) { - throw 'No CHECK blocks found in parity-checks.kql' - } - - $checks = @() - foreach ($block in @($parts | Select-Object -Skip 1)) { - $headerMatch = [regex]::Match($block, '(\d+):\s*(.+)') - if (-not $headerMatch.Success) { continue } - - $number = [int]$headerMatch.Groups[1].Value - $name = $headerMatch.Groups[2].Value.Trim() - - $expectMatch = [regex]::Match($block, '(?m)^// EXPECT:\s*(.+?)$') - $evalMatch = [regex]::Match($block, '(?m)^// EVAL:\s*(.+?)$') - $expect = if ($expectMatch.Success) { $expectMatch.Groups[1].Value.Trim() } else { '' } - $evaluator = if ($evalMatch.Success) { $evalMatch.Groups[1].Value.Trim() } else { '' } - - $kqlLines = @() - $inKql = $false - foreach ($line in ($block -split "`r?`n")) { - if (-not $inKql) { - if ($line.StartsWith('//') -or $line -match '^\d+:' -or -not $line.Trim()) { - continue - } - $inKql = $true - } - $kqlLines += $line - } - $kql = ($kqlLines -join "`n").Trim() - - $checkItem = [ParityCheck]::new($number, $name, $expect, $evaluator, $kql) - Set-Evaluator -CheckItem $checkItem - $checks += $checkItem - } - - return $checks -} - -function Invoke-KustoQuery { - param( - [Parameter(Mandatory)] [string] $Endpoint, - [Parameter(Mandatory)] [string] $Database, - [Parameter(Mandatory)] [string] $Kql - ) - - $url = if ($Endpoint -match '/v1/rest/query$') { $Endpoint } else { "$($Endpoint.TrimEnd('/'))/v1/rest/query" } - $body = @{ db = $Database; csl = $Kql } | ConvertTo-Json -Compress - $headers = @{ - 'Content-Type' = 'application/json' - 'x-ms-client-version' = 'Kusto.Python.Client:1.0.0' - } - - try { - return Invoke-RestMethod -Method Post -Uri $url -Headers $headers -Body $body -TimeoutSec 60 - } - catch { - $raw = $null - try { $raw = $_.ErrorDetails.Message } catch { } - if (-not $raw) { - try { - $stream = $_.Exception.Response.GetResponseStream() - $raw = (New-Object IO.StreamReader($stream)).ReadToEnd() - } catch { } - } - if (-not $raw) { $raw = $_.Exception.Message } - - $status = $null - try { $status = [int]$_.Exception.Response.StatusCode } catch { } - if ($status) { - throw "Kusto HTTP ${status}: $($raw.Substring(0, [Math]::Min(500, $raw.Length)))" - } - throw $raw - } -} - -function Get-PrimaryResult { - param([Parameter(Mandatory)] $Response) - - if (-not ($Response.PSObject.Properties.Name -contains 'Tables') -or -not $Response.Tables) { - $json = $Response | ConvertTo-Json -Depth 20 -Compress - throw "Unexpected response shape: $($json.Substring(0, [Math]::Min(200, $json.Length)))" - } - $table = $Response.Tables[0] - $cols = @($table.Columns | ForEach-Object { $_.ColumnName }) - $rows = @($table.Rows) - return [pscustomobject]@{ Columns = $cols; Rows = $rows } -} - -function Format-PythonValue { - param($Value) - - if ($null -eq $Value) { return 'None' } - if ($Value -is [string]) { return "'" + ($Value -replace "'", "\'") + "'" } - if ($Value -is [bool]) { return $(if ($Value) { 'True' } else { 'False' }) } - return [string]$Value -} - -function Format-PythonList { - param($Value) - - if ($null -eq $Value) { return 'None' } - if ($Value -is [System.Array]) { - $items = foreach ($item in $Value) { Format-PythonList $item } - return '[' + ($items -join ', ') + ']' - } - return (Format-PythonValue $Value) -} - -function Get-Cell { - param($Row, [int] $Index) - - if ($Row -is [System.Array]) { return $Row[$Index] } - return $Row[$Index] -} - -function Get-RowCount { - param($Rows) - if ($null -eq $Rows) { return 0 } - return @($Rows).Count -} - -function Get-RowSlice { - param($Rows, [int] $Count) - $allRows = @($Rows) - if ($allRows.Count -eq 0) { return @() } - $slice = @() - for ($i = 0; $i -lt [Math]::Min($Count, $allRows.Count); $i++) { - $slice += , $allRows[$i] - } - return , $slice -} - -function Test-ParityCheck { - param( - [Parameter(Mandatory)] [ParityCheck] $CheckItem, - [Parameter(Mandatory)] [string[]] $Columns, - [Parameter(Mandatory)] $Rows - ) - - $rowCount = Get-RowCount $Rows - - if ($CheckItem.EvalKind -eq 'zero') { - if ($rowCount -eq 0) { - return [pscustomobject]@{ Status = 'FAIL'; Explanation = 'no rows returned (expected one with value 0)' } - } - $value = Get-Cell -Row $Rows[0] -Index 0 - if ($null -eq $value -or $value -eq 0 -or $value -eq 0.0) { - return [pscustomobject]@{ Status = 'PASS'; Explanation = "result = $(Format-PythonValue $value)" } - } - return [pscustomobject]@{ Status = 'FAIL'; Explanation = "expected 0, got $value" } - } - - if ($CheckItem.EvalKind -eq 'within_pct') { - if (($Columns -contains 'delta_pct') -and $rowCount -gt 0) { - $delta = [Math]::Abs([double](Get-Cell -Row $Rows[0] -Index ([array]::IndexOf($Columns, 'delta_pct')))) - $tolerance = if ($null -ne $CheckItem.EvalArg) { [double]$CheckItem.EvalArg } else { 5.0 } - if ($delta -le $tolerance) { - return [pscustomobject]@{ Status = 'PASS'; Explanation = ('delta = {0:F4}% within {1}%' -f $delta, $tolerance) } - } - return [pscustomobject]@{ Status = 'FAIL'; Explanation = ('delta = {0:F4}% exceeds {1}%' -f $delta, $tolerance) } - } - if (($Columns -contains 'delta_abs') -and $rowCount -gt 0) { - $delta = [Math]::Abs([double](Get-Cell -Row $Rows[0] -Index ([array]::IndexOf($Columns, 'delta_abs')))) - if ($delta -lt 1.0) { - return [pscustomobject]@{ Status = 'PASS'; Explanation = ('delta = ${0:F6}' -f $delta) } - } - return [pscustomobject]@{ Status = 'FAIL'; Explanation = ('delta = ${0:F2} (too large for rounding)' -f $delta) } - } - $firstRow = if ($rowCount -gt 0) { Format-PythonList $Rows[0] } else { 'empty' } - return [pscustomobject]@{ Status = 'MANUAL'; Explanation = "cols=$(Format-PythonList $Columns), first_row=$firstRow" } - } - - if ($CheckItem.EvalKind -eq 'nonzero') { - if ($rowCount -eq 0) { - return [pscustomobject]@{ Status = 'FAIL'; Explanation = 'no rows returned (expected at least one)' } - } - for ($colIndex = 0; $colIndex -lt $Columns.Count; $colIndex++) { - try { - if ([double](Get-Cell -Row $Rows[0] -Index $colIndex) -gt 0) { - return [pscustomobject]@{ Status = 'PASS'; Explanation = "top row $($Columns[$colIndex]) = $(Get-Cell -Row $Rows[0] -Index $colIndex)" } - } - } - catch { - continue - } - } - return [pscustomobject]@{ Status = 'FAIL'; Explanation = "no positive numeric in top row: $(Format-PythonList $Rows[0])" } - } - - $preview = if ($rowCount -gt 0) { Get-RowSlice -Rows $Rows -Count 5 } else { @() } - return [pscustomobject]@{ Status = 'MANUAL'; Explanation = "cols=$(Format-PythonList $Columns), rows[:5]=$(Format-PythonList $preview)" } -} - -function Read-Options { - param([string[]] $Tokens) - - $opts = @{ checks = @() } - $i = 0 - while ($i -lt $Tokens.Count) { - switch ($Tokens[$i]) { - '--db' { $opts.db = $Tokens[++$i] } - '-Db' { $opts.db = $Tokens[++$i] } - '--endpoint' { $opts.endpoint = $Tokens[++$i] } - '-Endpoint' { $opts.endpoint = $Tokens[++$i] } - '--check' { $opts.checks += [int]$Tokens[++$i] } - '-Check' { $opts.checks += [int]$Tokens[++$i] } - default { throw "unrecognized arguments: $($Tokens[$i])" } - } - $i++ - } - return $opts -} - -function Test-NameParity { - param( - [Parameter(Mandatory)] [string] $Endpoint, - [Parameter(Mandatory)] [string] $IngestionDb, - [Parameter(Mandatory)] [string] $HubDb - ) - - $ingestionQueryUrl = if ($Endpoint -match '/v1/rest/query$') { $Endpoint } else { "$($Endpoint.TrimEnd('/'))/v1/rest/query" } - - function Invoke-NameQuery { - param([string] $Db, [string] $Csl) - $body = @{ db = $Db; csl = $Csl } | ConvertTo-Json -Compress - $headers = @{ 'Content-Type' = 'application/json'; 'x-ms-client-version' = 'Kusto.Python.Client:1.0.0' } - try { - $resp = Invoke-RestMethod -Method Post -Uri $ingestionQueryUrl -Headers $headers -Body $body -TimeoutSec 30 - $cols = @($resp.Tables[0].Columns | ForEach-Object { $_.ColumnName }) - $rows = @($resp.Tables[0].Rows) - return @{ Cols = $cols; Rows = $rows; Error = $null } - } - catch { return @{ Cols = @(); Rows = @(); Error = $_.Exception.Message } } - } - - $ingTables = Invoke-NameQuery -Db $IngestionDb -Csl '.show tables | project TableName' - if ($ingTables.Error) { - return [pscustomobject]@{ Status = 'FAIL'; Explanation = "cannot query Ingestion tables: $($ingTables.Error)" } - } - - $hubFunctions = Invoke-NameQuery -Db $HubDb -Csl '.show functions | project Name' - if ($hubFunctions.Error) { - return [pscustomobject]@{ Status = 'FAIL'; Explanation = "cannot query Hub functions: $($hubFunctions.Error)" } - } - - $tableNames = @($ingTables.Rows | ForEach-Object { [string]$_[0] }) - $funcNames = @($hubFunctions.Rows | ForEach-Object { [string]$_[0] }) - - # Assert no FtkLocal-prefixed or -suffixed objects exist - $ftkLocalObjects = @($tableNames | Where-Object { $_ -imatch 'FtkLocal' }) + - @($funcNames | Where-Object { $_ -imatch 'FtkLocal' }) - if ($ftkLocalObjects.Count -gt 0) { - return [pscustomobject]@{ Status = 'FAIL'; Explanation = "FtkLocal-named objects found: $($ftkLocalObjects -join ', ')" } - } - - # Assert expected core Ingestion tables are present - $requiredTables = @('Costs_raw','Prices_raw','Costs_final_v1_2','Prices_final_v1_2') - $missingTables = @($requiredTables | Where-Object { $tableNames -notcontains $_ }) - if ($missingTables.Count -gt 0) { - return [pscustomobject]@{ Status = 'FAIL'; Explanation = "Ingestion missing expected tables: $($missingTables -join ', ')" } - } - - # Assert expected core Hub functions are present - $requiredFuncs = @('Costs_v1_2','Prices_v1_2','Costs','Prices') - $missingFuncs = @($requiredFuncs | Where-Object { $funcNames -notcontains $_ }) - if ($missingFuncs.Count -gt 0) { - return [pscustomobject]@{ Status = 'FAIL'; Explanation = "Hub missing expected functions: $($missingFuncs -join ', ')" } - } - - return [pscustomobject]@{ Status = 'PASS'; Explanation = "Ingestion: $($tableNames.Count) table(s), Hub: $($funcNames.Count) function(s); no FtkLocal objects; all core names present" } -} - -function Main { - param([string[]] $Tokens) - - $opts = Read-Options -Tokens $Tokens - $endpoint = Get-QueryUrl $(if ($opts.ContainsKey('endpoint')) { $opts.endpoint } else { $null }) - $database = Get-Database $(if ($opts.ContainsKey('db')) { $opts.db } else { $null }) - - $text = Get-Content -Raw -Path $script:ParityFile - $checks = @(Get-ParityChecks -Text $text) - - # Route Hub-view checks to the Hub database regardless of --db default. - # CHECK 9: Costs_v1_2 aggregation (Hub function) - # CHECK 10: Costs_v1_2 vs Costs_final_v1_2 reconciliation (Hub, with cross-DB ref) - foreach ($checkItem in $checks) { - if ($checkItem.Number -in @(9, 10)) { - $checkItem.Database = 'Hub' - } - } - - if ($opts.checks.Count -gt 0) { - $selected = @{} - foreach ($checkNumber in $opts.checks) { $selected[$checkNumber] = $true } - $checks = @($checks | Where-Object { $selected.ContainsKey($_.Number) }) - } - - Write-Host "Loaded $($checks.Count) parity check(s) from $script:ParityFile`n" - - $passCount = 0 - $failCount = 0 - $manualCount = 0 - $failures = @() - - foreach ($checkItem in $checks) { - # Per-check database: Hub checks use Hub; all others use the $database default (Ingestion). - $checkDb = if ($checkItem.Database) { $checkItem.Database } else { $database } - Write-Host "━━━ CHECK $($checkItem.Number): $($checkItem.Name) [DB: $checkDb] ━━━" - Write-Host " EXPECT: $($checkItem.Expect)" - try { - $response = Invoke-KustoQuery -Endpoint $endpoint -Database $checkDb -Kql $checkItem.Kql - $result = Get-PrimaryResult -Response $response - $columns = $result.Columns - $rows = $result.Rows - $evaluation = Test-ParityCheck -CheckItem $checkItem -Columns $columns -Rows $rows - $status = $evaluation.Status - $explanation = $evaluation.Explanation - } - catch { - $status = 'FAIL' - $explanation = "query error: $($_.Exception.Message)" - $columns = @() - $rows = @() - } - - $marker = @{ PASS = '✓'; FAIL = '✗'; MANUAL = '?' }[$status] - Write-Host " $marker ${status}: $explanation" - if ((Get-RowCount $rows) -gt 0 -and $status -ne 'PASS') { - $preview = Get-RowSlice -Rows $rows -Count 3 - Write-Host " rows[:3] = $(Format-PythonList $preview)" - } - Write-Host '' - - if ($status -eq 'PASS') { - $passCount++ - } - elseif ($status -eq 'FAIL') { - $failCount++ - $failures += [pscustomobject]@{ Check = $checkItem; Explanation = $explanation; Columns = $columns; Rows = $rows } - } - else { - $manualCount++ - } - } - - # Synthetic CHECK 12: name-parity — Ingestion tables and Hub functions. - # Only skip if the user filtered to specific checks and didn't include 12. - $runNameParity = $opts.checks.Count -eq 0 -or ($opts.checks -contains 12) - if ($runNameParity) { - Write-Host "━━━ CHECK 12: Name parity — Ingestion tables + Hub functions [DB: Ingestion+Hub] ━━━" - Write-Host " EXPECT: No FtkLocal-named objects; required core tables and functions present." - try { - $evaluation = Test-NameParity -Endpoint $endpoint -IngestionDb $database -HubDb 'Hub' - $status = $evaluation.Status - $explanation = $evaluation.Explanation - } - catch { - $status = 'FAIL' - $explanation = "name-parity check error: $($_.Exception.Message)" - } - $marker = @{ PASS = '✓'; FAIL = '✗'; MANUAL = '?' }[$status] - Write-Host " $marker ${status}: $explanation" - Write-Host '' - if ($status -eq 'PASS') { $passCount++ } - elseif ($status -eq 'FAIL') { - $failCount++ - $dummyCheck = [ParityCheck]::new(12, 'Name parity', '', '', '') - $failures += [pscustomobject]@{ Check = $dummyCheck; Explanation = $explanation; Columns = @(); Rows = @() } - } - else { $manualCount++ } - } - - Write-Host ('━' * 60) - Write-Host "SUMMARY: $passCount pass, $failCount fail, $manualCount manual (of $($checks.Count + ($runNameParity ? 1 : 0)) total)" - if ($failures.Count -gt 0) { - Write-Host "`nFAILURES:" - foreach ($failure in $failures) { - Write-Host " CHECK $($failure.Check.Number) ($($failure.Check.Name)): $($failure.Explanation)" - } - Write-Host "`nWrite a gap report at notes/parity-gaps.md for each failure before re-running." - return 1 - } - if ($manualCount -gt 0) { - Write-Host "`nManual checks above require human inspection but did not block the run." - } - return 0 -} - -exit (Main -Tokens $args) diff --git a/src/templates/finops-hub-local/scripts/settings.ps1 b/src/templates/finops-hub-local/scripts/settings.ps1 deleted file mode 100644 index 3294accd6..000000000 --- a/src/templates/finops-hub-local/scripts/settings.ps1 +++ /dev/null @@ -1,100 +0,0 @@ -#!/usr/bin/env pwsh -#Requires -Version 7.0 -<# -.SYNOPSIS - Manage the local ftklocal settings.json (hub fidelity layer). - -.DESCRIPTION - settings.json mirrors the shape written by the deployed FinOps Hub to the - Azure Storage `config` container (schema https://aka.ms/finops/hubs/settings-schema). - It is data-like and gitignored so that scope / billing-account identifiers - are never committed. settings.example.json is the sanitized template. - - Subcommands (default: show): - show Print the resolved settings (from settings.json or built-in defaults). - init Copy settings.example.json → settings.json if the file is absent. - get Return a single dot-path value, e.g. 'retention.raw.days'. - -.EXAMPLE - pwsh scripts/settings.ps1 init - -.EXAMPLE - pwsh scripts/settings.ps1 show - -.EXAMPLE - pwsh scripts/settings.ps1 get retention.raw.days -#> -[CmdletBinding()] -param( - [Parameter(Position = 0)] - [ValidateSet('show', 'init', 'get')] - [string] $Subcommand = 'show', - - [Parameter(Position = 1)] - [string] $DotPath -) - -Set-StrictMode -Version Latest -$ErrorActionPreference = 'Stop' - -$script:FtkLocalHome = Split-Path -Parent (Split-Path -Parent $PSCommandPath) -$script:SettingsPath = Join-Path $script:FtkLocalHome 'settings.json' -$script:ExamplePath = Join-Path $script:FtkLocalHome 'settings.example.json' - -# Default settings (matches Copy-FileToAzureBlob.ps1 defaults, lines 66-86) -$script:Defaults = [ordered]@{ - '$schema' = 'https://aka.ms/finops/hubs/settings-schema' - type = 'HubInstance' - version = '0.0.0' - learnMore = 'https://aka.ms/finops/hubs' - scopes = @() - retention = [ordered]@{ - msexports = @{ days = 0 } - ingestion = @{ months = 13 } - raw = @{ days = 3650 } - final = @{ months = 13 } - } -} - -function Read-Settings { - if (Test-Path $script:SettingsPath) { - return Get-Content -Raw -Path $script:SettingsPath | ConvertFrom-Json - } - Write-Warning "settings.json not found — using built-in defaults. Run 'pwsh scripts/settings.ps1 init' to create it." - return $script:Defaults | ConvertTo-Json -Depth 10 | ConvertFrom-Json -} - -switch ($Subcommand) { - 'init' { - if (Test-Path $script:SettingsPath) { - Write-Host "settings.json already exists at: $script:SettingsPath" - Write-Host "Edit it to add your billing-account scopes and adjust retention values." - } - else { - if (-not (Test-Path $script:ExamplePath)) { - throw "settings.example.json not found at: $script:ExamplePath" - } - Copy-Item -Path $script:ExamplePath -Destination $script:SettingsPath - Write-Host "Created settings.json from settings.example.json." - Write-Host "Edit $script:SettingsPath to add real billing-account scopes if needed." - Write-Host "Note: settings.json is gitignored — it will not be committed." - } - } - - 'show' { - $s = Read-Settings - $s | ConvertTo-Json -Depth 10 - } - - 'get' { - if (-not $DotPath) { throw "Usage: settings.ps1 get e.g. 'retention.raw.days'" } - $s = Read-Settings - $parts = $DotPath -split '\.' - $value = $s - foreach ($part in $parts) { - if ($null -eq $value) { throw "Path '$DotPath' not found in settings (null at '$part')" } - $value = $value.$part - } - Write-Output $value - } -} diff --git a/src/templates/finops-hub-local/settings.example.json b/src/templates/finops-hub-local/settings.example.json deleted file mode 100644 index e9cface46..000000000 --- a/src/templates/finops-hub-local/settings.example.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "$schema": "https://aka.ms/finops/hubs/settings-schema", - "type": "HubInstance", - "version": "0.0.0", - "learnMore": "https://aka.ms/finops/hubs", - "scopes": [ - { "scope": "/providers/Microsoft.Billing/billingAccounts/" } - ], - "retention": { - "msexports": { "days": 0 }, - "ingestion": { "months": 13 }, - "raw": { "days": 3650 }, - "final": { "months": 13 } - } -} diff --git a/src/templates/finops-hub-local/tests/parity-checks.kql b/src/templates/finops-hub-local/tests/parity-checks.kql deleted file mode 100644 index d3839c665..000000000 --- a/src/templates/finops-hub-local/tests/parity-checks.kql +++ /dev/null @@ -1,164 +0,0 @@ -// ============================================================================ -// FTK-Local Parity Checks — two-DB topology (Ingestion + Hub) -// ============================================================================ -// Validates that data flowing Costs_raw → Costs_transform_v1_2 → Costs_final_v1_2 -// matches FTK ADX semantics. Each numbered check has an EXPECTED rule documented -// in the comment header. scripts/run-parity-checks.ps1 reads this file, executes -// each check against the appropriate database (Ingestion or Hub), evaluates the -// EVAL rule, and produces a pass/fail report. -// -// ROUTING (enforced by run-parity-checks.ps1 per-check DB assignment): -// Checks 1–8, 11–13 → Ingestion (raw tables, final tables) -// Checks 9, 10 → Hub (view functions cross-referencing Ingestion) -// -// CONVENTIONS -// - Each check is a block: // CHECK : , // EXPECT: , then the KQL. -// - The runner splits checks on the `// CHECK ` boundary. -// - Each block must end with a single tabular result; no multi-statement KQL. -// ============================================================================ - -// CHECK 1: row count parity -// EXPECT: final_count within 5% of raw_count (some rows may be filtered by transform). -// EVAL: abs(final_count - raw_count) / raw_count <= 0.05 -let raw_count = toscalar(Costs_raw | count); -let final_count = toscalar(Costs_final_v1_2 | count); -print - raw_count = raw_count, - final_count = final_count, - delta_pct = todouble(final_count - raw_count) / todouble(raw_count) * 100.0 - -// CHECK 2: BilledCost preservation -// EXPECT: sum(BilledCost) in final equals sum(BilledCost) in raw (no silent drops). -// EVAL: abs(raw_sum - final_sum) / abs(raw_sum) < 0.001 (allow rounding noise) -let raw_sum = toscalar(Costs_raw | summarize sum(BilledCost)); -let final_sum = toscalar(Costs_final_v1_2 | summarize sum(BilledCost)); -print - raw_sum_BilledCost = raw_sum, - final_sum_BilledCost = final_sum, - delta_abs = abs(raw_sum - final_sum) - -// CHECK 3: EffectiveCost zeroed for commitment-discount purchases -// EXPECT: sum(EffectiveCost) == 0 for ChargeCategory='Purchase' AND isnotempty(CommitmentDiscountId). -// This is an FTK bug-fix: Azure exports incorrectly set a non-zero EffectiveCost on -// the purchase row of a reservation; amortization spreads cost across usage rows. -// EVAL: result == 0 -Costs_final_v1_2 -| where ChargeCategory == 'Purchase' and isnotempty(CommitmentDiscountId) -| summarize sum_EffectiveCost = sum(EffectiveCost) - -// CHECK 4: data-quality flag dictionary populates -// EXPECT: x_SourceChanges contains documented codes (MissingContractedCost, -// InvalidChargeFrequency, etc.). Print top 10 codes by frequency. -// EVAL: print only — surface what flags fired. Failure mode is "no flags at all" -// which would mean the dictionary wiring is broken. -Costs_final_v1_2 -| where isnotempty(x_SourceChanges) -| extend first_code = tostring(split(x_SourceChanges, ',')[0]) -| summarize Count = count() by first_code -| order by Count desc -| take 10 - -// CHECK 5: AmortizationClass labels amortized usage -// EXPECT: ZERO rows where ChargeCategory='Usage' AND isnotempty(CommitmentDiscountId) -// AND x_AmortizationClass != 'Amortized Charge'. The transform should label every -// reservation-covered usage row as Amortized Charge. -// EVAL: result == 0 -Costs_final_v1_2 -| where ChargeCategory == 'Usage' - and isnotempty(CommitmentDiscountId) - and x_AmortizationClass != 'Amortized Charge' -| count - -// CHECK 6: TotalSavings populated for services with reservations -// EXPECT: sum(x_TotalSavings) > 0 for services that typically have RIs/SPs -// (Virtual Machines, Storage Accounts). Top 10 by sum_TotalSavings. -// EVAL: at least one row with sum > 0 for ServiceName in {Virtual Machines, -// Storage Accounts, Azure Reservations, Azure Savings Plan for Compute}. -Costs_final_v1_2 -| summarize sum_TotalSavings = sum(x_TotalSavings) by ServiceName -| order by sum_TotalSavings desc -| take 10 - -// CHECK 7: parse_resourceid populates x_ResourceGroupName when ARM path has one -// EXPECT: ZERO rows where isempty(x_ResourceGroupName) AND isnotempty(ResourceId) -// AND ResourceId contains '/resourcegroups/'. parse_resourceid is supposed to -// extract the RG from any ARM path that contains one. -// EVAL: result == 0 -Costs_final_v1_2 -| where isempty(x_ResourceGroupName) - and isnotempty(ResourceId) - and ResourceId contains '/resourcegroups/' -| count - -// CHECK 8: PublisherName normalized (no 'Microsoft Corporation') -// EXPECT: 'Microsoft Corporation' should be normalized to 'Microsoft' by the -// transform; the raw string should never appear in the final table. -// EVAL: result == 0 -Costs_final_v1_2 -| where PublisherName == 'Microsoft Corporation' -| count - -// CHECK 9: Hub view function returns plausible totals -// EXPECT: Costs_v1_2 | summarize sum(EffectiveCost) by ServiceCategory returns -// non-empty result. The Hub view is what consumers query; verifies it's wired. -// EVAL: at least one row with sum_EffectiveCost > 0. -// DB: Hub -Costs_v1_2 -| summarize sum_EffectiveCost = sum(EffectiveCost) by ServiceCategory -| order by sum_EffectiveCost desc - -// CHECK 10: Costs_v1_2 vs Costs_final_v1_2 BilledCost reconciliation -// EXPECT: sum(BilledCost) from Costs_v1_2 (Hub view) == sum from Costs_final_v1_2 -// (Ingestion table). The Hub view should not silently filter or duplicate. -// EVAL: abs(view_sum - final_sum) / abs(final_sum) < 0.001 -// DB: Hub — uses database('Ingestion').Costs_final_v1_2 cross-DB reference -let view_sum = toscalar(Costs_v1_2 | summarize sum(BilledCost)); -let final_sum_check10 = toscalar(database('Ingestion').Costs_final_v1_2 | summarize sum(BilledCost)); -print - view_sum_BilledCost = view_sum, - final_sum_BilledCost = final_sum_check10, - delta_abs = abs(view_sum - final_sum_check10) - -// CHECK 11: Prices row count parity per PriceType -// EXPECT: Consumption + ReservedInstance counts in Prices_final match Prices_raw -// exactly; SavingsPlan is allowed to drift by up to 2% per the documented FTK -// upstream lookup-fanout (see notes/parity-gaps.md §"T-2.3 root cause analysis"). -// Failure mode is Consumption or RI drift, which would indicate a real defect. -// EVAL: manual — print delta per PriceType; the runner flags it as MANUAL but -// the human-readable PASS criterion is: Consumption delta=0, RI delta=0, -// SP delta within +/-5% of SP raw count. -let raw_by = Prices_raw | summarize raw_n=count() by PriceType; -let final_by = Prices_final_v1_2 | summarize final_n=count() by PriceType=x_SkuPriceType; -raw_by -| join kind=fullouter final_by on PriceType -| project - PriceType = coalesce(PriceType, PriceType1), - raw_n = coalesce(raw_n, 0), - final_n = coalesce(final_n, 0), - delta = coalesce(final_n, 0) - coalesce(raw_n, 0), - delta_pct = iff(coalesce(raw_n, 0) > 0, - round(todouble(coalesce(final_n, 0) - coalesce(raw_n, 0)) / todouble(coalesce(raw_n, 1)) * 100.0, 4), - 0.0) -| order by raw_n desc - -// CHECK 13: Prices value parity — UnitPrice/ContractedUnitPrice (Consumption + RI) -// EXPECT: sum(UnitPrice) in Prices_raw equals sum(ContractedUnitPrice) in Prices_final_v1_2 -// for Consumption and ReservedInstance PriceTypes. These columns pass through the -// Prices transform unchanged. SavingsPlan is excluded — the transform enriches SP prices -// via cross-row lookups, so raw ≠ final for SP by design. -// Guards: (1) raw_n > 0 — rows for these types actually exist; (2) final_n == raw_n — -// no rows were lost or duplicated. If either guard fails, delta_pct is set to a large -// sentinel (9999) so the check FAILS rather than passing vacuously. -// EVAL: abs(raw_sum - final_sum) / abs(raw_sum) < 0.001 -let raw_n = toscalar(Prices_raw | where PriceType in ('Consumption', 'ReservedInstance') | count); -let final_n = toscalar(Prices_final_v1_2 | where x_SkuPriceType in ('Consumption', 'ReservedInstance') | count); -let raw_sum = toscalar(Prices_raw | where PriceType in ('Consumption', 'ReservedInstance') | summarize sum(UnitPrice)); -let final_sum = toscalar(Prices_final_v1_2 | where x_SkuPriceType in ('Consumption', 'ReservedInstance') | summarize sum(ContractedUnitPrice)); -print - raw_n = raw_n, - final_n = final_n, - raw_sum_UnitPrice = raw_sum, - final_sum_ContractedUnitPrice = final_sum, - delta_pct = iff(raw_n == 0 or final_n != raw_n, - 9999.0, - todouble(abs(raw_sum - final_sum)) / todouble(abs(raw_sum)) * 100.0) From bce7f658d793b01e83cea58c72d106ced5f4e3c2 Mon Sep 17 00:00:00 2001 From: MSBrett <24294904+MSBrett@users.noreply.github.com> Date: Thu, 18 Jun 2026 13:49:03 -0700 Subject: [PATCH 25/51] docs: clarify local hub prerequisites Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs-mslearn/toolkit/hubs/run-hubs-locally.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs-mslearn/toolkit/hubs/run-hubs-locally.md b/docs-mslearn/toolkit/hubs/run-hubs-locally.md index 4461bea9f..a0b412095 100644 --- a/docs-mslearn/toolkit/hubs/run-hubs-locally.md +++ b/docs-mslearn/toolkit/hubs/run-hubs-locally.md @@ -23,7 +23,7 @@ A local hub is useful to explore the FinOps hub data model, validate cost data, ## Prerequisites -- [Docker](https://docs.docker.com/get-docker/), to run the Kusto emulator container, with at least 16 GB of memory available to the container. +- [Docker](https://docs.docker.com/get-docker/), to run the Kusto emulator container, with at least 16 GB of memory available to the container. Start Docker before you run the commands and verify `docker info` works in the same PowerShell session. - [PowerShell 7](/powershell/scripting/install/installing-powershell) — every command in this article is PowerShell. - [Azure PowerShell](/powershell/azure/install-azure-powershell) (the `Az` modules) — only needed to download cost data from a storage account. - FOCUS cost exports as Parquet, either from a [FinOps hub storage account](configure-scopes.md) or [Cost Management exports](/cost-management-billing/costs/tutorial-improved-exports). @@ -31,7 +31,7 @@ A local hub is useful to explore the FinOps hub data model, validate cost data, For background on the emulator and its platform requirements, see the [Kusto emulator overview](/azure/data-explorer/kusto-emulator-overview) and [installation guide](/azure/data-explorer/kusto-emulator-install). > [!NOTE] -> This article assumes you're already signed in to Azure with `Connect-AzAccount` and have selected the subscription that holds your cost data. It never runs a sign-in command. If you aren't downloading data from a storage account, you can skip Azure sign-in entirely. +> This article assumes you're already signed in to Azure with `Connect-AzAccount` and have selected the subscription that holds your cost data. It never runs a sign-in command. If you aren't downloading data from a storage account, you can skip Azure sign-in entirely. If you download from Azure Storage with `-UseConnectedAccount`, your signed-in identity also needs storage data-plane access, such as **Storage Blob Data Reader**, on the storage account or container.
@@ -84,7 +84,7 @@ foreach ($db in 'Ingestion', 'Hub') { ## Load the schema -Download the hub's setup scripts from the latest toolkit release and load each into its database. These are the same `finops-hub-fabric-setup-*.kql` files used to set up a [Microsoft Fabric hub](deploy.md#set-up-microsoft-fabric) — each is a single `.execute database script` that creates every table, mapping, transform function, and update policy. +Download the hub's setup scripts from the latest toolkit release and load each into its database. These are the same `finops-hub-fabric-setup-*.kql` files used to set up a [Microsoft Fabric hub](deploy.md#optional-set-up-microsoft-fabric) — each is a single `.execute database script` that creates every table, mapping, transform function, and update policy. The Ingestion script has one placeholder, `$$rawRetentionInDays$$` (how long to keep raw data). Replace it with a number of days before loading. From 96bd18388c3ecddbfa0f274618aedbf48fd07c7c Mon Sep 17 00:00:00 2001 From: MSBrett <24294904+MSBrett@users.noreply.github.com> Date: Thu, 18 Jun 2026 19:18:20 -0700 Subject: [PATCH 26/51] docs: improve local hub ingestion guidance Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs-mslearn/toolkit/hubs/run-hubs-locally.md | 59 ++++++++++++++----- 1 file changed, 45 insertions(+), 14 deletions(-) diff --git a/docs-mslearn/toolkit/hubs/run-hubs-locally.md b/docs-mslearn/toolkit/hubs/run-hubs-locally.md index a0b412095..477ec5f74 100644 --- a/docs-mslearn/toolkit/hubs/run-hubs-locally.md +++ b/docs-mslearn/toolkit/hubs/run-hubs-locally.md @@ -37,13 +37,15 @@ For background on the emulator and its platform requirements, see the [Kusto emu ## Start the emulator -Create a working folder, then start the emulator. The `export` folder is mounted read-only into the container at `/data/export`; you'll download cost data into it later. For other ways to run the container, see [Install the Kusto emulator](/azure/data-explorer/kusto-emulator-install). +Create a working folder outside the repository, then start the emulator. The export folder is mounted read-only into the container at `/data/export`; you'll download cost data into it later. For other ways to run the container, see [Install the Kusto emulator](/azure/data-explorer/kusto-emulator-install). ```powershell -New-Item -ItemType Directory -Force -Path export | Out-Null +$exportPath = '../exports/local-hub' + +New-Item -ItemType Directory -Force -Path $exportPath | Out-Null docker run -d --name finops-hub-local --platform linux/amd64 ` -p 8082:8080 -m 16g -e ACCEPT_EULA=Y ` - -v "$($PWD.Path)/export:/data/export:ro" ` + -v "$((Resolve-Path $exportPath).Path):/data/export:ro" ` mcr.microsoft.com/azuredataexplorer/kustainer-linux:latest ``` @@ -90,15 +92,16 @@ The Ingestion script has one placeholder, `$$rawRetentionInDays$$` (how long to ```powershell $release = 'https://github.com/microsoft/finops-toolkit/releases/latest/download' -Invoke-WebRequest "$release/finops-hub-fabric-setup-Ingestion.kql" -OutFile ./setup-Ingestion.kql -Invoke-WebRequest "$release/finops-hub-fabric-setup-Hub.kql" -OutFile ./setup-Hub.kql +New-Item -ItemType Directory -Force -Path "$exportPath/setup" | Out-Null +Invoke-WebRequest "$release/finops-hub-fabric-setup-Ingestion.kql" -OutFile "$exportPath/setup/setup-Ingestion.kql" +Invoke-WebRequest "$release/finops-hub-fabric-setup-Hub.kql" -OutFile "$exportPath/setup/setup-Hub.kql" # Ingestion: raw tables, transforms, final tables, update policies -$ingestion = (Get-Content ./setup-Ingestion.kql -Raw) -replace '\$\$rawRetentionInDays\$\$', '90' +$ingestion = (Get-Content "$exportPath/setup/setup-Ingestion.kql" -Raw) -replace '\$\$rawRetentionInDays\$\$', '90' Invoke-Kusto Ingestion $ingestion # Hub: the Costs/Prices/Transactions view functions -Invoke-Kusto Hub (Get-Content ./setup-Hub.kql -Raw) +Invoke-Kusto Hub (Get-Content "$exportPath/setup/setup-Hub.kql" -Raw) ``` Each response is a table with one row per statement. The Hub functions reference `database('Ingestion').*`, which resolves because both databases exist. @@ -110,9 +113,9 @@ Each response is a table with one row per statement. The Hub functions reference FinOps hubs enrich cost data with open-data reference tables (regions, services, resource types, and pricing units). Download the reference CSVs from the toolkit release, then load them with the same commands a deployed hub uses. ```powershell -New-Item -ItemType Directory -Force -Path export/open-data | Out-Null +New-Item -ItemType Directory -Force -Path "$exportPath/open-data" | Out-Null foreach ($f in 'PricingUnits', 'Regions', 'ResourceTypes', 'Services') { - Invoke-WebRequest "$release/$f.csv" -OutFile "export/open-data/$f.csv" + Invoke-WebRequest "$release/$f.csv" -OutFile "$exportPath/open-data/$f.csv" } $openData = @' @@ -129,7 +132,7 @@ Invoke-Kusto Ingestion $openData ## Download your cost data -Download FOCUS cost and price exports from your storage account into the `export` folder, using your existing Azure sign-in. This works against either the **msexports** or **ingestion** container of a FinOps hub, or any container holding Cost Management exports. Only the data files (`manifest.json` and `*.parquet`) are downloaded. +Download FOCUS cost and price exports from your storage account into the export folder, using your existing Azure sign-in. This works against either the **msexports** or **ingestion** container of a FinOps hub, or any container holding Cost Management exports. Only the data files (`manifest.json` and `*.parquet`) are downloaded. Set these to match your storage account, then run the download: @@ -143,7 +146,7 @@ Get-AzStorageBlob -Container $container -Context $ctx -Prefix $prefix | Where-Object { $_.Name -match '(manifest\.json|\.parquet)$' } | ForEach-Object { Get-AzStorageBlobContent -Container $container -Context $ctx -Blob $_.Name ` - -Destination ./export -Force | Out-Null + -Destination $exportPath -Force | Out-Null } ``` @@ -156,25 +159,53 @@ Get-AzStorageBlob -Container $container -Context $ctx -Prefix $prefix | Ingest each Parquet file into the matching raw table. As rows land, the update policy created by the schema runs the FOCUS transform automatically and appends the result to the final table — the same mechanism a deployed hub uses. Cost exports go to `Costs_raw`; price sheets go to `Prices_raw`. +For large exports, use a small amount of parallelism. Start with about one ingestion thread per 8 GB of emulator memory. For the 16 GB container below, use two threads; if you increase the container to 32 GB, four threads is a good starting point. + ```powershell +$ingestConcurrency = 2 + function Invoke-Ingest { param([string]$Table, [string]$Mapping, [string]$File) - $rel = [IO.Path]::GetRelativePath((Resolve-Path ./export), $File) -replace '\\', '/' + $rel = [IO.Path]::GetRelativePath((Resolve-Path $exportPath), $File) -replace '\\', '/' $path = "/data/export/$rel" Invoke-Kusto Ingestion ".ingest into table $Table (h@'$path') with (format='parquet', ingestionMappingReference='$Mapping')" | Out-Null Write-Host " ingested $(Split-Path $File -Leaf)" } # Group exports by manifest type, then ingest each dataset's Parquet files -Get-ChildItem ./export -Recurse -Filter manifest.json | ForEach-Object { +$ingestJobs = Get-ChildItem $exportPath -Recurse -Filter manifest.json | ForEach-Object { $type = if ((Get-Content $_.FullName -Raw) -match '"type"\s*:\s*"(FocusCost|PriceSheet)"') { $Matches[1] } else { return } $target = if ($type -eq 'FocusCost') { 'Costs_raw', 'Costs_raw_mapping' } else { 'Prices_raw', 'Prices_raw_mapping' } Get-ChildItem $_.Directory -Filter *.parquet | ForEach-Object { - Invoke-Ingest $target[0] $target[1] $_.FullName + [PSCustomObject]@{ + Table = $target[0] + Mapping = $target[1] + File = $_.FullName + } } } + +$ingestJobs | ForEach-Object -Parallel { + $hub = $using:hub + $exportPath = $using:exportPath + function Invoke-Kusto { + param([string]$Database, [string]$Command, [ValidateSet('mgmt','query')][string]$Endpoint = 'mgmt') + Invoke-RestMethod -Uri "$hub/$Endpoint" -Method Post -ContentType 'application/json' ` + -Body (@{ db = $Database; csl = $Command } | ConvertTo-Json) + } + function Invoke-Ingest { + param([string]$Table, [string]$Mapping, [string]$File) + $rel = [IO.Path]::GetRelativePath((Resolve-Path $exportPath), $File) -replace '\\', '/' + $path = "/data/export/$rel" + Invoke-Kusto Ingestion ".ingest into table $Table (h@'$path') with (format='parquet', ingestionMappingReference='$Mapping')" | Out-Null + Write-Host " ingested $(Split-Path $File -Leaf)" + } + Invoke-Ingest $_.Table $_.Mapping $_.File +} -ThrottleLimit $ingestConcurrency ``` +If the emulator exits with code 137, it ran out of memory. Reduce `$ingestConcurrency`, increase the container memory, or ingest one month at a time and restart the emulator between batches. For very large imports, keep a per-file log so you can retry failed files without restarting the entire load. + Confirm the data landed and the final tables filled in: ```powershell From 7aee803ac9c2475c33582d933a213d60cbccd1eb Mon Sep 17 00:00:00 2001 From: MSBrett <24294904+MSBrett@users.noreply.github.com> Date: Thu, 18 Jun 2026 19:34:57 -0700 Subject: [PATCH 27/51] docs: broaden local hub positioning Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs-mslearn/toolkit/hubs/run-hubs-locally.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs-mslearn/toolkit/hubs/run-hubs-locally.md b/docs-mslearn/toolkit/hubs/run-hubs-locally.md index 477ec5f74..01be53578 100644 --- a/docs-mslearn/toolkit/hubs/run-hubs-locally.md +++ b/docs-mslearn/toolkit/hubs/run-hubs-locally.md @@ -1,6 +1,6 @@ --- title: Run FinOps hubs locally -description: Stand up a FinOps hub on your machine in a local container and ingest cost data, using the same KQL and open data as a deployed hub. +description: Stand up a FinOps hub on your own hardware in a local container and ingest cost data, using the same KQL and open data as a deployed hub. author: flanakin ms.author: micflan ms.date: 06/18/2026 @@ -15,9 +15,9 @@ ms.reviewer: micflan # Run FinOps hubs locally -This article shows how to run a FinOps hub on your own machine using the [Kusto emulator](/azure/data-explorer/kusto-emulator-overview), a free local container. It uses the same analytics KQL, transforms, and open data as a deployed [FinOps hub](finops-hubs-overview.md) — only the host changes from Azure Data Explorer to a local container. Run the PowerShell blocks below in order; each is a step you (or an agent) can run one after another. +This article shows how to run a FinOps hub on your own hardware using the [Kusto emulator](/azure/data-explorer/kusto-emulator-overview), a free local container. It uses the same analytics KQL, transforms, and open data as a deployed [FinOps hub](finops-hubs-overview.md) — only the host changes from Azure Data Explorer to a local container. Run the PowerShell blocks below in order; each is a step you (or an agent) can run one after another. -A local hub is useful to explore the FinOps hub data model, validate cost data, or work offline. It isn't a replacement for a deployed hub — there's no scheduled ingestion, networking, or sharing. +A local hub is useful when you need full hub analysis without managed daily refresh: exploring the data model, validating large customer datasets, supporting consulting deliveries, or working from on-premises, other-cloud, or disconnected environments where you can bring exported data to the machine. It isn't a replacement for a deployed hub — there's no scheduled ingestion, networking, security, or sharing.
@@ -25,8 +25,8 @@ A local hub is useful to explore the FinOps hub data model, validate cost data, - [Docker](https://docs.docker.com/get-docker/), to run the Kusto emulator container, with at least 16 GB of memory available to the container. Start Docker before you run the commands and verify `docker info` works in the same PowerShell session. - [PowerShell 7](/powershell/scripting/install/installing-powershell) — every command in this article is PowerShell. -- [Azure PowerShell](/powershell/azure/install-azure-powershell) (the `Az` modules) — only needed to download cost data from a storage account. -- FOCUS cost exports as Parquet, either from a [FinOps hub storage account](configure-scopes.md) or [Cost Management exports](/cost-management-billing/costs/tutorial-improved-exports). +- [Azure PowerShell](/powershell/azure/install-azure-powershell) (the `Az` modules) — only needed to download cost data from an Azure storage account. +- FOCUS cost exports as Parquet, either from a [FinOps hub storage account](configure-scopes.md), [Cost Management exports](/cost-management-billing/costs/tutorial-improved-exports), or another source staged in the same local folder structure. For background on the emulator and its platform requirements, see the [Kusto emulator overview](/azure/data-explorer/kusto-emulator-overview) and [installation guide](/azure/data-explorer/kusto-emulator-install). From 7e951a83b4428f844abc0778d29fe762e823ad61 Mon Sep 17 00:00:00 2001 From: MSBrett <24294904+MSBrett@users.noreply.github.com> Date: Thu, 18 Jun 2026 19:37:51 -0700 Subject: [PATCH 28/51] docs: remove unused local ingest helper Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs-mslearn/toolkit/hubs/run-hubs-locally.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/docs-mslearn/toolkit/hubs/run-hubs-locally.md b/docs-mslearn/toolkit/hubs/run-hubs-locally.md index 01be53578..a30daf4d9 100644 --- a/docs-mslearn/toolkit/hubs/run-hubs-locally.md +++ b/docs-mslearn/toolkit/hubs/run-hubs-locally.md @@ -164,14 +164,6 @@ For large exports, use a small amount of parallelism. Start with about one inges ```powershell $ingestConcurrency = 2 -function Invoke-Ingest { - param([string]$Table, [string]$Mapping, [string]$File) - $rel = [IO.Path]::GetRelativePath((Resolve-Path $exportPath), $File) -replace '\\', '/' - $path = "/data/export/$rel" - Invoke-Kusto Ingestion ".ingest into table $Table (h@'$path') with (format='parquet', ingestionMappingReference='$Mapping')" | Out-Null - Write-Host " ingested $(Split-Path $File -Leaf)" -} - # Group exports by manifest type, then ingest each dataset's Parquet files $ingestJobs = Get-ChildItem $exportPath -Recurse -Filter manifest.json | ForEach-Object { $type = if ((Get-Content $_.FullName -Raw) -match '"type"\s*:\s*"(FocusCost|PriceSheet)"') { $Matches[1] } else { return } From a32f12b27a80f39bd3d37ab3c8998dc53c340063 Mon Sep 17 00:00:00 2001 From: MSBrett <24294904+MSBrett@users.noreply.github.com> Date: Thu, 18 Jun 2026 19:44:26 -0700 Subject: [PATCH 29/51] docs: update mslearn dates Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs-mslearn/toolkit/changelog.md | 2 +- docs-mslearn/toolkit/hubs/run-hubs-locally.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs-mslearn/toolkit/changelog.md b/docs-mslearn/toolkit/changelog.md index a916e9894..170797346 100644 --- a/docs-mslearn/toolkit/changelog.md +++ b/docs-mslearn/toolkit/changelog.md @@ -3,7 +3,7 @@ title: FinOps toolkit changelog description: Review the latest features and enhancements in the FinOps toolkit, including updates to FinOps hubs, Power BI reports, and more. author: MSBrett ms.author: brettwil -ms.date: 06/18/2026 +ms.date: 06/19/2026 ms.topic: reference ms.service: finops ms.subservice: finops-toolkit diff --git a/docs-mslearn/toolkit/hubs/run-hubs-locally.md b/docs-mslearn/toolkit/hubs/run-hubs-locally.md index a30daf4d9..55621f0a4 100644 --- a/docs-mslearn/toolkit/hubs/run-hubs-locally.md +++ b/docs-mslearn/toolkit/hubs/run-hubs-locally.md @@ -3,7 +3,7 @@ title: Run FinOps hubs locally description: Stand up a FinOps hub on your own hardware in a local container and ingest cost data, using the same KQL and open data as a deployed hub. author: flanakin ms.author: micflan -ms.date: 06/18/2026 +ms.date: 06/19/2026 ms.topic: how-to ms.service: finops ms.subservice: finops-toolkit From 3fc373fb2c1e7cdcfcda2872fde7c4cf1e9d9756 Mon Sep 17 00:00:00 2001 From: MSBrett <24294904+MSBrett@users.noreply.github.com> Date: Fri, 19 Jun 2026 10:34:03 -0700 Subject: [PATCH 30/51] docs: add ftklocal agent skill Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- src/templates/agent-skills/ftklocal/SKILL.md | 119 +++++++ .../ftklocal/references/ftklocal-runbook.md | 298 ++++++++++++++++++ 2 files changed, 417 insertions(+) create mode 100644 src/templates/agent-skills/ftklocal/SKILL.md create mode 100644 src/templates/agent-skills/ftklocal/references/ftklocal-runbook.md diff --git a/src/templates/agent-skills/ftklocal/SKILL.md b/src/templates/agent-skills/ftklocal/SKILL.md new file mode 100644 index 000000000..01ce91350 --- /dev/null +++ b/src/templates/agent-skills/ftklocal/SKILL.md @@ -0,0 +1,119 @@ +--- +name: ftklocal +description: Deploy, operate, recover, and query a local FinOps hub on your own hardware with the Kusto emulator. Use when the user mentions ftklocal, local FinOps hub, running FinOps hubs locally, Kusto emulator hub, offline/on-prem/other-cloud FinOps analysis, local Hub database queries, large exported FOCUS datasets, or validating FinOps toolkit data without Azure-hosted ADX/Fabric. +license: MIT +compatibility: Requires Docker and PowerShell 7. Optional Azure PowerShell is only needed to download exports from Azure Storage. +metadata: + author: microsoft + version: "1.0" +--- + +# ftklocal + +Use this skill to run a FinOps hub analytics layer locally on your own hardware. The local hub uses the Azure Data Explorer Kusto emulator container and the same released FinOps hub setup KQL, schemas, transforms, update policies, open data, and Hub view functions used by deployed hubs. + +## What ftklocal is for + +- **Offline/customer delivery analysis**: Analyze full exported customer datasets without managed daily refresh. +- **Own-hardware/on-prem/other-cloud operation**: Run where Docker and exported data are available, including disconnected environments. +- **Agent UAT and troubleshooting**: Validate hub transforms, large ingestions, and KQL behavior without deploying Azure-hosted ADX or Fabric. +- **Scale shape**: Keep data in Kusto and query summarized results. Do not load tens of GB or hundreds of millions of rows into PowerShell objects. + +Not production: the Kusto emulator has no authentication, access control, encrypted connections, managed ingestion, or production support. It is provided as-is and is not intended for production workloads or benchmark claims. + +## Core model + +- Container endpoint: `http://localhost:8082/v1/rest` +- Databases: + - `Ingestion`: raw tables, final tables, transforms, update policies, open data + - `Hub`: user-facing view functions such as `Costs_v1_2` +- Data flow: + 1. Stage `manifest.json` and `*.parquet` exports under a local export folder. + 2. Mount that folder read-only into the container at `/data/export`. + 3. Load released setup KQL into `Ingestion` and `Hub`. + 4. Load open data CSVs. + 5. `.ingest` Parquet files into `Costs_raw` / `Prices_raw`; update policies fill final tables. + 6. Query the `Hub` database, not local Parquet files, for analysis. + +## Task routing + +Load [references/ftklocal-runbook.md](references/ftklocal-runbook.md) for deployment, recovery, ingestion, troubleshooting, or any non-trivial query/operations task. + +## Quick operations + +### Start existing local hub + +```powershell +$docker = 'C:\Program Files\Docker\Docker\resources\bin\docker.exe' +& $docker start finops-hub-local +``` + +Then wait for: + +```powershell +$hub = 'http://localhost:8082/v1/rest' +function Invoke-Kusto { + param([string]$Database, [string]$Command, [ValidateSet('mgmt','query')][string]$Endpoint = 'mgmt') + Invoke-RestMethod -Uri "$hub/$Endpoint" -Method Post -ContentType 'application/json' ` + -Body (@{ db = $Database; csl = $Command } | ConvertTo-Json) +} +Invoke-Kusto NetDefaultDB '.show version' +``` + +### Stop local hub + +```powershell +$docker = 'C:\Program Files\Docker\Docker\resources\bin\docker.exe' +& $docker stop finops-hub-local +``` + +### Check row counts + +```powershell +foreach ($t in 'Costs_raw', 'Costs_final_v1_2', 'Prices_raw', 'Prices_final_v1_2') { + $count = (Invoke-Kusto Ingestion "$t | count" -Endpoint query).Tables[0].Rows[0][0] + Write-Host "$t`: $count" +} +``` + +### Query analysis data + +```powershell +$result = Invoke-Kusto Hub @' +Costs_v1_2 +| summarize EffectiveCost = round(sum(EffectiveCost), 2) by ServiceCategory +| top 10 by EffectiveCost +'@ -Endpoint query +$result.Tables[0].Rows +``` + +## Operational rules for agents + +1. Prefer Kusto queries over reading Parquet/CSV into PowerShell objects. +2. Use the `Hub` database for analysis; use `Ingestion` only for row counts, raw/final validation, and troubleshooting. +3. For large imports, start around one ingest thread per 8 GB of emulator memory. +4. Exit code 137 means the emulator was OOM-killed: lower concurrency, increase memory, ingest one month at a time, and restart between batches. +5. For very large imports, keep per-file logs and idempotency tags so failed files can be retried. +6. Do not call this production, secure, or managed. It is a local analysis and validation tier. +7. Do not use local storage-export row materialization as the scalable analysis path. + +## Learn more + +| Topic | How to find | +|-------|-------------| +| FinOps hubs overview | `microsoft_docs_search(query="FinOps hubs overview Azure Data Explorer Fabric")` | +| Kusto emulator overview | `microsoft_docs_fetch(url="https://learn.microsoft.com/en-us/azure/data-explorer/kusto-emulator-overview")` | +| Kusto emulator install | `microsoft_docs_fetch(url="https://learn.microsoft.com/en-us/azure/data-explorer/kusto-emulator-install")` | +| Kusto query language | `microsoft_docs_search(query="Azure Data Explorer KQL summarize top project")` | + +## CLI alternative + +If the Learn MCP server is not available, use the `mslearn` CLI instead: + +| MCP tool | CLI command | +|----------|-------------| +| `microsoft_docs_search(query: "...")` | `mslearn search "..."` | +| `microsoft_docs_fetch(url: "...")` | `mslearn fetch "..."` | + +Run directly with `npx @microsoft/learn-cli ` or install globally with `npm install -g @microsoft/learn-cli`. + diff --git a/src/templates/agent-skills/ftklocal/references/ftklocal-runbook.md b/src/templates/agent-skills/ftklocal/references/ftklocal-runbook.md new file mode 100644 index 000000000..de0243275 --- /dev/null +++ b/src/templates/agent-skills/ftklocal/references/ftklocal-runbook.md @@ -0,0 +1,298 @@ +# ftklocal runbook + +This runbook describes how to deploy, manage, ingest, recover, and query a local FinOps hub with the Azure Data Explorer Kusto emulator. + +## Positioning + +ftklocal is the own-hardware path for FinOps hubs. It is useful for customer deliveries, consulting, offline analysis, on-premises/other-cloud hosts, disconnected environments, and agent UAT where managed daily refresh is not required but full-fidelity hub analysis is. + +Use Kusto as the execution engine. Avoid designs that download storage exports and materialize all rows as PowerShell objects; that does not scale to tens of GB or hundreds of millions of rows. + +## Requirements + +- Docker Desktop or Docker engine capable of running Linux containers. +- PowerShell 7. +- At least 16 GB container memory for small/medium imports; for large imports use roughly 8 GB per ingestion thread. +- Exported FOCUS data as Parquet with `manifest.json` files, staged locally or downloaded from a FinOps hub / Cost Management export storage account. +- Optional Az PowerShell modules if downloading from Azure Storage. + +Official Kusto emulator constraints: + +- Runs locally as a Linux Docker container. +- Exposes HTTP only. +- No Microsoft Entra authentication, access control, encrypted connections, managed ingestion, streaming ingestion, or production support. +- No Azure service provisioning or internet connectivity is required after image/data/artifacts are available locally. +- Not intended for production or benchmark claims. + +## Start Docker and existing container + +```powershell +$dockerDesktop = 'C:\Program Files\Docker\Docker\Docker Desktop.exe' +$docker = 'C:\Program Files\Docker\Docker\resources\bin\docker.exe' + +if (-not (Test-Path $docker)) { throw 'Docker CLI not found.' } + +try { & $docker info | Out-Null } catch { + if (Test-Path $dockerDesktop) { Start-Process -FilePath $dockerDesktop | Out-Null } +} + +for ($i = 0; $i -lt 60; $i++) { + & $docker info 2>$null | Out-Null + if ($LASTEXITCODE -eq 0) { break } + Start-Sleep -Seconds 5 +} + +& $docker start finops-hub-local +``` + +## Create a new container + +```powershell +$exportPath = 'D:\exports\local-hub' +New-Item -ItemType Directory -Force -Path $exportPath | Out-Null + +$docker = 'C:\Program Files\Docker\Docker\resources\bin\docker.exe' +& $docker run -d --name finops-hub-local --platform linux/amd64 ` + -p 8082:8080 -m 16g -e ACCEPT_EULA=Y ` + -v "$((Resolve-Path $exportPath).Path):/data/export:ro" ` + mcr.microsoft.com/azuredataexplorer/kustainer-linux:latest +``` + +Use more memory for large imports: + +```powershell +& $docker run -d --name finops-hub-local --platform linux/amd64 ` + -p 8082:8080 -m 32g -e ACCEPT_EULA=Y ` + -v "$((Resolve-Path $exportPath).Path):/data/export:ro" ` + mcr.microsoft.com/azuredataexplorer/kustainer-linux:latest +``` + +## Define the Kusto helper + +```powershell +$hub = 'http://localhost:8082/v1/rest' +function Invoke-Kusto { + param( + [string]$Database, + [string]$Command, + [ValidateSet('mgmt','query')][string]$Endpoint = 'mgmt' + ) + + Invoke-RestMethod -Uri "$hub/$Endpoint" -Method Post -ContentType 'application/json' ` + -Body (@{ db = $Database; csl = $Command } | ConvertTo-Json) +} +``` + +Wait for readiness: + +```powershell +do { + Start-Sleep -Seconds 3 + $ready = try { Invoke-Kusto NetDefaultDB '.show version'; $true } catch { $false } +} until ($ready) +``` + +## Create databases + +```powershell +foreach ($db in 'Ingestion', 'Hub') { + Invoke-Kusto NetDefaultDB ".create database $db persist (@`"/kustodata/dbs/$db/md`", @`"/kustodata/dbs/$db/data`")" | Out-Null +} +``` + +## Load released hub schema + +```powershell +$release = 'https://github.com/microsoft/finops-toolkit/releases/latest/download' +New-Item -ItemType Directory -Force -Path "$exportPath\setup" | Out-Null +Invoke-WebRequest "$release/finops-hub-fabric-setup-Ingestion.kql" -OutFile "$exportPath\setup\setup-Ingestion.kql" +Invoke-WebRequest "$release/finops-hub-fabric-setup-Hub.kql" -OutFile "$exportPath\setup\setup-Hub.kql" + +$ingestion = (Get-Content "$exportPath\setup\setup-Ingestion.kql" -Raw) -replace '\$\$rawRetentionInDays\$\$', '90' +Invoke-Kusto Ingestion $ingestion +Invoke-Kusto Hub (Get-Content "$exportPath\setup\setup-Hub.kql" -Raw) +``` + +## Load open data + +```powershell +New-Item -ItemType Directory -Force -Path "$exportPath\open-data" | Out-Null +foreach ($f in 'PricingUnits', 'Regions', 'ResourceTypes', 'Services') { + Invoke-WebRequest "$release/$f.csv" -OutFile "$exportPath\open-data\$f.csv" +} + +$openData = @' +.execute database script with (ContinueOnErrors=true) <| +.set-or-replace PricingUnits <| externaldata(x_PricingUnitDescription: string, AccountTypes: string, x_PricingBlockSize: real, PricingUnit: string)[@'/data/export/open-data/PricingUnits.csv'] with (format='csv', ignoreFirstRecord=true) | project-away AccountTypes +.set-or-replace Regions <| externaldata(ResourceLocation: string, RegionId: string, RegionName: string)[@'/data/export/open-data/Regions.csv'] with (format='csv', ignoreFirstRecord=true) +.set-or-replace ResourceTypes <| externaldata(x_ResourceType: string, SingularDisplayName: string, PluralDisplayName: string, LowerSingularDisplayName: string, LowerPluralDisplayName: string, IsPreview: bool, Description: string, IconUri: string, Links: string)[@'/data/export/open-data/ResourceTypes.csv'] with (format='csv', ignoreFirstRecord=true) | project-away Links +.set-or-replace Services <| externaldata(x_ConsumedService: string, x_ResourceType: string, ServiceName: string, ServiceCategory: string, ServiceSubcategory: string, PublisherName: string, x_PublisherCategory: string, x_Environment: string, x_ServiceModel: string)[@'/data/export/open-data/Services.csv'] with (format='csv', ignoreFirstRecord=true) +'@ +Invoke-Kusto Ingestion $openData +``` + +## Download exports from Azure Storage + +Only needed when the data is not already staged locally. + +```powershell +$account = '' +$container = 'ingestion' +$prefix = '' + +$ctx = New-AzStorageContext -StorageAccountName $account -UseConnectedAccount +Get-AzStorageBlob -Container $container -Context $ctx -Prefix $prefix | + Where-Object { $_.Name -match '(manifest\.json|\.parquet)$' } | + ForEach-Object { + Get-AzStorageBlobContent -Container $container -Context $ctx -Blob $_.Name ` + -Destination $exportPath -Force | Out-Null + } +``` + +Skip zero-byte hierarchical namespace marker blobs; download only `manifest.json` and `*.parquet`. + +## Ingest cost and price data + +```powershell +$ingestConcurrency = 2 + +$ingestJobs = Get-ChildItem $exportPath -Recurse -Filter manifest.json | ForEach-Object { + $type = if ((Get-Content $_.FullName -Raw) -match '"type"\s*:\s*"(FocusCost|PriceSheet)"') { $Matches[1] } else { return } + $target = if ($type -eq 'FocusCost') { 'Costs_raw', 'Costs_raw_mapping' } else { 'Prices_raw', 'Prices_raw_mapping' } + Get-ChildItem $_.Directory -Filter *.parquet | ForEach-Object { + [PSCustomObject]@{ + Table = $target[0] + Mapping = $target[1] + File = $_.FullName + } + } +} + +$ingestJobs | ForEach-Object -Parallel { + $hub = $using:hub + $exportPath = $using:exportPath + + function Invoke-Kusto { + param([string]$Database, [string]$Command, [ValidateSet('mgmt','query')][string]$Endpoint = 'mgmt') + Invoke-RestMethod -Uri "$hub/$Endpoint" -Method Post -ContentType 'application/json' ` + -Body (@{ db = $Database; csl = $Command } | ConvertTo-Json) + } + + $rel = [IO.Path]::GetRelativePath((Resolve-Path $exportPath), $_.File) -replace '\\', '/' + $path = "/data/export/$rel" + Invoke-Kusto Ingestion ".ingest into table $($_.Table) (h@'$path') with (format='parquet', ingestionMappingReference='$($_.Mapping)')" | Out-Null + Write-Host "ingested $(Split-Path $_.File -Leaf)" +} -ThrottleLimit $ingestConcurrency +``` + +## Large import guidance + +Validated pattern from a large import: + +- 734 Parquet files. +- About 41.9 GB of exported files. +- About 322M source rows. +- About 644M raw + final Kusto rows after transforms. +- Stable run used 32 GB emulator memory and 4 ingestion threads, with monthly batches and restarts between batches. + +Rules: + +- Start with one ingestion thread per 8 GB of emulator memory. +- If the container exits with code 137, it ran out of memory. +- For very large imports, ingest month-by-month and restart the emulator between batches. +- Keep a JSONL per-file log with status, elapsed time, row count if known, and error text. +- Use idempotency tags for safe retries when custom scripting direct `.ingest`. + +Correct tag syntax uses double quotes: + +```kusto +.ingest into table Costs_raw (h@'/data/export/.../part.parquet') + with (format='parquet', ingestionMappingReference='Costs_raw_mapping', + ingestIfNotExists='ingest-by:', tags='["ingest-by:"]') +``` + +## Validate ingestion + +```powershell +foreach ($t in 'Costs_raw', 'Costs_final_v1_2', 'Prices_raw', 'Prices_final_v1_2') { + $count = (Invoke-Kusto Ingestion "$t | count" -Endpoint query).Tables[0].Rows[0][0] + Write-Host "$t`: $count" +} +``` + +Check final vs raw rows. Costs should generally match exactly. Prices may differ after transforms depending on expansion/normalization behavior. + +## Query the Hub database + +Use `Hub` functions for analysis: + +```powershell +Invoke-Kusto Hub @' +Costs_v1_2 +| summarize EffectiveCost = round(sum(EffectiveCost), 2) by ServiceCategory +| top 10 by EffectiveCost +'@ -Endpoint query +``` + +Useful queries: + +```kusto +Costs_v1_2 +| summarize EffectiveCost=sum(EffectiveCost) by BillingPeriodStart +| order by BillingPeriodStart asc +``` + +```kusto +Costs_v1_2 +| summarize EffectiveCost=sum(EffectiveCost) by SubAccountId, ServiceName +| top 50 by EffectiveCost +``` + +```kusto +Prices_v1_2 +| summarize Prices=count() by BillingAccountId +``` + +## Connect tools + +- Azure Data Explorer web UI can connect to `http://localhost:8082`. +- The endpoint is HTTP and unauthenticated; configure clients to allow unsafe/no-auth connections where required. +- Power BI or custom tools should query the local Kusto endpoint, not read raw storage files for large datasets. + +## Stop and clean up + +Stop without deleting data: + +```powershell +$docker = 'C:\Program Files\Docker\Docker\resources\bin\docker.exe' +& $docker stop finops-hub-local +``` + +Remove the container: + +```powershell +& $docker rm -f finops-hub-local +``` + +Downloaded exports remain in `$exportPath`. Container-internal database persistence depends on how the container/database paths were created and whether volumes were mounted. + +## Troubleshooting + +| Symptom | Meaning | Action | +|---------|---------|--------| +| `docker` not recognized | Docker CLI not on PATH | Use `C:\Program Files\Docker\Docker\resources\bin\docker.exe` or refresh PATH | +| Docker pipe missing | Docker Desktop/daemon not running | Start Docker Desktop, wait for `docker info` | +| Endpoint not ready | Kusto still starting | Retry `.show version` every few seconds | +| Exit 137 | OOM kill | Reduce concurrency, increase memory, batch by month, restart between batches | +| Zero-byte marker issues | ADLS hierarchical namespace folder blobs | Download only `manifest.json` and `*.parquet` | +| Reingest uncertainty | Raw extents lack source-file lineage | Add idempotency tags and per-file logs in custom recovery scripts | +| PowerShell storage reader is slow/OOM | Rows are being materialized outside Kusto | Query Hub database directly | + +## Source + +- FinOps toolkit repo: `docs-mslearn/toolkit/hubs/run-hubs-locally.md` from `origin/features/ftklocal`, read 2026-06-19. +- FinOps toolkit repo: `docs-mslearn/toolkit/hubs/data-processing.md`, read 2026-06-19. +- FinOps toolkit repo: `docs-mslearn/toolkit/hubs/finops-hubs-overview.md`, read 2026-06-19. +- Microsoft Docs: `https://learn.microsoft.com/en-us/azure/data-explorer/kusto-emulator-overview`, read 2026-06-19. +- Microsoft Docs: `https://learn.microsoft.com/en-us/azure/data-explorer/kusto-emulator-install`, read 2026-06-19. + From 195869d4590b35afa75006fda2e22a0fec704c79 Mon Sep 17 00:00:00 2001 From: MSBrett <24294904+MSBrett@users.noreply.github.com> Date: Sun, 21 Jun 2026 12:54:50 -0700 Subject: [PATCH 31/51] feat(finops-hub): build-generated local open-data loader, Initialize-FinOpsHubLocal command, and guide Replace the rejected agent skill with a tested PowerShell command and a build-generated open-data artifact for the Run hubs locally guide. Verified end-to-end against a real Kusto emulator with real FOCUS data (7/7 integration tests, no mocks). - Add finops-hub-local-opendata.kql build artifact (.build.config) so the guide downloads it instead of hand-coding externaldata schemas. - Add Initialize-FinOpsHubLocal (downloads released setup scripts, applies retention, loads open data with verify-and-retry); no Docker management. - Order ingest prices-before-costs so the cost transform enriches from Prices_final_v1_2. - Add reference page, command list, TOC, and changelog entries; remove the ftklocal agent skill. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs-mslearn/TOC.yml | 2 + docs-mslearn/toolkit/changelog.md | 8 +- docs-mslearn/toolkit/hubs/run-hubs-locally.md | 115 ++++--- .../hubs/initialize-finopshublocal.md | 80 +++++ .../toolkit/powershell/powershell-commands.md | 3 +- .../Private/Invoke-FinOpsHubLocalCommand.ps1 | 49 +++ .../Public/Initialize-FinOpsHubLocal.ps1 | 169 ++++++++++ .../Initialize-FinOpsHubLocal.Tests.ps1 | 117 +++++++ .../Tests/Integration/Toolkit.Tests.ps1 | 1 + .../en-US/FinOpsToolkit.strings.psd1 | 3 + src/templates/agent-skills/ftklocal/SKILL.md | 119 ------- .../ftklocal/references/ftklocal-runbook.md | 298 ------------------ src/templates/finops-hub/.build.config | 6 + .../IngestionSetup_OpenDataExternal.kql | 17 + 14 files changed, 530 insertions(+), 457 deletions(-) create mode 100644 docs-mslearn/toolkit/powershell/hubs/initialize-finopshublocal.md create mode 100644 src/powershell/Private/Invoke-FinOpsHubLocalCommand.ps1 create mode 100644 src/powershell/Public/Initialize-FinOpsHubLocal.ps1 create mode 100644 src/powershell/Tests/Integration/Initialize-FinOpsHubLocal.Tests.ps1 delete mode 100644 src/templates/agent-skills/ftklocal/SKILL.md delete mode 100644 src/templates/agent-skills/ftklocal/references/ftklocal-runbook.md create mode 100644 src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/scripts/IngestionSetup_OpenDataExternal.kql diff --git a/docs-mslearn/TOC.yml b/docs-mslearn/TOC.yml index c11a155fb..caafb6f84 100644 --- a/docs-mslearn/TOC.yml +++ b/docs-mslearn/TOC.yml @@ -248,6 +248,8 @@ href: toolkit/powershell/hubs/deploy-finopshub.md - name: Initialize-FinOpsHubDeployment href: toolkit/powershell/hubs/initialize-finopshubdeployment.md + - name: Initialize-FinOpsHubLocal + href: toolkit/powershell/hubs/initialize-finopshublocal.md - name: Register-FinOpsHubProviders href: toolkit/powershell/hubs/register-finopshubproviders.md - name: Remove-FinOpsHub diff --git a/docs-mslearn/toolkit/changelog.md b/docs-mslearn/toolkit/changelog.md index 170797346..598aa13f4 100644 --- a/docs-mslearn/toolkit/changelog.md +++ b/docs-mslearn/toolkit/changelog.md @@ -3,7 +3,7 @@ title: FinOps toolkit changelog description: Review the latest features and enhancements in the FinOps toolkit, including updates to FinOps hubs, Power BI reports, and more. author: MSBrett ms.author: brettwil -ms.date: 06/19/2026 +ms.date: 06/21/2026 ms.topic: reference ms.service: finops ms.subservice: finops-toolkit @@ -49,12 +49,18 @@ _Released June 2026_ - **Added** - Added a [Run hubs locally](hubs/run-hubs-locally.md) guide to stand up a FinOps hub in a local Kusto emulator container and ingest cost data using the same KQL, transforms, and open data as a deployed hub. + - Added a build-generated `finops-hub-local-opendata.kql` release artifact that loads the open data reference tables from CSV, so the local hub guide stays in sync with published open data instead of hard-coding schemas. - **Changed** - Added a callout to the `config_RunBackfillJob` backfill option clarifying that it isn't supported on Microsoft Customer Agreement (MCA) billing accounts or billing profiles ([#2113](https://github.com/microsoft/finops-toolkit/issues/2113)). - **Fixed** - Fixed Data Factory ingestion memory pressure during emptiness filtering. - Replaced `isnotempty(strcat(x_SkuMeterId, x_SkuOfferId))` with separate `isnotempty()` checks in FinOps hub ingestion scripts to avoid temporary string allocation. +### [PowerShell module](powershell/powershell-commands.md) v15 + +- **Added** + - Added [Initialize-FinOpsHubLocal](powershell/hubs/initialize-finopshublocal.md) to set up a local FinOps hub in a running Kusto emulator with one command. + ### [Data dictionary](help/data-dictionary.md) updates - **Fixed** diff --git a/docs-mslearn/toolkit/hubs/run-hubs-locally.md b/docs-mslearn/toolkit/hubs/run-hubs-locally.md index 55621f0a4..7e7eda6cc 100644 --- a/docs-mslearn/toolkit/hubs/run-hubs-locally.md +++ b/docs-mslearn/toolkit/hubs/run-hubs-locally.md @@ -1,13 +1,13 @@ --- title: Run FinOps hubs locally description: Stand up a FinOps hub on your own hardware in a local container and ingest cost data, using the same KQL and open data as a deployed hub. -author: flanakin -ms.author: micflan -ms.date: 06/19/2026 +author: MSBrett +ms.author: brettwil +ms.date: 06/21/2026 ms.topic: how-to ms.service: finops ms.subservice: finops-toolkit -ms.reviewer: micflan +ms.reviewer: brettwil #customer intent: As a FinOps user, I want to run a FinOps hub locally so I can explore my cost data without deploying Azure resources. --- @@ -25,14 +25,11 @@ A local hub is useful when you need full hub analysis without managed daily refr - [Docker](https://docs.docker.com/get-docker/), to run the Kusto emulator container, with at least 16 GB of memory available to the container. Start Docker before you run the commands and verify `docker info` works in the same PowerShell session. - [PowerShell 7](/powershell/scripting/install/installing-powershell) — every command in this article is PowerShell. -- [Azure PowerShell](/powershell/azure/install-azure-powershell) (the `Az` modules) — only needed to download cost data from an Azure storage account. -- FOCUS cost exports as Parquet, either from a [FinOps hub storage account](configure-scopes.md), [Cost Management exports](/cost-management-billing/costs/tutorial-improved-exports), or another source staged in the same local folder structure. +- [Azure PowerShell](/powershell/azure/install-azure-powershell) (the `Az` modules), only needed to download cost data from an Azure storage account. Sign in first with `Connect-AzAccount` and select the subscription that holds your cost data — this article never runs a sign-in command. With `-UseConnectedAccount`, your identity also needs storage data-plane access, such as **Storage Blob Data Reader**, on the account or container. +- FOCUS cost exports as Parquet, available on the local machine. The emulator reads cost data from a local folder, so the files must be on disk — but you can download them from a [FinOps hub storage account](configure-scopes.md) or [Cost Management exports](/cost-management-billing/costs/tutorial-improved-exports) (shown below), or copy them from any other source into the same folder structure. For background on the emulator and its platform requirements, see the [Kusto emulator overview](/azure/data-explorer/kusto-emulator-overview) and [installation guide](/azure/data-explorer/kusto-emulator-install). -> [!NOTE] -> This article assumes you're already signed in to Azure with `Connect-AzAccount` and have selected the subscription that holds your cost data. It never runs a sign-in command. If you aren't downloading data from a storage account, you can skip Azure sign-in entirely. If you download from Azure Storage with `-UseConnectedAccount`, your signed-in identity also needs storage data-plane access, such as **Storage Blob Data Reader**, on the storage account or container. -
## Start the emulator @@ -72,6 +69,28 @@ Write-Host 'emulator ready'
+## Set up the hub + +Set up the databases, schema, and open data in one step with the [FinOps toolkit PowerShell module](../powershell/powershell-commands.md), or follow the next three sections to do it by hand. Both run the same released scripts and produce the same result. + +To use the module, stage the open data CSVs in the mounted folder first (the emulator reads them from there), then run one command: + +```powershell +New-Item -ItemType Directory -Force -Path "$exportPath/open-data" | Out-Null +foreach ($f in 'PricingUnits', 'Regions', 'ResourceTypes', 'Services') { + Invoke-WebRequest "https://github.com/microsoft/finops-toolkit/releases/latest/download/$f.csv" -OutFile "$exportPath/open-data/$f.csv" +} + +Install-Module -Name FinOpsToolkit -Scope CurrentUser +Initialize-FinOpsHubLocal -ClusterUri 'http://localhost:8082' -RawRetentionInDays 90 +``` + +`Initialize-FinOpsHubLocal` creates the `Ingestion` and `Hub` databases, applies the released setup scripts, and loads open data from `/data/export/open-data`. It doesn't manage the container or ingest cost data — start the emulator first and ingest your exports later. Add `-SkipOpenData` to skip open data. When it finishes, skip ahead to [Download your cost data](#download-your-cost-data). + +To set everything up by hand instead, continue with the steps below. + +
+ ## Create the two databases A FinOps hub uses two databases: `Ingestion` for raw and transformed data, and `Hub` for the view functions you query. @@ -110,22 +129,30 @@ Each response is a table with one row per statement. The Hub functions reference ## Load the open data -FinOps hubs enrich cost data with open-data reference tables (regions, services, resource types, and pricing units). Download the reference CSVs from the toolkit release, then load them with the same commands a deployed hub uses. +FinOps hubs enrich cost data with open-data reference tables (regions, services, resource types, and pricing units). Download the reference CSVs and the matching load script from the toolkit release, then run the script. The load script (`finops-hub-local-opendata.kql`) is generated by the toolkit build from the same open data a deployed hub uses, so its schema always matches the published CSVs. + +The emulator's first external-data read right after setup sometimes returns no rows without raising an error, so load the open data, check the tables filled, and retry if they didn't: ```powershell New-Item -ItemType Directory -Force -Path "$exportPath/open-data" | Out-Null foreach ($f in 'PricingUnits', 'Regions', 'ResourceTypes', 'Services') { Invoke-WebRequest "$release/$f.csv" -OutFile "$exportPath/open-data/$f.csv" } +Invoke-WebRequest "$release/finops-hub-local-opendata.kql" -OutFile "$exportPath/setup/load-open-data.kql" + +# Point the load script at the mounted open-data folder. +$openData = (Get-Content "$exportPath/setup/load-open-data.kql" -Raw) -replace '\$\$openDataPath\$\$', '/data/export/open-data' -$openData = @' -.execute database script with (ContinueOnErrors=true) <| -.set-or-replace PricingUnits <| externaldata(x_PricingUnitDescription: string, AccountTypes: string, x_PricingBlockSize: real, PricingUnit: string)[@'/data/export/open-data/PricingUnits.csv'] with (format='csv', ignoreFirstRecord=true) | project-away AccountTypes -.set-or-replace Regions <| externaldata(ResourceLocation: string, RegionId: string, RegionName: string)[@'/data/export/open-data/Regions.csv'] with (format='csv', ignoreFirstRecord=true) -.set-or-replace ResourceTypes <| externaldata(x_ResourceType: string, SingularDisplayName: string, PluralDisplayName: string, LowerSingularDisplayName: string, LowerPluralDisplayName: string, IsPreview: bool, Description: string, IconUri: string, Links: string)[@'/data/export/open-data/ResourceTypes.csv'] with (format='csv', ignoreFirstRecord=true) | project-away Links -.set-or-replace Services <| externaldata(x_ConsumedService: string, x_ResourceType: string, ServiceName: string, ServiceCategory: string, ServiceSubcategory: string, PublisherName: string, x_PublisherCategory: string, x_Environment: string, x_ServiceModel: string)[@'/data/export/open-data/Services.csv'] with (format='csv', ignoreFirstRecord=true) -'@ -Invoke-Kusto Ingestion $openData +# Load, verify the tables filled, and retry if the first read came back empty. +foreach ($attempt in 1..5) { + Invoke-Kusto Ingestion $openData | Out-Null + $empty = 'PricingUnits', 'Regions', 'ResourceTypes', 'Services' | Where-Object { + (Invoke-Kusto Ingestion "$_ | count" -Endpoint query).Tables[0].Rows[0][0] -eq 0 + } + if (-not $empty) { Write-Host 'open data loaded'; break } + if ($attempt -eq 5) { throw "Open data tables still empty: $($empty -join ', ')" } + Start-Sleep -Seconds 2 +} ```
@@ -136,6 +163,9 @@ Download FOCUS cost and price exports from your storage account into the export Set these to match your storage account, then run the download: +> [!NOTE] +> This download step is specific to Azure Storage. The local hub uses the same FOCUS transforms as a deployed hub, which are aligned to [FOCUS](../../focus/what-is-focus.md) and validated against Microsoft Cost Management cost data. If your FOCUS exports already live elsewhere, copy each dataset's `manifest.json` and `*.parquet` files into `$exportPath` and skip to [Ingest the data](#ingest-the-data). + ```powershell $account = '' $container = 'ingestion' # or: msexports @@ -161,39 +191,48 @@ Ingest each Parquet file into the matching raw table. As rows land, the update p For large exports, use a small amount of parallelism. Start with about one ingestion thread per 8 GB of emulator memory. For the 16 GB container below, use two threads; if you increase the container to 32 GB, four threads is a good starting point. +Ingest price sheets before cost data. The cost transform enriches rows with missing prices from `Prices_final_v1_2`, and the update policy runs as each file lands, so any prices must already be in place when costs are ingested. + ```powershell $ingestConcurrency = 2 # Group exports by manifest type, then ingest each dataset's Parquet files $ingestJobs = Get-ChildItem $exportPath -Recurse -Filter manifest.json | ForEach-Object { - $type = if ((Get-Content $_.FullName -Raw) -match '"type"\s*:\s*"(FocusCost|PriceSheet)"') { $Matches[1] } else { return } - $target = if ($type -eq 'FocusCost') { 'Costs_raw', 'Costs_raw_mapping' } else { 'Prices_raw', 'Prices_raw_mapping' } + # Default to FocusCost; treat as PriceSheet only when the manifest type or path says so. + $manifest = Get-Content $_.FullName -Raw + $isPrice = $manifest -match '"type"\s*:\s*"PriceSheet"' -or $_.FullName -match '(?i)price' + $target = if ($isPrice) { 'Prices_raw', 'Prices_raw_mapping' } else { 'Costs_raw', 'Costs_raw_mapping' } Get-ChildItem $_.Directory -Filter *.parquet | ForEach-Object { [PSCustomObject]@{ Table = $target[0] Mapping = $target[1] File = $_.FullName + # Prices first (0) then costs (1): the cost transform enriches from Prices_final_v1_2. + Phase = if ($isPrice) { 0 } else { 1 } } } } -$ingestJobs | ForEach-Object -Parallel { - $hub = $using:hub - $exportPath = $using:exportPath - function Invoke-Kusto { - param([string]$Database, [string]$Command, [ValidateSet('mgmt','query')][string]$Endpoint = 'mgmt') - Invoke-RestMethod -Uri "$hub/$Endpoint" -Method Post -ContentType 'application/json' ` - -Body (@{ db = $Database; csl = $Command } | ConvertTo-Json) - } - function Invoke-Ingest { - param([string]$Table, [string]$Mapping, [string]$File) - $rel = [IO.Path]::GetRelativePath((Resolve-Path $exportPath), $File) -replace '\\', '/' - $path = "/data/export/$rel" - Invoke-Kusto Ingestion ".ingest into table $Table (h@'$path') with (format='parquet', ingestionMappingReference='$Mapping')" | Out-Null - Write-Host " ingested $(Split-Path $File -Leaf)" - } - Invoke-Ingest $_.Table $_.Mapping $_.File -} -ThrottleLimit $ingestConcurrency +# Ingest prices, then costs. Each phase finishes before the next starts. +foreach ($phase in $ingestJobs | Group-Object Phase | Sort-Object Name) { + $phase.Group | ForEach-Object -Parallel { + $hub = $using:hub + $exportPath = $using:exportPath + function Invoke-Kusto { + param([string]$Database, [string]$Command, [ValidateSet('mgmt','query')][string]$Endpoint = 'mgmt') + Invoke-RestMethod -Uri "$hub/$Endpoint" -Method Post -ContentType 'application/json' ` + -Body (@{ db = $Database; csl = $Command } | ConvertTo-Json) + } + function Invoke-Ingest { + param([string]$Table, [string]$Mapping, [string]$File) + $rel = [IO.Path]::GetRelativePath((Resolve-Path $exportPath), $File) -replace '\\', '/' + $path = "/data/export/$rel" + Invoke-Kusto Ingestion ".ingest into table $Table (h@'$path') with (format='parquet', ingestionMappingReference='$Mapping')" | Out-Null + Write-Host " ingested $(Split-Path $File -Leaf)" + } + Invoke-Ingest $_.Table $_.Mapping $_.File + } -ThrottleLimit $ingestConcurrency +} ``` If the emulator exits with code 137, it ran out of memory. Reduce `$ingestConcurrency`, increase the container memory, or ingest one month at a time and restart the emulator between batches. For very large imports, keep a per-file log so you can retry failed files without restarting the entire load. @@ -227,7 +266,7 @@ You now have a working local FinOps hub. To explore it visually, connect the [Az ## Clean up -Stop and remove the container when you're done. Your downloaded data stays in the `export` folder. +Remove the container when you're done. Your downloaded data stays in the export folder you created (`$exportPath`), but removing the container deletes its databases — re-creating it means re-loading the schema and open data and re-ingesting. To keep the loaded data between sessions, use `docker stop finops-hub-local` instead and restart it later with `docker start finops-hub-local`. ```powershell docker rm -f finops-hub-local diff --git a/docs-mslearn/toolkit/powershell/hubs/initialize-finopshublocal.md b/docs-mslearn/toolkit/powershell/hubs/initialize-finopshublocal.md new file mode 100644 index 000000000..9ed4bc2f7 --- /dev/null +++ b/docs-mslearn/toolkit/powershell/hubs/initialize-finopshublocal.md @@ -0,0 +1,80 @@ +--- +title: Initialize-FinOpsHubLocal command +description: Set up a local FinOps hub in a running Kusto emulator using the Initialize-FinOpsHubLocal command in the FinOpsToolkit module. +author: MSBrett +ms.author: brettwil +ms.date: 06/21/2026 +ms.topic: reference +ms.service: finops +ms.subservice: finops-toolkit +ms.reviewer: brettwil +#customer intent: As a FinOps user, I want to set up a local FinOps hub with one command so I can explore my cost data without deploying Azure resources. +--- + +# Initialize-FinOpsHubLocal command + +The **Initialize-FinOpsHubLocal** command sets up a local FinOps hub in a running [Kusto emulator](/azure/data-explorer/kusto-emulator-overview). It creates the `Ingestion` and `Hub` databases, then downloads and applies the released FinOps hub setup scripts and open data load script, so the local hub uses the same KQL, transforms, and open data as a deployed hub. + +The command doesn't install Docker or manage the emulator container, and it doesn't ingest cost data. Start the emulator first, then run this command against its endpoint. For the full walkthrough, see [Run FinOps hubs locally](../../hubs/run-hubs-locally.md). + +
+ +## Syntax + +```powershell +Initialize-FinOpsHubLocal ` + [-ClusterUri ] ` + [-ReleaseUri ] ` + [-RawRetentionInDays ] ` + [-OpenDataPath ] ` + [-SkipOpenData] ` + [-Destination ] ` + [-WhatIf] +``` + +
+ +## Parameters + +| Name | Description | +| --------------------- | --------------------------------------------------------------------------------------------------------------- | +| `‑ClusterUri` | Optional. Base URI of the running Kusto emulator. Default = `http://localhost:8082`. | +| `‑ReleaseUri` | Optional. Base URI to download the setup scripts and open data load script from. Default = the latest FinOps toolkit GitHub release. Point this at a local file server or a specific release to run offline or pin a version. | +| `‑RawRetentionInDays` | Optional. Number of days to keep raw data in the `Ingestion` database. Default = `90`. | +| `‑OpenDataPath` | Optional. Path, as seen by the emulator, to the folder that holds the open data CSV files. Default = `/data/export/open-data`. | +| `‑SkipOpenData` | Optional. Skips loading the open data reference tables. Default = `false`. | +| `‑Destination` | Optional. Local folder used to download the setup scripts. Default = temp folder. | +| `‑WhatIf` | Optional. Shows what would happen if the command runs without actually running it. | + +
+ +## Examples + +### Set up a local hub + +```powershell +Initialize-FinOpsHubLocal +``` + +Sets up a local FinOps hub in the emulator at `http://localhost:8082` using the latest release. + +### Set up the schema only + +```powershell +Initialize-FinOpsHubLocal ` + -RawRetentionInDays 30 ` + -SkipOpenData +``` + +Creates the databases and applies the schema with 30-day raw retention, and skips loading open data. + +
+ +## Related content + +Related solutions: + +- [Run FinOps hubs locally](../../hubs/run-hubs-locally.md) +- [FinOps hubs](../../hubs/finops-hubs-overview.md) + +
diff --git a/docs-mslearn/toolkit/powershell/powershell-commands.md b/docs-mslearn/toolkit/powershell/powershell-commands.md index d80414150..5dbef4db0 100644 --- a/docs-mslearn/toolkit/powershell/powershell-commands.md +++ b/docs-mslearn/toolkit/powershell/powershell-commands.md @@ -3,7 +3,7 @@ title: FinOps toolkit PowerShell module description: Automate and scale your FinOps efforts using the FinOps toolkit PowerShell module, which includes commands to manage FinOps solutions. author: flanakin ms.author: micflan -ms.date: 04/01/2026 +ms.date: 06/21/2026 ms.topic: reference ms.service: finops ms.subservice: finops-toolkit @@ -64,6 +64,7 @@ The FinOps toolkit PowerShell module includes commands to manage FinOps solution - [Deploy-FinOpsHub](hubs/Deploy-FinOpsHub.md) – Deploy your first hub or update to the latest version. - [Get-FinOpsHub](hubs/Get-FinOpsHub.md) – Get details about your FinOps hub instance. - [Initialize-FinOpsHubDeployment](hubs/Initialize-FinOpsHubDeployment.md) – Initializes the deployment for FinOps hubs. +- [Initialize-FinOpsHubLocal](hubs/initialize-finopshublocal.md) – Set up a local FinOps hub in a running Kusto emulator. - [Register-FinOpsHubProviders](hubs/Register-FinOpsHubProviders.md) – Registers resource providers for FinOps hubs. - [Remove-FinOpsHub](hubs/Remove-FinOpsHub.md) – Deletes a FinOps hub instance. diff --git a/src/powershell/Private/Invoke-FinOpsHubLocalCommand.ps1 b/src/powershell/Private/Invoke-FinOpsHubLocalCommand.ps1 new file mode 100644 index 000000000..0e36c6d88 --- /dev/null +++ b/src/powershell/Private/Invoke-FinOpsHubLocalCommand.ps1 @@ -0,0 +1,49 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. + +<# + .SYNOPSIS + Sends a single command or query to a local FinOps hub (Kusto emulator) HTTP endpoint. + + .DESCRIPTION + Posts a management command or query to a running Kusto emulator over its REST API. Used by Initialize-FinOpsHubLocal to set up a local FinOps hub. This command does not manage the container; the emulator must already be running. + + .PARAMETER ClusterUri + Required. Base URI of the running Kusto emulator (for example, http://localhost:8082). + + .PARAMETER Database + Required. Name of the database to run the command against. + + .PARAMETER Command + Required. The management command (starting with '.') or query to run. + + .PARAMETER Endpoint + Optional. The REST endpoint to use: 'mgmt' for management commands (default) or 'query' for queries. +#> +function Invoke-FinOpsHubLocalCommand +{ + [CmdletBinding()] + param + ( + [Parameter(Mandatory)] + [string] + $ClusterUri, + + [Parameter(Mandatory)] + [string] + $Database, + + [Parameter(Mandatory)] + [string] + $Command, + + [Parameter()] + [ValidateSet('mgmt', 'query')] + [string] + $Endpoint = 'mgmt' + ) + + $uri = '{0}/v1/rest/{1}' -f $ClusterUri.TrimEnd('/'), $Endpoint + $body = @{ db = $Database; csl = $Command } | ConvertTo-Json -Compress + return Invoke-RestMethod -Uri $uri -Method 'Post' -ContentType 'application/json' -Body $body +} diff --git a/src/powershell/Public/Initialize-FinOpsHubLocal.ps1 b/src/powershell/Public/Initialize-FinOpsHubLocal.ps1 new file mode 100644 index 000000000..286c9d866 --- /dev/null +++ b/src/powershell/Public/Initialize-FinOpsHubLocal.ps1 @@ -0,0 +1,169 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. + +<# + .SYNOPSIS + Sets up a local FinOps hub in a running Kusto emulator. + + .DESCRIPTION + The Initialize-FinOpsHubLocal command configures a local FinOps hub in a running Kusto emulator. It creates the Ingestion and Hub databases, then downloads and applies the released FinOps hub setup scripts and open data load script so the local hub uses the same KQL, transforms, and open data as a deployed hub. + + This command does not install Docker or manage the emulator container. Start the emulator first, then run this command against its endpoint. It does not ingest cost data; stage and ingest your own exports after setup. + + .PARAMETER ClusterUri + Optional. Base URI of the running Kusto emulator. Default = http://localhost:8082. + + .PARAMETER ReleaseUri + Optional. Base URI to download the setup scripts and open data load script from. Default = the latest FinOps toolkit GitHub release. Point this at a local file server or a specific release to run offline or pin a version. + + .PARAMETER RawRetentionInDays + Optional. Number of days to keep raw data in the Ingestion database. Default = 90. + + .PARAMETER OpenDataPath + Optional. Path, as seen by the emulator, to the folder that holds the open data CSV files. Default = /data/export/open-data. + + .PARAMETER SkipOpenData + Optional. Skips loading the open data reference tables. Default = false. + + .PARAMETER Destination + Optional. Local folder used to download the setup scripts. Default = temp folder. + + .EXAMPLE + Initialize-FinOpsHubLocal + + Sets up a local FinOps hub in the emulator at http://localhost:8082 using the latest release. + + .EXAMPLE + Initialize-FinOpsHubLocal -ClusterUri 'http://localhost:8082' -RawRetentionInDays 30 -SkipOpenData + + Sets up the databases and schema with 30-day raw retention and skips loading open data. + + .LINK + https://aka.ms/ftk/Initialize-FinOpsHubLocal +#> +function Initialize-FinOpsHubLocal +{ + [CmdletBinding(SupportsShouldProcess)] + param + ( + [Parameter()] + [string] + $ClusterUri = 'http://localhost:8082', + + [Parameter()] + [string] + $ReleaseUri = 'https://github.com/microsoft/finops-toolkit/releases/latest/download', + + [Parameter()] + [ValidateRange(1, [int]::MaxValue)] + [int] + $RawRetentionInDays = 90, + + [Parameter()] + [string] + $OpenDataPath = '/data/export/open-data', + + [Parameter()] + [switch] + $SkipOpenData, + + [Parameter()] + [string] + $Destination = [System.IO.Path]::GetTempPath() + ) + + $progress = $ProgressPreference + $ProgressPreference = 'SilentlyContinue' + + try + { + # Verify the emulator is reachable. This command does not start the container. + try + { + $null = Invoke-FinOpsHubLocalCommand -ClusterUri $ClusterUri -Database 'NetDefaultDB' -Command '.show version' + } + catch + { + throw ($script:LocalizedData.HubLocal_Initialize_NotReachable -f $ClusterUri) + } + + # Download the required release assets by name from the release URI. + $assetNames = @('finops-hub-fabric-setup-Ingestion.kql', 'finops-hub-fabric-setup-Hub.kql') + if (-not $SkipOpenData) + { + $assetNames += 'finops-hub-local-opendata.kql' + } + + New-Directory -Path $Destination + $scripts = @{} + foreach ($assetName in $assetNames) + { + $filePath = Join-Path -Path $Destination -ChildPath $assetName + $null = Invoke-WebRequest -Uri "$($ReleaseUri.TrimEnd('/'))/$assetName" -OutFile $filePath -Verbose:$false + $scripts[$assetName] = Get-Content -Path $filePath -Raw + } + + # Create the Ingestion and Hub databases + foreach ($database in @('Ingestion', 'Hub')) + { + $createCommand = '.create database {0} persist (@"/kustodata/dbs/{0}/md", @"/kustodata/dbs/{0}/data")' -f $database + if ($PSCmdlet.ShouldProcess($database, 'Create database')) + { + $null = Invoke-FinOpsHubLocalCommand -ClusterUri $ClusterUri -Database 'NetDefaultDB' -Command $createCommand + } + } + + # Apply the Ingestion setup with the requested raw retention + $ingestionScript = $scripts['finops-hub-fabric-setup-Ingestion.kql'] -replace '\$\$rawRetentionInDays\$\$', $RawRetentionInDays + if ($PSCmdlet.ShouldProcess('Ingestion', 'Apply setup script')) + { + $null = Invoke-FinOpsHubLocalCommand -ClusterUri $ClusterUri -Database 'Ingestion' -Command $ingestionScript + } + + # Apply the Hub setup + if ($PSCmdlet.ShouldProcess('Hub', 'Apply setup script')) + { + $null = Invoke-FinOpsHubLocalCommand -ClusterUri $ClusterUri -Database 'Hub' -Command $scripts['finops-hub-fabric-setup-Hub.kql'] + } + + # Load the open data reference tables + if (-not $SkipOpenData) + { + $openDataScript = $scripts['finops-hub-local-opendata.kql'] -replace '\$\$openDataPath\$\$', $OpenDataPath + if ($PSCmdlet.ShouldProcess('Ingestion', 'Load open data')) + { + # The emulator's first external data read after setup can return no rows without + # raising an error. Load, verify the tables filled, and retry before giving up. + $openDataTables = @('PricingUnits', 'Regions', 'ResourceTypes', 'Services') + $maxAttempts = 5 + for ($attempt = 1; $attempt -le $maxAttempts; $attempt++) + { + $null = Invoke-FinOpsHubLocalCommand -ClusterUri $ClusterUri -Database 'Ingestion' -Command $openDataScript + $empty = @($openDataTables | Where-Object { + [int64](Invoke-FinOpsHubLocalCommand -ClusterUri $ClusterUri -Database 'Ingestion' -Command "$_ | count" -Endpoint 'query').Tables[0].Rows[0][0] -eq 0 + }) + + if ($empty.Count -eq 0) + { + break + } + + if ($attempt -eq $maxAttempts) + { + throw ($script:LocalizedData.HubLocal_Initialize_OpenDataEmpty -f $maxAttempts, ($empty -join ', ')) + } + + Start-Sleep -Seconds 2 + } + } + } + } + catch + { + throw $_.Exception.Message + } + finally + { + $ProgressPreference = $progress + } +} diff --git a/src/powershell/Tests/Integration/Initialize-FinOpsHubLocal.Tests.ps1 b/src/powershell/Tests/Integration/Initialize-FinOpsHubLocal.Tests.ps1 new file mode 100644 index 000000000..92b4f66df --- /dev/null +++ b/src/powershell/Tests/Integration/Initialize-FinOpsHubLocal.Tests.ps1 @@ -0,0 +1,117 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. + +# Integration tests for Initialize-FinOpsHubLocal. These run the command against a real +# running Kusto emulator and download real release artifacts over HTTP -- no mocks. +# +# They are skipped unless the following environment variables point at live infrastructure: +# FTK_LOCAL_EMULATOR_URI Base URI of a running Kusto emulator (for example, http://localhost:8087). +# FTK_LOCAL_RELEASE_URI Base URI serving the release artifacts (for example, a local file server). +# FTK_LOCAL_EXPORT_PATH Host path mounted into the emulator at /data/export, holding FOCUS exports +# and an open-data subfolder. + +if (-not (Get-Module -Name 'FinOpsToolkit')) +{ + $rootDirectory = ((Get-Item -Path $PSScriptRoot).Parent.Parent).FullName + $modulePath = (Get-ChildItem -Path $rootDirectory -Include 'FinOpsToolKit.psm1' -Recurse).FullName + Import-Module -FullyQualifiedName $modulePath +} + +$script:emulatorUri = $env:FTK_LOCAL_EMULATOR_URI +$script:releaseUri = $env:FTK_LOCAL_RELEASE_URI +$script:exportPath = $env:FTK_LOCAL_EXPORT_PATH +$script:ready = [bool]$script:emulatorUri -and [bool]$script:releaseUri -and [bool]$script:exportPath + +Describe 'Initialize-FinOpsHubLocal' { + BeforeAll { + $script:hub = $env:FTK_LOCAL_EMULATOR_URI + $script:release = $env:FTK_LOCAL_RELEASE_URI + $script:exports = $env:FTK_LOCAL_EXPORT_PATH + # Recompute readiness from the environment: discovery-scope variables do not flow into BeforeAll. + $script:ready = [bool]$script:hub -and [bool]$script:release -and [bool]$script:exports + + function Invoke-Q + { + param([string]$Database, [string]$Command, [ValidateSet('mgmt', 'query')][string]$Endpoint = 'query') + Invoke-RestMethod -Uri "$script:hub/v1/rest/$Endpoint" -Method 'Post' -ContentType 'application/json' -Body (@{ db = $Database; csl = $Command } | ConvertTo-Json) + } + + function Get-Count + { + param([string]$Database, [string]$Table) + [int64](Invoke-Q -Database $Database -Command "$Table | count" -Endpoint 'query').Tables[0].Rows[0][0] + } + + if ($script:ready) + { + # Real setup: download real artifacts over HTTP, create databases, apply schema, load open data. + Initialize-FinOpsHubLocal -ClusterUri $script:hub -ReleaseUri $script:release -RawRetentionInDays 90 -OpenDataPath '/data/export/open-data' -Destination (Join-Path $TestDrive 'setup') + + # Real ordered ingest: prices first, then costs (the documented pattern). + $jobs = Get-ChildItem $script:exports -Recurse -Filter manifest.json | ForEach-Object { + $manifest = Get-Content $_.FullName -Raw + $isPrice = $manifest -match '"type"\s*:\s*"PriceSheet"' -or $_.FullName -match '(?i)price' + $tableInfo = if ($isPrice) { 'Prices_raw', 'Prices_raw_mapping' } else { 'Costs_raw', 'Costs_raw_mapping' } + Get-ChildItem $_.Directory -Filter *.parquet | ForEach-Object { + [PSCustomObject]@{ Table = $tableInfo[0]; Mapping = $tableInfo[1]; File = $_.FullName; Phase = [int](-not $isPrice) } + } + } + + $script:pricesBeforeCostsPhase = $null + foreach ($phase in $jobs | Group-Object Phase | Sort-Object Name) + { + if ($phase.Name -eq '1') { $script:pricesBeforeCostsPhase = Get-Count -Database 'Ingestion' -Table 'Prices_raw' } + foreach ($job in $phase.Group) + { + $rel = [System.IO.Path]::GetRelativePath((Resolve-Path $script:exports), $job.File) -replace '\\', '/' + Invoke-Q -Database 'Ingestion' -Endpoint 'mgmt' -Command ".ingest into table $($job.Table) (h@'/data/export/$rel') with (format='parquet', ingestionMappingReference='$($job.Mapping)')" | Out-Null + } + } + } + } + + It 'creates the Ingestion and Hub databases' -Skip:(-not $script:ready) { + $databases = (Invoke-Q -Database 'NetDefaultDB' -Command '.show databases' -Endpoint 'mgmt').Tables[0].Rows | ForEach-Object { $_[0] } + $databases | Should -Contain 'Ingestion' + $databases | Should -Contain 'Hub' + } + + It 'loads all four open data tables with rows' -Skip:(-not $script:ready) { + Get-Count -Database 'Ingestion' -Table 'PricingUnits' | Should -BeGreaterThan 0 + Get-Count -Database 'Ingestion' -Table 'Regions' | Should -BeGreaterThan 0 + Get-Count -Database 'Ingestion' -Table 'ResourceTypes' | Should -BeGreaterThan 0 + Get-Count -Database 'Ingestion' -Table 'Services' | Should -BeGreaterThan 0 + } + + It 'applies the requested raw retention to Costs_raw' -Skip:(-not $script:ready) { + $policy = (Invoke-Q -Database 'Ingestion' -Command '.show table Costs_raw policy retention' -Endpoint 'mgmt').Tables[0].Rows[0][2] | ConvertFrom-Json + $policy.SoftDeletePeriod | Should -Be '90.00:00:00' + } + + It 'ingests prices before costs' -Skip:(-not $script:ready) { + # Captured just before the costs phase started: prices were already present. + $script:pricesBeforeCostsPhase | Should -BeGreaterThan 0 + } + + It 'fills the final cost and price tables through the update policy' -Skip:(-not $script:ready) { + $costsRaw = Get-Count -Database 'Ingestion' -Table 'Costs_raw' + $costsFinal = Get-Count -Database 'Ingestion' -Table 'Costs_final_v1_2' + $costsRaw | Should -BeGreaterThan 0 + $costsFinal | Should -Be $costsRaw + } + + It 'enriches costs with open data and prices' -Skip:(-not $script:ready) { + $total = Get-Count -Database 'Hub' -Table 'Costs_v1_2' + $enriched = Get-Count -Database 'Hub' -Table 'Costs_v1_2 | where isnotempty(ServiceCategory)' + $priced = Get-Count -Database 'Hub' -Table 'Costs_v1_2 | where isnotempty(ListUnitPrice) and ListUnitPrice > 0' + $total | Should -BeGreaterThan 0 + $enriched | Should -Be $total + $priced | Should -BeGreaterThan 0 + } + + It 'throws a clear error when the emulator is unreachable' -Skip:(-not $script:ready) { + # Real dead port -- no mock. + { Initialize-FinOpsHubLocal -ClusterUri 'http://localhost:1' -ReleaseUri $script:release -Destination (Join-Path $TestDrive 'unreachable') } | + Should -Throw '*Could not reach the Kusto emulator*' + } +} diff --git a/src/powershell/Tests/Integration/Toolkit.Tests.ps1 b/src/powershell/Tests/Integration/Toolkit.Tests.ps1 index c154dc5c6..6a43b02dd 100644 --- a/src/powershell/Tests/Integration/Toolkit.Tests.ps1 +++ b/src/powershell/Tests/Integration/Toolkit.Tests.ps1 @@ -49,6 +49,7 @@ Describe 'Get-FinOpsToolkitVersion' { CheckFile "finops-hub-dashboard.json" '0.8' $null CheckFile "finops-hub-fabric-setup-Hub.kql" '0.10' $null CheckFile "finops-hub-fabric-setup-Ingestion.kql" '0.10' $null + CheckFile "finops-hub-local-opendata.kql" '15.0' $null CheckFile "finops-hub-v$verStr.zip" $null $null CheckFile "finops-workbooks-v$verStr.zip" '0.6' $null CheckFile "governance-workbook-v$verStr.zip" '0.1' '0.5' diff --git a/src/powershell/en-US/FinOpsToolkit.strings.psd1 b/src/powershell/en-US/FinOpsToolkit.strings.psd1 index bf003b56b..fa4c05a8c 100644 --- a/src/powershell/en-US/FinOpsToolkit.strings.psd1 +++ b/src/powershell/en-US/FinOpsToolkit.strings.psd1 @@ -16,6 +16,9 @@ ConvertFrom-StringData -StringData @' Hub_Remove_Failed = FinOps hub could not be deleted. {0}. Hub_Remove_NotFound = FinOps hub '{0}' not found. + HubLocal_Initialize_NotReachable = Could not reach the Kusto emulator at '{0}'. Start the local hub container before running this command. See https://aka.ms/finops/hubs/local. + HubLocal_Initialize_OpenDataEmpty = Open data tables were still empty after {0} attempts: {1}. The emulator's first external data read after setup can return no rows; rerun the command to retry. + HubProviders_Register_AlreadyRegistered = Resource provider {0} is already registered. HubProviders_Register_Register = Registering resource provider {0}. HubProviders_Register_RegisterError = Error registering resource provider: {0}. diff --git a/src/templates/agent-skills/ftklocal/SKILL.md b/src/templates/agent-skills/ftklocal/SKILL.md deleted file mode 100644 index 01ce91350..000000000 --- a/src/templates/agent-skills/ftklocal/SKILL.md +++ /dev/null @@ -1,119 +0,0 @@ ---- -name: ftklocal -description: Deploy, operate, recover, and query a local FinOps hub on your own hardware with the Kusto emulator. Use when the user mentions ftklocal, local FinOps hub, running FinOps hubs locally, Kusto emulator hub, offline/on-prem/other-cloud FinOps analysis, local Hub database queries, large exported FOCUS datasets, or validating FinOps toolkit data without Azure-hosted ADX/Fabric. -license: MIT -compatibility: Requires Docker and PowerShell 7. Optional Azure PowerShell is only needed to download exports from Azure Storage. -metadata: - author: microsoft - version: "1.0" ---- - -# ftklocal - -Use this skill to run a FinOps hub analytics layer locally on your own hardware. The local hub uses the Azure Data Explorer Kusto emulator container and the same released FinOps hub setup KQL, schemas, transforms, update policies, open data, and Hub view functions used by deployed hubs. - -## What ftklocal is for - -- **Offline/customer delivery analysis**: Analyze full exported customer datasets without managed daily refresh. -- **Own-hardware/on-prem/other-cloud operation**: Run where Docker and exported data are available, including disconnected environments. -- **Agent UAT and troubleshooting**: Validate hub transforms, large ingestions, and KQL behavior without deploying Azure-hosted ADX or Fabric. -- **Scale shape**: Keep data in Kusto and query summarized results. Do not load tens of GB or hundreds of millions of rows into PowerShell objects. - -Not production: the Kusto emulator has no authentication, access control, encrypted connections, managed ingestion, or production support. It is provided as-is and is not intended for production workloads or benchmark claims. - -## Core model - -- Container endpoint: `http://localhost:8082/v1/rest` -- Databases: - - `Ingestion`: raw tables, final tables, transforms, update policies, open data - - `Hub`: user-facing view functions such as `Costs_v1_2` -- Data flow: - 1. Stage `manifest.json` and `*.parquet` exports under a local export folder. - 2. Mount that folder read-only into the container at `/data/export`. - 3. Load released setup KQL into `Ingestion` and `Hub`. - 4. Load open data CSVs. - 5. `.ingest` Parquet files into `Costs_raw` / `Prices_raw`; update policies fill final tables. - 6. Query the `Hub` database, not local Parquet files, for analysis. - -## Task routing - -Load [references/ftklocal-runbook.md](references/ftklocal-runbook.md) for deployment, recovery, ingestion, troubleshooting, or any non-trivial query/operations task. - -## Quick operations - -### Start existing local hub - -```powershell -$docker = 'C:\Program Files\Docker\Docker\resources\bin\docker.exe' -& $docker start finops-hub-local -``` - -Then wait for: - -```powershell -$hub = 'http://localhost:8082/v1/rest' -function Invoke-Kusto { - param([string]$Database, [string]$Command, [ValidateSet('mgmt','query')][string]$Endpoint = 'mgmt') - Invoke-RestMethod -Uri "$hub/$Endpoint" -Method Post -ContentType 'application/json' ` - -Body (@{ db = $Database; csl = $Command } | ConvertTo-Json) -} -Invoke-Kusto NetDefaultDB '.show version' -``` - -### Stop local hub - -```powershell -$docker = 'C:\Program Files\Docker\Docker\resources\bin\docker.exe' -& $docker stop finops-hub-local -``` - -### Check row counts - -```powershell -foreach ($t in 'Costs_raw', 'Costs_final_v1_2', 'Prices_raw', 'Prices_final_v1_2') { - $count = (Invoke-Kusto Ingestion "$t | count" -Endpoint query).Tables[0].Rows[0][0] - Write-Host "$t`: $count" -} -``` - -### Query analysis data - -```powershell -$result = Invoke-Kusto Hub @' -Costs_v1_2 -| summarize EffectiveCost = round(sum(EffectiveCost), 2) by ServiceCategory -| top 10 by EffectiveCost -'@ -Endpoint query -$result.Tables[0].Rows -``` - -## Operational rules for agents - -1. Prefer Kusto queries over reading Parquet/CSV into PowerShell objects. -2. Use the `Hub` database for analysis; use `Ingestion` only for row counts, raw/final validation, and troubleshooting. -3. For large imports, start around one ingest thread per 8 GB of emulator memory. -4. Exit code 137 means the emulator was OOM-killed: lower concurrency, increase memory, ingest one month at a time, and restart between batches. -5. For very large imports, keep per-file logs and idempotency tags so failed files can be retried. -6. Do not call this production, secure, or managed. It is a local analysis and validation tier. -7. Do not use local storage-export row materialization as the scalable analysis path. - -## Learn more - -| Topic | How to find | -|-------|-------------| -| FinOps hubs overview | `microsoft_docs_search(query="FinOps hubs overview Azure Data Explorer Fabric")` | -| Kusto emulator overview | `microsoft_docs_fetch(url="https://learn.microsoft.com/en-us/azure/data-explorer/kusto-emulator-overview")` | -| Kusto emulator install | `microsoft_docs_fetch(url="https://learn.microsoft.com/en-us/azure/data-explorer/kusto-emulator-install")` | -| Kusto query language | `microsoft_docs_search(query="Azure Data Explorer KQL summarize top project")` | - -## CLI alternative - -If the Learn MCP server is not available, use the `mslearn` CLI instead: - -| MCP tool | CLI command | -|----------|-------------| -| `microsoft_docs_search(query: "...")` | `mslearn search "..."` | -| `microsoft_docs_fetch(url: "...")` | `mslearn fetch "..."` | - -Run directly with `npx @microsoft/learn-cli ` or install globally with `npm install -g @microsoft/learn-cli`. - diff --git a/src/templates/agent-skills/ftklocal/references/ftklocal-runbook.md b/src/templates/agent-skills/ftklocal/references/ftklocal-runbook.md deleted file mode 100644 index de0243275..000000000 --- a/src/templates/agent-skills/ftklocal/references/ftklocal-runbook.md +++ /dev/null @@ -1,298 +0,0 @@ -# ftklocal runbook - -This runbook describes how to deploy, manage, ingest, recover, and query a local FinOps hub with the Azure Data Explorer Kusto emulator. - -## Positioning - -ftklocal is the own-hardware path for FinOps hubs. It is useful for customer deliveries, consulting, offline analysis, on-premises/other-cloud hosts, disconnected environments, and agent UAT where managed daily refresh is not required but full-fidelity hub analysis is. - -Use Kusto as the execution engine. Avoid designs that download storage exports and materialize all rows as PowerShell objects; that does not scale to tens of GB or hundreds of millions of rows. - -## Requirements - -- Docker Desktop or Docker engine capable of running Linux containers. -- PowerShell 7. -- At least 16 GB container memory for small/medium imports; for large imports use roughly 8 GB per ingestion thread. -- Exported FOCUS data as Parquet with `manifest.json` files, staged locally or downloaded from a FinOps hub / Cost Management export storage account. -- Optional Az PowerShell modules if downloading from Azure Storage. - -Official Kusto emulator constraints: - -- Runs locally as a Linux Docker container. -- Exposes HTTP only. -- No Microsoft Entra authentication, access control, encrypted connections, managed ingestion, streaming ingestion, or production support. -- No Azure service provisioning or internet connectivity is required after image/data/artifacts are available locally. -- Not intended for production or benchmark claims. - -## Start Docker and existing container - -```powershell -$dockerDesktop = 'C:\Program Files\Docker\Docker\Docker Desktop.exe' -$docker = 'C:\Program Files\Docker\Docker\resources\bin\docker.exe' - -if (-not (Test-Path $docker)) { throw 'Docker CLI not found.' } - -try { & $docker info | Out-Null } catch { - if (Test-Path $dockerDesktop) { Start-Process -FilePath $dockerDesktop | Out-Null } -} - -for ($i = 0; $i -lt 60; $i++) { - & $docker info 2>$null | Out-Null - if ($LASTEXITCODE -eq 0) { break } - Start-Sleep -Seconds 5 -} - -& $docker start finops-hub-local -``` - -## Create a new container - -```powershell -$exportPath = 'D:\exports\local-hub' -New-Item -ItemType Directory -Force -Path $exportPath | Out-Null - -$docker = 'C:\Program Files\Docker\Docker\resources\bin\docker.exe' -& $docker run -d --name finops-hub-local --platform linux/amd64 ` - -p 8082:8080 -m 16g -e ACCEPT_EULA=Y ` - -v "$((Resolve-Path $exportPath).Path):/data/export:ro" ` - mcr.microsoft.com/azuredataexplorer/kustainer-linux:latest -``` - -Use more memory for large imports: - -```powershell -& $docker run -d --name finops-hub-local --platform linux/amd64 ` - -p 8082:8080 -m 32g -e ACCEPT_EULA=Y ` - -v "$((Resolve-Path $exportPath).Path):/data/export:ro" ` - mcr.microsoft.com/azuredataexplorer/kustainer-linux:latest -``` - -## Define the Kusto helper - -```powershell -$hub = 'http://localhost:8082/v1/rest' -function Invoke-Kusto { - param( - [string]$Database, - [string]$Command, - [ValidateSet('mgmt','query')][string]$Endpoint = 'mgmt' - ) - - Invoke-RestMethod -Uri "$hub/$Endpoint" -Method Post -ContentType 'application/json' ` - -Body (@{ db = $Database; csl = $Command } | ConvertTo-Json) -} -``` - -Wait for readiness: - -```powershell -do { - Start-Sleep -Seconds 3 - $ready = try { Invoke-Kusto NetDefaultDB '.show version'; $true } catch { $false } -} until ($ready) -``` - -## Create databases - -```powershell -foreach ($db in 'Ingestion', 'Hub') { - Invoke-Kusto NetDefaultDB ".create database $db persist (@`"/kustodata/dbs/$db/md`", @`"/kustodata/dbs/$db/data`")" | Out-Null -} -``` - -## Load released hub schema - -```powershell -$release = 'https://github.com/microsoft/finops-toolkit/releases/latest/download' -New-Item -ItemType Directory -Force -Path "$exportPath\setup" | Out-Null -Invoke-WebRequest "$release/finops-hub-fabric-setup-Ingestion.kql" -OutFile "$exportPath\setup\setup-Ingestion.kql" -Invoke-WebRequest "$release/finops-hub-fabric-setup-Hub.kql" -OutFile "$exportPath\setup\setup-Hub.kql" - -$ingestion = (Get-Content "$exportPath\setup\setup-Ingestion.kql" -Raw) -replace '\$\$rawRetentionInDays\$\$', '90' -Invoke-Kusto Ingestion $ingestion -Invoke-Kusto Hub (Get-Content "$exportPath\setup\setup-Hub.kql" -Raw) -``` - -## Load open data - -```powershell -New-Item -ItemType Directory -Force -Path "$exportPath\open-data" | Out-Null -foreach ($f in 'PricingUnits', 'Regions', 'ResourceTypes', 'Services') { - Invoke-WebRequest "$release/$f.csv" -OutFile "$exportPath\open-data\$f.csv" -} - -$openData = @' -.execute database script with (ContinueOnErrors=true) <| -.set-or-replace PricingUnits <| externaldata(x_PricingUnitDescription: string, AccountTypes: string, x_PricingBlockSize: real, PricingUnit: string)[@'/data/export/open-data/PricingUnits.csv'] with (format='csv', ignoreFirstRecord=true) | project-away AccountTypes -.set-or-replace Regions <| externaldata(ResourceLocation: string, RegionId: string, RegionName: string)[@'/data/export/open-data/Regions.csv'] with (format='csv', ignoreFirstRecord=true) -.set-or-replace ResourceTypes <| externaldata(x_ResourceType: string, SingularDisplayName: string, PluralDisplayName: string, LowerSingularDisplayName: string, LowerPluralDisplayName: string, IsPreview: bool, Description: string, IconUri: string, Links: string)[@'/data/export/open-data/ResourceTypes.csv'] with (format='csv', ignoreFirstRecord=true) | project-away Links -.set-or-replace Services <| externaldata(x_ConsumedService: string, x_ResourceType: string, ServiceName: string, ServiceCategory: string, ServiceSubcategory: string, PublisherName: string, x_PublisherCategory: string, x_Environment: string, x_ServiceModel: string)[@'/data/export/open-data/Services.csv'] with (format='csv', ignoreFirstRecord=true) -'@ -Invoke-Kusto Ingestion $openData -``` - -## Download exports from Azure Storage - -Only needed when the data is not already staged locally. - -```powershell -$account = '' -$container = 'ingestion' -$prefix = '' - -$ctx = New-AzStorageContext -StorageAccountName $account -UseConnectedAccount -Get-AzStorageBlob -Container $container -Context $ctx -Prefix $prefix | - Where-Object { $_.Name -match '(manifest\.json|\.parquet)$' } | - ForEach-Object { - Get-AzStorageBlobContent -Container $container -Context $ctx -Blob $_.Name ` - -Destination $exportPath -Force | Out-Null - } -``` - -Skip zero-byte hierarchical namespace marker blobs; download only `manifest.json` and `*.parquet`. - -## Ingest cost and price data - -```powershell -$ingestConcurrency = 2 - -$ingestJobs = Get-ChildItem $exportPath -Recurse -Filter manifest.json | ForEach-Object { - $type = if ((Get-Content $_.FullName -Raw) -match '"type"\s*:\s*"(FocusCost|PriceSheet)"') { $Matches[1] } else { return } - $target = if ($type -eq 'FocusCost') { 'Costs_raw', 'Costs_raw_mapping' } else { 'Prices_raw', 'Prices_raw_mapping' } - Get-ChildItem $_.Directory -Filter *.parquet | ForEach-Object { - [PSCustomObject]@{ - Table = $target[0] - Mapping = $target[1] - File = $_.FullName - } - } -} - -$ingestJobs | ForEach-Object -Parallel { - $hub = $using:hub - $exportPath = $using:exportPath - - function Invoke-Kusto { - param([string]$Database, [string]$Command, [ValidateSet('mgmt','query')][string]$Endpoint = 'mgmt') - Invoke-RestMethod -Uri "$hub/$Endpoint" -Method Post -ContentType 'application/json' ` - -Body (@{ db = $Database; csl = $Command } | ConvertTo-Json) - } - - $rel = [IO.Path]::GetRelativePath((Resolve-Path $exportPath), $_.File) -replace '\\', '/' - $path = "/data/export/$rel" - Invoke-Kusto Ingestion ".ingest into table $($_.Table) (h@'$path') with (format='parquet', ingestionMappingReference='$($_.Mapping)')" | Out-Null - Write-Host "ingested $(Split-Path $_.File -Leaf)" -} -ThrottleLimit $ingestConcurrency -``` - -## Large import guidance - -Validated pattern from a large import: - -- 734 Parquet files. -- About 41.9 GB of exported files. -- About 322M source rows. -- About 644M raw + final Kusto rows after transforms. -- Stable run used 32 GB emulator memory and 4 ingestion threads, with monthly batches and restarts between batches. - -Rules: - -- Start with one ingestion thread per 8 GB of emulator memory. -- If the container exits with code 137, it ran out of memory. -- For very large imports, ingest month-by-month and restart the emulator between batches. -- Keep a JSONL per-file log with status, elapsed time, row count if known, and error text. -- Use idempotency tags for safe retries when custom scripting direct `.ingest`. - -Correct tag syntax uses double quotes: - -```kusto -.ingest into table Costs_raw (h@'/data/export/.../part.parquet') - with (format='parquet', ingestionMappingReference='Costs_raw_mapping', - ingestIfNotExists='ingest-by:', tags='["ingest-by:"]') -``` - -## Validate ingestion - -```powershell -foreach ($t in 'Costs_raw', 'Costs_final_v1_2', 'Prices_raw', 'Prices_final_v1_2') { - $count = (Invoke-Kusto Ingestion "$t | count" -Endpoint query).Tables[0].Rows[0][0] - Write-Host "$t`: $count" -} -``` - -Check final vs raw rows. Costs should generally match exactly. Prices may differ after transforms depending on expansion/normalization behavior. - -## Query the Hub database - -Use `Hub` functions for analysis: - -```powershell -Invoke-Kusto Hub @' -Costs_v1_2 -| summarize EffectiveCost = round(sum(EffectiveCost), 2) by ServiceCategory -| top 10 by EffectiveCost -'@ -Endpoint query -``` - -Useful queries: - -```kusto -Costs_v1_2 -| summarize EffectiveCost=sum(EffectiveCost) by BillingPeriodStart -| order by BillingPeriodStart asc -``` - -```kusto -Costs_v1_2 -| summarize EffectiveCost=sum(EffectiveCost) by SubAccountId, ServiceName -| top 50 by EffectiveCost -``` - -```kusto -Prices_v1_2 -| summarize Prices=count() by BillingAccountId -``` - -## Connect tools - -- Azure Data Explorer web UI can connect to `http://localhost:8082`. -- The endpoint is HTTP and unauthenticated; configure clients to allow unsafe/no-auth connections where required. -- Power BI or custom tools should query the local Kusto endpoint, not read raw storage files for large datasets. - -## Stop and clean up - -Stop without deleting data: - -```powershell -$docker = 'C:\Program Files\Docker\Docker\resources\bin\docker.exe' -& $docker stop finops-hub-local -``` - -Remove the container: - -```powershell -& $docker rm -f finops-hub-local -``` - -Downloaded exports remain in `$exportPath`. Container-internal database persistence depends on how the container/database paths were created and whether volumes were mounted. - -## Troubleshooting - -| Symptom | Meaning | Action | -|---------|---------|--------| -| `docker` not recognized | Docker CLI not on PATH | Use `C:\Program Files\Docker\Docker\resources\bin\docker.exe` or refresh PATH | -| Docker pipe missing | Docker Desktop/daemon not running | Start Docker Desktop, wait for `docker info` | -| Endpoint not ready | Kusto still starting | Retry `.show version` every few seconds | -| Exit 137 | OOM kill | Reduce concurrency, increase memory, batch by month, restart between batches | -| Zero-byte marker issues | ADLS hierarchical namespace folder blobs | Download only `manifest.json` and `*.parquet` | -| Reingest uncertainty | Raw extents lack source-file lineage | Add idempotency tags and per-file logs in custom recovery scripts | -| PowerShell storage reader is slow/OOM | Rows are being materialized outside Kusto | Query Hub database directly | - -## Source - -- FinOps toolkit repo: `docs-mslearn/toolkit/hubs/run-hubs-locally.md` from `origin/features/ftklocal`, read 2026-06-19. -- FinOps toolkit repo: `docs-mslearn/toolkit/hubs/data-processing.md`, read 2026-06-19. -- FinOps toolkit repo: `docs-mslearn/toolkit/hubs/finops-hubs-overview.md`, read 2026-06-19. -- Microsoft Docs: `https://learn.microsoft.com/en-us/azure/data-explorer/kusto-emulator-overview`, read 2026-06-19. -- Microsoft Docs: `https://learn.microsoft.com/en-us/azure/data-explorer/kusto-emulator-install`, read 2026-06-19. - diff --git a/src/templates/finops-hub/.build.config b/src/templates/finops-hub/.build.config index 2dd495e60..3b4f76c15 100644 --- a/src/templates/finops-hub/.build.config +++ b/src/templates/finops-hub/.build.config @@ -45,6 +45,12 @@ "modules/Microsoft.FinOpsHubs/Analytics/scripts/HubSetup_v1_2.kql", "modules/Microsoft.FinOpsHubs/Analytics/scripts/HubSetup_Latest.kql" ] + }, + { + "name": "finops-hub-local-opendata.kql", + "files": [ + "modules/Microsoft.FinOpsHubs/Analytics/scripts/IngestionSetup_OpenDataExternal.kql" + ] } ] } diff --git a/src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/scripts/IngestionSetup_OpenDataExternal.kql b/src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/scripts/IngestionSetup_OpenDataExternal.kql new file mode 100644 index 000000000..22ca67d06 --- /dev/null +++ b/src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/scripts/IngestionSetup_OpenDataExternal.kql @@ -0,0 +1,17 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +//====================================================================================================================== +// Ingestion database / Open data load (local emulator) +// Populates the open data tables (PricingUnits, Regions, ResourceTypes, Services) from CSV files staged on disk. +// A deployed hub loads these tables via Data Factory; this script is the local/emulator stand-in for that step and is +// published as a standalone release artifact (finops-hub-local-opendata.kql) for the "Run hubs locally" guide. +// The $$openDataPath$$ placeholder is replaced with the directory that holds the open data CSVs before the script runs. +//====================================================================================================================== + +// For allowed commands, see https://learn.microsoft.com/azure/data-explorer/database-script + +.set-or-replace PricingUnits <| externaldata(x_PricingUnitDescription: string, AccountTypes: string, x_PricingBlockSize: real, PricingUnit: string)[@'$$openDataPath$$/PricingUnits.csv'] with (format='csv', ignoreFirstRecord=true) | project-away AccountTypes +.set-or-replace Regions <| externaldata(ResourceLocation: string, RegionId: string, RegionName: string)[@'$$openDataPath$$/Regions.csv'] with (format='csv', ignoreFirstRecord=true) +.set-or-replace ResourceTypes <| externaldata(x_ResourceType: string, SingularDisplayName: string, PluralDisplayName: string, LowerSingularDisplayName: string, LowerPluralDisplayName: string, IsPreview: bool, Description: string, IconUri: string, Links: string)[@'$$openDataPath$$/ResourceTypes.csv'] with (format='csv', ignoreFirstRecord=true) | project-away Links +.set-or-replace Services <| externaldata(x_ConsumedService: string, x_ResourceType: string, ServiceName: string, ServiceCategory: string, ServiceSubcategory: string, PublisherName: string, x_PublisherCategory: string, x_Environment: string, x_ServiceModel: string)[@'$$openDataPath$$/Services.csv'] with (format='csv', ignoreFirstRecord=true) From b3c40a29c4af7f0f8be9abc775f7e5e10859e9db Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 25 Jun 2026 08:42:23 -0700 Subject: [PATCH 32/51] feat(hubs): add ftk-local-dashboard canvas extension 6-tab live FinOps dashboard for local Kusto emulator (port 8082): - Overview: total cost KPIs, top-10 services/regions, daily trend - Allocation: cost by resource group, subscription, department - Rate optimization: savings summary, commitment utilization (incl. col-5/col-7 grid fix) - Usage & unit economics: core-hours, unit cost trends - Anomalies & forecast: spend anomalies, 30-day forecast - AI tokenomics: token usage, model cost breakdown Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../ftk-local-dashboard/extension.mjs | 258 ++++ .../extensions/ftk-local-dashboard/kusto.mjs | 374 ++++++ .../ftk-local-dashboard/public/app.css | 300 +++++ .../ftk-local-dashboard/public/app.js | 1072 +++++++++++++++++ .../ftk-local-dashboard/public/index.html | 46 + 5 files changed, 2050 insertions(+) create mode 100644 .github/extensions/ftk-local-dashboard/extension.mjs create mode 100644 .github/extensions/ftk-local-dashboard/kusto.mjs create mode 100644 .github/extensions/ftk-local-dashboard/public/app.css create mode 100644 .github/extensions/ftk-local-dashboard/public/app.js create mode 100644 .github/extensions/ftk-local-dashboard/public/index.html diff --git a/.github/extensions/ftk-local-dashboard/extension.mjs b/.github/extensions/ftk-local-dashboard/extension.mjs new file mode 100644 index 000000000..417e496ee --- /dev/null +++ b/.github/extensions/ftk-local-dashboard/extension.mjs @@ -0,0 +1,258 @@ +// Extension: ftk-local-dashboard +// A FinOps dashboard canvas for the local (ftklocal) Kusto emulator. +// +// open() boots a per-instance loopback HTTP server that serves the static +// dashboard (public/) and two JSON endpoints (/api/config, /api/dashboard). +// The dashboard renderer fetches /api/dashboard, which runs the FinOps query +// layer (kusto.mjs) against the emulator's Hub database. + +import { createServer } from "node:http"; +import { readFile } from "node:fs/promises"; +import { joinSession, createCanvas, CanvasError } from "@github/copilot-sdk/extension"; +import { getDashboard, getTokenomics, getAllocation, getRate, getUsage, getAnomaly } from "./kusto.mjs"; + +const GETTERS = { + overview: getDashboard, + tokenomics: getTokenomics, + allocation: getAllocation, + rate: getRate, + usage: getUsage, + anomaly: getAnomaly, +}; + +const PUBLIC_DIR = new URL("./public/", import.meta.url); +const DEFAULT_CLUSTER = "http://localhost:8082"; +const DEFAULT_DB = "Hub"; + +const STATIC = { + "/": ["index.html", "text/html; charset=utf-8"], + "/index.html": ["index.html", "text/html; charset=utf-8"], + "/app.css": ["app.css", "text/css; charset=utf-8"], + "/app.js": ["app.js", "application/javascript; charset=utf-8"], +}; + +// instanceId -> { server, url, clusterUri, database } +const servers = new Map(); + +function sendJson(res, status, obj) { + const body = JSON.stringify(obj); + res.writeHead(status, { "Content-Type": "application/json; charset=utf-8", "Cache-Control": "no-store" }); + res.end(body); +} + +async function handleRequest(entry, req, res) { + const url = new URL(req.url, "http://127.0.0.1"); + const path = url.pathname; + + if (STATIC[path]) { + const [file, type] = STATIC[path]; + try { + const buf = await readFile(new URL(file, PUBLIC_DIR)); + res.writeHead(200, { "Content-Type": type, "Cache-Control": "no-store" }); + res.end(buf); + } catch (err) { + res.writeHead(500, { "Content-Type": "text/plain" }); + res.end(`Asset error: ${err.message}`); + } + return; + } + + if (path === "/api/config") { + sendJson(res, 200, { clusterUri: entry.clusterUri, database: entry.database, instanceId: entry.instanceId }); + return; + } + + if (path === "/api/dashboard") { + const preset = url.searchParams.get("preset") || "all"; + try { + const payload = await getDashboard(entry.clusterUri, entry.database, preset); + sendJson(res, 200, payload); + } catch (err) { + sendJson(res, 200, { error: String(err?.message ?? err), clusterUri: entry.clusterUri, database: entry.database }); + } + return; + } + + if (path === "/api/tokenomics") { + const preset = url.searchParams.get("preset") || "all"; + try { + const payload = await getTokenomics(entry.clusterUri, entry.database, preset); + sendJson(res, 200, payload); + } catch (err) { + sendJson(res, 200, { error: String(err?.message ?? err), clusterUri: entry.clusterUri, database: entry.database }); + } + return; + } + + if (path === "/api/view") { + const name = url.searchParams.get("name") || "overview"; + const preset = url.searchParams.get("preset") || "all"; + const getter = GETTERS[name]; + if (!getter) { sendJson(res, 200, { error: `Unknown view '${name}'` }); return; } + try { + const payload = await getter(entry.clusterUri, entry.database, preset); + sendJson(res, 200, payload); + } catch (err) { + sendJson(res, 200, { error: String(err?.message ?? err), clusterUri: entry.clusterUri, database: entry.database }); + } + return; + } + + res.writeHead(404, { "Content-Type": "text/plain" }); + res.end("Not found"); +} + +async function startServer(entry) { + const server = createServer((req, res) => { + handleRequest(entry, req, res).catch((err) => { + try { sendJson(res, 500, { error: String(err?.message ?? err) }); } catch { /* noop */ } + }); + }); + await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); + const addr = server.address(); + const port = typeof addr === "object" && addr ? addr.port : 0; + entry.server = server; + entry.url = `http://127.0.0.1:${port}/`; + return entry; +} + +// Compact headline KPIs for the agent-facing `summary` action. +function headline(payload) { + if (payload.empty) return { empty: true, window: payload.window }; + const d = payload.data; + const s = d.summary?.[0] || {}; + const list = s.List || 0, eff = s.Effective || 0, contracted = s.Contracted || 0; + const tag = Object.fromEntries((d.tagged || []).map((r) => [r._t, r.Cost || 0])); + const tagTotal = (tag.Tagged || 0) + (tag.Untagged || 0); + const price = Object.fromEntries((d.pricing || []).map((r) => [r.PricingCategory, r.Cost || 0])); + const priceTotal = Object.values(price).reduce((a, b) => a + b, 0); + return { + window: payload.window, + effectiveCost: Math.round(eff * 100) / 100, + billedCost: Math.round((s.Billed || 0) * 100) / 100, + totalSavings: Math.round((list - eff) * 100) / 100, + effectiveSavingsRate: list > 0 ? +((list - eff) / list).toFixed(4) : 0, + negotiatedSavings: Math.round((list - contracted) * 100) / 100, + commitmentSavings: Math.round((contracted - eff) * 100) / 100, + untaggedPercent: tagTotal > 0 ? +((tag.Untagged || 0) / tagTotal).toFixed(4) : 0, + commitmentCoverage: priceTotal > 0 ? +((price.Committed || 0) / priceTotal).toFixed(4) : 0, + resources: s.Resources || 0, + services: s.Services || 0, + subscriptions: s.Subscriptions || 0, + regions: s.Regions || 0, + topServices: (d.topServices || []).slice(0, 5).map((r) => ({ name: r.ServiceName, cost: Math.round((r.Cost || 0) * 100) / 100 })), + generatedAt: payload.generatedAt, + }; +} + +// Compact headline token KPIs for the agent-facing `tokenomics` action. +function tokenHeadline(payload) { + if (payload.empty) return { empty: true, window: payload.window }; + const d = payload.data; + const s = d.summary?.[0] || {}; + const tokens = s.Tokens || 0, eff = s.Effective || 0; + const cloud = d.totalCloud?.[0]?.Effective || 0; + const dir = Object.fromEntries((d.direction || []).map((r) => [r.Direction, r])); + const inTok = dir["Input"]?.Tokens || 0; + const cachedTok = dir["Cached input"]?.Tokens || 0; + return { + window: payload.window, + aiTokenCost: Math.round(eff * 100) / 100, + totalTokens: tokens, + blendedCostPerMillionTokens: tokens > 0 ? +((eff / tokens) * 1e6).toFixed(4) : 0, + cachedInputShareOfInputTokens: inTok + cachedTok > 0 ? +(cachedTok / (inTok + cachedTok)).toFixed(4) : 0, + aiShareOfCloudCost: cloud > 0 ? +(eff / cloud).toFixed(4) : 0, + modelCount: s.Models || 0, + directionMix: (d.direction || []).map((r) => ({ direction: r.Direction, tokens: r.Tokens, cost: Math.round((r.Cost || 0) * 100) / 100 })), + topModels: (d.models || []).slice(0, 5).map((r) => ({ + model: r.Model, tokens: r.Tokens, cost: Math.round((r.Cost || 0) * 100) / 100, + costPerMillionTokens: +((r.CostPer1K || 0) * 1000).toFixed(4), + })), + generatedAt: payload.generatedAt, + }; +} + +await joinSession({ + canvases: [ + createCanvas({ + id: "ftk-local-dashboard", + displayName: "FinOps hub local dashboard", + description: "Live FinOps dashboard for the local (ftklocal) Kusto emulator with six views: cost overview, allocation, rate optimization, usage & unit economics, anomalies & forecast, and AI tokenomics.", + inputSchema: { + type: "object", + properties: { + clusterUri: { type: "string", description: "Base URI of the Kusto emulator. Default http://localhost:8082." }, + database: { type: "string", description: "Database name. Default Hub." }, + }, + }, + actions: [ + { + name: "summary", + description: "Run the dashboard queries against the emulator and return headline FinOps KPIs (effective cost, savings, ESR, untagged %, commitment coverage, top services) for a time window.", + inputSchema: { + type: "object", + properties: { + preset: { type: "string", enum: ["all", "12m", "6m", "3m"], description: "Time window. Default all." }, + }, + }, + handler: async (ctx) => { + const entry = servers.get(ctx.instanceId); + const clusterUri = entry?.clusterUri || DEFAULT_CLUSTER; + const database = entry?.database || DEFAULT_DB; + const preset = ctx.input?.preset || "all"; + try { + const payload = await getDashboard(clusterUri, database, preset); + return headline(payload); + } catch (err) { + throw new CanvasError("query_failed", `Could not query ${clusterUri}/${database}: ${err?.message ?? err}`); + } + }, + }, + { + name: "tokenomics", + description: "Run the AI token-economics queries against the emulator and return headline token KPIs (AI token cost, total tokens, blended cost per 1M tokens, cached-input share, AI share of cloud, top models, direction mix) for a time window.", + inputSchema: { + type: "object", + properties: { + preset: { type: "string", enum: ["all", "12m", "6m", "3m"], description: "Time window. Default all." }, + }, + }, + handler: async (ctx) => { + const entry = servers.get(ctx.instanceId); + const clusterUri = entry?.clusterUri || DEFAULT_CLUSTER; + const database = entry?.database || DEFAULT_DB; + const preset = ctx.input?.preset || "all"; + try { + const payload = await getTokenomics(clusterUri, database, preset); + return tokenHeadline(payload); + } catch (err) { + throw new CanvasError("query_failed", `Could not query ${clusterUri}/${database}: ${err?.message ?? err}`); + } + }, + }, + ], + open: async (ctx) => { + const clusterUri = (ctx.input?.clusterUri || DEFAULT_CLUSTER).trim(); + const database = (ctx.input?.database || DEFAULT_DB).trim(); + let entry = servers.get(ctx.instanceId); + if (!entry) { + entry = { instanceId: ctx.instanceId, clusterUri, database }; + await startServer(entry); + servers.set(ctx.instanceId, entry); + } else { + // Reopen may carry updated connection input. + entry.clusterUri = clusterUri; + entry.database = database; + } + return { title: "FinOps hub · local", url: entry.url, status: `${clusterUri} · ${database}` }; + }, + onClose: async (ctx) => { + const entry = servers.get(ctx.instanceId); + if (entry) { + servers.delete(ctx.instanceId); + await new Promise((resolve) => entry.server.close(() => resolve())); + } + }, + }), + ], +}); diff --git a/.github/extensions/ftk-local-dashboard/kusto.mjs b/.github/extensions/ftk-local-dashboard/kusto.mjs new file mode 100644 index 000000000..50c0f9ce1 --- /dev/null +++ b/.github/extensions/ftk-local-dashboard/kusto.mjs @@ -0,0 +1,374 @@ +// KQL query layer for the FinOps hub local (ftklocal) Kusto emulator. +// +// Talks to the Kusto emulator HTTP API (/v1/rest/query) and parses the v1 +// response shape (Tables[0]) into plain row objects. The dashboard queries are +// grounded in the FinOps Framework domains and the FinOps toolkit query +// catalog (src/queries/INDEX.md, KPI.md, finops-hub-database-guide.md). + +const DEFAULT_TIMEOUT_MS = 20000; + +/** + * Run a single KQL query against the emulator and return rows as objects. + * Throws on transport error or Kusto error payload. + */ +export async function runQuery(clusterUri, database, csl, timeoutMs = DEFAULT_TIMEOUT_MS) { + const controller = new AbortController(); + const timer = setTimeout(() => controller.abort(), timeoutMs); + let res; + try { + res = await fetch(`${clusterUri.replace(/\/+$/, "")}/v1/rest/query`, { + method: "POST", + headers: { "Content-Type": "application/json", Accept: "application/json" }, + body: JSON.stringify({ db: database, csl }), + signal: controller.signal, + }); + } catch (err) { + if (err?.name === "AbortError") { + throw new Error(`Timed out after ${timeoutMs}ms reaching ${clusterUri}`); + } + throw new Error(`Could not reach Kusto emulator at ${clusterUri}: ${err?.message ?? err}`); + } finally { + clearTimeout(timer); + } + + if (!res.ok) { + const text = await res.text().catch(() => ""); + throw new Error(`Kusto returned HTTP ${res.status}. ${text.slice(0, 300)}`); + } + + const json = await res.json(); + // v1 query API returns { Tables: [ { TableName, Columns:[{ColumnName}], Rows:[[...]] }, ... ] } + // An error surfaces as an OneApiErrors / Exceptions payload instead. + if (json?.error || json?.Exceptions || json?.OneApiErrors) { + const msg = json?.error?.["@message"] || JSON.stringify(json).slice(0, 300); + throw new Error(`Kusto query error: ${msg}`); + } + const table = Array.isArray(json?.Tables) ? json.Tables[0] : null; + if (!table) return []; + const cols = table.Columns.map((c) => c.ColumnName); + return table.Rows.map((r) => Object.fromEntries(cols.map((c, i) => [c, r[i]]))); +} + +// --- date helpers (work in UTC to match Kusto datetimes) ---------------------- + +function startOfMonthUTC(d) { + return new Date(Date.UTC(d.getUTCFullYear(), d.getUTCMonth(), 1)); +} +function addMonthsUTC(d, n) { + return new Date(Date.UTC(d.getUTCFullYear(), d.getUTCMonth() + n, 1)); +} +function isoDay(d) { + return d.toISOString().slice(0, 10); +} + +/** + * Resolve a preset window (all | 12m | 6m | 3m) against the actual data range. + * Returns inclusive start and exclusive end ISO-day strings plus the data range. + */ +export async function resolveWindow(clusterUri, database, preset) { + const range = await runQuery( + clusterUri, + database, + "Costs() | summarize MinDate=min(ChargePeriodStart), MaxDate=max(ChargePeriodStart), Rows=count()" + ); + const row = range[0] ?? {}; + if (!row.MaxDate) { + return { start: null, end: null, dataMin: null, dataMax: null, rows: 0, empty: true }; + } + const dataMin = new Date(row.MinDate); + const dataMax = new Date(row.MaxDate); + const endExclusive = addMonthsUTC(startOfMonthUTC(dataMax), 1); // include the whole last month + const lastMonth = startOfMonthUTC(dataMax); + + let start; + switch (preset) { + case "3m": start = addMonthsUTC(lastMonth, -2); break; + case "6m": start = addMonthsUTC(lastMonth, -5); break; + case "12m": start = addMonthsUTC(lastMonth, -11); break; + case "all": + default: start = startOfMonthUTC(dataMin); break; + } + if (start < startOfMonthUTC(dataMin)) start = startOfMonthUTC(dataMin); + + return { + start: isoDay(start), + end: isoDay(endExclusive), + dataMin: isoDay(dataMin), + dataMax: isoDay(dataMax), + rows: row.Rows ?? 0, + empty: false, + }; +} + +// --- dashboard data ----------------------------------------------------------- + +/** + * Run all dashboard queries in parallel for the resolved window and shape the + * result into a single payload the renderer consumes. + */ +export async function getDashboard(clusterUri, database, preset = "all") { + const win = await resolveWindow(clusterUri, database, preset); + if (win.empty) { + return { window: win, empty: true, generatedAt: new Date().toISOString() }; + } + + const period = `| where ChargePeriodStart >= datetime(${win.start}) and ChargePeriodStart < datetime(${win.end})`; + + const queries = { + // KPI totals — Understand Usage & Cost + Quantify Business Value + summary: `Costs() ${period} | summarize Billed=sum(BilledCost), Effective=sum(EffectiveCost), List=sum(ListCost), Contracted=sum(ContractedCost), Resources=dcount(ResourceId), Services=dcount(ServiceName), Subscriptions=dcount(SubAccountId), Regions=dcount(RegionId), Rows=count()`, + // Allocation KPI — percentage-untagged-costs + tagged: `Costs() ${period} | extend _t=iff(isnull(Tags) or array_length(bag_keys(Tags))==0,'Untagged','Tagged') | summarize Cost=sum(EffectiveCost) by _t`, + // Rate Optimization — commitment coverage (Committed vs Standard pricing) + pricing: `Costs() ${period} | summarize Cost=sum(EffectiveCost) by PricingCategory`, + // Reporting & Analytics — monthly-cost-trend (Billed vs Effective) + trend: `Costs() ${period} | summarize Billed=sum(BilledCost), Effective=sum(EffectiveCost) by Month=format_datetime(startofmonth(ChargePeriodStart),'yyyy-MM') | order by Month asc`, + // Understand Usage & Cost — cost by service category + serviceCategory: `Costs() ${period} | summarize Cost=sum(EffectiveCost) by ServiceCategory | where Cost > 0 | order by Cost desc`, + // top-services-by-cost + topServices: `Costs() ${period} | summarize Cost=sum(EffectiveCost) by ServiceName | top 10 by Cost desc`, + // top-resource-groups-by-cost + topResourceGroups: `Costs() ${period} | where isnotempty(x_ResourceGroupName) | summarize Cost=sum(EffectiveCost) by x_ResourceGroupName | top 10 by Cost desc`, + // cost-by-region-trend (top regions by cost) + topRegions: `Costs() ${period} | where isnotempty(RegionId) | summarize Cost=sum(EffectiveCost) by RegionId | top 12 by Cost desc`, + // Charge category mix (Usage / Purchase / Adjustment) + chargeCategory: `Costs() ${period} | summarize Cost=sum(EffectiveCost) by ChargeCategory | where Cost != 0 | order by Cost desc`, + }; + + const entries = Object.entries(queries); + const results = await Promise.all( + entries.map(([, csl]) => runQuery(clusterUri, database, csl)) + ); + const data = Object.fromEntries(entries.map(([key], i) => [key, results[i]])); + + return { + window: win, + empty: false, + data, + generatedAt: new Date().toISOString(), + }; +} + +// --- tokenomics (AI / Azure OpenAI token economics) --------------------------- +// +// Grounded in the FinOps toolkit AI query catalog (ai-token-usage-breakdown, +// ai-model-cost-comparison, ai-daily-trend) and the FinOps Foundation +// "Token Consumption Metrics" KPI (Cost per Token = Total Cost / Tokens Used). +// +// Token meters are scoped to Azure OpenAI subcategories whose SKU description +// is denominated in tokens, which excludes non-token AI meters (image/media, +// Cognitive Search). ConsumedQuantity is the token count per the catalog. +const AI_SCOPE = `| where x_SkuMeterSubcategory has 'OpenAI' and x_SkuDescription contains 'Token'`; + +// Direction: descriptions use abbreviations (Inp / Outp / cached Inp), so the +// canonical contains "Input"/"Output" test is replaced with term/substring +// matching that also splits cached input out as its own (cheaper) bucket. +const DIRECTION = `extend Direction = case( + x_SkuDescription has 'Outp' or x_SkuDescription contains 'Output', 'Output', + x_SkuDescription contains 'cached', 'Cached input', + x_SkuDescription has 'Inp' or x_SkuDescription contains 'Input', 'Input', + 'Other')`; + +// Collapse verbose SKU descriptions to a clean model family, e.g. +// "Azure OpenAI - gpt 4.1 cached Inp glbl Tokens - US East 2" -> "GPT 4.1". +const MODEL_FAMILY = `extend Model = x_SkuDescription +| extend Model = replace_regex(Model, @'^Azure OpenAI(?: GPT5)?\\s*-\\s*', '') +| extend Model = replace_regex(Model, @'(?i)[\\s-]+(cached[\\s-]+)?(inp|inpt|outp|out|chat|media)([\\s-].*)?$', '') +| extend Model = replace_regex(trim(@'[\\s-]+', Model), @'(?i)^gpt', 'GPT')`; + +export async function getTokenomics(clusterUri, database, preset = "all") { + const win = await resolveWindow(clusterUri, database, preset); + if (win.empty) { + return { window: win, empty: true, generatedAt: new Date().toISOString() }; + } + const period = `| where ChargePeriodStart >= datetime(${win.start}) and ChargePeriodStart < datetime(${win.end})`; + + const queries = { + // Token KPI totals — Token Consumption Metrics + summary: `Costs() ${period} ${AI_SCOPE} | summarize Tokens=sum(ConsumedQuantity), Effective=sum(EffectiveCost), List=sum(ListCost), Models=dcount(x_SkuDescription), Resources=dcount(ResourceId), Rows=count()`, + // Total cloud effective cost in the window — for AI share-of-spend + totalCloud: `Costs() ${period} | summarize Effective=sum(EffectiveCost)`, + // ai-token-usage-breakdown — direction mix (input/cached/output) + direction: `Costs() ${period} ${AI_SCOPE} | ${DIRECTION} | summarize Tokens=sum(ConsumedQuantity), Cost=sum(EffectiveCost) by Direction`, + // ai-model-cost-comparison — by model family with cost per 1K tokens + models: `Costs() ${period} ${AI_SCOPE} | ${MODEL_FAMILY} | summarize Tokens=sum(ConsumedQuantity), Cost=sum(EffectiveCost), List=sum(ListCost) by Model | extend CostPer1K=iff(Tokens==0, 0.0, Cost/Tokens*1000) | top 12 by Cost desc`, + // ai-daily-trend (monthly variant) — token volume + AI cost over time + trend: `Costs() ${period} ${AI_SCOPE} | summarize Tokens=sum(ConsumedQuantity), Cost=sum(EffectiveCost) by Month=format_datetime(startofmonth(ChargePeriodStart),'yyyy-MM') | order by Month asc`, + }; + + const entries = Object.entries(queries); + const results = await Promise.all(entries.map(([, csl]) => runQuery(clusterUri, database, csl))); + const data = Object.fromEntries(entries.map(([key], i) => [key, results[i]])); + + const tokenRows = data.summary?.[0]?.Rows ?? 0; + return { + window: win, + empty: tokenRows === 0, + data, + generatedAt: new Date().toISOString(), + }; +} + +// --- shared page runner ------------------------------------------------------- +// Resolves the window, builds a named map of KQL queries from the period clause, +// runs them in parallel, and returns { window, empty, data, generatedAt }. +async function runPage(clusterUri, database, preset, buildQueries) { + const win = await resolveWindow(clusterUri, database, preset); + if (win.empty) return { window: win, empty: true, generatedAt: new Date().toISOString() }; + const period = `| where ChargePeriodStart >= datetime(${win.start}) and ChargePeriodStart < datetime(${win.end})`; + const queries = buildQueries(period, win); + const entries = Object.entries(queries); + const results = await Promise.all(entries.map(([, csl]) => runQuery(clusterUri, database, csl))); + const data = Object.fromEntries(entries.map(([key], i) => [key, results[i]])); + return { window: win, empty: false, data, generatedAt: new Date().toISOString() }; +} + +// --- Allocation page ---------------------------------------------------------- +// FinOps "Allocation" capability. Grounded in catalog queries: +// percentage-untagged-costs, percentage-unallocated-costs, tagging-policy-compliance, +// allocation-accuracy-index, cost-by-financial-hierarchy. Tag policy keys are tuned +// to this estate's taxonomy (CostCenter/env/org); allocation evidence also honours +// the enriched x_CostCenter / x_CostAllocationRuleName columns. +const NON_PURCHASE = `| where not(ChargeCategory == 'Purchase' and isnotempty(CommitmentDiscountCategory))`; + +export async function getAllocation(clusterUri, database, preset = "all") { + return runPage(clusterUri, database, preset, (period) => ({ + // Single-pass core: total, untagged, attributed (AAI), compliant + core: `let req=dynamic(['CostCenter','env','org']); +let ev=dynamic(['cost-center','team','owner','application','product','CostCenter','org','env','Project']); +Costs() ${period} ${NON_PURCHASE} +| extend tk=coalesce(bag_keys(Tags), dynamic([])) +| extend isUntagged = array_length(tk)==0 +| extend hasEvidence = isnotempty(x_CostAllocationRuleName) or isnotempty(x_CostCenter) or array_length(set_intersect(tk,ev))>0 +| extend isCompliant = array_length(set_intersect(tk,req))==array_length(req) +| summarize Total=sum(EffectiveCost), Untagged=sumif(EffectiveCost,isUntagged), Attributed=sumif(EffectiveCost,hasEvidence), Compliant=sumif(EffectiveCost,isCompliant), Subs=dcount(SubAccountId)`, + // cost-by-financial-hierarchy (tuned to org/Project/env taxonomy) + hierarchy: `Costs() ${period} +| extend Org=tostring(Tags['org']), Project=tostring(Tags['Project']), Env=tostring(Tags['env']) +| summarize Cost=sum(EffectiveCost) by Org, Project, Env +| where Cost > 0 | top 12 by Cost desc`, + // Tag-key coverage — cost touched by each tag key + tagKeys: `Costs() ${period} +| mv-expand k=bag_keys(Tags) to typeof(string) +| where isnotempty(k) and k !in ('ftk-tool','ftk-version','cm-resource-parent','costanalysis-parent') +| summarize Cost=sum(EffectiveCost) by k +| top 12 by Cost desc`, + // Cost by subscription (SubAccountName) + bySubscription: `Costs() ${period} | where isnotempty(SubAccountName) | summarize Cost=sum(EffectiveCost) by SubAccountName | top 10 by Cost desc`, + })); +} + +// --- Rate optimization page --------------------------------------------------- +// FinOps "Rate Optimization" capability. Grounded in catalog queries: +// savings-summary-report, commitment-discount-waste, compute-spend-commitment-coverage, +// commitment-discount-utilization. (cost-optimization-index/COIN is omitted because it +// depends on Recommendations(), which is empty in this estate, so it would always read 100.) +// Commitment utilization is derived as the effective-cost complement of waste, the cleanest +// single-basis definition for a grand-total KPI. +export async function getRate(clusterUri, database, preset = "all") { + return runPage(clusterUri, database, preset, (period) => ({ + // savings-summary-report — ESR + negotiated/commitment/total savings + savings: `Costs() ${period} ${NON_PURCHASE} +| extend neg=iff(ListCost0, cores*ConsumedQuantity, toreal('')) +| extend t=iff(isempty(CommitmentDiscountType),'On Demand',CommitmentDiscountType) +| summarize CoreHours=sum(ch) by t | where CoreHours > 0 | order by CoreHours desc`, + // Per-commitment waste — which reservations/plans are underutilized + byCommitment: `Costs() ${period} | where isnotempty(CommitmentDiscountName) ${NON_PURCHASE} +| summarize Unused=sumif(EffectiveCost,CommitmentDiscountStatus=='Unused'), Total=sum(EffectiveCost) by CommitmentDiscountName +| where Unused > 0 | top 10 by Unused desc`, + })); +} + +// --- Usage & unit economics page ---------------------------------------------- +// FinOps "Usage Optimization" + "Unit Economics". Grounded in catalog queries: +// compute-cost-per-core, cost-per-gb-stored, storage-tier-distribution, top-resource-types-by-cost. +export async function getUsage(clusterUri, database, preset = "all") { + return runPage(clusterUri, database, preset, (period) => ({ + // compute-cost-per-core + compute: `Costs() ${period} ${NON_PURCHASE} +| extend vm = x_SkuMeterCategory in ('Virtual Machines','Virtual Machine Licenses') and ChargeCategory=='Usage' +| extend isComputeCommit = x_SkuMeterCategory in ('Virtual Machines','Virtual Machine Licenses') +| extend cores = iff(vm, toint(coalesce(x_SkuDetails.VCPUs, x_SkuDetails.vCores)), toint('')) +| extend ch = iff(vm and isnotempty(cores), toreal(cores*ConsumedQuantity), toreal('')) +| summarize ComputeEff=sumif(EffectiveCost,vm), UnusedCommit=sumif(EffectiveCost, CommitmentDiscountStatus=='Unused' and isnotempty(CommitmentDiscountCategory) and isComputeCommit), CoreHours=sum(ch)`, + // cost-per-gb-stored + storage: `Costs() ${period} | where ServiceCategory=='Storage' and ChargeCategory=='Usage' +| extend gb = case(ConsumedUnit endswith 'PB', toreal(ConsumedQuantity)*1048576.0, ConsumedUnit endswith 'TB', toreal(ConsumedQuantity)*1024.0, ConsumedUnit endswith 'MB', toreal(ConsumedQuantity)/1024.0, toreal(ConsumedQuantity)) +| summarize Cost=sum(EffectiveCost), GBMonths=sum(gb)`, + // storage-tier-distribution + storageTiers: `Costs() ${period} | where ServiceCategory=='Storage' and ChargeCategory=='Usage' +| extend Tier = case( + x_SkuTier in ('Hot','Standard','Premium'), 'Frequent', + x_SkuTier in ('Cool','Cold','Archive'), 'Infrequent', + x_SkuMeterSubcategory has_any ('Hot','Standard','Premium','Frequent'), 'Frequent', + x_SkuMeterSubcategory has_any ('Cool','Cold','Archive'), 'Infrequent', + 'Unclassified') +| summarize Cost=sum(EffectiveCost) by Tier | where Cost > 0 | order by Cost desc`, + // top-resource-types-by-cost + topResourceTypes: `Costs() ${period} | where isnotempty(ResourceType) | summarize Count=count(), Cost=sum(EffectiveCost) by ResourceType | top 10 by Cost desc`, + // compute-cost-per-core grouped by VM series — where the expensive cores are + perCoreSeries: `Costs() ${period} | where x_SkuMeterCategory in ('Virtual Machines','Virtual Machine Licenses') and ChargeCategory=='Usage' +| extend cores=toint(coalesce(x_SkuDetails.VCPUs, x_SkuDetails.vCores)) +| extend ch=iff(isnotempty(cores), toreal(cores*ConsumedQuantity), toreal('')) +| summarize Eff=sum(EffectiveCost), CH=sum(ch) by x_SkuMeterSubcategory +| where CH > 100 | extend PerCore=Eff/CH | top 10 by Eff desc`, + // grand total for share-of-cost on the resource-type table + total: `Costs() ${period} | summarize Total=sum(EffectiveCost)`, + })); +} + +// --- Anomalies & forecast page ------------------------------------------------ +// FinOps "Anomaly Management" + "Forecasting" + data-freshness (Data Ingestion). +// Grounded in catalog queries: cost-anomaly-detection, anomaly-detection-rate, +// anomaly-variance-total, monthly-cost-change-percentage, cost-forecasting-model, +// data-update-frequency, cost-visibility-delay. Time-series array outputs are +// flattened with mv-expand so the renderer can chart them. +export async function getAnomaly(clusterUri, database, preset = "all") { + return runPage(clusterUri, database, preset, (period, win) => { + // forecast uses full history for accuracy; horizon = 4 months past the last data month + const dmax = new Date(win.dataMax); + const monthStart = new Date(Date.UTC(dmax.getUTCFullYear(), dmax.getUTCMonth(), 1)); + const horizon = new Date(Date.UTC(dmax.getUTCFullYear(), dmax.getUTCMonth() + 4, 1)); + const isoH = horizon.toISOString().slice(0, 10); + const fcDays = Math.round((horizon - monthStart) / 86400000); + return { + // cost-anomaly-detection + anomaly-variance-total (flattened daily series) + daily: `let s=datetime(${win.start}); let e=datetime(${win.end}); +Costs() | where ChargePeriodStart>=s and ChargePeriodStart=s +| summarize Eff=sum(EffectiveCost) by bin(ChargePeriodStart,1d) +| make-series Actual=sum(Eff) default=0.0 on ChargePeriodStart from s to datetime(${isoH}) step 1d +| extend Fc=series_decompose_forecast(Actual,${fcDays}) +| mv-expand Day=ChargePeriodStart to typeof(datetime), Actual to typeof(real), Fc to typeof(real) +| extend M=startofmonth(Day) +| summarize Actual=sum(toreal(Actual)), Forecast=sum(toreal(Fc)) by M +| order by M asc | project Month=format_datetime(M,'yyyy-MM'), Actual, Forecast`, + // data-update-frequency + cost-visibility-delay + freshness: `Costs() ${period} | where isnotnull(x_IngestionTime) +| summarize LastUpdate=max(x_IngestionTime), Rows=count(), P50=percentile(todouble((x_IngestionTime-ChargePeriodEnd)/1h),50), P90=percentile(todouble((x_IngestionTime-ChargePeriodEnd)/1h),90)`, + }; + }); +} diff --git a/.github/extensions/ftk-local-dashboard/public/app.css b/.github/extensions/ftk-local-dashboard/public/app.css new file mode 100644 index 000000000..f22d5ac97 --- /dev/null +++ b/.github/extensions/ftk-local-dashboard/public/app.css @@ -0,0 +1,300 @@ +:root { + --accent: #3b82f6; + --pos: #10b981; + --neg: #ef4444; + --warn: #f59e0b; + --grid: var(--border-color-default, rgba(128, 128, 128, 0.22)); + --muted: var(--text-color-muted, #6b7280); + --card-bg: var(--background-color-default, #ffffff); + --radius: 12px; + --gap: 16px; +} + +* { box-sizing: border-box; } + +body { + margin: 0; + padding: 0 20px 28px; + background: var(--background-color-default, #f6f8fa); + color: var(--text-color-default, #1f2328); + font-family: var(--font-sans, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif); + font-size: var(--text-body-medium, 14px); + line-height: var(--leading-body-medium, 20px); + -webkit-font-smoothing: antialiased; +} + +.muted { color: var(--muted); } + +/* ---------- header ---------- */ +.app-header { + position: sticky; + top: 0; + z-index: 5; + display: flex; + align-items: flex-end; + justify-content: space-between; + gap: 16px; + padding: 18px 0 14px; + margin-bottom: 6px; + background: linear-gradient(var(--background-color-default, #f6f8fa) 78%, transparent); + flex-wrap: wrap; +} +.title-block h1 { + margin: 0; + font-size: var(--text-title-large, 24px); + font-weight: var(--font-weight-semibold, 650); + letter-spacing: -0.01em; +} +.title-block .sub { + margin: 4px 0 0; + font-size: 12.5px; + color: var(--muted); +} +.title-block .sub code { + font-family: var(--font-mono, ui-monospace, monospace); + font-size: 11.5px; + padding: 1px 5px; + border-radius: 5px; + background: color-mix(in srgb, var(--muted) 14%, transparent); +} + +.controls { display: flex; align-items: center; gap: 10px; } +.seg { + display: inline-flex; + border: 1px solid var(--grid); + border-radius: 9px; + overflow: hidden; +} +.seg button { + appearance: none; + border: 0; + background: transparent; + color: var(--text-color-default, #1f2328); + padding: 6px 13px; + font-size: 12.5px; + font-weight: 600; + cursor: pointer; + border-left: 1px solid var(--grid); +} +.seg button:first-child { border-left: 0; } +.seg button.active { background: var(--accent); color: #fff; } +.seg button:not(.active):hover { background: color-mix(in srgb, var(--accent) 12%, transparent); } + +.btn { + appearance: none; + border: 1px solid var(--grid); + background: var(--card-bg); + color: var(--text-color-default, #1f2328); + padding: 6px 13px; + border-radius: 9px; + font-size: 12.5px; + font-weight: 600; + cursor: pointer; +} +.btn:hover { background: color-mix(in srgb, var(--accent) 10%, transparent); } +.btn:active { transform: translateY(1px); } + +/* ---------- tabs ---------- */ +.tabs { + display: flex; + gap: 4px; + margin: 2px 0 18px; + border-bottom: 1px solid var(--grid); +} +.tabs button { + appearance: none; + border: 0; + background: transparent; + color: var(--muted); + font-size: 13.5px; + font-weight: 620; + padding: 9px 14px 11px; + cursor: pointer; + position: relative; + border-radius: 8px 8px 0 0; +} +.tabs button:hover { color: var(--text-color-default, #1f2328); background: color-mix(in srgb, var(--accent) 8%, transparent); } +.tabs button.active { color: var(--text-color-default, #1f2328); } +.tabs button.active::after { + content: ""; + position: absolute; + left: 10px; right: 10px; bottom: -1px; + height: 2.5px; + border-radius: 2px; + background: var(--accent); +} + +/* ---------- KPI cards ---------- */ +.kpi-grid { + display: grid; + grid-template-columns: repeat(auto-fit, minmax(180px, 1fr)); + gap: var(--gap); + margin-bottom: 22px; +} +.kpi { + background: var(--card-bg); + border: 1px solid var(--grid); + border-radius: var(--radius); + padding: 15px 16px 14px; + position: relative; + overflow: hidden; +} +.kpi::before { + content: ""; + position: absolute; + left: 0; top: 0; bottom: 0; + width: 3px; + background: var(--kpi-accent, var(--accent)); +} +.kpi .label { + font-size: 11.5px; + text-transform: uppercase; + letter-spacing: 0.04em; + color: var(--muted); + font-weight: 600; +} +.kpi .value { + font-size: 26px; + font-weight: 680; + margin-top: 6px; + letter-spacing: -0.02em; + font-variant-numeric: tabular-nums; +} +.kpi .meta { margin-top: 5px; font-size: 12px; color: var(--muted); } +.kpi .meta .pos { color: var(--pos); font-weight: 650; } +.kpi .meta .neg { color: var(--neg); font-weight: 650; } +.kpi .meta .warn { color: var(--warn); font-weight: 650; } + +/* ---------- sections & panels ---------- */ +.section-title { + display: flex; + align-items: baseline; + gap: 10px; + margin: 26px 0 12px; +} +.section-title h2 { + margin: 0; + font-size: 15px; + font-weight: 650; + letter-spacing: -0.01em; +} +.section-title .domain { + font-size: 11px; + color: var(--muted); + text-transform: uppercase; + letter-spacing: 0.05em; +} + +.panel-grid { + display: grid; + gap: var(--gap); + grid-template-columns: repeat(12, 1fr); +} +.panel { + background: var(--card-bg); + border: 1px solid var(--grid); + border-radius: var(--radius); + padding: 16px; + min-width: 0; +} +.panel.col-12 { grid-column: span 12; } +.panel.col-9 { grid-column: span 9; } +.panel.col-8 { grid-column: span 8; } +.panel.col-7 { grid-column: span 7; } +.panel.col-6 { grid-column: span 6; } +.panel.col-5 { grid-column: span 5; } +.panel.col-4 { grid-column: span 4; } +.panel.col-3 { grid-column: span 3; } +@media (max-width: 900px) { + .panel.col-9, .panel.col-8, .panel.col-7, + .panel.col-6, .panel.col-5, .panel.col-4, + .panel.col-3 { grid-column: span 12; } +} +.panel h3 { + margin: 0 0 2px; + font-size: 13.5px; + font-weight: 640; +} +.panel .panel-sub { + margin: 0 0 12px; + font-size: 11.5px; + color: var(--muted); +} + +/* ---------- charts ---------- */ +svg { display: block; width: 100%; overflow: visible; } +.axis-label, .tick { fill: var(--muted); font-size: 11px; } +.grid-line { stroke: var(--grid); stroke-width: 1; } + +.legend { display: flex; flex-wrap: wrap; gap: 10px 16px; margin-top: 10px; } +.legend .item { display: inline-flex; align-items: center; gap: 6px; font-size: 12px; } +.legend .swatch { width: 10px; height: 10px; border-radius: 3px; flex: none; } +.legend .lv { color: var(--muted); font-variant-numeric: tabular-nums; } + +.donut-wrap { display: flex; align-items: center; gap: 18px; flex-wrap: wrap; } +.donut-center .big { font-size: 18px; font-weight: 680; letter-spacing: -0.02em; } +.donut-center .small { font-size: 11px; fill: var(--muted); } + +.hbar-row { font-variant-numeric: tabular-nums; } +.hbar-row .name { fill: var(--text-color-default, #1f2328); font-size: 12px; } +.hbar-row .val { fill: var(--muted); font-size: 11.5px; } +.bar:hover, .arc:hover, .hbar:hover { opacity: 0.82; cursor: default; } + +/* ---------- data table ---------- */ +.dtable { width: 100%; border-collapse: collapse; font-size: 12.5px; font-variant-numeric: tabular-nums; } +.dtable th, .dtable td { padding: 9px 10px; text-align: right; border-bottom: 1px solid var(--grid); white-space: nowrap; } +.dtable th:first-child, .dtable td:first-child { text-align: left; } +.dtable thead th { color: var(--muted); font-weight: 600; font-size: 11px; text-transform: uppercase; letter-spacing: 0.03em; } +.dtable tbody tr:hover { background: color-mix(in srgb, var(--accent) 6%, transparent); } +.dtable .model { display: inline-flex; align-items: center; gap: 7px; font-weight: 580; } +.dtable .swatch { width: 9px; height: 9px; border-radius: 3px; flex: none; } +.dtable .barcell { position: relative; } +.dtable .minibar { display: inline-block; height: 7px; border-radius: 3px; vertical-align: middle; margin-left: 6px; } + +/* ---------- states ---------- */ +.loading, .error { + padding: 40px; + text-align: center; + color: var(--muted); +} +.error { + border: 1px solid var(--grid); + border-radius: var(--radius); + background: var(--card-bg); + max-width: 640px; + margin: 40px auto; +} +.error h2 { color: var(--neg); margin: 0 0 8px; font-size: 16px; } +.error code { + font-family: var(--font-mono, ui-monospace, monospace); + font-size: 12px; + background: color-mix(in srgb, var(--muted) 14%, transparent); + padding: 2px 6px; + border-radius: 5px; +} +.error pre { + text-align: left; + white-space: pre-wrap; + font-size: 12px; + background: color-mix(in srgb, var(--muted) 10%, transparent); + padding: 10px 12px; + border-radius: 8px; + margin-top: 14px; +} + +/* ---------- footer ---------- */ +.app-footer { + display: flex; + justify-content: space-between; + gap: 12px; + flex-wrap: wrap; + margin-top: 26px; + padding-top: 14px; + border-top: 1px solid var(--grid); + font-size: 11.5px; + color: var(--muted); +} +.app-footer #footer-meta { font-variant-numeric: tabular-nums; } + +.spin { animation: spin 0.8s linear infinite; display: inline-block; } +@keyframes spin { to { transform: rotate(360deg); } } diff --git a/.github/extensions/ftk-local-dashboard/public/app.js b/.github/extensions/ftk-local-dashboard/public/app.js new file mode 100644 index 000000000..38e435456 --- /dev/null +++ b/.github/extensions/ftk-local-dashboard/public/app.js @@ -0,0 +1,1072 @@ +/* FinOps hub local dashboard — client renderer. + Dependency-free: KPIs + SVG charts (line, horizontal bar, donut). + Data comes from the extension's loopback /api endpoints. */ + +"use strict"; + +const PALETTE = [ + "#3b82f6", "#10b981", "#8b5cf6", "#f59e0b", "#ef4444", "#06b6d4", + "#ec4899", "#84cc16", "#f97316", "#6366f1", "#14b8a6", "#a855f7", +]; + +const state = { preset: "all", tab: "overview", loading: false, cache: {} }; + +/* ------------------------------------------------------------------ utils */ + +function fmtMoney(n) { + if (n == null || isNaN(n)) return "$0"; + const sign = n < 0 ? "-" : ""; + const a = Math.abs(n); + if (a >= 1e6) return `${sign}$${(a / 1e6).toFixed(2)}M`; + if (a >= 1e3) return `${sign}$${(a / 1e3).toFixed(1)}K`; + return `${sign}$${a.toFixed(a < 100 ? 2 : 0)}`; +} +function fmtMoneyFull(n) { + if (n == null || isNaN(n)) return "$0"; + return n.toLocaleString("en-US", { style: "currency", currency: "USD", maximumFractionDigits: 0 }); +} +function fmtPct(x, d = 1) { + if (x == null || isNaN(x)) return "—"; + return `${(x * 100).toFixed(d)}%`; +} +function fmtInt(n) { + return (n ?? 0).toLocaleString("en-US"); +} +function fmtTokens(n) { + if (n == null || isNaN(n)) return "0"; + const a = Math.abs(n); + if (a >= 1e9) return `${(n / 1e9).toFixed(2)}B`; + if (a >= 1e6) return `${(n / 1e6).toFixed(1)}M`; + if (a >= 1e3) return `${(n / 1e3).toFixed(1)}K`; + return `${Math.round(n)}`; +} +function fmtPerM(costPer1K) { + // costPer1K is $ per 1,000 tokens -> show $ per 1,000,000 tokens + const v = (costPer1K || 0) * 1000; + return `$${v.toFixed(2)}`; +} +function fmtMonth(ym) { + // "2025-04" -> "Apr ’25" + const [y, m] = ym.split("-"); + const names = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"]; + return `${names[+m - 1]} ’${y.slice(2)}`; +} +function fmtDayRange(min, max) { + const f = (s) => { + const d = new Date(s); + return d.toLocaleDateString("en-US", { month: "short", year: "numeric", timeZone: "UTC" }); + }; + return `${f(min)} – ${f(max)}`; +} +function esc(s) { + return String(s ?? "").replace(/[&<>"]/g, (c) => ({ "&": "&", "<": "<", ">": ">", '"': """ }[c])); +} +function trunc(s, n) { + s = String(s ?? ""); + return s.length > n ? s.slice(0, n - 1) + "…" : s; +} +function el(id) { return document.getElementById(id); } +function svgEl(w, h, body) { + return `${body}`; +} + +/* ----------------------------------------------------------------- charts */ + +function lineChart(rows) { + // rows: [{Month, Billed, Effective}] + const W = 760, H = 280; + const m = { l: 56, r: 18, t: 16, b: 34 }; + const iw = W - m.l - m.r, ih = H - m.t - m.b; + if (!rows || rows.length === 0) return emptyChart(W, H); + const max = Math.max(...rows.map((r) => Math.max(r.Billed || 0, r.Effective || 0)), 1); + const yMax = max * 1.12; + const n = rows.length; + const x = (i) => m.l + (n === 1 ? iw / 2 : (i / (n - 1)) * iw); + const y = (v) => m.t + ih - (v / yMax) * ih; + + let g = ""; + // gridlines + y ticks + const ticks = 4; + for (let t = 0; t <= ticks; t++) { + const val = (yMax / ticks) * t; + const yy = y(val); + g += ``; + g += `${fmtMoney(val)}`; + } + // x labels (thin out if crowded) + const step = Math.ceil(n / 12); + rows.forEach((r, i) => { + if (i % step === 0 || i === n - 1) { + g += `${esc(fmtMonth(r.Month))}`; + } + }); + // area under effective + const ptsE = rows.map((r, i) => `${x(i)},${y(r.Effective || 0)}`); + const area = `M${m.l},${y(0)} L${ptsE.join(" L")} L${x(n - 1)},${y(0)} Z`; + g += ``; + // billed line (muted, dashed) + const ptsB = rows.map((r, i) => `${x(i)},${y(r.Billed || 0)}`).join(" L"); + g += ``; + // effective line + g += ``; + // dots + hover titles + rows.forEach((r, i) => { + g += `${esc(fmtMonth(r.Month))}\nEffective ${fmtMoneyFull(r.Effective)}\nBilled ${fmtMoneyFull(r.Billed)}`; + }); + const legend = legendHtml([ + { label: "Effective cost", color: PALETTE[0] }, + { label: "Billed cost", color: "var(--muted)" }, + ]); + return svgEl(W, H, g) + legend; +} + +function hbar(rows, nameKey, valKey, opts = {}) { + const data = (rows || []).map((r) => ({ name: String(r[nameKey] ?? "—"), val: +r[valKey] || 0 })) + .filter((r) => r.val > 0); + if (data.length === 0) return `

No data in range.

`; + const max = Math.max(...data.map((d) => d.val), 1); + const total = data.reduce((s, d) => s + d.val, 0); + const rowH = 30, padR = 64, nameW = opts.nameW ?? 142; + const W = 540, H = data.length * rowH + 6; + const barX = nameW + 8, barW = W - barX - padR; + let g = ""; + data.forEach((d, i) => { + const yy = i * rowH + 4; + const cy = yy + rowH / 2; + const w = Math.max(2, (d.val / max) * barW); + const color = opts.color || PALETTE[i % PALETTE.length]; + const pct = total > 0 ? (d.val / total) : 0; + g += ``; + g += `${esc(trunc(d.name, 20))}${esc(d.name)}`; + g += `${esc(d.name)}\n${fmtMoneyFull(d.val)} · ${fmtPct(pct)}`; + g += `${fmtMoney(d.val)}`; + g += ``; + }); + return svgEl(W, H, g); +} + +function donut(slices, opts = {}) { + const data = (slices || []).filter((s) => (+s.value || 0) > 0); + const total = data.reduce((s, d) => s + (+d.value || 0), 0); + if (total <= 0) return `

No data in range.

`; + const size = 180, cx = size / 2, cy = size / 2, R = 80, r = 50; + let a0 = 0, g = ""; + if (data.length === 1) { + g += `${esc(data[0].label)}\n${fmtMoneyFull(data[0].value)} · 100%`; + } else { + data.forEach((d) => { + const frac = d.value / total; + const a1 = a0 + frac * 360; + g += `${esc(d.label)}\n${fmtMoneyFull(d.value)} · ${fmtPct(frac)}`; + a0 = a1; + }); + } + const centerBig = opts.centerBig ?? fmtMoney(total); + const centerSmall = opts.centerSmall ?? "total"; + g += `${esc(centerBig)}`; + g += `${esc(centerSmall)}`; + const legend = legendHtml(data.map((d) => ({ + label: d.label, color: d.color, + value: opts.valueFmt ? opts.valueFmt(d) : `${fmtMoney(d.value)} · ${fmtPct(d.value / total)}`, + }))); + return `
${svgEl(size, size, g)}
${legend}
`; +} + +function donutSeg(cx, cy, R, r, a0, a1) { + const polar = (rad, ang) => { + const a = ((ang - 90) * Math.PI) / 180; + return [cx + rad * Math.cos(a), cy + rad * Math.sin(a)]; + }; + const large = a1 - a0 > 180 ? 1 : 0; + const [x0, y0] = polar(R, a0), [x1, y1] = polar(R, a1); + const [x2, y2] = polar(r, a1), [x3, y3] = polar(r, a0); + return `M${x0} ${y0} A${R} ${R} 0 ${large} 1 ${x1} ${y1} L${x2} ${y2} A${r} ${r} 0 ${large} 0 ${x3} ${y3} Z`; +} + +function legendHtml(items) { + return `
${items.map((it) => + `${esc(it.label)}${ + it.value ? `${esc(it.value)}` : ""}`).join("")}
`; +} + +// Generic data table. cols: [{label, align?, get:(row,i)=>htmlString}]. rows: any[]. +function tableHtml(cols, rows, emptyMsg = "No data in range.") { + if (!rows || rows.length === 0) return `

${esc(emptyMsg)}

`; + const head = cols.map((c) => `${esc(c.label)}`).join(""); + const body = rows.map((r, i) => `${cols.map((c) => `${c.get(r, i)}`).join("")}`).join(""); + return `${head}${body}
`; +} + +function emptyChart(W, H) { + return svgEl(W, H, `No data`); +} + +function tokenTrendChart(rows) { + // rows: [{Month, Tokens, Cost}] — bars = token volume (left axis), line = AI cost (right axis) + const W = 760, H = 280; + const m = { l: 56, r: 58, t: 16, b: 34 }; + const iw = W - m.l - m.r, ih = H - m.t - m.b; + if (!rows || rows.length === 0) return emptyChart(W, H); + const tokMax = Math.max(...rows.map((r) => r.Tokens || 0), 1) * 1.14; + const costMax = Math.max(...rows.map((r) => r.Cost || 0), 1) * 1.14; + const n = rows.length; + const cx = (i) => m.l + ((i + 0.5) / n) * iw; + const yTok = (v) => m.t + ih - (v / tokMax) * ih; + const yCost = (v) => m.t + ih - (v / costMax) * ih; + const bw = Math.max(4, (iw / n) * 0.62); + + let g = ""; + const ticks = 4; + for (let t = 0; t <= ticks; t++) { + const yy = m.t + ih - (ih / ticks) * t; + g += ``; + g += `${fmtTokens((tokMax / ticks) * t)}`; + g += `${fmtMoney((costMax / ticks) * t)}`; + } + // token bars (left axis) + rows.forEach((r, i) => { + const top = yTok(r.Tokens || 0); + const h = Math.max(0, m.t + ih - top); + g += `${esc(fmtMonth(r.Month))}\n${fmtTokens(r.Tokens)} tokens\n${fmtMoneyFull(r.Cost)}`; + }); + // cost line (right axis) + const pts = rows.map((r, i) => `${cx(i)},${yCost(r.Cost || 0)}`); + g += ``; + rows.forEach((r, i) => { + g += `${esc(fmtMonth(r.Month))}\n${fmtMoneyFull(r.Cost)}`; + }); + // x labels + const step = Math.ceil(n / 12); + rows.forEach((r, i) => { + if (i % step === 0 || i === n - 1) { + g += `${esc(fmtMonth(r.Month))}`; + } + }); + const legend = legendHtml([ + { label: "Token volume", color: PALETTE[2] }, + { label: "AI effective cost", color: PALETTE[3] }, + ]); + return svgEl(W, H, g) + legend; +} + +function anomalyChart(rows) { + // rows: [{Day, Cost, Flag, Baseline}] + const W = 760, H = 280; + const m = { l: 56, r: 18, t: 16, b: 34 }; + const iw = W - m.l - m.r, ih = H - m.t - m.b; + if (!rows || rows.length === 0) return emptyChart(W, H); + const max = Math.max(...rows.map((r) => Math.max(r.Cost || 0, r.Baseline || 0)), 1) * 1.12; + const n = rows.length; + const x = (i) => m.l + (n === 1 ? iw / 2 : (i / (n - 1)) * iw); + const y = (v) => m.t + ih - (v / max) * ih; + let g = ""; + const ticks = 4; + for (let t = 0; t <= ticks; t++) { + const yy = m.t + ih - (ih / ticks) * t; + g += ``; + g += `${fmtMoney((max / ticks) * t)}`; + } + // month x labels + let lastMonth = ""; + rows.forEach((r, i) => { + const mo = String(r.Day).slice(0, 7); + if (mo !== lastMonth) { + lastMonth = mo; + g += `${esc(fmtMonth(mo))}`; + } + }); + // baseline (dashed) + cost line + const base = rows.map((r, i) => `${x(i)},${y(r.Baseline || 0)}`).join(" L"); + g += ``; + const cost = rows.map((r, i) => `${x(i)},${y(r.Cost || 0)}`).join(" L"); + g += ``; + // anomaly markers + rows.forEach((r, i) => { + if (r.Flag !== 0) { + const up = r.Flag > 0; + g += `${esc(String(r.Day).slice(0, 10))}\n${fmtMoneyFull(r.Cost)} (${up ? "spike" : "drop"})\nbaseline ${fmtMoneyFull(r.Baseline)}`; + } + }); + const legend = legendHtml([ + { label: "Daily effective cost", color: PALETTE[0] }, + { label: "Expected baseline", color: "var(--muted)" }, + { label: "Spike", color: PALETTE[4] }, + { label: "Drop", color: PALETTE[5] }, + ]); + return svgEl(W, H, g) + legend; +} + +function momBars(rows) { + // rows: [{Month, EffChangePct}] — diverging bars (cost up = red, down = green) + const W = 760, H = 240; + const m = { l: 44, r: 14, t: 14, b: 34 }; + const iw = W - m.l - m.r, ih = H - m.t - m.b; + const data = (rows || []).filter((r) => isFinite(r.EffChangePct)); + if (data.length === 0) return emptyChart(W, H); + const maxAbs = Math.max(...data.map((r) => Math.abs(r.EffChangePct)), 5); + const n = data.length; + const y0 = m.t + ih / 2; // zero line + const cx = (i) => m.l + ((i + 0.5) / n) * iw; + const bw = Math.max(5, (iw / n) * 0.6); + const yScale = (v) => (v / maxAbs) * (ih / 2); + let g = ``; + data.forEach((r, i) => { + const v = r.EffChangePct; + const h = Math.abs(yScale(v)); + const yTop = v >= 0 ? y0 - h : y0; + const color = v > 0 ? PALETTE[4] : PALETTE[1]; + g += `${esc(fmtMonth(r.Month))}\n${v > 0 ? "+" : ""}${v.toFixed(1)}%`; + const step = Math.ceil(n / 12); + if (i % step === 0 || i === n - 1) { + g += `${esc(fmtMonth(r.Month))}`; + } + }); + return svgEl(W, H, g); +} + +function forecastChart(rows, splitMonth) { + // rows: [{Month, Actual, Forecast}] — actual solid up to splitMonth, forecast dashed onward + const W = 760, H = 280; + const m = { l: 56, r: 18, t: 16, b: 34 }; + const iw = W - m.l - m.r, ih = H - m.t - m.b; + if (!rows || rows.length === 0) return emptyChart(W, H); + const max = Math.max(...rows.map((r) => Math.max(r.Actual || 0, r.Forecast || 0)), 1) * 1.12; + const n = rows.length; + const x = (i) => m.l + (n === 1 ? iw / 2 : (i / (n - 1)) * iw); + const y = (v) => m.t + ih - (v / max) * ih; + let g = ""; + const ticks = 4; + for (let t = 0; t <= ticks; t++) { + const yy = m.t + ih - (ih / ticks) * t; + g += ``; + g += `${fmtMoney((max / ticks) * t)}`; + } + const splitIdx = rows.findIndex((r) => r.Month >= splitMonth); + const sIdx = splitIdx < 0 ? n - 1 : splitIdx; + // shaded forecast region + g += ``; + // actual line up to split + const actualPts = rows.slice(0, sIdx + 1).map((r, i) => `${x(i)},${y(r.Actual || 0)}`); + if (actualPts.length > 1) g += ``; + // forecast line from split onward + const fcPts = rows.slice(sIdx).map((r, i) => `${x(sIdx + i)},${y(r.Forecast || 0)}`); + if (fcPts.length > 1) g += ``; + // x labels + const step = Math.ceil(n / 12); + rows.forEach((r, i) => { + if (i % step === 0 || i === n - 1) { + g += `${esc(fmtMonth(r.Month))}`; + } + }); + rows.forEach((r, i) => { + const isFc = i >= sIdx; + g += `${esc(fmtMonth(r.Month))}\n${isFc ? "forecast " + fmtMoneyFull(r.Forecast) : "actual " + fmtMoneyFull(r.Actual)}`; + }); + const legend = legendHtml([ + { label: "Actual", color: PALETTE[0] }, + { label: "Forecast", color: PALETTE[3] }, + ]); + return svgEl(W, H, g) + legend; +} + +/* --------------------------------------------------------------- KPI calc */ + +function deriveKpis(d) { + const s = d.summary?.[0] || {}; + const list = s.List || 0, eff = s.Effective || 0, contracted = s.Contracted || 0, billed = s.Billed || 0; + const savings = list - eff; + const esr = list > 0 ? savings / list : 0; + const negotiated = list - contracted; + const commitment = contracted - eff; + + const tagMap = Object.fromEntries((d.tagged || []).map((r) => [r._t, r.Cost || 0])); + const tagged = tagMap.Tagged || 0, untagged = tagMap.Untagged || 0; + const tagTotal = tagged + untagged; + const untaggedPct = tagTotal > 0 ? untagged / tagTotal : 0; + + const priceMap = Object.fromEntries((d.pricing || []).map((r) => [r.PricingCategory, r.Cost || 0])); + const committed = priceMap.Committed || 0; + const priceTotal = Object.values(priceMap).reduce((a, b) => a + b, 0); + const coverage = priceTotal > 0 ? committed / priceTotal : 0; + + const trend = d.trend || []; + let mom = null, lastMonthVal = null, lastMonthLabel = null; + if (trend.length >= 1) { + const last = trend[trend.length - 1]; + lastMonthVal = last.Effective || 0; + lastMonthLabel = fmtMonth(last.Month); + if (trend.length >= 2) { + const prev = trend[trend.length - 2].Effective || 0; + mom = prev > 0 ? (lastMonthVal - prev) / prev : null; + } + } + + return { + billed, eff, list, contracted, savings, esr, negotiated, commitment, + tagged, untagged, untaggedPct, committed, coverage, + resources: s.Resources || 0, services: s.Services || 0, + subscriptions: s.Subscriptions || 0, regions: s.Regions || 0, + mom, lastMonthVal, lastMonthLabel, + }; +} + +function kpiCard(label, value, meta, accent) { + return `
+
${esc(label)}
+
${value}
+
${meta}
+
`; +} + +/* ---------------------------------------------------------------- render */ + +function renderOverview(p) { + const content = el("content"); + if (!p) return; + if (p.error) return renderError(p); + if (p.empty) { + content.innerHTML = `

No cost data

The Hub database has no rows yet. Ingest cost data, then refresh.

`; + return; + } + const k = deriveKpis(p.data); + + const momClass = k.mom == null ? "" : k.mom > 0 ? "neg" : "pos"; // cost up = bad + const momTxt = k.mom == null ? "—" : `${k.mom > 0 ? "▲" : "▼"} ${fmtPct(Math.abs(k.mom))}`; + + const kpis = [ + kpiCard("Effective cost", fmtMoney(k.eff), `Billed ${fmtMoney(k.billed)}`, PALETTE[0]), + kpiCard("Total savings", fmtMoney(k.savings), + `${fmtPct(k.esr)} effective savings rate`, PALETTE[1]), + kpiCard("Untagged cost", fmtPct(k.untaggedPct), + `${fmtMoney(k.untagged)} on untagged resources`, PALETTE[3]), + kpiCard("Commitment coverage", fmtPct(k.coverage), + `${fmtMoney(k.committed)} on committed rates`, PALETTE[5]), + kpiCard("Tracked resources", fmtInt(k.resources), + `${fmtInt(k.services)} services · ${fmtInt(k.subscriptions)} subs · ${fmtInt(k.regions)} regions`, PALETTE[2]), + kpiCard("Latest month", k.lastMonthVal == null ? "—" : fmtMoney(k.lastMonthVal), + k.mom == null ? esc(k.lastMonthLabel || "") : `${momTxt} vs prior · ${esc(k.lastMonthLabel)}`, PALETTE[4]), + ].join(""); + + const d = p.data; + const html = ` +
${kpis}
+ +

Understand usage & cost

FinOps Framework
+
+
+

Monthly cost trend

+

Billed vs effective cost by month — executive run-rate view.

+ ${lineChart(d.trend)} +
+
+

Top services by cost

+

Effective cost by Azure service.

+ ${hbar(d.topServices, "ServiceName", "Cost")} +
+
+

Cost by service category

+

Where spend concentrates across categories.

+ ${hbar(d.serviceCategory, "ServiceCategory", "Cost")} +
+
+ +

Optimize usage & cost

FinOps Framework
+
+
+

Top resource groups

+

Largest cost owners for allocation & accountability.

+ ${hbar(d.topResourceGroups, "x_ResourceGroupName", "Cost")} +
+
+

Cost by region

+

Regional spend for placement & sustainability review.

+ ${hbar(d.topRegions, "RegionId", "Cost")} +
+
+ +

Quantify business value

FinOps Framework
+
+
+

Rate coverage

+

Committed vs on-demand (standard) effective cost.

+ ${donut([ + { label: "Committed", value: k.committed, color: PALETTE[1] }, + { label: "On-demand", value: Math.max(0, k.eff - k.committed), color: PALETTE[0] }, + ], { centerBig: fmtPct(k.coverage), centerSmall: "covered" })} +
+
+

Savings breakdown

+

List → effective, by discount type.

+ ${savingsTable(k)} +
+
+

Cost allocation

+

Tagged vs untagged effective cost.

+ ${donut([ + { label: "Tagged", value: k.tagged, color: PALETTE[1] }, + { label: "Untagged", value: k.untagged, color: PALETTE[3] }, + ], { centerBig: fmtPct(1 - k.untaggedPct), centerSmall: "tagged" })} +
+
+ `; + content.innerHTML = html; +} + +function savingsTable(k) { + const row = (label, val, accent) => + `
+ ${ + accent ? `` : ""}${esc(label)} + ${fmtMoney(val)}
`; + return `
+ ${row("List cost", k.list, "var(--muted)")} + ${row("Negotiated savings", k.negotiated, PALETTE[8])} + ${row("Commitment savings", k.commitment, PALETTE[1])} + ${row("Effective cost", k.eff, PALETTE[0])} +
+ Effective savings rate + ${fmtPct(k.esr)} +
+
`; +} + +/* ----------------------------------------------------- tokenomics render */ + +function deriveTokenKpis(d) { + const s = d.summary?.[0] || {}; + const tokens = s.Tokens || 0, eff = s.Effective || 0; + const cloud = d.totalCloud?.[0]?.Effective || 0; + const dir = Object.fromEntries((d.direction || []).map((r) => [r.Direction, r])); + const inTok = dir["Input"]?.Tokens || 0; + const cachedTok = dir["Cached input"]?.Tokens || 0; + const cachedShare = inTok + cachedTok > 0 ? cachedTok / (inTok + cachedTok) : 0; + return { + tokens, eff, cloud, + blendedPer1K: tokens > 0 ? eff / tokens * 1000 : 0, + cachedShare, + aiShare: cloud > 0 ? eff / cloud : 0, + models: s.Models || 0, + resources: s.Resources || 0, + }; +} + +function renderTokenomics(p) { + const content = el("content"); + if (!p) return; + if (p.error) return renderError(p); + if (p.empty) { + content.innerHTML = `

No AI token data

+

No Azure OpenAI token meters were found in the Hub database for this period.

+

Tokenomics tracks meters where x_SkuMeterSubcategory contains “OpenAI” and the SKU is billed in tokens. Ingest Azure OpenAI usage, then refresh.

`; + return; + } + const d = p.data; + const k = deriveTokenKpis(d); + + const dirColors = { "Input": PALETTE[0], "Cached input": PALETTE[1], "Output": PALETTE[3], "Other": PALETTE[6] }; + const dirSlices = (d.direction || []).map((r) => ({ + label: r.Direction, value: r.Tokens || 0, cost: r.Cost || 0, color: dirColors[r.Direction] || PALETTE[6], + })); + + const kpis = [ + kpiCard("AI token cost", fmtMoney(k.eff), `${fmtPct(k.aiShare, 2)} of all cloud cost`, PALETTE[2]), + kpiCard("Total tokens", fmtTokens(k.tokens), `across ${fmtInt(k.models)} model SKUs`, PALETTE[0]), + kpiCard("Blended rate", fmtPerM(k.blendedPer1K), `per 1M tokens (effective)`, PALETTE[5]), + kpiCard("Cached input", fmtPct(k.cachedShare), + `${fmtPct(k.cachedShare)} of input tokens cached`, PALETTE[1]), + kpiCard("AI resources", fmtInt(k.resources), `Azure OpenAI deployments`, PALETTE[4]), + kpiCard("Models in use", fmtInt(k.models), `distinct token SKUs`, PALETTE[8]), + ].join(""); + + content.innerHTML = ` +
${kpis}
+ +

AI token economics

Token Consumption Metrics KPI
+
+
+

Token volume & AI cost trend

+

Monthly token consumption (bars) and effective AI cost (line).

+ ${tokenTrendChart(d.trend)} +
+
+

AI cost by model

+

Effective cost per model family.

+ ${hbar((d.models || []).map((m) => ({ Model: m.Model, Cost: m.Cost })), "Model", "Cost")} +
+
+

Token direction mix

+

Input vs cached input vs output — by token volume.

+ ${donut(dirSlices, { + centerBig: fmtTokens(k.tokens), centerSmall: "tokens", + valueFmt: (s) => `${fmtTokens(s.value)} · ${fmtMoney(s.cost)}`, + })} +
+
+ +

Model efficiency

Rate & usage optimization
+
+
+

Cost per 1M tokens by model

+

Unit economics for model selection — sorted by effective cost.

+ ${tokenModelTable(d.models, k.eff)} +
+
+ `; +} + +function tokenModelTable(models, totalCost) { + const rows = (models || []).filter((m) => (m.Tokens || 0) > 0); + if (rows.length === 0) return `

No token data in range.

`; + const maxPer1K = Math.max(...rows.map((m) => m.CostPer1K || 0), 1e-9); + const body = rows.map((m, i) => { + const color = PALETTE[i % PALETTE.length]; + const share = totalCost > 0 ? (m.Cost || 0) / totalCost : 0; + const barW = Math.max(2, ((m.CostPer1K || 0) / maxPer1K) * 90); + return ` + ${esc(m.Model)} + ${fmtTokens(m.Tokens)} + ${fmtMoneyFull(m.Cost)} + ${fmtPerM(m.CostPer1K)} + ${fmtPct(share)} + `; + }).join(""); + return ` + + ${body} +
ModelTokensEffective cost$ / 1M tokens% of AI cost
`; +} + +/* --------------------------------------------- anomalies & forecast render */ + +function renderAnomaly(p) { + const content = el("content"); + if (!p) return; + if (p.error) return renderError(p); + if (p.empty) { + content.innerHTML = `

No cost data

The Hub database has no rows yet.

`; + return; + } + const d = p.data; + const daily = d.daily || []; + const anomDays = daily.filter((r) => r.Flag !== 0); + const totalCost = daily.reduce((a, r) => a + (r.Cost || 0), 0); + const anomCost = anomDays.reduce((a, r) => a + (r.Cost || 0), 0); + const variance = Math.abs(anomDays.reduce((a, r) => a + ((r.Cost || 0) - (r.Baseline || 0)), 0)); + const rate = totalCost > 0 ? anomCost / totalCost : 0; + + const fc = d.forecast || []; + const dataMaxMonth = (p.window?.dataMax || "").slice(0, 7); + const nextFc = fc.find((r) => r.Month > dataMaxMonth); + + const mc = (d.monthlyChange || []).filter((r) => isFinite(r.EffChangePct)); + // last complete month (skip the partial dataMax month for the headline KPI) + const completeMc = mc.filter((r) => r.Month < dataMaxMonth); + const lastMc = completeMc[completeMc.length - 1] || mc[mc.length - 1]; + + const fr = d.freshness?.[0] || {}; + const p50Days = fr.P50 != null ? fr.P50 / 24 : null; + + const kpis = [ + kpiCard("Anomaly detection rate", fmtPct(rate, 2), + `${anomDays.length} flagged of ${daily.length} days`, PALETTE[4]), + kpiCard("Unpredicted variance", fmtMoney(variance), + `net spend vs baseline on anomaly days`, PALETTE[3]), + kpiCard("Anomaly days", fmtInt(anomDays.length), + `${fmtMoney(anomCost)} on flagged days`, PALETTE[5]), + kpiCard("Last month change", lastMc ? `${lastMc.EffChangePct > 0 ? "+" : ""}${lastMc.EffChangePct.toFixed(1)}%` : "—", + lastMc ? `effective cost · ${esc(fmtMonth(lastMc.Month))}` : "", PALETTE[lastMc && lastMc.EffChangePct > 0 ? 4 : 1]), + kpiCard("Forecast next month", nextFc ? fmtMoney(nextFc.Forecast) : "—", + nextFc ? `projected · ${esc(fmtMonth(nextFc.Month))}` : "", PALETTE[0]), + kpiCard("Visibility delay", p50Days != null ? `${p50Days.toFixed(0)}d` : "—", + `median ingestion lag (P50)`, PALETTE[2]), + ].join(""); + + content.innerHTML = ` +
${kpis}
+ +

Cost anomalies

Anomaly Management capability
+
+
+

Daily cost & detected anomalies

+

Daily effective cost vs the expected baseline (STL decomposition); markers flag spikes & drops.

+ ${anomalyChart(daily)} +
+
+ +

Trend & forecast

Forecasting · Data freshness
+
+
+

Month-over-month change

+

Effective cost % change vs prior month (red = increase).

+ ${momBars(mc)} +
+
+

Cost forecast

+

Monthly effective cost, actual vs forecast (next 3 months).

+ ${forecastChart(fc, dataMaxMonth)} +
+
+ `; +} + +/* ----------------------------------------------- usage & unit economics render */ + +function renderUsage(p) { + const content = el("content"); + if (!p) return; + if (p.error) return renderError(p); + if (p.empty) { + content.innerHTML = `

No cost data

The Hub database has no rows yet.

`; + return; + } + const d = p.data; + const c = d.compute?.[0] || {}; + const s = d.storage?.[0] || {}; + const coreHours = c.CoreHours || 0; + const hourlyPerCore = coreHours > 0 ? c.ComputeEff / coreHours : 0; + const effPerCore = coreHours > 0 ? (c.ComputeEff + (c.UnusedCommit || 0)) / coreHours : 0; + const gbMonths = s.GBMonths || 0; + const perGB = gbMonths > 0 ? s.Cost / gbMonths : 0; + const total = d.total?.[0]?.Total || 0; + + const kpis = [ + kpiCard("Hourly cost / core", `$${hourlyPerCore.toFixed(3)}`, + `per consumed vCPU-hour`, PALETTE[0]), + kpiCard("Effective cost / core", `$${effPerCore.toFixed(3)}`, + `incl. unused commitment`, PALETTE[2]), + kpiCard("Compute core-hours", fmtTokens(coreHours), + `${fmtMoney(c.ComputeEff)} VM usage`, PALETTE[1]), + kpiCard("Storage rate", `$${(perGB * 1024).toFixed(3)}`, + `per TB-month (effective)`, PALETTE[5]), + kpiCard("Storage volume", `${fmtTokens(gbMonths)}`, + `GB-months stored`, PALETTE[8]), + kpiCard("Storage cost", fmtMoney(s.Cost), + `effective storage spend`, PALETTE[3]), + ].join(""); + + const typeRows = (d.topResourceTypes || []).map((r) => ({ + type: r.ResourceType, count: r.Count || 0, cost: r.Cost || 0, + pct: total > 0 ? (r.Cost || 0) / total : 0, + })); + const typeTable = tableHtml([ + { label: "Resource type", align: "left", get: (r, i) => `${esc(r.type)}` }, + { label: "Resources", get: (r) => fmtInt(r.count) }, + { label: "Effective cost", get: (r) => fmtMoneyFull(r.cost) }, + { label: "% of total", get: (r) => fmtPct(r.pct) }, + ], typeRows); + + const tierColors = { "Frequent": PALETTE[1], "Infrequent": PALETTE[5], "Unclassified": PALETTE[6] }; + const tierSlices = (d.storageTiers || []).map((r) => ({ label: r.Tier, value: r.Cost || 0, color: tierColors[r.Tier] || PALETTE[6] })); + const freqShare = (() => { + const t = tierSlices.reduce((a, x) => a + x.value, 0); + const f = (d.storageTiers || []).find((r) => r.Tier === "Frequent"); + return t > 0 ? (f?.Cost || 0) / t : 0; + })(); + + content.innerHTML = ` +
${kpis}
+ +

Usage & unit economics

Usage Optimization · Unit Economics
+
+
+

Top resource types by cost

+

Resource count and effective spend per resource type.

+ ${typeTable} +
+
+

Compute cost per core by VM series

+

Effective cost per vCPU-hour — highlights expensive (e.g. GPU) cores.

+ ${hbar(d.perCoreSeries, "x_SkuMeterSubcategory", "PerCore")} +
+
+

Storage tier distribution

+

Effective storage cost by access tier (${fmtPct(freqShare)} classified frequent).

+ ${donut(tierSlices, { centerBig: fmtMoney(s.Cost), centerSmall: "storage" })} +
+
+ `; +} + +function renderRate(p) { + const content = el("content"); + if (!p) return; + if (p.error) return renderError(p); + if (p.empty) { + content.innerHTML = `

No cost data

The Hub database has no rows yet.

`; + return; + } + const d = p.data; + const s = d.savings?.[0] || {}; + const cm = d.commitment?.[0] || {}; + const cc = d.computeCoverage?.[0] || {}; + const esr = s.List > 0 ? s.Total / s.List : 0; + const cmTotal = cm.Total || 0; + const util = cmTotal > 0 ? (cmTotal - (cm.Unused || 0)) / cmTotal : 0; + const waste = cmTotal > 0 ? (cm.Unused || 0) / cmTotal : 0; + const coverage = cc.Contracted > 0 ? cc.Committed / cc.Contracted : 0; + const coreTotal = (d.coreHours || []).reduce((a, r) => a + (r.CoreHours || 0), 0); + const committedCore = (d.coreHours || []).filter((r) => r.t !== "On Demand").reduce((a, r) => a + (r.CoreHours || 0), 0); + const coreShare = coreTotal > 0 ? committedCore / coreTotal : 0; + + const kpis = [ + kpiCard("Effective savings rate", fmtPct(esr), + `${fmtMoney(s.Total)} total savings`, PALETTE[1]), + kpiCard("Total savings", fmtMoney(s.Total), + `of ${fmtMoney(s.List)} list cost`, PALETTE[2]), + kpiCard("Commitment utilization", fmtPct(util), + `${fmtMoney(cmTotal - (cm.Unused || 0))} of ${fmtMoney(cmTotal)} used`, PALETTE[0]), + kpiCard("Commitment waste", fmtPct(waste), + `${fmtMoney(cm.Unused)} unused`, PALETTE[3]), + kpiCard("Compute coverage", fmtPct(coverage), + `compute spend on commitments`, PALETTE[5]), + kpiCard("Committed core-hours", fmtPct(coreShare), + `RI + savings plan vs on-demand`, PALETTE[8]), + ].join(""); + + const savingsBreak = `
+ ${rateRow("List cost", s.List, "var(--muted)")} + ${rateRow("Negotiated savings", s.Negotiated, PALETTE[8])} + ${rateRow("Commitment savings", s.Commitment, PALETTE[1])} + ${rateRow("Effective cost", s.Effective, PALETTE[0])} +
+ Effective savings rate + ${fmtPct(esr)} +
+
`; + + const coreColors = { "On Demand": PALETTE[0], "Reservation": PALETTE[1], "Savings Plan": PALETTE[4] }; + const coreSlices = (d.coreHours || []).map((r) => ({ label: r.t, value: r.CoreHours || 0, color: coreColors[r.t] || PALETTE[6] })); + + content.innerHTML = ` +
${kpis}
+ +

Rate optimization

Rate Optimization capability
+
+
+

Savings breakdown

+

List → effective cost by discount type (effective savings rate).

+ ${savingsBreak} +
+
+

Commitment utilization

+

Used vs unused commitment effective cost.

+ ${donut([ + { label: "Used", value: cmTotal - (cm.Unused || 0), color: PALETTE[1] }, + { label: "Unused (waste)", value: cm.Unused || 0, color: PALETTE[3] }, + ], { centerBig: fmtPct(util), centerSmall: "utilized" })} +
+
+

Core-hour coverage

+

Consumed core-hours by commitment type.

+ ${donut(coreSlices, { + centerBig: fmtPct(coreShare), centerSmall: "committed", + valueFmt: (s) => `${fmtTokens(s.value)} core-hrs`, + })} +
+
+

Underutilized commitments

+

Reservations & plans with the most unused cost.

+ ${hbar(d.byCommitment, "CommitmentDiscountName", "Unused")} +
+
+ `; +} + +function rateRow(label, val, accent) { + return `
+ ${ + accent ? `` : ""}${esc(label)} + ${fmtMoney(val)}
`; +} + +/* ----------------------------------------------------- allocation render */ + +function renderAllocation(p) { + const content = el("content"); + if (!p) return; + if (p.error) return renderError(p); + if (p.empty) { + content.innerHTML = `

No cost data

The Hub database has no rows yet.

`; + return; + } + const d = p.data; + const c = d.core?.[0] || {}; + const total = c.Total || 0; + const aai = total > 0 ? c.Attributed / total : 0; + const untaggedPct = total > 0 ? c.Untagged / total : 0; + const unallocPct = total > 0 ? (total - c.Attributed) / total : 0; + const compliancePct = total > 0 ? c.Compliant / total : 0; + + const kpis = [ + kpiCard("Allocation accuracy", fmtPct(aai), + `directly attributed effective cost`, PALETTE[1]), + kpiCard("Untagged cost", fmtPct(untaggedPct), + `${fmtMoney(c.Untagged)} with no tags`, PALETTE[3]), + kpiCard("Unallocated cost", fmtPct(unallocPct), + `${fmtMoney(total - c.Attributed)} lacks allocation evidence`, PALETTE[4]), + kpiCard("Tag policy compliance", fmtPct(compliancePct), + `keys: CostCenter · env · org`, PALETTE[5]), + kpiCard("Subscriptions", fmtInt(c.Subs), + `billing scopes in range`, PALETTE[0]), + kpiCard("Allocated cost", fmtMoney(c.Attributed), + `of ${fmtMoney(total)} total`, PALETTE[2]), + ].join(""); + + const hierRows = (d.hierarchy || []).map((r) => ({ + org: r.Org || "—", project: r.Project || "—", env: r.Env || "—", cost: r.Cost || 0, + pct: total > 0 ? (r.Cost || 0) / total : 0, + })); + const hierTable = tableHtml([ + { label: "Org", align: "left", get: (r, i) => `${esc(r.org)}` }, + { label: "Project", align: "left", get: (r) => esc(r.project) }, + { label: "Environment", align: "left", get: (r) => esc(r.env) }, + { label: "Effective cost", get: (r) => fmtMoneyFull(r.cost) }, + { label: "% of total", get: (r) => fmtPct(r.pct) }, + ], hierRows); + + content.innerHTML = ` +
${kpis}
+ +

Cost allocation

Allocation capability
+
+
+

Cost by financial hierarchy

+

Org → project → environment (from resource tags), with share of total.

+ ${hierTable} +
+
+

Tagging coverage

+

Tagged vs untagged effective cost.

+ ${donut([ + { label: "Tagged", value: total - c.Untagged, color: PALETTE[1] }, + { label: "Untagged", value: c.Untagged, color: PALETTE[3] }, + ], { centerBig: fmtPct(1 - untaggedPct), centerSmall: "tagged" })} +
+
+

Cost by tag key

+

Effective cost touched by each governance tag.

+ ${hbar(d.tagKeys, "k", "Cost")} +
+
+

Cost by subscription

+

Spend per billing scope for showback.

+ ${hbar(d.bySubscription, "SubAccountName", "Cost")} +
+
+ `; +} + +function renderError(p) { + el("content").innerHTML = `
+

Can’t reach the FinOps hub

+

The dashboard queried ${esc(p.clusterUri || "")} (database ${esc(p.database || "Hub")}) but the request failed.

+

Start the Kusto emulator and run Initialize-FinOpsHubLocal, then refresh.

+ 
${esc(p.error)}
+
`; +} + +/* ----------------------------------------------------------------- driver */ + +const ENDPOINT = { + overview: "/api/view?name=overview", + allocation: "/api/view?name=allocation", + rate: "/api/view?name=rate", + usage: "/api/view?name=usage", + anomaly: "/api/view?name=anomaly", + tokenomics: "/api/view?name=tokenomics", +}; + +function currentPayload() { + return state.cache[state.tab]?.[state.preset]; +} + +function render() { + const p = currentPayload(); + if (!p) return; + if (state.tab === "tokenomics") renderTokenomics(p); + else if (state.tab === "allocation") renderAllocation(p); + else if (state.tab === "rate") renderRate(p); + else if (state.tab === "usage") renderUsage(p); + else if (state.tab === "anomaly") renderAnomaly(p); + else renderOverview(p); +} + +async function load() { + const tab = state.tab, preset = state.preset; + if (state.cache[tab]?.[preset]) { updateChrome(); render(); return; } + state.cache[tab] = state.cache[tab] || {}; + state.loading = true; + setRefreshSpinning(true); + el("content").innerHTML = `
Loading ${tab} data…
`; + try { + const res = await fetch(`${ENDPOINT[tab]}&preset=${encodeURIComponent(preset)}`); + state.cache[tab][preset] = await res.json(); + } catch (err) { + state.cache[tab][preset] = { error: String(err && err.message ? err.message : err) }; + } + state.loading = false; + setRefreshSpinning(false); + updateChrome(); + render(); +} + +function setRefreshSpinning(on) { + const b = el("refresh"); + if (b) b.innerHTML = on ? `↻ Refresh` : `↻ Refresh`; +} + +function updateChrome() { + const p = currentPayload(); + const w = p && p.window; + if (w && w.dataMin) { + el("source-line").innerHTML = + `Hub database · ${esc(window.__cfg?.clusterUri || "localhost:8082")} · ${esc(fmtDayRange(w.dataMin, w.dataMax))}`; + let meta; + if (state.tab === "tokenomics" && p.data) { + const s = p.data.summary?.[0] || {}; + meta = `${fmtTokens(s.Tokens)} tokens · ${fmtMoney(s.Effective)} AI cost`; + } else { + meta = `${fmtInt(w.rows)} rows in range`; + } + el("footer-meta").textContent = + `${meta} · window ${w.start} → ${w.end} · refreshed ${new Date(p.generatedAt).toLocaleTimeString()}`; + } else if (p && p.error) { + el("source-line").textContent = "Connection failed — see panel below."; + el("footer-meta").textContent = ""; + } +} + +function wireControls() { + el("preset").addEventListener("click", (e) => { + const btn = e.target.closest("button[data-preset]"); + if (!btn || state.loading) return; + state.preset = btn.dataset.preset; + [...el("preset").querySelectorAll("button")].forEach((b) => b.classList.toggle("active", b === btn)); + load(); + }); + el("tabs").addEventListener("click", (e) => { + const btn = e.target.closest("button[data-tab]"); + if (!btn || state.loading || btn.dataset.tab === state.tab) return; + state.tab = btn.dataset.tab; + [...el("tabs").querySelectorAll("button")].forEach((b) => b.classList.toggle("active", b === btn)); + load(); + }); + el("refresh").addEventListener("click", () => { + if (state.loading) return; + if (state.cache[state.tab]) delete state.cache[state.tab][state.preset]; // force re-query + load(); + }); + let t; + window.addEventListener("resize", () => { clearTimeout(t); t = setTimeout(render, 180); }); +} + +async function init() { + try { + const cfg = await fetch("/api/config").then((r) => r.json()); + window.__cfg = cfg; + } catch { window.__cfg = {}; } + wireControls(); + load(); +} + +init(); diff --git a/.github/extensions/ftk-local-dashboard/public/index.html b/.github/extensions/ftk-local-dashboard/public/index.html new file mode 100644 index 000000000..24771763d --- /dev/null +++ b/.github/extensions/ftk-local-dashboard/public/index.html @@ -0,0 +1,46 @@ + + + + + + FinOps hub — local dashboard + + + +
+
+

FinOps hub · local

+

Connecting to the Kusto emulator…

+
+
+
+ + + + +
+ +
+
+ + + +
+
Loading cost data…
+
+ +
+ + Grounded in the FinOps Framework domains & the FinOps toolkit query catalog. +
+ + + + From a133f6618e34f6977902fc1c33829f375eec69ae Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 25 Jun 2026 09:34:23 -0700 Subject: [PATCH 33/51] feat(dashboard): add KPI interpretive annotations (T-5001.1) - Add kpiThreshold() helper: maps pct to threshold-green/amber/red - Add isPartialMonth() helper: true when today < last day of current month - Extend kpiCard() with optional 5th thresholdClass param - Overview: Untagged cost gets threshold-green/amber/red badge (<10/10-25/>25%) - Overview: Commitment coverage meta updated to 'of compute spend' - Overview: Latest month appends 'partial month' (amber) when month is incomplete - Rate tab: Effective savings rate meta adds 'vs. list price' sub-label - Rate tab: Commitment waste gets threshold badge (<10/10-20/>20%) + 'of commitment spend' sub-label - Allocation tab: Untagged cost gets same threshold badge as overview - CSS: add .threshold-green/.threshold-amber/.threshold-red border-color rules Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../ftk-local-dashboard/public/app.css | 39 +++++++++++------- .../ftk-local-dashboard/public/app.js | 41 ++++++++++++++----- 2 files changed, 55 insertions(+), 25 deletions(-) diff --git a/.github/extensions/ftk-local-dashboard/public/app.css b/.github/extensions/ftk-local-dashboard/public/app.css index f22d5ac97..9fb866bbd 100644 --- a/.github/extensions/ftk-local-dashboard/public/app.css +++ b/.github/extensions/ftk-local-dashboard/public/app.css @@ -70,7 +70,8 @@ body { border: 0; background: transparent; color: var(--text-color-default, #1f2328); - padding: 6px 13px; + padding: 10px 13px; + min-height: 44px; font-size: 12.5px; font-weight: 600; cursor: pointer; @@ -85,7 +86,8 @@ body { border: 1px solid var(--grid); background: var(--card-bg); color: var(--text-color-default, #1f2328); - padding: 6px 13px; + padding: 10px 13px; + min-height: 44px; border-radius: 9px; font-size: 12.5px; font-weight: 600; @@ -108,7 +110,8 @@ body { color: var(--muted); font-size: 13.5px; font-weight: 620; - padding: 9px 14px 11px; + padding: 12px 14px 14px; + min-height: 44px; cursor: pointer; position: relative; border-radius: 8px 8px 0 0; @@ -135,21 +138,13 @@ body { background: var(--card-bg); border: 1px solid var(--grid); border-radius: var(--radius); + box-shadow: inset 0 3px 0 var(--kpi-accent, var(--accent)); padding: 15px 16px 14px; position: relative; overflow: hidden; } -.kpi::before { - content: ""; - position: absolute; - left: 0; top: 0; bottom: 0; - width: 3px; - background: var(--kpi-accent, var(--accent)); -} .kpi .label { font-size: 11.5px; - text-transform: uppercase; - letter-spacing: 0.04em; color: var(--muted); font-weight: 600; } @@ -165,6 +160,11 @@ body { .kpi .meta .neg { color: var(--neg); font-weight: 650; } .kpi .meta .warn { color: var(--warn); font-weight: 650; } +/* ---------- KPI threshold badges ---------- */ +.threshold-green { border-color: var(--pos); } +.threshold-amber { border-color: var(--warn); } +.threshold-red { border-color: var(--neg); } + /* ---------- sections & panels ---------- */ .section-title { display: flex; @@ -181,8 +181,6 @@ body { .section-title .domain { font-size: 11px; color: var(--muted); - text-transform: uppercase; - letter-spacing: 0.05em; } .panel-grid { @@ -244,7 +242,7 @@ svg { display: block; width: 100%; overflow: visible; } .dtable { width: 100%; border-collapse: collapse; font-size: 12.5px; font-variant-numeric: tabular-nums; } .dtable th, .dtable td { padding: 9px 10px; text-align: right; border-bottom: 1px solid var(--grid); white-space: nowrap; } .dtable th:first-child, .dtable td:first-child { text-align: left; } -.dtable thead th { color: var(--muted); font-weight: 600; font-size: 11px; text-transform: uppercase; letter-spacing: 0.03em; } +.dtable thead th { color: var(--muted); font-weight: 600; font-size: 11px; } .dtable tbody tr:hover { background: color-mix(in srgb, var(--accent) 6%, transparent); } .dtable .model { display: inline-flex; align-items: center; gap: 7px; font-weight: 580; } .dtable .swatch { width: 9px; height: 9px; border-radius: 3px; flex: none; } @@ -298,3 +296,14 @@ svg { display: block; width: 100%; overflow: visible; } .spin { animation: spin 0.8s linear infinite; display: inline-block; } @keyframes spin { to { transform: rotate(360deg); } } + +/* ---------- accessibility ---------- */ +*:focus-visible { + outline: 2px solid var(--accent); + outline-offset: 2px; + border-radius: 3px; +} + +@media (prefers-reduced-motion: reduce) { + .spin { animation: none; } +} diff --git a/.github/extensions/ftk-local-dashboard/public/app.js b/.github/extensions/ftk-local-dashboard/public/app.js index 38e435456..0fa9de4ef 100644 --- a/.github/extensions/ftk-local-dashboard/public/app.js +++ b/.github/extensions/ftk-local-dashboard/public/app.js @@ -410,8 +410,20 @@ function deriveKpis(d) { }; } -function kpiCard(label, value, meta, accent) { - return `
+function kpiThreshold(pct, greenMax, amberMax) { + if (pct < greenMax) return "threshold-green"; + if (pct < amberMax) return "threshold-amber"; + return "threshold-red"; +} + +function isPartialMonth() { + const now = new Date(); + return now.getDate() < new Date(now.getFullYear(), now.getMonth() + 1, 0).getDate(); +} + +function kpiCard(label, value, meta, accent, thresholdClass) { + const cls = thresholdClass ? ` ${thresholdClass}` : ""; + return `
${esc(label)}
${value}
${meta}
@@ -433,18 +445,21 @@ function renderOverview(p) { const momClass = k.mom == null ? "" : k.mom > 0 ? "neg" : "pos"; // cost up = bad const momTxt = k.mom == null ? "—" : `${k.mom > 0 ? "▲" : "▼"} ${fmtPct(Math.abs(k.mom))}`; + const partialHtml = isPartialMonth() ? ` · partial month` : ""; + const kpis = [ kpiCard("Effective cost", fmtMoney(k.eff), `Billed ${fmtMoney(k.billed)}`, PALETTE[0]), kpiCard("Total savings", fmtMoney(k.savings), `${fmtPct(k.esr)} effective savings rate`, PALETTE[1]), kpiCard("Untagged cost", fmtPct(k.untaggedPct), - `${fmtMoney(k.untagged)} on untagged resources`, PALETTE[3]), + `${fmtMoney(k.untagged)} on untagged resources`, PALETTE[3], + kpiThreshold(k.untaggedPct, 0.10, 0.25)), kpiCard("Commitment coverage", fmtPct(k.coverage), - `${fmtMoney(k.committed)} on committed rates`, PALETTE[5]), + `${fmtMoney(k.committed)} of compute spend`, PALETTE[5]), kpiCard("Tracked resources", fmtInt(k.resources), `${fmtInt(k.services)} services · ${fmtInt(k.subscriptions)} subs · ${fmtInt(k.regions)} regions`, PALETTE[2]), kpiCard("Latest month", k.lastMonthVal == null ? "—" : fmtMoney(k.lastMonthVal), - k.mom == null ? esc(k.lastMonthLabel || "") : `${momTxt} vs prior · ${esc(k.lastMonthLabel)}`, PALETTE[4]), + k.mom == null ? `${esc(k.lastMonthLabel || "")}${partialHtml}` : `${momTxt} vs prior · ${esc(k.lastMonthLabel)}${partialHtml}`, PALETTE[4]), ].join(""); const d = p.data; @@ -810,13 +825,14 @@ function renderRate(p) { const kpis = [ kpiCard("Effective savings rate", fmtPct(esr), - `${fmtMoney(s.Total)} total savings`, PALETTE[1]), + `${fmtMoney(s.Total)} total savings · vs. list price`, PALETTE[1]), kpiCard("Total savings", fmtMoney(s.Total), `of ${fmtMoney(s.List)} list cost`, PALETTE[2]), kpiCard("Commitment utilization", fmtPct(util), `${fmtMoney(cmTotal - (cm.Unused || 0))} of ${fmtMoney(cmTotal)} used`, PALETTE[0]), kpiCard("Commitment waste", fmtPct(waste), - `${fmtMoney(cm.Unused)} unused`, PALETTE[3]), + `${fmtMoney(cm.Unused)} unused · of commitment spend`, PALETTE[3], + kpiThreshold(waste, 0.10, 0.20)), kpiCard("Compute coverage", fmtPct(coverage), `compute spend on commitments`, PALETTE[5]), kpiCard("Committed core-hours", fmtPct(coreShare), @@ -901,7 +917,8 @@ function renderAllocation(p) { kpiCard("Allocation accuracy", fmtPct(aai), `directly attributed effective cost`, PALETTE[1]), kpiCard("Untagged cost", fmtPct(untaggedPct), - `${fmtMoney(c.Untagged)} with no tags`, PALETTE[3]), + `${fmtMoney(c.Untagged)} with no tags`, PALETTE[3], + kpiThreshold(untaggedPct, 0.10, 0.25)), kpiCard("Unallocated cost", fmtPct(unallocPct), `${fmtMoney(total - c.Attributed)} lacks allocation evidence`, PALETTE[4]), kpiCard("Tag policy compliance", fmtPct(compliancePct), @@ -1002,7 +1019,8 @@ async function load() { const res = await fetch(`${ENDPOINT[tab]}&preset=${encodeURIComponent(preset)}`); state.cache[tab][preset] = await res.json(); } catch (err) { - state.cache[tab][preset] = { error: String(err && err.message ? err.message : err) }; + console.error("[ftk-dashboard] fetch failed:", err); + state.cache[tab][preset] = { error: "Could not load data. Check that the Kusto emulator is running." }; } state.loading = false; setRefreshSpinning(false); @@ -1048,7 +1066,10 @@ function wireControls() { const btn = e.target.closest("button[data-tab]"); if (!btn || state.loading || btn.dataset.tab === state.tab) return; state.tab = btn.dataset.tab; - [...el("tabs").querySelectorAll("button")].forEach((b) => b.classList.toggle("active", b === btn)); + [...el("tabs").querySelectorAll("button")].forEach((b) => { + b.classList.toggle("active", b === btn); + b.setAttribute("aria-selected", b === btn ? "true" : "false"); + }); load(); }); el("refresh").addEventListener("click", () => { From 13b389c9574b29f877e0fbc04f3275a10aed1371 Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 25 Jun 2026 09:43:11 -0700 Subject: [PATCH 34/51] feat(dashboard): add triage framing to Overview tab (T-5001.4) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add 3-tile triage strip (Anomalies | Overspend | Savings Opportunities) above the KPI row on the Overview tab only - Each tile shows count, severity badge, and one-line action cue - Tile color uses threshold-green/amber/red classes from Task 1 - Anomaly count: reads from state.cache['anomaly'] when loaded - Overspend count: months in trend data where effective cost rose >20% MoM - Savings count: underutilized commitments from state.cache['rate'] when loaded - Clicking any tile fires switchTab() to deep-link to the relevant tab - Zero count renders 'None' in green; unloaded cache renders '—' (neutral) - Extract switchTab() from wireControls so triage tiles can reuse it - Add responsive 1-column layout at ≤640 px Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../ftk-local-dashboard/public/app.css | 68 ++++++++++++++++++ .../ftk-local-dashboard/public/app.js | 71 +++++++++++++++++-- 2 files changed, 132 insertions(+), 7 deletions(-) diff --git a/.github/extensions/ftk-local-dashboard/public/app.css b/.github/extensions/ftk-local-dashboard/public/app.css index 9fb866bbd..e739cfd5c 100644 --- a/.github/extensions/ftk-local-dashboard/public/app.css +++ b/.github/extensions/ftk-local-dashboard/public/app.css @@ -165,6 +165,74 @@ body { .threshold-amber { border-color: var(--warn); } .threshold-red { border-color: var(--neg); } +/* ---------- triage strip ---------- */ +.triage-strip { + display: grid; + grid-template-columns: repeat(3, 1fr); + gap: var(--gap); + margin-bottom: 22px; +} +.triage-tile { + appearance: none; + background: var(--card-bg); + border: 2px solid var(--grid); + border-radius: var(--radius); + padding: 16px 18px; + cursor: pointer; + text-align: left; + display: flex; + flex-direction: column; + gap: 4px; + transition: opacity 0.15s; + color: var(--text-color-default, #1f2328); +} +.triage-tile:hover { opacity: 0.82; } +.triage-tile:active { transform: translateY(1px); } +.triage-title { + font-size: 11px; + font-weight: 650; + color: var(--muted); + text-transform: uppercase; + letter-spacing: 0.05em; +} +.triage-count { + font-size: 30px; + font-weight: 700; + letter-spacing: -0.02em; + font-variant-numeric: tabular-nums; + line-height: 1.1; +} +.triage-badge { + display: inline-block; + font-size: 11px; + font-weight: 650; + padding: 2px 9px; + border-radius: 20px; + width: fit-content; + background: color-mix(in srgb, var(--muted) 12%, transparent); + color: var(--muted); +} +.triage-tile.threshold-green .triage-badge { + background: color-mix(in srgb, var(--pos) 15%, transparent); + color: var(--pos); +} +.triage-tile.threshold-amber .triage-badge { + background: color-mix(in srgb, var(--warn) 15%, transparent); + color: var(--warn); +} +.triage-tile.threshold-red .triage-badge { + background: color-mix(in srgb, var(--neg) 15%, transparent); + color: var(--neg); +} +.triage-cue { + font-size: 12px; + color: var(--muted); + margin-top: 2px; +} +@media (max-width: 640px) { + .triage-strip { grid-template-columns: 1fr; } +} + /* ---------- sections & panels ---------- */ .section-title { display: flex; diff --git a/.github/extensions/ftk-local-dashboard/public/app.js b/.github/extensions/ftk-local-dashboard/public/app.js index 0fa9de4ef..905002fdb 100644 --- a/.github/extensions/ftk-local-dashboard/public/app.js +++ b/.github/extensions/ftk-local-dashboard/public/app.js @@ -416,6 +416,68 @@ function kpiThreshold(pct, greenMax, amberMax) { return "threshold-red"; } +function switchTab(tabId) { + if (state.loading || tabId === state.tab) return; + state.tab = tabId; + [...el("tabs").querySelectorAll("button")].forEach((b) => { + b.classList.toggle("active", b.dataset.tab === tabId); + b.setAttribute("aria-selected", b.dataset.tab === tabId ? "true" : "false"); + }); + load(); +} + +/* --------------------------------------------------------- triage strip */ + +function buildTriageTile(title, count, cue, tabId) { + const cls = count === null ? "" : count === 0 ? "threshold-green" : count <= 4 ? "threshold-amber" : "threshold-red"; + const badge = count === null ? "Unknown" : count === 0 ? "Good" : count <= 4 ? "Review" : "Urgent"; + const display = count === null ? "—" : count === 0 ? "None" : fmtInt(count); + return ``; +} + +function renderTriageStrip(d) { + // Anomalies: reuse anomaly tab cache when loaded + const anomPayload = state.cache["anomaly"]?.[state.preset]; + const anomCount = anomPayload?.data?.daily !== undefined + ? anomPayload.data.daily.filter((r) => r.Flag !== 0).length + : null; + const anomCue = anomCount === null ? "Visit Anomalies & forecast tab to load" + : anomCount === 0 ? "No anomalies detected" + : "Review flagged cost days"; + + // Overspend: months in trend where effective cost rose >20% vs prior month + const trend = d.trend || []; + let overspendCount = 0; + for (let i = 1; i < trend.length; i++) { + const prev = trend[i - 1].Effective || 0; + const curr = trend[i].Effective || 0; + if (prev > 0 && curr > prev * 1.20) overspendCount++; + } + const overspendCue = overspendCount === 0 + ? "Spend within expected range" + : `${overspendCount} month${overspendCount === 1 ? "" : "s"} with >20% spike`; + + // Savings opportunities: underutilized commitments from rate tab cache when loaded + const ratePayload = state.cache["rate"]?.[state.preset]; + const savingsCount = ratePayload?.data?.byCommitment !== undefined + ? ratePayload.data.byCommitment.filter((r) => (r.Unused || 0) > 0).length + : null; + const savingsCue = savingsCount === null ? "Visit Rate optimization tab to load" + : savingsCount === 0 ? "Commitments fully utilized" + : "Underutilized commitments found"; + + return `
+ ${buildTriageTile("Anomalies", anomCount, anomCue, "anomaly")} + ${buildTriageTile("Overspend", overspendCount, overspendCue, "anomaly")} + ${buildTriageTile("Savings Opportunities", savingsCount, savingsCue, "rate")} +
`; +} + function isPartialMonth() { const now = new Date(); return now.getDate() < new Date(now.getFullYear(), now.getMonth() + 1, 0).getDate(); @@ -464,6 +526,7 @@ function renderOverview(p) { const d = p.data; const html = ` + ${renderTriageStrip(d)}
${kpis}

Understand usage & cost

FinOps Framework
@@ -1064,13 +1127,7 @@ function wireControls() { }); el("tabs").addEventListener("click", (e) => { const btn = e.target.closest("button[data-tab]"); - if (!btn || state.loading || btn.dataset.tab === state.tab) return; - state.tab = btn.dataset.tab; - [...el("tabs").querySelectorAll("button")].forEach((b) => { - b.classList.toggle("active", b === btn); - b.setAttribute("aria-selected", b === btn ? "true" : "false"); - }); - load(); + if (btn) switchTab(btn.dataset.tab); }); el("refresh").addEventListener("click", () => { if (state.loading) return; From 435bc00aa6ccb6cfd13befa7d3fa65c38fb8b609 Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 25 Jun 2026 09:47:44 -0700 Subject: [PATCH 35/51] fix(dashboard): correct Overspend tab target and null guards in triage strip - Bug 1: Fix Overspend tile navigating to 'anomaly' tab instead of 'usage' - Bug 2: Replace null-check pattern for anomaly/savings counts with || [] pattern Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/extensions/ftk-local-dashboard/public/app.js | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/.github/extensions/ftk-local-dashboard/public/app.js b/.github/extensions/ftk-local-dashboard/public/app.js index 905002fdb..e90445d4f 100644 --- a/.github/extensions/ftk-local-dashboard/public/app.js +++ b/.github/extensions/ftk-local-dashboard/public/app.js @@ -443,9 +443,8 @@ function buildTriageTile(title, count, cue, tabId) { function renderTriageStrip(d) { // Anomalies: reuse anomaly tab cache when loaded const anomPayload = state.cache["anomaly"]?.[state.preset]; - const anomCount = anomPayload?.data?.daily !== undefined - ? anomPayload.data.daily.filter((r) => r.Flag !== 0).length - : null; + const daily = anomPayload?.data?.daily || []; + const anomCount = anomPayload ? daily.filter((r) => r.Flag !== 0).length : null; const anomCue = anomCount === null ? "Visit Anomalies & forecast tab to load" : anomCount === 0 ? "No anomalies detected" : "Review flagged cost days"; @@ -464,16 +463,15 @@ function renderTriageStrip(d) { // Savings opportunities: underutilized commitments from rate tab cache when loaded const ratePayload = state.cache["rate"]?.[state.preset]; - const savingsCount = ratePayload?.data?.byCommitment !== undefined - ? ratePayload.data.byCommitment.filter((r) => (r.Unused || 0) > 0).length - : null; + const byCommitment = ratePayload?.data?.byCommitment || []; + const savingsCount = ratePayload ? byCommitment.filter((r) => (r.Unused || 0) > 0).length : null; const savingsCue = savingsCount === null ? "Visit Rate optimization tab to load" : savingsCount === 0 ? "Commitments fully utilized" : "Underutilized commitments found"; return `
${buildTriageTile("Anomalies", anomCount, anomCue, "anomaly")} - ${buildTriageTile("Overspend", overspendCount, overspendCue, "anomaly")} + ${buildTriageTile("Overspend", overspendCount, overspendCue, "usage")} ${buildTriageTile("Savings Opportunities", savingsCount, savingsCue, "rate")}
`; } From 7f627ca76313afa0a9f80de9fc2e37b9365032c7 Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 25 Jun 2026 09:52:15 -0700 Subject: [PATCH 36/51] feat(dashboard): elevate source line to diagnostic rail (T-5001.2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add module-level queryState { rows, health, refreshedAt, dataset } - Add fmtRelativeTime() using Intl.RelativeTimeFormat (no dependencies) - Add renderDiagnosticRail() — populates #diagnostic-rail below content - Update updateChrome(): populate queryState from payload, call rail; health=warn on 0 rows, health=error on fetch failure, health=ok otherwise - Simplify #source-line to cluster URI only; clear row/refresh data from #footer-meta (window range kept); all data metadata moves to rail - Add #diagnostic-rail div to index.html between and
- Add .diagnostic-rail CSS: 24px max-height, muted text, flex layout; .rail-health--ok/warn/error colour-coded; dotted underline on time tooltip Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../ftk-local-dashboard/public/app.css | 20 ++++++++ .../ftk-local-dashboard/public/app.js | 50 +++++++++++++++---- .../ftk-local-dashboard/public/index.html | 16 +++--- 3 files changed, 69 insertions(+), 17 deletions(-) diff --git a/.github/extensions/ftk-local-dashboard/public/app.css b/.github/extensions/ftk-local-dashboard/public/app.css index e739cfd5c..3cee81234 100644 --- a/.github/extensions/ftk-local-dashboard/public/app.css +++ b/.github/extensions/ftk-local-dashboard/public/app.css @@ -348,6 +348,26 @@ svg { display: block; width: 100%; overflow: visible; } margin-top: 14px; } +/* ---------- diagnostic rail ---------- */ +.diagnostic-rail { + display: flex; + align-items: center; + gap: 8px; + height: 24px; + max-height: 24px; + overflow: hidden; + font-size: 11.5px; + color: var(--muted); + padding: 0 2px; + margin-top: 12px; + font-variant-numeric: tabular-nums; +} +.rail-sep { opacity: 0.45; user-select: none; } +.rail-health--ok { color: var(--pos); } +.rail-health--warn { color: var(--warn); } +.rail-health--error { color: var(--neg); } +.rail-time { cursor: default; text-decoration: underline; text-decoration-style: dotted; } + /* ---------- footer ---------- */ .app-footer { display: flex; diff --git a/.github/extensions/ftk-local-dashboard/public/app.js b/.github/extensions/ftk-local-dashboard/public/app.js index e90445d4f..f432ca02a 100644 --- a/.github/extensions/ftk-local-dashboard/public/app.js +++ b/.github/extensions/ftk-local-dashboard/public/app.js @@ -10,6 +10,7 @@ const PALETTE = [ ]; const state = { preset: "all", tab: "overview", loading: false, cache: {} }; +const queryState = { rows: 0, health: "ok", refreshedAt: null, dataset: "Hub database" }; /* ------------------------------------------------------------------ utils */ @@ -66,6 +67,16 @@ function trunc(s, n) { return s.length > n ? s.slice(0, n - 1) + "…" : s; } function el(id) { return document.getElementById(id); } +function fmtRelativeTime(date) { + if (!date) return "—"; + const rtf = new Intl.RelativeTimeFormat("en", { numeric: "auto" }); + const diffSec = (date - Date.now()) / 1000; + const abs = Math.abs(diffSec); + if (abs < 60) return rtf.format(Math.round(diffSec), "second"); + if (abs < 3600) return rtf.format(Math.round(diffSec / 60), "minute"); + if (abs < 86400) return rtf.format(Math.round(diffSec / 3600), "hour"); + return rtf.format(Math.round(diffSec / 86400), "day"); +} function svgEl(w, h, body) { return `${body}`; } @@ -1094,24 +1105,43 @@ function setRefreshSpinning(on) { if (b) b.innerHTML = on ? `↻ Refresh` : `↻ Refresh`; } +function renderDiagnosticRail() { + const railEl = el("diagnostic-rail"); + if (!railEl) return; + const { rows, health, refreshedAt, dataset } = queryState; + const relTime = fmtRelativeTime(refreshedAt); + const absTime = refreshedAt ? refreshedAt.toLocaleString() : ""; + const rowTxt = `${fmtInt(rows)} rows`; + const healthLabel = health === "error" ? "● error" : health === "warn" ? "● warn" : "● ok"; + railEl.innerHTML = + `${esc(dataset)}` + + `` + + `${rowTxt}` + + `` + + `${healthLabel}` + + `` + + `${esc(relTime)}`; +} + function updateChrome() { const p = currentPayload(); const w = p && p.window; if (w && w.dataMin) { el("source-line").innerHTML = - `Hub database · ${esc(window.__cfg?.clusterUri || "localhost:8082")} · ${esc(fmtDayRange(w.dataMin, w.dataMax))}`; - let meta; - if (state.tab === "tokenomics" && p.data) { - const s = p.data.summary?.[0] || {}; - meta = `${fmtTokens(s.Tokens)} tokens · ${fmtMoney(s.Effective)} AI cost`; - } else { - meta = `${fmtInt(w.rows)} rows in range`; - } - el("footer-meta").textContent = - `${meta} · window ${w.start} → ${w.end} · refreshed ${new Date(p.generatedAt).toLocaleTimeString()}`; + `Hub database · ${esc(window.__cfg?.clusterUri || "localhost:8082")}`; + queryState.dataset = `Hub database · ${fmtDayRange(w.dataMin, w.dataMax)}`; + queryState.rows = w.rows || 0; + queryState.health = queryState.rows === 0 ? "warn" : "ok"; + queryState.refreshedAt = p.generatedAt ? new Date(p.generatedAt) : new Date(); + el("footer-meta").textContent = `window ${w.start} → ${w.end}`; + renderDiagnosticRail(); } else if (p && p.error) { el("source-line").textContent = "Connection failed — see panel below."; el("footer-meta").textContent = ""; + queryState.health = "error"; + queryState.refreshedAt = new Date(); + queryState.dataset = "Hub database"; + renderDiagnosticRail(); } } diff --git a/.github/extensions/ftk-local-dashboard/public/index.html b/.github/extensions/ftk-local-dashboard/public/index.html index 24771763d..2aa6d3014 100644 --- a/.github/extensions/ftk-local-dashboard/public/index.html +++ b/.github/extensions/ftk-local-dashboard/public/index.html @@ -24,18 +24,20 @@

FinOps hub · local

-
+
Loading cost data…
+
+
Grounded in the FinOps Framework domains & the FinOps toolkit query catalog. From 807c02c3f0d8738677acf1c22bf84febcab9bf4c Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 25 Jun 2026 10:23:53 -0700 Subject: [PATCH 37/51] feat(dashboard): add open KQL escape hatch per chart (T-5001.5) - openKqlDialog(panelId): reads PANEL_KQL, populates textarea, calls showModal() - executeKql(): POST /api/kql, renders result table inside dialog - showKqlResult(): displays up to 200 rows in scrollable table - wireControls(): wires kql-close, kql-copy, kql-run, and .kql-btn delegation - panelHtml(): adds data-panel-id to button, fixes aria-label to 'View KQL query' - All 26 PANEL_KQL panels migrated to panelHtml() (tokenomics, anomaly, usage, rate, allocation) - index.html: remove readonly from textarea (edit+run AC requires editable) - app.css: add .btn-primary, .kql-result, .kql-result-scroll, .kql-dialog-footer styles Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../ftk-local-dashboard/public/app.css | 113 +++++++ .../ftk-local-dashboard/public/app.js | 296 +++++++++--------- .../ftk-local-dashboard/public/index.html | 15 + 3 files changed, 284 insertions(+), 140 deletions(-) diff --git a/.github/extensions/ftk-local-dashboard/public/app.css b/.github/extensions/ftk-local-dashboard/public/app.css index 3cee81234..4b63fd20e 100644 --- a/.github/extensions/ftk-local-dashboard/public/app.css +++ b/.github/extensions/ftk-local-dashboard/public/app.css @@ -395,3 +395,116 @@ svg { display: block; width: 100%; overflow: visible; } @media (prefers-reduced-motion: reduce) { .spin { animation: none; } } + +/* ---------- panel-header (KQL escape hatch) ---------- */ +.panel-header { + display: flex; + align-items: flex-start; + justify-content: space-between; + gap: 8px; + margin-bottom: 10px; +} +.panel-header > div { min-width: 0; } +.panel-header h3 { margin-bottom: 2px; } +.kql-btn { + flex: none; + background: transparent; + border: 1px solid var(--grid); + border-radius: 4px; + color: var(--muted); + cursor: pointer; + font: 600 10px/1 ui-monospace, monospace; + padding: 4px 7px; + opacity: 0.7; + transition: opacity 0.15s, background 0.15s, color 0.15s; +} +.kql-btn:hover { + opacity: 1; + background: color-mix(in srgb, var(--accent) 8%, transparent); + color: var(--accent); + border-color: var(--accent); +} + +/* ---------- KQL dialog ---------- */ +#kql-dialog { + border: 1px solid var(--grid); + border-radius: var(--radius); + background: var(--card-bg); + color: inherit; + padding: 0; + width: min(780px, 96vw); +} +#kql-dialog::backdrop { + background: rgba(0, 0, 0, 0.55); + backdrop-filter: blur(2px); +} +.kql-dialog-inner { + display: flex; + flex-direction: column; + max-height: 90vh; +} +.kql-dialog-header { + display: flex; + align-items: center; + justify-content: space-between; + padding: 14px 18px 12px; + border-bottom: 1px solid var(--grid); + gap: 12px; +} +.kql-dialog-header h3 { margin: 0; font-size: 14px; } +#kql-text { + flex: 1; + border: none; + border-bottom: 1px solid var(--grid); + background: transparent; + color: inherit; + font: 12px/1.55 ui-monospace, Menlo, Consolas, monospace; + padding: 14px 18px; + resize: vertical; + min-height: 200px; + white-space: pre; + overflow-wrap: normal; + overflow-x: auto; +} +.kql-dialog-error { + margin: 0; + padding: 6px 18px 0; + font-size: 11.5px; + color: var(--neg); + min-height: 22px; +} +.kql-dialog-footer { + display: flex; + align-items: center; + justify-content: flex-end; + gap: 8px; + padding: 10px 18px; +} +.btn-primary { + background: var(--accent); + color: #fff; + border-color: var(--accent); +} +.btn-primary:hover { + background: color-mix(in srgb, var(--accent) 85%, #000); +} + +/* ---------- KQL result table ---------- */ +.kql-result { + border-top: 1px solid var(--grid); + overflow: hidden; +} +.kql-result-meta { + margin: 0; + padding: 6px 18px; + font-size: 11.5px; + color: var(--muted); +} +.kql-result-scroll { + overflow-x: auto; + padding: 0 18px 14px; + max-height: 260px; + overflow-y: auto; +} + + diff --git a/.github/extensions/ftk-local-dashboard/public/app.js b/.github/extensions/ftk-local-dashboard/public/app.js index f432ca02a..8acd4a77f 100644 --- a/.github/extensions/ftk-local-dashboard/public/app.js +++ b/.github/extensions/ftk-local-dashboard/public/app.js @@ -12,6 +12,45 @@ const PALETTE = [ const state = { preset: "all", tab: "overview", loading: false, cache: {} }; const queryState = { rows: 0, health: "ok", refreshedAt: null, dataset: "Hub database" }; +/* ----------------------------------------------------------------- KQL templates */ + +const PERIOD = "| where ChargePeriodStart >= datetime({start}) and ChargePeriodStart < datetime({end})"; +const NON_PURCH = "| where not(ChargeCategory == 'Purchase' and isnotempty(CommitmentDiscountCategory))"; +const AI_SCOPE = "| where x_SkuMeterSubcategory has 'OpenAI' and x_SkuDescription contains 'Token'"; + +/* eslint-disable max-len */ +const PANEL_KQL = { + "overview-trend": ["Costs()", PERIOD, "| summarize Billed=sum(BilledCost), Effective=sum(EffectiveCost)", " by Month=format_datetime(startofmonth(ChargePeriodStart),'yyyy-MM')", "| order by Month asc"].join("\n"), + "overview-top-services": ["Costs()", PERIOD, "| summarize Cost=sum(EffectiveCost) by ServiceName", "| top 10 by Cost desc"].join("\n"), + "overview-service-category": ["Costs()", PERIOD, "| summarize Cost=sum(EffectiveCost) by ServiceCategory", "| where Cost > 0 | order by Cost desc"].join("\n"), + "overview-top-rgs": ["Costs()", PERIOD, "| where isnotempty(x_ResourceGroupName)", "| summarize Cost=sum(EffectiveCost) by x_ResourceGroupName", "| top 10 by Cost desc"].join("\n"), + "overview-top-regions": ["Costs()", PERIOD, "| where isnotempty(RegionId)", "| summarize Cost=sum(EffectiveCost) by RegionId", "| top 12 by Cost desc"].join("\n"), + "overview-rate-coverage": ["Costs()", PERIOD, "| summarize Cost=sum(EffectiveCost) by PricingCategory"].join("\n"), + "overview-savings": ["Costs()", PERIOD, NON_PURCH, "| extend neg=iff(ListCost=s and ChargePeriodStart 100 | extend PerCore=Eff/CH | top 10 by Eff desc"].join("\n"), + "usage-storage-tiers": ["Costs()", PERIOD, "| where ServiceCategory=='Storage' and ChargeCategory=='Usage'", "| extend Tier=case(x_SkuTier has_any('Hot','Standard','Premium'),'Frequent',x_SkuTier has_any('Cool','Cold','Archive'),'Infrequent','Unclassified')", "| summarize Cost=sum(EffectiveCost) by Tier", "| where Cost > 0 | order by Cost desc"].join("\n"), + "rate-savings": ["Costs()", PERIOD, NON_PURCH, "| extend neg=iff(ListCost0, cores*ConsumedQuantity, toreal(''))", "| extend t=iff(isempty(CommitmentDiscountType),'On Demand',CommitmentDiscountType)", "| summarize CoreHours=sum(ch) by t", "| where CoreHours > 0 | order by CoreHours desc"].join("\n"), + "rate-underutil": ["Costs()", PERIOD, "| where isnotempty(CommitmentDiscountName)", NON_PURCH, "| summarize Unused=sumif(EffectiveCost,CommitmentDiscountStatus=='Unused'), Total=sum(EffectiveCost) by CommitmentDiscountName", "| where Unused > 0 | top 10 by Unused desc"].join("\n"), + "alloc-hierarchy": ["Costs()", PERIOD, "| extend Org=tostring(Tags['org']), Project=tostring(Tags['Project']), Env=tostring(Tags['env'])", "| summarize Cost=sum(EffectiveCost) by Org, Project, Env", "| where Cost > 0 | top 12 by Cost desc"].join("\n"), + "alloc-tagging": ["Costs()", PERIOD, "| extend _t=iff(isnull(Tags) or array_length(bag_keys(Tags))==0,'Untagged','Tagged')", "| summarize Cost=sum(EffectiveCost) by _t"].join("\n"), + "alloc-tag-keys": ["Costs()", PERIOD, "| mv-expand k=bag_keys(Tags) to typeof(string)", "| where isnotempty(k) and k !in ('ftk-tool','ftk-version','cm-resource-parent','costanalysis-parent')", "| summarize Cost=sum(EffectiveCost) by k", "| top 12 by Cost desc"].join("\n"), + "alloc-by-subscription": ["Costs()", PERIOD, "| where isnotempty(SubAccountName)", "| summarize Cost=sum(EffectiveCost) by SubAccountName", "| top 10 by Cost desc"].join("\n"), +}; +/* eslint-enable max-len */ + +let _kqlPanelId = null; + /* ------------------------------------------------------------------ utils */ function fmtMoney(n) { @@ -503,6 +542,67 @@ function kpiCard(label, value, meta, accent, thresholdClass) { /* ---------------------------------------------------------------- render */ +function panelHtml(id, span, title, sub, body) { + const subHtml = sub ? `

${sub}

` : ""; + return `
+

${title}

${subHtml}
+
${body}
+
`; +} + +function openKqlDialog(panelId) { + _kqlPanelId = panelId; + el("kql-text").value = PANEL_KQL[panelId] || ""; + el("kql-error").textContent = ""; + const prev = document.getElementById("kql-result"); + if (prev) prev.remove(); + el("kql-dialog").showModal(); +} + +async function executeKql() { + const kql = el("kql-text").value.trim(); + const errEl = el("kql-error"); + const runBtn = el("kql-run"); + if (!kql) return; + errEl.textContent = ""; + const prev = document.getElementById("kql-result"); + if (prev) prev.remove(); + runBtn.disabled = true; + runBtn.textContent = "Running…"; + try { + const res = await fetch("/api/kql", { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ kql }), + }); + const data = await res.json(); + if (data.error) { + errEl.textContent = data.error; + } else { + showKqlResult(data.rows || []); + } + } catch (err) { + errEl.textContent = "Request failed: " + err.message; + } finally { + runBtn.disabled = false; + runBtn.textContent = "Run"; + } +} + +function showKqlResult(rows) { + if (!rows.length) { el("kql-error").textContent = "Query returned no rows."; return; } + const cols = Object.keys(rows[0]); + const head = cols.map((c) => `${esc(c)}`).join(""); + const body = rows.slice(0, 200).map((r) => + `${cols.map((c) => `${esc(String(r[c] ?? ""))}`).join("")}` + ).join(""); + const div = document.createElement("div"); + div.id = "kql-result"; + div.className = "kql-result"; + div.innerHTML = `

${rows.length} rows${rows.length > 200 ? " (showing first 200)" : ""}

${head}${body}
`; + el("kql-dialog").querySelector(".kql-dialog-footer").before(div); +} + function renderOverview(p) { const content = el("content"); if (!p) return; @@ -540,60 +640,28 @@ function renderOverview(p) {

Understand usage & cost

FinOps Framework
-
-

Monthly cost trend

-

Billed vs effective cost by month — executive run-rate view.

- ${lineChart(d.trend)} -
-
-

Top services by cost

-

Effective cost by Azure service.

- ${hbar(d.topServices, "ServiceName", "Cost")} -
-
-

Cost by service category

-

Where spend concentrates across categories.

- ${hbar(d.serviceCategory, "ServiceCategory", "Cost")} -
+ ${panelHtml("overview-trend", 12, "Monthly cost trend", "Billed vs effective cost by month — executive run-rate view.", lineChart(d.trend))} + ${panelHtml("overview-top-services", 6, "Top services by cost", "Effective cost by Azure service.", hbar(d.topServices, "ServiceName", "Cost"))} + ${panelHtml("overview-service-category", 6, "Cost by service category", "Where spend concentrates across categories.", hbar(d.serviceCategory, "ServiceCategory", "Cost"))}

Optimize usage & cost

FinOps Framework
-
-

Top resource groups

-

Largest cost owners for allocation & accountability.

- ${hbar(d.topResourceGroups, "x_ResourceGroupName", "Cost")} -
-
-

Cost by region

-

Regional spend for placement & sustainability review.

- ${hbar(d.topRegions, "RegionId", "Cost")} -
+ ${panelHtml("overview-top-rgs", 6, "Top resource groups", "Largest cost owners for allocation & accountability.", hbar(d.topResourceGroups, "x_ResourceGroupName", "Cost"))} + ${panelHtml("overview-top-regions", 6, "Cost by region", "Regional spend for placement & sustainability review.", hbar(d.topRegions, "RegionId", "Cost"))}

Quantify business value

FinOps Framework
-
-

Rate coverage

-

Committed vs on-demand (standard) effective cost.

- ${donut([ - { label: "Committed", value: k.committed, color: PALETTE[1] }, - { label: "On-demand", value: Math.max(0, k.eff - k.committed), color: PALETTE[0] }, - ], { centerBig: fmtPct(k.coverage), centerSmall: "covered" })} -
-
-

Savings breakdown

-

List → effective, by discount type.

- ${savingsTable(k)} -
-
-

Cost allocation

-

Tagged vs untagged effective cost.

- ${donut([ - { label: "Tagged", value: k.tagged, color: PALETTE[1] }, - { label: "Untagged", value: k.untagged, color: PALETTE[3] }, - ], { centerBig: fmtPct(1 - k.untaggedPct), centerSmall: "tagged" })} -
+ ${panelHtml("overview-rate-coverage", 4, "Rate coverage", "Committed vs on-demand (standard) effective cost.", donut([ + { label: "Committed", value: k.committed, color: PALETTE[1] }, + { label: "On-demand", value: Math.max(0, k.eff - k.committed), color: PALETTE[0] }, + ], { centerBig: fmtPct(k.coverage), centerSmall: "covered" }))} + ${panelHtml("overview-savings", 4, "Savings breakdown", "List → effective, by discount type.", savingsTable(k))} + ${panelHtml("overview-cost-allocation", 4, "Cost allocation", "Tagged vs untagged effective cost.", donut([ + { label: "Tagged", value: k.tagged, color: PALETTE[1] }, + { label: "Untagged", value: k.untagged, color: PALETTE[3] }, + ], { centerBig: fmtPct(1 - k.untaggedPct), centerSmall: "tagged" }))}
`; content.innerHTML = html; @@ -670,33 +738,19 @@ function renderTokenomics(p) {

AI token economics

Token Consumption Metrics KPI
-
-

Token volume & AI cost trend

-

Monthly token consumption (bars) and effective AI cost (line).

- ${tokenTrendChart(d.trend)} -
-
-

AI cost by model

-

Effective cost per model family.

- ${hbar((d.models || []).map((m) => ({ Model: m.Model, Cost: m.Cost })), "Model", "Cost")} -
-
-

Token direction mix

-

Input vs cached input vs output — by token volume.

- ${donut(dirSlices, { + ${panelHtml("token-trend", 12, "Token volume & AI cost trend", "Monthly token consumption (bars) and effective AI cost (line).", tokenTrendChart(d.trend))} + ${panelHtml("token-by-model", 6, "AI cost by model", "Effective cost per model family.", + hbar((d.models || []).map((m) => ({ Model: m.Model, Cost: m.Cost })), "Model", "Cost"))} + ${panelHtml("token-direction", 6, "Token direction mix", "Input vs cached input vs output — by token volume.", + donut(dirSlices, { centerBig: fmtTokens(k.tokens), centerSmall: "tokens", valueFmt: (s) => `${fmtTokens(s.value)} · ${fmtMoney(s.cost)}`, - })} -
+ }))}

Model efficiency

Rate & usage optimization
-
-

Cost per 1M tokens by model

-

Unit economics for model selection — sorted by effective cost.

- ${tokenModelTable(d.models, k.eff)} -
+ ${panelHtml("token-model-table", 12, "Cost per 1M tokens by model", "Unit economics for model selection — sorted by effective cost.", tokenModelTable(d.models, k.eff))}
`; } @@ -773,25 +827,13 @@ function renderAnomaly(p) {

Cost anomalies

Anomaly Management capability
-
-

Daily cost & detected anomalies

-

Daily effective cost vs the expected baseline (STL decomposition); markers flag spikes & drops.

- ${anomalyChart(daily)} -
+ ${panelHtml("anomaly-daily", 12, "Daily cost & detected anomalies", "Daily effective cost vs the expected baseline (STL decomposition); markers flag spikes & drops.", anomalyChart(daily))}

Trend & forecast

Forecasting · Data freshness
-
-

Month-over-month change

-

Effective cost % change vs prior month (red = increase).

- ${momBars(mc)} -
-
-

Cost forecast

-

Monthly effective cost, actual vs forecast (next 3 months).

- ${forecastChart(fc, dataMaxMonth)} -
+ ${panelHtml("anomaly-mom", 6, "Month-over-month change", "Effective cost % change vs prior month (red = increase).", momBars(mc))} + ${panelHtml("anomaly-forecast", 6, "Cost forecast", "Monthly effective cost, actual vs forecast (next 3 months).", forecastChart(fc, dataMaxMonth))}
`; } @@ -855,21 +897,11 @@ function renderUsage(p) {

Usage & unit economics

Usage Optimization · Unit Economics
-
-

Top resource types by cost

-

Resource count and effective spend per resource type.

- ${typeTable} -
-
-

Compute cost per core by VM series

-

Effective cost per vCPU-hour — highlights expensive (e.g. GPU) cores.

- ${hbar(d.perCoreSeries, "x_SkuMeterSubcategory", "PerCore")} -
-
-

Storage tier distribution

-

Effective storage cost by access tier (${fmtPct(freqShare)} classified frequent).

- ${donut(tierSlices, { centerBig: fmtMoney(s.Cost), centerSmall: "storage" })} -
+ ${panelHtml("usage-top-types", 12, "Top resource types by cost", "Resource count and effective spend per resource type.", typeTable)} + ${panelHtml("usage-per-core-series", 6, "Compute cost per core by VM series", "Effective cost per vCPU-hour — highlights expensive (e.g. GPU) cores.", + hbar(d.perCoreSeries, "x_SkuMeterSubcategory", "PerCore"))} + ${panelHtml("usage-storage-tiers", 6, `Storage tier distribution`, `Effective storage cost by access tier (${fmtPct(freqShare)} classified frequent).`, + donut(tierSlices, { centerBig: fmtMoney(s.Cost), centerSmall: "storage" }))}
`; } @@ -930,32 +962,19 @@ function renderRate(p) {

Rate optimization

Rate Optimization capability
-
-

Savings breakdown

-

List → effective cost by discount type (effective savings rate).

- ${savingsBreak} -
-
-

Commitment utilization

-

Used vs unused commitment effective cost.

- ${donut([ + ${panelHtml("rate-savings", 7, "Savings breakdown", "List → effective cost by discount type (effective savings rate).", savingsBreak)} + ${panelHtml("rate-commit-util", 5, "Commitment utilization", "Used vs unused commitment effective cost.", + donut([ { label: "Used", value: cmTotal - (cm.Unused || 0), color: PALETTE[1] }, { label: "Unused (waste)", value: cm.Unused || 0, color: PALETTE[3] }, - ], { centerBig: fmtPct(util), centerSmall: "utilized" })} -
-
-

Core-hour coverage

-

Consumed core-hours by commitment type.

- ${donut(coreSlices, { + ], { centerBig: fmtPct(util), centerSmall: "utilized" }))} + ${panelHtml("rate-core-hours", 6, "Core-hour coverage", "Consumed core-hours by commitment type.", + donut(coreSlices, { centerBig: fmtPct(coreShare), centerSmall: "committed", valueFmt: (s) => `${fmtTokens(s.value)} core-hrs`, - })} -
-
-

Underutilized commitments

-

Reservations & plans with the most unused cost.

- ${hbar(d.byCommitment, "CommitmentDiscountName", "Unused")} -
+ }))} + ${panelHtml("rate-underutil", 6, "Underutilized commitments", "Reservations & plans with the most unused cost.", + hbar(d.byCommitment, "CommitmentDiscountName", "Unused"))}
`; } @@ -1018,29 +1037,14 @@ function renderAllocation(p) {

Cost allocation

Allocation capability
-
-

Cost by financial hierarchy

-

Org → project → environment (from resource tags), with share of total.

- ${hierTable} -
-
-

Tagging coverage

-

Tagged vs untagged effective cost.

- ${donut([ + ${panelHtml("alloc-hierarchy", 8, "Cost by financial hierarchy", "Org → project → environment (from resource tags), with share of total.", hierTable)} + ${panelHtml("alloc-tagging", 4, "Tagging coverage", "Tagged vs untagged effective cost.", + donut([ { label: "Tagged", value: total - c.Untagged, color: PALETTE[1] }, { label: "Untagged", value: c.Untagged, color: PALETTE[3] }, - ], { centerBig: fmtPct(1 - untaggedPct), centerSmall: "tagged" })} -
-
-

Cost by tag key

-

Effective cost touched by each governance tag.

- ${hbar(d.tagKeys, "k", "Cost")} -
-
-

Cost by subscription

-

Spend per billing scope for showback.

- ${hbar(d.bySubscription, "SubAccountName", "Cost")} -
+ ], { centerBig: fmtPct(1 - untaggedPct), centerSmall: "tagged" }))} + ${panelHtml("alloc-tag-keys", 6, "Cost by tag key", "Effective cost touched by each governance tag.", hbar(d.tagKeys, "k", "Cost"))} + ${panelHtml("alloc-by-subscription", 6, "Cost by subscription", "Spend per billing scope for showback.", hbar(d.bySubscription, "SubAccountName", "Cost"))}
`; } @@ -1162,6 +1166,18 @@ function wireControls() { if (state.cache[state.tab]) delete state.cache[state.tab][state.preset]; // force re-query load(); }); + + // KQL dialog controls + el("kql-close").addEventListener("click", () => el("kql-dialog").close()); + el("kql-copy").addEventListener("click", () => navigator.clipboard.writeText(el("kql-text").value)); + el("kql-run").addEventListener("click", executeKql); + + // KQL escape-hatch buttons (event delegation — buttons injected by panelHtml) + document.addEventListener("click", (e) => { + const btn = e.target.closest(".kql-btn[data-panel-id]"); + if (btn) openKqlDialog(btn.dataset.panelId); + }); + let t; window.addEventListener("resize", () => { clearTimeout(t); t = setTimeout(render, 180); }); } diff --git a/.github/extensions/ftk-local-dashboard/public/index.html b/.github/extensions/ftk-local-dashboard/public/index.html index 2aa6d3014..b16074a3f 100644 --- a/.github/extensions/ftk-local-dashboard/public/index.html +++ b/.github/extensions/ftk-local-dashboard/public/index.html @@ -43,6 +43,21 @@

FinOps hub · local

Grounded in the FinOps Framework domains & the FinOps toolkit query catalog.
+ +
+
+

KQL query

+ +
+ + +
+ + +
+
+ + From 4ebc90e762b903712538116b14b22a1119983d40 Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 25 Jun 2026 10:28:31 -0700 Subject: [PATCH 38/51] fix(dashboard): wire Run button to chart re-render and add HTTP error guard - executeKql() now closes the dialog on success and calls renderKqlResultInPanel() which injects the result table directly into the matching panel's .panel-body (found via [data-panel-id] DOM attribute), satisfying AC3 - Replaced showKqlResult() (in-dialog table injection) with renderKqlResultInPanel() - Added res.ok guard before res.json() so HTTP 4xx/5xx surfaces as 'Server error N' Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../ftk-local-dashboard/public/app.js | 20 +++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/.github/extensions/ftk-local-dashboard/public/app.js b/.github/extensions/ftk-local-dashboard/public/app.js index 8acd4a77f..1a03fdbcc 100644 --- a/.github/extensions/ftk-local-dashboard/public/app.js +++ b/.github/extensions/ftk-local-dashboard/public/app.js @@ -575,11 +575,18 @@ async function executeKql() { headers: { "Content-Type": "application/json" }, body: JSON.stringify({ kql }), }); + if (!res.ok) { errEl.textContent = `Server error ${res.status}`; return; } const data = await res.json(); if (data.error) { errEl.textContent = data.error; } else { - showKqlResult(data.rows || []); + const rows = data.rows || []; + if (!rows.length) { + errEl.textContent = "Query returned no rows."; + } else { + el("kql-dialog").close(); + renderKqlResultInPanel(_kqlPanelId, rows); + } } } catch (err) { errEl.textContent = "Request failed: " + err.message; @@ -589,18 +596,15 @@ async function executeKql() { } } -function showKqlResult(rows) { - if (!rows.length) { el("kql-error").textContent = "Query returned no rows."; return; } +function renderKqlResultInPanel(panelId, rows) { + const panelBody = document.querySelector(`[data-panel-id="${panelId}"] .panel-body`); + if (!panelBody) return; const cols = Object.keys(rows[0]); const head = cols.map((c) => `${esc(c)}`).join(""); const body = rows.slice(0, 200).map((r) => `${cols.map((c) => `${esc(String(r[c] ?? ""))}`).join("")}` ).join(""); - const div = document.createElement("div"); - div.id = "kql-result"; - div.className = "kql-result"; - div.innerHTML = `

${rows.length} rows${rows.length > 200 ? " (showing first 200)" : ""}

${head}${body}
`; - el("kql-dialog").querySelector(".kql-dialog-footer").before(div); + panelBody.innerHTML = `

${rows.length} rows${rows.length > 200 ? " (showing first 200)" : ""}

${head}${body}
`; } function renderOverview(p) { From 4e23bce3f4ae99e621ebb8ac1549aa64c0cf9741 Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 25 Jun 2026 10:36:08 -0700 Subject: [PATCH 39/51] feat(dashboard): introduce visual hierarchy between primary and reference KPIs (T-5001.3) - Add kpi--primary and kpi--reference CSS classes to distinguish KPI importance - Modify kpiCard() to auto-detect primary/reference status based on KPI label - Primary KPIs: Untagged cost, Commitment waste, Effective savings rate (grid-column: span 2, accent border) - Reference KPIs: Commitment coverage, Compute coverage, Total tokens, Anomaly days (85% opacity, muted border) - Primary KPIs render at full size with accent borders for visibility - Reference KPIs render smaller/muted for context without distraction - Integrates with Task 1's threshold badge logic (coexist in same class string) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../ftk-local-dashboard/public/app.css | 4 ++++ .../extensions/ftk-local-dashboard/public/app.js | 16 +++++++++++++++- 2 files changed, 19 insertions(+), 1 deletion(-) diff --git a/.github/extensions/ftk-local-dashboard/public/app.css b/.github/extensions/ftk-local-dashboard/public/app.css index 4b63fd20e..217629c6b 100644 --- a/.github/extensions/ftk-local-dashboard/public/app.css +++ b/.github/extensions/ftk-local-dashboard/public/app.css @@ -160,6 +160,10 @@ body { .kpi .meta .neg { color: var(--neg); font-weight: 650; } .kpi .meta .warn { color: var(--warn); font-weight: 650; } +/* ---------- KPI hierarchy (primary vs reference) ---------- */ +.kpi--primary { grid-column: span 2; border-top: 3px solid var(--accent); } +.kpi--reference { opacity: 0.85; border-top: 1px solid var(--border-muted); } + /* ---------- KPI threshold badges ---------- */ .threshold-green { border-color: var(--pos); } .threshold-amber { border-color: var(--warn); } diff --git a/.github/extensions/ftk-local-dashboard/public/app.js b/.github/extensions/ftk-local-dashboard/public/app.js index 1a03fdbcc..cbf9e3950 100644 --- a/.github/extensions/ftk-local-dashboard/public/app.js +++ b/.github/extensions/ftk-local-dashboard/public/app.js @@ -532,7 +532,21 @@ function isPartialMonth() { } function kpiCard(label, value, meta, accent, thresholdClass) { - const cls = thresholdClass ? ` ${thresholdClass}` : ""; + // Determine hierarchy class based on label + const primaryLabels = ["Untagged cost", "Commitment waste", "Effective savings rate"]; + const referenceLabels = ["Commitment coverage", "Compute coverage", "Total tokens", "Anomaly days"]; + + let hierarchyClass = ""; + if (primaryLabels.includes(label)) { + hierarchyClass = "kpi--primary"; + } else if (referenceLabels.includes(label)) { + hierarchyClass = "kpi--reference"; + } + + // Combine threshold and hierarchy classes + const classArray = [thresholdClass, hierarchyClass].filter(Boolean); + const cls = classArray.length > 0 ? ` ${classArray.join(" ")}` : ""; + return `
${esc(label)}
${value}
From db3531f85485f3ee08fba126e3ac7165797a0ea1 Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 25 Jun 2026 10:43:11 -0700 Subject: [PATCH 40/51] fix(dashboard): fix KPI ordering, grid span, and border stacking (T-5001.3) - Reorder KPI arrays in renderOverview, renderRate, renderTokenomics, renderAnomaly, renderAllocation so primaries appear first, then references, then supporting KPIs - Remove grid-column:span 2 from .kpi--primary to eliminate auto-fit layout holes on narrow viewports - Fix double top-border: .kpi--primary now uses thicker inset box-shadow (4px) instead of a competing border-top; .kpi--reference uses --kpi-accent:var(--border-muted) + opacity:0.75 instead of border-top - Add --border-muted custom property to :root with GitHub token fallback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../ftk-local-dashboard/public/app.css | 12 +++++-- .../ftk-local-dashboard/public/app.js | 36 +++++++++++++------ 2 files changed, 35 insertions(+), 13 deletions(-) diff --git a/.github/extensions/ftk-local-dashboard/public/app.css b/.github/extensions/ftk-local-dashboard/public/app.css index 217629c6b..78cf55012 100644 --- a/.github/extensions/ftk-local-dashboard/public/app.css +++ b/.github/extensions/ftk-local-dashboard/public/app.css @@ -8,6 +8,7 @@ --card-bg: var(--background-color-default, #ffffff); --radius: 12px; --gap: 16px; + --border-muted: var(--border-color-muted, rgba(128, 128, 128, 0.3)); } * { box-sizing: border-box; } @@ -161,8 +162,15 @@ body { .kpi .meta .warn { color: var(--warn); font-weight: 650; } /* ---------- KPI hierarchy (primary vs reference) ---------- */ -.kpi--primary { grid-column: span 2; border-top: 3px solid var(--accent); } -.kpi--reference { opacity: 0.85; border-top: 1px solid var(--border-muted); } +.kpi--primary { + box-shadow: inset 0 4px 0 var(--kpi-accent, var(--accent)); + font-size: 1.05em; + font-weight: 600; +} +.kpi--reference { + opacity: 0.75; + --kpi-accent: var(--border-muted); +} /* ---------- KPI threshold badges ---------- */ .threshold-green { border-color: var(--pos); } diff --git a/.github/extensions/ftk-local-dashboard/public/app.js b/.github/extensions/ftk-local-dashboard/public/app.js index cbf9e3950..fb43e57f0 100644 --- a/.github/extensions/ftk-local-dashboard/public/app.js +++ b/.github/extensions/ftk-local-dashboard/public/app.js @@ -637,14 +637,18 @@ function renderOverview(p) { const partialHtml = isPartialMonth() ? ` · partial month` : ""; const kpis = [ - kpiCard("Effective cost", fmtMoney(k.eff), `Billed ${fmtMoney(k.billed)}`, PALETTE[0]), - kpiCard("Total savings", fmtMoney(k.savings), - `${fmtPct(k.esr)} effective savings rate`, PALETTE[1]), + // primary KPIs first kpiCard("Untagged cost", fmtPct(k.untaggedPct), `${fmtMoney(k.untagged)} on untagged resources`, PALETTE[3], kpiThreshold(k.untaggedPct, 0.10, 0.25)), + // supporting KPIs + kpiCard("Effective cost", fmtMoney(k.eff), `Billed ${fmtMoney(k.billed)}`, PALETTE[0]), + kpiCard("Total savings", fmtMoney(k.savings), + `${fmtPct(k.esr)} effective savings rate`, PALETTE[1]), + // reference KPIs kpiCard("Commitment coverage", fmtPct(k.coverage), `${fmtMoney(k.committed)} of compute spend`, PALETTE[5]), + // supporting KPIs kpiCard("Tracked resources", fmtInt(k.resources), `${fmtInt(k.services)} services · ${fmtInt(k.subscriptions)} subs · ${fmtInt(k.regions)} regions`, PALETTE[2]), kpiCard("Latest month", k.lastMonthVal == null ? "—" : fmtMoney(k.lastMonthVal), @@ -742,8 +746,10 @@ function renderTokenomics(p) { })); const kpis = [ - kpiCard("AI token cost", fmtMoney(k.eff), `${fmtPct(k.aiShare, 2)} of all cloud cost`, PALETTE[2]), + // reference KPIs first (no primaries in this tab) kpiCard("Total tokens", fmtTokens(k.tokens), `across ${fmtInt(k.models)} model SKUs`, PALETTE[0]), + // supporting KPIs + kpiCard("AI token cost", fmtMoney(k.eff), `${fmtPct(k.aiShare, 2)} of all cloud cost`, PALETTE[2]), kpiCard("Blended rate", fmtPerM(k.blendedPer1K), `per 1M tokens (effective)`, PALETTE[5]), kpiCard("Cached input", fmtPct(k.cachedShare), `${fmtPct(k.cachedShare)} of input tokens cached`, PALETTE[1]), @@ -826,12 +832,14 @@ function renderAnomaly(p) { const p50Days = fr.P50 != null ? fr.P50 / 24 : null; const kpis = [ + // reference KPIs first (no primaries in this tab) + kpiCard("Anomaly days", fmtInt(anomDays.length), + `${fmtMoney(anomCost)} on flagged days`, PALETTE[5]), + // supporting KPIs kpiCard("Anomaly detection rate", fmtPct(rate, 2), `${anomDays.length} flagged of ${daily.length} days`, PALETTE[4]), kpiCard("Unpredicted variance", fmtMoney(variance), `net spend vs baseline on anomaly days`, PALETTE[3]), - kpiCard("Anomaly days", fmtInt(anomDays.length), - `${fmtMoney(anomCost)} on flagged days`, PALETTE[5]), kpiCard("Last month change", lastMc ? `${lastMc.EffChangePct > 0 ? "+" : ""}${lastMc.EffChangePct.toFixed(1)}%` : "—", lastMc ? `effective cost · ${esc(fmtMonth(lastMc.Month))}` : "", PALETTE[lastMc && lastMc.EffChangePct > 0 ? 4 : 1]), kpiCard("Forecast next month", nextFc ? fmtMoney(nextFc.Forecast) : "—", @@ -946,17 +954,21 @@ function renderRate(p) { const coreShare = coreTotal > 0 ? committedCore / coreTotal : 0; const kpis = [ + // primary KPIs first kpiCard("Effective savings rate", fmtPct(esr), `${fmtMoney(s.Total)} total savings · vs. list price`, PALETTE[1]), + kpiCard("Commitment waste", fmtPct(waste), + `${fmtMoney(cm.Unused)} unused · of commitment spend`, PALETTE[3], + kpiThreshold(waste, 0.10, 0.20)), + // supporting KPIs kpiCard("Total savings", fmtMoney(s.Total), `of ${fmtMoney(s.List)} list cost`, PALETTE[2]), kpiCard("Commitment utilization", fmtPct(util), `${fmtMoney(cmTotal - (cm.Unused || 0))} of ${fmtMoney(cmTotal)} used`, PALETTE[0]), - kpiCard("Commitment waste", fmtPct(waste), - `${fmtMoney(cm.Unused)} unused · of commitment spend`, PALETTE[3], - kpiThreshold(waste, 0.10, 0.20)), + // reference KPIs kpiCard("Compute coverage", fmtPct(coverage), `compute spend on commitments`, PALETTE[5]), + // supporting KPIs kpiCard("Committed core-hours", fmtPct(coreShare), `RI + savings plan vs on-demand`, PALETTE[8]), ].join(""); @@ -1023,11 +1035,13 @@ function renderAllocation(p) { const compliancePct = total > 0 ? c.Compliant / total : 0; const kpis = [ - kpiCard("Allocation accuracy", fmtPct(aai), - `directly attributed effective cost`, PALETTE[1]), + // primary KPIs first kpiCard("Untagged cost", fmtPct(untaggedPct), `${fmtMoney(c.Untagged)} with no tags`, PALETTE[3], kpiThreshold(untaggedPct, 0.10, 0.25)), + // supporting KPIs + kpiCard("Allocation accuracy", fmtPct(aai), + `directly attributed effective cost`, PALETTE[1]), kpiCard("Unallocated cost", fmtPct(unallocPct), `${fmtMoney(total - c.Attributed)} lacks allocation evidence`, PALETTE[4]), kpiCard("Tag policy compliance", fmtPct(compliancePct), From 64dddf7c43ae78fd6e8502f0b8c39c72bf3ad5b7 Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 25 Jun 2026 10:47:25 -0700 Subject: [PATCH 41/51] fix(dashboard): fix reference KPI shadow cascade and remove dead font-size rule (T-5001.3) - Replace --kpi-accent re-assignment in .kpi--reference with explicit box-shadow override - Remove ineffective font-size: 1.05em from .kpi--primary (children use absolute px units) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/extensions/ftk-local-dashboard/public/app.css | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/.github/extensions/ftk-local-dashboard/public/app.css b/.github/extensions/ftk-local-dashboard/public/app.css index 78cf55012..fa0e8f2e6 100644 --- a/.github/extensions/ftk-local-dashboard/public/app.css +++ b/.github/extensions/ftk-local-dashboard/public/app.css @@ -164,12 +164,11 @@ body { /* ---------- KPI hierarchy (primary vs reference) ---------- */ .kpi--primary { box-shadow: inset 0 4px 0 var(--kpi-accent, var(--accent)); - font-size: 1.05em; font-weight: 600; } .kpi--reference { opacity: 0.75; - --kpi-accent: var(--border-muted); + box-shadow: inset 0 3px 0 var(--border-muted); } /* ---------- KPI threshold badges ---------- */ From 2fa184681fb6b9d22af37ab2cbf7be5c0a907a48 Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 25 Jun 2026 10:58:11 -0700 Subject: [PATCH 42/51] fix(dashboard): reset row count on error state and fix orphaned separator --- .github/extensions/ftk-local-dashboard/public/app.js | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/extensions/ftk-local-dashboard/public/app.js b/.github/extensions/ftk-local-dashboard/public/app.js index fb43e57f0..92485c9eb 100644 --- a/.github/extensions/ftk-local-dashboard/public/app.js +++ b/.github/extensions/ftk-local-dashboard/public/app.js @@ -652,7 +652,7 @@ function renderOverview(p) { kpiCard("Tracked resources", fmtInt(k.resources), `${fmtInt(k.services)} services · ${fmtInt(k.subscriptions)} subs · ${fmtInt(k.regions)} regions`, PALETTE[2]), kpiCard("Latest month", k.lastMonthVal == null ? "—" : fmtMoney(k.lastMonthVal), - k.mom == null ? `${esc(k.lastMonthLabel || "")}${partialHtml}` : `${momTxt} vs prior · ${esc(k.lastMonthLabel)}${partialHtml}`, PALETTE[4]), + k.mom == null ? (k.lastMonthLabel ? `${esc(k.lastMonthLabel)}${partialHtml}` : (isPartialMonth() ? `partial month` : "")) : `${momTxt} vs prior · ${esc(k.lastMonthLabel)}${partialHtml}`, PALETTE[4]), ].join(""); const d = p.data; @@ -1174,6 +1174,7 @@ function updateChrome() { } else if (p && p.error) { el("source-line").textContent = "Connection failed — see panel below."; el("footer-meta").textContent = ""; + queryState.rows = 0; queryState.health = "error"; queryState.refreshedAt = new Date(); queryState.dataset = "Hub database"; From db01317252c58a50dd3a6553b137b227d2435fdc Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 25 Jun 2026 11:06:51 -0700 Subject: [PATCH 43/51] feat(dashboard): add /api/kql endpoint, refactor error handling (T-5001.5) - Add runQuery import from kusto.mjs - Add /api/kql POST handler for open KQL escape hatch - Extract logError + sendQueryError helpers - Sanitize error messages to avoid leaking stack traces to browser - Harden unhandled rejection path (res.headersSent guard) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../ftk-local-dashboard/extension.mjs | 53 ++++++++++++++++--- 1 file changed, 45 insertions(+), 8 deletions(-) diff --git a/.github/extensions/ftk-local-dashboard/extension.mjs b/.github/extensions/ftk-local-dashboard/extension.mjs index 417e496ee..0411cf516 100644 --- a/.github/extensions/ftk-local-dashboard/extension.mjs +++ b/.github/extensions/ftk-local-dashboard/extension.mjs @@ -9,7 +9,7 @@ import { createServer } from "node:http"; import { readFile } from "node:fs/promises"; import { joinSession, createCanvas, CanvasError } from "@github/copilot-sdk/extension"; -import { getDashboard, getTokenomics, getAllocation, getRate, getUsage, getAnomaly } from "./kusto.mjs"; +import { runQuery, getDashboard, getTokenomics, getAllocation, getRate, getUsage, getAnomaly } from "./kusto.mjs"; const GETTERS = { overview: getDashboard, @@ -40,6 +40,19 @@ function sendJson(res, status, obj) { res.end(body); } +function logError(context, err) { + console.error(`[ftk-local-dashboard] ${context}`, err); +} + +function sendQueryError(res, entry, viewName, err) { + logError(`Could not query ${viewName} for ${entry.clusterUri}/${entry.database}`, err); + sendJson(res, 200, { + error: "Could not query the local FinOps hub. Check the extension logs for details.", + clusterUri: entry.clusterUri, + database: entry.database, + }); +} + async function handleRequest(entry, req, res) { const url = new URL(req.url, "http://127.0.0.1"); const path = url.pathname; @@ -51,8 +64,9 @@ async function handleRequest(entry, req, res) { res.writeHead(200, { "Content-Type": type, "Cache-Control": "no-store" }); res.end(buf); } catch (err) { + logError(`Could not serve asset ${file}`, err); res.writeHead(500, { "Content-Type": "text/plain" }); - res.end(`Asset error: ${err.message}`); + res.end("Asset error"); } return; } @@ -68,7 +82,7 @@ async function handleRequest(entry, req, res) { const payload = await getDashboard(entry.clusterUri, entry.database, preset); sendJson(res, 200, payload); } catch (err) { - sendJson(res, 200, { error: String(err?.message ?? err), clusterUri: entry.clusterUri, database: entry.database }); + sendQueryError(res, entry, "overview", err); } return; } @@ -79,7 +93,7 @@ async function handleRequest(entry, req, res) { const payload = await getTokenomics(entry.clusterUri, entry.database, preset); sendJson(res, 200, payload); } catch (err) { - sendJson(res, 200, { error: String(err?.message ?? err), clusterUri: entry.clusterUri, database: entry.database }); + sendQueryError(res, entry, "tokenomics", err); } return; } @@ -93,7 +107,23 @@ async function handleRequest(entry, req, res) { const payload = await getter(entry.clusterUri, entry.database, preset); sendJson(res, 200, payload); } catch (err) { - sendJson(res, 200, { error: String(err?.message ?? err), clusterUri: entry.clusterUri, database: entry.database }); + sendQueryError(res, entry, name, err); + } + return; + } + + if (path === "/api/kql" && req.method === "POST") { + let body = ""; + for await (const chunk of req) body += chunk; + let kql, database; + try { ({ kql, database } = JSON.parse(body)); } catch { sendJson(res, 400, { error: "Invalid JSON" }); return; } + if (!kql) { sendJson(res, 400, { error: "Missing kql" }); return; } + try { + const rows = await runQuery(entry.clusterUri, database || entry.database, kql); + sendJson(res, 200, { rows }); + } catch (err) { + logError("Custom KQL error", err); + sendJson(res, 200, { error: err.message || "Query failed" }); } return; } @@ -105,7 +135,12 @@ async function handleRequest(entry, req, res) { async function startServer(entry) { const server = createServer((req, res) => { handleRequest(entry, req, res).catch((err) => { - try { sendJson(res, 500, { error: String(err?.message ?? err) }); } catch { /* noop */ } + logError("Unhandled dashboard request failure", err); + if (res.headersSent) { + res.destroy(); + } else { + sendJson(res, 500, { error: "Unexpected dashboard server error" }); + } }); }); await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); @@ -204,7 +239,8 @@ await joinSession({ const payload = await getDashboard(clusterUri, database, preset); return headline(payload); } catch (err) { - throw new CanvasError("query_failed", `Could not query ${clusterUri}/${database}: ${err?.message ?? err}`); + logError(`Could not query summary for ${clusterUri}/${database}`, err); + throw new CanvasError("query_failed", "Could not query the local FinOps hub. Check the extension logs for details."); } }, }, @@ -226,7 +262,8 @@ await joinSession({ const payload = await getTokenomics(clusterUri, database, preset); return tokenHeadline(payload); } catch (err) { - throw new CanvasError("query_failed", `Could not query ${clusterUri}/${database}: ${err?.message ?? err}`); + logError(`Could not query tokenomics for ${clusterUri}/${database}`, err); + throw new CanvasError("query_failed", "Could not query the local FinOps hub. Check the extension logs for details."); } }, }, From bce4ee4dec4ad327cda2d62e922d3b2b100e4453 Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 25 Jun 2026 11:21:28 -0700 Subject: [PATCH 44/51] fix(dashboard): fix tainted format string in logError (CodeQL js/tainted-format-string) Pass context as a separate argument to console.error so it is never used as a printf-style format string. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/extensions/ftk-local-dashboard/extension.mjs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/extensions/ftk-local-dashboard/extension.mjs b/.github/extensions/ftk-local-dashboard/extension.mjs index 0411cf516..00b648bfd 100644 --- a/.github/extensions/ftk-local-dashboard/extension.mjs +++ b/.github/extensions/ftk-local-dashboard/extension.mjs @@ -41,7 +41,7 @@ function sendJson(res, status, obj) { } function logError(context, err) { - console.error(`[ftk-local-dashboard] ${context}`, err); + console.error("[ftk-local-dashboard]", context, err); } function sendQueryError(res, entry, viewName, err) { From b60441d529472921e1e99395b00ba3ed3cf13cc0 Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 25 Jun 2026 11:51:42 -0700 Subject: [PATCH 45/51] feat(dashboard): add CUS, MACC, top-txns, AI-by-app from INDEX.md catalog - Rate tab: commitment-utilization-score (formal KPI per catalog) per-commitment table + grand-total score; top-commitment-transactions table (top 10 RI/SP purchases) - Overview tab: macc-consumption-vs-commitment KPI card (graceful N/A when no Transactions() data) - Tokenomics tab: ai-cost-by-application showback table by app/team/env/cost-center Closes 5 catalog query gaps identified in INDEX.md + KPI.md coverage review. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../extensions/ftk-local-dashboard/kusto.mjs | 28 ++++++ .../ftk-local-dashboard/public/app.js | 87 ++++++++++++++++++- 2 files changed, 113 insertions(+), 2 deletions(-) diff --git a/.github/extensions/ftk-local-dashboard/kusto.mjs b/.github/extensions/ftk-local-dashboard/kusto.mjs index 50c0f9ce1..db7734fa3 100644 --- a/.github/extensions/ftk-local-dashboard/kusto.mjs +++ b/.github/extensions/ftk-local-dashboard/kusto.mjs @@ -133,6 +133,11 @@ export async function getDashboard(clusterUri, database, preset = "all") { topRegions: `Costs() ${period} | where isnotempty(RegionId) | summarize Cost=sum(EffectiveCost) by RegionId | top 12 by Cost desc`, // Charge category mix (Usage / Purchase / Adjustment) chargeCategory: `Costs() ${period} | summarize Cost=sum(EffectiveCost) by ChargeCategory | where Cost != 0 | order by Cost desc`, + // macc-consumption-vs-commitment — MACC burn rate (graceful: returns CommitmentAmount=0 if no MACC data) + macc: `let con = toscalar(Costs() ${period} | where not(ChargeCategory == 'Purchase' and isnotempty(CommitmentDiscountCategory)) | summarize sum(EffectiveCost)); +let com = toscalar(Transactions() | where isnotnull(x_MonetaryCommitment) | summarize sum(x_MonetaryCommitment)); +let com0 = coalesce(todouble(com), 0.0); +print ConsumptionAmount=con, CommitmentAmount=com0, CommitmentBurnPercent=iff(com0 > 0, con / com0 * 100.0, 0.0)`, }; const entries = Object.entries(queries); @@ -194,6 +199,15 @@ export async function getTokenomics(clusterUri, database, preset = "all") { models: `Costs() ${period} ${AI_SCOPE} | ${MODEL_FAMILY} | summarize Tokens=sum(ConsumedQuantity), Cost=sum(EffectiveCost), List=sum(ListCost) by Model | extend CostPer1K=iff(Tokens==0, 0.0, Cost/Tokens*1000) | top 12 by Cost desc`, // ai-daily-trend (monthly variant) — token volume + AI cost over time trend: `Costs() ${period} ${AI_SCOPE} | summarize Tokens=sum(ConsumedQuantity), Cost=sum(EffectiveCost) by Month=format_datetime(startofmonth(ChargePeriodStart),'yyyy-MM') | order by Month asc`, + // ai-cost-by-application — AI cost showback by app/team/env/cost-center + byApplication: `Costs() ${period} ${AI_SCOPE} +| extend Application = tostring(Tags['application']), Team = tostring(Tags['team']) +| extend CostCenter = coalesce(tostring(Tags['cost-center']), tostring(Tags['CostCenter']), '') +| extend Environment = tostring(Tags['environment']) +| summarize TokenCount=sum(ConsumedQuantity), EffectiveCost=sum(EffectiveCost) + by Application, Team, CostCenter, Environment +| extend CostPer1KTokens = iff(TokenCount == 0, 0.0, EffectiveCost / TokenCount * 1000) +| top 12 by EffectiveCost desc`, }; const entries = Object.entries(queries); @@ -289,6 +303,20 @@ export async function getRate(clusterUri, database, preset = "all") { byCommitment: `Costs() ${period} | where isnotempty(CommitmentDiscountName) ${NON_PURCHASE} | summarize Unused=sumif(EffectiveCost,CommitmentDiscountStatus=='Unused'), Total=sum(EffectiveCost) by CommitmentDiscountName | where Unused > 0 | top 10 by Unused desc`, + // commitment-utilization-score (formal KPI) — per-commitment and grand-total utilization score + commitmentUtilScore: `let rows = materialize(Costs() ${period} | where isnotempty(CommitmentDiscountId) +| extend Potential = case(ChargeCategory == 'Purchase', toreal(0), isnotempty(CommitmentDiscountCategory), toreal(EffectiveCost), toreal(0)) +| extend Amount = iff(CommitmentDiscountStatus == 'Used', Potential, toreal(0))); +let byCommit = rows | summarize Amount=sum(Amount), Potential=sum(Potential) by CommitmentDiscountName, CommitmentDiscountCategory, CommitmentDiscountType +| extend Score = iff(Potential > 0, Amount / Potential * 100.0, 0.0); +union byCommit, (byCommit | summarize Amount=sum(Amount), Potential=sum(Potential) +| extend CommitmentDiscountName='(Grand Total)', CommitmentDiscountCategory='', CommitmentDiscountType='', Score=iff(Potential>0, Amount/Potential*100.0, 0.0)) +| project CommitmentDiscountName, CommitmentDiscountCategory, CommitmentDiscountType, Amount, Potential, Score | order by Potential desc`, + // top-commitment-transactions — largest RI/SP purchases + topCommitmentTxns: `Costs() ${period} | where ChargeCategory != 'Usage' and isnotempty(CommitmentDiscountType) and BilledCost > 0 +| summarize BilledCost=sum(BilledCost), EffectiveCost=sum(EffectiveCost) + by CommitmentDiscountName, CommitmentDiscountType, CommitmentDiscountCategory +| top 10 by BilledCost desc`, })); } diff --git a/.github/extensions/ftk-local-dashboard/public/app.js b/.github/extensions/ftk-local-dashboard/public/app.js index 92485c9eb..79b804dac 100644 --- a/.github/extensions/ftk-local-dashboard/public/app.js +++ b/.github/extensions/ftk-local-dashboard/public/app.js @@ -635,6 +635,7 @@ function renderOverview(p) { const momTxt = k.mom == null ? "—" : `${k.mom > 0 ? "▲" : "▼"} ${fmtPct(Math.abs(k.mom))}`; const partialHtml = isPartialMonth() ? ` · partial month` : ""; + const maccRow = p.data.macc?.[0] || { ConsumptionAmount: 0, CommitmentAmount: 0, CommitmentBurnPercent: 0 }; const kpis = [ // primary KPIs first @@ -653,6 +654,13 @@ function renderOverview(p) { `${fmtInt(k.services)} services · ${fmtInt(k.subscriptions)} subs · ${fmtInt(k.regions)} regions`, PALETTE[2]), kpiCard("Latest month", k.lastMonthVal == null ? "—" : fmtMoney(k.lastMonthVal), k.mom == null ? (k.lastMonthLabel ? `${esc(k.lastMonthLabel)}${partialHtml}` : (isPartialMonth() ? `partial month` : "")) : `${momTxt} vs prior · ${esc(k.lastMonthLabel)}${partialHtml}`, PALETTE[4]), + // macc-consumption-vs-commitment — MACC burn rate + kpiCard("MACC burn rate", + maccRow.CommitmentAmount > 0 ? fmtPct(maccRow.CommitmentBurnPercent / 100) : "N/A", + maccRow.CommitmentAmount > 0 + ? `${fmtMoney(maccRow.ConsumptionAmount)} of ${fmtMoney(maccRow.CommitmentAmount)} committed` + : "No Microsoft Azure Consumption Commitment data", + PALETTE[7]), ].join(""); const d = p.data; @@ -776,6 +784,11 @@ function renderTokenomics(p) {
${panelHtml("token-model-table", 12, "Cost per 1M tokens by model", "Unit economics for model selection — sorted by effective cost.", tokenModelTable(d.models, k.eff))}
+ +

AI cost allocation

Showback & chargeback · ai-cost-by-application
+
+ ${panelHtml("token-by-app", 12, "AI cost by application", "Azure OpenAI effective cost and token volume by application, team, environment, and cost center.", aiByAppTable(d.byApplication))} +
`; } @@ -801,6 +814,29 @@ function tokenModelTable(models, totalCost) { `; } +function aiByAppTable(rows) { + const data = (rows || []).filter((r) => (r.EffectiveCost || 0) > 0); + if (data.length === 0) return `

No tagged AI cost data. Tag Azure OpenAI resources with application, team, or environment tags.

`; + const totalCost = data.reduce((a, r) => a + (r.EffectiveCost || 0), 0); + return ` + + ${data.map((r, i) => { + const share = totalCost > 0 ? (r.EffectiveCost || 0) / totalCost : 0; + const color = PALETTE[i % PALETTE.length]; + return ` + + + + + + + + + `; + }).join("")} +
ApplicationTeamEnvironmentCost centerTokensEffective cost$/1M tokens% of AI
${esc(r.Application || "(untagged)")}${esc(r.Team || "—")}${esc(r.Environment || "—")}${esc(r.CostCenter || "—")}${fmtTokens(r.TokenCount)}${fmtMoney(r.EffectiveCost)}${fmtPerM(r.CostPer1KTokens)}${fmtPct(share)}
`; +} + /* --------------------------------------------- anomalies & forecast render */ function renderAnomaly(p) { @@ -963,8 +999,15 @@ function renderRate(p) { // supporting KPIs kpiCard("Total savings", fmtMoney(s.Total), `of ${fmtMoney(s.List)} list cost`, PALETTE[2]), - kpiCard("Commitment utilization", fmtPct(util), - `${fmtMoney(cmTotal - (cm.Unused || 0))} of ${fmtMoney(cmTotal)} used`, PALETTE[0]), + (() => { + const cusRow = (d.commitmentUtilScore || []).find((r) => r.CommitmentDiscountName === '(Grand Total)'); + const cusScore = cusRow ? cusRow.Score / 100 : util; + return kpiCard("Commitment utilization", fmtPct(cusScore), + cusRow + ? `${fmtMoney(cusRow.Amount)} utilized of ${fmtMoney(cusRow.Potential)} potential` + : `${fmtMoney(cmTotal - (cm.Unused || 0))} of ${fmtMoney(cmTotal)} used`, + PALETTE[0]); + })(), // reference KPIs kpiCard("Compute coverage", fmtPct(coverage), `compute spend on commitments`, PALETTE[5]), @@ -1006,6 +1049,12 @@ function renderRate(p) { ${panelHtml("rate-underutil", 6, "Underutilized commitments", "Reservations & plans with the most unused cost.", hbar(d.byCommitment, "CommitmentDiscountName", "Unused"))}
+ +

Commitment transactions

Rate optimization · Commitment purchasing
+
+ ${panelHtml("rate-commit-score", 6, "Commitment utilization score", "Per-commitment utilization (used vs potential) from the formal CUS KPI.", commitUtilTable(d.commitmentUtilScore))} + ${panelHtml("rate-top-txns", 6, "Top commitment transactions", "Largest RI and savings plan purchases by billed cost.", topCommitTxnTable(d.topCommitmentTxns))} +
`; } @@ -1016,6 +1065,40 @@ function rateRow(label, val, accent) { ${fmtMoney(val)}
`; } +function commitUtilTable(rows) { + const data = (rows || []).filter((r) => r.CommitmentDiscountName !== '(Grand Total)' && (r.Potential || 0) > 0); + if (data.length === 0) return `

No commitment data in range.

`; + return ` + + ${data.map((r) => { + const score = r.Score || 0; + const cls = score < 70 ? "neg" : score < 90 ? "warn" : "pos"; + const barW = Math.max(2, (score / 100) * 90); + return ` + + + + + + `; + }).join("")} +
CommitmentTypeScoreUtilizedPotential
${esc(r.CommitmentDiscountName)}${esc(r.CommitmentDiscountType || r.CommitmentDiscountCategory || "")}${fmtPct(score / 100)}${fmtMoney(r.Amount)}${fmtMoney(r.Potential)}
`; +} + +function topCommitTxnTable(rows) { + const data = rows || []; + if (data.length === 0) return `

No commitment transactions in range.

`; + return ` + + ${data.map((r) => ` + + + + + `).join("")} +
CommitmentTypeBilled costEffective cost
${esc(r.CommitmentDiscountName || "(unknown)")}${esc(r.CommitmentDiscountType || "")}${fmtMoney(r.BilledCost)}${fmtMoney(r.EffectiveCost)}
`; +} + /* ----------------------------------------------------- allocation render */ function renderAllocation(p) { From c5a23e117b65ea911c39c40064d00e30fd1ac2f4 Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 25 Jun 2026 12:05:17 -0700 Subject: [PATCH 46/51] feat(canvas): add interactive click-to-filter to dashboard MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Filter state: state.filters {dim -> string[]} + filterKey()/cacheKey()/filterParam() - hbar(): filterable rows get data-filter-dim/val attrs, CSS classes hbar-filterable/selected/dimmed - renderFilterBar(): renders chips (×-removable) + Reset all button in #filter-bar - wireControls(): event delegation for hbar click, chip remove, reset - Server-side KQL injection: buildFilterWhere() in kusto.mjs appends | where Col in (...) - extension.mjs parseFilters() reads ?filters= and forwards to all 6 getters - Cache keyed by preset+filterKey() so filtered/unfiltered results are cached separately - tagKeys hbar opts out of filtering (synthetic key column) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../extensions/ftk-local-dashboard/PRODUCT.md | 36 +++++ .../ftk-local-dashboard/extension.mjs | 20 ++- .../extensions/ftk-local-dashboard/kusto.mjs | 48 ++++--- .../ftk-local-dashboard/public/app.css | 83 ++++++++++++ .../ftk-local-dashboard/public/app.js | 127 ++++++++++++++++-- .../ftk-local-dashboard/public/index.html | 6 + 6 files changed, 289 insertions(+), 31 deletions(-) create mode 100644 .github/extensions/ftk-local-dashboard/PRODUCT.md diff --git a/.github/extensions/ftk-local-dashboard/PRODUCT.md b/.github/extensions/ftk-local-dashboard/PRODUCT.md new file mode 100644 index 000000000..65ab9a021 --- /dev/null +++ b/.github/extensions/ftk-local-dashboard/PRODUCT.md @@ -0,0 +1,36 @@ +# Product + +## Register + +product + +## Users + +FinOps practitioners, cloud engineers, and consultants who need to analyze Azure cost data locally — without deploying Azure resources. They run this inside GitHub Copilot as a canvas panel while they work: exploring the data model, validating large datasets, or doing FinOps analysis in disconnected / on-premises environments. They are data-fluent, comfortable with KQL and Azure concepts, and expect density and precision over decoration. They are in a task when they open this — they want numbers fast. + +## Product Purpose + +A local FinOps hub dashboard that connects to a Kusto emulator running in Docker. Provides the same six analysis views as a deployed FinOps hub (cost overview, allocation, rate optimization, usage & unit economics, anomalies & forecast, AI tokenomics) — but entirely on-device. Success means a practitioner can load cost data, switch views, and spot the insight they need in one sitting without touching Azure. + +## Brand Personality + +Precise. Grounded. Efficient. The interface should feel like a well-calibrated instrument, not a product pitch. Numbers are the hero; the chrome disappears. + +## Anti-references + +- Consumer personal finance dashboards (Mint, Copilot Money) — too soft, too colorful +- SaaS marketing dashboards (hero metric templates, gradient text, glassmorphism cards) +- Over-designed BI tools with heavy chrome, deep sidebars, and modal-heavy workflows +- Any interface that prioritizes looking impressive over being immediately useful + +## Design Principles + +1. **Numbers first** — KPIs and data are the primary visual element. Supporting chrome (headers, tabs, labels) recedes. +2. **GitHub-native** — Use GitHub design tokens (`--background-color-default`, `--text-color-default`, etc.) so the panel feels like an extension of Copilot, not a foreign app. +3. **Density is a virtue** — FinOps data is inherently multi-dimensional. Don't sacrifice information density for whitespace. +4. **State is explicit** — Loading, error, empty, and no-data states are real states, not afterthoughts. Every panel handles all of them. +5. **Zero ceremony** — No animated intros, no onboarding tours. Open panel → see data. + +## Accessibility & Inclusion + +WCAG AA minimum. SVG charts include `` elements for screen-reader context. Interactive controls (tabs, segmented control, refresh) have ARIA roles and labels. Reduced motion is respected via `@media (prefers-reduced-motion)`. diff --git a/.github/extensions/ftk-local-dashboard/extension.mjs b/.github/extensions/ftk-local-dashboard/extension.mjs index 00b648bfd..89ecd798b 100644 --- a/.github/extensions/ftk-local-dashboard/extension.mjs +++ b/.github/extensions/ftk-local-dashboard/extension.mjs @@ -34,6 +34,17 @@ const STATIC = { // instanceId -> { server, url, clusterUri, database } const servers = new Map(); +/** Parse `?filters=<JSON>` from a URL search params; returns plain object (column->array). */ +function parseFilters(url) { + const raw = url.searchParams.get("filters"); + if (!raw) return {}; + try { + const parsed = JSON.parse(raw); + if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) return parsed; + } catch { /* ignore malformed */ } + return {}; +} + function sendJson(res, status, obj) { const body = JSON.stringify(obj); res.writeHead(status, { "Content-Type": "application/json; charset=utf-8", "Cache-Control": "no-store" }); @@ -78,8 +89,9 @@ async function handleRequest(entry, req, res) { if (path === "/api/dashboard") { const preset = url.searchParams.get("preset") || "all"; + const filters = parseFilters(url); try { - const payload = await getDashboard(entry.clusterUri, entry.database, preset); + const payload = await getDashboard(entry.clusterUri, entry.database, preset, filters); sendJson(res, 200, payload); } catch (err) { sendQueryError(res, entry, "overview", err); @@ -89,8 +101,9 @@ async function handleRequest(entry, req, res) { if (path === "/api/tokenomics") { const preset = url.searchParams.get("preset") || "all"; + const filters = parseFilters(url); try { - const payload = await getTokenomics(entry.clusterUri, entry.database, preset); + const payload = await getTokenomics(entry.clusterUri, entry.database, preset, filters); sendJson(res, 200, payload); } catch (err) { sendQueryError(res, entry, "tokenomics", err); @@ -101,10 +114,11 @@ async function handleRequest(entry, req, res) { if (path === "/api/view") { const name = url.searchParams.get("name") || "overview"; const preset = url.searchParams.get("preset") || "all"; + const filters = parseFilters(url); const getter = GETTERS[name]; if (!getter) { sendJson(res, 200, { error: `Unknown view '${name}'` }); return; } try { - const payload = await getter(entry.clusterUri, entry.database, preset); + const payload = await getter(entry.clusterUri, entry.database, preset, filters); sendJson(res, 200, payload); } catch (err) { sendQueryError(res, entry, name, err); diff --git a/.github/extensions/ftk-local-dashboard/kusto.mjs b/.github/extensions/ftk-local-dashboard/kusto.mjs index db7734fa3..be4ab832e 100644 --- a/.github/extensions/ftk-local-dashboard/kusto.mjs +++ b/.github/extensions/ftk-local-dashboard/kusto.mjs @@ -100,19 +100,35 @@ export async function resolveWindow(clusterUri, database, preset) { }; } -// --- dashboard data ----------------------------------------------------------- +/** + * Build a KQL `| where` clause from a filters object `{ ColumnName: ["val1","val2"] }`. + * Returns an empty string when there are no filters. + */ +function buildFilterWhere(filters) { + if (!filters || typeof filters !== "object") return ""; + const clauses = Object.entries(filters) + .filter(([, vals]) => Array.isArray(vals) && vals.length > 0) + .map(([col, vals]) => { + const quoted = vals.map((v) => `'${String(v).replace(/'/g, "''")}'`).join(", "); + return `| where ${col} in (${quoted})`; + }); + return clauses.length > 0 ? "\n" + clauses.join("\n") : ""; +} + + /** * Run all dashboard queries in parallel for the resolved window and shape the * result into a single payload the renderer consumes. */ -export async function getDashboard(clusterUri, database, preset = "all") { +export async function getDashboard(clusterUri, database, preset = "all", filters = {}) { const win = await resolveWindow(clusterUri, database, preset); if (win.empty) { return { window: win, empty: true, generatedAt: new Date().toISOString() }; } - const period = `| where ChargePeriodStart >= datetime(${win.start}) and ChargePeriodStart < datetime(${win.end})`; + const filterWhere = buildFilterWhere(filters); + const period = `| where ChargePeriodStart >= datetime(${win.start}) and ChargePeriodStart < datetime(${win.end})${filterWhere}`; const queries = { // KPI totals — Understand Usage & Cost + Quantify Business Value @@ -181,12 +197,13 @@ const MODEL_FAMILY = `extend Model = x_SkuDescription | extend Model = replace_regex(Model, @'(?i)[\\s-]+(cached[\\s-]+)?(inp|inpt|outp|out|chat|media)([\\s-].*)?$', '') | extend Model = replace_regex(trim(@'[\\s-]+', Model), @'(?i)^gpt', 'GPT')`; -export async function getTokenomics(clusterUri, database, preset = "all") { +export async function getTokenomics(clusterUri, database, preset = "all", filters = {}) { const win = await resolveWindow(clusterUri, database, preset); if (win.empty) { return { window: win, empty: true, generatedAt: new Date().toISOString() }; } - const period = `| where ChargePeriodStart >= datetime(${win.start}) and ChargePeriodStart < datetime(${win.end})`; + const filterWhere = buildFilterWhere(filters); + const period = `| where ChargePeriodStart >= datetime(${win.start}) and ChargePeriodStart < datetime(${win.end})${filterWhere}`; const queries = { // Token KPI totals — Token Consumption Metrics @@ -226,10 +243,11 @@ export async function getTokenomics(clusterUri, database, preset = "all") { // --- shared page runner ------------------------------------------------------- // Resolves the window, builds a named map of KQL queries from the period clause, // runs them in parallel, and returns { window, empty, data, generatedAt }. -async function runPage(clusterUri, database, preset, buildQueries) { +async function runPage(clusterUri, database, preset, buildQueries, filters = {}) { const win = await resolveWindow(clusterUri, database, preset); if (win.empty) return { window: win, empty: true, generatedAt: new Date().toISOString() }; - const period = `| where ChargePeriodStart >= datetime(${win.start}) and ChargePeriodStart < datetime(${win.end})`; + const filterWhere = buildFilterWhere(filters); + const period = `| where ChargePeriodStart >= datetime(${win.start}) and ChargePeriodStart < datetime(${win.end})${filterWhere}`; const queries = buildQueries(period, win); const entries = Object.entries(queries); const results = await Promise.all(entries.map(([, csl]) => runQuery(clusterUri, database, csl))); @@ -245,7 +263,7 @@ async function runPage(clusterUri, database, preset, buildQueries) { // the enriched x_CostCenter / x_CostAllocationRuleName columns. const NON_PURCHASE = `| where not(ChargeCategory == 'Purchase' and isnotempty(CommitmentDiscountCategory))`; -export async function getAllocation(clusterUri, database, preset = "all") { +export async function getAllocation(clusterUri, database, preset = "all", filters = {}) { return runPage(clusterUri, database, preset, (period) => ({ // Single-pass core: total, untagged, attributed (AAI), compliant core: `let req=dynamic(['CostCenter','env','org']); @@ -269,7 +287,7 @@ Costs() ${period} ${NON_PURCHASE} | top 12 by Cost desc`, // Cost by subscription (SubAccountName) bySubscription: `Costs() ${period} | where isnotempty(SubAccountName) | summarize Cost=sum(EffectiveCost) by SubAccountName | top 10 by Cost desc`, - })); + }), filters); } // --- Rate optimization page --------------------------------------------------- @@ -279,7 +297,7 @@ Costs() ${period} ${NON_PURCHASE} // depends on Recommendations(), which is empty in this estate, so it would always read 100.) // Commitment utilization is derived as the effective-cost complement of waste, the cleanest // single-basis definition for a grand-total KPI. -export async function getRate(clusterUri, database, preset = "all") { +export async function getRate(clusterUri, database, preset = "all", filters = {}) { return runPage(clusterUri, database, preset, (period) => ({ // savings-summary-report — ESR + negotiated/commitment/total savings savings: `Costs() ${period} ${NON_PURCHASE} @@ -317,13 +335,13 @@ union byCommit, (byCommit | summarize Amount=sum(Amount), Potential=sum(Potentia | summarize BilledCost=sum(BilledCost), EffectiveCost=sum(EffectiveCost) by CommitmentDiscountName, CommitmentDiscountType, CommitmentDiscountCategory | top 10 by BilledCost desc`, - })); + }), filters); } // --- Usage & unit economics page ---------------------------------------------- // FinOps "Usage Optimization" + "Unit Economics". Grounded in catalog queries: // compute-cost-per-core, cost-per-gb-stored, storage-tier-distribution, top-resource-types-by-cost. -export async function getUsage(clusterUri, database, preset = "all") { +export async function getUsage(clusterUri, database, preset = "all", filters = {}) { return runPage(clusterUri, database, preset, (period) => ({ // compute-cost-per-core compute: `Costs() ${period} ${NON_PURCHASE} @@ -355,7 +373,7 @@ export async function getUsage(clusterUri, database, preset = "all") { | where CH > 100 | extend PerCore=Eff/CH | top 10 by Eff desc`, // grand total for share-of-cost on the resource-type table total: `Costs() ${period} | summarize Total=sum(EffectiveCost)`, - })); + }), filters); } // --- Anomalies & forecast page ------------------------------------------------ @@ -364,7 +382,7 @@ export async function getUsage(clusterUri, database, preset = "all") { // anomaly-variance-total, monthly-cost-change-percentage, cost-forecasting-model, // data-update-frequency, cost-visibility-delay. Time-series array outputs are // flattened with mv-expand so the renderer can chart them. -export async function getAnomaly(clusterUri, database, preset = "all") { +export async function getAnomaly(clusterUri, database, preset = "all", filters = {}) { return runPage(clusterUri, database, preset, (period, win) => { // forecast uses full history for accuracy; horizon = 4 months past the last data month const dmax = new Date(win.dataMax); @@ -398,5 +416,5 @@ Costs() | where ChargePeriodStart>=s and ChargePeriodStart<e ${NON_PURCHASE} freshness: `Costs() ${period} | where isnotnull(x_IngestionTime) | summarize LastUpdate=max(x_IngestionTime), Rows=count(), P50=percentile(todouble((x_IngestionTime-ChargePeriodEnd)/1h),50), P90=percentile(todouble((x_IngestionTime-ChargePeriodEnd)/1h),90)`, }; - }); + }, filters); } diff --git a/.github/extensions/ftk-local-dashboard/public/app.css b/.github/extensions/ftk-local-dashboard/public/app.css index fa0e8f2e6..ed9fd858b 100644 --- a/.github/extensions/ftk-local-dashboard/public/app.css +++ b/.github/extensions/ftk-local-dashboard/public/app.css @@ -519,3 +519,86 @@ svg { display: block; width: 100%; overflow: visible; } } +/* ---------- filter bar ---------- */ +.filter-bar { + display: flex; + align-items: center; + gap: 8px; + padding: 7px 20px; + margin: -12px 0 14px; + background: color-mix(in srgb, var(--accent) 7%, transparent); + border: 1px solid color-mix(in srgb, var(--accent) 22%, transparent); + border-radius: 10px; + flex-wrap: wrap; +} +.filter-label { + font-size: 11.5px; + font-weight: 600; + color: var(--muted); + white-space: nowrap; +} +.filter-chips { + display: flex; + gap: 6px; + flex-wrap: wrap; + flex: 1; +} +.filter-chip { + display: inline-flex; + align-items: center; + gap: 1px; + background: color-mix(in srgb, var(--accent) 14%, var(--card-bg)); + border: 1px solid color-mix(in srgb, var(--accent) 35%, transparent); + border-radius: 20px; + padding: 2px 2px 2px 10px; + font-size: 12px; + color: var(--text-color-default, #1f2328); + line-height: 1.5; +} +.filter-chip strong { font-weight: 650; } +.chip-label { display: flex; gap: 4px; align-items: baseline; } +.chip-remove { + all: unset; + display: flex; + align-items: center; + justify-content: center; + cursor: pointer; + color: var(--muted); + font-size: 13px; + width: 22px; + height: 22px; + border-radius: 50%; + margin-left: 2px; +} +.chip-remove:hover { + background: color-mix(in srgb, var(--muted) 18%, transparent); + color: var(--text-color-default, #1f2328); +} +.filter-reset { + appearance: none; + background: transparent; + border: 1px solid var(--grid); + border-radius: 6px; + cursor: pointer; + color: var(--muted); + font-size: 11.5px; + font-weight: 600; + padding: 3px 10px; + white-space: nowrap; +} +.filter-reset:hover { + border-color: color-mix(in srgb, var(--muted) 70%, transparent); + color: var(--text-color-default, #1f2328); +} + +/* ---------- interactive chart elements ---------- */ +svg .hbar-filterable { cursor: pointer; } +svg .hbar-filterable:hover rect.hbar { filter: brightness(1.12); } +svg .hbar-filterable:hover text.name { fill: var(--accent); } +svg .hbar-dimmed { opacity: 0.25; } +svg .hbar-selected rect.hbar { stroke: var(--text-color-default, #1f2328); stroke-width: 1.5; stroke-opacity: 0.35; } +svg .hbar-selected text.name { font-weight: 700; fill: var(--accent); } + +@media (prefers-reduced-motion: no-preference) { + svg .hbar-dimmed { transition: opacity 0.12s ease-out; } +} diff --git a/.github/extensions/ftk-local-dashboard/public/app.js b/.github/extensions/ftk-local-dashboard/public/app.js index 79b804dac..49795c6d3 100644 --- a/.github/extensions/ftk-local-dashboard/public/app.js +++ b/.github/extensions/ftk-local-dashboard/public/app.js @@ -9,9 +9,20 @@ const PALETTE = [ "#ec4899", "#84cc16", "#f97316", "#6366f1", "#14b8a6", "#a855f7", ]; -const state = { preset: "all", tab: "overview", loading: false, cache: {} }; +const state = { preset: "all", tab: "overview", loading: false, cache: {}, filters: {} }; const queryState = { rows: 0, health: "ok", refreshedAt: null, dataset: "Hub database" }; +/** Human-readable labels for filter dimensions (used in chips). */ +const FILTER_LABELS = { + ServiceName: "Service", + ServiceCategory: "Category", + RegionId: "Region", + x_ResourceGroupName: "Resource group", + SubAccountName: "Subscription", + CommitmentDiscountName: "Commitment", + x_SkuMeterSubcategory: "Meter", +}; + /* ----------------------------------------------------------------- KQL templates */ const PERIOD = "| where ChargePeriodStart >= datetime({start}) and ChargePeriodStart < datetime({end})"; @@ -51,6 +62,69 @@ const PANEL_KQL = { let _kqlPanelId = null; +/* ------------------------------------------------------------------ filter management */ + +function filterKey() { + const entries = Object.entries(state.filters) + .filter(([, arr]) => arr && arr.length > 0) + .sort(([a], [b]) => a.localeCompare(b)); + return entries.length > 0 ? "|" + JSON.stringify(entries) : ""; +} + +function cacheKey() { + return state.preset + filterKey(); +} + +function filterParam() { + const entries = Object.entries(state.filters) + .filter(([, arr]) => arr && arr.length > 0); + if (entries.length === 0) return ""; + return "&filters=" + encodeURIComponent(JSON.stringify(Object.fromEntries(entries))); +} + +function toggleFilter(dim, val) { + const arr = state.filters[dim] || []; + const idx = arr.indexOf(val); + if (idx >= 0) { + const next = arr.filter((v) => v !== val); + if (next.length === 0) delete state.filters[dim]; + else state.filters[dim] = next; + } else { + state.filters[dim] = [...arr, val]; + } + renderFilterBar(); + load(); +} + +function clearFilters() { + state.filters = {}; + renderFilterBar(); + load(); +} + +function renderFilterBar() { + const bar = document.getElementById("filter-bar"); + const chips = document.getElementById("filter-chips"); + if (!bar || !chips) return; + const entries = Object.entries(state.filters).filter(([, arr]) => arr && arr.length > 0); + if (entries.length === 0) { + bar.hidden = true; + chips.innerHTML = ""; + return; + } + bar.hidden = false; + chips.innerHTML = entries.flatMap(([dim, vals]) => + vals.map((val) => { + const label = FILTER_LABELS[dim] || dim; + return `<span class="filter-chip" data-dim="${esc(dim)}" data-val="${esc(val)}">` + + `<span class="chip-label"><strong>${esc(label)}</strong> ${esc(val)}</span>` + + `<button class="chip-remove" data-dim="${esc(dim)}" data-val="${esc(val)}" ` + + `aria-label="Remove filter ${esc(label)}: ${esc(val)}" type="button">×</button>` + + `</span>`; + }) + ).join(""); +} + /* ------------------------------------------------------------------ utils */ function fmtMoney(n) { @@ -179,6 +253,10 @@ function hbar(rows, nameKey, valKey, opts = {}) { const rowH = 30, padR = 64, nameW = opts.nameW ?? 142; const W = 540, H = data.length * rowH + 6; const barX = nameW + 8, barW = W - barX - padR; + // filterDim: by default use nameKey; pass null to opt-out of filtering + const filterDim = "filterDim" in opts ? opts.filterDim : nameKey; + const activeVals = filterDim && state.filters[filterDim]; + const hasFilter = activeVals && activeVals.length > 0; let g = ""; data.forEach((d, i) => { const yy = i * rowH + 4; @@ -186,7 +264,14 @@ function hbar(rows, nameKey, valKey, opts = {}) { const w = Math.max(2, (d.val / max) * barW); const color = opts.color || PALETTE[i % PALETTE.length]; const pct = total > 0 ? (d.val / total) : 0; - g += `<g class="hbar-row">`; + const isSelected = hasFilter && activeVals.includes(d.name); + const isDimmed = hasFilter && !isSelected; + let cls = "hbar-row"; + if (filterDim) cls += " hbar-filterable"; + if (isSelected) cls += " hbar-selected"; + if (isDimmed) cls += " hbar-dimmed"; + const dimAttr = filterDim ? ` data-filter-dim="${esc(filterDim)}" data-filter-val="${esc(d.name)}"` : ""; + g += `<g class="${cls}"${dimAttr}>`; g += `<text class="name" x="0" y="${cy + 4}">${esc(trunc(d.name, 20))}<title>${esc(d.name)}`; g += `${esc(d.name)}\n${fmtMoneyFull(d.val)} · ${fmtPct(pct)}`; g += `${fmtMoney(d.val)}`; @@ -491,8 +576,8 @@ function buildTriageTile(title, count, cue, tabId) { } function renderTriageStrip(d) { - // Anomalies: reuse anomaly tab cache when loaded - const anomPayload = state.cache["anomaly"]?.[state.preset]; + // Anomalies: reuse anomaly tab cache when loaded (use same cache key for consistency) + const anomPayload = state.cache["anomaly"]?.[cacheKey()]; const daily = anomPayload?.data?.daily || []; const anomCount = anomPayload ? daily.filter((r) => r.Flag !== 0).length : null; const anomCue = anomCount === null ? "Visit Anomalies & forecast tab to load" @@ -512,7 +597,7 @@ function renderTriageStrip(d) { : `${overspendCount} month${overspendCount === 1 ? "" : "s"} with >20% spike`; // Savings opportunities: underutilized commitments from rate tab cache when loaded - const ratePayload = state.cache["rate"]?.[state.preset]; + const ratePayload = state.cache["rate"]?.[cacheKey()]; const byCommitment = ratePayload?.data?.byCommitment || []; const savingsCount = ratePayload ? byCommitment.filter((r) => (r.Unused || 0) > 0).length : null; const savingsCue = savingsCount === null ? "Visit Rate optimization tab to load" @@ -1158,7 +1243,7 @@ function renderAllocation(p) { { label: "Tagged", value: total - c.Untagged, color: PALETTE[1] }, { label: "Untagged", value: c.Untagged, color: PALETTE[3] }, ], { centerBig: fmtPct(1 - untaggedPct), centerSmall: "tagged" }))} - ${panelHtml("alloc-tag-keys", 6, "Cost by tag key", "Effective cost touched by each governance tag.", hbar(d.tagKeys, "k", "Cost"))} + ${panelHtml("alloc-tag-keys", 6, "Cost by tag key", "Effective cost touched by each governance tag.", hbar(d.tagKeys, "k", "Cost", { filterDim: null }))} ${panelHtml("alloc-by-subscription", 6, "Cost by subscription", "Spend per billing scope for showback.", hbar(d.bySubscription, "SubAccountName", "Cost"))}
`; @@ -1185,7 +1270,7 @@ const ENDPOINT = { }; function currentPayload() { - return state.cache[state.tab]?.[state.preset]; + return state.cache[state.tab]?.[cacheKey()]; } function render() { @@ -1200,18 +1285,18 @@ function render() { } async function load() { - const tab = state.tab, preset = state.preset; - if (state.cache[tab]?.[preset]) { updateChrome(); render(); return; } + const tab = state.tab, key = cacheKey(); + if (state.cache[tab]?.[key]) { updateChrome(); render(); return; } state.cache[tab] = state.cache[tab] || {}; state.loading = true; setRefreshSpinning(true); el("content").innerHTML = `
Loading ${tab} data…
`; try { - const res = await fetch(`${ENDPOINT[tab]}&preset=${encodeURIComponent(preset)}`); - state.cache[tab][preset] = await res.json(); + const res = await fetch(`${ENDPOINT[tab]}&preset=${encodeURIComponent(state.preset)}${filterParam()}`); + state.cache[tab][key] = await res.json(); } catch (err) { console.error("[ftk-dashboard] fetch failed:", err); - state.cache[tab][preset] = { error: "Could not load data. Check that the Kusto emulator is running." }; + state.cache[tab][key] = { error: "Could not load data. Check that the Kusto emulator is running." }; } state.loading = false; setRefreshSpinning(false); @@ -1279,7 +1364,7 @@ function wireControls() { }); el("refresh").addEventListener("click", () => { if (state.loading) return; - if (state.cache[state.tab]) delete state.cache[state.tab][state.preset]; // force re-query + if (state.cache[state.tab]) delete state.cache[state.tab][cacheKey()]; // force re-query load(); }); @@ -1292,6 +1377,22 @@ function wireControls() { document.addEventListener("click", (e) => { const btn = e.target.closest(".kql-btn[data-panel-id]"); if (btn) openKqlDialog(btn.dataset.panelId); + // hbar click-to-filter + const hbarRow = e.target.closest(".hbar-filterable[data-filter-dim]"); + if (hbarRow) { + const dim = hbarRow.dataset.filterDim; + const val = hbarRow.dataset.filterVal; + if (dim && val) toggleFilter(dim, val); + } + // chip remove + const chipRemove = e.target.closest(".chip-remove[data-dim]"); + if (chipRemove) { + const dim = chipRemove.dataset.dim; + const val = chipRemove.dataset.val; + if (dim && val) toggleFilter(dim, val); + } + // reset all + if (e.target.closest("#filter-reset")) clearFilters(); }); let t; diff --git a/.github/extensions/ftk-local-dashboard/public/index.html b/.github/extensions/ftk-local-dashboard/public/index.html index b16074a3f..dcdfed198 100644 --- a/.github/extensions/ftk-local-dashboard/public/index.html +++ b/.github/extensions/ftk-local-dashboard/public/index.html @@ -32,6 +32,12 @@

FinOps hub · local

+ +
Loading cost data…
From c215872a17c0b810a2b76ca82bd92dc36792a902 Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 25 Jun 2026 12:31:23 -0700 Subject: [PATCH 47/51] chore: add PRODUCT.md and ignore impeccable cache Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .gitignore | 3 +++ PRODUCT.md | 42 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+) create mode 100644 PRODUCT.md diff --git a/.gitignore b/.gitignore index 198952770..42ce37628 100644 --- a/.gitignore +++ b/.gitignore @@ -387,3 +387,6 @@ src/templates/finops-hub-copilot-studio/knowledge/query-catalog.md todo/ done/ release/scloud-occurrence-report.md + +# impeccable design skill cache +.impeccable/ diff --git a/PRODUCT.md b/PRODUCT.md new file mode 100644 index 000000000..07fbf365d --- /dev/null +++ b/PRODUCT.md @@ -0,0 +1,42 @@ +# FinOps hub local — product context + +## What this is + +A live cost-analytics dashboard embedded in GitHub Copilot CLI as a canvas extension. +It connects to a locally-running Kusto emulator (ftklocal) loaded with the FinOps hub +schema and renders interactive KPI cards, charts, and triage signals — all grounded in +the FinOps Framework and the finops-toolkit query catalog. + +## Register + +product — design serves the product, the dashboard is the tool, not the brand. + +## Users + +FinOps practitioners and cloud engineers who run `Initialize-FinOpsHubLocal` and want +to explore their Azure cost data locally without provisioning cloud infrastructure. +They are technical (comfortable with KQL, PowerShell, Azure), time-conscious, and +treat the dashboard as a professional diagnostic surface rather than a consumer app. + +## Goals + +- Show the six FinOps Framework capability areas (overview, allocation, rate, usage, + anomaly, tokenomics) with actionable KPIs and charts. +- Let engineers drill down by clicking chart elements to slice the full dataset by + service, category, region, resource group, or subscription. +- Surface triage signals (anomalies, overspend, savings gaps) at a glance. +- Stay snappy: all data is local, no network calls beyond the loopback emulator. + +## Non-goals + +- Not a production monitoring tool (no alerting, no SLAs). +- Not a multi-user shared dashboard (single-engineer local only). +- Not a replacement for Cost Management in the Azure portal. + +## Design constraints + +- Zero external dependencies: vanilla JS/CSS, no build step, no npm packages. +- Served by a Node.js HTTP server inside the Copilot extension process. +- Embeds in the Copilot CLI canvas panel (variable width, ~800–1400 px typical). +- Must respect GitHub's CSS custom properties for light/dark theming. +- All interactivity must be keyboard-accessible and meet WCAG 2.1 AA contrast. From b1b66540128dbd8b87061d54b42d35d17a3a24e8 Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 25 Jun 2026 13:42:08 -0700 Subject: [PATCH 48/51] =?UTF-8?q?feat(dashboard):=20impeccable=20polish=20?= =?UTF-8?q?=E2=80=94=20P1-P3=20design=20fixes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit P1: remove KPI hero-metric top-stripe (box-shadow inset); replace with threshold value-text coloring (threshold-green/amber/red → .value) P1: remove false hover affordance from non-filterable hbar rows (.hbar:hover removed from opacity rule) P2: fix triage-title uppercase/tracking eyebrow anti-pattern (text-transform:uppercase + letter-spacing removed) P2: replace loading text with shimmer skeleton grid P3: restructure error state with copyable Initialize-FinOpsHubLocal command block and collapsible error detail P3: add filter-bar show/hide CSS transition P3: normalize all non-standard font-weights (680→700, 650/640/620→600, 580→500, var fallback 650→600) P3: remove dotted underline from .rail-time (misleading affordance) P3: add tab overflow-x:auto + white-space:nowrap for narrow canvas widths Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../ftk-local-dashboard/public/app.css | 121 +++++++++++++----- .../ftk-local-dashboard/public/app.js | 24 +++- 2 files changed, 107 insertions(+), 38 deletions(-) diff --git a/.github/extensions/ftk-local-dashboard/public/app.css b/.github/extensions/ftk-local-dashboard/public/app.css index ed9fd858b..0cc4d699f 100644 --- a/.github/extensions/ftk-local-dashboard/public/app.css +++ b/.github/extensions/ftk-local-dashboard/public/app.css @@ -43,7 +43,7 @@ body { .title-block h1 { margin: 0; font-size: var(--text-title-large, 24px); - font-weight: var(--font-weight-semibold, 650); + font-weight: var(--font-weight-semibold, 600); letter-spacing: -0.01em; } .title-block .sub { @@ -103,16 +103,22 @@ body { gap: 4px; margin: 2px 0 18px; border-bottom: 1px solid var(--grid); + overflow-x: auto; + -webkit-overflow-scrolling: touch; + scrollbar-width: none; } +.tabs::-webkit-scrollbar { display: none; } .tabs button { appearance: none; border: 0; background: transparent; color: var(--muted); font-size: 13.5px; - font-weight: 620; + font-weight: 600; padding: 12px 14px 14px; min-height: 44px; + white-space: nowrap; + flex-shrink: 0; cursor: pointer; position: relative; border-radius: 8px 8px 0 0; @@ -139,7 +145,6 @@ body { background: var(--card-bg); border: 1px solid var(--grid); border-radius: var(--radius); - box-shadow: inset 0 3px 0 var(--kpi-accent, var(--accent)); padding: 15px 16px 14px; position: relative; overflow: hidden; @@ -151,30 +156,27 @@ body { } .kpi .value { font-size: 26px; - font-weight: 680; + font-weight: 700; margin-top: 6px; letter-spacing: -0.02em; font-variant-numeric: tabular-nums; } .kpi .meta { margin-top: 5px; font-size: 12px; color: var(--muted); } -.kpi .meta .pos { color: var(--pos); font-weight: 650; } -.kpi .meta .neg { color: var(--neg); font-weight: 650; } -.kpi .meta .warn { color: var(--warn); font-weight: 650; } +.kpi .meta .pos { color: var(--pos); font-weight: 600; } +.kpi .meta .neg { color: var(--neg); font-weight: 600; } +.kpi .meta .warn { color: var(--warn); font-weight: 600; } /* ---------- KPI hierarchy (primary vs reference) ---------- */ -.kpi--primary { - box-shadow: inset 0 4px 0 var(--kpi-accent, var(--accent)); - font-weight: 600; -} -.kpi--reference { - opacity: 0.75; - box-shadow: inset 0 3px 0 var(--border-muted); -} +.kpi--primary .value { font-size: 28px; } +.kpi--reference { opacity: 0.75; } -/* ---------- KPI threshold badges ---------- */ +/* ---------- KPI threshold state: value text carries the signal ---------- */ .threshold-green { border-color: var(--pos); } .threshold-amber { border-color: var(--warn); } .threshold-red { border-color: var(--neg); } +.threshold-green .value { color: var(--pos); } +.threshold-amber .value { color: var(--warn); } +.threshold-red .value { color: var(--neg); } /* ---------- triage strip ---------- */ .triage-strip { @@ -200,11 +202,9 @@ body { .triage-tile:hover { opacity: 0.82; } .triage-tile:active { transform: translateY(1px); } .triage-title { - font-size: 11px; - font-weight: 650; + font-size: 11.5px; + font-weight: 600; color: var(--muted); - text-transform: uppercase; - letter-spacing: 0.05em; } .triage-count { font-size: 30px; @@ -216,10 +216,7 @@ body { .triage-badge { display: inline-block; font-size: 11px; - font-weight: 650; - padding: 2px 9px; - border-radius: 20px; - width: fit-content; + font-weight: 600; background: color-mix(in srgb, var(--muted) 12%, transparent); color: var(--muted); } @@ -254,7 +251,7 @@ body { .section-title h2 { margin: 0; font-size: 15px; - font-weight: 650; + font-weight: 600; letter-spacing: -0.01em; } .section-title .domain { @@ -290,7 +287,7 @@ body { .panel h3 { margin: 0 0 2px; font-size: 13.5px; - font-weight: 640; + font-weight: 600; } .panel .panel-sub { margin: 0 0 12px; @@ -309,13 +306,13 @@ svg { display: block; width: 100%; overflow: visible; } .legend .lv { color: var(--muted); font-variant-numeric: tabular-nums; } .donut-wrap { display: flex; align-items: center; gap: 18px; flex-wrap: wrap; } -.donut-center .big { font-size: 18px; font-weight: 680; letter-spacing: -0.02em; } +.donut-center .big { font-size: 18px; font-weight: 700; letter-spacing: -0.02em; } .donut-center .small { font-size: 11px; fill: var(--muted); } .hbar-row { font-variant-numeric: tabular-nums; } .hbar-row .name { fill: var(--text-color-default, #1f2328); font-size: 12px; } .hbar-row .val { fill: var(--muted); font-size: 11.5px; } -.bar:hover, .arc:hover, .hbar:hover { opacity: 0.82; cursor: default; } +.bar:hover, .arc:hover { opacity: 0.82; cursor: default; } /* ---------- data table ---------- */ .dtable { width: 100%; border-collapse: collapse; font-size: 12.5px; font-variant-numeric: tabular-nums; } @@ -323,18 +320,16 @@ svg { display: block; width: 100%; overflow: visible; } .dtable th:first-child, .dtable td:first-child { text-align: left; } .dtable thead th { color: var(--muted); font-weight: 600; font-size: 11px; } .dtable tbody tr:hover { background: color-mix(in srgb, var(--accent) 6%, transparent); } -.dtable .model { display: inline-flex; align-items: center; gap: 7px; font-weight: 580; } +.dtable .model { display: inline-flex; align-items: center; gap: 7px; font-weight: 500; } .dtable .swatch { width: 9px; height: 9px; border-radius: 3px; flex: none; } .dtable .barcell { position: relative; } .dtable .minibar { display: inline-block; height: 7px; border-radius: 3px; vertical-align: middle; margin-left: 6px; } /* ---------- states ---------- */ -.loading, .error { +.error { padding: 40px; text-align: center; color: var(--muted); -} -.error { border: 1px solid var(--grid); border-radius: var(--radius); background: var(--card-bg); @@ -358,6 +353,26 @@ svg { display: block; width: 100%; overflow: visible; } border-radius: 8px; margin-top: 14px; } +.error-action { + margin: 14px 0; + text-align: left; +} +.error-cmd { + display: flex; + align-items: center; + gap: 8px; + background: color-mix(in srgb, var(--muted) 10%, transparent); + border: 1px solid var(--grid); + border-radius: 8px; + padding: 8px 12px; + margin: 6px 0; +} +.error-cmd code { + flex: 1; + background: transparent; + padding: 0; +} +.error-detail { margin-top: 14px; } /* ---------- diagnostic rail ---------- */ .diagnostic-rail { @@ -377,7 +392,7 @@ svg { display: block; width: 100%; overflow: visible; } .rail-health--ok { color: var(--pos); } .rail-health--warn { color: var(--warn); } .rail-health--error { color: var(--neg); } -.rail-time { cursor: default; text-decoration: underline; text-decoration-style: dotted; } +.rail-time { cursor: default; } /* ---------- footer ---------- */ .app-footer { @@ -555,7 +570,7 @@ svg { display: block; width: 100%; overflow: visible; } color: var(--text-color-default, #1f2328); line-height: 1.5; } -.filter-chip strong { font-weight: 650; } +.filter-chip strong { font-weight: 600; } .chip-label { display: flex; gap: 4px; align-items: baseline; } .chip-remove { all: unset; @@ -602,3 +617,41 @@ svg .hbar-selected text.name { font-weight: 700; fill: var(--accent); } @media (prefers-reduced-motion: no-preference) { svg .hbar-dimmed { transition: opacity 0.12s ease-out; } } + +/* ---------- filter bar show/hide transition ---------- */ +.filter-bar { + transition: opacity 0.15s ease-out, transform 0.15s ease-out; +} +.filter-bar[hidden] { + display: none !important; +} + +/* ---------- loading skeleton ---------- */ +@keyframes shimmer { + 0% { background-position: -600px 0; } + 100% { background-position: 600px 0; } +} +.skeleton-card { + background: linear-gradient( + 90deg, + var(--grid) 25%, + color-mix(in srgb, var(--grid) 40%, transparent) 50%, + var(--grid) 75% + ); + background-size: 1200px 100%; + animation: shimmer 1.4s linear infinite; + border-radius: var(--radius); + border: 1px solid var(--grid); +} +.skeleton-kpi-grid { + display: grid; + grid-template-columns: repeat(auto-fit, minmax(180px, 1fr)); + gap: var(--gap); + margin-bottom: var(--gap); +} +.skeleton-kpi-grid .skeleton-card { height: 88px; } +.skeleton-panel-lg { height: 220px; margin-bottom: var(--gap); } +.skeleton-panel-sm { height: 160px; margin-bottom: var(--gap); } +@media (prefers-reduced-motion: reduce) { + .skeleton-card { animation: none; } +} diff --git a/.github/extensions/ftk-local-dashboard/public/app.js b/.github/extensions/ftk-local-dashboard/public/app.js index 49795c6d3..6aa80cf88 100644 --- a/.github/extensions/ftk-local-dashboard/public/app.js +++ b/.github/extensions/ftk-local-dashboard/public/app.js @@ -632,7 +632,7 @@ function kpiCard(label, value, meta, accent, thresholdClass) { const classArray = [thresholdClass, hierarchyClass].filter(Boolean); const cls = classArray.length > 0 ? ` ${classArray.join(" ")}` : ""; - return `
+ return `
${esc(label)}
${value}
${meta}
@@ -1253,8 +1253,18 @@ function renderError(p) { el("content").innerHTML = `

Can’t reach the FinOps hub

The dashboard queried ${esc(p.clusterUri || "")} (database ${esc(p.database || "Hub")}) but the request failed.

-

Start the Kusto emulator and run Initialize-FinOpsHubLocal, then refresh.

- 
${esc(p.error)}
+
+

Start the Kusto emulator, then run:

+
+ Initialize-FinOpsHubLocal + +
+

Then refresh this dashboard.

+
+
+ Show error detail + 
${esc(p.error)}
+
`; } @@ -1290,7 +1300,13 @@ async function load() { state.cache[tab] = state.cache[tab] || {}; state.loading = true; setRefreshSpinning(true); - el("content").innerHTML = `
Loading ${tab} data…
`; + el("content").innerHTML = ` +
+ ${'
'.repeat(6)} +
+
+
+ `; try { const res = await fetch(`${ENDPOINT[tab]}&preset=${encodeURIComponent(state.preset)}${filterParam()}`); state.cache[tab][key] = await res.json(); From 6f1366c88d66748f1d8a005772094aa5692244cd Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 25 Jun 2026 14:03:04 -0700 Subject: [PATCH 49/51] =?UTF-8?q?feat(dashboard):=20harden=20=E2=80=94=20a?= =?UTF-8?q?bort=20races,=20error=20boundary,=20a11y,=20null=20guards?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - AbortController: cancel in-flight fetch when tab/preset changes; discard AbortError silently - aria-busy on #content during load; aria-live="polite" on main element (screen reader announcements) - Render error boundary: catch any renderXxx() throw and show structured error instead of blank screen - fmtMonth/fmtDayRange: null/type guards to prevent crash on sparse Kusto data - Filterable hbar rows: tabindex="0" + role="button" + aria-label for keyboard access; Enter/Space delegation in wireControls - KQL copy button: success/failure feedback with 1.5s label swap - svgEl: optional aria-label parameter for chart accessibility Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../ftk-local-dashboard/public/app.js | 70 +++++++++++++++---- .../ftk-local-dashboard/public/index.html | 2 +- 2 files changed, 56 insertions(+), 16 deletions(-) diff --git a/.github/extensions/ftk-local-dashboard/public/app.js b/.github/extensions/ftk-local-dashboard/public/app.js index 6aa80cf88..ce84c05bb 100644 --- a/.github/extensions/ftk-local-dashboard/public/app.js +++ b/.github/extensions/ftk-local-dashboard/public/app.js @@ -61,6 +61,7 @@ const PANEL_KQL = { /* eslint-enable max-len */ let _kqlPanelId = null; +let _loadAbort = null; /* ------------------------------------------------------------------ filter management */ @@ -161,13 +162,17 @@ function fmtPerM(costPer1K) { } function fmtMonth(ym) { // "2025-04" -> "Apr ’25" + if (!ym || typeof ym !== "string") return String(ym ?? "—"); const [y, m] = ym.split("-"); + if (!y || !m) return ym; const names = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"]; - return `${names[+m - 1]} ’${y.slice(2)}`; + return `${names[+m - 1] || m} '${y.slice(2)}`; } function fmtDayRange(min, max) { + if (!min || !max) return "—"; const f = (s) => { const d = new Date(s); + if (isNaN(d)) return String(s); return d.toLocaleDateString("en-US", { month: "short", year: "numeric", timeZone: "UTC" }); }; return `${f(min)} – ${f(max)}`; @@ -190,8 +195,9 @@ function fmtRelativeTime(date) { if (abs < 86400) return rtf.format(Math.round(diffSec / 3600), "hour"); return rtf.format(Math.round(diffSec / 86400), "day"); } -function svgEl(w, h, body) { - return `${body}`; +function svgEl(w, h, body, label = "") { + const ariaAttr = label ? ` aria-label="${esc(label)}"` : ""; + return `${body}`; } /* ----------------------------------------------------------------- charts */ @@ -271,7 +277,8 @@ function hbar(rows, nameKey, valKey, opts = {}) { if (isSelected) cls += " hbar-selected"; if (isDimmed) cls += " hbar-dimmed"; const dimAttr = filterDim ? ` data-filter-dim="${esc(filterDim)}" data-filter-val="${esc(d.name)}"` : ""; - g += ``; + const interactiveAttrs = filterDim ? ` tabindex="0" role="button" aria-label="Filter by ${esc(d.name)}, ${fmtMoney(d.val)}"` : ""; + g += ``; g += `${esc(trunc(d.name, 20))}${esc(d.name)}`; g += `${esc(d.name)}\n${fmtMoneyFull(d.val)} · ${fmtPct(pct)}`; g += `${fmtMoney(d.val)}`; @@ -1286,21 +1293,34 @@ function currentPayload() { function render() { const p = currentPayload(); if (!p) return; - if (state.tab === "tokenomics") renderTokenomics(p); - else if (state.tab === "allocation") renderAllocation(p); - else if (state.tab === "rate") renderRate(p); - else if (state.tab === "usage") renderUsage(p); - else if (state.tab === "anomaly") renderAnomaly(p); - else renderOverview(p); + try { + if (state.tab === "tokenomics") renderTokenomics(p); + else if (state.tab === "allocation") renderAllocation(p); + else if (state.tab === "rate") renderRate(p); + else if (state.tab === "usage") renderUsage(p); + else if (state.tab === "anomaly") renderAnomaly(p); + else renderOverview(p); + } catch (err) { + console.error("[ftk-dashboard] render error:", err); + renderError({ error: `Render error in ${state.tab}: ${err.message}` }); + } } async function load() { const tab = state.tab, key = cacheKey(); if (state.cache[tab]?.[key]) { updateChrome(); render(); return; } + + // Cancel any in-flight request for a superseded tab/preset + if (_loadAbort) _loadAbort.abort(); + _loadAbort = new AbortController(); + const { signal } = _loadAbort; + state.cache[tab] = state.cache[tab] || {}; state.loading = true; setRefreshSpinning(true); - el("content").innerHTML = ` + const contentEl = el("content"); + contentEl.setAttribute("aria-busy", "true"); + contentEl.innerHTML = `
${'
'.repeat(6)}
@@ -1308,14 +1328,17 @@ async function load() {
`; try { - const res = await fetch(`${ENDPOINT[tab]}&preset=${encodeURIComponent(state.preset)}${filterParam()}`); + const res = await fetch(`${ENDPOINT[tab]}&preset=${encodeURIComponent(state.preset)}${filterParam()}`, { signal }); state.cache[tab][key] = await res.json(); } catch (err) { + if (err.name === "AbortError") return; // superseded by a newer load(); discard silently console.error("[ftk-dashboard] fetch failed:", err); state.cache[tab][key] = { error: "Could not load data. Check that the Kusto emulator is running." }; + } finally { + state.loading = false; + setRefreshSpinning(false); + el("content")?.setAttribute("aria-busy", "false"); } - state.loading = false; - setRefreshSpinning(false); updateChrome(); render(); } @@ -1386,7 +1409,12 @@ function wireControls() { // KQL dialog controls el("kql-close").addEventListener("click", () => el("kql-dialog").close()); - el("kql-copy").addEventListener("click", () => navigator.clipboard.writeText(el("kql-text").value)); + el("kql-copy").addEventListener("click", () => { + const btn = el("kql-copy"); + navigator.clipboard.writeText(el("kql-text").value) + .then(() => { btn.textContent = "Copied!"; setTimeout(() => { btn.textContent = "Copy"; }, 1500); }) + .catch(() => { btn.textContent = "Failed"; setTimeout(() => { btn.textContent = "Copy"; }, 1500); }); + }); el("kql-run").addEventListener("click", executeKql); // KQL escape-hatch buttons (event delegation — buttons injected by panelHtml) @@ -1411,6 +1439,18 @@ function wireControls() { if (e.target.closest("#filter-reset")) clearFilters(); }); + // Keyboard activation for filterable hbar rows (Enter / Space) + document.addEventListener("keydown", (e) => { + if (e.key !== "Enter" && e.key !== " ") return; + const hbarRow = e.target.closest(".hbar-filterable[data-filter-dim]"); + if (hbarRow) { + e.preventDefault(); + const dim = hbarRow.dataset.filterDim; + const val = hbarRow.dataset.filterVal; + if (dim && val) toggleFilter(dim, val); + } + }); + let t; window.addEventListener("resize", () => { clearTimeout(t); t = setTimeout(render, 180); }); } diff --git a/.github/extensions/ftk-local-dashboard/public/index.html b/.github/extensions/ftk-local-dashboard/public/index.html index dcdfed198..3e1a68977 100644 --- a/.github/extensions/ftk-local-dashboard/public/index.html +++ b/.github/extensions/ftk-local-dashboard/public/index.html @@ -38,7 +38,7 @@

FinOps hub · local

-
+
Loading cost data…
From ef634f4d9748db91109494497c2b99743ed6d690 Mon Sep 17 00:00:00 2001 From: msbrett Date: Thu, 25 Jun 2026 14:15:18 -0700 Subject: [PATCH 50/51] =?UTF-8?q?feat(dashboard):=20P1+P2=20UX=20polish=20?= =?UTF-8?q?=E2=80=94=20KPI=20tooltips,=20triage=20callouts,=20aria-pressed?= =?UTF-8?q?,=20KQL=20button?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - P1: add KPI_TIPS map with threshold rationale for 7 KPIs; render as native title-tooltip '?' button (kpi-tip) on each labelled card — no JS required - P2: add aria-pressed to filterable hbar rows so AT announces toggle state - P2: replace KQL button icon-only '' with text label 'KQL'; bump default opacity to 0.85 (was 0.7) so the button is always discoverable - P2: triage arrival context — anomaly tab shows amber callout banner when anomalous days detected; rate tab shows callout when underutilized commitments exist; both echo the triage-tile finding to anchor the user Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../ftk-local-dashboard/public/app.css | 42 ++++++++++++++++++- .../ftk-local-dashboard/public/app.js | 36 +++++++++++++--- 2 files changed, 71 insertions(+), 7 deletions(-) diff --git a/.github/extensions/ftk-local-dashboard/public/app.css b/.github/extensions/ftk-local-dashboard/public/app.css index 0cc4d699f..ad772efa0 100644 --- a/.github/extensions/ftk-local-dashboard/public/app.css +++ b/.github/extensions/ftk-local-dashboard/public/app.css @@ -170,6 +170,30 @@ body { .kpi--primary .value { font-size: 28px; } .kpi--reference { opacity: 0.75; } +/* ---------- KPI threshold tooltip ---------- */ +.kpi-tip { + display: inline-flex; + align-items: center; + justify-content: center; + width: 14px; + height: 14px; + border-radius: 50%; + border: 1px solid currentColor; + background: transparent; + color: var(--muted); + cursor: pointer; + font: 600 9px/1 var(--font-sans, -apple-system, sans-serif); + padding: 0; + margin-left: 4px; + vertical-align: middle; + transition: color 0.15s, border-color 0.15s; +} +.kpi-tip:hover, .kpi-tip:focus-visible { + color: var(--accent); + outline: 2px solid var(--accent); + outline-offset: 1px; +} + /* ---------- KPI threshold state: value text carries the signal ---------- */ .threshold-green { border-color: var(--pos); } .threshold-amber { border-color: var(--warn); } @@ -178,6 +202,22 @@ body { .threshold-amber .value { color: var(--warn); } .threshold-red .value { color: var(--neg); } +/* ---------- triage arrival context banner ---------- */ +.triage-callout { + display: flex; + align-items: center; + gap: 10px; + background: color-mix(in srgb, var(--warn) 10%, transparent); + border: 1px solid color-mix(in srgb, var(--warn) 28%, transparent); + border-radius: 8px; + padding: 10px 14px; + font-size: 13px; + font-weight: 500; + color: color-mix(in srgb, var(--warn) 75%, #000); + margin-bottom: 16px; +} +.triage-callout-icon { font-size: 15px; flex-shrink: 0; line-height: 1; } + /* ---------- triage strip ---------- */ .triage-strip { display: grid; @@ -441,7 +481,7 @@ svg { display: block; width: 100%; overflow: visible; } cursor: pointer; font: 600 10px/1 ui-monospace, monospace; padding: 4px 7px; - opacity: 0.7; + opacity: 0.85; transition: opacity 0.15s, background 0.15s, color 0.15s; } .kql-btn:hover { diff --git a/.github/extensions/ftk-local-dashboard/public/app.js b/.github/extensions/ftk-local-dashboard/public/app.js index ce84c05bb..c133dd6e2 100644 --- a/.github/extensions/ftk-local-dashboard/public/app.js +++ b/.github/extensions/ftk-local-dashboard/public/app.js @@ -277,7 +277,7 @@ function hbar(rows, nameKey, valKey, opts = {}) { if (isSelected) cls += " hbar-selected"; if (isDimmed) cls += " hbar-dimmed"; const dimAttr = filterDim ? ` data-filter-dim="${esc(filterDim)}" data-filter-val="${esc(d.name)}"` : ""; - const interactiveAttrs = filterDim ? ` tabindex="0" role="button" aria-label="Filter by ${esc(d.name)}, ${fmtMoney(d.val)}"` : ""; + const interactiveAttrs = filterDim ? ` tabindex="0" role="button" aria-pressed="${isSelected ? 'true' : 'false'}" aria-label="Filter by ${esc(d.name)}, ${fmtMoney(d.val)}"` : ""; g += ``; g += `${esc(trunc(d.name, 20))}${esc(d.name)}`; g += `${esc(d.name)}\n${fmtMoneyFull(d.val)} · ${fmtPct(pct)}`; @@ -623,24 +623,37 @@ function isPartialMonth() { return now.getDate() < new Date(now.getFullYear(), now.getMonth() + 1, 0).getDate(); } +const KPI_TIPS = { + "Untagged cost": "% of spend on resources missing tags. Target: <10% · Review: <25% · Urgent: ≥25%. Tagging enables accurate showback and chargeback.", + "Commitment waste": "% of RI/savings-plan spend on unused capacity. Target: <5% · Review: <10% · Urgent: ≥10%. Idle commitments erode net savings.", + "Effective savings rate": "Negotiated + commitment savings as % of list price. Higher = better. Enterprise customers typically target ≥15–20%.", + "Commitment coverage": "Compute spend covered by RIs or savings plans. Target: ≥60% for steady workloads. Higher coverage → lower effective rate.", + "Compute coverage": "On-demand core-hours offset by commitments. Target: ≥60%. Tracks whether savings plan scope is sufficient.", + "MACC burn rate": "Microsoft Azure Consumption Commitment utilization. Target: ≥90% to avoid forfeiting unused balance at term end.", + "Anomaly days": "Days where daily cost deviated significantly from the expected baseline (STL decomposition). Review flagged dates for unexpected spend.", +}; + function kpiCard(label, value, meta, accent, thresholdClass) { // Determine hierarchy class based on label const primaryLabels = ["Untagged cost", "Commitment waste", "Effective savings rate"]; const referenceLabels = ["Commitment coverage", "Compute coverage", "Total tokens", "Anomaly days"]; - + let hierarchyClass = ""; if (primaryLabels.includes(label)) { hierarchyClass = "kpi--primary"; } else if (referenceLabels.includes(label)) { hierarchyClass = "kpi--reference"; } - + // Combine threshold and hierarchy classes const classArray = [thresholdClass, hierarchyClass].filter(Boolean); const cls = classArray.length > 0 ? ` ${classArray.join(" ")}` : ""; - + + const tip = KPI_TIPS[label]; + const tipHtml = tip ? ` ` : ""; + return `
-
${esc(label)}
+
${esc(label)}${tipHtml}
${value}
${meta}
`; @@ -651,7 +664,7 @@ function kpiCard(label, value, meta, accent, thresholdClass) { function panelHtml(id, span, title, sub, body) { const subHtml = sub ? `

${sub}

` : ""; return `
-

${title}

${subHtml}
+

${title}

${subHtml}
${body}
`; } @@ -976,7 +989,12 @@ function renderAnomaly(p) { `median ingestion lag (P50)`, PALETTE[2]), ].join(""); + const triageCallout = anomDays.length > 0 + ? `
${fmtInt(anomDays.length)} anomal${anomDays.length === 1 ? "y day" : "y days"} detected — ${fmtMoney(anomCost)} in flagged spend. Review the chart below.
` + : ""; + content.innerHTML = ` + ${triageCallout}
${kpis}

Cost anomalies

Anomaly Management capability
@@ -1122,7 +1140,13 @@ function renderRate(p) { const coreColors = { "On Demand": PALETTE[0], "Reservation": PALETTE[1], "Savings Plan": PALETTE[4] }; const coreSlices = (d.coreHours || []).map((r) => ({ label: r.t, value: r.CoreHours || 0, color: coreColors[r.t] || PALETTE[6] })); + const underutilCount = (d.byCommitment || []).filter((r) => (r.Unused || 0) > 0).length; + const rateCallout = underutilCount > 0 + ? `
${fmtInt(underutilCount)} underutilized commitment${underutilCount === 1 ? "" : "s"} found — ${fmtMoney(cm.Unused)} in unused spend. See the commitments panel below.
` + : ""; + content.innerHTML = ` + ${rateCallout}
${kpis}

Rate optimization

Rate Optimization capability
From bfd2d815cf6f2adcf2c32cbdab6ca0541e8d82f6 Mon Sep 17 00:00:00 2001 From: msbrett Date: Mon, 29 Jun 2026 11:43:44 -0700 Subject: [PATCH 51/51] fix(docs): mount kustodata volume in docker run so databases survive container restarts Without -v /kustodata, databases are stored in the container's writable layer. docker rm (or Docker Desktop recreation) wipes everything, forcing the user to redo the full setup. Add $dataPath + the volume flag so persistence is durable by default. Also document the restart path (docker start). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs-mslearn/toolkit/hubs/run-hubs-locally.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/docs-mslearn/toolkit/hubs/run-hubs-locally.md b/docs-mslearn/toolkit/hubs/run-hubs-locally.md index 7e7eda6cc..4632e8640 100644 --- a/docs-mslearn/toolkit/hubs/run-hubs-locally.md +++ b/docs-mslearn/toolkit/hubs/run-hubs-locally.md @@ -3,7 +3,7 @@ title: Run FinOps hubs locally description: Stand up a FinOps hub on your own hardware in a local container and ingest cost data, using the same KQL and open data as a deployed hub. author: MSBrett ms.author: brettwil -ms.date: 06/21/2026 +ms.date: 06/29/2026 ms.topic: how-to ms.service: finops ms.subservice: finops-toolkit @@ -34,18 +34,23 @@ For background on the emulator and its platform requirements, see the [Kusto emu ## Start the emulator -Create a working folder outside the repository, then start the emulator. The export folder is mounted read-only into the container at `/data/export`; you'll download cost data into it later. For other ways to run the container, see [Install the Kusto emulator](/azure/data-explorer/kusto-emulator-install). +Create a working folder outside the repository, then start the emulator. Two folders are mounted into the container: exports (read-only at `/data/export`) and a data folder where the engine persists its databases. Mounting the data folder means your databases survive container restarts and removals. For other ways to run the container, see [Install the Kusto emulator](/azure/data-explorer/kusto-emulator-install). ```powershell $exportPath = '../exports/local-hub' +$dataPath = '../exports/local-hub-data' New-Item -ItemType Directory -Force -Path $exportPath | Out-Null +New-Item -ItemType Directory -Force -Path $dataPath | Out-Null docker run -d --name finops-hub-local --platform linux/amd64 ` -p 8082:8080 -m 16g -e ACCEPT_EULA=Y ` -v "$((Resolve-Path $exportPath).Path):/data/export:ro" ` + -v "$((Resolve-Path $dataPath).Path):/kustodata" ` mcr.microsoft.com/azuredataexplorer/kustainer-linux:latest ``` +To restart after stopping the container, run `docker start finops-hub-local`. Your databases reload automatically from the data folder — no need to repeat the setup steps. + All later steps talk to the emulator's HTTP endpoint, so define a small helper to send a command. Management commands (those starting with `.`) go to `/v1/rest/mgmt`; queries go to `/v1/rest/query`. The endpoint has no authentication. ```powershell