Migrate instance size flexibility ratios from deprecated CSV to Azure Catalogs API

[RolandKrummenacher]

Summary

Microsoft is deprecating the static instance size flexibility (ISF) ratio CSV file hosted at ccmstorageprod.blob.core.windows.net. The toolkit must migrate to the new Azure Catalogs API before the file is removed.

Timeline

• 9 May 2026 – CSV file will no longer be updated
• 30 August 2026 – CSV file will be removed entirely

Current usage

Power BI (direct dependency)

The InstanceSizeFlexibility table in both storage and KQL datasets loads the CSV directly:

• src/power-bi/storage/Shared.Dataset/definition/tables/InstanceSizeFlexibility.tmdl
• src/power-bi/kql/Shared.Dataset/definition/tables/InstanceSizeFlexibility.tmdl

Source: AzureStorage.BlobContents("https://ccmstorageprod.blob.core.windows.net/costmanagementconnector-data/AutofitComboMeterData.csv")

KQL ingestion (indirect dependency)

The KQL ingestion scripts (IngestionSetup_v1_0.kql, v1_2.kql, RawTables.kql) currently consume InstanceFlexibilityGroup and InstanceFlexibilityRatio from Cost Management export data (reservation details and reservation recommendations). This works for existing reservations, but does not cover the full ISF ratio catalog needed for recommending new reservations in the future.

Optimization Engine

Workbooks (reservations-usage, reservations-potential, benefits-simulation) reference flexibility data and may need updates.

Proposed solution

1. Create a new open data dataset for instance size flexibility ratios, sourced from the Azure Catalogs API
2. Update Power BI InstanceSizeFlexibility table to consume from the new open data dataset instead of the deprecated CSV
3. Integrate into FinOps hubs so the full ISF ratio catalog is available in ADX/Fabric for future reservation recommendation scenarios (not just existing reservations from CM exports)
4. Update Optimization Engine workbooks if they reference the CSV directly

References

• Azure notification: "Access to instance size flexibility ratios is changing" (received April 2026)
• Azure Catalogs API documentation: referenced in the notification
• Current CSV: https://ccmstorageprod.blob.core.windows.net/costmanagementconnector-data/AutofitComboMeterData.csv

🤖 Generated with Claude Code

[microsoft-github-policy-service]

Thank you for submitting this issue! We try to respond within 3 business days. In the meantime:

• 📖 Check the troubleshooting guide for common issues
• 💬 Ask questions in Discussions
• 📅 Join our biweekly office hours for live help from the team

[RolandKrummenacher]

https://learn.microsoft.com/en-us/azure/cost-management-billing/reservations/instance-size-flexibility#extract-instance-size-flexibility-ratios-using-azure-catalogs-api

[helderpinto]

This one will be hard to replace in AOE workbooks. Being the Catalog APIs authenticated, it'll be complex (if not impossible) to implement the OAuth flow with the logged in user context before calling the Catalog APIs. It will probably be better to ingest ISF data into Log Analytics as an additional export runbook.

[RolandKrummenacher]

Additional dependency: a second deprecated ISF blob (isfratioblob.csv / aka.ms/isf)

While auditing our own KQL dashboards against this deprecation, I found that the same retiring storage account hosts two distinct ISF CSVs, and this issue currently only tracks one of them:

Blob	Container	Consumed by
AutofitComboMeterData.csv	costmanagementconnector-data	Power BI InstanceSizeFlexibility table (already covered above)
isfratioblob.csv	instancesizeflexibility-data	Optimization Engine workbooks (not yet enumerated)

The short link https://aka.ms/isf is a 301 redirect to the second blob:

https://aka.ms/isf
  -> https://ccmstorageprod.blob.core.windows.net/instancesizeflexibility-data/isfratioblob.csv

AOE workbooks that hardcode this path

All three reference instancesizeflexibility-data/isfratioblob.csv directly:

• src/optimization-engine/views/workbooks/benefits-simulation.json
• src/optimization-engine/views/workbooks/reservations-potential.json
• src/optimization-engine/views/workbooks/reservations-usage.json

Why it matters for this migration

• Both blobs sit on ccmstorageprod.blob.core.windows.net and are therefore both subject to the timeline in this issue (no updates after 9 May 2026, removed 30 Aug 2026).
• The two files differ in shape and freshness:
  • isfratioblob.csv — 3 columns (InstanceSizeFlexibilityGroup, ArmSkuName, Ratio), currently maintained (Last-Modified 2026-03-16), ~3.3k rows / 434 groups / 2,524 SKUs.
  • AutofitComboMeterData.csv — 9 columns, frozen since 2023-02-22, ~58k rows but only 132 groups / 721 SKUs.
• Note that aka.ms/isf is referred to as the "official public URL" for ISF data in [Power BI] Update instance size flexibility data #971 — so the proposed open-data replacement should ideally take over that short link to avoid breaking external consumers that already point at it.

⚠️ Heads-up: "we use the same file as the connector, so we're safe" does not hold

It's tempting to assume that because Power BI reads AutofitComboMeterData.csv — the same file the Cost Management connector uses — we're covered as long as the connector keeps working. That assumption is wrong on two counts:

1. The file is already frozen. AutofitComboMeterData.csv has a Last-Modified of 2023-02-22 — it has not been updated in ~3 years. So the Power BI ISF data is already stale (132 groups / 721 SKUs vs. 434 / 2,524 in the maintained isfratioblob.csv), independent of the upcoming removal. SKUs introduced since early 2023 are simply missing.
2. Connector parity ≠ safety. Both files live on the same ccmstorageprod account that is being decommissioned per the timeline above, and the connector itself is in maintenance mode with support ending. Tying our fate to the connector just means we inherit its retirement.

Net: the migration to a maintained open-data source is needed for both blobs regardless of connector status, and for AutofitComboMeterData.csv it also fixes a pre-existing staleness problem.

Suggested addition to the proposed solution

The "new open data dataset" (step 1) should replace both blob paths, and the Optimization Engine workbook updates (step 4) should specifically repoint the three workbooks above off aka.ms/isf / isfratioblob.csv.

[helderpinto]

@RolandKrummenacher, if we go the open data set path as suggested, does this mean we are going to generate at each release a new open data CSV file that mimics the currently existing ISF file? That would definitely make things much easier than implementing Azure Catalogs API data extraction logic in each tool.

[RolandKrummenacher]

@helderpinto yes — exactly that, and I've opened #2199 implementing it.

The generator (Update-InstanceSizeFlexibility.ps1) runs as a weekly workflow, extracts ISF ratios from the Catalogs API, and commits a public open data CSV (InstanceSizeFlexibility.csv). All the authentication complexity stays in that one generator — Power BI, hubs, and the Optimization Engine then consume the public CSV with no Azure credentials. That directly sidesteps the earlier concern about implementing an OAuth flow inside the AOE workbooks: they just point at the open data file like they would any other.

A few findings from building/validating it against the live Catalogs API:

• Only four reserved resource types actually expose ISF ratios (ReservationsAutofitGroup/ReservationsAutofitRatio): Virtual Machines, Block Blob, Redis Cache, and Dedicated Host. The other ~20 reserved types I probed are 1:1 and carry no flexibility groups.
• Unioning those four across a broad region set yields 2,437 SKUs / 558 groups — more groups than the deprecated isfratioblob.csv (434), and well above the frozen AutofitComboMeterData.csv Power BI uses today (132). Block Blob groups are region-specific, which is why the broad region sweep matters.
• Ratios are region-stable for the same group/SKU, so the union is safe.
• I kept the raw Microsoft ratios (not normalized) for drop-in parity with the deprecated files, since downstream math depends on that scale. Normalization is an opt-in switch.

One maintainer action: unlike the other open data workflows, the Catalogs API is authenticated (Microsoft.Capacity/catalogs/read), so the workflow needs an OIDC service principal + repo secrets, and the subscription it uses needs the Microsoft.Capacity resource provider registered. Details are in the workflow header.

#2199 is foundation only — the new dataset + generator + workflow. Repointing Power BI and the three AOE workbooks off the retiring blobs is a deliberate follow-up, to be done once this dataset ships in a release (so they don't reference a not-yet-published URL).