cost-analysis-export: Support multiple CSV schemas#5579
Merged
Conversation
Add dynamic query builder to handle different column names across: - Legacy EA/PAYG (InstanceId, UsageDateTime, UsageQuantity, PreTaxCost) - Modern EA 2024 (ResourceId, Date, Quantity, CostInBillingCurrency) - MCA/MCA Partner/CSP (camelCase: resourceId, date, quantity, costInBillingCurrency) - FOCUS (ResourceId, ChargePeriodStart, ConsumedQuantity, BilledCost) Key changes: - Detect ResourceID and Date columns dynamically for JOIN - Multiply all cost/quantity columns by split fraction - Preserve original column names in output - Add test cases for Modern EA, MCA, and FOCUS schemas
Automatically append trailing slash to AZURE_STORAGE_AKS_DATA_PREFIX and AZURE_STORAGE_COST_EXPORT_PREFIX if missing. Also add logging when no AKS export files are found to aid debugging.
r2k1
changed the title
r2k1
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
This PR enhances the cost analysis export tool to support multiple Azure cost export CSV schemas, making it compatible with Legacy EA/PAYG, MCA (Microsoft Customer Agreement), and FOCUS (FinOps Open Cost and Usage Specification) formats. The implementation dynamically detects the schema by inspecting column names rather than requiring explicit configuration.
Changes:
- Added dynamic schema detection that identifies resource ID and date columns from a list of known variants across different Azure billing schemas
- Fixed a bug where missing trailing slashes in storage prefix paths caused file discovery failures
- Added logging to aid debugging when no AKS export files are found
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| examples/cost-analysis-export/main.go | Implements dynamic column detection, adds prefix path normalization in config validation, adds file processing logging, and builds SQL queries dynamically based on detected schema |
| examples/cost-analysis-export/main_test.go | Updates test assertions to check for column presence rather than exact order/content, adds comprehensive test cases for Modern EA, MCA, and FOCUS schemas |
MCA cost exports use MM/DD/YYYY format while AKS exports use YYYY-MM-DD, causing INNER JOIN to return zero rows. Added date normalization during CSV import to convert all dates to YYYY-MM-DD format.
Helps debug issues when join returns empty results.
charleswool
approved these changes
Mar 11, 2026
Fei-Guo
approved these changes
Mar 11, 2026
wangyira
approved these changes
Mar 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary