[Hubs] msexports_ExecuteETL drops ReservationDetails ingestion when manifest dataRowCount is null (manifestVersion 2024-04-01)

[haflidif]

🐛 Problem

msexports_ExecuteETL silently skips all Cost Management ReservationDetails ingestion when the manifest emits dataRowCount: null (the default for manifestVersion: 2024-04-01). The Set Has No Rows SetVariable activity sets hasNoRows = true, which causes the downstream Detect Channel switch to route to "ignore", skipping every Copy/Ingest activity. The pipeline reports Succeeded, so there's no operator signal that data was dropped.

Affects FinOps hubs v14.0 (and likely earlier v13.x — same expression is in main).

Current expression in src/templates/finops-hub/modules/Microsoft.CostManagement/Exports/app.bicep for the Set Has No Rows activity:

@or(
  equals(activity('Read Manifest').output.firstRow.blobCount, null),
  equals(activity('Read Manifest').output.firstRow.blobCount, 0),
  and(
    contains(activity('Read Manifest').output.firstRow, 'dataRowCount'),
    or(
      equals(activity('Read Manifest').output.firstRow.dataRowCount, null),   <-- treats null as zero
      equals(activity('Read Manifest').output.firstRow.dataRowCount, 0)
    )
  )
)

manifestVersion: 2024-04-01 emits dataRowCount: null to mean "row count not pre-computed", not "zero rows". blobCount and byteCount are always populated when data exists, so it's incorrect to treat null as zero rows.

👣 Repro steps

1. Create a Cost Management export of type ReservationDetails (dataVersion: 2023-03-01) on an EA billing account scope (Microsoft.Billing/billingAccounts/{enrollment}).
2. Let it run on its daily schedule, or trigger any "Export selected dates" run.
3. Inspect storage at msexports/billingAccounts/{enrollment}/{exportName}/{YYYYMMDD-YYYYMMDD}/{run}/{guid}/manifest.json — manifest shows manifestVersion: 2024-04-01, byteCount > 0, blobCount: 1, dataRowCount: null, with a real CSV alongside.
4. In ADF, the msexports_ExecuteETL run reports Succeeded. Drill into activity outputs:
   • Set Has No Rows → hasNoRows = true
   • Detect Channel → output "ignore"
   • Copy CSV → Parquet activities — skipped
5. Confirm CommitmentDiscountUsage_raw (and therefore CommitmentDiscountUsage_final_v1_2) in ADX stays empty despite multiple succeeded export runs.

Reproduced on FinOps hubs v14.0 with dataVersion: 2023-03-01, apiVersion: 2025-03-01, EA billing account scope.

🤔 Expected

The pipeline should ingest the data because real data exists (byteCount > 0, blobCount: 1, real CSV in storage). Either:

• Remove the dataRowCount check entirely and rely on blobCount (and optionally byteCount), or
• Stop treating dataRowCount: null as 0 — only the explicit numeric value 0 should mean "no rows".

Suggested replacement expression (simplest):

@or(
  equals(coalesce(activity('Read Manifest').output.firstRow.blobCount, 0), 0),
  equals(coalesce(activity('Read Manifest').output.firstRow.byteCount, 0), 0)
)

Alternative if you want to preserve the dataRowCount short-circuit for older manifest versions:

@or(
  equals(coalesce(activity('Read Manifest').output.firstRow.blobCount, 0), 0),
  and(
    contains(activity('Read Manifest').output.firstRow, 'dataRowCount'),
    equals(activity('Read Manifest').output.firstRow.dataRowCount, 0)   // explicit zero only
  )
)

🔧 Environment

• FinOps hub version: 14.0
• Billing account type: EA (Enterprise Agreement)
• Power BI report type: ADX (Azure Data Explorer)
• Cost Management export: ReservationDetails, dataVersion: 2023-03-01, apiVersion: 2025-03-01, manifestVersion: 2024-04-01

ℹ️ Additional context

Sample failing manifest:

{
  "manifestVersion": "2024-04-01",
  "byteCount": 1972417,
  "blobCount": 1,
  "dataRowCount": null,
  "exportConfig": {
    "exportName": "...-daily-reservationdetails",
    "dataVersion": "2023-03-01",
    "apiVersion": "2025-03-01",
    "type": "ReservationDetails",
    "timeFrame": "MonthToDate",
    "granularity": "Daily"
  },
  "blobs": [
    { "blobName": ".../<file>.csv", "byteCount": 1972417 }
  ]
}

Activity outputs from a failing pipeline run:

• Set Has No Rows → hasNoRows: true
• Detect Channel → "ignore"
• All downstream Copy/Ingest activities skipped (zero data flows through)

Workaround we applied: edited the Set Has No Rows expression in ADF Studio to use blobCount + byteCount only and dropped the dataRowCount check. After publishing, we re-triggered msexports_ExecuteETL per historical month folder via az datafactory pipeline create-run. ~11 months / 21 runs reprocessed successfully and CommitmentDiscountUsage_final_v1_2 populated as expected (~40k rows so far, monthly cadence of ~5,500–6,500 rows per month, ~65–99 distinct reservations). The manual edit will be overwritten by the next Deploy-FinOpsHub, so an upstream fix would be very welcome.

Why this only affected ReservationDetails in our hub: other EA-scope exports on the same factory (FocusCost, PriceSheet, ReservationRecommendations, ReservationTransactions) kept ingesting correctly. Either their manifests still emit a numeric dataRowCount, or they omit the field entirely (which makes contains(...) false and skips the third branch).

🙋‍♀️ Ask for the community

We could use your help:

1. Please vote this issue up (👍) to prioritize it.
2. If you see CommitmentDiscountUsage_raw / CommitmentDiscountUsage_final_v1_2 empty in your hub despite the export running, drop a comment with your hub version, scope type, and the dataRowCount value from one of your ReservationDetails manifests.

[RolandKrummenacher]

Hi @haflidif — thank you for the exceptionally detailed report (manifest sample, activity trace, and a validated workaround).

Your mechanism is correct: the Set Has No Rows check treats dataRowCount: null the same as 0, sets hasNoRows = true, and the run then skips ingestion while still reporting Succeeded.

One important framing on the supported configuration: FinOps hubs currently document support for Cost Management export API version 2023-07-01-preview (data ingestion docs), whose manifest carries a numeric dataRowCount. Your export uses apiVersion: 2025-03-01, which produces the newer manifestVersion: 2024-04-01 manifest where dataRowCount defaults to null — a manifest shape the hub doesn't yet handle. That's why only this export drops data while your other datasets keep ingesting.

Workaround that stays on the supported path: recreate the ReservationDetails export at API version 2023-07-01-preview (the toolkit's New-FinOpsCostExport uses that version by default). Its manifest emits a numeric dataRowCount, so ingestion works without editing the factory — and it survives the next Deploy-FinOpsHub (unlike the ADF Studio edit).

Could you confirm whether recreating the export at 2023-07-01-preview resolves it on your side? That tells us whether to treat this purely as forward-compat hardening for newer manifests.

#needs-info

[RolandKrummenacher]

🔍 Root-cause analysis (engineering)

Root cause (confirmed): Set Has No Rows (src/templates/finops-hub/modules/Microsoft.CostManagement/Exports/app.bicep:335, v14) treats dataRowCount: null as zero rows; hasNoRows = true empties the For Each Blob items list (:897) and skips Copy Manifest (:968) → silent drop, pipeline reports Succeeded. The Detect Channel → ignore route (:443) is non-causal (always ignore for ReservationDetails, which only selects the schema file).

Supported-usage gate: Docs (docs-mslearn/toolkit/hubs/data-processing.md:351-358, 513-520) document supported export API version 2023-07-01-preview, with a numeric dataRowCount manifest model (:332). The reporter's export is apiVersion: 2025-03-01 → manifestVersion: 2024-04-01 (null dataRowCount) — an undocumented/unsupported manifest shape. Hub-managed exports use 2023-07-01-preview (ManagedExports/app.bicep:40) → numeric dataRowCount, so the supported path is not broken. This reproduces only on a newer, unsupported export API version.

Forward-compat risk: The null → no-rows logic is a latent defect that becomes a supported-path bug once managed exports adopt a 2024-04-01-style manifest. The failure is silent (pipeline = Succeeded).

Scope/impact: Today: only user/unmanaged exports on API versions newer than 2023-07-01-preview that emit a null dataRowCount. Hub-managed exports unaffected.

Fix options:

1. Proactive hardening — drop only the equals(…dataRowCount, null) clause, keep the numeric-0 short-circuit. Low risk, future-proofs, preserves PR [Hubs] Don't ingest empty export files #2042's empty-file fix.
2. Officially support 2024-04-01+/newer API versions across all datasets — broader, needs testing per dataset.
3. Document-only — state the supported API version + the recreate-export workaround, defer the code change until the hub adopts newer manifests.

Recommendation: Not an urgent supported-path bug. Lean to option 1 (cheap, safe, proactive hardening) or option 3 (document the supported API version now). Reclassify from a "Status: Ready" bug to forward-compat hardening / needs-info pending the reporter's API-version confirmation.

[RolandKrummenacher]

Risk note: this conflicts with the fix from #2042 (#1535/#1738) — confirm manifest behavior before changing

The dataRowCount clause in the Set Has No Rows activity (Microsoft.CostManagement/Exports/app.bicep, the hasNoRows expression) was added deliberately in #2042 to fix #1535/#1738:

When Cost Management exports a file with column headers but zero data rows, the manifest reports blobCount = 1 (the file exists), but the pipeline tries to read the first data row for channel detection and convert the empty file to parquet — both fail.

So line ~335 now has to satisfy two opposing requirements:

• [Hubs] Don't ingest empty export files #2042: header-only blob exists (blobCount = 1) but is empty → must skip, or the pipeline crashes.
• [Hubs] msexports_ExecuteETL drops ReservationDetails ingestion when manifest dataRowCount is null (manifestVersion 2024-04-01) #2173: real data exists but manifestVersion 2024-04-01 reports dataRowCount: null → must not skip, or data is silently dropped.

byteCount can't disambiguate these — a header-only CSV still has byteCount > 0.

Current expression:

@or(
  equals(blobCount, null),
  equals(blobCount, 0),
  and(contains(firstRow,'dataRowCount'),
      or(equals(dataRowCount, null), equals(dataRowCount, 0))))

What to confirm first

Capture a manifest.json for a genuinely empty (header-only) reservationdetails export emitted as manifestVersion: 2024-04-01, and check the dataRowCount value.

Candidate A — if empty 2024-04-01 exports emit dataRowCount: 0

Drop only the null check; keep == 0. Safe and minimal:

@or(
  equals(blobCount, null),
  equals(blobCount, 0),
  and(contains(firstRow,'dataRowCount'),
      equals(dataRowCount, 0)))

• null (rows present, count not pre-computed) → not skipped ✅ (fixes [Hubs] msexports_ExecuteETL drops ReservationDetails ingestion when manifest dataRowCount is null (manifestVersion 2024-04-01) #2173)
• 0 (genuinely empty) → skipped ✅ (keeps [Hubs] Don't ingest empty export files #2042)

Candidate B — if empty 2024-04-01 exports also emit dataRowCount: null

Then the manifest is genuinely ambiguous (null means both "empty" and "has data"), and no expression change is correct. The fix would instead be to make the downstream tolerant of header-only files — channel detection (the Check for MCA Column Lookup) and the CSV→Parquet copy must not crash on a headers-only blob — after which hasNoRows can be made conservative (skip only on blobCount 0/null or a real dataRowCount == 0). This is a larger change than a one-line expression edit.

Testing caveat

This pipeline has no automated unit/integration coverage (#2042 was manually verified only), so a regression here would not be caught by CI. Any fix needs a deploy that exercises both an empty header-only export and a real-data 2024-04-01 export.