Costs_final_v1_2 loses extents when PropagateIngestionProperties=true

[Hardik29]

Description

Environment

• FinOps Toolkit version: 12.0

• ADX update policy:

  • Source: Costs_raw
  • Query: Costs_transform_v1_2()
  • IsTransactional: true
  • PropagateIngestionProperties: true

Problem

After ingesting multiple parquet files into Costs_raw, only some extents appear in Costs_final_v1_2.

Example:

Costs_raw

March export:

• part_0_0001 – 270 rows
• part_1_0001 – 329,262 rows
• part_2_0001 – 355,983 rows
• part_3_0001 – 356,014 rows
• part_4_0001 – 196,431 rows

Total rows:

1,237,960

Costs_final_v1_2

Only the last extent was present:

• part_4_0001 – 196,431 rows

The same behavior occurred for another export where only the last part survived.

No update policy failures or ingestion failures were reported.

Investigation performed

• Verified Costs_raw extents were healthy.
• Verified Costs_transform_v1_2() returns correct results when executed manually.
• Verified row counts and total cost values.
• Verified there were no update policy failures.
• Verified there were no ingestion failures.
• Tables were recreated from scratch.

Root cause

The issue appears to be caused by:

"PropagateIngestionProperties": true

with propagated drop-by: tags.

Workaround

Changing the update policy to:

{
  "IsEnabled": true,
  "Source": "Costs_raw",
  "Query": "Costs_transform_v1_2()",
  "IsTransactional": true,
  "PropagateIngestionProperties": false
}

resolved the issue.

After changing the setting:

• Costs_raw row count = 1,237,960
• Costs_final_v1_2 row count = 1,237,960

Total cost values also matched.

[RolandKrummenacher]

Thanks for the thorough writeup and workaround, @Hardik29 — and good instinct isolating it to PropagateIngestionProperties.

This is actually expected behavior for custom ingestion, and it's documented. When you place parquet files directly in the ingestion container, FinOps hubs use an ingestion ID to decide which extents belong to the current run versus a previous one. The ingestion ID is the part of the file name before __, and every file in a single run must share the same <ingestionId>__<originalFileName>.parquet name — otherwise each file is treated as a separate run and the per-folder cleanup drops the earlier ones, leaving only the last. Your files (part_0_0001, part_1_0001, …) have no shared <ingestionId>__ prefix, so each part is seen as a different run — exactly the "only the last part survives" symptom.

See the docs:

• Ingest from other data sources — the <ingestionId>__<originalFileName>.parquet naming rule and the "Each run must replace the full contents of its folder path" callout: "each file's pre-ingest cleanup drops extents tagged with a different ingestionId — only the files sharing the last ingestionId processed survive in Data Explorer. The managed Cost Management export pipeline handles this cleanup automatically, but custom ingestion workflows must implement it themselves."
• About the ingestion container — the ingestion ID "must be … consistent across all files for the ingested dataset."

The fix is to name all parts of one export with the same ingestion ID, e.g. 20260312T1200__part_0_0001.parquet, 20260312T1200__part_1_0001.parquet, … (a timestamp or GUID — any value, as long as it's identical across the run), and replace the full folder contents per run. Your PropagateIngestionProperties: false workaround also stops the symptom, but note it disables Costs_final_v1_2 retention (the cleanup reads the month from those same tags), so the naming convention is the recommended path.

One thing to confirm so we're sure this isn't a product bug: are you ingesting these parquet files yourself (custom workflow into the ingestion container), or are they coming from the managed Cost Management export pipeline? If it's the managed pipeline producing un-shared ingestion IDs, that would be a bug and we'll dig in.

#needs-info

[RolandKrummenacher]

🔍 Root-cause analysis (engineering)

Disposition: by-design / documented (pending confirmation the reporter is on custom ingestion, which their filenames indicate). Not a toolkit bug in the supported path.

Documented contract: Custom ingestion requires <ingestionId>__<originalFileName>.parquet with the ingestionId identical across all files of a run; the pipeline derives it by splitting on __, and extents from prior runs in the folder are dropped once the new run succeeds — docs: deploy.md "Ingest from other data sources", data-processing.md "About the ingestion container". Documented to prevent silent data loss in #2096.

Code mechanism (v12, consistent with the docs): The managed pipeline sets ingestionId = export manifest runId (dataFactory.bicep:3608), shared across all parts. Cleanup at :4549 (raw) / :4619 (_final_v*) drops folder extents whose tags !has drop-by:<ingestionId>. With distinct per-file ingestionIds (the reporter's part_N_0001 — no shared prefix), each part's cleanup evicts the earlier siblings → only the last survives. PropagateIngestionProperties: true (IngestionSetup_v1_2.kql:1075-1084) propagates the tags to the final table; :4619 also uses the folderPath tag for finalRetentionMonths, so disabling propagation removes the symptom but breaks final-table retention.

Action: None on the product if custom ingestion is confirmed — point to docs. If the reporter is on the managed pipeline, re-open as a real bug.