ingestion_ManifestAdded trigger not firing — msexports_ETL_ingestion pipeline never executes

[alsanch]

🐛 Problem

The msexports_ETL_ingestion pipeline is never executed because the ingestion_ManifestAdded trigger does not fire, despite being in Started state.

✅ Cost Management export runs successfully — files are written to the storage account
✅ Pipeline msexports_ExecuteETL is executing correctly
✅ Trigger ingestion_ManifestAdded shows as Started in Data Factory Studio
❌ Trigger ingestion_ManifestAdded never fires (no trigger runs recorded in Monitor)
❌ Pipeline msexports_ETL_ingestion is never invoked

👣 Repro steps

Deploy FinOps hub
Configure Cost Management export — export runs successfully and writes to storage
Go to Data Factory Studio → Monitor → Trigger runs
Filter by ingestion_ManifestAdded — no runs are recorded
Go to Monitor → Pipeline runs — msexports_ETL_ingestion has never been triggered

🤔 Expected

The ingestion_ManifestAdded trigger should automatically invoke the msexports_ETL_ingestion pipeline.

🔧 Environment

• FinOps hub version: deployed on the 21-05-26
• Billing account type: MCA
• Power BI report type: ADX (Azure Data Explorer)
• Cost Management export: all of them

[RolandKrummenacher]

Hi @alsanch, thanks for the detailed report and for the clear checklist of what is and isn't working — that's really helpful.

A bit of context on how the pipeline is wired, because the two pipeline names in your report sit on different "hops" of the chain:

1. Cost Management writes a manifest to the msexports container → the msexports_ManifestAdded trigger fires → msexports_ExecuteETL runs.
2. Inside msexports_ExecuteETL, a "For Each Blob" loop calls msexports_ETL_ingestion once per exported data file, then a "Copy Manifest" step copies the manifest into the ingestion container.
3. That copied manifest is what fires ingestion_ManifestAdded → ingestion_ExecuteETL → ADX ingestion.

There's an important detail in step 2: if the export manifest reports zero blobs or zero data rows, msexports_ExecuteETL still completes successfully, but it intentionally skips both the msexports_ETL_ingestion calls and the "Copy Manifest" step (added in #1535 to avoid ingesting empty exports). When that happens, no manifest lands in the ingestion container, so ingestion_ManifestAdded legitimately has nothing to fire on — which matches your observation of zero trigger runs even though the trigger is Started.

The other common cause for a Started storage-event trigger that never records runs is the Microsoft.EventGrid resource provider not being registered (or the storage-side event subscription not being created), since these are Event Grid Blob-created triggers. Our troubleshooting guide covers this: https://aka.ms/ftk/trouble (Step 2).

To pin down which path you're hitting, could you share:

• Does the msexports_ManifestAdded trigger record runs (Monitor → Trigger runs)? Or is it only ingestion_ManifestAdded that shows none?
• In a successful msexports_ExecuteETL run, what do the "Set Has No Rows" / "For Each Blob" / "Copy Manifest" activities show (skipped vs. executed)?
• Are there parquet files appearing in the ingestion container, or is it empty?
• The blobCount / dataRowCount values from a recent export manifest.json in the msexports container.
• Confirmation that Microsoft.EventGrid is registered on the hub's subscription, and whether re-starting/re-running the export changes anything.

With that we can tell whether this is the expected empty-export skip or a genuine trigger/Event Grid problem. #needs-info

[RolandKrummenacher]

🔍 Root-cause analysis (engineering)

Root cause: Not root-caused — two equally-supported candidates, and the report gives conflicting/insufficient signal to choose:

(A) Event Grid storage-event trigger not delivering. Both ingestion_ManifestAdded and msexports_ManifestAdded are BlobEventsTrigger (Event Grid) — src/templates/finops-hub/modules/fx/hub-eventTrigger.bicep:75-84 (type BlobEventsTrigger, events ['Microsoft.Storage.BlobCreated'], scope = storageAccount.id). A Started trigger with zero runs is the documented symptom of Microsoft.EventGrid RP not registered / storage event subscription missing — docs-mslearn/toolkit/help/troubleshooting.md:72-75. hub.bicep:17-18 even carries commented-out scaffolding for forcing EventGrid RP registration.

(B) Intentional empty-export skip (by-design). msexports_ExecuteETL sets hasNoRows = blobCount null/0 OR dataRowCount null/0 (Exports/app.bicep:333-336). When true, "For Each Blob" iterates an empty array so msexports_ETL_ingestion is never called (app.bicep:897), and "Copy Manifest" is gated by @not(hasNoRows) so the manifest is never copied to the ingestion container (app.bicep:953-968) → ingestion_ManifestAdded never gets an event. This exactly reproduces "msexports_ExecuteETL succeeds, msexports_ETL_ingestion never runs, ingestion_ManifestAdded records no runs." Added deliberately in #1535 (docs-mslearn/toolkit/changelog.md:118).

Scope / impact: Candidate B is intended behavior on all versions since the #1535 fix (incl. v14). Candidate A is an environment/prerequisite condition, not a code defect. Reporter is MCA + "all exports"; several MCA datasets (e.g. price sheet, reservation transactions/recommendations) can legitimately produce zero-row manifests for a given period, which would surface as B and look like a bug.

Fix options:

1. Gather the missing diagnostics (msexports_ManifestAdded run history, activity skip status, manifest blobCount/dataRowCount, ingestion-container contents, EventGrid RP state) and re-triage — distinguishes A from B definitively. Trade-off: round-trip with reporter.
2. If B: documentation only — clarify in data-processing.md/troubleshooting.md that zero-row exports intentionally skip msexports_ETL_ingestion and ingestion_ManifestAdded. Trade-off: no code change; UX still confusing in Monitor.
3. If A: treat as Resolution: External / prerequisite (register Microsoft.EventGrid), already documented at troubleshooting.md:72-75. Trade-off: optionally harden deploy to auto-register the RP (hub.bicep:17-18 scaffolding).

Recommendation: Hold for info (Option 1). Do not accept as a code bug yet — the most likely explanation is either by-design empty-export skipping (B) or an Event Grid RP prerequisite (A), neither of which is a defect. Decide A vs B after the reporter confirms whether msexports_ManifestAdded fires and whether parquet files reach the ingestion container.