Skip to content

IFC-2334: Webhook cache invalidation on KeyValue changes#8602

Merged
polmichel merged 15 commits intopmi-20260227-webhooks-ifc-2272from
pmi-20260312-webhooks-cache-invalidation-mngmt
Mar 17, 2026
Merged

IFC-2334: Webhook cache invalidation on KeyValue changes#8602
polmichel merged 15 commits intopmi-20260227-webhooks-ifc-2272from
pmi-20260312-webhooks-cache-invalidation-mngmt

Conversation

@polmichel
Copy link
Contributor

@polmichel polmichel commented Mar 14, 2026

Why

When a CoreKeyValue node (used as a webhook header) is created, updated, or deleted, the cached webhook configuration becomes stale. Without invalidation, webhooks continue using outdated header values until the cache naturally expires or the server restarts.

Based on #8590 and #8597.

Closes IFC-2334

What changed

  • Cache invalidation flow: New Prefect task (invalidate_webhook_headers) that resolves which webhooks reference a given KeyValue node and clears their cache entries
  • Cypher query: KeyValueGetWebhooksQuery traverses webhook__headers relationships to find affected webhooks
  • Trigger registration (bug fix): TRIGGER_KEYVALUE_WEBHOOK_INVALIDATE has been added to catalogue.py
  • Tests: Unit tests for the invalidation task, component tests for the query + cache integration, functional test verifying end-to-end cache clearing

How to review

Since this PR is based on other PRs, commits to be reviewed start from Encapsulate raw strings into small detectable objects

  1. backend/infrahub/webhook/query.py — Cypher query joining KeyValue → Relationship → Webhook
  2. backend/infrahub/webhook/tasks/invalidate.py + cache.py — the Prefect flow and cache deletion task
  3. backend/infrahub/webhook/triggers.py — trigger definition for KeyValue events
  4. Tests: unit/webhook/test_invalidate.py, component/webhook/test_query.py, functional/webhook/test_invalidate.py

How to test

# Unit tests
uv run pytest backend/tests/unit/webhook/test_invalidate.py -v
uv run pytest backend/tests/unit/trigger/test_catalogue.py -v

# Component tests (requires Neo4j)
uv run pytest backend/tests/component/webhook/test_query.py -v

# Functional tests (requires Prefect + Neo4j)
uv run pytest backend/tests/functional/webhook/test_invalidate.py -v

Impact & rollout

Checklist

  • Tests added/updated
  • External docs updated (if user-facing or ops-facing change)
  • Internal .md docs updated (internal knowledge and AI code tools knowledge)

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Mar 14, 2026

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (4)
  • main
  • stable
  • develop
  • release-.*

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: e63bd938-b45d-4ed4-80b6-4fe796b7a350

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Coding Plan
  • Generate coding plan for human review comments

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions github-actions bot added group/backend Issue related to the backend (API Server, Git Agent) group/frontend Issue related to the frontend (React) labels Mar 14, 2026
@codspeed-hq
Copy link

codspeed-hq bot commented Mar 14, 2026

Merging this PR will not alter performance

✅ 12 untouched benchmarks


Comparing pmi-20260312-webhooks-cache-invalidation-mngmt (a7fe5e6) with stable (43d16b8)1

Open in CodSpeed

Footnotes

  1. No successful run was found on pmi-20260227-webhooks-ifc-2272 (d8323d0) during the generation of this report, so stable (43d16b8) was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@polmichel polmichel force-pushed the pmi-20260312-webhooks-cache-invalidation-mngmt branch from 7e55860 to 221c269 Compare March 14, 2026 09:32
@polmichel polmichel changed the title Webhook cache invalidation on KeyValue changes IFC-2334: Webhook cache invalidation on KeyValue changes Mar 14, 2026
@polmichel polmichel self-assigned this Mar 14, 2026
@polmichel polmichel force-pushed the pmi-20260312-webhooks-cache-invalidation-mngmt branch 2 times, most recently from 100f6e0 to 8280b6e Compare March 14, 2026 09:51
@polmichel polmichel marked this pull request as ready for review March 14, 2026 09:51
@polmichel polmichel requested review from a team as code owners March 14, 2026 09:51
@polmichel polmichel force-pushed the pmi-20260312-webhooks-cache-invalidation-mngmt branch from 8280b6e to b8dc72a Compare March 14, 2026 10:00
@polmichel polmichel force-pushed the pmi-20260312-webhooks-cache-invalidation-mngmt branch from 1894daa to ec78b6f Compare March 16, 2026 22:16
@github-actions github-actions bot removed the group/frontend Issue related to the frontend (React) label Mar 16, 2026
@polmichel polmichel force-pushed the pmi-20260312-webhooks-cache-invalidation-mngmt branch from ec78b6f to d582061 Compare March 17, 2026 09:49
@polmichel polmichel marked this pull request as draft March 17, 2026 09:53
@polmichel polmichel force-pushed the pmi-20260312-webhooks-cache-invalidation-mngmt branch from d582061 to efd1e5b Compare March 17, 2026 09:55
@polmichel polmichel marked this pull request as ready for review March 17, 2026 09:56
@polmichel
Copy link
Contributor Author

polmichel commented Mar 17, 2026

While doing the application testing, there is a corner case I am thinking about regarding cache management.

When a KeyValue update fires NodeUpdatedEvent, both webhook-process (if the webhook listens to KeyValue changes) and webhook-invalidate-headers are triggered by the same event. Unless Prefect guarantees ordering between independent automations triggered by the same event, webhook-process could execute before webhook-invalidate-headers and serve stale cached headers.

One possible solution could be to have the webhook-process flow watch to see if the node that triggered the flow is a KeyValue. If so, it could force the data to be retrieved from the database.

This is a hacky solution. I am not sure if the use case I am describing could be a critical issue for customers, i.e. using old headers in the update request of a KeyValue node.

@ogenstad
Copy link
Contributor

While doing the application testing, there is a corner case I am thinking about regarding cache management.

I'm not sure we need to care about that. The way I see it is that we don't guarantee that a config change takes place immediately. A similar situation would be if you create a webhook from the first place and have it trigger on node creation events then we won't fire a webhook for that specific event.

Even with the changes you are making here it could be that we add a header to a webhook and then immediately afterwards we update some other unrelated node. There's a chance that one webhook would fire with the old headers before the new config is applied. I think this is fine as long as the convergence time is short.

@polmichel polmichel force-pushed the pmi-20260312-webhooks-cache-invalidation-mngmt branch from efd1e5b to 148d8f4 Compare March 17, 2026 13:37
@polmichel polmichel force-pushed the pmi-20260312-webhooks-cache-invalidation-mngmt branch from 7120f45 to a7fe5e6 Compare March 17, 2026 15:05
@polmichel polmichel merged commit 7668317 into pmi-20260227-webhooks-ifc-2272 Mar 17, 2026
48 checks passed
@polmichel polmichel deleted the pmi-20260312-webhooks-cache-invalidation-mngmt branch March 17, 2026 17:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

group/backend Issue related to the backend (API Server, Git Agent)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants