Add finding lifecycle provenance events (3/3)

[devGregA]

Part 3/3 of a visibility series: #15150 · #15151 · #15152 (this PR)

⚠️ Merge-order note (interacts with #15150)

This PR and #15150 both update the pinned query-count baselines in unittests/test_importers_performance.py and unittests/test_tag_inheritance_perf.py, each measured against dev without the other. #15150 must merge first: this PR's migration 0279_finding_lifecycle_event depends on 0278_finding_processing_status (added in #15150), so applying #15152 on its own fails with NodeNotFoundError. They are otherwise functionally independent. Separately, whichever merges second needs a rebase with re-measured baselines, because the deltas add up. Example: EXPECTED_ZAP_IMPORT_V2 is 287 on dev, 288 in each PR alone, and 289 once both are in (+1 for this PR's created-events batch, +1 for #15150's batch stamp). I'll push the combined-baseline rebase on whichever PR lands second.

---

Description

Adds a finding lifecycle provenance ledger: an append-only record of the semantic transitions in a finding's life, answering the questions behind a large class of support tickets — "why did this finding close?", "why is this a duplicate?", "when was this ticketed?"

New model Finding_Lifecycle_Event (finding, actor_type, action, detail JSON, created), written at five capture points:

Event	Where	Detail
created	importer + reimporter (only findings actually created)	test id, scan type, import vs reimport
closed	mitigate_finding (close_old_findings / re-upload)	close reason, test id
reopened	reimporter reactivation	reason, test id
marked_duplicate	set_duplicate (covers batch dedupe; transitive re-points record their own event)	original finding id, hash_code
pushed_jira	add_jira_issue success	JIRA issue key

This complements — deliberately does not duplicate — existing history: pghistory triggers capture field-level diffs, and Test_Import_Finding_Action records per-import actions. Neither can express why: which key matched, which re-upload closed it, what it's a duplicate of. That's what this table records.

API: GET /api/v2/findings/{id}/lifecycle_events/ — the finding's provenance timeline, newest first (read-only).

Performance / operational design (this table must never become a problem):

• Transition-only writes: a reimport that matches findings unchanged writes zero rows (covered by an explicit test). Event volume tracks churn, not scan cadence.
• Batched: importers bulk_create events at the existing 1,000-finding batch boundaries; no signals, no per-row saves; detail values truncated to 256 chars.
• Delete-safe: the FK has db_constraint=False + on_delete=DO_NOTHING, so bulk finding deletion never touches this table (no ORM cascade collection, no DB cascade). Orphans are swept by retention.
• Two indexes only: (finding, created) for the timeline read; (created) for the purge.
• Retention: nightly beat task purges events older than DD_FINDING_LIFECYCLE_EVENTS_RETENTION_DAYS (default 540), batched deletes.
• Kill switch: DD_FINDING_LIFECYCLE_EVENTS_ENABLED (default true) turns all writes into no-ops.
• Measured cost, pinned by the perf baselines: +1 query per import batch (bulk-created CREATED events), +0 on unchanged-match reimports, +1 per finding closed by close_old_findings, +1 per duplicate marked by dedupe. The query-count baselines in test_importers_performance.py and test_tag_inheritance_perf.py are updated accordingly and now double as a regression tripwire for this table's write discipline.

Test results

New module unittests/test_finding_lifecycle_events.py (5 tests), including a full reimport cycle over the semgrep close-old fixtures (created → matched-with-zero-events → closed with reason on unique-id change → reactivated), dedupe originals, the API endpoint, retention purge, and the kill switch.

Regression: test_importers_closeold, test_importers_deduplication, test_deduplication_logic (135 tests) and test_rest_framework.FindingsTest (27 tests) all pass against PostgreSQL via the unit-test compose image. makemigrations --check clean; ruff (0.15.20, repo config) clean.

Not covered by an automated test: the pushed_jira event (requires a mocked JIRA stack; the capture is three lines on the existing success path). Happy to extend a JIRA test if maintainers prefer.

Documentation

Additive feature; the API action is schema-annotated. Happy to add a docs page (finding lifecycle events + settings reference) in this PR or as a follow-up, whichever maintainers prefer.

🤖 Generated with Claude Code

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[github-actions]

Conflicts have been resolved. A maintainer will review the pull request shortly.

[Maffooch]

Fixed, tested, and approving (milestone 3.1.100) — must merge after #15150.

Issues found in review:

1. set_duplicate recorded a MARKED_DUPLICATE lifecycle event on every call, so a product-wide re-deduplication appended a redundant event for findings that were already duplicates of the same original (breaking the "transition-only" provenance guarantee).
2. The PR body claimed the three parts "can merge in either order," but migration 0279_finding_lifecycle_event hard-depends on 0278_finding_processing_status from Add post-import processing lifecycle to Finding (processing_status) (1/3) #15150 — confirmed by makemigrations --check failing with NodeNotFoundError when Add finding lifecycle provenance events (3/3) #15152 is applied without Add post-import processing lifecycle to Finding (processing_status) (1/3) #15150.

Fixes:

• Added a guard in set_duplicate so the MARKED_DUPLICATE event is only recorded on a genuine transition (skips when new_finding is already a duplicate of that same existing_finding).
• Added regression test test_rededupe_does_not_append_duplicate_marked_duplicate_event.
• Corrected the PR body's merge-order note to state Add post-import processing lifecycle to Finding (processing_status) (1/3) #15150 must merge first.

Verified (in the combined #15150 + #15152 state, which is what production will have):

• Full test_finding_lifecycle_events suite: 6/6 passing, including the new re-dedup guard test (re-running dedup keeps the event count at 1).
• The existing test_dedupe_records_marked_duplicate_with_original still passes (normal dedup still records exactly one event).

The lifecycle provenance feature still meets its goal; the re-dedup double-count is closed and the migration ordering is documented correctly. Looks good now (merge after #15150).

[valentijnscholten]

During the selection of the django-auditlog replacement we found it a benefit of pghistory because it allows for adding extra context to data edits and/or group edit together and show them as a single event. I can understand the benefit of having a "life cycle event" only table. It would thought be the third table capturing finding events next to Test_Import_Finding_Action and pghistory. I thin it's worth a small exploration if we could achieve the same nice "life cycle event" view on the existing pghistory pages and by using pghistory contexts to enrich the events.