Save final state for runs; flexible input for "next" cron run

[taylordowns2000]

Please look at this super quick slide show for quick context. When testing this PR, note that it goes best with OpenFn/kit#1306 so you can see the performance enhancements, but you're welcome to test on the current worker too.

Overview

This PR saves the run's final state and lets users choose which step's output the cron scheduler uses as input for the next execution. Closes #4485.

Final state preservation: When the worker sends run:complete, we now persist the run's final state by setting final_dataclip_id on the runs table. Previously final_state was only used for webhook response broadcasting, then discarded. The handler accepts two mutually exclusive fields from the worker:

• final_dataclip_id — reuse an existing step output dataclip (single-leaf workflows, no duplication)
• final_state — a new map to persist as a separate dataclip (multi-leaf workflows with merged output)

Neither field is required — backward compatible with the current worker.

Cron input source: Each cron trigger now has an optional cron_cursor_job_id. When set, the scheduler uses that specific job's last successful step output. When null (the new default for fresh triggers), it uses the run's final_dataclip_id — the merged final state from above. A backfill migration points all existing cron triggers at their first downstream job, preserving current behavior.

Changes

• Migration: Adds nullable final_dataclip_id FK on runs → dataclips
• Migration: Adds nullable cron_cursor_job_id FK on triggers → jobs, with backfill for existing cron triggers
• Run schema: New belongs_to :final_dataclip; complete/2 casts the new field
• CompleteRun handler: resolve_final_dataclip/2 either reuses an existing ID or inserts a new dataclip. Respects save_dataclips: false. Wrapped in Repo.transact
• Run channel: Injects project_id into the run:complete payload so new dataclips can be associated with the correct project
• Trigger schema: New belongs_to :cron_cursor_job; cleared automatically when type changes to webhook/kafka
• Scheduler: Replaced last_state_for_job/1 with Invocation.get_next_cron_run_dataclip/1, which branches on cron_cursor_job_id — either last_run_final_dataclip/1 (final run state) or last_successful_step_dataclip/1 (specific job output)
• Invocation: Refactored cron dataclip resolution to be trigger-aware instead of job-aware
• TriggerForm UI: New "Cron Input Source" dropdown listing workflow jobs, with tooltip
• ManualRunPanel: Fixed stale-closure bug in cron auto-selection; disableAutoSelection now respects manuallyUnselectedDataclip instead of always being true in fullscreen mode
• Workflow serializer/snapshot: Round-trips cron_cursor_job_id through Y.Doc and snapshots

Performance considerations

• Single-leaf workflows (most common): Worker sends final_dataclip_id pointing to the last step's output dataclip. Zero additional storage — just sets the FK.
• Multi-leaf workflows: One extra dataclip INSERT per run. The merged body doesn't exist elsewhere, so this is net-new data.
• Transaction scope: CompleteRun now wraps in Repo.transact. For the ID-reuse path this adds no extra queries. For the final_state path it's one INSERT + one UPDATE in a short transaction.
• I was super worried about a race condition where the step:complete with the dataclip would be processed after the run:complete with a reference to the dataclip ID, but Claude tells me this isn't gonna happen... in a channel all events are processed in order.

How to test

1. Run any workflow — final_dataclip_id will be null until the worker is updated (backward compatible)
2. After worker change: run a single-leaf workflow, verify final_dataclip_id equals the last step's output_dataclip_id (no duplicate)
3. After worker change: run a branching workflow, verify final_dataclip_id points to a new dataclip with the merged output
4. Open a cron-triggered workflow — the trigger inspector should show a "Cron Input Source" dropdown
5. Default ("Final run state") uses the run's final_dataclip_id; selecting a specific job uses that job's last successful step output
6. Verify the manual run panel auto-selects the correct next cron dataclip

AI Usage

Please disclose whether you've used AI anywhere in this PR (it's cool, we just
want to know!):

• I have used Claude Code
• I have used another model
• I have not used AI

You can read more details in our
Responsible AI Policy

Pre-submission checklist

• I have performed an AI review of my code (we recommend using /review with Claude Code)
• I have implemented and tested all related authorization policies. (e.g., :owner, :admin, :editor, :viewer)
• I have updated the changelog.
• I have ticked a box in "AI usage" in this PR

[codecov]

Codecov Report

❌ Patch coverage is 94.28571% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.49%. Comparing base (f242061) to head (f2e2916).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
lib/lightning/runs/handlers.ex	92.30%	1 Missing ⚠️
lib/lightning_web/live/run_live/show.ex	85.71%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4531      +/-   ##
==========================================
- Coverage   89.55%   89.49%   -0.07%     
==========================================
  Files         425      425              
  Lines       20344    20365      +21     
==========================================
+ Hits        18220    18226       +6     
- Misses       2124     2139      +15     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[midigofrank]

@taylordowns2000 is this ready for review?

[taylordowns2000]

@taylordowns2000 is this ready for review?

It is, @midigofrank , but Stu's doing it with me this morning. Thanks for looking out :-)

[github-actions]

Security Review

⚠️ Automated security review did not complete.

Claude hit the max-turns limit or encountered an error before posting findings.
A manual review of S0 (project-scoped data access), S1 (authorization policies),
and S2 (audit trail coverage) is recommended for this PR.

See the workflow run for details.