Lightning to use sync v2 formats

[taylordowns2000]

Description

Closes #4718.

Switches Lightning to the v2 (portability spec) YAML format for the four user-facing surfaces below. The wire shape is now fully aligned with the OpenFn portability spec: @openfn/kit/packages/lexicon/portability.d.ts.

1. Canvas → "View workflow as code"

CodeViewPanel serializes the live workflow state to v2 YAML via assets/js/yaml/format.ts (delegates to assets/js/yaml/v2.ts).

1a. Export button

Same code path — the Download button writes the v2 YAML the panel renders.

1b. Publish as template button

TemplatePublishPanel writes new templates in v2. Existing v1 templates remain readable on the Pick Template path (see 2a).

2. Create new → "Import code as workflow"

YAMLImportPanel uses the format-aware façade: detects v1 vs v2 from pasted/uploaded YAML and dispatches to the matching parser. Empty-editor placeholder shows the v2 shape so new users land on the new format.

2a. Pick template

TemplatePanel runs every template through the same façade. v1 templates load (lenient v1 parser), v2 templates parse strictly. A test asserts that the v1 and v2 versions of the same scenario produce structurally equivalent workflow state.

3. Settings → "Export project" button

Projects.export_project/2 calls Lightning.Workflows.YamlFormat.serialize_project/2, which emits v2 YAML. The old v1 emitter (Lightning.ExportUtils) is removed.

4. GitHub sync — block sandbox/ancestor branch collisions

A sandbox project can no longer link to the same (repo, branch) as any of its ancestors:

• Projects.ancestor_ids/1 walks the parent_id chain via a recursive CTE.
• ProjectRepoConnection.changeset/2 validates against ancestor branches and surfaces an inline error in the Save form.
• VersionControl.create_github_connection/2 re-checks the same condition transactionally so a concurrent insert can't race past the validation.

Spec coverage

Fully implemented from the spec:

Type	Field	Notes
ProjectSpec	id (required)	hyphenated project name
ProjectSpec	name?, description?
ProjectSpec	workflows: WorkflowSpec[]	YAML sequence
ProjectSpec	credentials?: Credential[]	YAML sequence; entries without resolvable owner email are dropped
ProjectSpec	collections?: string[]	YAML sequence of names, alphabetically sorted
WorkflowSpec	id?, name?	id is the hyphenated name
WorkflowSpec	steps: Array<Job | Trigger>	unified array
Step	id?, name?, next?
StepEdge	boolean | string | ConditionalStepEdge	parser accepts all three; emitter prefers the object form
ConditionalStepEdge	condition? (JS body), label?, disabled?
Trigger	enabled?
Credential	name, owner (both required)
Job	adaptors?: string[]	Lightning state has a single adaptor; collapsed/wrapped to a 1-element array
Job	expression?
Job	configuration?: object | string	string form only (Lightning credentials are <email>|<name> strings)

Edge condition mapping

Lightning's internal condition_type enum ↔ spec condition (a JS expression body):

Lightning enum	Wire condition
:always	omitted (or next: <target> shortcut)
:on_job_success	'!state.errors'
:on_job_failure	'!!state.errors'
:js_expression	<user JS body> (literal block when multi-line)

Parser also accepts the spec's shortcuts:

• next: { foo: true } → :always
• next: { foo: false } → :js_expression body "false" (Lightning has no :never)
• next: { foo: "<js>" } → :js_expression body verbatim

Lightning extensions (documented)

The portability spec marks Trigger fields explicitly TODO (// TODO a trigger supports many more keys which the spec must support). Until the spec author fills these in, we keep two trigger-only extensions:

• type: discriminator (webhook / cron / kafka)
• openfn: block carrying cron expression, cron cursor reference, webhook reply mode, and Kafka config

These are commented as Lightning extensions in both v2.ex and v2.ts.

What does NOT work yet (out of scope for this PR)

• Server-side YAML parsing. This PR only emits v2 on the server. The provisioning controller still accepts the legacy provisioner-shape JSON unchanged. Accepting v2 YAML/JSON over the API (and the matching @openfn/cli integration) is intentionally deferred — there is no Phase 5 import bridge in this branch.
• Spec fields not yet mapped: WorkflowSpec.globals, WorkflowSpec.start, WorkflowSpec.options, Step.previous, Job.configuration as object. Lightning has no internal equivalents today.
• test/integration/cli_deploy_test.exs is broken by the export cutover (it compares against a v1 fixture). Tagged :integration and disabled by default; will be updated when the @openfn/cli suite is next exercised.

Validation steps

1. Open a workflow → Code panel → confirm id:, name:, steps: array, no jobs:/edges:, jobs use adaptors: [...], expression bodies under expression: |.
2. Click Download → file matches the panel content.
3. Publish as template → the saved template's code is v2.
4. Pick Template → both a v1 (legacy) and a v2 template load into a working canvas.
5. Import code → paste v1 YAML, paste v2 YAML, both import.
6. Project settings → Export → downloaded file has top-level id:, workflows: is a sequence (- id: …), credentials: is a sequence of {name, owner}, collections: is a sequence of names.
7. Edge conditions: a workflow with on-success / on-failure edges round-trips through Code panel → reload as !state.errors / !!state.errors.
8. GitHub sync: connect a parent project to repo:main, create a sandbox of it, attempt to connect the sandbox to repo:main → inline error, Save button disabled. Switch the sandbox to repo:dev → Save enabled.

AI Usage

Please disclose whether you've used AI anywhere in this PR (it's cool, we just
want to know!):

• I have used Claude Code
• I have used another model
• I have not used AI

You can read more details in our
Responsible AI Policy

Pre-submission checklist

• I have performed an AI review of my code (we recommend using /review
  with Claude Code)
• I have implemented and tested all related authorization policies.
  (e.g., :owner, :admin, :editor, :viewer)
• I have updated the changelog.
• I have ticked a box in "AI usage" in this PR

[codecov]

Codecov Report

❌ Patch coverage is 86.78571% with 37 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.85%. Comparing base (7cedc62) to head (e383134).

Files with missing lines	Patch %	Lines
lib/lightning/workflows/yaml_format/v2.ex	84.61%	36 Missing ⚠️
...ghtning/version_control/project_repo_connection.ex	96.15%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4722      +/-   ##
==========================================
- Coverage   89.93%   89.85%   -0.09%     
==========================================
  Files         444      444              
  Lines       21979    22110     +131     
==========================================
+ Hits        19767    19866      +99     
- Misses       2212     2244      +32     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Security Review ✅

• S0 (project scoping): New root_id/1 walks parent_id via CTE keyed on the caller's project; tree_branch_conflict?/4 filters by root_project_id; validate event in github_sync_component.ex:30 now puts project_id from socket assigns rather than form params — a tightening, not a regression. The V2 YAML serializer is reached only through Projects.export_project/3, whose two callers (downloads_controller.ex:14 and provisioning_controller.ex:191) still gate on Permissions.can/4 before calling.
• S1 (authorization): No new web-layer mutating actions; existing can_install_github / can_initiate_github_sync gates on save, delete-connection, reconnect, initiate-sync are unchanged, and the new validate path performs no DB writes.
• S2 (audit trail): create_github_connection still wraps the insert and Audit.repo_connection(:created, user) in a single Repo.transact; the new insert_repo_connection/1 only translates the unique-index violation into :branch_used_in_project_tree and does not bypass the audit step.

[taylordowns2000]

@josephjclark , @midigofrank , sorry... dug deeper and this needs a few more tweaks. will be ready for review shortly.

[josephjclark]

High level QA notes for this branch:

• Export workflow yaml to a file and execute through CLI
• Copy a workflow yaml from production (with the old format) and import as a new workflow on this branch. Should be identical
• Copy a workflow yaml from this branch (with the new format) and import as a new workflow
• Export a project, paste the yaml locally, run a workflow through the CLI

I've run a smoke test and it looks great - the format looks right and I can run a workflow yaml through the latest CLI version

EDIT: QA looks good! Just one issue executing from the project, see below

[josephjclark]

@taylordowns2000 I notice an issue here where when viewing workflow yaml, the credential is not included. It's fine when exporting the project.

But this affects main too! So I'm going to raise it out as a bug, No action needed here.

[taylordowns2000]

@taylordowns2000 I notice an issue here where when viewing workflow yaml, the credential is not included. It's fine when exporting the project.

But this affects main too! So I'm going to raise it out as a bug, No action needed here.

This is super interesting @josephjclark - the reason is likely that credentials are not included in templates and "view" is the precursor to "publish as template". How should we handle this?

(a) Leave it as is
(b) Show the credential in view, strip it out in publish?
(c) Somehow leave it in templates, and strip it out on create?
(d) Something else?

[josephjclark]

@taylordowns2000 I'll take that convo over to #4731

[josephjclark]

@taylordowns2000 one change I'm going to recommend on both sides: let's add something like schema_version to the top of the project.yaml (or, if a workflow.yaml is standalone, let's add it there too).

The CLI currently writes a cli.version: 2 flag to help differentiate state v1 and state v2 files. But after this work that's not just a CLI thing - it's global.

This helps tools like the CLI and maybe later the provisioner understand and anticipate the structure they are about to parse.

The practical upshot of this is that the CLI is erroring if you do openfn checkout project-spec.yaml from a downloaded spec.That's a result of a bad assumption in the CLI code - but without a version flag I'm struggling to override that assumption.

Now, we should probably align that version with the portability spec v4. And we should probably considering semvering it (at least major and minor, maybe not patch)

So I recommend:

• Top level schema_version key
• Lightning needs to set this on project and workflow export
• Set the version value to 4.0
• Patch CLI to synchronise to the same version number

This will also make it much easier to coordinate the next schema change we want to make.

BTW I've made a separate issue for the "should portability spec be a JSON schema" thing. Still want to defer that for that though