feat(ws-worker): support multi-root @local adaptor resolution#1397
feat(ws-worker): support multi-root @local adaptor resolution#1397jeremi wants to merge 1 commit into
Conversation
OPENFN_ADAPTORS_REPO (and the --monorepo-dir / -m flag) now accept a colon-separated list of monorepo roots. When a job pins an adaptor to @Local, the worker walks the configured roots in order and resolves to the first root whose `packages/<shortName>/package.json` exists. This matches Lightning's AdaptorRegistry precedence so the registry view and the worker's execution path agree on which root supplies a given adaptor. Single-path values keep behaving exactly as before. When no root contains the adaptor the worker still surfaces a candidate path under the first root, so the runtime emits a clean "missing adaptor" error rather than crashing on a malformed colon-joined string. This unblocks the multi-root flow on the Lightning side, where the AdaptorRegistry already accepts the colon-separated form but the worker was rejecting it with ENOENT on @Local execution.
josephjclark
left a comment
josephjclark
left a comment
There was a problem hiding this comment.
Hi @jeremi, just taking a closer look at this. The solution is basically fine but it strikes me that this approach won't work for windows users, who have paths like c:/repo/adaptors.
I think a comma separated list will do just as well, and be a bit more accessible to other users
| ): { plan: ExecutionPlan; options: WorkerRunOptions; input: Lazy<State> } => { | ||
| const { collectionsVersion, monorepoPath } = options; | ||
|
|
||
| // monorepoPath is a colon-separated list of monorepo roots, mirroring how |
There was a problem hiding this comment.
The most obvious problem that strikes me here is that this will break on windows machines.
It's convenient that the colon approach mimics a unix path. And I'm not aware we have many windows users doing this stuff - although occasionally someone crops up.
Anyway I would prefer a comma or semi-colon separated list here so that it's safer
| return candidate; | ||
| } | ||
| } | ||
| // Fall back to the first root's resolved candidate path. The directory |
There was a problem hiding this comment.
Way too many comments here, and in the wrong place. But I can tidy that up before merge to main. I presume these are AI generated?
| import { Job } from '@openfn/lexicon'; | ||
|
|
||
| // Builds a temporary monorepo root with package.json files in each named adaptor. | ||
| const makeMonorepo = (adaptors: string[]) => { |
There was a problem hiding this comment.
Not wild about the file IO stuff in these tests to be honest. Mostly in this repo we use the mock-fs library to mock out the file system. If you're using an AI agent can you ask it to regenerate like that?
2 participants
Summary
Lets
@openfn/ws-workerresolve@localadaptors against a colon-separatedOPENFN_ADAPTORS_REPO, so a private adaptor monorepo can be loaded alongside the canonical OpenFn adaptors monorepo without forking.packages/<shortName>/package.json, the earlier root wins.Companion to Lightning PR
This is the executor-side half of OpenFn/lightning#4714. Without this patch, colon-separated
OPENFN_ADAPTORS_REPOonly widens what Lightning'sAdaptorRegistryexposes (picker UI, metadata, version validation); the worker still tries to load the literal joined path and dies withENOENT. The two PRs ship together so multi-root @Local execution works end-to-end.The first-wins semantics here mirror Lightning's
AdaptorRegistry: a private adaptor repo listed before the canonical OpenFn monorepo will override individual adaptors locally, and the registry view and the worker's execution path stay consistent.Test plan
pnpm --filter @openfn/ws-worker test— 30/30convert-lightning-plantests green, including 5 new cases:@localagainst a single existing rootpnpm --filter @openfn/ws-worker test:typescleanpnpm prettier --checkclean for the touched files@openfn/language-publicschema@localwas resolved by the patched worker to/private-adaptors/packages/publicschema/dist/index.js(the first root in the colon list), and the runtime loggedResolved adaptor @openfn/language-publicschema to version 0.0.1.Changeset
minorbump for@openfn/ws-worker.