Supporting Private Workflow Registry #20708

patrickhuie19 · 2026-01-03T18:54:16Z

Supporting Private Workflow Registry

This change is part of a series of changes to allow the workflow engine to pull workflows from metadata sources other than the WorkflowRegistry.

For broader context, check out https://docs.google.com/document/d/1K5lLtczMe_HyrLTbdLNmb_jxj40lPCmMZW4OBjn7K5U/edit?tab=t.t17fuwqoj0w9#bookmark=id.i3vs01f9lvvr

For the workflow engine specific changes, this diagram is most relevant:

Series

[merged] chainlink-protos: smartcontractkit/chainlink-protos#255
[merged] chainlink-common: smartcontractkit/chainlink-common#1749
[In-review] chainlink: #20708

Changes specific to chainlink

Supports AlternativeWorkflowMetadata sources configured via TOML: grpc brokered and file brokered (for on-prem deployments)
Each source is responsible for fetching active + paused workflows. They present events directly to the reconciliation loop, so that if there is a failure on their source (RPC read failure, grpc server unavailable), they can gracefully wait for the next polling cycle w/o churning engine deletion.
The GRPC workflow metadata source will be deployed to a server. JWT CSA-key based allowlist auth. Retries on retryable errors.
Workflow execution and deployment events now include the workflow metadata source name to give us flexibility in our analytics and UI consumers to filter data by source.
Adds metrics for future dashboards/alerting.

github-actions · 2026-01-05T03:02:53Z

I see you updated files related to core. Please run pnpm changeset in the root directory to add a changeset as well as in the text include at least one of the following tags:

#added For any new functionality added.
#breaking_change For any functionality that requires manual action for the node to boot.
#bugfix For bug fixes.
#changed For any change to the existing functionality.
#db_update For any feature that introduces updates to database schema.
#deprecation_notice For any upcoming deprecation functionality.
#internal For changesets that need to be excluded from the final changelog.
#nops For any feature that is NOP facing and needs to be in the official Release Notes for the release.
#removed For any functionality/config that is removed.
#updated For any functionality that is updated.
#wip For any change that is not ready yet and external communication about it should be held off till it is feature complete.

trunk-io · 2026-01-05T04:30:46Z

Failed Test	Failure Summary	Logs
`Test_CRE_GRPCSource_Lifecycle`	The test failed during the gRPC source lifecycle, but the specific error causing the failure is not indicated in the logs.	Logs ↗︎
`Test_CRE_GRPCSource_Lifecycle`	The test failed without a specific error message, but it appears to be related to issues in the gRPC source lifecycle workflow or environment setup.	Logs ↗︎

_{View Full Report ↗︎ ⋅ Docs}

trunk-io · 2026-01-05T04:30:46Z

Failed Test	Failure Summary	Logs
`Test_CRE_V2_Cron_Regression`	The test named Test_CRE_V2_Cron_Regression failed to complete successfully.	Logs ↗︎
`Test_CRE_V2_EVM_EstimateGas_Invalid_To_Address_Regression/[v2]_EVM.EstimateGas_-_invalid_'to'_address_fails_with_not_authored_con...`	The test failed because it could not connect to the local node, causing environment setup to fail.	Logs ↗︎
`Test_CRE_V2_EVM_EstimateGas_Invalid_To_Address_Regression/[v2]_EVM.EstimateGas_-_invalid_'to'_address_fails_with_cut_hex`	The test failed because it could not connect to the local node, causing environment setup to fail.	Logs ↗︎
`Test_CRE_V2_EVM_FilterLogs_Invalid_Addresses_Regression/[v2]_EVM.FilterLogs_-_invalid_addresses_fails_with_invalid_address`	The test failed because it could not connect to the local node, causing environment setup to fail.	Logs ↗︎

... and 48 more

View Full Report ↗︎ ⋅ Docs

cedric-cordenier · 2026-01-05T11:39:15Z

core/services/workflows/syncer/v2/multi_source.go

+// Graceful degradation: Even if all sources fail, we return an empty list and nil error
+// to allow retry on the next polling cycle. Errors are logged at appropriate levels
+// (WARN when all sources fail, ERROR for individual source failures).
+func (m *MultiSourceWorkflowAggregator) ListWorkflowMetadata(ctx context.Context, don capabilities.DON) ([]WorkflowMetadataView, *commontypes.Head, error) {


@patrickhuie19 I think you should lift this up into the syncer -- you're having to deal with inconsistent heads here, but if you just treated each source independently you'd avoid that

What I'm proposing is that in the syncUsingReconciliationStrategy you just:

loop over the sources

for each source -> fetch workflows and store associated head

once you've fetched from all sources -> merge the result (and assert no overlapping!)

then put it through the reconcile logic

handled a little differently: a117d58. updated the diagram in the description of the PR to match. syncUsingReconciliation loops over each source. each source will call ListWorkflowMetadata to get workflows, and then will send active + paused events to generateReconciliationEvents if there are no failures on the List action. In this way theres no deleting engine churn if on one polling cycle the eth mainnet rpc is down or the grpc server is unavailable after a few retries.

cedric-cordenier · 2026-01-05T11:42:15Z

core/config/docs/core.toml

 # ArtifactStorageHost is the host name that, when present within the workflow metadata binary or config URL, designates that a signed URL should be retrieved from the workflow storage service.
 ArtifactStorageHost = 'artifact.cre.chain.link' # Example

+[[Capabilities.WorkflowRegistry.AlternativeSources]]


Is Additional a better name here? (Alternative to me implies "replacement of", whereas we want to read from both)

hmm. we have to keep TOML backwards compatible, so it will take a few node revisions to deprecate the old TOML config section in favor of a more flexible section name like [[Capabilities.WorkflowRegistry.Source]] - i wanted to handle any such migration later on once we see usage.

I'm with Cedric on this. How about "Offchain Sources"?

cedric-cordenier · 2026-01-05T11:43:36Z

core/config/toml/types.go

+}
+
+// MaxAlternativeSources is the maximum number of alternative workflow sources
+const MaxAlternativeSources = 1


I would avoid enforcing a limit here (if we have to have a limit, I'd raise it to eg. 5); I'm anticipating that this will probably need to change

Some context: Our polling cycle in the syncer is 12 seconds, so I think when we see this feature getting a lot of use, we would want to move to an individual polling cycle per source. I don't anticipate that happening for the next 12 months though, so putting a strawman limit on the alternative sources to help us not scatter metadata seems useful. Given that, would you still propose raising the limit?

Raised to 5: a117d58

cedric-cordenier · 2026-01-05T11:46:46Z

core/config/toml/types.go

+	MaxConfigSize            *utils.FileSize
+	SyncStrategy             *string
+	WorkflowStorage          WorkflowStorage
+	AlternativeSourcesConfig []AlternativeWorkflowSource `toml:"AlternativeSources"`


Ideally we would deprecate the old fields (Address, NetworkID, ContractVersion, ChainID)
And introduce a []SourcesConfig that supports both contract + grpc sources
Then in setFrom you can read from the deprecated fields and add a contract source to the sources

Everywhere else in the application uses sources going forward

Will schedule for a follow up ticket, good idea.

https://smartcontract-it.atlassian.net/browse/CRE-1634

It will be hard to keep all sources together because config format is very different. How about a split into OnchainSources and OffchainSources? Potentially add XYZSources in the future.

cedric-cordenier · 2026-01-05T11:49:07Z

core/services/workflows/syncer/v2/contract_workflow_source.go

+
+// TryInitialize attempts to initialize the contract reader without blocking.
+// Returns true if initialization succeeded or was already done.
+func (c *ContractWorkflowSource) TryInitialize(ctx context.Context) bool {


Does this need to be public if it's only used one line 52?

good catch, thanks

cedric-cordenier · 2026-01-05T11:51:06Z

core/services/workflows/syncer/v2/contract_workflow_source.go

+// TryInitialize attempts to initialize the contract reader without blocking.
+// Returns true if initialization succeeded or was already done.
+func (c *ContractWorkflowSource) TryInitialize(ctx context.Context) bool {
+	c.mu.Lock()


Do you need a lock here? (Same question inside the List function)

This won't get called in parallel (the only thing calling this is the singleton sync routine)

I kept the lock to match the old pattern, which IFIUC had the lock in case the node was running the vault don functionality associated with the contract reader in addition to the engine deployment functionality. could remove; enables parallelization in tests w/ no cost for a singleton routine.

cedric-cordenier · 2026-01-05T11:56:15Z

core/services/workflows/syncer/v2/workflow_registry.go

+			w.lggr.Debugw("fetching workflow metadata from all sources", "don", don.Families)
+
+			// Use the multi-source aggregator to fetch workflows from all configured sources
+			allWorkflowsMetadata, head, err := w.workflowSources.ListWorkflowMetadata(ctx, don)


One edge case to be careful of here is that if you fail to get a result from one source, but get a result from another and then try to reconcile that partial list of workflows, it will lead to any workflows from the failing source to be deleted

That probably isn't desirable. You'll either need to:

just error at this point (even if you fetched one of the sources successfully)

or do reconciliation based on the source alone (i.e. all workflows are tagged with a source and you reconcile each in turn)

Probably the second one is what we want

Agreed, thanks for the catch - the second option looks best to me as well

…conciliation per source in syncUsingReconciliationStrategy directly

…of head in grpc source

cedric-cordenier · 2026-01-06T16:32:01Z

core/services/workflows/syncer/v2/grpc_workflow_source.go

+}
+
+// TryInitialize returns the current ready state (GRPC client initialized in constructor).
+func (g *GRPCWorkflowSource) TryInitialize(_ context.Context) bool {


Nit: TryInitialize shouldn't be part of any public interface -- we should treat this as an internal detail of each source

cedric-cordenier · 2026-01-06T16:39:03Z

core/services/workflows/syncer/v2/types.go

+	Ready() error
+
+	// TryInitialize attempts lazy initialization. Returns true if ready.
+	TryInitialize(ctx context.Context) bool


Can we move this out of the interface since it's only ever called internally (and in fact for some sources it does nothing)?

Yes good catch, thanks

cedric-cordenier · 2026-01-06T16:41:22Z

core/services/workflows/syncer/v2/workflow_registry.go

 		switch wfMeta.Status {
 		case WorkflowStatusActive:
 			switch engineFound {
-			// we can't tell the difference between an activation and registration without holding


@patrickhuie19 Lots of comments are being removed here... some of which contains some useful information about what we're doing -- do we need to remove?

no, we want them, thanks - I thought i added back in.

cl-sonarqube-production · 2026-01-07T00:47:02Z

Quality Gate passed

Issues
4 New issues
2 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

bolekk · 2026-01-07T04:03:38Z

core/config/docs/core.toml

 # ArtifactStorageHost is the host name that, when present within the workflow metadata binary or config URL, designates that a signed URL should be retrieved from the workflow storage service.
 ArtifactStorageHost = 'artifact.cre.chain.link' # Example

+[[Capabilities.WorkflowRegistry.AlternativeSources]]


I'm with Cedric on this. How about "Offchain Sources"?

bolekk · 2026-01-07T04:04:38Z

core/config/toml/types.go

+	MaxConfigSize            *utils.FileSize
+	SyncStrategy             *string
+	WorkflowStorage          WorkflowStorage
+	AlternativeSourcesConfig []AlternativeWorkflowSource `toml:"AlternativeSources"`


It will be hard to keep all sources together because config format is very different. How about a split into OnchainSources and OffchainSources? Potentially add XYZSources in the future.

bolekk · 2026-01-07T04:06:19Z

core/scripts/cre/environment/configs/workflow-gateway-don-grpc-source.toml

@@ -0,0 +1,105 @@
+# Workflow Gateway DON configuration with gRPC alternative workflow source enabled.


Can you change the comment to "workflow DON with Gateway"? The name of the file is already confusing.

bolekk · 2026-01-07T04:06:36Z

core/scripts/cre/environment/README.md

   - [Debugging core nodes](#debugging-core-nodes)
   - [Debugging capabilities (mac)](#debugging-capabilities-mac)
   - [Workflow Commands](#workflow-commands)
+   - [Alternative Workflow Sources](#alternative-workflow-sources)


"offchain"?

bolekk · 2026-01-07T04:07:04Z

core/scripts/cre/environment/README.md

+- `--binary-url-prefix`: Prefix for the binary URL in the output (e.g., `file:///home/chainlink/workflows/`)
+- `--config-url-prefix`: Prefix for the config URL in the output
+
+### Deploying a File-Source Workflow


bolekk · 2026-01-07T04:09:11Z

core/services/workflows/syncer/v2/grpc_workflow_source.go

+	}
+}
+
+// calculateBackoff calculates the backoff duration for a given attempt with jitter.


This is probably a ready Go library which core already imports. @jmank88 ?

bolekk · 2026-01-07T04:11:09Z

core/services/workflows/syncer/v2/contract_workflow_source.go

+
+// ContractWorkflowSource implements WorkflowMetadataSource by reading from the on-chain
+// workflow registry contract.
+type ContractWorkflowSource struct {


How about OnchainWorkflowSource?

bolekk · 2026-01-07T04:14:38Z

core/services/workflows/syncer/v2/contract_workflow_source.go

+
+// ListWorkflowMetadata fetches workflow metadata from the on-chain contract.
+// It lazily initializes the contract reader on first call.
+func (c *ContractWorkflowSource) ListWorkflowMetadata(ctx context.Context, don capabilities.DON) ([]WorkflowMetadataView, *commontypes.Head, error) {


Is this pretty much a copy of the old "getAllWorkflowsMetadata"? Can you preserve the TODO comment that got removed please?

patrickhuie19 added 2 commits December 30, 2025 10:43

MVP of multi-source workflows

c60eee1

adding jwt auth to grpc workflow source and smoke tests

5ed54bc

patrickhuie19 changed the title ~~Feat/multi source workflows~~ Supporting Private Workflow Registry Jan 3, 2026

minor cleanup + refactoring

ba21308

This was referenced Jan 3, 2026

adding protos for private workflow registry source smartcontractkit/chainlink-protos#255

Merged

adding grpc workflow metadata source client and related types smartcontractkit/chainlink-common#1749

Merged

patrickhuie19 added 2 commits January 4, 2026 21:50

cleanup + refactoring

8377374

Merge branch 'develop' into feat/multi-source-workflows

358fadd

patrickhuie19 added 3 commits January 4, 2026 22:16

lint

9ac9032

fixing regression in unit tests

5076067

further lint

89e1c14

further lint

986a8cf

patrickhuie19 marked this pull request as ready for review January 5, 2026 04:42

patrickhuie19 requested review from a team as code owners January 5, 2026 04:42

product-security-plaid-production bot requested review from justinkaseman and skudasov January 5, 2026 04:42

cedric-cordenier reviewed Jan 5, 2026

View reviewed changes

refactoring out MultiWorkflowSourceAggregator in favor of handling re…

a117d58

…conciliation per source in syncUsingReconciliationStrategy directly

patrickhuie19 added 5 commits January 5, 2026 12:42

Merge branch 'develop' into feat/multi-source-workflows

5700b11

removing redundant code in GenerateWorkflowEventsWithSource, and use …

f385cf3

…of head in grpc source

bumping protos + common + lint

f24657f

bumping common; make gomodtidy

bb2098f

cleaning up source service after head removal

5f1f8af

cedric-cordenier reviewed Jan 6, 2026

View reviewed changes

patrickhuie19 added 4 commits January 6, 2026 11:51

further docker changes for system tests

1626e3a

Merge branch 'develop' into feat/multi-source-workflows

9ac7f2c

cleaning TryInitialize out of the iface

1204d1f

skipping smoke test for now

d170011

bolekk reviewed Jan 7, 2026

View reviewed changes

		@@ -0,0 +1,105 @@
		# Workflow Gateway DON configuration with gRPC alternative workflow source enabled.

Supporting Private Workflow Registry #20708

Are you sure you want to change the base?

Supporting Private Workflow Registry #20708

Uh oh!

Conversation

patrickhuie19 commented Jan 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Supporting Private Workflow Registry

Series

Changes specific to chainlink

Uh oh!

github-actions bot commented Jan 5, 2026

Uh oh!

trunk-io bot commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trunk-io bot commented Jan 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cl-sonarqube-production bot commented Jan 7, 2026

Quality Gate passed

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

patrickhuie19 commented Jan 3, 2026 •

edited

Loading

trunk-io bot commented Jan 5, 2026 •

edited

Loading