Certified Nodes Catalogue Not Updating on Website

[n-lark]

Fix: Certified Nodes Catalogue Not Updating on Website

Summary

The Node-RED workflow that powers the certified nodes listing on the FlowFuse website is broken or outdated. As a result, https://catalog.flowfuse.com/catalogue.json is not being updated, and not all certified nodes are appearing on the website.

Background

The Node-RED flow hosted at https://ff-integrations.flowfuse.cloud builds an API for node catalogue data. It is supposed to:

1. Periodically fetch the full node catalogue from the Node-RED Flow Library
2. Fetch the list of FlowFuse-certified nodes
3. Store both in global context
4. Serve GET /api/nodes — filtering the catalogue and flagging which nodes are FlowFuse-certified

The website then consumes this data to populate the integrations/certified nodes pages.

Flow: https://ff-integrations.flowfuse.cloud/?#flow/ee3a6dc032a838f4

Catalogue endpoint (not updating): https://catalog.flowfuse.com/catalogue.json

How the Website Consumes This Data

The website has two data pipelines that both depend on the upstream endpoints:

1. src/_data/certifiedNodes.js

• Fetches https://catalog.flowfuse.com/catalogue.json at build time using @11ty/eleventy-fetch
• Cache duration: 4 hours
• Reads data.modules[] — each item must have id, and optionally url and description
• Transforms each entry: strips node-red-contrib-, node-red-node-, @flowfuse/ prefixes, title-cases the remainder
• Returns an array of { title, id, url, description } objects

2. src/_data/integrations.js

• Fetches https://ff-integrations.flowfuse.cloud/api/nodes at build time, cache: 1 hour
• Expects response shape: { catalogue: [ { _id, name, description, version, downloads: { week }, categories, npmScope, npmOwners }, ... ] }
• Calls certifiedNodes() and builds a lookup map keyed by node.id → ffNodesMap
• Takes the full response.catalogue, sorts by downloads.week descending, slices top 50
• Merge step: iterates certified node IDs; any not already in the top-50 map are looked up in response.catalogue by n._id === certifiedId and appended — ID matching is exact string equality
• For each node, fetches https://registry.npmjs.org/${node._id} (cache: 1h) to enrich: author, maintainers, homepage, bugs, repository, time, lastUpdated, created, license
• Parses repository.url via regex to extract githubOwner / githubRepo
• Attempts to fetch https://api.github.com/repos/{owner}/{repo}/contents/examples — filters for .json files and fetches their raw content; silently skips if the folder doesn't exist
• Rewrites relative image paths in readmes to raw.githubusercontent.com/{owner}/{repo}/master/... — hardcoded to master branch
• Final sort: ffCertified nodes first, then descending by downloads.week

3. src/integrations/index.njk (client-side)

• At page load, fetches https://ff-integrations.flowfuse.cloud/api/nodes again directly in the browser
• Uses a generatedNodeIds Set (injected at SSG build time) to decide: internal link /integrations/{id} vs external https://flows.nodered.org/node/{id}
• Category filter keys are hardcoded: ai, communication, data-and-analytics, database, hardware, home-automation, industrial, security, storage, tools, ui, utility
• Filtering checks node.categories.includes(key) against the raw API response (not build-time prefixed values)
• Certified filter checks node.ffCertified on the live API response — this flag must be set by the Node-RED flow itself

4. src/integrations/integrations.njk

• Generates one static page per node via Eleventy pagination over the integrations data collection
• Page permalink: /integrations/{node._id}/
• Renders integration.readme via the md filter and rewriteIntegrationLinks filter

Relevant Website Files

• src/integrations/integrations.njk
• src/_data/integrations.js
• src/_data/certifiedNodes.js
• src/integrations/index.njk

Problem

• catalogue.json is stale — it is not being refreshed by the Node-RED flow
• Not all FlowFuse-certified nodes appear on the website
• The periodic fetch mechanism in the flow appears to be broken or misconfigured

Tasks

• Investigate the Node-RED flow at ff-integrations.flowfuse.cloud — identify why the periodic catalogue fetch is failing
• Confirm whether the issue is in the fetch trigger (e.g. inject/cron node), the HTTP request to the Flow Library, or the certified nodes list fetch
• Fix or update the flow so catalogue.json is refreshed on schedule
• Verify the GET /api/nodes response correctly includes all certified nodes with the ffCertified: true flag (this flag is consumed by the client-side filter in index.njk)
• Confirm certified node id values in catalogue.json (data.modules[].id) exactly match _id values in the /api/nodes catalogue response — the merge in integrations.js uses strict equality
• Confirm the website integrations pages reflect the full, up-to-date certified node list after the fix

Success Criteria

• https://catalog.flowfuse.com/catalogue.json is being updated on its expected schedule (confirm via Last-Modified header or timestamp in payload)
• All FlowFuse-certified nodes appear in the GET /api/nodes response with ffCertified: true
• The website integrations pages (/integrations) show the complete and current list of certified nodes
• No certified nodes that exist in the source list are missing from the website
• The Node-RED flow runs without errors (no error nodes triggered, no failed HTTP requests in the debug log)

Additional Context

Raised in Slack here. The flow logic is understood, the issue is likely a broken trigger or a stale/incorrect source URL.