Crawler support to automatically create/update package yaml files

[kmturley]

Currently yaml files are manually added/updated via PR request. This requires substantial time to ensure the files are formatted correctly and the links are correct.

Crawlers such as https://crawlee.dev allow scripts to read an API or webpage, and extract data needed.

Running a crawler could be done in a pipeline, or as a developer step manually. However Crawlers can check many sites/pages and automate the work required to create data files.

Example workflow:

This would allow Crawlers to be used to generate yaml files, which could then be submitted via PRs and go through the validation/virus scanning checks.

[DropSnorz]

I've implemented crawlers in the past to generate OwlPlug compatible feed from already existing plugins collection. Detecting creation and deletion can be quite challenging depending on how the target website is implemented. It's also a legal grey area... as it can be seen as a way to "steal content" and bypass website traffic (ads, donation, ...)

Few month ago, we mentionned implementing a Github Action to integrate with developer workflow. But instead, we can crawl/call Github APIs to look for new releases for plugins already referenced in the registry. If a new release is detected, we can try to generate metadata based on previous data in the OAS registry, and automatically open a PR.

• PluginVersion
  • name : Retrieved from version (n-1) from registry feed
  • author: Retrieved from version (n-1) from registry feed
  • license: Retrieved from version (n-1) from registry feed
  • type: Retrieved from version (n-1) from registry feed
  • tags: Retrieved from version (n-1) from registry feed
  • url: Retrieved from version (n-1) from registry feed
  • audio: Retrieved from version (n-1) from registry feed
  • image: Retrieved from version (n-1) from registry feed
  • date: Computed from release (n) from Github API using created_at.
  • changes: Computed from release (n) from Github API using body.
  • files: Try to map files from registry version (n-1) and Github API release (n) based on a string similarity rating between File.url and Github's assets[].url.
    • systems: Retrieved from version (n-1) from registry feed
    • contains: Retrieved from version (n-1) from registry feed
    • sha256: Computed from release (n) from Github API using assets[].digest
    • size: Computed from release (n) from Github API using assets[].size
    • type Retrieved from version (n-1) from registry feed or Computed from release (n) from Github using assets[].url file extension
    • url : Computed from release (n) from Github API using assets[].url

I think something similar was in place in the studiorack-registry. This can work well to be notified about patches and updates. A manual review is still mandatory as:

• Major changes might require to update screenshot, audio, description, ...
• Some changes introduced in Github Release might not be detected (new packaging model, new kind of files, ...)

One step back, I think we can provide 3 kind of complementary automation:

• Reproducible builds, with publicly visible logs for them. #28 - A push-mode, a Github Action for developers to integrate into their release workflow and automate PR opening.
• This issue - A pull-mode, a Github release crawler for submitting updates as PR. We can introduce a 7d delay to not conflict with the push-mode action workflow managed by plugin developer. If developer cannot integrate the action (external CI, no time, no knowledge, ...) it provides "optimistic" updates notification.
• Scheduled registry maintenance job #26 - A scheduled job for registry feed maintenance (deleted files, invalid url, ...) and error reporting

[kmturley]

The solutions can solve different uses-cases:

1. Individual package - Site form, hand-coded yaml with pull request
2. Multiple packages - Crawler script or Release API script automation.

The issue with automated scripts in the registry build, is that they can hit API limits, take a long time or even error.
If we add too many steps to the registry build workflow, then we could block the registry build.
Package files need to be Virus scanned and checksums, which is resource-intensive to run every build.

My suggestion is for the Crawler / Release API script to generate yaml files upfront, any failures or bad data can be caught in the PR review. Files can be Virus scanned and checksum before submission. Keeping the registry build very fast and snappy.

We can have the monthly job, start to automate this by running those and submitting a PR

[kmturley]

I have been prototyping this workflow in a separate repo here:
https://github.com/kmturley/static-api-generator

Here is an updated diagram of the steps:

import.ts - contains multiple sources (Api, File, Site), when run this has one Target which is yaml files. These changes can be committed as changes and reviewed in a PR, before merging.

main.ts - contains one source (yaml files), and exports one Target (json files) for the static api, which is deployed to GitHub pages.

I like this pattern of Sources and Targets as it is extendable. For example we could have a GitHub repo source like you mentioned. Which loops through the github repo ids, and pulls the release date to create the yaml files automatically.

I'm going to iterate a little bit more on this prototype before merging anythign in Open Audio Stack. but this could be a huge time-saver for keeping plugin/preset metadata up-to-date!

[kmturley]

Realized this could also support Large-language models, which could generate the package metadata based on a prompt.
Here is the outline of an example: kmturley/static-api-generator@1958bdc

The model used does not generate good data, need to try a few options to find the best free open-source model for package metadata generation.

[RustoMCSpit]

are there API equivalents for codeberg, gitlab, etc.?