Plugin navigation by JFRudzinski · Pull Request #186 · FAIRmat-NFDI/nomad-docs

JFRudzinski · 2025-12-01T14:29:30Z

Add Automated NOMAD Plugin Registry

This PR implements an automated plugin registry documentation page that queries and displays all NOMAD plugins from the FAIRmat-NFDI and nomad-coe GitHub organizations.

Features

Automated Data Collection: Queries the NOMAD Example Oasis nomad-plugins app for plugin metadata
Multi-Organization Support: Aggregates plugins from FAIRmat-NFDI and nomad-coe
Comprehensive Metadata:
- Plugin details: name, description, repository, GitHub stars
- Deployment status: PyPI availability, NOMAD Central, Example Oasis
- Entry point types, authors, maintainers, and dependencies
Visual Statistics: Overview metrics and Mermaid pie chart for plugin type distribution
Monthly Automation: GitHub Actions workflow regenerates the page on the 1st of each month

Implementation

New Files:

scripts/generate_plugin_registry.py - API query and markdown generation script
.github/workflows/update-plugin-registry.yml - Monthly automation workflow
docs/examples/plugin_registry.md - Generated registry page
scripts/README.md - Script documentation

Workflow:

Schedule: Monthly on the 1st at 00:00 UTC
Triggers: Scheduled run, manual dispatch, or push to main (for script changes)
Creates PR for scheduled runs to allow review before merge

Manual regeneration:

python scripts/generate_plugin_registry.py

github-actions · 2025-12-01T14:29:41Z

🤖 Hi @JFRudzinski, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

JFRudzinski · 2025-12-01T14:31:04Z

@DanielYang59 Before I send to others for review, could you please assess my overall approach here, provide feedback on the code, especially the github workflow which I have not tested yet.

github-actions · 2025-12-01T16:37:48Z

PR Preview Action v1.8.1
🚀 View preview at https://FAIRmat-NFDI.github.io/nomad-docs/pr-preview/pr-186/
Built to branch `gh-pages` at 2026-02-13 19:45 UTC. Preview will be ready when the GitHub Pages deployment is complete.

JFRudzinski · 2026-01-22T10:46:53Z

Thank you @lukaspie and @ahm531 for the feedback. Below I address individual comments and note changes so far:

From @lukaspie:

Plugin overview

Not sure how editable the table format is, but the plugin overview table could maybe be a bit smaller by adjusting the > column widths (first and last two could be smaller and the others wider).

Done. (According to Ahmed's suggestion)

Detailed Plugin Information

Maybe instead of the complete link to the repo, we could just write Repository | Docs and make each of them a link. I think linking the docs would be great in any case.

Done.

Is stars a good stat to display? I get that it is a rough measure for how "successful" a plugin is, but a lot of these plugins have 0-2 stars, which may make them look undesirable for users. But I don't have a better proxy for how widely a package is adapated.

I don't really have a strong opinion myself on this myself. I see that at the moment they are not very useful, but perhaps in the future they are more indicative of usage? I left them now for further comments from others.

Not sure I like authors and maintainers as two separate categories. A lot of these plugins also use "The NOMAD authors", which kind of defeats the purpose. This may be a more general discussion point: shall we always use this generic "The NOMAD authors" or really the actual authors? The latter is of course helpful if you want to talk to the devs directly, but may be harder to maintain with people leaving, etc.

Can you explain why you don't like 2 separate categories? This is well-defined in GitHub. For me the author list is for acknowledgement. The maintainer list for contact. It is also mentioned at the top of the page that FAIRmat owns and maintains all these plugins. Perhaps that should be made clearer.

Usage of "The NOMAD Authors" appears to be largely used by nomad-coe plugins which are or will eventually be archived.

We will need to add a check for archived repos and perhaps just list them concisely at the bottom of the page or something.

The issue of maintaining author lists is independent of this page. But I think it is something that we want to do, at least while FAIRmat exists, as it is a major way we can acknowledge our co-workers work, through publications of repos on Zenodo.

For the list of entry points, we could give for each of them their type (e..g bandstructurenormalizer (Normalizer), then we could skip the item Plugin Types. But either way is fine.

I am fine with this but some plugins have many entry points which will blow up the text a little. I like the way Ahmed did it cause the Plugin Types are in the overview, and the detailed list only in the drop down. Also, often the type is also in the entry point name, as in the example you gave. I am open to more opinions on this though.

General

I discussed this with @hampusnasstrom already, the check whether a plugin is available on a given deployment is broke. But this is more an issue for the nomad-plugins code. EDIT: saw there is an https://github.com/FAIRmat-NFDI/nomad-plugins/issues/22for this already 👍

👍

I would prefer if the automated script was run in shorter intervals, weekly maybe. We are still developing a lot of new plugins, would be great if they showed up on this page early.

Done. (There is a check that only opens a PR if changes are found, so this should have no negative side effect).

Does it make sense to have a similar page for non-FAIRmat plugins? This may help to connect the community.

Yes, but let's finalize the format first.

Create an analogous page for external plugins

From @ahm531

I took the format suggestions from test_plugin_registry.md directly. If there is some other consideration that I missed, please let me know.

JFRudzinski · 2026-01-22T11:26:16Z

Oh .. The location of the page let's discuss later

lukaspie · 2026-01-22T17:02:40Z

From @lukaspie:

Not sure I like authors and maintainers as two separate categories. A lot of these plugins also use "The NOMAD authors", which kind of defeats the purpose. This may be a more general discussion point: shall we always use this generic "The NOMAD authors" or really the actual authors? The latter is of course helpful if you want to talk to the devs directly, but may be harder to maintain with people leaving, etc.

Can you explain why you don't like 2 separate categories? This is well-defined in GitHub. For me the author list is for acknowledgement. The maintainer list for contact. It is also mentioned at the top of the page that FAIRmat owns and maintains all these plugins. Perhaps that should be made clearer.
Usage of "The NOMAD Authors" appears to be largely used by nomad-coe plugins which are or will eventually be archived.
The issue of maintaining author lists is independent of this page. But I think it is something that we want to do, at least while FAIRmat exists, as it is a major way we can acknowledge our co-workers work, through publications of repos on Zenodo.

I think it's fine to have both categories, as long as we agree to fill these correctly. That means to me that we do not use "The NOMAD authors", but highlight the actual people who did the work and who are the maintainers. For now, the categories are pretty redundant if they just contain these generic placeholders in both categories. But okay, let's keep and make it a separate point to acknowdledge properly. I think the main issue is the inconsistent use of authors and maintainers in pyproject.toml (where often just "The NOMAD Authors" is used) and in the CITATION.cff for Zenodo, where often the proper list is given.

On a separate note, the maintainers category may be tricky because these get outdated with coworkers leaving.

We will need to add a check for archived repos and perhaps just list them concisely at the bottom of the page or something.

agree

For the list of entry points, we could give for each of them their type (e..g bandstructurenormalizer (Normalizer), then we could skip the item Plugin Types. But either way is fine.

I am fine with this but some plugins have many entry points which will blow up the text a little. I like the way Ahmed did it cause the Plugin Types are in the overview, and the detailed list only in the drop down. Also, often the type is also in the entry point name, as in the example you gave. I am open to more opinions on this though.

agree, I like Ahmed's solution

lukaspie · 2026-01-26T10:43:59Z

One more thing that I forgot (maybe it was already discussed, if so, please ignore): would it be possible to have at the top of the plugins overview the possiblity to filter by entry point? So e.g., all NORTH tools or all Parsers could be shown?

Or do we let the NOMAD plugins app handle that?

lauri-codes · 2026-01-27T07:15:14Z

I do think that having the list of plugins cleanly in very clear webpage is a good idea. To me this also feels much more natural than having a separate app for plugins. Although functionality-wise the app is great, it is very hard to justify that approach because it is too tighlty coupled to a deployment and requires quite a bit of mental overhead to understand for new people ("I'm now in a NOMAD deployment, which has a NOMAD App that showcases NOMAD plugins." vs. "I'm now reading the NOMAD docs and looking at a list of plugins").

Functionality-wise and visually this can certainly be improved (I would ask Berfin about the visuals), but I think there are more fundamental questions:

Should the list include also third-party plugins (I think so, maybe in a separate view or at least so that they can be easily also fitlered out).
Should this completely replace the plugins app (I think so)
Should this table be tied to the docs, or do we put this into the nomad-lab.eu homepage? The homepage is absolutely not tied to any installation (the docs are), and I think nomad-lab.eu is really the page for decision-makers: people who want to learn about the infrastructure.
In the deployment-specific docs we could still pull information about the plugins that are installed to that specific deployment if that is something we wish to do.

JFRudzinski · 2026-01-27T11:45:48Z

I do think that having the list of plugins cleanly in very clear webpage is a good idea. To me this also feels much more natural than having a separate app for plugins. Although functionality-wise the app is great, it is very hard to justify that approach because it is too tighlty coupled to a deployment and requires quite a bit of mental overhead to understand for new people ("I'm now in a NOMAD deployment, which has a NOMAD App that showcases NOMAD plugins." vs. "I'm now reading the NOMAD docs and looking at a list of plugins").

Functionality-wise and visually this can certainly be improved (I would ask Berfin about the visuals), but I think there are more fundamental questions:

Should the list include also third-party plugins (I think so, maybe in a separate view or at least so that they can be easily also fitlered out).

Should this completely replace the plugins app (I think so)

Should this table be tied to the docs, or do we put this into the nomad-lab.eu homepage? The homepage is absolutely not tied to any installation (the docs are), and I think nomad-lab.eu is really the page for decision-makers: people who want to learn about the infrastructure.

In the deployment-specific docs we could still pull information about the plugins that are installed to that specific deployment if that is something we wish to do.

Based on these comments, I think there are some fundamental design choices to discuss. I will arrange a dedicated meeting for this.

* collecting material first * working on the description of the tools * updates * add context and explanation for further tools * done with walking through all containers, past and present ones * initial proof-reading, needs feedback now * work in feedback from the last tf meeting, work on previously untouched footer and header sections, reduce reporting of the history behind the tools, mark for each tool what should happen to these and where that tool-specific part of the documentation should go * typos, a proof-reading round, ready for review * add connection to #186 plugin registry page * worked in comments from lukaspie * remaining reviewer comment --------- Co-authored-by: mkuehbach <markus.kuehbach@physik.hu-berlin.de>

lukaspie · 2026-02-03T17:20:42Z

I just came across this, we should probably remove it as well or link to the plugin navigation page instead: https://nomad-lab.eu/prod/v1/docs/howto/plugins/types/schema_packages.html#schema-packages-developed-by-fairmat

JFRudzinski · 2026-02-13T15:00:28Z

@DanielYang59 I merged develop into this branch, and now I get a ton of warnings when I do mkdocs serve which I guess are coming from the changes from removing the example data maybe. Have you seen this? Can you check it for me?

JFRudzinski · 2026-02-13T15:19:19Z

Current status:

JFRudzinski · 2026-02-13T15:21:44Z

TODOs:

extend plugin metadata for further filtering (archived, categories)
extend filter to cover domain-specific examples and then (if successful) remove examples section entirely

DanielYang59 · 2026-02-13T15:49:12Z

@DanielYang59 I merged develop into this branch, and now I get a ton of warnings when I do mkdocs serve which I guess are coming from the changes from removing the example data maybe. Have you seen this? Can you check it for me?

Can you show me the warnings you got? I only saw some deprecation warnings from fastapi side but was fixed in https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/2836

INFO    -  FastAPIDeprecationWarning: `example` has been deprecated, please use `examples` instead
             File "/home/yang/nomad/nomad-docs/.venv/lib/python3.12/site-packages/fastapi/params.py", line 521, in __init__
               warnings.warn(
             File "/home/yang/nomad/nomad-docs/.venv/lib/python3.12/site-packages/nomad/app/v1/models/models.py", line 380, in
               class WithQuery(BaseModel):
INFO    -  FastAPIDeprecationWarning: `example` has been deprecated, please use `examples` instead
             File "/home/yang/nomad/nomad-docs/.venv/lib/python3.12/site-packages/fastapi/params.py", line 521, in __init__
               warnings.warn(
             File "/home/yang/nomad/nomad-docs/.venv/lib/python3.12/site-packages/nomad/app/v1/models/models.py", line 1134, in
               class WithQueryAndPagination(WithQuery):
INFO    -  FastAPIDeprecationWarning: `example` has been deprecated, please use `examples` instead
             File "/home/yang/nomad/nomad-docs/.venv/lib/python3.12/site-packages/fastapi/params.py", line 521, in __init__
               warnings.warn(
             File "/home/yang/nomad/nomad-docs/.venv/lib/python3.12/site-packages/nomad/app/v1/models/models.py", line 1140, in
               class Metadata(WithQueryAndPagination):

JFRudzinski · 2026-02-13T15:55:44Z

yeah same:

INFO    -  The following pages exist in the docs directory, but are not included in
           the "nav" configuration:
             - 404.md
             - aitoolkit.md
             - todo.md
             - tutorials_template.md
             - writing_guide.md
             - examples/experiment_data/nexus.md
             - howto/manage/program/graph-api/basics.md
             - tutorial/access_api.md
             - tutorial/nomad_repo.md
INFO    -  FastAPIDeprecationWarning: `example` has been deprecated, please use
           `examples` instead
             File
           "/home/jfrudzinski/work/soft/plugins/nomad-docs/.venv/lib/python3.12/site-packages/fastapi/params.py",
           line 521, in __init__
               warnings.warn(
             File
           "/home/jfrudzinski/work/soft/plugins/nomad-docs/.venv/lib/python3.12/site-packages/nomad/app/v1/models/models.py",
           line 380, in
               class WithQuery(BaseModel):
INFO    -  FastAPIDeprecationWarning: `example` has been deprecated, please use
           `examples` instead
             File
           "/home/jfrudzinski/work/soft/plugins/nomad-docs/.venv/lib/python3.12/site-packages/fastapi/params.py",
           line 521, in __init__
               warnings.warn(
             File
           "/home/jfrudzinski/work/soft/plugins/nomad-docs/.venv/lib/python3.12/site-packages/nomad/app/v1/models/models.py",
           line 1134, in
               class WithQueryAndPagination(WithQuery):
INFO    -  FastAPIDeprecationWarning: `example` has been deprecated, please use
           `examples` instead
             File
           "/home/jfrudzinski/work/soft/plugins/nomad-docs/.venv/lib/python3.12/site-packages/fastapi/params.py",
           line 521, in __init__
               warnings.warn(
             File
           "/home/jfrudzinski/work/soft/plugins/nomad-docs/.venv/lib/python3.12/site-packages/nomad/app/v1/models/models.py",
           line 1140, in
               class Metadata(WithQueryAndPagination):

but also these files not in nav. I guess maybe I messed up the merge?

DanielYang59 · 2026-02-13T16:04:57Z

I think there're two separate warnings here:

The following pages exist in the docs directory, but are not included the "nav" configuration: for some markdowns in the docs directory but not in the nav section of mkdoc.yml
The rest FastAPIDeprecationWarning is from nomad-lab side and has been fixed, the lock files was not yet updated (i would do this later)

nomad-docs/uv.lock

Lines 1837 to 1838 in ec2ba55

name = "nomad-lab"

version = "1.4.1.dev224+gbb5ea6b9e"

You could run uv sync --upgrade and the deprecation would be gone

JFRudzinski added 4 commits November 27, 2025 18:40

initial plugin registry implementation

7b961cc

expand to nomad-coe

a0a683a

formatting with button

017fe3d

no star icons

14ec827

JFRudzinski requested a review from DanielYang59 December 1, 2025 14:29

JFRudzinski added 2 commits December 1, 2025 15:43

ruff

de9563d

fix link errors and ruff

d9272f5

JFRudzinski added 2 commits December 1, 2025 21:10

linting

b3dafa9

further linting

344b75d