Merge pull request #18 from FAIRmat-NFDI/update-to-nuw-0.2.0

custom entry to ELNFileManager, custom workflow update for v0.2.0

Author: JFRudzinski

docs/api.md

@@ -436,7 +436,7 @@ Let's also add the drag and drop upload with the MD data to our dataset. First,
 with open(os.path.join('<path to PIDs>', 'PIDs.json')) as f:
     pids_dict = json.load(f)
 
-md_upload_id = pids_dict.get('upload_ids').get('md')
+md_upload_id = pids_dict.get('upload_ids').get('md-workflow')
 metadata_new = {'upload_name': f'Test Upload - MD-equilibration', 'datasets': dataset_id}
 edit_upload_metadata(md_upload_id, url='test', upload_metadata=metadata_new)
 ```

docs/assets/Custom_ELN_Entries.zip

@@ -101,7 +101,7 @@ This example data has been pre-uploaded and published on NOMAD. Go to the [examp
 
 Create a workspace folder for this tutorial, e.g., `workspace_nomad_tutorial_workflows/`, and then move the downloaded zip to this folder. We suggest also creating sub-folders `Part-1`-`Part-4` for organizational purposes.
 
-Now go to the [Test NOMAD Deployment](https://nomad-lab.eu/prod/v1/test/gui/search/entries){:target="\_blank"}.
+**Now go to the [Test NOMAD Deployment](https://nomad-lab.eu/prod/v1/test/gui/search/entries){:target="\_blank"}.**
 
 !!! warning "Attention"
 
@@ -147,6 +147,10 @@ We will need both the `upload_id` and the `entry_id` for this entry later. Copy
 }
 ```
 
+!!! warning "Attention"
+
+    The JSON format requires double quotations!
+
 Production Simulation:
 
 <video width="100%" controls>

docs/assets/ELNFileManager.webm

@@ -3,9 +3,9 @@
 ## 🎯 What You Will Learn
 
 - How to store data that is **not supported by existing NOMAD parsers**
-- How to define a custom **YAML schema**
 - How to use the **NOMAD ELN (Electronic Lab Notebook)** interface
-- How to connect these custom entries into a workflow graph
+- How to annotate and reference files as part of a workflow
+- How to connect entries into a workflow graph
 
 ---
 
@@ -79,87 +79,36 @@ Uploading this yaml to the test deployment results in an entry with the overview
     </label>
 </div>
 
-## Customizing the schema
-
-To document our simulation setup workflow, we need to reference files within our ELN entry. For standardization and search capabilities, it is best practice to use existing classes in the MetaInfo. However, NOMAD also allows users to customize the schema to their own specific needs. Let's create our own schema to store annotated files within an ELN.
-
-!!! warning "Attention"
-    Understanding the details of the customized schema below is beyond the scope of this tutorial, and is not necessarily the adviced route for the most robust customization. The important take away is that you can in principal create you own schema. In this case we have done so to enable storage of annotated files.
-
-Create a file `ELNFiles.archive.yaml` with the following contents:
-
-<h4><code>ELNFiles.archive.yaml</code></h4>
-```yaml
-definitions: # Use the defintions section to create your schema
-  name: 'ELN-Annotated-File-List'
-  sections:
-    AnnotatedFile: # A subsection for storing an annotated file
-      m_annotations:
-        eln:
-          overview: True # Displays this quantity in the overview page of the entry
-      quantities:
-        file: # a quantity for storing the actualy file reference
-          type: str
-          description: single workflow files
-          m_annotations:
-            browser:
-              adaptor: RawFileAdaptor  # Allows to navigate to files in the data browser
-            eln:
-              component: FileEditQuantity # Allows editing with the GUI
-        description: # a quantity for storing the annotation
-          type: str
-          description: describe the file
-          m_annotations:
-            eln:
-              component: StringEditQuantity # Allows editing within the GUI
-    ELNAnnotatedFiles: # Define a subsection for storing files
-      base_sections:
-      - 'nomad.datamodel.metainfo.eln.ElnBaseSection' # inherits from the basic ELN class
-      - 'nomad.datamodel.data.EntryData' # necessary when a class will be the root of our archive
-      m_annotations:
-        eln:
-          hide: ['lab_id'] # hides the lab_id quantity that we will not use
-      sub_sections:
-        Files:
-          repeats: True # makes the subsection repeating (i.e., a list)
-          section: '#/AnnotatedFile' # this subsection will include the quantities defined within the `AnnotedFile` class defined above
-```
-
-The section `AnnotationFile` contains 2 quantities `file` and `description` for storing a file reference and annotation, respectively. The section `ELNAnnotatedFiles` extends the most basic ELN implementation (`ElnBaseSection`) with a repeating subsection of type `AnnotatedFile`. In this way, our ELN will be able to store a list of annotated files.
-
-??? tip "More on custom schemas"
-
-    The YAML approach is a quick and dirty way to customize your NOMAD entries.
-    To disincentivise the proliferation of _ad-hoc_ schemas and remain FAIR, YAML sections or quantities only have partial support.
-    Seamless integration of new quantities happens when schemas are organized in **plugins** (python packages) and installed in NOMAD during deployment.
-    See [NOMAD Docs > How to write a YAML schema package](https://nomad-lab.eu/prod/v1/test/docs/howto/customization/basics.html){:target="_blank"} for more details about defining custom schemas in this way.
+## The `ELNFileManager`
 
-    The more robust and powerful approach for creating custom schemas is to create a *schema plugin* (see [NOMAD Docs > How to get started with plugins](https://nomad-lab.eu/prod/v1/test/docs/howto/plugins/plugins.html){:target="_blank"}).
+`ELNFileManager` is a built-in schema for referencing and annotating files within an ELN entry. You can create an `ELNFileManager` either from the GUI or via the YAML approach, in the same ways as described above.
 
-    Useful resources for plugin developers are the [Plugin Template](https://github.com/FAIRmat-NFDI/nomad-plugin-template) and the [NOMAD Distro Template](https://github.com/FAIRmat-NFDI/nomad-distro-template).
-
-We can now use these definitions to create an entry file for the step of creating the force field file (as illustrated in the image above):
+We can now use these definitions to create an entry file for the step of creating the force field, and link to the output force field file from this step in the workflow:
 
 <h4><code>create_force_field.archive.yaml</code></h4>
 ```yaml
 data:
-  m_def: '../upload/raw/Custom_ELN_Entries/ELNFiles.archive.yaml#ELNAnnotatedFiles'
+  m_def: 'nomad.datamodel.metainfo.eln.ElnFileManager'
   name: 'Create force field'
   description: 'The force field is defined for input to the MD simulation engine.'
   Files:
   - file: 'Custom_ELN_Entries/water.top'
     description: 'The force field file for simulation input.'
 ```
 
-Here we define the data section using our `ELNFiles.archive.yaml` schema. The given path is a relative path assuming that we will upload these 2 files (i.e., `ELNFiles.archive.yaml` and `create_force_field.archive.yaml`) within the same upload with a root folder called `Custom_ELN_Entries`.
+Uploading to NOMAD should result in the following entry display:
+
+<video width="100%" controls>
+  <source src="../assets/ELNFileManager.webm" alt="" type="video/mp4">
+</video>
 
 You can now create analogous files `create_box.archive.yaml`, `insert_water.archive.yaml`, `workflow_parameters.archive.yaml`, `workflow_scripts.archive.yaml`:
 
 ??? success "`create_box.archive.yaml`"
 
     ```yaml
     data:
-      m_def: '../upload/raw/Custom_ELN_Entries/ELNFiles.archive.yaml#ELNAnnotatedFiles'
+      m_def: 'nomad.datamodel.metainfo.eln.ElnFileManager'
       name: 'Create box'
       description: 'The initial simulation box is created.'
       Files:
@@ -171,7 +120,7 @@ You can now create analogous files `create_box.archive.yaml`, `insert_water.arch
 
     ```yaml
     data:
-      m_def: '../upload/raw/Custom_ELN_Entries/ELNFiles.archive.yaml#ELNAnnotatedFiles'
+      m_def: 'nomad.datamodel.metainfo.eln.ElnFileManager'
       name: 'Insert water'
       description: 'Water is inserted into the simulation box, creating the structure file for simulation input.'
       Files:
@@ -192,7 +141,7 @@ You can now create analogous files `create_box.archive.yaml`, `insert_water.arch
 
     ```yaml
     data:
-      m_def: '../upload/raw/Custom_ELN_Entries/ELNFiles.archive.yaml#ELNAnnotatedFiles'
+      m_def: 'nomad.datamodel.metainfo.eln.ElnFileManager'
       name: 'Workflow Scripts'
       description: 'All the scripts run during setup of the MD simulation.'
       Files:
@@ -202,9 +151,22 @@ You can now create analogous files `create_box.archive.yaml`, `insert_water.arch
         description: 'Creates the appropriate force field files for the simulation engine.'
     ```
 
+## Customizing the schema
+
+For standardization and search capabilities, it is best practice to use existing classes in the MetaInfo. However, NOMAD also allows users to customize the schema to their own specific needs. For example, you can customize one of the existing ELN schemas by adding specific sections and quantities relevant for your use case.
+
+Customization of schemas is beyond the scope of the present tutorial, however, the tip box below will direct you to further resources if you are interested in learning more.
+
+??? tip "Customizing the schema"
+    Schema customization can be achieved via 2 approaches:
+
+    1. By including a `YAML`-based schema alongside your `YAML` entry file: This is a quick and dirty way to customize your NOMAD entries, and only provides partial support in terms of, e.g., search capabilities. See [NOMAD Docs > How to write a YAML schema package](https://nomad-lab.eu/prod/v1/test/docs/howto/customization/basics.html){:target="_blank"}.
+
+    2. By creating a *schema plugin*: This more robust and powerful approach integrates your new schema into NOMAD via python class definitions See [NOMAD Docs > How to get started with plugins](https://nomad-lab.eu/prod/v1/test/docs/howto/plugins/plugins.html){:target="_blank"}. The following tools are available to assist in plugin creation and development: [Plugin Template](https://github.com/FAIRmat-NFDI/nomad-plugin-template); [NOMAD Distro Template](https://github.com/FAIRmat-NFDI/nomad-distro-template).
+
 ## Creating a custom workflow in NOMAD
 
-NOMAD allows users to connect entries into a workflow, i.e., a directed graph structure. This is achieved using the same parsing functionality as demonstrated with the custom schemas above. In this case, we simply populate the `workflow2` section instead of the `data` section. When uploaded to NOMAD, a new _workflow_ entry will be created, with references to each of the workflow tasks, and also an interactive workflow graph for easy navigation of the entire workflow. Learn more about the [archive file structure](https://nomad-lab.eu/prod/v1/test/docs/explanation/data.html#archive-files-a-shared-entry-structure) in the official NOMAD documentation.
+NOMAD allows users to connect entries into a workflow, i.e., a directed graph structure. This is achieved using the same parsing functionality as demonstrated with the custom schemas above. In this case, we simply populate the `workflow2` section instead of the `data` section. When uploaded to NOMAD, a new _workflow_ entry will be created, with references to each of the workflow tasks, and also an interactive workflow graph for easy navigation of the entire workflow.
 
 Let's construct this workflow yaml piece by piece, starting with the section definition and global inputs/outputs:
 

docs/assets/Generate_Workflow_Yaml.ipynb

@@ -107,7 +107,7 @@ pip install uv
 Install the latest pypi version of the plugin using pip:
 
 ```bash
-uv pip install "nomad-utility-workflows[vis]>=0.1.0"
+uv pip install "nomad-utility-workflows[vis]>=0.2.0"
 ```
 
 Install `python-dotenv` package.

docs/core.md

@@ -186,7 +186,7 @@ import networkx as nx
 from nomad_utility_workflows.utils.workflows import (
     NodeAttributes,
     NodeAttributesUniverse,
-    build_nomad_workflow,
+    NomadWorkflow
     nodes_to_graph,
 )
 ```
@@ -316,34 +316,21 @@ gv.d3(
     </label>
 </div>
 
-## Generate the workflow yaml
+## Filling the workflow graph with default connections
 
-First designate the full path and filename for your YAML and a name for your workflow:
+Before creating the workflow YAML, `nomad-utility-workflows` will fill the workflow graph generated in the previous step with certain "default" connections (i.e., inputs and outputs) based on the types of nodes in the workflow. This happens automatically upon instantiation of the `NomadWorkflow` class:
 
 ```python
-workflow_metadata = {
-    'destination_filename': './project_workflow.archive.yaml',
-    'workflow_name': 'NOMAD Tutorial Workflows Project',
-}
-```
-
-The workflow name will show up on top of the workflow graph visualization on the overview page of the workflow entry that we will create.
-
-We can now use the generated graph to create the workflow YAML using the `build_nomad_workflow()` function:
-
-```
-workflow_graph_output = build_nomad_workflow(
-    workflow_metadata=workflow_metadata,
-    workflow_graph=nx.DiGraph(workflow_graph_input),
-    write_to_yaml=True,
+nomad_workflow = NomadWorkflow(
+    workflow_graph=workflow_graph_input,
 )
 ```
 
-Another workflow graph is returned by this function:
+Here, we create a `NomadWorkflow` using the workflow graph generated in the previous step. The resulting "filled" workflow graph can be visualized with:
 
-```python
+```javascript
 gv.d3(
-    workflow_graph_output,
+    nomad_workflow.workflow_graph,
     node_label_data_source='name',
     edge_label_data_source='name',
     zoom_factor=1.5,
@@ -358,7 +345,19 @@ gv.d3(
     </label>
 </div>
 
-We see that our output graph looks signficantly different than the input. That's because `nomad-utility-workflow` is automatically adding some default inputs/outputs to ensure the proper node connections within the workflow visualizer. For nodes without an `entry_type`, these default connections work by simply adding inputs/outputs that point to the mainfile of one of the nodes connected by an edge in the graph.
+We see that our output graph looks signficantly different than the input. The additional inputs/outputs ensure the proper node connections within NOMAD's workflow visualizer. For nodes without an `entry_type`, these default connections work by simply adding inputs/outputs that point to the mainfile of one of the nodes connected by an edge in the graph.
+
+## Generating the workflow yaml
+
+Finally, we can add some workflow metadata to the [`NomadWorkflow`](../reference/workflows.md#nomadworkflow) (i.e., the filename of the output yaml and the name of the workflow) and generate the workflow YAML file with the class method [`build_workflow_yaml()`](../reference/workflows.md#build_workflow_yaml):
+
+```python
+nomad_workflow.destination_filename = './project_workflow.archive.yaml'
+nomad_workflow.name = 'NOMAD Tutorial Workflows Project'
+nomad_workflow.build_workflow_yaml()
+```
+
+The workflow name will show up on top of the workflow graph visualization on the overview page of the workflow entry that we will create.
 
 ??? success "example `project_workflow.archive.yaml`"