ScienceMesh PoC integration

[ljocha]

Copy-and-paste from Confluence Dispatcher draft page

The Dispatcher receives the RO-Crate and recognizes it as a package intended for processing on ScienceMesh. The Dispatcher utilizes an HTTP client to POST a signed OCM Share Request (signed http request) containing the embedded RO-Crate to sciencemesh.cern.ch/ocm/shares/. The ScienceMesh instance receives the OCM share request, verifies it as originating from a trusted ScienceMesh node (rasmus.oscar.welander@dispatcher.egi), confirms receipt, and lists it under the user’s “Shared with me” tab. The user accesses the “Shared with me” tab, identifies the special share type, and activates the “Execute Workflow” button. Upon activation, the ScienceMesh node resolves remote references and downloads required resources (.csv and .ipynb). The user is redirected to SWAN (JupyterLab), enabling workflow execution using HPC resources.

The RO-Crate could contain (example):

• Sharing Party: rasmus.oscar.welander@matchmaker.egi
• Receiving Party: rasmus.oscar.welander@sciencemesh.cern.ch
• ScienceMesh Address: sciencemesh.cern.ch
• Remote Resources:
  • CSV: some/remote/foo.csv
  • Notebook: some/remote/bar.ipynb

Additions to the Dispatcher:

• The Dispatcher needs to implement logic for embedding the RO-Crate in the OCM Share Request.
• it will use an HTTP Client in order to POST the OCM Share Request to sciencemesh.cern.ch/ocm/shares/.

[ljocha]

I'm a bit concerned with "originating from a trusted ScienceMesh node" -- does it mean that we will need a particular service identity of the Dispatcher to talk to talk to sciencemesh.cern.ch/ocm/shares? Or can we still stick with the user's identity, using OAuth token exchange?

Otherwise, carry on. Provide the following, please:

• an example ROCrate in test/
• a class similar to VREGalaxy (on this branch, it's going to be merged with Adding celery support for galaxy #9 to implement the logic described above

[rawe0]

I’m sure Giuseppe could explain this better, but I’ll do my best :)

In general, a "Trusted ScienceMesh Node" refers to a node that is part of the ScienceMesh federation. This typically implies the existence of a central repository that allows each node to verify whether a request originates from within the federation.

Initially, such a central repository will likely not be available. However, each ScienceMesh node is responsible for deciding which other nodes it trusts. For example, the CERN ScienceMesh node can manually configure the dispatcher as a trusted node.

Example Scenario

Suppose I (identity: rasmus.oscar.welander@dispatcher.egi.eu) find research through the MatchMaker that I want to reproduce. When I click "Process on ScienceMesh" (or a similar button), the dispatcher creates an OCM Share to rasmus.oscar.welander@cern.ch.

This share includes the RO-Crate, embedded directly into the request—effectively allowing the user to share the RO-Crate with themselves. Because the CERN ScienceMesh node trusts the dispatcher, it accepts the request.

The user at CERN then sees the request listed under incoming shares in the ScienceMesh interface. Upon accepting the request, the system begins processing the RO-Crate—resolving all references and executing the defined workflow.

Dispatcher as a ScienceMesh Node

The role of the dispatcher as a ScienceMesh node can be slightly confusing. However, in this case, it simply sends HTTP requests using its identity (e.g., someone@dispatcher.egi.eu) and does not receive any requests. This makes it relatively straightforward to implement.

The main requirement is to generate a valid OCM Share Request with the RO-Crate embedded.

Authentication Notes

• No authentication is required between ScienceMesh nodes when sharing data.
• Authentication is required when logging into:
  • The CERN ScienceMesh node
  • The MatchMaker

Dispatcher Discovery Endpoint

The dispatcher must expose static discovery metadata at a URL like:

https://dispatcher.egi.eu/.well-known/ocm

{
  "enabled": true,
  "apiVersion": "1.2.0",
  "endPoint": "https://dispatcher.egi.eu/ ",
  "resourceTypes": [
    {
      "name": "ro-crate",
      "shareTypes": [
        "user"
      ],
      "protocols": {
        "http": "embedded"
      }
    }
  ]
}

This allows the CERN ScienceMesh node to validate the origin of a request when the sender is, for example, someone@dispatcher.egi.eu.

Future considerations

Although OCM 1.2 (not yet implemented) introduces the options for signed HTTP requests, this is not a priority for the current proof of concept, but should come later.

[ljocha]

Thanks for the clarification but sorry for carrying on with ignorant questions:

For example, the CERN ScienceMesh node can manually configure the dispatcher as a trusted node.

What exactly does this mean technically? Just putting the Dispatcher hostname on some white list, or dealing with some private-public key pair like ssh?

Suppose I (identity: rasmus.oscar.welander@dispatcher.egi.eu)

I'm still confused what this identity means. Assuming we authenticate via EGI CheckIn, my identity would be 84d9a1ec23cd397eb806ca8562307cb2af7eb904506f091b6e3f37469ff6a523@egi.eu, which is just mine, not related to Dispatcher specifically.

When I click "Process on ScienceMesh" (or a similar button), the dispatcher creates an OCM Share to rasmus.oscar.welander@cern.ch.

I understand there must be a user mapping stage, which links that ugly hash above @egi.eu to some identity Ales.Krenek@cern.ch. On the other hand, doing so at Dispatcher is one of cans of worms I prefer not to open.

For PoC we can rely on the user to claim his/her ScienceMesh username, though.

Because the CERN ScienceMesh node trusts the dispatcher, it accepts the request.

Does this mean the ScienceMesh node trusts Dispatcher blindly, accepting the share for any user? I'd guess so based on your "No authentication is required between ScienceMesh nodes when sharing data."

The main requirement is to generate a valid OCM Share Request with the RO-Crate embedded.

Let's focus on this. Once I see a specific example, answers to my questions may become obvious.

The dispatcher must expose static discovery metadata at a URL like:

As long as it's static, it should be trivial to include in the config.

[glpatcern]

Thanks for the clarification but sorry for carrying on with ignorant questions:

Questions are totally welcome Ales, apologies for the late reply as I was on vacation.

For example, the CERN ScienceMesh node can manually configure the dispatcher as a trusted node.

What exactly does this mean technically? Just putting the Dispatcher hostname on some white list, or dealing with some private-public key pair like ssh?

In first approximation yes, it would be a whitelist. The point here is that the OCM is unauthenticated, and trust is established by users "off-band" and hardened via HTTP signatures (which will come at a later stage).

Suppose I (identity: rasmus.oscar.welander@dispatcher.egi.eu)

I'm still confused what this identity means. Assuming we authenticate via EGI CheckIn, my identity would be 84d9a1ec23cd397eb806ca8562307cb2af7eb904506f091b6e3f37469ff6a523@egi.eu, which is just mine, not related to Dispatcher specifically.

OK, that's the one that would be included in the sender field on the OCM payload.

When I click "Process on ScienceMesh" (or a similar button), the dispatcher creates an OCM Share to rasmus.oscar.welander@cern.ch.

I understand there must be a user mapping stage, which links that ugly hash above @egi.eu to some identity Ales.Krenek@cern.ch. On the other hand, doing so at Dispatcher is one of cans of worms I prefer not to open.

To be exact, the OCM share is created to rasmus@cernbox.cern.ch, and if I can say that's the "magic" of OCM: the share is (anonymously) posted to https://cernbox.cern.ch/ocm/shares, i.e. <recipient server>/ocm/shares. Now, the receiver has the option to discard such a offered share that potentially might come from "nobody@nowhere" (to also address your next question).

Then, OCM includes mechanisms to discover recipients and improve trust via what we call the Invitation flow, and servers may implement policies to only accept shares from trusted parties. We can expand on that in some upcoming meeting.

The main requirement is to generate a valid OCM Share Request with the RO-Crate embedded.

Let's focus on this. Once I see a specific example, answers to my questions may become obvious.

Yes, we will be working on this in the coming weeks and hopefully clarify ourselves if the approach is OK.

The dispatcher must expose static discovery metadata at a URL like:

As long as it's static, it should be trivial to include in the config.

The only non-static part would be the endPoint, which should really be https://dispatcher.egi.eu/ in the example above.