RFE: module/library for resolving an RO-Crate

[glpatcern]

Following discussions in WP6 and within CERN, as we better define the PoC for a ScienceMesh VRE, we believe it would make sense to factor out a library/module that resolves an RO-Crate and downloads the referenced data sets (and apologies if this is "duplicate" and was brought up in some other meeting).

The idea is to give as input to this module:

• The RO-Crate
  • Including any eventual authN credentials to access the referenced datasets
• An HTTP WebDAV endpoint with a bearer token that allows to write to that endpoint (via HTTP PUT, COPY etc)

The module would (recursively) download the datasets referenced in the RO-Crate, and push them to the given WebDAV endpoint. A possible enhancement would be to execute third-party HTTP COPY as opposed to download + write. Ideally, this effort for Data Access is beneficial for the entire EOSC-DC project.

The way we think to "consume" it for the ScienceMesh VRE is as per this Sequence Diagram:

sequenceDiagram
participant User
participant Dispatcher
participant ScienceMeshVRE
participant RO-Crate-Resolver
participant Remote as Remote repository

User->>Dispatcher:Run in ScienceMesh
Note right of User: Via MatchMaker GUI
Dispatcher->>ScienceMeshVRE: POST /ocm/shares (RO-Crate)
ScienceMeshVRE->>ScienceMeshVRE: Parse OCM share
ScienceMeshVRE->>Dispatcher: GET /.well-known/ocm for validation
Dispatcher-->>ScienceMeshVRE: OCM discovery payload
ScienceMeshVRE->>ScienceMeshVRE: store RO-Crate for the given user recipient
ScienceMeshVRE-->>Dispatcher: HTTP 201
Dispatcher-->>User: OK, redirect to ScienceMesh node of choice
User->>ScienceMeshVRE: Auth, Resolve and Download to /cephfs/my/path
ScienceMeshVRE->>ScienceMeshVRE: Fetch RO-Crate, mint token for /cephfs/my/path
ScienceMeshVRE->>RO-Crate-Resolver: Download (RO-Crate, token)
RO-Crate-Resolver-->>ScienceMeshVRE: ACK
ScienceMeshVRE->>User: OK, data will be made available asynchronously
loop For each element in the given RO-Crate
  RO-Crate-Resolver->>Remote: HTTP GET (auth)
  Remote-->>RO-Crate-Resolver: content
  RO-Crate-Resolver->>ScienceMeshVRE: HTTP PUT (token)
end
RO-Crate-Resolver->>ScienceMeshVRE: Download complete
ScienceMeshVRE->>User: Notify

Loading

@ljocha please comment if this is already being considered within WP6 or in other WPs.

[glpatcern]

Let me add that in the PoC targeting November we plan to use a trivial RO-Crate with a publicly available referenced dataset, which we can download "by hand" in the ScienceMesh node backend itself.

But this could/should serve as a first spec basis for this Data Access component to be integrated at a later stage.