As a User/Registered User, I want to Access data/datasets (Products) referenced in EarthCODE Catalogue on CDSE, so that I can view them on CDSE

[JanssenBrm]

Acceptance criteria:

• As a User/Registered User I can use the EarthCODE Catalog to discover the STAC metadata for a product and then use the reference to the product to develop a new experiment/workflow in CDSE.

[JanssenBrm]

@GarinSmith - What would be the use case or user flow for the discovery of the products on the EarthCODE catalogue? Are users expected to go to the EarthCODE catalogue to discover what is there? How is this different from the experiment result, mentioned in #10?

[JanssenBrm]

Comment from @GarinSmith

@JanssenBrm - This is a good point.
I think we have some flexibility to agree how this works as this forms part of the 3rd milestone of work.
I think this user story is more focused on the experimental result itself.
I think #10 could be more focused on the experiment itself.

As such, a done criteria for this story could be either:
Done criteria 1 : "As a User/Registered User, I can see the STAC metadata in the catalog that was published from CDSE for a product."
However, I think this could be part of the done criteria for https://github.com/orgs/ESA-EarthCODE/projects/7/views/12?pane=issue&itemId=82643597

Done criteria 2: "As a User/Registered User I can use the Catalog to discover the STAC metadata that was published from CDSE for a product and then access that product in CDSE".
Note that later on this might also be implemented in the PRR (unless we decide to do this first).

@edobrowolska and @dgiomo please comment. We can also discuss this at the next Scrum of Scrums. This is partly informed by the metadata examples that the platforms use to describe produts and experimemst/workflows.

[JanssenBrm]

I believe "Done Criteria 1" could also be covered in #9. For "Done criteria 2" we should define the scope of "assessing the product". From our side, it could mean:

As a User/Registered User I can use the Catalog to discover the STAC metadata that was published from CDSE for a product and then use the reference to the product to develop a new experiment/workflow in CDSE".

Similar to the comment at #10 (comment)

[GarinSmith]

I agree with the points above. Done criteria 2 is applicable to this user story. Done criteria 1 applicable elsewhere.
Also, this story is related to Products and so I will update it to reflect that.

[JanssenBrm]

Great, thank you for the quick feedback @GarinSmith - I have updated the main description of this ticket accordingly. Feel free to comment if any other criteria are applicable.

[JanssenBrm]

In summary, the main acceptance criteria from a platform perspective is defined as follows:

As a User/Registered User, I can reference a product in the EarthCODE Catalogue to develop a new experiment/workflow in CDSE.

Assuming the product is stored in permanent storage and accessible through the EarthCODE catalogue, as outlined in #9, the product is available as a STAC item in the catalogue. The openEO Python client tools enable users to load external STAC items using the load_stac feature (https://processes.openeo.org/#load_stac). Here are examples demonstrating how this feature can be integrated with other openEO processes to create a new workflow/experiment:

• https://github.com/Open-EO/openeo-community-examples/blob/main/python/LoadStac/load-stac-item-example.ipynb
• https://github.com/Open-EO/openeo-community-examples/blob/main/python/LoadStac/LoadBiomassSTAC.ipynb

@GarinSmith, do you think the load_stac feature meets the requirements of this ticket?

[GarinSmith]

@JanssenBrm. Yes, I think it meets the requirements of this ticket.
I think it is important to demo this to ESA to confirm this, but the expectation is that we re-use existing functionality where possible and this seems appropriate in that regard.
I think it might be good if possible to

1. Demo this using the existing storage solution
2. Then demo this using the permanent storage solution (PRR), unless this has a similar amount of effort to implement.

[JanssenBrm]

@GarinSmith - I wanted to share with you a notebook on how the load_stac works to prove that it can be used to load external datasets into openEO: https://github.com/Open-EO/openeo-community-examples/tree/e30aec1e2a19466f94720e2a2a88cabac0510c4d/python/LoadStac

However, as I was trying to create an example using data on the ESA PRR, I found the following issues:

• There is no consistency in the metadata that is provided on both collection and item level in the PRR. This means that some of the collection do not work with the load_stac as they don't have sufficient data for openEO to read the file and load it as datacube. For example, for this to work, the EO stac extension should be implemented as a minimum.
• The download paths in the PRR contain a relative reference (e.g. /d/worldsoils-soc/2018/03/01/worldsoils-soc-europe_aggr-orgc_00-020_201803-202010/europe_aggr-orgc_00-020_mean_100_201803-202010.tif) whereas the openEO load_stac expects an absolute URL to be able to download the files.
• In general almost (if not all) alll of the collections in the PRR are marked as "invalid" by the browser.

So, in general, to make this type of requirement work I believe there is some alignment needed between how we can expect the data to be made available in a consistent and clear way so that it can be used in external tooling. Without any alignment it is difficult for platforms to know what to do with the data. However, as shown in the example, the concept works if the STAC data is valid and complete.

[JanssenBrm]

As promised, here are a couple of examples

Working Example: Microsoft Planetary Computer

Collection

https://planetarycomputer.microsoft.com/api/stac/v1/collections/landsat-c2-l2

Items

https://planetarycomputer.microsoft.com/api/stac/v1/collections/landsat-c2-l2/items

{
   "assets":{
      "red":{
         "href":"https://landsateuwest.blob.core.windows.net/landsat-c2/level-2/standard/oli-tirs/2025/090/107/LC09_L2SR_090107_20250918_20250919_02_T2/LC09_L2SR_090107_20250918_20250919_02_T2_SR_B4.TIF",
         "eo:bands":[
            {
               "name":"OLI_B4",
               "center_wavelength":0.65,
               "full_width_half_max":0.04,
               "common_name":"red",
               "description":"Visible red"
            }
         ],
         "description":"Collection 2 Level-2 Red Band (SR_B4) Surface Reflectance",
         "type":"image/tiff; application=geotiff; profile=cloud-optimized",
         "roles":[
            "data",
            "reflectance"
         ],
         "title":"Red Band",
         "raster:bands":[
            {
               "scale":0.0000275,
               "nodata":0,
               "offset":-0.2,
               "data_type":"uint16",
               "spatial_resolution":30
            }
         ]
      }
      ...
   }
   ...
}

Example: ESA PRR - WorldSOILS

Collection

https://eoresults.esa.int/stac/collections/worldsoils-soc

Items

https://eoresults.esa.int/stac/collections/worldsoils-soc/items

❌ Uses local paths in assets:

"assets": {
     "mean": {
     "href": "/d/worldsoils-soc/2020/03/01/worldsoils-soc-wallonia_aggr-orgc_00-020_202003-202210/wallonia_aggr-orgc_00-020_mean_50_202003-202210.tif",
     "type": "image/tiff; application=geotiff",
     "roles": [
        "data"
     ],
    ...
}

Summary

Currently, the most identified issues are:

• Usage of relative paths makes it hard to identify the full URL and protocol (http, s3, ...) that is required to access the data in a transparent way.
• The raster extension is missing to allow openEO to accurately determine the bands of the products.
• An item_assets on collection level, describing the assets that are included also help with identifying which assets are included in the items.

[JanssenBrm]

@edobrowolska / @GarinSmith / @u3ks - FYI - Above is a working example from Microsoft's Planetary Computer and one of the examples from the PRR.