WIP implementation for patch level world cover embeddings

[lillythomas]

Relates to https://github.com/orgs/Clay-foundation/discussions/291

[lauracchen]

Hi @lillythomas ! Thanks so much for this. Would you also be able to update the worldcover script to include the option to generate embeddings with all bands included (not just partial inputs)? The app team is wondering if generating with only a couple bands is having an impact on performance.

cc @antymoro @lukaszknasiecki

[lillythomas]

Hi @lauracchen sure thing!

[lillythomas]

Actually @lauracchen it appears the Worldcover composite we are using is just RGB-Nir (https://esa-worldcover.org/en/data-access). There is a SWIR composite provided as well, which we can stack on the RGB-Nir, but this still misses out on 4/10 Sentinel-2 inputs. We can similarly stack the Sentinel-1 composite as well.

[lillythomas]

Patch-level sim search with the currently-in-place RGB-Nir worldcover model is working, albeit the results are still a bit random:

Example 1:

Example 2:

Working on the adjustments per #231 (comment)

[antymoro]

hi @lillythomas

if we stack these 4 files, I think we might cover for all sentinel-2 bounds... correct?

https://esa-worldcover-s2.s3.amazonaws.com/swir/2020/N55/ESA_WorldCover_10m_2020_v100_N55E014_SWIR.tif
https://esa-worldcover-s2.s3.amazonaws.com/ndvi/2020/N55/ESA_WorldCover_10m_2020_v100_N55E014_NDVI.tif
https://esa-worldcover-s1.s3.amazonaws.com/gamma0/2020/N55/ESA_WorldCover_10m_2020_v100_N55E014_GAMMA0.tif
https://esa-worldcover-s1.s3.amazonaws.com/vvvhratio/2020/N55/ESA_WorldCover_10m_2020_v100_N55E014_S1VVVHratio.tif

[lillythomas]

@antymoro I've implemented the concatenation of the RGB-Nir composite with the SWIR and SAR composites (see the new script run_msi_sar.py). Feel free to try it on your end.

[yellowcap]

We are updating a lot of the examples we have to run with v1. So I think this would make sense as well to be updated to v1. But not sure if it's worth the additional effort now that we have seen the limitations of patch level embeddings, and with the flexibility in terms of image size for v1, where smaller embeddings can be generated without relying on patches.