Add tutorial for creating embeddings over AOI

[yellowcap]

Show how to use the clay torchscript encoder to create embeddings over AOI.

[srmsoumya]

This is so lean. Few minor suggestions but please don't hesitate to merge without them, I am just nitpicking here.

• Link missing in the first cell, line number 6
• The compiled version of Clay encoder is built using torch compile & not torchscript.
• May be a few lines to explain why we need to normalize the timesteps & latlon

[rbavery]

very cool stuff! at this line

datacube = (
    torch.tensor(pixels, dtype=torch.float32, device="cuda"),
    torch.tensor(np.hstack((week_norm, hour_norm)), dtype=torch.float32, device="cuda"),
    torch.tensor(np.hstack((lat_norm, lon_norm)), dtype=torch.float32, device="cuda"),
    torch.tensor(waves, dtype=torch.float32, device="cuda"),
    torch.tensor(gsd, dtype=torch.float32, device="cuda"),
)

I get

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[/home/rave/clay-model/docs/tutorials/create-embeddings-over-aoi.ipynb](https://vscode-remote+ssh-002dremote-002b10-002e0-002e0-002e156.vscode-resource.vscode-cdn.net/home/rave/clay-model/docs/tutorials/create-embeddings-over-aoi.ipynb) Cell 14 line 2
      [1](vscode-notebook-cell://ssh-remote%2B10.0.0.156/home/rave/clay-model/docs/tutorials/create-embeddings-over-aoi.ipynb#X16sdnNjb2RlLXJlbW90ZQ%3D%3D?line=0) datacube = (
----> [2](vscode-notebook-cell://ssh-remote%2B10.0.0.156/home/rave/clay-model/docs/tutorials/create-embeddings-over-aoi.ipynb#X16sdnNjb2RlLXJlbW90ZQ%3D%3D?line=1)     torch.tensor(pixels, dtype=torch.float32, device="cuda"),
      [3](vscode-notebook-cell://ssh-remote%2B10.0.0.156/home/rave/clay-model/docs/tutorials/create-embeddings-over-aoi.ipynb#X16sdnNjb2RlLXJlbW90ZQ%3D%3D?line=2)     torch.tensor(np.hstack((week_norm, hour_norm)), dtype=torch.float32, device="cuda"),
      [4](vscode-notebook-cell://ssh-remote%2B10.0.0.156/home/rave/clay-model/docs/tutorials/create-embeddings-over-aoi.ipynb#X16sdnNjb2RlLXJlbW90ZQ%3D%3D?line=3)     torch.tensor(np.hstack((lat_norm, lon_norm)), dtype=torch.float32, device="cuda"),
      [5](vscode-notebook-cell://ssh-remote%2B10.0.0.156/home/rave/clay-model/docs/tutorials/create-embeddings-over-aoi.ipynb#X16sdnNjb2RlLXJlbW90ZQ%3D%3D?line=4)     torch.tensor(waves, dtype=torch.float32, device="cuda"),
      [6](vscode-notebook-cell://ssh-remote%2B10.0.0.156/home/rave/clay-model/docs/tutorials/create-embeddings-over-aoi.ipynb#X16sdnNjb2RlLXJlbW90ZQ%3D%3D?line=5)     torch.tensor(gsd, dtype=torch.float32, device="cuda"),
      [7](vscode-notebook-cell://ssh-remote%2B10.0.0.156/home/rave/clay-model/docs/tutorials/create-embeddings-over-aoi.ipynb#X16sdnNjb2RlLXJlbW90ZQ%3D%3D?line=6) )

TypeError: can't convert np.ndarray of type numpy.uint16. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.

after running the notebook end to end after installing the environment from the lockfile. Once I install pytorch 2.3 this error doesn't happen because I think they recently introduced uint16 conversion from numpy. So maybe the install instructions and environment could be udpated to 2.3 (or 2.4 which was released today).

I also get an authentication error when fetching the chips if my AWS SSO token expires. Even though no auth is needed to fetch the chips, I think no sign request needs to be explicitly set for this to work in all cases.

import os
os.environ['AWS_NO_SIGN_REQUEST'] = 'YES'

[rbavery]

also

!wget https://huggingface.co/made-with-clay/Clay/resolve/main/clay-v1-encoder.pt

--2024-07-26 16:18:36--  https://huggingface.co/made-with-clay/Clay/resolve/main/clay-v1-encoder.pt
Resolving huggingface.co (huggingface.co)... 2600:9000:234c:8e00:17:b174:6d00:93a1, 2600:9000:234c:6000:17:b174:6d00:93a1, 2600:9000:234c:3400:17:b174:6d00:93a1, ...
Connecting to huggingface.co (huggingface.co)|2600:9000:234c:8e00:17:b174:6d00:93a1|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2024-07-26 16:18:37 ERROR 404: Not Found.

I noticed that there's now a torch compiled model on HF. I've loaded the .so directly before but for .pt2 I'm not sure how it was packaged or how to load these. looks like a recent solution was merged to pytorch: https://github.com/pytorch/pytorch/pull/129895/files#diff-9f4fab52b6c89a91bfe60aa3eab8f8c40ba88bc27394449e9735b2659a409dc9R8

[4242psherman4242]

Hey @yellowcap, I'm getting:

`---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[19], line 147
137 pixels = transform(pixels)
139 datacube = (
140 torch.tensor(pixels, dtype=torch.float32, device="cuda"),
141 torch.tensor(np.hstack((week_norm, hour_norm)), dtype=torch.float32, device="cuda"),
(...)
144 torch.tensor(gsd, dtype=torch.float32, device="cuda"),
145 )
--> 147 clay_encoder = torch.export.load("clay-v1-encoder.pt").module()
149 # Run the clay encoder
150 with torch.no_grad():

ValueError: invalid literal for int() with base 10: b'5.1'`

when running the notebook. I tried changing to torch.load("clay-v1-encoder.pt").module(), but that didn't seem to help either. Any ideas?

[srmsoumya]

@4242psherman4242 which pytorch version are you using to run the notebook?

[4242psherman4242]

@4242psherman4242 which pytorch version are you using to run the notebook?

Hey @srmsoumya this is using version 2.1.2 of pytorch

[srmsoumya]

@4242psherman4242 Can you bump the pytorch version to ~2.3? torch.export is still under development & has breaking changes in minor versions.

[rbavery]

very cool stuff! at this line

datacube = (
    torch.tensor(pixels, dtype=torch.float32, device="cuda"),
    torch.tensor(np.hstack((week_norm, hour_norm)), dtype=torch.float32, device="cuda"),
    torch.tensor(np.hstack((lat_norm, lon_norm)), dtype=torch.float32, device="cuda"),
    torch.tensor(waves, dtype=torch.float32, device="cuda"),
    torch.tensor(gsd, dtype=torch.float32, device="cuda"),
)

I get

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[/home/rave/clay-model/docs/tutorials/create-embeddings-over-aoi.ipynb](https://vscode-remote+ssh-002dremote-002b10-002e0-002e0-002e156.vscode-resource.vscode-cdn.net/home/rave/clay-model/docs/tutorials/create-embeddings-over-aoi.ipynb) Cell 14 line 2
      [1](vscode-notebook-cell://ssh-remote%2B10.0.0.156/home/rave/clay-model/docs/tutorials/create-embeddings-over-aoi.ipynb#X16sdnNjb2RlLXJlbW90ZQ%3D%3D?line=0) datacube = (
----> [2](vscode-notebook-cell://ssh-remote%2B10.0.0.156/home/rave/clay-model/docs/tutorials/create-embeddings-over-aoi.ipynb#X16sdnNjb2RlLXJlbW90ZQ%3D%3D?line=1)     torch.tensor(pixels, dtype=torch.float32, device="cuda"),
      [3](vscode-notebook-cell://ssh-remote%2B10.0.0.156/home/rave/clay-model/docs/tutorials/create-embeddings-over-aoi.ipynb#X16sdnNjb2RlLXJlbW90ZQ%3D%3D?line=2)     torch.tensor(np.hstack((week_norm, hour_norm)), dtype=torch.float32, device="cuda"),
      [4](vscode-notebook-cell://ssh-remote%2B10.0.0.156/home/rave/clay-model/docs/tutorials/create-embeddings-over-aoi.ipynb#X16sdnNjb2RlLXJlbW90ZQ%3D%3D?line=3)     torch.tensor(np.hstack((lat_norm, lon_norm)), dtype=torch.float32, device="cuda"),
      [5](vscode-notebook-cell://ssh-remote%2B10.0.0.156/home/rave/clay-model/docs/tutorials/create-embeddings-over-aoi.ipynb#X16sdnNjb2RlLXJlbW90ZQ%3D%3D?line=4)     torch.tensor(waves, dtype=torch.float32, device="cuda"),
      [6](vscode-notebook-cell://ssh-remote%2B10.0.0.156/home/rave/clay-model/docs/tutorials/create-embeddings-over-aoi.ipynb#X16sdnNjb2RlLXJlbW90ZQ%3D%3D?line=5)     torch.tensor(gsd, dtype=torch.float32, device="cuda"),
      [7](vscode-notebook-cell://ssh-remote%2B10.0.0.156/home/rave/clay-model/docs/tutorials/create-embeddings-over-aoi.ipynb#X16sdnNjb2RlLXJlbW90ZQ%3D%3D?line=6) )

TypeError: can't convert np.ndarray of type numpy.uint16. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.

after running the notebook end to end after installing the environment from the lockfile. Once I install pytorch 2.3 this error doesn't happen because I think they recently introduced uint16 conversion from numpy. So maybe the install instructions and environment could be udpated to 2.3 (or 2.4 which was released today).

I also get an authentication error when fetching the chips if my AWS SSO token expires. Even though no auth is needed to fetch the chips, I think no sign request needs to be explicitly set for this to work in all cases.

import os
os.environ['AWS_NO_SIGN_REQUEST'] = 'YES'

[lauracchen]

Hi @yellowcap is it possible to resolve the aforementioned conflicts soon? Would be great to get this super clean way of running inference merged