Able to cache catalog

[thopri]

I need to search the whole catalog to check if there is suitable data e.g. get the highest spatial and temporal resolution data there is for the spatial and temporal extent of interest. This is easy enough but currently it takes a long time as it will always rebuild an index from the catalog which is not really that efficent/required

Is it possible to generate a local cache that is updated at a defined interval? ideally it can also be forced if the user knows there is new data.

[oj-tooth]

Hi Tom,

Thanks for raising this issue. Is this specifically related to the catalog.search() method? Or related to .open_dataset()?

Do you still experience this long search time with the latest v0.1.0 release? My experience has been that dropping the path-like representation of Collections & Sub-Catalogs has considerably improved the speed of searching the entire catalog using item_name or standard_name parameters for example.

[thopri]

Yes, the catalog.search() it is much quicker than it was. My approach with other datasources e.g. CMEMS is to generate the whole catalog (using a broad search term) and then parse it for the model ids I need. This works reasonably well as CMEMS caches its catalog. If its faster to search for each parameter, indidually I will try that.

[oj-tooth]

Do you have an example / template code for your workflow? It may be possible to first search on the Collection of interest and then filter on the STAC Items for each property of interest?

Alternatively, we may be able to add more properties into the .search() method if this is inefficient.

[thopri]

It is currently something like:

• search catalog (using "global" to only get entries with global in them)
• check if variable or alias is present
• check spatial and temporal extent
• check field type e.g. monthly means

Its a little more complicated than that but that is the jist, if everything matches then the entry is saved to download later, if say a better entry is found e.g. daily means then this the entry to download is updated.

So the catalog.search() being say 20 seconds is not a big deal, just annoying for development :)

[thopri]

Just to update this, I have added a simple caching mechanism to my library where it uses jsonpickle to serialise/encode the catalog to a json file. If this file is present my code will decode it and use it as the catalog. This was quite easy to implement (still need to add stuff like force update and time expiry) so its debatable if it should be part of this library or not.

[thopri]

Ok another update, I wanted to add an expiry function to the local catalog json file in case of updates to the server catalog, my code now checks the OceanDataCatalog.Catalog.extra_fields['last_update'] field to see if it is newer than the one stored locally, if it is then it will update it. I mention this as I would like to clarify if this field will get updated when the STAC catalog is?