Missing climate models and ensemble members

[jirvin16]

Thank you for putting together an amazing dataset for the AI+climate community!

It looks like the dataset hosted on huggingface is missing several files. It only seems to have 21 climate models (rather than 36 stated in the paper) and from the included climate models, several ensemble members seem to be missing (e.g. CAMS-CSM1-0 only has 1 but the paper states it has 2). I believe several scenarios are missing as well.

Would it be possible to upload the missing data, or was their exclusion intentional?

Thanks again.

[liellnima]

Hi Jeremy,

I am happy you find ClimateSet helpful!

Yes, the dataset is indeed missing several files (and having some issues here and there still). To separate the issues:

• Climate Models: We have only included 21 climate models because we run into some issues with the remaining ones that we need to track down. Of the 21 climate models I am only recommending using the following 15 ones (since our data loader had issues in the past with the other ones): AWI-CM-1-1-MR, BCC-CSM2-MR, CAS-ESM2-0, CNRM-CM6-1-HR, EC-Earth3, EC-Earth3-Veg-LR, FGOALS-f3-L, GFDL-ESM4, INM-CM4-8, INM-CM5-0, MPI-ESM1-2-HR, MRI-ESM2-0, NorESM2-LM, NorESM2-MM, TaiESM1.
• Ensemble members: Right now, we are providing only 1 ensemble member per climate model in the core dataset, since we want to make sure that one climate model is not overrepresented in the data. However, we want to add e.g. the 97 ensemble members of the EC-Earth3-Veg model, so it can be used to assess intra-model variability.

In summary: The exclusion is intentional, however, we would like to add the missing data.

We are currently working on re-doing the whole ClimateSet pipeline and hope to be able to provide a ClimateSet python package that includes the full dataset and a smooth pipeline by the end of this year (2024). Unfortunately, the folks working on this (including me) are doing this as a side thing and have all other main tasks / research projects keeping us occupied.

I think that our new approach will help us to have a cleaner setup / dataset and track down the issues of the currently missing datasets :)

If you want to contribute and accelerate things, please let me know - I am super happy to include anyone who has time for this :)