Cloud-native storage formats

[PaulC91]

Hi Max. I just discovered your field maps project and it looks like a super useful resource!

I work at Epicentre which is an epidemiological unit of MSF, and we are often collating admin boundary data for field teams to use for patient origin data entry and mapping. Normally I take OCHA data from HDX on a country by country basis, but it's great that you have a created a single resource for global edge-matched sub-national boundaries.

I see each admin level is available to download as a GeoPackage or GeoDatabase file, but have you considered storing the data in a cloud-native format such as hive-partitioned geoparquet in s3 or other cloud storage systems? This would allow for efficient querying and downloading of subsets of data with tools such as duckdb (example) and avoid the need of having to download the entire admin layer, which can be very large.

Interested to hear your thoughts.

Thanks,
Paul

[brianmcdonald]

Hi Max. I hope you're keeping well.

I was downloading a .gpkg file and had the same thought as Paul. Serving the data as a single GeoParquet file could give an API-like experience (without the need for an API) where the user is only retrieving the data they need for their map or analysis. The user can specify their bounding box and/or variables that they want to use without needing to download all the data.

Happy to discuss

[PaulC91]

Hi @maxmalynowsky. I recently noticed you're now hosting GeoParquet versions of the admin boundaries - thanks a lot for that!

I developed a quick R package that uses a DuckDB engine to query the remote parquet files and pull different admin levels by country. I know a lot of people are using R in the humanitarian data world so hopefully it might prove useful to others. Let me know if you have any feedback... https://github.com/epicentre-msf/fieldmaps

cc: @brianmcdonald

Thanks,
Paul

[maxmalynowsky]

Hey @PaulC91 and @brianmcdonald , sorry I missed your earlier messages, and great work on the R package! I don't work with R, so I'll trust you implemented it well. I'm at UN OCHA (HDX) now, and we're also using DuckDB for some of our data science workflows, which is what prompted me to publish in GeoParquet. At the moment, I'm working on 2 big projects right now which are probably relevant to you. One is developing a pipeline to generate global admin boundaries directly from within OCHA, bringing all the experience from FieldMaps to the source (which will make this dataset much easier to keep up to date). Also, I'm working on a Geo Data API for HDX to make all datasets there accessible through GDAL programs via an API. For example, making gdal vector convert available as an API service like /api/gdal/vector/convert?input={hdx_uuid}&output=test.gpkg or gdal vector filter as /api/gdal/vector/filter?input={hdx_uuid}&output=test.gpkg&input-layer=admin2&where=iso3=SSD. Some slides showing the concepts we're working to adopt.