Can I register a custom CRS to a user-defined EPSG code to use with RS_SetSRID?

[samlamont]

Hello!

I am reading netcdf files from s3 and saving locally as geotiff using Sedona and PySpark. If I do not explicitly set the SRID with RS_SetSRID I get an error saying it must be defined when writing to disk. However it seems RS_SetSRID only accepts an integer EPSG code and the netcdf data are in a non-standard CRS with no EPSG code. I do know the proj4/WKT strings of the CRS.

So is there a way I can register the WKT string of the netcdf data to a user-defined EPSG code (ex. 99999) to use with the RS_SetSRID in Sedona?

The netcdf data is the National Water Model v3.0 retrospective rainfall data, ex: https://noaa-nwm-retrospective-3-0-pds.s3.amazonaws.com/index.html#CONUS/netcdf/FORCING/1979/

I'm using PySpark 4.0, Sedona 1.8.0, and the GeoTools extension (org.datasyslab:geotools-wrapper:1.8.0-33.1)

Ex. read:

raster_sdf = (
    spark
    .read
    .format("binaryFile")
    .load(s3_glob_pattern)
    .selectExpr("RS_FromNetCDF(content, 'RAINRATE', 'x', 'y') as raster", "path as filepath")
    .selectExpr("RS_SetSRID(raster, 99999) as raster", "filepath")  # <-- can I set 99999 to a custom WKT string?
    .selectExpr("RS_AsGeoTiff(raster, 'LZW', 1) as raster", "filepath")
)

Then writing to geotiff like:

raster_sdf.write.format("raster").mode("overwrite").partitionBy("value_time").save(f"{output_dir.as_posix()}")

Any tips/guidance would be greatly appreciated, thanks!

[jiayuasu]

Does each raster have its own WKT string or all rasters in the table have the same CRS?

[samlamont]

Hi @jiayuasu thanks for the reply, all rasters have the same CRS, which is available as a WKT or proj4 string

[samlamont]

Hi just wondering if anyone has any insight here? Thanks in advance!

[jiayuasu]

@samlamont thanks for reporting this.

Custom CRS support for vector geometry is already merged and will be released in the upcoming Sedona 1.9.0.

We've started to work on this support in Raster as well. I've created a ticket for this: #2674

[samlamont]

Hi @jiayuasu sounds great thanks for the feedback.

Just as some general context of our use-case, we are working on a framework (python package plus a suite of cloud services) called TEEHR built on PySpark, Sedona, and Iceberg (among many others) to enable hydrologic model evaluation, data management, and dashboards at scale.

One of the variables of interest is basin-averaged rainfall or temperature (basically zonal stats between grids and polygons but accounting for fractional pixel coverage), and we have had some success using Sedona for this; reading National Water Model data directly from S3. We also found Sedona to be very efficient at writing a netcdf variable to geotiff (which could support visualization or other analyses) which is the reason for the original question here.

This is all just to say thanks for your work! I'm excited to learn more about Sedona and its capabilities