rgee has problems with 64-bit integers

[spittssj]

• rgee version: 1.0.9
• R version: 4.0.3
• Operating System: MacOS X 11.2.3

At submit an issue, please attached the following information of your rgee session:

• You have the Python API installed (from terminal):

earthengine -h

• You can find the credentials file on your system:

library(rgee)
ee_path <- path.expand("~/.config/earthengine/credentials")
file.exists(ee_path)

• You can run a simple EE command from R:

library(rgee)

# Initialize the Earth Engine module.
ee_Initialize()

# Print metadata for a DEM dataset.
print(ee$Image('USGS/SRTMGL1_003')$getInfo())

Attach your Python (reticulate) configuration:

library(reticulate)
py_config()

python:         /Users/stephenpitts/miniconda3/envs/r_geospatial/bin/python
libpython:      /Users/stephenpitts/miniconda3/envs/r_geospatial/lib/libpython3.9.dylib
pythonhome:     /Users/stephenpitts/miniconda3/envs/r_geospatial:/Users/stephenpitts/miniconda3/envs/r_geospatial
version:        3.9.2 | packaged by conda-forge | (default, Feb 21 2021, 05:02:20)  [Clang 11.0.1 ]
numpy:          /Users/stephenpitts/miniconda3/envs/r_geospatial/lib/python3.9/site-packages/numpy
numpy_version:  1.20.2

Description

I am downloading NDVI time series LandSat data and the time field does not come through correctly. On
investigation, it is a 64-bit unsigned integer in Python that gets converted to a 32-bit signed integer in R.
According to rstudio/reticulate#590 this is a known problem in reticulate.

What I Did

parcelas <- ee$FeatureCollection("users/spittssj/parcelas")
allParcelas = parcelas$union()$first()

LandSatNDVI <- (ee$ImageCollection("LANDSAT/LC08/C01/T1_8DAY_NDVI")
                $filterDate("2013-01-01", "2021-3-31")
                $filterBounds(allParcelas$geometry())
                $select("NDVI"))

p1_NDVI <- LandSatNDVI$getRegion(allParcelas$geometry(), 30)$getInfo()

Later on, once I have converted p1_NDVI from a list of lists to a tibble, here is the output

> p1_ndvi_tibble
# A tibble: 103,812 x 6
   id       longitude latitude        time  NDVI date      
   <chr>        <dbl>    <dbl>       <int> <dbl> <date>    
 1 20130415     -92.3     17.0   184399872 0.467 2013-04-15
 2 20130501     -92.3     17.0  1566799872 0.619 2013-05-01
 3 20130517     -92.3     17.0 -1345767424 0.629 2013-05-17
 4 20130618     -92.3     17.0  1419032576 0.656 2013-06-18
 5 20130720     -92.3     17.0  -111134720 0.775 2013-07-20
 6 20130805     -92.3     17.0  1271265280 0.747 2013-08-05
 7 20130821     -92.3     17.0 -1641302016 0.803 2013-08-21
 8 20130906     -92.3     17.0  -258902016 0.752 2013-09-06
 9 20130922     -92.3     17.0  1123497984 0.765 2013-09-22
10 20131109     -92.3     17.0   975730688 0.386 2013-11-09
# … with 103,802 more rows

I expect time above to be a double or something that preserves the unsigned integers. The same code in Python preserves the values. For now I don't need time since I will just use the date field but I wonder if anyone else has encountered this problem and has a workaround.

Thank you for a wonderful package. The above computation takes 15 seconds for my entire region of interest!

[csaybar]

Hi @spittssj, happy to see that rgee is useful :). You are right, datetime Python and R conversion is a problem, we talk about this here.

You can solve your problem as follow:

allParcelas <- ee$Geometry$Point(c(-92.3, 17.0))$buffer(100)
LandSatNDVI <- ee$ImageCollection("LANDSAT/LC08/C01/T1_8DAY_NDVI") %>% 
                  ee$ImageCollection$filterDate("2013-01-01", "2015-3-31") %>% 
                  ee$ImageCollection$filterBounds(allParcelas) %>% 
                  ee$ImageCollection$select("NDVI")
dates <- ee_get_date_ic(LandSatNDVI)
dates

[spittssj]

My fault. I just started using rgee this morning and I have not had a chance to peruse all of the documentation because I have been too busy using it! I will continue to read through the documentation.

Do you have any interest in putting it on conda-forge? I use conda to manage my R installation because in the conda environment (r_geospatial) R, python, and my whole geospatial toolchain (like PostgreSQL + PostGIS) play nicely. So I created a local r-rgee package for consistency's sake to install it but if others are interested it might be worth it. Also, I custom-configured my RStudio to use the python installation in the conda environment per the instructions I found here

I just exported the tibble of 100000 rows to a PostgreSQL table, built a geometry column, and now I'm querying against my dataset (smallholder coffee plots in Chiapas, Mexico -> hence they are called parcelas) to build time series graphs of average NDVI over 8 years for them. Nothing has taken more than 10s.

This is a pretty cool research tool. I will definitely cite you in my qualifying paper.

Te mando saludos desde el norte de los EEUU.

[csaybar]

Hi @spittssj I never heard about conda-forge, but it has a lot of sense to me. I promise to put effort to make this happen in rgee v.1.1.0. I'm really happy you find rgee useful for your research :). The sync PostGIS + rgee sounds quite interesting! 🚀

Muchas gracias por los saludos :D