Describe the bug
The error occurs when using extractEnv for some variables (e.g., "rs_sst_interpolate") when the pull_var() internal function is unable to find the specific raster file using the constructed URL catalog. This is due to some raster files that do not exist on the IMOS server.
To Reproduce
Using the example dataset, and modifying it to extract data from certain days that IMOS doesn't have 'rs_sst_interpolated' data for.
qc_data <-
TownsvilleReefQC %>%
unnest(cols = c(QC)) %>%
ungroup() %>%
filter(Detection_QC %in% c(1,2)) %>%
filter(filename == unique(filename)[1]) %>%
slice(5:8) %>%
mutate(detection_datetime = ymd_hms(c("2013-08-18 11:08:06", "2013-08-18 21:35:36", "2014-04-27 00:00:00", "2023-08-1400:00:00")))
data_with_sst <-
extractEnv(df = qc_data,
X = "receiver_deployment_longitude",
Y = "receiver_deployment_latitude",
datetime = "detection_datetime",
env_var = "rs_sst_interpolated",
cache_layers = FALSE,
crop_layers = TRUE,
full_timeperiod = FALSE,
fill_gaps = TRUE,
folder_name = "test",
.parallel = FALSE)
Expected behavior
The expected behaviour would be to return a NA for those dates IMOS does not have files for. Additional detail of what dates these errors were for can be provided in a message in the console would be also useful for the user to know which dates there was no data for (and potentially follow up with AODN to fix). Currently, the function terminates with an error when it cannot find the URL that was built by the build_url() function.
Screenshots
Desktop (please complete the following information):
R version 4.4.0 (2024-04-24)
Platform: x86_64-apple-darwin20
Running under: macOS Sonoma 14.4.1
Additional context
Potential solution could be running an additional check in the build_url() function to provide a validity check on each URL produced.
A simple function to check each URL in the url_df prior to returning it at the end of the function:
valid_url <-
function(url_in, t = 2){
con <- url(url_in)
check <- suppressWarnings(try(open.connection(con,open="rt",timeout=t),silent=T)[1])
suppressWarnings(try(close.connection(con),silent=T))
ifelse(is.null(check),TRUE,FALSE)
}
url_df <-
url_df %>%
mutate(valid = sapply(url_name, valid_url)) %>%
filter(valid %in% TRUE)
There may be a more efficient way to validate URLs, but this will prevent the subsequent pull_var() function from failing.
Describe the bug
The error occurs when using extractEnv for some variables (e.g., "rs_sst_interpolate") when the pull_var() internal function is unable to find the specific raster file using the constructed URL catalog. This is due to some raster files that do not exist on the IMOS server.
To Reproduce
Using the example dataset, and modifying it to extract data from certain days that IMOS doesn't have 'rs_sst_interpolated' data for.
qc_data <-
TownsvilleReefQC %>%
unnest(cols = c(QC)) %>%
ungroup() %>%
filter(Detection_QC %in% c(1,2)) %>%
filter(filename == unique(filename)[1]) %>%
slice(5:8) %>%
mutate(detection_datetime = ymd_hms(c("2013-08-18 11:08:06", "2013-08-18 21:35:36", "2014-04-27 00:00:00", "2023-08-1400:00:00")))
data_with_sst <-
extractEnv(df = qc_data,
X = "receiver_deployment_longitude",
Y = "receiver_deployment_latitude",
datetime = "detection_datetime",
env_var = "rs_sst_interpolated",
cache_layers = FALSE,
crop_layers = TRUE,
full_timeperiod = FALSE,
fill_gaps = TRUE,
folder_name = "test",
.parallel = FALSE)
Expected behavior
The expected behaviour would be to return a NA for those dates IMOS does not have files for. Additional detail of what dates these errors were for can be provided in a message in the console would be also useful for the user to know which dates there was no data for (and potentially follow up with AODN to fix). Currently, the function terminates with an error when it cannot find the URL that was built by the build_url() function.
Screenshots
Desktop (please complete the following information):
R version 4.4.0 (2024-04-24)
Platform: x86_64-apple-darwin20
Running under: macOS Sonoma 14.4.1
Additional context
Potential solution could be running an additional check in the build_url() function to provide a validity check on each URL produced.
A simple function to check each URL in the url_df prior to returning it at the end of the function:
valid_url <-
function(url_in, t = 2){
con <- url(url_in)
check <- suppressWarnings(try(open.connection(con,open="rt",timeout=t),silent=T)[1])
suppressWarnings(try(close.connection(con),silent=T))
ifelse(is.null(check),TRUE,FALSE)
}
url_df <-
url_df %>%
mutate(valid = sapply(url_name, valid_url)) %>%
filter(valid %in% TRUE)
There may be a more efficient way to validate URLs, but this will prevent the subsequent pull_var() function from failing.