We are using the gcs_upload function to upload CSVs to a GCS bucket. The sizes vary, but they are rarely more than 20MB. They are sometimes overwriting each other.
We have noticed the call hangs for a very long time — say, 5-10 minutes — before completing. Sometimes it just fails entirely. We aren't able to discern any rhyme or reason to it — it does not seem correlated with file size, for example, or whether it is overwriting a pre-existing file. We have tried both upload_type = simple and upload_type = resumable, with no success.
Because it happens infrequently, it's been impossible to create repro steps. But are there things we could be logging that could help?
Below is our current call to gcs_upload.
googleCloudStorageR::gcs_upload(
data,
object_function = function(input, output) readr::write_csv(x = input, file = output),
type = "text/csv",
name = path,
bucket = bucket,
predefinedAcl = "bucketLevel"
)
We are using the gcs_upload function to upload CSVs to a GCS bucket. The sizes vary, but they are rarely more than 20MB. They are sometimes overwriting each other.
We have noticed the call hangs for a very long time — say, 5-10 minutes — before completing. Sometimes it just fails entirely. We aren't able to discern any rhyme or reason to it — it does not seem correlated with file size, for example, or whether it is overwriting a pre-existing file. We have tried both upload_type = simple and upload_type = resumable, with no success.
Because it happens infrequently, it's been impossible to create repro steps. But are there things we could be logging that could help?
Below is our current call to gcs_upload.