Upload fails with large file, works with smaller ones (nondescriptive readerror)

[JulianRein]

Hi there,

I am using easyDataverse and thus pydvuploader.
However, when uploading a "larger" 2.1 GB file (fastq.gz), the upload stalls and fails with a "ReadError" and nothing else.
It works for smaller files in the same directory.
It also fails when renaming the file to remove the .fastq.gz
Likely unrelated: wierd filenames like "index.html?C=N;O=D" also fail. Here, renaming helps.

Any idea how to solve this?
I attached the (very long) error log, which unfortunately does not help me at all. (to avoid confusion: streamlit is the name of my conda-env, it does not run as an actual webapp at the moment, but in an .ipynb within positron.
However, running it in a standard console throws the same error -> both example runs attached)

My code (besides the console run):

`
import dvuploader as dv

path = "testProjectX/SDFSF_3232423_24R388-1_S143_L000_R2_001.fastq.gz" (large file)
"#path = "uploadCode.tmp" WORKED (small file)"
"# path = "testProjectX/exampleHtml2" # WORKED (small file) "
"# path = "testProjectX/index.html?C=D;O=D" did NOT work (small file)"
"# path = "testProjectX/R2fastqCopied.fastq.gz" NOT working either... takes too long?" (large file)
"# path = "testProjectX/R2fastqCopied" # Nope..." (large file)

"# Add file individually"
files = [
dv.File(directory_label="fastq", filepath=path, tab_ingest = False),
]

dvuploader = dv.DVUploader(files=files)
dvuploader.upload(
api_token=dataset.API_TOKEN,
dataverse_url=dataset.DATAVERSE_URL,
persistent_id=dataset.p_id,
n_parallel_uploads=1, # Whatever your instance can handle
`

longErrorPyDVuploader.txt

[JR-1991]

@JulianRein, thank you for submitting the issue. I was able to replicate the error even locally. Upon closer inspection of the passed file handlers, including a sanity check for size and correctness, I found that the correct file is being passed to the request library. I suspect that this is not a specific issue with python-dvuploader, but rather with either httpx or Dataverse itself.

I’ll check if the issue works on previous Dataverse versions. If it does, we can exclude this as the source of the problem. It’s possible that larger files are restricted because Dataverse recommends HTTP-based uploads up to 2GB. In general, it’s advisable to use the S3 direct upload (which is also supported here) to upload larger volumes, as HTTP uploads can be unreliable with large files.

I’ll get back to you once I have more information.

[JulianRein]

@JR-1991 Thanks again for the quick reaction.
As a side note: I did (try to) increase the dataverse file limit to > 5GB with below command, to no avail:

curl -X PUT -d 6000000000 http://localhost:8080/api/admin/settings/:MaxFileUploadSizeInBytes
{"status":"OK","data":{":MaxFileUploadSizeInBytes":"6000000000"}}

[JR-1991]

@JulianRein I have added an intermediate streaming layer to track the upload progress, because somehow with the new httpx version this did not work as before, and it seemed to have resolved the error (local and remote). I suspect that the ReadError has stemmed from too frequent readouts, causing an error, but I am not sure. I will put this more to the test and will get back to you with a PR to test on your machine.

[JulianRein]

@JR-1991 Thanks for working on it. Any updates on the current status?

[JR-1991]

@JulianRein apologies for my delayed response, could not get back earlier. I’ve pushed the changes that resolved the error locally to a new branch and opened a new draft pull request (#48) for testing. Can you try it again with these changes?

[JulianRein]

@JR-1991 thanks for the changes.

The upload progresses further now. It uploads the 2 GB test file (slight delay at 30% completion in the progress bar, but that is a minor issue), takes about 30 sec to 100% completion (local "upload").
However, it just hangs after, waiting for something?
After several minutes, if fails. Any idea why?

Here is my test code, error log attached ( tenacity.RetryError: RetryError[<Future at 0x72aa0033b530 state=finished raised ReadError>])

from easyDataverse import Dataverse \n
dataverse = Dataverse(
  server_url="http://localhost:8080",
  api_token="XXXXXXX",
)

dataset = dataverse.load_dataset(
    pid="perma:DV/AXSME6",
)

dataset.add_file(local_path="testProjectX/DX130_305732_24R388-1_S143_L000_R2_001.fastq.gz", dv_path = "fastq/", tab_ingest=False)
dataset.update()

Note: "streamlit" is my conda env
bugfixError.txt

[JR-1991]

@JulianRein, thanks for the feedback!

Upon reviewing the files you’ve uploaded, the TSV file might trigger a dataset lock, preventing any retries initiated by tenacity. Although you’ve explicitly stated that you don’t want to perform tabular ingestion, there was a recent bug that caused the tab ingest to fail. This bug was fixed and merged into the main branch in #46, but this branch hasn’t been updated yet. I’ve merged the main branch with the fix into this PR.

Could you try uploading the files with and without the TSV file to ensure that this isn’t a general issue? I’ve tested a 2GB mock file on our Stuttgart instance, and the upload works. So, I believe this might be the solution to this problem. :)

[JulianRein]

@JR-1991 Thanks for the quick reply!
I deleted the .tsv and the .tab from the project, see below. (I assumed that I should just test, not update to the new ingestion-fix PR).
However, the hangup still persists with the same error. Is there any use to try it on a new project? Any other ideas?