Server not sending back 200 for uploads on time

[0xCardiE]

This is about when you just upload the file to 100% to bee node and getting the 200 response.
As you can see on video on testing infra this happens short after upload that we get 200 response that its uploaded.
On other video 200 happens only when file is actually chunked and available.

Problem is that we cant inform users properly of what is going on in this case. Upload progress and chunk distribution is combined in 200 response.
So lets say we are uploading 1GB it will be very bad as upload and distribution will be joined and then UX will suck

Vids.

https://github.com/user-attachments/assets/46561936-e14c-4195-a25f-5d603fbe0238
https://github.com/user-attachments/assets/bc6e0c91-a61b-418e-975d-fd7a5fd3ca9c

[Cafe137]

What I can see while testing locally is that Beeport UI sends the request with header Swarm-Deferred-Upload: false, a.k.a direct upload

This is defined in constants.ts

// Swarm upload configuration
export const SWARM_DEFERRED_UPLOAD = 'false'; // Use direct upload to Swarm network for better performance

Direct uploads hold the request on purpose (so no HTTP 200 OK) until it is fully synced.

I understand the videos show a different behaviour, but it seems Beeport, as of now, is misconfigured. We intend to use deferred, not direct uploads, right?

[0xCardiE]

I am not sure anymore, think we at one point said we wanted non deferred but it makes more sense to try deferred.
So lets try with that then.

Putting some reference here

[Cafe137]

Agreed. With deferred we get the HTTP 200 response as soon as all the data has reached the Bee node, and we can continue by using the tags API to get accurate information on the sync progress, which usually takes the majority of time.

[darkobas2]

This comes with a few additional questions about how things work

• asuming deferred files uploaded by users are stored on the bee until upload completes:
  • where are they stored? Assuming tmp or ram
  • what happens when this storage is full ?
  • what happens if this storage usage causes bee to reboot (oom)

[0xCardiE]

Good points. @Cafe137 think we decided before on deferred = false to have more reliable dapp and not depend on possible node pitfalls

[Cafe137]

@0xCardiE OK, this is good information, let's stay on the safe side.

In this case, my suggestion for #33 is just displaying a static message, something like Syncing takes approximately ${uploadSizeInGigabytes * 10} minutes. Your upload has been added to the history, it is safe to leave the page and check back later. (as per of #32)

[0xCardiE]

@0xCardiE OK, this is good information, let's stay on the safe side.

In this case, my suggestion for #33 is just displaying a static message, something like Syncing takes approximately ${uploadSizeInGigabytes * 10} minutes. Your upload has been added to the history, it is safe to leave the page and check back later. (as per of #32)

ok but I do wonder how do we get here 200 before and get some estimates?
https://swarming.site/

its same code, its also deferred false. I dont think
we have override somewhere for upload types on bee node or did I set some afaik(can double check)

[Cafe137]

Should it be reproducible on swarming.site right now?

I just tested it, but request was pending until the very end.

See "pending":

Was syncing for 35 seconds before receiving HTTP 200 OK:

[0xCardiE]

hmm yes, interesting, having the same thing now. let me check some things...

[0xCardiE]

yeah, its the same on swarming. But you can see my video from OP it was done using swarming node and deferred was false as it was for months. Seems its something else there that maybe could be with all of this. Will try to dig it up

[Cafe137]

@0xCardiE Is there any chance of an accidental "double syncing" in the video? I see it starts with 5 seconds of the static distribution. Then it switches to the dynamic distribution step for 21 seconds. But it's not obvious from the video if the static distribution was only 5 seconds, or more which is not part of the video. Do you maybe remember something about this?

I am also trying to reproduce this, and also looking at the Bee source code to see how this could be triggered, but no luck so far and I have no other theories at the moment.

[crtahlin]

What are the next steps on this issue? Is it stale or waiting for something from DevOps? @0xCardiE ?