Plugin for direct download of datasets needed

[jotelha]

lookup server needs a plugin for direct download of dataset.

[pastewka]

What I learned from topobank is that upload and download should never go through the web server itself, as uploading massive data just blocks the server. This means this feature would be restricted to specific storage brokers.

For example, if the data resides on S3 it is straightforward to generate a presigned url that allows download of a specific file. For SMB storage backends, this would need to be funnelled through the lookup server. I'd suggest to restrict this to brokers that support generating presigned URLs. This means this needs to be implemented in the core dtool first.

I don't know if any of the other storage brokers could in principle support this. My guess is that Azure does, I don't know what IRODS can do.

Note that uploading could work the same way, i.e. dtool generates a bunch of presigned URLs for PUT requests to S3, the user uploads and then the lookup server compares hashes to client-generated ones.

Once this is implemented, we could implement upload and download of data into the webapp.