Is your feature request related to a problem? Please describe.
Currently, VFS only provides support for deleting individual files. If deleting more than a handful of files, this causes many network roundtrips and is a generally inefficient method of doing things, especially since S3 provides native batch deletion operations.
We are currently deleting multiple files in a for loop, calling Delete on each one. This is also a waste of money, since we pay for each S3 request.
Is this something that is on the VFS team's radar?
Describe the solution you'd like
One option is to add a new function DeletePrefix(prefix string) that takes a prefix and deletes all objects matching that prefix in as few API calls as possible.
Another option would be DeleteMultiple(objects []string) that deletes all objects in string array objects, again using as few API calls as possible. This fits into the existing ListWithPrefix function, and ideally the output of this function could be directly passed into DeleteMultiple
Describe alternatives you've considered
Not including this feature
Additional context
N/A
Is your feature request related to a problem? Please describe.
Currently, VFS only provides support for deleting individual files. If deleting more than a handful of files, this causes many network roundtrips and is a generally inefficient method of doing things, especially since S3 provides native batch deletion operations.
We are currently deleting multiple files in a for loop, calling
Deleteon each one. This is also a waste of money, since we pay for each S3 request.Is this something that is on the VFS team's radar?
Describe the solution you'd like
One option is to add a new function
DeletePrefix(prefix string)that takes a prefix and deletes all objects matching that prefix in as few API calls as possible.Another option would be
DeleteMultiple(objects []string)that deletes all objects in string arrayobjects, again using as few API calls as possible. This fits into the existingListWithPrefixfunction, and ideally the output of this function could be directly passed intoDeleteMultipleDescribe alternatives you've considered
Not including this feature
Additional context
N/A