Skip to content

Support batch deletion operation in S3 backend #135

@arestifo

Description

@arestifo

Is your feature request related to a problem? Please describe.
Currently, VFS only provides support for deleting individual files. If deleting more than a handful of files, this causes many network roundtrips and is a generally inefficient method of doing things, especially since S3 provides native batch deletion operations.

We are currently deleting multiple files in a for loop, calling Delete on each one. This is also a waste of money, since we pay for each S3 request.

Is this something that is on the VFS team's radar?

Describe the solution you'd like
One option is to add a new function DeletePrefix(prefix string) that takes a prefix and deletes all objects matching that prefix in as few API calls as possible.

Another option would be DeleteMultiple(objects []string) that deletes all objects in string array objects, again using as few API calls as possible. This fits into the existing ListWithPrefix function, and ideally the output of this function could be directly passed into DeleteMultiple

Describe alternatives you've considered
Not including this feature

Additional context
N/A

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions