-
Notifications
You must be signed in to change notification settings - Fork 320
Storage Partitioned Transfer Base #3340
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
sdk/storage/azure_storage_blob/src/streams/partitioned_stream.rs
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR implements foundational components for partitioned upload and copy operations in Azure Storage Blob SDK, introducing stream partitioning and concurrent operation execution capabilities.
- Adds
PartitionedStreamthat converts aSeekableStreaminto partitionedByteschunks for block operations - Implements
run_all_with_concurrency_limit()for executing async operations with configurable concurrency - Includes comprehensive test coverage for both components
Reviewed Changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 16 comments.
Show a summary per file
| File | Description |
|---|---|
| sdk/storage/azure_storage_blob/src/streams/partitioned_stream.rs | New implementation of PartitionedStream with Stream and FusedStream traits, plus test suite |
| sdk/storage/azure_storage_blob/src/streams/mod.rs | Module declaration for streams |
| sdk/storage/azure_storage_blob/src/partitioned_transfer/mod.rs | New implementation of run_all_with_concurrency_limit() with concurrency control logic and tests |
| sdk/storage/azure_storage_blob/src/lib.rs | Module declarations for new partitioned_transfer and streams modules |
| sdk/storage/azure_storage_blob/Cargo.toml | Added bytes and futures dependencies (and rand for tests) |
| Cargo.lock | Lock file updates reflecting new dependencies |
sdk/storage/azure_storage_blob/src/streams/partitioned_stream.rs
Outdated
Show resolved
Hide resolved
sdk/storage/azure_storage_blob/src/streams/partitioned_stream.rs
Outdated
Show resolved
Hide resolved
sdk/storage/azure_storage_blob/src/streams/partitioned_stream.rs
Outdated
Show resolved
Hide resolved
Accept useful generated comments. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
generated docs tried to write a doctest for a non-public function.
sdk/storage/azure_storage_blob/src/streams/partitioned_stream.rs
Outdated
Show resolved
Hide resolved
sdk/storage/azure_storage_blob/src/streams/partitioned_stream.rs
Outdated
Show resolved
Hide resolved
sdk/storage/azure_storage_blob/src/streams/partitioned_stream.rs
Outdated
Show resolved
Hide resolved
sdk/storage/azure_storage_blob/src/streams/partitioned_stream.rs
Outdated
Show resolved
Hide resolved
API Change CheckAPIView identified API level changes in this PR and created the following API reviews |
sdk/storage/azure_storage_blob/src/streams/partitioned_stream.rs
Outdated
Show resolved
Hide resolved
|
@heaths I think this is pretty stable now that your core comments are addressed. Can I get another review on this? |
|
I will, but in general I was letting your team sign off on the rest of the changes and Storage PRs in general. I've only been taking a peek more recently since your team has been much more comfortable with Rust. I'm mainly just looking at more architectural concerns. But you can always mention me specifically in reviews or contact me offline too. |
Implements two foundation components to implement partitioned upload and copy.
PartitionedStream: Consumes aBox<dyn SeekableStream>and converts it to aStream<Item = Result<Bytes, Error>>where eachOk(Bytes)returned is a contiguously buffered partition to be used for a put block or equivalent request.run_all_with_concurrency_limit(): Takes a sequence of async jobs (impl FnOnce() -> impl Future<Output = Result<(), Error>>). These will be sequences of put block operations or equivalent requests.