Skip to content

Comments

[WIP] Feat/multipart upload#45

Open
IonesioJunior wants to merge 9 commits intomainfrom
feat/multipart-upload
Open

[WIP] Feat/multipart upload#45
IonesioJunior wants to merge 9 commits intomainfrom
feat/multipart-upload

Conversation

@IonesioJunior
Copy link
Member

@IonesioJunior IonesioJunior commented Jun 30, 2025

This pull request introduces multipart upload functionality to the blob handling system, including API endpoints, request/response types, and validation tests. The changes enhance the system's ability to handle large file uploads efficiently by splitting them into smaller parts. Key updates include implementing multipart upload and completion logic, adding request/response structures, and comprehensive validation testing.

Multipart Upload Functionality

  • Implementation of multipart upload logic: Added UploadMultipart and UploadComplete methods to BlobHandler to initiate and complete multipart uploads, including validation of paths, permissions, and request parameters. (internal/server/handlers/blob/blob_handler.go, internal/server/handlers/blob/blob_handler.goR70-R179)

  • Integration with S3 backend: Updated CompleteMultipartUpload in S3Backend to retrieve object metadata and invoke hooks for post-upload processing. (internal/server/blob/blob_backend_s3.go, internal/server/blob/blob_backend_s3.goL225-R252)

Request and Response Structures

  • New types for multipart uploads: Added MultipartUploadRequest, MultipartUploadResponse, CompleteUploadRequest, and CompleteUploadResponse to define request and response formats for multipart upload operations. (internal/server/handlers/blob/blob_handler_types.go, internal/server/handlers/blob/blob_handler_types.goR66-R96)

Validation and Testing

Affected Dependencies

List any dependencies that are required for this change.

How has this been tested?

  • Describe the tests that you ran to verify your changes.
  • Provide instructions so we can reproduce.
  • List any relevant details for your test configuration.

Checklist

Define data structures for multipart upload requests and responses including validation rules
Implement CompleteMultipartUpload with proper metadata retrieval and index updates
Add handler for initiating multipart uploads with permission checks and presigned URL generation
Add handler for completing multipart uploads with part assembly and metadata response
Clean up placeholder implementations now that actual handlers are implemented
Add comprehensive unit tests for request validation and reserved path checking
@IonesioJunior IonesioJunior changed the title Feat/multipart upload [WIP] Feat/multipart upload Jun 30, 2025
Move UploadMultipart and UploadComplete implementations from separate
files (blob_handler_multipart.go and blob_handler_complete.go) into
the main blob_handler.go file. This consolidation improves code
organization and maintainability by keeping all blob-related handlers
in a single location.

- Move UploadMultipart method to handle multipart upload initiation
- Move UploadComplete method to handle multipart upload completion
- Remove redundant separate handler files
- Add required fmt import for error formatting
@yashgorana
Copy link
Contributor

yashgorana commented Jun 30, 2025

holding off on this change - there's a bigger pr that's coming in on the client's syftsdk. also multipart upload ETags are different than standard ETags and require some care on both blob index and local client sync journal to work correctly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants