Skip to content

Commit b8b8f1e

Browse files
MagicFun1241MagicFun1241
committed
feat: enhance multi-part upload API with uploadToken for session-specific authentication
1 parent 827b718 commit b8b8f1e

4 files changed

Lines changed: 58 additions & 229 deletions

File tree

‎MULTIPART_UPLOAD.md‎

Lines changed: 14 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -8,13 +8,17 @@ The server provides three endpoints:
88
- Upload a single part: POST /videos/multiparts/:uploadId/parts/:partIndex
99
- Check upload status: GET /videos/multiparts/:uploadId
1010

11-
Security: All multi-part endpoints require the `token` query parameter matching the `APP_TOKEN` configuration value.
12-
Uploading must also be enabled via `APP_UPLOADING_ENABLED=true` and S3 must be configured.
13-
Redis is required for upload tracking when using multi-part uploads.
11+
Security:
12+
- The initialization endpoint requires `token` query parameter matching `APP_TOKEN`.
13+
- Part upload requires `uploadToken` (a unique token generated per upload session, returned by init).
14+
- Status check requires `token` query parameter matching `APP_TOKEN`.
15+
- Uploading must be enabled via `APP_UPLOADING_ENABLED=true` and S3 must be configured.
16+
- Redis is required for upload tracking when using multi-part uploads.
1417

1518
## Key concepts
1619

1720
- uploadId: A server-generated identifier for the multi-part upload session. Returned by the initialization endpoint.
21+
- uploadToken: A secure random token generated per upload session. Use this to authenticate part uploads (not APP_TOKEN).
1822
- parts: The file is split into N parts according to the configured chunk size. Each part has index, offset and size.
1923
- partIndex: Zero-based index of a part in the upload session.
2024
- chunkSize: The per-part size in bytes. Default is 80 MB (80 * 1024 * 1024). Can be overridden during init.
@@ -49,6 +53,7 @@ Response (200 OK)
4953
```json
5054
{
5155
"uploadId": "<upload-id>",
56+
"uploadToken": "<secure-random-token>",
5257
"location": "videos/user123/my-video.mp4",
5358
"totalSize": 157286400,
5459
"chunkSize": 83886080,
@@ -69,26 +74,28 @@ Errors
6974

7075
Notes
7176
- The server computes partsCount = ceil(size / chunkSize) and returns an array of part metadata with offsets and sizes.
77+
- The server generates a unique `uploadToken` for this session. Save this token - you'll need it to upload parts.
7278
- Upload tracking is stored in Redis under the key `upload:{uploadId}` and kept until the upload `expiresAt` (or a configured TTL).
7379

7480
## 2) Upload a part
7581

7682
Endpoint
7783
```
78-
POST /videos/multiparts/:uploadId/parts/:partIndex?token={token}
84+
POST /videos/multiparts/:uploadId/parts/:partIndex?uploadToken={uploadToken}
7985
```
8086

8187
Path parameters
8288
- uploadId (required) — upload session id returned by init
8389
- partIndex (required) — zero-based part index to upload
8490

8591
Query parameters
86-
- token (required) — must match `APP_TOKEN`
92+
- uploadToken (required) — the unique token returned by the init endpoint (NOT APP_TOKEN)
8793

8894
Form data
8995
- video (required) — the multipart `video` field containing raw bytes for this part. The server expects the part size to exactly match the declared size for this part.
9096

9197
Behavior
98+
- The server validates the `uploadToken` against the stored upload session.
9299
- The server reads the `uploadId` info from Redis, validates the `partIndex` and the part size.
93100
- The part is stored in S3 under `{location}.part{partIndex}` using the configured S3 client.
94101
- The server marks the part as uploaded in Redis.
@@ -97,7 +104,6 @@ Behavior
97104
Response (200 OK)
98105
```json
99106
{
100-
"success": true,
101107
"uploadId": "1234567890-videos/user123/video.mp4",
102108
"partIndex": 0,
103109
"size": 83886080,
@@ -108,12 +114,10 @@ Response (200 OK)
108114
When the last part is uploaded:
109115
```json
110116
{
111-
"success": true,
112117
"uploadId": "1234567890-videos/user123/video.mp4",
113118
"partIndex": 1,
114119
"size": 73400320,
115-
"complete": true,
116-
"message": "upload complete, parts will be merged"
120+
"complete": true
117121
}
118122
```
119123

@@ -185,7 +189,7 @@ curl -X POST \
185189
### cURL: upload part 0
186190
```bash
187191
curl -X POST \
188-
"http://localhost:3000/videos/multiparts/<UPLOAD_ID>/parts/0?token=${APP_TOKEN}" \
192+
"http://localhost:3000/videos/multiparts/<UPLOAD_ID>/parts/0?uploadToken=${UPLOAD_TOKEN}" \
189193
-F "video=@part0.bin" \
190194
-i
191195
```

‎README.md‎

Lines changed: 1 addition & 197 deletions
Original file line numberDiff line numberDiff line change
@@ -293,7 +293,6 @@ POST /videos?deadline={unix_timestamp}&location={base64_location}&signature={hma
293293
**Response (201 Created):**
294294
```json
295295
{
296-
"success": true,
297296
"location": "videos/user123/my-video.mp4",
298297
"size": 1048576
299298
}
@@ -380,202 +379,7 @@ For more details, see [VIDEO_UPLOAD.md](VIDEO_UPLOAD.md).
380379

381380
For large video files, the service supports multi-part uploads with progress tracking via Redis.
382381

383-
#### Configuration
384-
```bash
385-
export APP_UPLOADING_ENABLED=true
386-
export APP_TOKEN="your-upload-token"
387-
export REDIS_ENABLED=true
388-
export REDIS_ADDR="localhost:6379"
389-
export S3_ENABLED=true
390-
export S3_ENDPOINT="s3.amazonaws.com"
391-
export S3_ACCESS_KEY_ID="your-access-key"
392-
export S3_SECRET_ACCESS_KEY="your-secret-key"
393-
export S3_BUCKET="your-bucket"
394-
export APP_CHUNK_SIZE=83886080 # 80MB chunks
395-
```
396-
397-
#### Step 1: Initialize Multi-Part Upload
398-
```
399-
POST /videos/multiparts?token={token}&deadline={unix_timestamp}&location={location}&size={bytes}&contentType={mime_type}&chunkSize={bytes}
400-
```
401-
402-
**Query Parameters:**
403-
- `token`: Authentication token (required)
404-
- `deadline`: Unix timestamp when upload expires (required)
405-
- `location`: S3 object key where video will be stored (required)
406-
- `size`: Total file size in bytes (required)
407-
- `contentType`: Video MIME type (required)
408-
- `chunkSize`: Optional chunk size in bytes (default: 80MB)
409-
410-
**Response (200 OK):**
411-
```json
412-
{
413-
"uploadId": "1234567890-videos/user123/video.mp4",
414-
"location": "videos/user123/video.mp4",
415-
"totalSize": 157286400,
416-
"chunkSize": 83886080,
417-
"partsCount": 2,
418-
"parts": [
419-
{ "index": 0, "offset": 0, "size": 83886080 },
420-
{ "index": 1, "offset": 83886080, "size": 73400320 }
421-
],
422-
"expiresAt": "2025-11-03T12:00:00Z"
423-
}
424-
```
425-
426-
#### Step 2: Upload Each Part
427-
```
428-
POST /videos/multiparts/{uploadId}/parts/{partIndex}?token={token}
429-
```
430-
431-
**Path Parameters:**
432-
- `uploadId`: Upload ID from initialization (required)
433-
- `partIndex`: Zero-based part index (required)
434-
435-
**Query Parameters:**
436-
- `token`: Authentication token (required)
437-
438-
**Form Data:**
439-
- `video`: The video part data (required)
440-
441-
**Response (200 OK):**
442-
```json
443-
{
444-
"success": true,
445-
"uploadId": "1234567890-videos/user123/video.mp4",
446-
"partIndex": 0,
447-
"size": 83886080,
448-
"complete": false
449-
}
450-
```
451-
452-
When the last part is uploaded:
453-
```json
454-
{
455-
"success": true,
456-
"uploadId": "1234567890-videos/user123/video.mp4",
457-
"partIndex": 1,
458-
"size": 73400320,
459-
"complete": true,
460-
"message": "upload complete, parts will be merged"
461-
}
462-
```
463-
464-
#### Step 3: Check Upload Status
465-
```
466-
GET /videos/multiparts/{uploadId}?token={token}
467-
```
468-
469-
**Path Parameters:**
470-
- `uploadId`: Upload ID from initialization (required)
471-
472-
**Query Parameters:**
473-
- `token`: Authentication token (required)
474-
475-
**Response (200 OK):**
476-
```json
477-
{
478-
"uploadId": "1234567890-videos/user123/video.mp4",
479-
"location": "videos/user123/video.mp4",
480-
"totalSize": 157286400,
481-
"partsCount": 2,
482-
"uploadedParts": [0, 1],
483-
"uploadedCount": 2,
484-
"complete": true,
485-
"progress": 100,
486-
"contentType": "video/mp4",
487-
"createdAt": "2025-11-03T11:00:00Z",
488-
"expiresAt": "2025-11-03T12:00:00Z"
489-
}
490-
```
491-
492-
#### Example: Multi-Part Upload (Node.js)
493-
```javascript
494-
const fs = require('fs');
495-
const FormData = require('form-data');
496-
497-
const CHUNK_SIZE = 80 * 1024 * 1024; // 80MB
498-
const token = 'your-upload-token';
499-
const baseUrl = 'http://localhost:3000';
500-
501-
async function uploadVideoMultipart(filePath, location) {
502-
// Get file stats
503-
const stats = fs.statSync(filePath);
504-
const fileSize = stats.size;
505-
const contentType = 'video/mp4';
506-
507-
// Calculate deadline (5 hours from now)
508-
const deadline = Math.floor(Date.now() / 1000) + (5 * 3600);
509-
510-
// Step 1: Initialize upload
511-
const initUrl = new URL(`${baseUrl}/videos/multiparts`);
512-
initUrl.searchParams.set('token', token);
513-
initUrl.searchParams.set('deadline', deadline);
514-
initUrl.searchParams.set('location', location);
515-
initUrl.searchParams.set('size', fileSize);
516-
initUrl.searchParams.set('contentType', contentType);
517-
initUrl.searchParams.set('chunkSize', CHUNK_SIZE);
518-
519-
const initResponse = await fetch(initUrl, { method: 'POST' });
520-
const uploadInfo = await initResponse.json();
521-
522-
console.log(`Upload initialized: ${uploadInfo.uploadId}`);
523-
console.log(`Total parts: ${uploadInfo.partsCount}`);
524-
525-
// Step 2: Upload each part
526-
const fileStream = fs.createReadStream(filePath);
527-
528-
for (const part of uploadInfo.parts) {
529-
console.log(`Uploading part ${part.index + 1}/${uploadInfo.partsCount}...`);
530-
531-
// Read chunk from file
532-
const buffer = Buffer.alloc(part.size);
533-
const fd = fs.openSync(filePath, 'r');
534-
fs.readSync(fd, buffer, 0, part.size, part.offset);
535-
fs.closeSync(fd);
536-
537-
// Create form data
538-
const formData = new FormData();
539-
formData.append('video', buffer, { filename: 'part.mp4' });
540-
541-
// Upload part
542-
const uploadUrl = new URL(`${baseUrl}/videos/multiparts/${uploadInfo.uploadId}/parts/${part.index}`);
543-
uploadUrl.searchParams.set('token', token);
544-
545-
const partResponse = await fetch(uploadUrl, {
546-
method: 'POST',
547-
body: formData
548-
});
549-
550-
const result = await partResponse.json();
551-
console.log(`Part ${part.index} uploaded (${result.complete ? 'COMPLETE' : 'in progress'})`);
552-
}
553-
554-
// Step 3: Check final status
555-
const statusUrl = new URL(`${baseUrl}/videos/multiparts/${uploadInfo.uploadId}`);
556-
statusUrl.searchParams.set('token', token);
557-
558-
const statusResponse = await fetch(statusUrl);
559-
const status = await statusResponse.json();
560-
561-
console.log(`Upload complete: ${status.complete}`);
562-
console.log(`Progress: ${status.progress}%`);
563-
564-
return status;
565-
}
566-
567-
// Usage
568-
uploadVideoMultipart('./large-video.mp4', 'videos/user123/large-video.mp4')
569-
.then(status => console.log('Done!', status))
570-
.catch(err => console.error('Error:', err));
571-
```
572-
573-
#### Benefits of Multi-Part Upload
574-
- **Resume capability**: Track which parts have been uploaded
575-
- **Parallel uploads**: Upload multiple parts simultaneously
576-
- **Progress tracking**: Monitor upload progress in real-time
577-
- **Large file support**: Handle files larger than server memory limits
578-
- **Fault tolerance**: Retry individual parts on failure
382+
For detailed documentation including API endpoints, parameters, examples, and security model, see [MULTIPART_UPLOAD.md](MULTIPART_UPLOAD.md).
579383

580384
## URL Encoding for Path-based Format
581385

‎routes/multipart_upload.go‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@ package routes
22

33
import (
44
"context"
5+
"crypto/rand"
6+
"encoding/hex"
57
"encoding/json"
68
"fmt"
79
"time"
@@ -28,6 +30,7 @@ type UploadPart struct {
2830
// UploadInfo represents the multi-part upload tracking information
2931
type UploadInfo struct {
3032
UploadID string `json:"uploadId"`
33+
UploadToken string `json:"uploadToken"` // Token specific to this upload session
3134
Location string `json:"location"`
3235
TotalSize int64 `json:"totalSize"`
3336
ChunkSize int64 `json:"chunkSize"`
@@ -75,6 +78,15 @@ func (r *RedisUploadTracker) Close() error {
7578
return r.client.Close()
7679
}
7780

81+
// generateUploadToken generates a secure random token for upload authentication
82+
func generateUploadToken() (string, error) {
83+
bytes := make([]byte, 32) // 256-bit token
84+
if _, err := rand.Read(bytes); err != nil {
85+
return "", err
86+
}
87+
return hex.EncodeToString(bytes), nil
88+
}
89+
7890
// InitializeUpload creates upload tracking information and returns part details
7991
func (r *RedisUploadTracker) InitializeUpload(ctx context.Context, uploadID, location string, totalSize int64, chunkSize int64, contentType string, deadline time.Time) (*UploadInfo, error) {
8092
if r == nil || r.client == nil {
@@ -85,6 +97,12 @@ func (r *RedisUploadTracker) InitializeUpload(ctx context.Context, uploadID, loc
8597
chunkSize = DefaultChunkSize
8698
}
8799

100+
// Generate upload-specific token
101+
uploadToken, err := generateUploadToken()
102+
if err != nil {
103+
return nil, fmt.Errorf("failed to generate upload token: %w", err)
104+
}
105+
88106
// Calculate parts
89107
partsCount := int((totalSize + chunkSize - 1) / chunkSize) // Ceiling division
90108

@@ -108,6 +126,7 @@ func (r *RedisUploadTracker) InitializeUpload(ctx context.Context, uploadID, loc
108126

109127
uploadInfo := &UploadInfo{
110128
UploadID: uploadID,
129+
UploadToken: uploadToken,
111130
Location: location,
112131
TotalSize: totalSize,
113132
ChunkSize: chunkSize,

0 commit comments

Comments
 (0)