Skip to content

README_CHUNKING_FIX.md

Latest commit

 

History

History
123 lines (95 loc) · 3.78 KB

README_CHUNKING_FIX.md

File metadata and controls

123 lines (95 loc) · 3.78 KB

Traffic Log Chunking Fix

Problem Resolved

Fixed the "Value too large (max 65536 bytes)" error that occurred when calling exposed APIs via Postman. The error was happening when storing traffic logs in Deno KV because large API responses were exceeding the storage limit.

Root Cause

  • Issue: Traffic logs containing large API responses (>65KB) couldn't be stored in Deno KV
  • Location: MockApiRepository.storeTrafficLog() method in src/services/mockApiService.ts
  • Impact: API calls with large responses were failing with 500 errors

Solution Implemented

1. Traffic Log Chunking

  • Extended the existing chunking mechanism to traffic logs
  • Large response bodies are automatically chunked into smaller pieces
  • Chunks are stored separately and reassembled when retrieved

2. Smart Chunking Strategy

// Three-tier approach:
1. Normal storage: Logs < 65KB stored normally
2. Response chunking: Large response bodies (>32KB) are chunked
3. Truncation fallback: Other large data is truncated if chunking doesn't help

3. Automatic Reconstruction

  • Traffic logs are automatically reconstructed when retrieved
  • No changes needed to existing UI or API consumers
  • Transparent to users - they see complete data

Technical Details

Storage Strategy

// Large response body chunking
if (responseBodySize > MAX_VALUE_SIZE / 2) {
  // Split response into chunks
  const chunks = ChunkingUtils.chunkResponseData(responseBody);
  // Store chunks separately with keys like: traffic-{logId}:0, traffic-{logId}:1
}

Retrieval Strategy

// Automatic reconstruction in getTrafficLogs()
private async reconstructTrafficLog(log: TrafficLog): Promise<TrafficLog> {
  if (log.response.body.__chunked) {
    // Retrieve and combine all chunks
    return reconstructedLog;
  }
  return log; // Return as-is if not chunked
}

Cleanup Strategy

// Traffic log cleanup includes chunk cleanup
async clearTrafficLogs(): Promise<number> {
  // Delete main logs AND their associated chunks
}

Benefits

  1. No More 500 Errors: Large API responses no longer cause storage failures
  2. Transparent Operation: Users see complete data without knowing about chunking
  3. Automatic Fallback: Multiple strategies ensure data is always stored
  4. Performance Optimized: Only chunks data when necessary
  5. Memory Efficient: Doesn't load all chunks unless needed

What's Fixed

  • ✅ Large API responses can now be logged without errors
  • ✅ Traffic monitoring works for all API sizes
  • ✅ Postman calls to large-response APIs now succeed
  • ✅ Performance metrics are captured for all requests
  • ✅ Debugging capabilities maintained for large responses

Configuration

The chunking is automatic and requires no configuration:

// Constants in mockApiService.ts
const MAX_VALUE_SIZE = 60000; // Conservative limit below 65536
const CHUNK_SIZE = 50000;     // Chunk size for large data

Monitoring

Look for these log messages to see chunking in action:

Traffic log {id} is too large ({size} bytes), chunking response body...
Cleaned up {count} chunks for traffic log {id}
Missing chunk for traffic log: {chunkKey}

Before vs After

Before (Failing)

TypeError: Value too large (max 65536 bytes)
  at MockApiRepository.storeTrafficLog

After (Working)

✅ Traffic log stored successfully (chunked into 3 parts)
✅ API response: 200 OK
✅ Performance metrics captured

Backward Compatibility

  • ✅ Existing traffic logs: Continue working normally
  • ✅ Small responses: No change in behavior
  • ✅ API contracts: No breaking changes
  • ✅ UI components: Show complete data transparently

The fix is production-ready and handles edge cases gracefully.