Skip to content

feat: optimize Horreum data loading with concurrency#168

Merged
jhutar merged 2 commits into
redhat-performance:mainfrom
deekshith-24:feat/optimize-horreum-data-loading
Mar 31, 2026
Merged

feat: optimize Horreum data loading with concurrency#168
jhutar merged 2 commits into
redhat-performance:mainfrom
deekshith-24:feat/optimize-horreum-data-loading

Conversation

@deekshith-24
Copy link
Copy Markdown
Contributor

@deekshith-24 deekshith-24 commented Mar 31, 2026

Summary

Optimizes Horreum data loading by implementing HTTP connection pooling and concurrent processing.

Performance improvement: ~19x faster

Changes

  1. HTTP Session Pooling - Reuses TCP connections instead of creating new ones for each request
  2. Concurrent Trashed Run Checking - Checks 5 runs in parallel instead of sequentially
  3. Concurrent Alerting Variable Checking - Checks all alerting variables in parallel

Technical Details

  • Added requests.Session() for connection pooling
  • Used ThreadPoolExecutor for concurrent processing (5 workers for trashed checks, 10 for alerting)
  • Replaced all requests.get/post/put/delete with session methods
  • Early termination when duplicate/change is found

Testing

  • ✅ Validated with 75 successful uploads
  • ✅ Measured average upload time: 10.7 seconds (down from ~120+ seconds)
  • ✅ Result determination: 5.9 seconds (down from ~115 seconds)

Co-Authored-By: Claude Sonnet 4.5 noreply@anthropic.com

Implements connection pooling and concurrent processing for trashed runs and alerting variables.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@deekshith-24 deekshith-24 requested a review from jhutar March 31, 2026 07:21
@jhutar jhutar merged commit 1f5336a into redhat-performance:main Mar 31, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants