Skip to content

Conversation

@skyflow-shravan
Copy link
Contributor

@skyflow-shravan skyflow-shravan commented Sep 18, 2025

This PR modifies insert method to utilise sessions provided by requests library.

Why

  • On larger scale, the simple requests.post leads to choking and connection errors.
  • therefore, we needed to optimise connection pooling by using sessions instead.
  • We also needed to add retries for any connection related and 500 errors.

Goal

  • Optimised connection pooling by utilising sessions
  • Retry any connection related and/or 500 errors.

Testing

  • Tested the changes manually
  • Load tested the changes for 2M records using locust.

@github-actions
Copy link

Gitleaks Findings: No secrets detected. Safe to proceed!

@github-actions
Copy link

Semgrep Findings: Issues with Error level severity are found (Error is Highest severity in Semgrep), Please resolve the issues before merging.

@skyflow-shravan skyflow-shravan merged commit 57d594b into main Sep 18, 2025
7 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants