Skip to content

Conversation

@skyflow-shravan
Copy link
Contributor

This PR modifies insert method to utilise sessions provided by requests library.

Why

  • On larger scale, the simple requests.post leads to choking and connection errors.
  • therefore, we needed to optimise connection pooling by using sessions instead.
  • We also needed to add retries for any connection related and 500 errors.

Goal

  • Optimised connection pooling by utilising sessions
  • Retry any connection related and/or 500 errors.

Testing

  • Tested the changes manually
  • Load tested the changes for 2M records using locust.

@github-actions
Copy link

Gitleaks Findings: No secrets detected. Safe to proceed!

@github-actions
Copy link

Semgrep Findings: Issues with Error level severity are found (Error is Highest severity in Semgrep), Please resolve the issues before merging.

@skyflow-shravan skyflow-shravan changed the base branch from main to release/25.9.14 September 22, 2025 17:53
@skyflow-shravan skyflow-shravan changed the base branch from release/25.9.14 to skyflow-vivek/SK-2296-use-sessions-for-insert September 22, 2025 17:54
@skyflow-shravan skyflow-shravan changed the base branch from skyflow-vivek/SK-2296-use-sessions-for-insert to main September 22, 2025 17:55
@github-actions
Copy link

Gitleaks Findings: No secrets detected. Safe to proceed!

@github-actions
Copy link

Semgrep Findings: Issues with Error level severity are found (Error is Highest severity in Semgrep), Please resolve the issues before merging.

@skyflow-shravan skyflow-shravan merged commit ada8b74 into main Sep 23, 2025
7 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants