Skip to content

Conversation

@skyflow-vivek
Copy link
Collaborator

This PR modifies insert method to utilise sessions provided by requests library.

Why

  • On larger scale, the simple requests.post leads to choking and connection errors.
  • therefore, we needed to optimise connection pooling by using sessions instead.
  • We also needed to add retries for any connection related and 500 errors.

Goal

  • Optimised connection pooling by utilising sessions
  • Retry any connection related and/or 500 errors.

Testing

  • Tested the changes manually
  • Load tested the changes for 2M records using locust.

skyflow-vivek and others added 2 commits September 16, 2025 18:11
- Optimise connection pooling by usine sessions
- Add retry for connection and 500 errors
@github-actions
Copy link

Gitleaks Findings: No secrets detected. Safe to proceed!

@github-actions
Copy link

Semgrep Findings: Issues with Error level severity are found (Error is Highest severity in Semgrep), Please resolve the issues before merging.

@skyflow-vivek skyflow-vivek changed the base branch from main to release/25.9.14 September 16, 2025 16:33
@skyflow-shravan skyflow-shravan merged commit 1c45049 into release/25.9.14 Sep 16, 2025
7 of 8 checks passed
skyflow-shravan added a commit that referenced this pull request Sep 18, 2025
* Merge pull request #199 from skyflowapi/skyflow-vivek/SK-2296-use-sessions-for-insert

SK-2296 Use sessions for insert method

* [AUTOMATED] Private Release 1.15.4.dev0+1c45049

---------

Co-authored-by: skyflow-vivek <121149418+skyflow-vivek@users.noreply.github.com>
Co-authored-by: skyflow-shravan <skyflow-shravan@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants