[BUG] Salt worker thread overflow

[x1125]

Description
Having a reactor configured to do HTTP notifications for jobs, the amount of worker threads will increase indefinitely and eventually stop Salt from working. This wasn't the case in earlier versions.

Setup
Check PoC below.

Steps to Reproduce the behavior
I've built a PoC for that. Checkout this repository

Expected behavior
Salt shouldn't stop working after a while. We have an installation with Salt 3005.5+ds-1 running this code for years without problems

Screenshots
Check PoC

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3007.3
 
Python Version:
        Python: 3.10.17 (main, May 11 2025, 04:07:13) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.16.0
      cherrypy: unknown
  cryptography: 42.0.5
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.19.2
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.4.2
           ZMQ: 4.3.4
 
Salt Package Information:
  Package Type: onedir
 
System Versions:
          dist: ubuntu 24.04.2 noble
        locale: utf-8
       machine: x86_64
       release: 6.15.2-arch1-1
        system: Linux
       version: Ubuntu 24.04.2 noble

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Salt’s Contributor Guide
• Join our Community Discord
• Salt Project YouTube channel
• Community Wiki

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject.pdl@broadcom.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[x1125]

This problem started with 3007.2. With 3007.1 this problem doesn't exist. Might be caused by #66330?

[dwoz]

This problem started with 3007.2. With 3007.1 this problem doesn't exist. Might be caused by #66330?

Yes, I think it's very likely cause by #66330, have you tried reverting that change against 3007.x ? If that resolve it, I'd like to revert the change to salt.utils.http and come up wiith a different solution for #66194

[x1125]

This problem started with 3007.2. With 3007.1 this problem doesn't exist. Might be caused by #66330?

Yes, I think it's very likely cause by #66330, have you tried reverting that change against 3007.x ? If that resolve it, I'd like to revert the change to salt.utils.http and come up wiith a different solution for #66194

Yes! I reverted 0b92bfdf80ab40787072d546b6962c77dd24e0b7 and the bug is gone :)

[dwoz]

Likely fixed by open PR #68456 (base 3007.x), which wraps the SyncWrapper(AsyncHTTPClient, ...) in a with block so close() runs after each http.query call. That matches the leak path the reporter identified (revert of 0b92bfd fixes it).