[BUG] 3007.1 - salt-proxy - "there is no current event loop in thread"

[ITJamie]

Description
attempt to run a state.apply against a salt-sproxy device.

Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

Please be as specific as possible and give set-up details.

• on-prem machine
• VM (Virtualbox, KVM, etc. please specify)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

Steps to Reproduce the behavior
setup a salt-sproxy host + master setup. both version 3007.1 (in my case i was using the napalm nxos sproxy config)
confirm that basic commands like test.ping work to the salt-proxy host.
when you attempt to apply the following error occurs:

salt "sproxyminionname" state.apply timeout=600 test=true
sproxyminionname:
    The minion function caused an exception: Traceback (most recent call last):
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/metaproxy/proxy.py", line 483, in thread_return
        return_data = minion_instance.executors[fname](
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 160, in __call__
        ret = self.loader.run(run_func, *args, **kwargs)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1269, in run
        return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1284, in _run_as
        return _func_or_method(*args, **kwargs)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/executors/direct_call.py", line 10, in execute
        return func(*args, **kwargs)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 160, in __call__
        ret = self.loader.run(run_func, *args, **kwargs)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1269, in run
        return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1284, in _run_as
        return _func_or_method(*args, **kwargs)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/state.py", line 835, in apply_
        return highstate(**kwargs)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/state.py", line 1154, in highstate
        st_ = salt.state.HighState(
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 5007, in __init__
        self.client = salt.fileclient.get_file_client(self.opts)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 55, in get_file_client
        return {"remote": RemoteClient, "local": FSClient, "pillar": PillarClient}.get(
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1129, in __init__
        self.channel = salt.channel.client.ReqChannel.factory(self.opts)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 56, in factory
        return SyncWrapper(
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 76, in __init__
        self.obj = cls(*args, **kwargs)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 134, in factory
        auth = salt.crypt.AsyncAuth(opts, io_loop=io_loop)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 695, in __new__
        auth.__singleton_init__(opts, io_loop=io_loop)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 742, in __singleton_init__
        self._authenticate_future = tornado.concurrent.Future()
      File "/opt/saltstack/salt/lib/python3.10/asyncio/events.py", line 656, in get_event_loop
        raise RuntimeError('There is no current event loop in thread %r.'
    RuntimeError: There is no current event loop in thread 'ProcessPayload(jid=20240801160004816387)'.
ERROR: Minions returned with non-zero exit code

Expected behavior
state to apply and respond cleanly

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3007.1

Python Version:
        Python: 3.10.14 (main, Apr  3 2024, 21:30:09) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.16.0
      cherrypy: unknown
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.4
       libgit2: 1.7.1
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.1
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: 1.13.1
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.16.0
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.3.3
           ZMQ: 4.3.4

Salt Package Information:
  Package Type: onedir

System Versions:
          dist: debian 11.10 bullseye
        locale: utf-8
       machine: x86_64
       release: 5.10.0-30-amd64
        system: Linux
       version: Debian GNU/Linux 11.10 bullseye

Additional context
downgrading the salt-minion (and thus salt-sproxy) to 3006.9 fixes and state.apply's from the salt 3007.1 master work correctly

[r-mettner-skaylink]

any update here ?
i see a similar issue if i try to use salt.client.Caller() within a thread, looks like a general issue with threading and salt 3007.x.
This worked with 3006.x.

[frebib]

We've just hit more or less the same issue with __slot__s and the fileclient. Applying this patch seems to fix it

diff --git salt/utils/asynchronous.py salt/utils/asynchronous.py
index 316fc5c478d..2db45ec49bc 100644
--- salt/utils/asynchronous.py
+++ salt/utils/asynchronous.py
@@ -73,7 +73,8 @@ class SyncWrapper:
         self.cls = cls
         if loop_kwarg:
             kwargs[self.loop_kwarg] = self.io_loop
-        self.obj = cls(*args, **kwargs)
+        with current_ioloop(self.io_loop):
+            self.obj = cls(*args, **kwargs)
         self._async_methods = list(
             set(async_methods + getattr(self.obj, "async_methods", []))
         )

[r-mettner-skaylink]

^can confirm that diff also fixes my issue with salt.client.Caller() within a thread.

[ITJamie]

@frebib would you consider opening a PR for that or would you be ok with me opening one with that change?

[frebib]

I can open a PR 👍🏻

[dwoz]

The changes from this PR #67096 should also resolve this once merged forward.

[ITJamie]

still likely an issue #67891