[BUG] file.exists + "retry"+ "parallel" in SLS causes recursion error

[ReubenM]

Description
When using multiple names file.exists in combination with retries and parallel, a recursion error takes place

Setup
The following will cause an exception:

pre_tomcat:
  module.run:
    - service.systemctl_reload:

tomcat:
  service.running:
    - enable: False
    - require:
      - module: pre_tomcat

tomcat_logs:
  file.exists:
    - names:
      - "/usr/cps/apache-tomcat/tomcat.pid"
      - "/var/cps/access_log.txt"
      - "/var/cps/catalina.log"
      - "/var/cps/catalina.out"
    - retry:
        attempts: 300
        interval: 1
    - parallel: True
    - require:
      - service: tomcat

The following with the retry section removed will return results:

pre_tomcat:
  module.run:
    - service.systemctl_reload:

tomcat:
  service.running:
    - enable: False
    - require:
      - module: pre_tomcat

tomcat_logs:
  file.exists:
    - names:
      - "/usr/cps/apache-tomcat/tomcat.pid"
      - "/var/cps/access_log.txt"
      - "/var/cps/catalina.log"
      - "/var/cps/catalina.out"
    - parallel: True
    - require:
      - service: tomcat

Trace

   The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/salt/minion.py", line 1913, in _thread_return
        function_name, function_args, executors, opts, data
      File "/usr/lib/python3.6/site-packages/salt/minion.py", line 1869, in _execute_job_function
        return_data = self.executors[fname](opts, data, func, args, kwargs)
      File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
        return self.loader.run(run_func, *args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
        return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
      File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
        return callable(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
        return _func_or_method(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/executors/direct_call.py", line 10, in execute
        return func(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
        return self.loader.run(run_func, *args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
        return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
      File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
        return callable(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
        return _func_or_method(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/modules/state.py", line 1392, in sls
        ret = st_.state.call_high(high_, orchestration_jid)
      File "/usr/lib/python3.6/site-packages/salt/state.py", line 3279, in call_high
        ret = self.call_chunks(chunks)
      File "/usr/lib/python3.6/site-packages/salt/state.py", line 2497, in call_chunks
        running = self.call_chunk(low, running, chunks)
      File "/usr/lib/python3.6/site-packages/salt/state.py", line 3003, in call_chunk
        running[tag] = self.call(low, chunks, running)
      File "/usr/lib/python3.6/site-packages/salt/utils/decorators/state.py", line 45, in _func
        result = func(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/state.py", line 2270, in call
        retry_ret = self.call(low, chunks, running, retries=retries + 1)
[ ... ]
     File "/usr/lib/python3.6/site-packages/salt/state.py", line 2270, in call
        retry_ret = self.call(low, chunks, running, retries=retries + 1)
      File "/usr/lib/python3.6/site-packages/salt/utils/decorators/state.py", line 45, in _func
        result = func(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/state.py", line 2267, in call
        interval,
      File "/usr/lib64/python3.6/logging/__init__.py", line 1308, in info
        self._log(INFO, msg, args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/_logging/impl.py", line 298, in _log
        stack_info=stack_info,
      File "/usr/lib64/python3.6/logging/__init__.py", line 1443, in _log
        exc_info, func, extra, sinfo)
      File "/usr/lib/python3.6/site-packages/salt/_logging/impl.py", line 368, in makeRecord
        self, name, level, fn, lno, _msg, _args, exc_info, func, sinfo
      File "/usr/lib64/python3.6/logging/__init__.py", line 1413, in makeRecord
        sinfo)
      File "/usr/lib/python3.6/site-packages/salt/_logging/impl.py", line 129, in __init__
        SaltLogRecord.__init__(self, *args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/_logging/impl.py", line 121, in __init__
        logging.LogRecord.__init__(self, *args, **kwargs)
      File "/usr/lib64/python3.6/logging/__init__.py", line 277, in __init__
        if (args and len(args) == 1 and isinstance(args[0], collections.Mapping)
      File "/usr/lib64/python3.6/abc.py", line 184, in __instancecheck__
        if subclass in cls._abc_cache:
    RecursionError: maximum recursion depth exceeded

Versions Report
Salt Version:
Salt: 3004.2

Dependency Versions:
cffi: 1.9.1
cherrypy: Not Installed
dateutil: 2.8.2
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.11.1
libgit2: Not Installed
M2Crypto: 0.35.2
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.14
pycrypto: Not Installed
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.6.8 (default, Nov 18 2021, 10:07:16)
python-gnupg: Not Installed
PyYAML: 3.13
PyZMQ: 17.0.0
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.1.4

System Versions:
dist: ol 7.9
locale: UTF-8
machine: x86_64
release: 5.4.17-2136.305.5.3.el7uek.x86_64
system: Linux
version: Oracle Linux Server 7.9

[ReubenM]

I initially thought the issue was the combination of names and file.exists since I was sure I had used file.exists with names elsewhere, but it was actually file.managed.

Realized this was not the case when my attempt to use jinja templating for a work-around also causes a recursion exceptiion:

pre_tomcat:
  module.run:
    - service.systemctl_reload:

tomcat:
  service.running:
    - enable: False
    - require:
      - module: pre_tomcat

{%- for file in [ "/usr/cps/apache-tomcat/tomcat.pid", "/var/cps/access_log.txt", "/var/cps/catalina.log", "/var/cps/catalina.out" ] %}

{{ file }}:
  file.exists:
    - parallel: True
    - retry:
        attempts: 300
        interval: 1
    - require:
      - service: tomcat

{%- endfor %}

[ReubenM]

One more update:
Seems to be the combination of "retry" and "parallel" withing "file.exists"

This works:

re_tomcat:
  module.run:
    - service.systemctl_reload:

tomcat:
  service.running:
    - enable: False
    - require:
      - module: pre_tomcat

tomcat_logs:
  file.exists:
    - names:
      - "/usr/cps/apache-tomcat/tomcat.pid"
      - "/var/cps/access_log.txt"
      - "/var/cps/catalina.log"
      - "/var/cps/catalina.out"
    - retry:
        attempts: 300
        interval: 1
    - require:
      - service: tomcat

[ReubenM]

trace from minion:

  File "/usr/lib/python3.6/site-packages/salt/utils/process.py", line 1034, in wrapped_run_func
    return run_func()
  File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/site-packages/salt/state.py", line 1990, in _call_parallel_target
    self.format_slots(cdata)
  File "/usr/lib/python3.6/site-packages/salt/state.py", line 2371, in format_slots
    arg = salt.utils.data.decode(arg, keep=True)
  File "/usr/lib/python3.6/site-packages/salt/utils/data.py", line 248, in decode
    if isinstance(data, Mapping):
  File "/usr/lib64/python3.6/abc.py", line 184, in __instancecheck__
    if subclass in cls._abc_cache:
  File "/usr/lib64/python3.6/_weakrefset.py", line 72, in __contains__
    wr = ref(item)

[co-cy]

Hi, I'm not 100% sure, but this might be related to #69028. Was there high CPU usage and minion freezes?

I opened an issue about this: #68940