[BUG] salt-api server dies on bad /login request

[carrysauce]

Description
Salt-master times out when doing salt-api /auth endpoint request without username or password fields but with eauth field present. Can lead to salt-api DDoS.

Setup
Really the most generic salt setup you can ever create, just enabled salt-api with external_auth (config below)

• VM KVM Master and slave

Steps to Reproduce the behavior

• Install salt-master, salt-minion and salt-api packages
• Enable external_auth in master config

Example master config

external_auth:
  auto:
    admin:
      - .*

rest_cherrypy:
  port: 8000
  disable_ssl: True

- Try to do salt-api `/login` request without password field

Example curl

curl -X "POST" "http://{{salt_endpoint}}:8000/login" -H 'Content-Type: application/json; charset=utf-8' -d '{"eauth": "auto",  "username": "admin","token": "whatever"}'

- See salt-api getting stuck, master errors in logs, retries in salt-api (every retry takes 1 minute - why is this default?)

Salt-api stuck logs

2022-06-17 10:20:11,965 [salt.loaded.int.netapi.rest_cherrypy.app:747 ][INFO    ][3027991] [api_acl] Authentication not checked for user admin from IP 192.168.41.101
2022-06-17 10:20:11,974 [salt.transport.zeromq:158 ][DEBUG   ][3027991] Connecting the Minion to the Master URI (for the return server): tcp://127.0.0.1:4506
2022-06-17 10:20:11,976 [salt.transport.zeromq:1233][DEBUG   ][3027991] Trying to connect to: tcp://127.0.0.1:4506
2022-06-17 10:21:12,036 [salt.transport.zeromq:1291][DEBUG   ][3027991] SaltReqTimeoutError, retrying. (1/3)
2022-06-17 10:22:12,094 [salt.transport.zeromq:1291][DEBUG   ][3027991] SaltReqTimeoutError, retrying. (2/3)
2022-06-17 10:23:12,156 [salt.transport.zeromq:1291][DEBUG   ][3027991] SaltReqTimeoutError, retrying. (3/3)
2022-06-17 10:24:12,212 [salt.transport.zeromq:1261][DEBUG   ][3027991] Re-init ZMQ socket: Message timed out
2022-06-17 10:24:12,214 [salt.transport.zeromq:1233][DEBUG   ][3027991] Trying to connect to: tcp://127.0.0.1:4506
2022-06-17 10:24:12,216 [salt.transport.zeromq:179 ][DEBUG   ][3027991] Closing AsyncZeroMQReqChannel instance
2022-06-17 10:24:12,218 [cherrypy.access.140230875023728:284 ][INFO    ][3027991] 192.168.41.101 - - [17/Jun/2022:10:24:12] "POST /login HTTP/1.1" 503 742 "" "curl/7.79.1"

Salt-master logs

022-06-17 10:20:09,059 [salt.config      :1917][DEBUG   ][3027939] Reading configuration from /etc/salt/master
2022-06-17 10:20:09,062 [salt.config      :2080][DEBUG   ][3027939] Including configuration from '/etc/salt/minion.d/_schedule.conf'
2022-06-17 10:20:09,062 [salt.config      :1917][DEBUG   ][3027939] Reading configuration from /etc/salt/minion.d/_schedule.conf
2022-06-17 10:20:09,063 [salt.config      :2080][DEBUG   ][3027939] Including configuration from '/etc/salt/minion.d/local_master.conf'
2022-06-17 10:20:09,063 [salt.config      :1917][DEBUG   ][3027939] Reading configuration from /etc/salt/minion.d/local_master.conf
2022-06-17 10:20:09,064 [salt.config      :3520][DEBUG   ][3027939] Using cached minion ID from /etc/salt/minion_id: vm-3d-salt-test.pine.ly
2022-06-17 10:20:09,066 [salt.loader      :868 ][DEBUG   ][3027939] Grains refresh requested. Refreshing grains.
2022-06-17 10:20:09,066 [salt.config      :1917][DEBUG   ][3027939] Reading configuration from /etc/salt/master
2022-06-17 10:20:09,086 [salt.utils.entrypoints:57  ][DEBUG   ][3027939] Using importlib_metadata to load entry points
2022-06-17 10:20:09,141 [salt.loader.lazy :857 ][DEBUG   ][3027939] Override  __utils__: <module 'salt.loaded.int.grains.zfs' from '/usr/lib/python3/dist-packages/salt/grains/zfs.py'>
2022-06-17 10:20:09,170 [salt.modules.network:2132][DEBUG   ][3027939] Elapsed time getting FQDNs: 0.017315387725830078 seconds
2022-06-17 10:20:09,335 [salt.utils.lazy  :99  ][DEBUG   ][3027939] LazyLoaded zfs.is_supported
2022-06-17 10:20:09,398 [salt.utils.lazy  :99  ][DEBUG   ][3027939] LazyLoaded local_cache.clean_old_jobs
2022-06-17 10:20:09,435 [salt.utils.lazy  :99  ][DEBUG   ][3027939] LazyLoaded localfs.list_tokens
2022-06-17 10:20:09,437 [salt.utils.verify:454 ][DEBUG   ][3027939] This salt-master instance has accepted 1 minion keys.
2022-06-17 10:20:11,981 [salt.utils.lazy  :99  ][DEBUG   ][3027950] LazyLoaded auto.auth
2022-06-17 10:20:11,982 [salt.transport.zeromq:795 ][ERROR   ][3027950] Some exception handling a payload from minion
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/transport/zeromq.py", line 791, in handle_message
    ret, req_opts = yield self.payload_handler(payload)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 294, in wrapper
    result = func(*args, **kwargs)
  File "/usr/lib/python3.8/types.py", line 278, in wrapped
    coro = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/master.py", line 1023, in _handle_payload
    ret = {"aes": self._handle_aes, "clear": self._handle_clear}[key](load)
  File "/usr/lib/python3/dist-packages/salt/master.py", line 1064, in _handle_clear
    ret = method(load), {"fun": "send_clear"}
  File "/usr/lib/python3/dist-packages/salt/master.py", line 2130, in mk_token
    token = self.loadauth.mk_token(clear_load)
  File "/usr/lib/python3/dist-packages/salt/auth/__init__.py", line 209, in mk_token
    if not self.authenticate_eauth(load):
  File "/usr/lib/python3/dist-packages/salt/auth/__init__.py", line 312, in authenticate_eauth
    if not self.time_auth(load):
  File "/usr/lib/python3/dist-packages/salt/auth/__init__.py", line 115, in time_auth
    ret = self.__auth_call(load)
  File "/usr/lib/python3/dist-packages/salt/auth/__init__.py", line 98, in __auth_call
    fcall = salt.utils.args.format_call(
  File "/usr/lib/python3/dist-packages/salt/utils/args.py", line 470, in format_call
    raise SaltInvocationError(
salt.exceptions.SaltInvocationError: auth takes at least 2 arguments (1 given)
2022-06-17 10:20:11,990 [tornado.general  :467 ][ERROR   ][3027950] Uncaught exception in zmqstream callback
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/zmq/sugar/socket.py", line 435, in send_multipart
    memoryview(msg)
TypeError: memoryview: a bytes-like object is required, not 'str'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/zmq/eventloop/zmqstream.py", line 460, in _handle_events
    self._handle_send()
  File "/usr/lib/python3/dist-packages/zmq/eventloop/zmqstream.py", line 499, in _handle_send
    status = self.socket.send_multipart(msg, **kwargs)
  File "/usr/lib/python3/dist-packages/zmq/sugar/socket.py", line 440, in send_multipart
    raise TypeError(
TypeError: Frame 0 ('Some exception handling minion ...) does not support the buffer interface.
2022-06-17 10:20:11,994 [tornado.application:640 ][ERROR   ][3027950] Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x7f8617aeb5e0>)
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/zmq/sugar/socket.py", line 435, in send_multipart
    memoryview(msg)
TypeError: memoryview: a bytes-like object is required, not 'str'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 606, in _run_callback
    ret = callback()
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
    return fn(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/zmq/eventloop/zmqstream.py", line 542, in <lambda>
    self.io_loop.add_callback(lambda : self._handle_events(self.socket, 0))
  File "/usr/lib/python3/dist-packages/zmq/eventloop/zmqstream.py", line 460, in _handle_events
    self._handle_send()
  File "/usr/lib/python3/dist-packages/zmq/eventloop/zmqstream.py", line 499, in _handle_send
    status = self.socket.send_multipart(msg, **kwargs)
  File "/usr/lib/python3/dist-packages/zmq/sugar/socket.py", line 440, in send_multipart
    raise TypeError(
TypeError: Frame 0 ('Some exception handling minion ...) does not support the buffer interface.

- All further requests to salt-api via rest_cherrypy salt-api server become stuck - Further bogus requests prolong server downtime - If we stop doing bad requests server will come back

Expected behavior

• Do bad request to salt-api (/login without password field)
• See bad request error and not connection getting stuck

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3004.1
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: unknown
      dateutil: 2.7.3
     docker-py: Not Installed
         gitdb: 2.0.6
     gitpython: 3.0.7
        Jinja2: 2.10.1
       libgit2: 0.28.3
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: 2.6.1
  pycryptodome: 3.6.1
        pygit2: 1.0.3
        Python: 3.8.10 (default, Mar 15 2022, 12:22:08)
  python-gnupg: 0.4.5
        PyYAML: 5.3.1
         PyZMQ: 18.1.1
         smmap: 2.0.5
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.2
 
System Versions:
          dist: ubuntu 20.04 focal
        locale: utf-8
       machine: x86_64
       release: 5.13.0-41-generic
        system: Linux
       version: Ubuntu 20.04 focal

Additional context
Problem can be easily solved by placing try block before format_call() func here: https://github.com/saltstack/salt/blob/master/salt/auth/__init__.py#L98-L101

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[carrysauce]

Hey, kindly reminder, is it possible someone look into this report?

On my setup salt-api gets stuck for all incoming requests, could easily lead to ddos :(

[Sxderp]

For what it's worth. I'm not seeing this crash on 3007.11.