Skip to content

[Bug]: Minion process/service can't be stopped with SIGTERM when stuck at master DNS lookup #69466

@hurzhurz

Description

@hurzhurz

What happened?

When the minion is started and a hostname is configured for master, it repeatedly tries to do a DNS lookup.
In this state, it is not possible to stop it with a SIGTERM, a hard SIGKILL is needed.

For example, if you freshly install a minion from a package, the service is usually automatically started.
With the default configuration, it wants to connect to the master via the hostname salt by default.
If this hostname doesn't exist, the log repeatedly shows these messages:

[ERROR   ] DNS lookup or connection check of 'salt' failed.
[ERROR   ] Master hostname: 'salt' not found or not responsive. Retrying in 30 seconds

If you then change the minion config to the correct master address and want to restart the service, e.g. via systemctl restart salt-minion, it doesn't work properly:
systemd tries to stop the process by sending a SIGTERM. After 90 seconds it gives up, uses a SIGKILL and marks the service as failed.

This behavior seems to be the case since 3006.15, I suspect this change: 5c8df36 (PR #68209)
@barneysowood maybe you could have a look at this?

The async stop probably doesn't work because the io_loop is stuck here:

salt/salt/minion.py

Lines 142 to 160 in 0a9f560

while True:
if retry_dns_count is not None:
if retry_dns_count == 0:
raise SaltMasterUnresolvableError
retry_dns_count -= 1
log.error(
"Master hostname: '%s' not found or not responsive. "
"Retrying in %s seconds",
opts["master"],
opts["retry_dns"],
)
time.sleep(opts["retry_dns"])
try:
ret["master_ip"] = salt.utils.network.dns_check(
opts["master"], int(opts["master_port"]), True, opts["ipv6"]
)
break
except SaltClientError:
pass

Type of salt install

Official deb

Major version

3006.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

ubuntu-24.04

salt --versions-report output

Salt Version:
          Salt: 3006.25

Python Version:
        Python: 3.10.20 (main, Apr 22 2026, 05:54:37) [GCC 11.2.0]

Dependency Versions:
          cffi: 2.0.0
      cherrypy: unknown
  cryptography: 46.0.7
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: 4.0.12
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.23.0
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.22.8
         smmap: 5.0.2
       timelib: 0.3.0
       Tornado: 6.5.5
           ZMQ: 4.3.4

System Versions:
          dist: ubuntu 24.04.4 noble
        locale: utf-8
       machine: x86_64
       release: 6.18.33.1-microsoft-standard-WSL2
        system: Linux
       version: Ubuntu 24.04.4 noble

Metadata

Metadata

Assignees

Labels

bugbroken, incorrect, or confusing behaviorneeds-triage

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions