Skip to content

Bugfix 67214 minion swarm#67783

Open
dnessett wants to merge 6 commits into
saltstack:masterfrom
dnessett:bugfix-67214
Open

Bugfix 67214 minion swarm#67783
dnessett wants to merge 6 commits into
saltstack:masterfrom
dnessett:bugfix-67214

Conversation

@dnessett

@dnessett dnessett commented Mar 1, 2025

Copy link
Copy Markdown

What does this PR do?

Fixes bugs in tests/minionswarm.py that were preventing it to run.

What issues does this PR fix or reference?

Fixes #67214

Previous Behavior

minionswarm failed to run

New Behavior

minionswarm now runs correctly

Merge requirements satisfied?

[NOTICE] Bug fixes or features added to Salt require tests.

Commits signed with GPG?

No

@dnessett dnessett requested a review from a team as a code owner March 1, 2025 20:35
@welcome

welcome Bot commented Mar 1, 2025

Copy link
Copy Markdown

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject.pdl@broadcom.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

Comment thread tests/support/runtests.py Outdated
import salt.utils.path
import salt.utils.platform
import tests.support.paths as paths
import support.paths as paths

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this needed?

@dnessett dnessett Mar 8, 2025

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The original threw an unfound error. runtests.py exists within salt/tests/support. Relative to runtests.py, paths is not located at tests.support.paths, but rather at support.paths. This may have resulted from a reorganization of the directory structure sometime after the 2015 release in which minionswarm.py and runtests.py were moved (that's a guess).

@dnessett dnessett Mar 8, 2025

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just spent some time reading about relative pathnames in the import statement. I used "import support.paths" simply because it worked. However, after reading, I'm not sure why it works. Perhaps "import paths" would be better, since paths.py exists in the same directory as runtests.py.

Added later: OK, I guess I am getting above my head on this. I just tried changing "import support.paths as paths" to "import paths" and minionswarm.py threw an unfound error. So, someone who has more experience with import statements will have to advise on the best way to import paths.py when it is in the same directory as runtests.py

@dwoz dwoz changed the title Bugfix 67214 Bugfix 67214 minion swarm Mar 7, 2025
@dmurphy18 dmurphy18 added this to the Argon v3008.0 milestone Mar 13, 2025
@twangboy twangboy added the test:full Run the full test suite label May 14, 2025
twangboy
twangboy previously approved these changes May 14, 2025
@twangboy

Copy link
Copy Markdown
Contributor

Looks like there are some pre-commit failures.

@dnessett

Copy link
Copy Markdown
Author

Looks like there are some pre-commit failures.

OK. I will take a look at it. It may take some time since I am reorganizing my development environment.

@dnessett

Copy link
Copy Markdown
Author

I am very new to salt development, so I very possibly am missing something, I ran nox -e lint-tests in bugfix-67214, which is what is pushed to https://github.com/dnessett/salt and what is the basis of the PR. The results are:

(salt) dan@Eclipse-salt:~/bugfix-67214/salt$ nox -e lint-tests
nox > Running session lint-tests
nox > Creating virtual environment (virtualenv) using python3 in .nox/lint-tests
nox > python -m pip install --progress-bar=off -U setuptools pip wheel
nox > python -m pip install --progress-bar=off -r requirements/static/ci/py3.10/linux.txt -r requirements/static/ci/py3.10/lint.txt
nox > pylint --rcfile=.pylintrc --disable=I tests/

------------------------------------
Your code has been rated at 10.00/10

nox > Session lint-tests was successful.

To check I was in the correct branch, I ran:

(salt) dan@Eclipse-salt:~/bugfix-67214/salt$ git log
commit aeb11b3e615066804d9debd581b8663234a4bffc (HEAD -> bugfix-67214, origin/bugfix-67214)
Author: dnessett <dnessett@yahoo.cm>
Date:   Sat Mar 1 13:24:48 2025 -0700

    Eliminated some trailing whitespace errors revealed by nox -e lint-tests

commit ed91f4ceecfb92f773096951dd8a2ac4ee1f73a8
Author: dnessett <dnessett@yahoo.cm>
Date:   Sat Mar 1 12:24:11 2025 -0700

    added changelog file

commit 05f5e132d2d90844df3fbc432802a864c42c8dc2
Author: dnessett <dnessett@yahoo.cm>
Date:   Fri Feb 28 13:19:32 2025 -0700

    cleaned up some problems with the parser.add_options and added a
    clarificationn to the config-dir help text.

commit 85afd494938c53934e941eb7cb658a9cbcd42062
Author: dnessett <dnessett@yahoo.cm>
Date:   Thu Feb 27 11:31:06 2025 -0700

    Fix bugs in minionswarm. Updated help with significant enhancements. Added username to salt-master command. Added ability to turn off open_mode.

commit 2530c8f1d5b1440b70860d4dea453b6657bdf92b
Author: Daniel A. Wozniak <daniel.wozniak@broadcom.com>
Date:   Wed May 7 01:33:13 2025 -0700

    Do not fail when cached grains no longer exit

and:

(salt) dan@Eclipse-salt:~/bugfix-67214/salt$ git branch
* bugfix-67214
  master

I then checked the changelog directory in case the changelog file was faulty, but it seems not:

(salt) dan@Eclipse-salt:~/bugfix-67214/salt/changelog$ cat 67214.fixed.md 
Fixed bugs in minionswarm and added ability turn off open-mode.

So, I am stuck finding out what is causing pre-commit to fail.

@lkubb

lkubb commented May 18, 2025

Copy link
Copy Markdown
Contributor

lint-tests/lint-salt run pylint only. The pre-commit hooks that are failing are black/isort though. You'd run them via

pre-commit run black --files tests/minionswarm.py,tests/support/runtests.py
pre-commit run isort --files tests/minionswarm.py,tests/support/runtests.py

In general, you should consider installing the pre-commit hook (the git hook) in your local salt checkout for pre-commit (the program) via pre-commit install (or python -m pre_commit install, if you're in a venv and want to ensure you're using the pre-commit that's installed inside the venv). This would ensure you cannot commit before all hooks are fixed, thus avoiding the failing check in CI entirely.

@dnessett

Copy link
Copy Markdown
Author

lint-tests/lint-salt run pylint only. The pre-commit hooks that are failing are black/isort though. You'd run them via

pre-commit run black --files tests/minionswarm.py,tests/support/runtests.py pre-commit run isort --files tests/minionswarm.py,tests/support/runtests.py

In general, you should consider installing the pre-commit hook (the git hook) in your local salt checkout for pre-commit (the program) via pre-commit install (or python -m pre_commit install, if you're in a venv and want to ensure you're using the pre-commit that's installed inside the venv). This would ensure you cannot commit before all hooks are fixed, thus avoiding the failing check in CI entirely.

Thanks.

@dnessett

Copy link
Copy Markdown
Author

lint-tests/lint-salt run pylint only. The pre-commit hooks that are failing are black/isort though. You'd run them via
pre-commit run black --files tests/minionswarm.py,tests/support/runtests.py pre-commit run isort --files tests/minionswarm.py,tests/support/runtests.py
In general, you should consider installing the pre-commit hook (the git hook) in your local salt checkout for pre-commit (the program) via pre-commit install (or python -m pre_commit install, if you're in a venv and want to ensure you're using the pre-commit that's installed inside the venv). This would ensure you cannot commit before all hooks are fixed, thus avoiding the failing check in CI entirely.
pre-commit run black --files tests/minionswarm.py,tests/support/runtests.py pre-commit run isort --files tests/minionswarm.py,tests/support/runtests.py
Thanks.

Sorry for the delay, I had some emergencies to resolve in another area of my volunteer work.

When I run: pre-commit run black/isort
The result is:

(salt) dan@Eclipse-salt:~/bugfix-67214/salt$ pre-commit run black --files tests/minionswarm.py,tests/support/runtests.py
black................................................(no files to check)Skipped
(salt) dan@Eclipse-salt:~/bugfix-67214/salt$ pre-commit run isort --files tests/minionswarm.py,tests/support/runtests.py
isort................................................(no files to check)Skipped
(salt) dan@Eclipse-salt:~/bugfix-67214/salt$ 

There doesn't seem to be any errors from black or isort. Perhaps this is because the two files in question have been committed and so black/isort doesn't regard them as changed? (This is a guess, since I don't know what black/iisort checks for)

@lkubb

lkubb commented May 21, 2025

Copy link
Copy Markdown
Contributor

@dnessett $PWD being ~/bugfix-67214/salt indicates to me you're inside the salt subdirectory of the repository when running the command, not in the repository root, which the paths I gave you are relative to. From pre-commit's perspective, they don't exist, so there's nothing to do. It's not about being committed, this generally works.

@dnessett

dnessett commented May 21, 2025

Copy link
Copy Markdown
Author

@dnessett $PWD being ~/bugfix-67214/salt indicates to me you're inside the salt subdirectory of the repository when running the command, not in the repository root, which the paths I gave you are relative to. From pre-commit's perspective, they don't exist, so there's nothing to do. It's not about being committed, this generally works.

I'm in the git source directory when I execute the black/isort functions. I am reorganizing my development environment, so I cloned a copy of my github repository to a directory in my home directory in order to address this problem while the reorganization is in process.

(salt) dan@Eclipse-salt:~/bugfix-67214/salt$ ls
AUTHORS       CODE_OF_CONDUCT.md  doc          noxfile.py   pyproject.toml  rfcs         setup.cfg    tests
changelog     conf                LICENSE      pkg          pytest.ini      salt         setup.py     tools
CHANGELOG.md  CONTRIBUTING.rst    MANIFEST.in  POLICY.rst   README.rst      scripts      SUPPORT.rst
cicd          DEPENDENCIES.md     NOTICE       __pycache__  requirements    SECURITY.md  templates
(salt) dan@Eclipse-salt:~/bugfix-67214/salt$ pre-commit run black --files tests/minionswarm.py,tests/support/runtests.py
black................................................(no files to check)Skipped
(salt) dan@Eclipse-salt:~/bugfix-67214/salt$ pre-commit run isort --files tests/minionswarm.py,tests/support/runtests.py
isort................................................(no files to check)Skipped
(salt) dan@Eclipse-salt:~/bugfix-67214/salt$

Just to demonstrate the files are in the right place:

(salt) dan@Eclipse-salt:~/bugfix-67214/salt/tests$ ls
buildpackage.py      eventlisten.py    __init__.py     modparser.py  runtests.py      support       zypp_plugin.py
committer_parser.py  eventlisten.sh    integration     packdump.py   saltsh.py        unit
conftest.py          filename_map.yml  minionswarm.py  pytests       salt-tcpdump.py  wheeltest.py
(salt) dan@Eclipse-salt:~/bugfix-67214/salt/tests$

(salt) dan@Eclipse-salt:~/bugfix-67214/salt/tests$ cd support
(salt) dan@Eclipse-salt:~/bugfix-67214/salt/tests/support$ ls
case.py         gitfs.py     kernelpkg.py  napalm.py  pkg.py       sminion.py  win_installer.py
cli_scripts.py  helpers.py   mixins.py     netapi.py  pytest       unit.py     xmlunit.py
events.py       __init__.py  mock.py       paths.py   runtests.py  virt.py     zfs.py
(salt) dan@Eclipse-salt:~/bugfix-67214/salt/tests/support$ 


Comment thread tests/minionswarm.py
default="minion",
help="Give the minions an alternative id prefix, this is used "
"when minions from many systems are being aggregated onto "
"a single master. (default = minion)",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't duplicate a default value in the help text of an OptionParser argument (the same applies to ArgumentParser). Use %default in the help text. With ArgumentParser you can show the defaults for all arguments by setting formatter_class=argparse.ArgumentDefaultsHelpFormatter. I am not sure if OptionParser has similar functionality or not since I haven't really used because ArgumentParser superseded it over a decade ago.

dnessett and others added 6 commits June 8, 2026 22:09
…Added username to salt-master command. Added ability to turn off open_mode.
clarificationn to the config-dir help text.
Address review feedback on saltstack#67214:

* tests/support/runtests.py: restore canonical package-qualified
  import. ``import support.paths as paths`` only resolved when the
  CWD was ``tests/`` and broke ``from tests.support.runtests import
  RUNTIME_VARS`` -- used by 50+ conftests/tests. Use
  ``from tests.support import paths`` instead. This was the root
  cause of the 393 failing Test Salt / Test Package jobs.
* tests/minionswarm.py: use ``from tests.support import runtests``
  and add a sys.path bootstrap so the script can still be executed
  directly via ``python tests/minionswarm.py`` (the original
  ``cd tests/`` reproduction from saltstack#67214 also still works).
* tests/minionswarm.py: collapse duplicated ``machine_id``
  assignment, fix invalid ``sys.exit([1])`` -> ``sys.exit(1)``.
* tests/minionswarm.py: master ``sock_dir`` pointed at
  ``var/run/salt/minion`` instead of ``var/run/salt/master``.
* tests/minionswarm.py: ``--open-mode`` was declared with
  ``default=True`` and no action, so ``--open-mode=False`` parsed
  to the string ``"False"`` (truthy). Switched to
  ``--no-open-mode`` (action="store_false") so the flag does what
  the help text claims.
* tests/minionswarm.py: typo fixes (``/tmp/srooot`` -> ``/tmp/sroot``,
  ``manaaged`` -> ``managed``, ``alt-master`` -> ``salt-master``).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

has-failing-test test:full Run the full test suite

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Executing tests/minionswarm.py fails

6 participants