salt-ssh: permission denied on thin_dir when set in roster

[baby-gnu]

Description

When I set the thin_dir of a roster host, I can execute state commands:

salt-ssh test-machine-1 state.show_top
[ERROR   ] An Exception occurred while executing state.show_top: [Errno 13] Permission denied: '/root/.cache'
test-machine-1:
    An Exception occurred while executing state.show_top: [Errno 13] Permission denied: '/root/.cache'

Setup

Here is my salt-ssh personal configuration:

~/.salt/Saltfile

# -*- yaml -*-

salt-ssh:
  config_dir: ~/.salt

~/.salt/master

# -*- mode: yaml; coding: utf-8 -*-

####
#### Global parameters
####

# Unfortunately, salt does not support ~ expension
# We need to use absolute path for `root_dir`
root_dir: /home/me/.salt
pki_dir: pki
cachedir: cache
sock_dir: run
pidfile: pids

log_file: logs/master.log
key_logfile: logs/key.log

# Global level logged in file
log_level_logfile: error

# Unfortunately, salt does not support ~ expension
# We need to use absolute path for `file_roots` and `pillar_roots`
file_roots:
  base:
    - /home/me/.salt/srv/salt/

pillar_roots:
  base:
    - /home/me/.salt/srv/pillar/

####
#### Salt-SSH specific configuration
####
ssh_minion_opts:
  log_level: debug

ssh_log_file: logs/ssh.log
ssh_use_home_key: True
ssh_timeout: 5

roster_defaults:
  # Use ssh-agent authentication
  priv: agent-forwarding

~/.salt/roster

test-machine-1:
  host: 192.168.0.100
  user: root
  thin_dir: .cache/salt/thin

Please be as specific as possible and give set-up details.

• on-prem machine
• VM (KVM)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

Steps to Reproduce the behavior

Create the salt-ssh configuration, no need of any top.sls or existing states.

Expected behavior

The state.show_top should return an empty dict:

salt-ssh test-machine-1 state.show_top --output yaml

test-machine-1: {}

Screenshots

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3006.2
 
Python Version:
        Python: 3.10.12 (main, Aug  3 2023, 21:47:10) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: 1.6.4
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: 1.12.1
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.13.3
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: debian n/a trixie
        locale: utf-8
       machine: x86_64
       release: 6.4.0-4-amd64
        system: Linux
       version: Debian GNU/Linux n/a trixie

Additional context

This is a reopen of #46891 which is still valid for 3006.2.

I made different tests:

• /var/cache/salt/thin is not working but the directory is created and populated
• /tmp/thin is working

[depeele]

We are also experiencing this problem with salt-ssh 3005.1

There seem to be permission issues with the thin_dir when using a non-root user in the roster with sudo: true and targeting the host on which salt-ssh is being run.

Ref: #46891 (comment)

Success
If the first such salt-ssh command is performed with roster sudo: false, then all subsequent salt-ssh commands succeed even when changing back to roster sudo: true

Failure
If the first such salt-ssh command is performed with roster sudo: true, then subsequent calls that require access to run state (e.g. salt-ssh 'salt-master' state.show_top) will fail with [Errno 13] Permission denied: /var/tmp/.<user>_<id>_salt/running_data/var.

[vladimir-lu]

This behavior happens even when not targeting the host that is running salt-ssh "master". It's probably due to some user ids being incorrectly synced:

-rw-r--r--    1 1000     1000             1 Jan  3 21:27 .thin-gen-py-version
-rw-r--r--    1 1000     1000            65 Jan  3 21:27 code-checksum
-rw-r--r--    1 1000     1000            40 Jan  3 15:16 ext_version
-rw-r--r--    1 root     root           163 Jan  3 21:27 minion
drwx------    1 root     root           206 Jan  3 21:27 py3
drwx------    1 root     root            48 Jan  3 21:27 pyall
drwx------    1 root     root            32 Jan  3 21:27 running_data
-rw-r--r--    1 1000     1000           757 Jan  3 21:27 salt-call
-rw-r--r--    1 1000     1000             8 Jan  3 21:27 supported-versions
-rw-r--r--    1 1000     1000             6 Jan  3 21:27 version

where 1000 is the user id of the unprivileged user that's running salt-ssh that does not exist on the target (target is running as root with sudo: false). Not setting a persistent thin_dir in the roster does not trigger the error.

Edit: in fact this is the same comment as #46891 (comment)
Edit2: Oh so the actual issue is that salt is trying to create the thin_dir in the localhost environment and obviously my unprivileged user does not have permissions to do it there. If I mount tmpfs on it that seems to be ok - why does it not use the cache dir instead? Weird.

[Rudd-O]

I am hit by the same issue as well.

[Rudd-O]

I can get it to work by running:

sudo chmod -R go+rwX /tmp/salt-thin-root/

(where /tmp/salt-thin-root is the thin_dir I set for the minion.)

Note this problem happens even when no thin_dir is set on the minion or in roster_defaults.

It seems like the upload of the thin files is happening as root, but further consultation of the files down the road (during state execution) happens as the regular user Salt logged in as.

[pmuller]

Same here with salt-ssh 3007.0

[baby-gnu]

Even with 3007.1 without setting thin_dir I got this issue, I tracked it to:

1. the cachedir does not exists when caching grains
2. the second problem I have is the update of roots because __opts__["cachedir"] is set to /var/tmp/.root_8dccac_salt/running_data/var/cache/salt/minion but the update is executed locally

As a temporary workaround, I chown to my user account my local /var/tmp/.root_8dccac_salt/.