No such file or directory : ~/.ansible/tmp/ansible-tmp-.../wait_for.py

[cdggcali]

Hi,

Im testing to stackstorm-ansible to test if we could use it to update Ubuntu servers.

I have a good know playbook that I can successfully run using the standard ansible-playbook command, to run that playbook using Strackstorm Im using the following command:

# st2 run ansible.playbook playbook=/root/git/stash/linux-update/linux-update.yml inventory_file=/root/git/stash/linux-update/linux-update.ini

The task is the following:

---
  - name: Wait for host to boot
    become: false
    local_action: wait_for
    args:
      host: "{{ inventory_hostname }}"
      port: 22
      state: started
      delay: 30
      timeout: 200

And it is failing with the following error:

TASK [linux-update : Wait for host to boot] ************************************
fatal: [192.168.56.101]: FAILED! =>
{"msg": "failed to transfer file to ~/.ansible/tmp/ansible-tmp-1524245385.09-134125845791648/wait_for.py:
[Errno 2] No such file or directory: '~/.ansible/tmp/ansible-tmp-1524245385.09-134125845791648/wait_for.py'"}
	to retry, use: --limit @/root/git/stash/linux-update/linux-update.retry

Is it possible the path is not correct since Im using Stackstrom?

Regards,
Carlos

Update: The Workaround/Solution

There is a bug in Ansible (ansible/ansible#21562) and they got really a lot of reports like this. They're trying to put everything together and fix it in some future release: ansible/ansible#31022

Until that:

Workaround 1

Set in ansible.cfg:

remote_tmp = $HOME/.ansible/tmp
local_tmp = $HOME/.ansible/tmp

Workaround 2

Pass those environment variables to ansible command:

ANSIBLE_LOCAL_TEMP=$HOME/.ansible/tmp
ANSIBLE_REMOTE_TEMP=$HOME/.ansible/tmp

Workaround 3

Downgrade of Ansible to version 2.2.0.

[cognifloyd]

What version of ansible are you using manually?
What version of ansible is installed with StackStorm?

Maybe try testing with StackStorm's copy of ansible: /opt/stackstorm/virtualenvs/ansible/bin/ansible

[arm4b]

That error looks familiar to me from Ansible, see reports:

• https://stackoverflow.com/questions/23899028/ansible-failed-to-transfer-file-to-command
• failed to transfer file to ~/.ansible/tmp/ansible-tmp-xxx/setup.py: [Errno 2] No such file or directory ansible/ansible#21562
• file transfer failure after upgrade from 2.2.0 to 2.2.1 ansible/ansible#20332

where people are still having this issue.

I think it depends on a mix of local/remote server configuration, Ansible version, ssh settings and so on.

---

As @cognifloyd noted, first thing is that StackStorm runs Ansible binary directly, installed into isolated virtualenv:

/opt/stackstorm/virtualenvs//ansible/bin/ansible-playbook

The main idea is: if you could run that binary, - StackStorm would be able to run too.

Having that path to executable, - try to run the same command with your params from CLI, add more verbosity to reveal what's happening behind the hood. Is the same command running manually failing or not?
Try to run the same command from stanley user, - is there any diff?

Hope that helps as a starting point for debugging this.

[cdggcali]

@cognifloyd

What version of ansible is installed with StackStorm?

I'm testing using the Docker image and it looks like it uses 2.5.1

root@a0108173f049:~/git/stash/linux-update# /opt/stackstorm/virtualenvs/ansible/bin/ansible-playbook --version
ansible-playbook 2.5.1
...

And just to let you know, running the playbook just using the ansible-playbook on the virtualenv it works:

root@a0108173f049:~/git/stash/linux-update# /opt/stackstorm/virtualenvs/ansible/bin/ansible-playbook -i linux-update.ini linux-update.yml

PLAY [all] *********************************************************************************************************************************************************************************************************

TASK [Gathering Facts] *********************************************************************************************************************************************************************************************
ok: [192.168.56.101]

TASK [linux-update : Check /boot available space] ******************************************************************************************************************************************************************
skipping: [192.168.56.101] => (item={u'block_used': 602333, u'uuid': u'8889ebb8-ee3b-4408-bed5-747f4795aa85', u'size_total': 5699186688, u'block_total': 1391403, u'inode_available': 261526, u'block_available': 789070, u'size_available': 3232030720, u'fstype': u'ext4', u'inode_total': 362160, u'options': u'rw,relatime,errors=remount-ro,data=ordered', u'device': u'/dev/mapper/repo01--vg-root', u'inode_used': 100634, u'block_size': 4096, u'mount': u'/'})
ok: [192.168.56.101] => (item={u'block_used': 133598, u'uuid': u'ab9a187e-d52a-41d6-8c6a-244ebc0cfca6', u'size_total': 494512128, u'block_total': 482922, u'inode_available': 124618, u'block_available': 349324, u'size_available': 357707776, u'fstype': u'ext2', u'inode_total': 124928, u'options': u'rw,relatime,block_validity,barrier,user_xattr,acl', u'device': u'/dev/sda1', u'inode_used': 310, u'block_size': 1024, u'mount': u'/boot'}) => {
"changed": false,
"item": {
"block_available": 349324,
"block_size": 1024,
"block_total": 482922,
"block_used": 133598,
"device": "/dev/sda1",
"fstype": "ext2",
"inode_available": 124618,
"inode_total": 124928,
"inode_used": 310,
"mount": "/boot",
"options": "rw,relatime,block_validity,barrier,user_xattr,acl",
"size_available": 357707776,
"size_total": 494512128,
"uuid": "ab9a187e-d52a-41d6-8c6a-244ebc0cfca6"
},
"msg": "All assertions passed"
}

TASK [linux-update : Running apt autoremove] ***********************************************************************************************************************************************************************
skipping: [192.168.56.101]

TASK [linux-update : Refresh apt cache] ****************************************************************************************************************************************************************************
changed: [192.168.56.101]

TASK [linux-update : Update all packages] **************************************************************************************************************************************************************************
changed: [192.168.56.101]

TASK [linux-update : Copy liveness test file] **********************************************************************************************************************************************************************
changed: [192.168.56.101]

TASK [linux-update : Remove liveness test file] ********************************************************************************************************************************************************************
changed: [192.168.56.101]

TASK [linux-update : Sleeping 5 minutes] ***************************************************************************************************************************************************************************
Pausing for 300 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
Press 'C' to continue the play or 'A' to abort
ok: [192.168.56.101]

TASK [linux-update : Stop Apache2 service] *************************************************************************************************************************************************************************
changed: [192.168.56.101]

TASK [linux-update : Refresh apt cache] ****************************************************************************************************************************************************************************
changed: [192.168.56.101]

TASK [linux-update : Update all packages] **************************************************************************************************************************************************************************
changed: [192.168.56.101]

TASK [linux-update : Reboot server] ********************************************************************************************************************************************************************************
changed: [192.168.56.101]

TASK [linux-update : Wait for host to boot] ************************************************************************************************************************************************************************
ok: [192.168.56.101 -> localhost]

TASK [linux-update : Sanity check] *********************************************************************************************************************************************************************************
changed: [192.168.56.101]

TASK [linux-update : Remove useless packages from the cache] *******************************************************************************************************************************************************
ok: [192.168.56.101]

TASK [linux-update : Remove dependencies that are no longer required] **********************************************************************************************************************************************
ok: [192.168.56.101]

TASK [linux-update : Copy liveness test file] **********************************************************************************************************************************************************************
changed: [192.168.56.101]

TASK [linux-update : Remove liveness test file] ********************************************************************************************************************************************************************
changed: [192.168.56.101]

TASK [linux-update : Start Apache2 service] ************************************************************************************************************************************************************************
ok: [192.168.56.101]

PLAY RECAP *********************************************************************************************************************************************************************************************************
192.168.56.101 : ok=18 changed=11 unreachable=0 failed=0

root@a0108173f049:~/git/stash/linux-update#

[arm4b]

That's a good start!

Add more verbosity -vvvvv to reveal what's happening behind the hood. Please include both successful/unsuccessful runs for us to compare. We do care only about the Wait for host to boot task, so please exclude everything else from the log.

Try to run the same virtualenv command from stanley user, - is there any diff?

[cdggcali]

Hi Eugen,

Thanks for your help, here is the information you have requested:
Successful

run: virtualenvs-ansible-playbook.txt
Unsuccessful run: Stackstorm_debug.txt

Im not sure what you meant about running it as from stanley user, but I did the following and it failed:

stanley@a0108173f049:~$ st2 run ansible.playbook playbook=/root/git/stash/linux-update/linux-update.yml inventory_file=/root/git/stash/linux-update/linux-update.ini
ERROR: 401 Client Error: Unauthorized
MESSAGE: Unauthorized - One of Token or API key required. for url: http://127.0.0.1:9101/v1/actions/ansible.playbook

stanley@a0108173f049:~$

Regards,
Carlos

[cdggcali]

virtualenvs-ansible-playbook.txt
Stackstorm_debug.txt

[arm4b]

@cdggcali
ERROR: 401 Client Error: Unauthorized error message means that you forgot to provide the auth creds for st2 command. This could be st2 username/password pair or StackStorm API key or API token.
See: https://docs.stackstorm.com/reference/cli.html#authentication-and-auth-token-caching as example.

---

Returning back to Ansible issue.

Try this. Create /etc/ansible/ansible.cfg with the following contents:

[defaults]
remote_tmp = $HOME/.ansible/tmp
local_tmp = $HOME/.ansible/tmp

Re-run the st2 ansible.playbook command and let us know.

[cdggcali]

@armab, that solved the issue.

I have a couple questions:

1. Whats the difference between running the playbook using stackstorm and directly running using /opt/stackstorm/virtualenvs/ansible/bin/ansible-playbook?
2. Why running the playbook using Stackstorm requires having the /etc/ansible/ansible.cfg and /opt/stackstorm/virtualenvs/ansible/bin/ansible-playbook doesn't?
3. I'm using the docker container, so should this file be added to the container?
4. If we actually install Stackstorm manually, do we also need to do this?

I'm trying to determine if I fail to follow the installation instructions, if is assumed that you will have Ansible installed on the servers or if it should be added to the installation instructions?

Regards,
Carlos

[arm4b]

@cdggcali Thanks for confirming!

Yes, you'll probably need to add this workaround to your container + any custom st2 installation.
The difference is that Ansible works differently if you invoke it from interactive user console or via automated way (cloud-init, cron, from another script, etc where environment may differ).

First time I was caught by this issue when added ansible-pull into cron, eg. StackStorm is not involved.

This is a bug in Ansible (ansible/ansible#21562) and they got really a lot of reports. They're trying to put everything together and fix it in some future release: ansible/ansible#31022

I'll dig a bit further and see if we could provide workarounds in st2 as well, but we really do pass $HOME env var which ATM is considered by Ansible team a reason for this bug to happen.

[arm4b]

@cdggcali Since the workaround works for you, I've updated the first message in this issue to include possible solutions (3) - so it'll be easier for others to find it: #25 (comment)

---

Let's also gather more feedback, - maybe someone has a better idea why it's happening.
We may include this in Ansible pack README in "Known Issues", but the behavior could depend on Ansible version and possible configuration.

@brainstorm looks like had the same issue before (ansible/ansible#21562 (comment))