All own/custom container have been deleted

[dbanck]

I don't know if this is an indented behaviour, but all my custom containers have been deleted.

I had a couple of data and application containers, now there is nothing left to start for my systemd services :o

[kdomanski]

Well there's only one docker rm in the script and it removes a container named configure, so it might be something else.

[kdomanski]

btw. seems to be connected to #41

[hinnerk]

I'm sorry that happened to you. It's certainly not intended and we've been looking into it but so far weren't able to reproduce it. Can you reproduce it with your setup?

That said, the changes in /experimental-platform/platform-configure#18 might have fixed it and will make it into the next alpha channel update that's scheduled for tomorrow.

Besides, our convention for app containers is to store data in /data/<containername> which is mounted inside the container on /data by a dokku plugin. Is it viable for you to modify your custom containers in such a way?

[hinnerk]

This should be fixed with todays release. Please re-open the ticket if it isn't.

[dbanck]

Same error again. All I can say so far is that the containers are gone, but the images are still there.

Let me dig through the logs, maybe I can find something.

[dbanck]

There was a reboot of the box on dec 27th.

Dec 27 23:49:04 zeus dockerd[586]: time="2015-12-27T23:49:04.699592691Z" level=error msg="Failed to load container 0fce97c8e4c24ccdd76edfd6cb9b50b3e2efafedd977c0717147fa8ede3206c1: open /var/
Dec 27 23:49:04 zeus dockerd[586]: time="2015-12-27T23:49:04.739773981Z" level=error msg="Failed to load container 340d96bbb8e86e5efeb72e82e6d4818ae3e9bd2d912f24062921448f5eb544e9: open /var/
Dec 27 23:49:04 zeus dockerd[586]: time="2015-12-27T23:49:04.779232530Z" level=error msg="Failed to load container b7a6929bcac623c86c99aaf5a4031fe0a1b7c6c9e0a3cea52a0162ac4d17692a: open /var/
Dec 27 23:49:04 zeus docker[587]: vpn1
Dec 27 23:49:04 zeus docker[698]: vpn1
Dec 27 23:49:04 zeus systemd[1]: Started VPN.
[...]
Dec 27 23:49:05 zeus dockerd[586]: time="2015-12-27T23:49:05.298535771Z" level=error msg="Handler for POST /containers/create returned error: no such id: ovpn-data"
Dec 27 23:49:05 zeus dockerd[586]: time="2015-12-27T23:49:05.298940045Z" level=error msg="HTTP Error" err="no such id: ovpn-data" statusCode=404
Dec 27 23:49:05 zeus docker[705]: Error response from daemon: no such id: ovpn-data
Dec 27 23:49:05 zeus systemd[1]: vpn.service: Main process exited, code=exited, status=1/FAILURE
Dec 27 23:49:05 zeus systemd[1]: vpn.service: Unit entered failed state.
Dec 27 23:49:05 zeus systemd[1]: vpn.service: Failed with result 'exit-code'.
Dec 27 23:49:05 zeus systemd[1]: Started Initial Protonet Experimental Plattform.
Dec 27 23:49:05 zeus systemd[1]: Started Start removal of unused docker images..
Dec 27 23:49:05 zeus systemd[1]: Starting Start removal of unused docker images..
Dec 27 23:49:05 zeus systemd[1]: Reached target Timers.
Dec 27 23:49:05 zeus systemd[1]: Starting Timers.
Dec 27 23:49:05 zeus systemd[1]: Starting Run 'central gateway' service...
Dec 27 23:49:05 zeus systemd[1]: Starting Run 'pulseaudio' service...
Dec 27 23:49:05 zeus systemd[1]: Starting Run 'dokku' service...
Dec 27 23:49:05 zeus systemd[1]: Starting Run systemd-proxy (sproxy) service...
Dec 27 23:49:05 zeus systemd[1]: Starting Run 'monitoring' service...
Dec 27 23:49:05 zeus systemd[1]: Starting Run 'app manager' service...
Dec 27 23:49:05 zeus systemd[1]: Starting Remove unused docker images...
[...]

I didn't notice the container didn't come up back then and turned the box off a few days later.

After booting it up today the errors are somewhat identical.

Jan 04 13:50:19 zeus dockerd[524]: time="2016-01-04T13:50:19.375289875Z" level=error msg="Failed to load container 0fce97c8e4c24ccdd76edfd6cb9b50b3e2efafedd977c0717147fa8ede3206c1: open /var/
Jan 04 13:50:19 zeus dockerd[524]: time="2016-01-04T13:50:19.423739487Z" level=error msg="Failed to load container 340d96bbb8e86e5efeb72e82e6d4818ae3e9bd2d912f24062921448f5eb544e9: open /var/
Jan 04 13:50:19 zeus dockerd[524]: time="2016-01-04T13:50:19.480921109Z" level=error msg="Failed to load container b7a6929bcac623c86c99aaf5a4031fe0a1b7c6c9e0a3cea52a0162ac4d17692a: open /var/
Jan 04 13:50:19 zeus dockerd[524]: time="2016-01-04T13:50:19.498150587Z" level=error msg="Handler for POST /containers/{name:.*}/kill returned error: no such id: vpn1"
Jan 04 13:50:19 zeus dockerd[524]: time="2016-01-04T13:50:19.498897049Z" level=error msg="HTTP Error" err="no such id: vpn1" statusCode=404
Jan 04 13:50:19 zeus docker[525]: Error response from daemon: no such id: vpn1
Jan 04 13:50:19 zeus docker[525]: Error: failed to kill containers: [vpn1]
Jan 04 13:50:19 zeus dockerd[524]: time="2016-01-04T13:50:19.957205301Z" level=error msg="Handler for DELETE /containers/{name:.*} returned error: no such id: vpn1"
Jan 04 13:50:19 zeus dockerd[524]: time="2016-01-04T13:50:19.957260182Z" level=error msg="HTTP Error" err="no such id: vpn1" statusCode=404
Jan 04 13:50:19 zeus docker[747]: Error response from daemon: no such id: vpn1
Jan 04 13:50:19 zeus docker[747]: Error: failed to remove containers: [vpn1]
Jan 04 13:50:19 zeus systemd[1]: Started VPN.
[...]
Jan 04 13:50:23 zeus dockerd[524]: time="2016-01-04T13:50:23.061102467Z" level=error msg="Handler for POST /containers/create returned error: no such id: ovpn-data"
Jan 04 13:50:23 zeus dockerd[524]: time="2016-01-04T13:50:23.061561298Z" level=error msg="HTTP Error" err="no such id: ovpn-data" statusCode=404
Jan 04 13:50:23 zeus docker[766]: Error response from daemon: no such id: ovpn-data
Jan 04 13:50:23 zeus systemd[1]: vpn.service: Main process exited, code=exited, status=1/FAILURE
Jan 04 13:50:23 zeus systemd[1]: vpn.service: Unit entered failed state.
Jan 04 13:50:23 zeus systemd[1]: vpn.service: Failed with result 'exit-code'.

I haven't found anything requesting the deletion of the container. Are there separate logs (outside journalctl for experimental-platform?