Skip to content

Restarting replicated after using the studio results in endless creation of piper containers #4

@naderman

Description

@naderman

This starts if I restart the replicated service on the host, or e.g. reboot the machine, and doesn't stop.

45512590407e41415fdacc66b8fb   0 B (virtual 127.8 MB)
fcd9defc57f2        f08b0774069e                                     "entrypoint.sh piper "   20 minutes ago       Exited (255) 20 minutes ago                                                           piper-b7855307564140966f884c95b8034790   0 B (virtual 127.8 MB)
c45e7045012a        f08b0774069e                                     "entrypoint.sh piper "   20 minutes ago       Exited (255) 20 minutes ago                                                           piper-91ccb35e545841db5b6f5be7c28a6130   0 B (virtual 127.8 MB)
6999c92834ad        f08b0774069e                                     "entrypoint.sh piper "   20 minutes ago       Exited (255) 20 minutes ago                                                           piper-81e1e92cd41d495d5639944d46d5f695   0 B (virtual 127.8 MB)
6d0840c29577        f08b0774069e                                     "entrypoint.sh piper "   20 minutes ago       Exited (255) 20 minutes ago                                                           piper-4f91d4a78af146bf77d05bae2402f950   0 B (virtual 127.8 MB)
c3193643526a        f08b0774069e                                     "entrypoint.sh piper "   21 minutes ago       Exited (255) 20 minutes ago                                                           piper-7d64e58dceeb41635456110f289218c8   0 B (virtual 127.8 MB)
8bd035d25ff9        f08b0774069e                                     "entrypoint.sh piper "   21 minutes ago       Exited (255) 21 minutes ago                                                           piper-8d6751fb8a8043f77c7f505ab0552a4b   0 B (virtual 127.8 MB)
82bd806e8a12        f08b0774069e                                     "entrypoint.sh piper "   21 minutes ago       Exited (255) 21 minutes ago                                                           piper-e390ad746aaf41ea7f2a9ccee9be2569   0 B (virtual 127.8 MB)
d8671164d257        f08b0774069e                                     "entrypoint.sh piper "   21 minutes ago       Exited (255) 21 minutes ago                                                           piper-54a32b80bc3c411572052b6761857970   0 B (virtual 127.8 MB)
b905c9e905de        f08b0774069e                                     "entrypoint.sh piper "   21 minutes ago       Exited (255) 21 minutes ago                                                           piper-e503dede69164e8871647da64d61cf80   0 B (virtual 127.8 MB)
d4bdc9ccf598        f08b0774069e                                     "entrypoint.sh piper "   21 minutes ago       Exited (255) 21 minutes ago                                                           piper-8102ac3fa490407c75701c57315a505e   0 B (virtual 127.8 MB)
7a734b5eff74        f08b0774069e                                     "entrypoint.sh piper "   21 minutes ago       Exited (255) 21 minutes ago                                                           piper-17c3bea5278f4fe174557fcfc97a7ea9   0 B (virtual 127.8 MB)
f7d30235cf25        f08b0774069e                                     "entrypoint.sh piper "   21 minutes ago       Exited (255) 21 minutes ago                                                           piper-4e65c7e17f6246b547eb2def3a9a395c   0 B (virtual 127.8 MB)
d65de917bb0e        f08b0774069e                                     "entrypoint.sh piper "   21 minutes ago       Exited (255) 21 minutes ago                                                           piper-8067e4dd5a0a41ac77729bbb27562005   0 B (virtual 127.8 MB)
2ed24b71528b        f08b0774069e                                     "entrypoint.sh piper "   21 minutes ago       Exited (255) 21 minutes ago                                                           piper-33aa25fe650b4b085fecb48126902e45   0 B (virtual 127.8 MB)
47549816ede1        f08b0774069e                                     "entrypoint.sh piper "   21 minutes ago       Exited (255) 21 minutes ago                                                           piper-3aa0f9f952e245d26b27016393413578   0 B (virtual 127.8 MB)
dfdaf8197c78        f08b0774069e                                     "entrypoint.sh piper "   22 minutes ago       Exited (255) 21 minutes ago                                                           piper-0ebe5e7cab9a4f5875e424bd7f56d9f2   0 B (virtual 127.8 MB)
ad2c25803a42        f08b0774069e                                     "entrypoint.sh piper "   22 minutes ago       Exited (255) 22 minutes ago                                                           piper-a5d4ddcf0d6b4687450597bdeb49acaf   0 B (virtual 127.8 MB)
3af9546c963d        f08b0774069e                                     "entrypoint.sh piper "   22 minutes ago       Exited (255) 22 minutes ago                                                           piper-b4a56122535445f6632f4473b234265d   0 B (virtual 127.8 MB)
cf6656ecd71d        f08b0774069e                                     "entrypoint.sh piper "   22 minutes ago       Exited (255) 22 minutes ago                                                           piper-a4b33b54dfd246824efd11dc343b41e1   0 B (virtual 127.8 MB)
7bcd37bdb8c7        f08b0774069e                                     "entrypoint.sh piper "   22 minutes ago       Exited (255) 22 minutes ago                                                           piper-9ed529bbf6ac44fe5e6b3dcb05bc90bb   0 B (virtual 127.8 MB)
9e87b6e9601c        f08b0774069e                                     "entrypoint.sh piper "   22 minutes ago       Exited (255) 22 minutes ago                                                           piper-580000db9001430d57af04f384f21c3f   0 B (virtual 127.8 MB)
1144237a2407        f08b0774069e                                     "entrypoint.sh piper "   22 minutes ago       Exited (255) 22 minutes ago                                                           piper-946b6effc142419b4a013cffef9d6cfc   0 B (virtual 127.8 MB)
01c849b1258d        f08b0774069e                                     "entrypoint.sh piper "   22 minutes ago       Exited (255) 22 minutes ago                                                           piper-559a

Syslog:

Mar 12 20:39:31 ubuntu-xenial docker[2805]: ERROR 2017/03/12 20:39:31 premkit/log/gin.go:52 [GIN] 500 | 263.971221ms |  | GET     /v0.1/scheduler/snapshots/cluster-node-restore/stream                            
Mar 12 20:39:31 ubuntu-xenial docker[2771]: ERROR 2017/03/12 20:39:31 eventsource/eventsource.go:54 Eventsource stream https://replicated/v0.1/scheduler/snapshots/cluster-node-restore/stream?appid=[app id]&version=42 subscription error: 500:                                                                                                                                                             
Mar 12 20:39:31 ubuntu-xenial kernel: [ 1959.621553] docker0: port 2(veth26fcf92) entered disabled state                                                                                                           
Mar 12 20:39:31 ubuntu-xenial kernel: [ 1959.621614] veth809a8c1: renamed from eth0                                                                                                                                
Mar 12 20:39:31 ubuntu-xenial kernel: [ 1959.647837] docker0: port 2(veth26fcf92) entered disabled state                                                                                                           
Mar 12 20:39:31 ubuntu-xenial kernel: [ 1959.652369] device veth26fcf92 left promiscuous mode                                                                                                                      
Mar 12 20:39:31 ubuntu-xenial kernel: [ 1959.652374] docker0: port 2(veth26fcf92) entered disabled state                                                                                                           
Mar 12 20:39:33 ubuntu-xenial kernel: [ 1961.492449] aufs au_opts_verify:1597:dockerd[6312]: dirperm1 breaks the protection by the permission bits on the lower branch                                             
Mar 12 20:39:33 ubuntu-xenial kernel: [ 1961.515465] aufs au_opts_verify:1597:dockerd[6312]: dirperm1 breaks the protection by the permission bits on the lower branch                                             
Mar 12 20:39:33 ubuntu-xenial kernel: [ 1961.536889] aufs au_opts_verify:1597:dockerd[5506]: dirperm1 breaks the protection by the permission bits on the lower branch                                             
Mar 12 20:39:33 ubuntu-xenial kernel: [ 1961.546442] device vethe499e34 entered promiscuous mode                                                                                                                   
Mar 12 20:39:33 ubuntu-xenial kernel: [ 1961.546494] IPv6: ADDRCONF(NETDEV_UP): vethe499e34: link is not ready                                                                                                     
Mar 12 20:39:33 ubuntu-xenial systemd-udevd[4851]: Could not generate persistent MAC address for veth9ed8cb0: No such file or directory                                                                            
Mar 12 20:39:33 ubuntu-xenial systemd-udevd[4852]: Could not generate persistent MAC address for vethe499e34: No such file or directory                                                                            
Mar 12 20:39:33 ubuntu-xenial kernel: [ 1961.649525] eth0: renamed from veth9ed8cb0                                                                                                                                
Mar 12 20:39:33 ubuntu-xenial kernel: [ 1961.660773] IPv6: ADDRCONF(NETDEV_CHANGE): vethe499e34: link becomes ready                                                                                                
Mar 12 20:39:33 ubuntu-xenial kernel: [ 1961.660794] docker0: port 2(vethe499e34) entered forwarding state                                                                                                         
Mar 12 20:39:33 ubuntu-xenial kernel: [ 1961.660802] docker0: port 2(vethe499e34) entered forwarding state                                                                                                         
Mar 12 20:39:33 ubuntu-xenial docker[2805]: INFO 2017-03-12T20:39:33+00:00 [pkg/piper] cmd.go:164 Piper result failed for file                                                                                     
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .dashboard .btn-warning {                                                                                                                                              
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #f0ad4e;                                                                                                                                           
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   border-color: #eea236;                                                                                                                                               
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   color: #fff;                                                                                                                                                         
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }                                                                                                                                                                      
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .dashboard .btn-warning:hover {                                                                                                                                        
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   color:#fff;                                                                                                                                                          
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color:#ec971f;                                                                                                                                            
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   border-color:#d58512;                                                                                                                                                
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }                                                                                                                                                                      
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .react-grid-item {                                                                                                                                                     
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   border: 0;                                                                                                                                                           
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }                                                                                                                                                                      
Mar 12 20:39:33 ubuntu-xenial docker[2805]: /* cpu graphs */                                                                                                                                                       
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-metrics-cpuacct {                                                                                                                                              
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #ec663c;                                                                                                                                           
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }                                                                                                                                                                      
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-metrics-cpuacct .widget-metrics-chart {                                                                                                                        
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #ec663c;                                                                                                                                           
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }                                                                                                                                                                      
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-cpu-metrics > .text-muted {                                                                                                                                    
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   color: #ccc;                                                                                                                                                         
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }                                                                                                                                                                      
Mar 12 20:39:33 ubuntu-xenial docker[2805]: /* memory graphs */                                                                                                                                                    
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-metrics-memory {                                                                                                                                               
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #8A4F7D;                                                                                                                                           
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }                                                                                                                                                                      
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-metrics-memory .widget-metrics-chart {                                                                                                                         
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #8A4F7D;                                                                                                                                           
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }                                                                                                                                                                      
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-memory-metrics > .text-muted {                                                                                                                                 
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   color: #ccc;                                                                                                                                                         
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }                                                                                                                                                                      
Mar 12 20:39:33 ubuntu-xenial docker[2805]: /* App status tile */                                                                                                                                                  
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-appstatus {                                                                                                                                                    
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #2A747B;                                                                                                                                           
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }                                                                                                                                                                      
Mar 12 20:39:33 ubuntu-xenial docker[2805]: /* App udpate tile */                                                                                                                                                  
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-appupdate {
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #12B0C5;
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }
Mar 12 20:39:33 ubuntu-xenial docker[2805]: /* App backup tile */
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-appbackup {
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #887880;
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }
Mar 12 20:39:33 ubuntu-xenial docker[2805]: /manifest.json: lstat /host/
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .dashboard .btn-warning {
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #f0ad4e;
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   border-color: #eea236;
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   color: #fff;
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .dashboard .btn-warning:hover {
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   color:#fff;
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color:#ec971f;
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   border-color:#d58512;
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .react-grid-item {
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   border: 0;
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }
Mar 12 20:39:33 ubuntu-xenial docker[2805]: /* cpu graphs */
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-metrics-cpuacct {
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #ec663c;
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-metrics-cpuacct .widget-metrics-chart {
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #ec663c;
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-cpu-metrics > .text-muted {
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   color: #ccc;
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }
Mar 12 20:39:33 ubuntu-xenial docker[2805]: /* memory graphs */
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-metrics-memory {
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #8A4F7D;
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-metrics-memory .widget-metrics-chart {
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #8A4F7D;
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-memory-metrics > .text-muted {
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   color: #ccc;
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }
Mar 12 20:39:33 ubuntu-xenial docker[2805]: /* App status tile */
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-appstatus {
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #2A747B;
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }
Mar 12 20:39:33 ubuntu-xenial docker[2805]: /* App udpate tile */
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-appupdate {
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #12B0C5;
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }
Mar 12 20:39:33 ubuntu-xenial docker[2805]: /* App backup tile */
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-appbackup {
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #887880;
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }
Mar 12 20:39:33 ubuntu-xenial docker[2805]: /manifest.json: no such file or directory
Mar 12 20:39:33 ubuntu-xenial docker[2805]: ERROR 2017/03/12 20:39:33 premkit/log/gin.go:52 [GIN] 500 | 262.991283ms |  | GET     /v0.1/scheduler/snapshots/cluster-node-restore/stream
Mar 12 20:39:33 ubuntu-xenial docker[2771]: ERROR 2017/03/12 20:39:33 eventsource/eventsource.go:54 Eventsource stream https://replicated/v0.1/scheduler/snapshots/cluster-node-restore/stream?appid=[app id][&version=42 subscription error: 500:
Mar 12 20:39:33 ubuntu-xenial kernel: [ 1961.769419] docker0: port 2(vethe499e34) entered disabled state
Mar 12 20:39:33 ubuntu-xenial kernel: [ 1961.769505] veth9ed8cb0: renamed from eth0
Mar 12 20:39:33 ubuntu-xenial kernel: [ 1961.793862] docker0: port 2(vethe499e34) entered disabled state
Mar 12 20:39:33 ubuntu-xenial kernel: [ 1961.796901] device vethe499e34 left promiscuous mode
Mar 12 20:39:33 ubuntu-xenial kernel: [ 1961.796905] docker0: port 2(vethe499e34) entered disabled state

This, too loops endlessly

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions