Insane cgroup mounts in /sys/fs/cgroup/systemd/libpod_parent with podman+bind mounts

[aalba6675]

1. Firstly, this issue is only triggered when using bind mounts with podman.
2. When a bind mount is used, the cgroups mounted in /sys/fs/cgroup/systemd/libpod_parent has an infinite recursion after a few stop/start cycles. In fact there are 86(!) mounts as below, but only four unique mount paths. These directories don't exist and the cgroup cannot be unmounted. Furthermore, after the third time the container cannot be started. No other container can be started (even those without bind mounts).
3. @mheon in CGroup Handling Enhancements containers/podman#507 suggests that the bug is here as these crazy mount paths are not created by podman.
4. Crazy mounts: https://github.com/projectatomic/libpod/files/1991848/cgroup.zip
5. Any pointers as to who/what is recursively adding to the cgroup mount path?

Reproducer:

# this kludge is necessary otherwise we cannot MS_MOVE the mount
mount --make-private /tmp
podman run --name bobby_silver -v /volumes/podman/home:/home:z --entrypoint /sbin/init fedora:28
podman stop bobby_silver
podman start bobby_silver
podman stop boddy_silver
podman start bobby_silver # third time fails

single:
cgroup on /sys/fs/cgroup/systemd/libpod_parent/libpod-cd8be22a52efaed7e2790d2eb3421c00542c3eb9763bfe715c3ad23647c419e0/ctr

doubled:

cgroup on /sys/fs/cgroup/systemd/libpod_parent/libpod-cd8be22a52efaed7e2790d2eb3421c00542c3eb9763bfe715c3ad23647c419e0/ctr/libpod_parent/libpod-cd8be22a52efaed7e2790d2eb3421c00542c3eb9763bfe715c3ad23647c419e0/ctr type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,name=systemd)

tripled:
cgroup on /sys/fs/cgroup/systemd/libpod_parent/libpod-cd8be22a52efaed7e2790d2eb3421c00542c3eb9763bfe715c3ad23647c419e0/ctr/libpod_parent/libpod-cd8be22a52efaed7e2790d2eb3421c00542c3eb9763bfe715c3ad23647c419e0/ctr/libpod_parent/libpod-cd8be22a52efaed7e2790d2eb3421c00542c3eb9763bfe715c3ad23647c419e0/ctr type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,name=systemd)

quadrupled:
cgroup on /sys/fs/cgroup/systemd/libpod_parent/libpod-cd8be22a52efaed7e2790d2eb3421c00542c3eb9763bfe715c3ad23647c419e0/ctr/libpod_parent/libpod-cd8be22a52efaed7e2790d2eb3421c00542c3eb9763bfe715c3ad23647c419e0/ctr/libpod_parent/libpod-cd8be22a52efaed7e2790d2eb3421c00542c3eb9763bfe715c3ad23647c419e0/ctr/libpod_parent/libpod-cd8be22a52efaed7e2790d2eb3421c00542c3eb9763bfe715c3ad23647c419e0/ctr type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,name=systemd)

[mheon]

The first potential problem here is that we're mounting libpod's cgroupfs CGroups into systemd's CGroup hierarchy. That strikes me as wrong (and potentially dangerous). Systemd CGroup support in libpod is still very much in progress, but once it is complete we might want to consider making it mandatory for oci-systemd-hook (it'll be moving to default once it's ready to match CRI-O's current default)

[aalba6675]

Unrelated to cleanup: on start up we also seem to have a bug: the host sees 3 mounts, two on the single path and one on the double path (specific to containers created with -v).

cgroup on /sys/fs/cgroup/systemd/
libpod_parent/libpod-9ffaff1cdcab235dc1dabdb25d6d1e209f044957b02b533874e0aaf17c0200db/ctr 
type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,name=systemd)

cgroup on /sys/fs/cgroup/systemd/
libpod_parent/libpod-9ffaff1cdcab235dc1dabdb25d6d1e209f044957b02b533874e0aaf17c0200db/ctr
type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,name=systemd)

cgroup on /sys/fs/cgroup/systemd/
libpod_parent/libpod-9ffaff1cdcab235dc1dabdb25d6d1e209f044957b02b533874e0aaf17c0200db/ctr/
libpod_parent/libpod-9ffaff1cdcab235dc1dabdb25d6d1e209f044957b02b533874e0aaf17c0200db/ctr
type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,name=systemd)

After the container is stopped, if these paths are manually unmounted, then the container can be restarted indefinitely, i.e, it doesn't trap itself in tripled/quad paths etc.

This may explain the triple/quad path - when the container is restarted without proper cleanup it just keeps adding to the cgroup path ad inifinitum...

[aalba6675]

The weird thing is the containers that do not use -v see the single/single/double mount and the host does not see the mount. When the container exits, these mounts are gracefully deleted.

This container does not use -v; it sees

cgroup on /sys/fs/cgroup/systemd/libpod_parent/libpod-9ffaff1cdcab235dc1dabdb25d6d1e209f044957b02b533874e0aaf17c0200db/ctr type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,name=systemd)
cgroup on /sys/fs/cgroup/systemd/libpod_parent/libpod-9ffaff1cdcab235dc1dabdb25d6d1e209f044957b02b533874e0aaf17c0200db/ctr type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,name=systemd)
cgroup on /sys/fs/cgroup/systemd/libpod_parent/libpod-9ffaff1cdcab235dc1dabdb25d6d1e209f044957b02b533874e0aaf17c0200db/ctr/libpod_parent/libpod-9ffaff1cdcab235dc1dabdb25d6d1e209f044957b02b533874e0aaf17c0200db/ctr type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,name=systemd)

The host sees nothing.

[mheon]

@rhatdan This definitely sounds like a separate bug than Podman not firing the cleanup hooks (unless we want a duplicated CGroup path for some reason)?

I'm betting that mount propogation is the reason for only seeing this with volume mounts - we probably need to change our mount propogation to mount volumes into the container.

[aalba6675]

Normally a non--v container also sees one single-path mount, the single/single/double appears when a -v container is started after it.

Non--v container is running:

# this is a non-`-v` container
cgroup on /sys/fs/cgroup/systemd/libpod_parent/libpod-9075fe81752a0a9383e587ba9af6de76d546cfec3f3d23683d1de165c69ed96f/ctr type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,name=systemd)

Start a -v container; then check again. Three mounts of the -v container are propagated into the non--v container

## three of these mounts are from another container
cgroup on /sys/fs/cgroup/systemd/libpod_parent/libpod-9075fe81752a0a9383e587ba9af6de76d546cfec3f3d23683d1de165c69ed96f/ctr type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,name=systemd)
cgroup on /sys/fs/cgroup/systemd/libpod_parent/libpod-9ffaff1cdcab235dc1dabdb25d6d1e209f044957b02b533874e0aaf17c0200db/ctr type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,name=systemd)
cgroup on /sys/fs/cgroup/systemd/libpod_parent/libpod-9ffaff1cdcab235dc1dabdb25d6d1e209f044957b02b533874e0aaf17c0200db/ctr type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,name=systemd)
cgroup on /sys/fs/cgroup/systemd/libpod_parent/libpod-9ffaff1cdcab235dc1dabdb25d6d1e209f044957b02b533874e0aaf17c0200db/ctr/libpod_parent/libpod-9ffaff1cdcab235dc1dabdb25d6d1e209f044957b02b533874e0aaf17c0200db/ctr type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,name=systemd)

[aalba6675]

Filed as #95