Unable to find a master for /19 (/dev/console)

[stgraber]

I recently updated Ubuntu from CRIU 2.9 to 2.12, in the process, we've noticed that CRIU now systematically fails to dump state for LXD containers.

The typical symptoms are:

error: snapshot dump failed
(00.123326) Warn  (criu/arch/x86/crtools.c:138): Will restore 25111 with interrupted system call
(00.653660) Warn  (criu/arch/x86/crtools.c:138): Will restore 25396 with interrupted system call
(01.517694) Error (criu/tty.c:2230): tty: Unable to find a master for /19
(01.520239) Error (criu/cr-dump.c:1641): Dumping FAILED.

I've tested CRIU 2.11.1 which is unaffected, then ran a bisect from the v2.11.1 tag to the v2.12 tag which shows aa26237 as the culprit.
The complete log can be found here: http://paste.ubuntu.com/24226041/

It's failing on /dev/console in this case which as with any /dev device in unprivileged LXC is setup as bind-mount of the host's own device or in this case, a host pts device.

~ # cat /proc/self/mountinfo  | grep console
847 820 0:19 /19 /dev/console rw,nosuid,noexec,relatime master:3 - devpts devpts rw,gid=5,mode=620,ptmxmode=000

And full CRIU command as run by LXC:

criu dump --tcp-established --file-locks --link-remap --manage-cgroups=full --ext-mount-map auto --enable-external-sharing --enable-external-masters --enable-fs hugetlbfs --enable-fs tracefs -D /var/lib/lxd/containers/c1/state -o /var/lib/lxd/containers/c1/state/dump.log --cgroup-root name=systemd:/lxc/c1-3 --cgroup-root memory:/lxc/c1-3 --cgroup-root perf_event:/lxc/c1-3 --cgroup-root freezer:/lxc/c1-3 --cgroup-root hugetlb:/lxc/c1-3 --cgroup-root net_cls,net_prio:/lxc/c1-3 --cgroup-root blkio:/lxc/c1-3 --cgroup-root devices:/lxc/c1-3 --cgroup-root pids:/lxc/c1-3 --cgroup-root cpu,cpuacct:/lxc/c1-3 --cgroup-root cpuset:/lxc/c1-3 -vvvvvv --ext-mount-map /dev/fuse:dev/fuse --ext-mount-map /dev/net/tun:dev/net/tun --ext-mount-map /proc/sys/fs/binfmt_misc:proc/sys/fs/binfmt_misc --ext-mount-map /sys/firmware/efi/efivars:sys/firmware/efi/efivars --ext-mount-map /sys/fs/fuse/connections:sys/fs/fuse/connections --ext-mount-map /sys/fs/pstore:sys/fs/pstore --ext-mount-map /sys/kernel/debug:sys/kernel/debug --ext-mount-map /sys/kernel/security:sys/kernel/security --ext-mount-map /dev/mqueue:dev/mqueue --ext-mount-map /dev/lxd:dev/lxd --ext-mount-map /dev/.lxd-mounts:dev/.lxd-mounts -t 25111 --freeze-cgroup /sys/fs/cgroup/freezer///lxc/c1-3 --ext-mount-map /dev/console:console --external tty[8813:13] --ghost-limit 268435456 --leave-running

[xemul]

Cc @avagin (and @cyrillos )

[avagin]

(00.113829) type devpts source devpts mnt_id 847 s_dev 0x13 /19 @ ./dev/console flags 0x28000a options gid=-1,mode=620,ptmxmode=000

@stgraber where is a master for this pty slave (19)?

[stgraber]

The master for that pty is the monitor process on the host (the parent of the container's PID1).

From what I remember (@brauner may know better), LXC essentially does:

• Create a pty pair on the host devpts
• Keeps the master fd around (in our monitor process) for use by lxc-console
• Bind-mounts the pts path from /dev/pts/XYZ on the host to /dev/console in the container

So in the case above, /dev/console is a bind-mount of /dev/pts/19 on the host. With the master fd of that pts device being held by the container's parent (monitor process).

A new instance of that monitor process is what will then be used to restore the CRIU dump.

Getting that part working was a bit tricky and took @tych0 quite a while to figure out as I remember us having to disable /dev/console entirely for a while in the early days of checkpoint/restore with LXC/LXD.

[stgraber]

On restore, we call CRIU like so:

criu restore --tcp-established --file-locks --link-remap --manage-cgroups=full --ext-mount-map auto --enable-external-sharing --enable-external-masters --enable-fs hugetlbfs --enable-fs tracefs -D /var/lib/lxd/containers/c1/state -o /var/lib/lxd/containers/c1/state/restore.log --cgroup-root name=systemd:/lxc/c1-3 --cgroup-root memory:/lxc/c1-3 --cgroup-root perf_event:/lxc/c1-3 --cgroup-root freezer:/lxc/c1-3 --cgroup-root hugetlb:/lxc/c1-3 --cgroup-root net_cls,net_prio:/lxc/c1-3 --cgroup-root blkio:/lxc/c1-3 --cgroup-root devices:/lxc/c1-3 --cgroup-root pids:/lxc/c1-3 --cgroup-root cpu,cpuacct:/lxc/c1-3 --cgroup-root cpuset:/lxc/c1-3 -vvvvvv --ext-mount-map dev/fuse:/dev/fuse --ext-mount-map dev/net/tun:/dev/net/tun --ext-mount-map proc/sys/fs/binfmt_misc:/proc/sys/fs/binfmt_misc --ext-mount-map sys/firmware/efi/efivars:/sys/firmware/efi/efivars --ext-mount-map sys/fs/fuse/connections:/sys/fs/fuse/connections --ext-mount-map sys/fs/pstore:/sys/fs/pstore --ext-mount-map sys/kernel/debug:/sys/kernel/debug --ext-mount-map sys/kernel/security:/sys/kernel/security --ext-mount-map dev/mqueue:/dev/mqueue --ext-mount-map dev/lxd:/var/lib/lxd/devlxd --ext-mount-map dev/.lxd-mounts:/var/lib/lxd/shmounts/c1 --root /usr/lib/x86_64-linux-gnu/lxc --restore-detached --restore-sibling --inherit-fd fd[5]:tty[8813:13] --ext-mount-map console:/dev/pts/19 --lsm-profile apparmor:lxd-c1_</var/lib/lxd>//&:lxd-c1_<var-lib-lxd>: --veth-pair eth0=vethEIKS87@lxdbr0

[avagin]

@stgraber Thank you for the explanation. Could you try out this patch avagin@87f4472?

[stgraber]

@avagin that did the trick!

[avagin]

Recently I added support /dev/console in runc containers, but there a terminal pair is allocate in a container and then a master terminal is sent back to host via a unix socket:
opencontainers/runc#1355

[avagin]

@stgraber good. Thank you. I think we can release v2.12.1 with this patch.

[stgraber]

Yeah, we've been staying away from container allocated pts pairs so far due to fear of security issues when the user can unmount /dev/pts and mount a look-alike fuse/cuse filesystem in its place.

It wouldn't be an issue for /dev/console itself since we can set that one up before init is spawned in the container, but it's an issue for pts pairs we allocate on demand later on when attaching to the container.