Run nixos container on nixos host with shared nix store using incus.
Inspired by nixos-container and nixcloud-container.
nixos-container lets container share host's nix store. But it uses systemd-nspawn, which I am not familiar with. And it might be less functional than other prevalent lxc managers.
nixcloud-container project brings the concept of sharing nix store to LXD, but hasn't been maintained for a long time. During the years, incus took the place of LXD (project history). And that is the target of inx-container.
Tip: When encontering problems, add a environment variable VERBOSE=1 might help. That tells inx-container to show what it does step by step.
Firstly, manually create a directory /nix/var/nix/profiles/inx-container (more details). I personally suggest make it writable by incus-admin group, letting a user in the group uses inx-container without dedicated permission.
Then, there is an example folder in this repo, cd into example folder,
inx-container create inx-ex .
The system should be built and placed under /nix/var/nix/profiles/inx-container/inx-ex. And a incus container named "inx-ex" should be created but not started.
From now on, "inx-ex" is just a normal incus container, not much different from others. You can start it now, or customize before first run.
Next time after editing the configration,
inx-container update inx-ex .
Incus - Official NixOS Wiki talks about how to create custom container image. It includes a module from "${inputs.nixpkgs}/nixos/modules/virtualisation/lxc-container.nix", which requires a /sbin folder under root directory. But we are not gonna build rootfs from the lxc-container configuration. So please DO NOT follow that. Instead, add boot.isContainer = true; manually to your module.
Example in this repo provides a fully functional instance.
nix commands are supported in container, although not recommended. There is a known issue with nix garbage collection, which nixos-container also confronts (details here).
That said, nix-shell and even nixos-rebuild switch work fine in container. Though it might be preferred to use inx-container update form host rather than nixos-rebuild switch in container.
There is also a flaw with sudo (same as nixos-container), that you cannot directly sudo a nix command. It can be easily solved.
inx-container stores container system profiles in /nix/var/nix/profiles/inx-container/$INSTANCE, which should be kept persistent if the host system uses a volatile rootfs.
nixos-container has some key components, a cli written in perl (/run/current-system/sw/bin/nixos-container) and a systemd service template (/etc/systemd/system/container@.service). There is also a nixos-containers.nix module for containers defined with containers option, which is pretty much the same. They provide key clues to find out how nixos-container works.
nixos-container mounts /nix/var/nix/gcroots/per-container/$INSTANCE on host to /nix/var/nix/gcroots in container. Which seems reasonable, but actually does nothing.
-
booted-systemandcurrent-systemin that gcroots directory are symlinks to/run/booted-systemand/run/current-system. From the perspective of nix daemon, they refer to the host system instead of container. Even so it is not a big deal because/nix/var/nix/profilesis also treated as gc root. There are symlinks point directly to absolute path in nix store, which will keep the container system from being deleted. -
When nix performs a build, it automatically leaves a symlink in
/nix/var/nix/gcroots/autopointing to build result (more details here). But if the build is requested from container, the symlink will point to a path inside container, unreachable from host, causing build result get garbage collected when gc. The problem has existed for a long time (a discussion in 2021), and it is still there.
Inside nixos-container, NIX_REMOTE=daemon is absent in sudo environment. But both root and non-root user's environments are correct. Only sudo a nix command will fail.
inx-container is just a command line helper, calling nix and incus to set up container properly. It does not offer the declarative that containers option provides. Which could be improved to some extent.