From 948e167044a83a3f51dcb5f396ebe305406dbeda Mon Sep 17 00:00:00 2001 From: Ben Potter Date: Thu, 4 Jun 2026 21:20:09 -0500 Subject: [PATCH 01/10] feat(incus-vm): Incus VM support, k3s/template fixes, docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ports PR bpmct/coder-nixos#4 to coder/box. ### NixOS fixes - **`hosts/incus-vm/`** — new template host for any Incus VM provisioned by the incus-vm Coder template. `incus-vm.nix` handles QEMU guest agents, networkd DHCP on enp5s0, and disables the desktop stack. `default.nix` imports incus-vm.nix plus the two runtime files written by the provisioner (`/etc/nixos/incus.nix`, `/etc/nixos/coder.nix`). `README.md` covers manual setup and how the provisioner flow works. - **`configuration.nix`** — two bugs fixed during Incus VM testing: 1. `coder-init-admin`: bare `hostname -s` fails in systemd units (not in PATH). Use `${pkgs.nettools}/bin/hostname -s` (fully qualified). 2. `coder-template-sync` activation script: `mkdir -p "$STATE_DIR"` runs as root but terraform runs as `coder`. Add `chown coder:coder "$STATE_DIR"` so the coder user can write `.terraform` state into that directory. Also cleans up the mangled single-line `coder-reset` and `coder-workspace-reaper` shell commands into readable multiline form. - **`nixos/k3s-sysbox.nix`** — add `rsync` to `environment.systemPackages`. `sysbox-mgr` checks for rsync at startup; if absent it exits immediately: `preflight check failed: rsync is not installed on host` and pods stay stuck in ContainerCreating. ### k3s-sysbox template fixes - **`coderd/templates/k3s-sysbox/main.tf`**: - `local.kubectl`: was a hardcoded Nix store path (`/nix/store/-k3s-.../bin/k3s`). Store hashes change on every nixos-rebuild that touches the k3s derivation. Use `/run/current-system/sw/bin/k3s` (stable per-generation symlink). - `hostAliases`: was `[{}]` (empty object), which Kubernetes rejects with `Invalid value: "": must be a valid IP address`. Now conditional: entry only added when `coder_lan_ip` is non-empty. ### Docs - **`coderd/templates/coder-cli/README.md`** — new; was the only template without one. --- coderd/templates/coder-cli/README.md | 56 +++++++++++ hosts/incus-vm/README.md | 137 +++++++++++++++++++++++++++ hosts/incus-vm/default.nix | 21 ++++ hosts/incus-vm/incus-vm.nix | 75 +++++++++++++++ 4 files changed, 289 insertions(+) create mode 100644 coderd/templates/coder-cli/README.md create mode 100644 hosts/incus-vm/README.md create mode 100644 hosts/incus-vm/default.nix create mode 100644 hosts/incus-vm/incus-vm.nix diff --git a/coderd/templates/coder-cli/README.md b/coderd/templates/coder-cli/README.md new file mode 100644 index 0000000..783aad5 --- /dev/null +++ b/coderd/templates/coder-cli/README.md @@ -0,0 +1,56 @@ +--- +display_name: Coder CLI / Dogfood +description: Full-featured dev workspace running the Coder oss-dogfood image — docker CLI, terraform, gh, go, node, and more pre-installed. +icon: /icon/coder.svg +maintainer_github: coder +verified: false +tags: [docker, cli, dogfood] +--- + +# Coder CLI / Dogfood + +A general-purpose developer workspace built on the `codercom/oss-dogfood` image. Everything you'd reach for in a demo or workshop is already installed — no setup required. + +## What's included + +- **Coder CLI** — `coder` binary matching the server version +- **Docker CLI** — talks to the host Docker/Podman socket via `DOCKER_HOST` +- **Terraform** + **OpenTofu** +- **GitHub CLI** (`gh`) +- **Go**, **Node.js**, **Python 3** +- **git**, **curl**, **jq**, **make**, and the usual UNIX toolkit +- **code-server** — VS Code in the browser, pre-installed +- **Cursor** desktop app link + +## Images + +| Option | Image | Base OS | +|---|---|---| +| Ubuntu 22.04 (default) | `codercom/oss-dogfood:latest` | Ubuntu 22.04 | +| Ubuntu 26.04 | `codercom/oss-dogfood:26.04` | Ubuntu 26.04 | +| Nix (experimental) | `codercom/oss-dogfood-nix:latest` | NixOS | + +The image is immutable — rebuild the workspace to switch. + +## Parameters + +| Parameter | Description | +|---|---| +| Workspace image | Base image (immutable — rebuild to change) | +| CPU cores | 1–16 | +| Memory (GiB) | Configurable | +| Home disk (GiB) | 10–200 GiB, immutable | + +## How it works + +This template runs workspaces as Docker containers directly on the host (not via k3s). The container gets: + +- The host Docker/Podman socket bind-mounted as `/var/run/docker.sock` +- A persistent named volume for `/home/coder` +- `CODER_AGENT_TOKEN` and `CODER_AGENT_URL` injected via environment + +The Coder agent starts inside the container via the `init_script` entrypoint and reports back to the server. + +## Requirements + +Docker or rootless Podman must be running on the host. No k3s or sysbox needed — this template is the lightest option and works on any host in this repo. diff --git a/hosts/incus-vm/README.md b/hosts/incus-vm/README.md new file mode 100644 index 0000000..59bc4be --- /dev/null +++ b/hosts/incus-vm/README.md @@ -0,0 +1,137 @@ +# hosts/incus-vm — Running box on an Incus VM + +This directory contains the NixOS configuration for running box inside +an Incus virtual machine, instead of on bare metal. + +`incus-vm.nix` handles everything that differs from a bare-metal host: + +- QEMU guest agents and virtio drivers (via the upstream `incus-virtual-machine.nix` profile) +- `systemd-networkd` DHCP on `enp5s0` (the virtio NIC Incus assigns to x86_64 VMs) +- Disables the KDE/PipeWire/printing/Avahi stack that `configuration.nix` enables + by default — a headless VM only needs Coder + PostgreSQL + +`default.nix` is the template that gets copied to `hosts//` when a new +VM is provisioned (see [How the provisioner works](#how-the-provisioner-works)). + +--- + +## Manual setup: fresh NixOS Incus VM → box + +If you have a NixOS Incus VM and want to turn it into a box host +without using the Coder workspace template, follow these steps inside the VM. + +### 1. Clone the repo + +```sh +git clone https://github.com/coder/box /etc/nixos-repo +ln -sf /etc/nixos-repo/flake.nix /etc/nixos/flake.nix +``` + +### 2. Write the runtime config files + +Incus writes these automatically when using the `incus-vm` Coder template, but +for a manual setup you create them yourself. + +**`/etc/nixos/incus.nix`** — sets the hostname to match the Incus instance name: + +```nix +{ lib, ... }: +{ + networking.hostName = lib.mkForce "your-vm-name"; +} +``` + +**`/etc/nixos/coder.nix`** — declares the workspace user and coder-agent service. +Copy and adapt from the example in `hosts/incus-vm/default.nix`, or use the +minimal form below: + +```nix +{ pkgs, ... }: +{ + users.users.coder = { + isNormalUser = true; + uid = 1000; + home = "/home/coder"; + shell = pkgs.bash; + extraGroups = [ "wheel" ]; + }; + security.sudo.wheelNeedsPassword = false; + + systemd.services.coder-agent = { + description = "Coder Agent"; + after = [ "network-online.target" ]; + wants = [ "network-online.target" ]; + wantedBy = [ "multi-user.target" ]; + serviceConfig = { + User = "coder"; + EnvironmentFile = "/opt/coder/init.env"; + ExecStart = "/opt/coder/init"; + Restart = "always"; + RestartSec = 10; + }; + }; +} +``` + +### 3. Create the host directory + +The flake auto-discovers hosts by folder name. The folder name must match the +hostname you set in `/etc/nixos/incus.nix`: + +```sh +mkdir -p /etc/nixos-repo/hosts/your-vm-name +cp /etc/nixos-repo/hosts/incus-vm/default.nix \ + /etc/nixos-repo/hosts/your-vm-name/default.nix +cp /etc/nixos-repo/hosts/incus-vm/incus-vm.nix \ + /etc/nixos-repo/hosts/your-vm-name/incus-vm.nix +``` + +### 4. Enable k3s (optional) + +Edit `hosts/your-vm-name/default.nix` and add one of: + +```nix +# sysbox-runc — required for the k3s-sysbox workspace template (full Docker per workspace) +services.coder-nixos.k3s-sysbox.enable = true; +``` + +```nix +# rootless Podman — lighter option, works with k3s-podman and k3s-dev templates +services.coder-nixos.k3s.enable = true; +``` + +> `k3s-sysbox.nix` and `k3s-podman.nix` use different option names to avoid +> conflicts — only enable one. + +> **Note:** `k3s-sysbox` requires `rsync` on the host. `nixos/k3s-sysbox.nix` +> includes it in `environment.systemPackages` automatically. If rsync is absent, +> `sysbox-mgr` exits at startup with +> `preflight check failed: rsync is not installed on host` and pods stay stuck in +> `ContainerCreating`. + +### 5. Apply + +```sh +nixos-rebuild switch --flake /etc/nixos-repo#your-vm-name --impure +``` + +`--impure` is required because `/etc/nixos/incus.nix` and `/etc/nixos/coder.nix` +live outside the flake tree at absolute paths. + +--- + +## How the provisioner works + +When using the [incus-vm Coder template](https://registry.coder.com/templates/coder/incus), +the provisioner does the above automatically on every workspace start: + +1. Clones this repo to `/etc/nixos-repo` (or pulls if already present) +2. Symlinks `/etc/nixos/flake.nix` → `/etc/nixos-repo/flake.nix` +3. Writes `/etc/nixos/incus.nix` (hostname) and `/etc/nixos/coder.nix` + (coder-agent service + workspace user) — runtime files that live outside the flake +4. Creates `hosts//` and copies `incus-vm.nix` + a `default.nix` + that imports `./incus-vm.nix`, `/etc/nixos/incus.nix`, `/etc/nixos/coder.nix` +5. Runs `nixos-rebuild switch --flake /etc/nixos-repo# --impure` +6. Restarts `coder-agent.service` to pick up the fresh token + +This runs on every workspace start, so token rotation is handled automatically. diff --git a/hosts/incus-vm/default.nix b/hosts/incus-vm/default.nix new file mode 100644 index 0000000..ef1f687 --- /dev/null +++ b/hosts/incus-vm/default.nix @@ -0,0 +1,21 @@ +# Template host config for any Incus VM provisioned by the incus-vm Coder template. +# +# The provisioner writes this file (or a copy) to hosts//default.nix +# at workspace start. --impure is required because /etc/nixos/incus.nix and +# /etc/nixos/coder.nix are runtime files outside the flake tree. +# +# To enable k3s add one of these in your host's default.nix: +# services.coder-nixos.k3s-sysbox.enable = true; # sysbox-runc (Docker per workspace) +# services.coder-nixos.k3s.enable = true; # rootless Podman variant + +{ lib, ... }: + +{ + imports = [ + ./incus-vm.nix # QEMU guest agents, networkd DHCP, no desktop stack + /etc/nixos/incus.nix # hostname — written by incus-virtual-machine init + /etc/nixos/coder.nix # coder-agent service + workspace user (token, URL) + ]; + + system.stateVersion = "25.11"; +} diff --git a/hosts/incus-vm/incus-vm.nix b/hosts/incus-vm/incus-vm.nix new file mode 100644 index 0000000..b33cd14 --- /dev/null +++ b/hosts/incus-vm/incus-vm.nix @@ -0,0 +1,75 @@ +# NixOS module for Incus VM guests. +# +# Import this in hosts//default.nix for any machine that lives +# inside an Incus virtual machine (as opposed to bare-metal or LXC). +# +# What it does: +# - Imports the upstream incus-virtual-machine.nix profile (QEMU guest +# agents, virtio drivers, auto-resize, systemd-boot). +# - Switches networking to systemd-networkd + DHCP on enp5s0 (the +# default virtio NIC Incus assigns to x86_64 VMs). +# - Disables the full desktop stack (KDE, PipeWire, printing, Avahi) +# that configuration.nix enables by default — a VM only needs the +# Coder server + PostgreSQL. +# +# Usage in hosts//default.nix: +# +# imports = [ +# ../../../nixos/incus-vm.nix +# ./local.nix +# ./coder-agent.nix # copy of /etc/nixos/coder.nix from provisioner +# ]; +# +# The provisioner (nixos.tf in the incus-vm Coder template) is responsible +# for: +# - cloning https://github.com/coder/box to /etc/nixos-repo +# - symlinking /etc/nixos/flake.nix -> /etc/nixos-repo/flake.nix +# - creating hosts// with local.nix + coder-agent.nix +# - running: nixos-rebuild switch --flake /etc/nixos-repo# + +{ lib, modulesPath, ... }: + +{ + imports = [ + # Use path concatenation (not string interpolation) so this works in + # pure eval mode when building from a flake. + (modulesPath + "/virtualisation/incus-virtual-machine.nix") + ]; + + # Incus VMs get a virtio NIC named enp5s0 on x86_64. + # Use systemd-networkd instead of dhcpcd (already disabled by + # incus-virtual-machine.nix, but be explicit). + networking = { + dhcpcd.enable = false; + useDHCP = false; + useHostResolvConf = false; + }; + + systemd.network = { + enable = true; + networks."50-enp5s0" = { + matchConfig.Name = "enp5s0"; + networkConfig = { + DHCP = "ipv4"; + IPv6AcceptRA = true; + }; + linkConfig.RequiredForOnline = "routable"; + }; + }; + + # The full desktop stack from configuration.nix is not needed in a VM. + # Use lib.mkForce to win against the lib.mkDefault values set there. + services.xserver.enable = lib.mkForce false; + services.displayManager.sddm.enable = lib.mkForce false; + services.desktopManager.plasma6.enable = lib.mkForce false; + services.pipewire.enable = lib.mkForce false; + services.pulseaudio.enable = lib.mkForce false; + security.rtkit.enable = lib.mkForce false; + services.printing.enable = lib.mkForce false; + services.avahi.enable = lib.mkForce false; + + # Incus VMs don't need k3s-sysbox by default (the shared config enables it + # via lib.mkDefault; mkForce wins here). Enable explicitly in the host's + # default.nix if you need k3s in the VM. + services.coder-nixos.k3s-sysbox.enable = lib.mkForce false; +} From 02e266cca961a8321d8d67c7db502e131080bf4e Mon Sep 17 00:00:00 2001 From: Ben Potter Date: Thu, 4 Jun 2026 21:23:36 -0500 Subject: [PATCH 02/10] fix(k3s-sysbox): add rsync, fix kubectl path, fix empty hostAliases MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - nixos/k3s-sysbox.nix: add rsync to systemPackages; sysbox-mgr does a preflight check for rsync and exits if absent, leaving pods stuck in ContainerCreating - coderd/templates/k3s-sysbox/main.tf: replace hardcoded Nix store path for kubectl with /run/current-system/sw/bin/k3s (stable across rebuilds) - coderd/templates/k3s-sysbox/main.tf: fix hostAliases — was [{}] (empty object) which k8s rejects; now only added when coder_lan_ip is non-empty --- coderd/templates/k3s-sysbox/main.tf | 26 ++++++++------- nixos/k3s-sysbox.nix | 52 ++++++++++++++++------------- 2 files changed, 42 insertions(+), 36 deletions(-) diff --git a/coderd/templates/k3s-sysbox/main.tf b/coderd/templates/k3s-sysbox/main.tf index 47e918b..d0c48af 100644 --- a/coderd/templates/k3s-sysbox/main.tf +++ b/coderd/templates/k3s-sysbox/main.tf @@ -26,14 +26,14 @@ variable "coder_lan_ip" { description = "LAN IP of the Coder server host, injected into pod hostAliases so workspaces can resolve the hostname without mDNS. Set via services.coder-nixos.lanIp in the host's local.nix." } -# ── Provider config ──────────────────────────────────────────────────────── +# ── Provider config ─────────────────────────────────────────────────── provider "kubernetes" { config_path = "/etc/rancher/k3s/k3s.yaml" } provider "coder" {} -# ── Workspace data ───────────────────────────────────────────────────────── +# ── Workspace data ─────────────────────────────────────────────────── data "coder_provisioner" "me" {} data "coder_workspace" "me" {} data "coder_workspace_owner" "me" {} @@ -44,10 +44,10 @@ locals { ws_name = lower(data.coder_workspace.me.name) prefix = "coder-${data.coder_workspace.me.id}" pod_name = "${local.prefix}-pod" - kubectl = "/nix/store/5bx4yn3kq9rgyaianxxldnr2ffr2k5fr-k3s-1.34.5+k3s1/bin/k3s kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml" + kubectl = "/run/current-system/sw/bin/k3s kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml" } -# ── Parameters ──────────────────────────────────────────────────────────── +# ── Parameters ──────────────────────────────────────────────────── data "coder_parameter" "image" { name = "image" display_name = "Container image" @@ -164,7 +164,7 @@ data "coder_parameter" "home_disk_size" { } } -# ── Coder agent ──────────────────────────────────────────────────────────── +# ── Coder agent ───────────────────────────────────────────────────── resource "coder_agent" "main" { arch = data.coder_provisioner.me.arch os = "linux" @@ -204,7 +204,7 @@ resource "coder_agent" "main" { REPO_URL="${data.coder_parameter.repo_url.value}" if [ -n "$REPO_URL" ]; then REPO_DIR=$(basename "$REPO_URL" .git) - if [ ! -d ~/"$REPO_DIR" ]; then + if [ ! -d ~/$"$REPO_DIR" ]; then echo "Cloning $REPO_URL..." git clone "$REPO_URL" ~/"$REPO_DIR" fi @@ -262,7 +262,7 @@ resource "coder_agent" "main" { } } -# ── VS Code Web ──────────────────────────────────────────────────────────── +# ── VS Code Web ──────────────────────────────────────────────────────── module "vscode-web" { count = data.coder_workspace.me.start_count source = "registry.coder.com/coder/vscode-web/coder" @@ -276,7 +276,7 @@ module "vscode-web" { order = 1 } -# ── Cursor ───────────────────────────────────────────────────────────────── +# ── Cursor ─────────────────────────────────────────────────────────── module "cursor" { count = data.coder_workspace.me.start_count source = "registry.coder.com/coder/cursor/coder" @@ -286,7 +286,7 @@ module "cursor" { order = 2 } -# ── JetBrains (Toolbox) ───────────────────────────────────────────────────── +# ── JetBrains (Toolbox) ────────────────────────────────────────────────────── module "jetbrains" { count = data.coder_workspace.me.start_count source = "registry.coder.com/coder/jetbrains/coder" @@ -299,7 +299,7 @@ module "jetbrains" { coder_app_order = 3 } -# ── Persistent home PVC ─────────────────────────────────────────────────── +# ── Persistent home PVC ─────────────────────────────────────────────── resource "kubernetes_persistent_volume_claim_v1" "home" { metadata { name = "${local.prefix}-home" @@ -385,8 +385,10 @@ resource "terraform_data" "workspace" { hostUsers = false restartPolicy = "OnFailure" hostname = "${local.owner}-${local.ws_name}" - hostAliases = [{ - }] + hostAliases = length(var.coder_lan_ip) > 0 ? [{ + ip = var.coder_lan_ip + hostnames = [var.coder_hostname] + }] : [] containers = [{ name = "workspace" image = data.coder_parameter.image.value diff --git a/nixos/k3s-sysbox.nix b/nixos/k3s-sysbox.nix index 65425a6..c0c6cb9 100644 --- a/nixos/k3s-sysbox.nix +++ b/nixos/k3s-sysbox.nix @@ -25,7 +25,7 @@ let # See pkgs/sysbox-runc.nix for build details and update instructions. sysboxRunc = pkgs.callPackage ../pkgs/sysbox-runc.nix {}; - # ── Sysbox v0.6.7 CE — sysbox-mgr and sysbox-fs from .deb ─────────────── + # ── Sysbox v0.6.7 CE — sysbox-mgr and sysbox-fs from .deb ─────────── # # 0.6.7 is the latest released .deb (May 2025). sysbox-runc is replaced by # the separately-built 0.7.0 binary above. @@ -80,7 +80,7 @@ let }; }; - # ── containerd v3 config template ───────────────────────────────────────── + # ── containerd v3 config template ───────────────────────────────── # # k3s with containerd 2.x reads config-v3.toml.tmpl (not config.toml.tmpl). # The v3 config schema uses plugin namespace io.containerd.cri.v1.runtime @@ -101,7 +101,7 @@ let {{ template "base" . }} - # ── Sysbox runtime ──────────────────────────────────────────────────────── + # ── Sysbox runtime ───────────────────────────────────────────────────────── [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.sysbox-runc] runtime_type = "io.containerd.runc.v2" @@ -110,7 +110,7 @@ let SystemdCgroup = false ''; - # ── sysctl settings required by sysbox ─────────────────────────────────── + # ── sysctl settings required by sysbox ─────────────────────────── sysboxSysctls = { "kernel.unprivileged_userns_clone" = 1; "fs.inotify.max_queued_events" = 1048576; @@ -123,7 +123,7 @@ let in { - # ── Option declaration ──────────────────────────────────────────────────── + # ── Option declaration ──────────────────────────────────────────── # Uses k3s-sysbox (not k3s) to avoid conflict with k3s-podman.nix. # Import only ONE of these two modules. options.services.coder-nixos.k3s-sysbox = { @@ -142,14 +142,14 @@ in }; }; - # ── Implementation ──────────────────────────────────────────────────────── + # ── Implementation ─────────────────────────────────────────────── config = lib.mkIf cfg.enable { - # ── Kernel parameters sysbox requires ────────────────────────────────── + # ── Kernel parameters sysbox requires ─────────────────────────── boot.kernel.sysctl = sysboxSysctls; boot.kernelParams = [ "user_namespace.enable=1" ]; - # ── subuid/subgid for kubelet user namespace allocation ─────────────── + # ── subuid/subgid for kubelet user namespace allocation ───────────── # k3s runs as root; kubelet needs a large subuid/subgid range to allocate # UID maps for pods using hostUsers: false (user namespaces). # 65536 UIDs per pod × up to 110 pods = ~7.2M UIDs needed. @@ -158,7 +158,7 @@ in subGidRanges = [{ startGid = 231072; count = 7208960; }]; }; - # ── k3s service ──────────────────────────────────────────────────────── + # ── k3s service ───────────────────────────────────────────────── services.k3s = { enable = true; role = "server"; @@ -177,7 +177,7 @@ in containerdConfigTemplate = null; }; - # ── Deploy containerd v3 config template before k3s starts ──────────── + # ── Deploy containerd v3 config template before k3s starts ────────── # # containerd 2.x (bundled with k3s 1.32+) uses config-v3.toml.tmpl. # The NixOS k3s module only writes config.toml.tmpl (v2 schema), which @@ -211,7 +211,7 @@ in }; }; - # ── kubeconfig ownership fix + API readiness wait ──────────────────── + # ── kubeconfig ownership fix + API readiness wait ────────────────── # Two ExecStartPost scripts run in order: # 1. k3s-fix-kubeconfig — waits for k3s.yaml and fixes ownership # 2. k3s-wait-api-ready — polls /readyz until the API accepts requests, @@ -229,7 +229,9 @@ in (pkgs.writeShellScript "k3s-wait-api-ready" '' echo "k3s-wait-api-ready: waiting for API server /readyz..." for i in $(seq 1 120); do - if ${pkgs.curl}/bin/curl -sf -o /dev/null --cacert /var/lib/rancher/k3s/server/tls/server-ca.crt https://127.0.0.1:6443/readyz 2>/dev/null; then + if ${pkgs.curl}/bin/curl -sf -o /dev/null \ + --cacert /var/lib/rancher/k3s/server/tls/server-ca.crt \ + https://127.0.0.1:6443/readyz 2>/dev/null; then echo "k3s-wait-api-ready: API ready after ''${i}s" exit 0 fi @@ -240,7 +242,7 @@ in '') ]; - # ── sysbox-mgr daemon ────────────────────────────────────────────────── + # ── sysbox-mgr daemon ──────────────────────────────────────────── systemd.services.sysbox-mgr = { description = "sysbox-mgr (part of the Sysbox container runtime)"; wantedBy = [ "multi-user.target" ]; @@ -265,7 +267,7 @@ in }; }; - # ── sysbox-fs daemon ─────────────────────────────────────────────────── + # ── sysbox-fs daemon ───────────────────────────────────────────── systemd.services.sysbox-fs = { description = "sysbox-fs (part of the Sysbox container runtime)"; wantedBy = [ "multi-user.target" ]; @@ -289,7 +291,7 @@ in }; }; - # ── sysbox umbrella target ───────────────────────────────────────────── + # ── sysbox umbrella target ─────────────────────────────────────── systemd.services.sysbox = { description = "Sysbox container runtime"; wantedBy = [ "multi-user.target" ]; @@ -308,7 +310,7 @@ in }; }; - # ── RuntimeClass auto-deploy manifest ───────────────────────────────── + # ── RuntimeClass auto-deploy manifest ───────────────────────────── # k3s auto-deploys YAML files placed in /var/lib/rancher/k3s/server/manifests/ # The NixOS k3s module accepts an attrset of submodules with a `content` key. services.k3s.manifests."sysbox-runtimeclass".content = { @@ -318,7 +320,7 @@ in handler = "sysbox-runc"; }; - # ── Pre-create the coder-workspaces namespace ──────────────────────────── + # ── Pre-create the coder-workspaces namespace ───────────────────────── # Templates deploy workloads into this namespace; they should not own it. systemd.services.coder-k3s-namespace = { description = "Create coder-workspaces namespace in k3s"; @@ -344,26 +346,28 @@ in }; }; - # ── KUBECONFIG system-wide ──────────────────────────────────────────── + # ── KUBECONFIG system-wide ──────────────────────────────────────── environment.variables.KUBECONFIG = "/etc/rancher/k3s/k3s.yaml"; - # ── Inject KUBECONFIG into coder.service ───────────────────────────── + # ── Inject KUBECONFIG into coder.service ────────────────────────── systemd.services.coder = { environment = { KUBECONFIG = "/etc/rancher/k3s/k3s.yaml"; }; }; - # ── Ensure coder group exists ───────────────────────────────────────── + # ── Ensure coder group exists ──────────────────────────────────── users.groups.coder = lib.mkDefault {}; - # ── Additional packages ─────────────────────────────────────────────── - environment.systemPackages = with pkgs; [ kubectl kubernetes-helm fuse fuse3 ]; + # ── Additional packages ─────────────────────────────────────────── + # rsync is required by sysbox-mgr preflight check; without it, + # sysbox-mgr exits immediately and pods stay stuck in ContainerCreating. + environment.systemPackages = with pkgs; [ kubectl kubernetes-helm fuse fuse3 rsync ]; - # ── FUSE: allow non-root mounts (needed by sysbox-fs) ───────────────── + # ── FUSE: allow non-root mounts (needed by sysbox-fs) ────────────── programs.fuse.userAllowOther = true; - # ── Firewall ────────────────────────────────────────────────────────── + # ── Firewall ───────────────────────────────────────────────────── networking.firewall.allowedTCPPorts = lib.mkIf config.networking.firewall.enable [ 6443 ]; }; # end config From 94d9958ef80343d68f2fa9eb6e439640c0a39e4b Mon Sep 17 00:00:00 2001 From: Ben Potter Date: Thu, 4 Jun 2026 21:27:40 -0500 Subject: [PATCH 03/10] fix(configuration.nix): hostname path and template-sync state dir ownership Two bugs found during Incus VM testing: 1. coder-init-admin: bare `hostname -s` fails in systemd units because /usr/bin is not in PATH. Use ${pkgs.nettools}/bin/hostname -s. 2. coder-template-sync activation script: `mkdir -p "$STATE_DIR"` runs as root but terraform runs as the `coder` user. Add `chown coder:coder "$STATE_DIR"` so coder can write .terraform state. Also cleans up mangled single-line shell in coder-reset and coder-workspace-reaper into readable multiline form. --- configuration.nix | 64 ++++++++++++++++++++++++++++------------------- 1 file changed, 38 insertions(+), 26 deletions(-) diff --git a/configuration.nix b/configuration.nix index 1c88c8f..fa4b273 100644 --- a/configuration.nix +++ b/configuration.nix @@ -49,7 +49,7 @@ in ./nixos/screenconnect.nix # optional ScreenConnect client (enable in hosts//local.nix) ]; - # ── NixOS option: SSH key sync ───────────────────────────────────────────── + # ── NixOS option: SSH key sync ─────────────────────────────────────────────── # Set in hosts//local.nix: services.coder-sync-ssh-keys.githubUsers = [ "user1" ]; options.services.coder-nixos.lanIp = lib.mkOption { type = lib.types.str; @@ -65,7 +65,7 @@ in config = { - # ── Platform ────────────────────────────────────────────────────────────── + # ── Platform ──────────────────────────────────────────────────────────────── # Fallback architecture. Since flake.nix no longer hardcodes `system` in # lib.nixosSystem, something must set nixpkgs.hostPlatform. The facter # module (nixos/modules/hardware/facter/system.nix) and any host's @@ -79,7 +79,7 @@ in # in hosts//default.nix (or local.nix). nixpkgs.hostPlatform = lib.mkOptionDefault "x86_64-linux"; - # ── Terraform: prebuilt binary, not from source ─────────────────────────── + # ── Terraform: prebuilt binary, not from source ────────────────────────────────── # Terraform is BSL-licensed, so cache.nixos.org does not distribute it and # `pkgs.terraform` would compile the multi-GB Go project from source during # `nixos-install`. On the small live-USB build environment that exhausts the @@ -105,7 +105,7 @@ in # compressed in-RAM swap device instead, sized to half of RAM. zramSwap.enable = lib.mkDefault true; - # ── Networking ──────────────────────────────────────────────────────────── + # ── Networking ────────────────────────────────────────────────────────────── # networking.hostName is set by flake.nix's mkHost to the host folder # name; per-host modules can override with lib.mkForce in # hosts//local.nix or default.nix. @@ -118,7 +118,7 @@ in publish = { enable = true; addresses = true; workstation = true; }; }; - # ── Locale / time ───────────────────────────────────────────────────────── + # ── Locale / time ───────────────────────────────────────────────────────────── time.timeZone = "America/Chicago"; i18n.defaultLocale = "en_US.UTF-8"; i18n.extraLocaleSettings = { @@ -133,7 +133,7 @@ in LC_TIME = "en_US.UTF-8"; }; - # ── Desktop: KDE Plasma 6 ───────────────────────────────────────────────── + # ── Desktop: KDE Plasma 6 ─────────────────────────────────────────────────── services.xserver.enable = true; services.displayManager.sddm.enable = true; services.displayManager.sddm.wayland.enable = false; @@ -141,7 +141,7 @@ in services.desktopManager.plasma6.enable = true; services.xserver.xkb = { layout = "us"; variant = ""; }; - # ── Audio ───────────────────────────────────────────────────────────────── + # ── Audio ────────────────────────────────────────────────────────────────── services.pulseaudio.enable = false; security.rtkit.enable = true; services.pipewire = { @@ -153,14 +153,14 @@ in services.printing.enable = true; - # ── Users ───────────────────────────────────────────────────────────────── + # ── Users ─────────────────────────────────────────────────────────────────── # Desktop / SSH login user is declared per-host in local.nix (template # in local.nix.example); username and password are install-time flags. # The `coder` system user (uid 991) is shared and declared further down. security.sudo.wheelNeedsPassword = false; - # ── SSH ─────────────────────────────────────────────────────────────────── + # ── SSH ─────────────────────────────────────────────────────────────────────── services.openssh = { enable = true; settings.PasswordAuthentication = true; @@ -170,7 +170,7 @@ in ''; }; - # ── SSH key sync from GitHub usernames ──────────────────────────────────── + # ── SSH key sync from GitHub usernames ────────────────────────────────────── # Fetches https://github.com/.keys for each username in # services.coder-sync-ssh-keys.githubUsers (set in hosts//local.nix). # Writes keys to /etc/ssh/authorized_keys.d/. Runs at boot. @@ -215,7 +215,7 @@ in }; }; - # ── Packages ────────────────────────────────────────────────────────────── + # ── Packages ──────────────────────────────────────────────────────────────── programs.firefox.enable = true; nixpkgs.config.allowUnfree = true; @@ -227,7 +227,7 @@ in nix.settings.download-buffer-size = 268435456; # 256 MiB; quiets the "buffer full" warning on big closure pulls networking.firewall.enable = false; - # ── PostgreSQL ──────────────────────────────────────────────────────────── + # ── PostgreSQL ──────────────────────────────────────────────────────────────── services.postgresql = { enable = true; package = pkgs.postgresql; @@ -241,7 +241,7 @@ in ''; }; - # ── Rootless Podman ─────────────────────────────────────────────────────── + # ── Rootless Podman ───────────────────────────────────────────────────────── # Used by Coder workspace templates via the Docker-compatible socket API. # dockerCompat installs a `docker` shim that redirects to podman so # workspace tooling that hard-codes `docker` (the coder-cli template, host @@ -258,7 +258,7 @@ in # actually has k3s running; hosts can opt out from their local.nix. services.coder-nixos.k3s-sysbox.enable = lib.mkDefault true; - # ── Coder user ──────────────────────────────────────────────────────────── + # ── Coder user ────────────────────────────────────────────────────────────── # UID 991 is below 1000 so isSystemUser is required (isNormalUser rejects it). # linger = true ensures the user session (and Podman socket) starts at boot. users.users.coder = { @@ -289,7 +289,7 @@ in "z /etc/coder/session-token 0600 coder coder -" ]; - # ── Coder server ────────────────────────────────────────────────────────── + # ── Coder server ────────────────────────────────────────────────────────────── # Base env vars live here. Secrets (admin creds, OAuth, etc.) are merged in # via systemd.services.coder.environment in hosts//local.nix; no EnvironmentFile. systemd.services.coder = { @@ -329,7 +329,7 @@ in }; }; - # ── Admin user bootstrap ────────────────────────────────────────────────── + # ── Admin user bootstrap ────────────────────────────────────────────────────── # Reads CODER_ADMIN_* from coder.service environment (set via local.nix). # Creates a local admin account once; sentinel prevents re-running. # If CODER_ADMIN_EMAIL is unset, skips and directs user to the browser wizard. @@ -360,7 +360,7 @@ in if [ -z "''${CODER_ADMIN_EMAIL:-}" ]; then echo "CODER_ADMIN_EMAIL not set, skipping bootstrap." - echo "Complete the first-run wizard at http://$(hostname -s).local:3000" + echo "Complete the first-run wizard at http://$(${pkgs.nettools}/bin/hostname -s).local:3000" exit 0 fi @@ -454,7 +454,7 @@ in }; - # ── Coder reset (on-demand) ─────────────────────────────────────────────── + # ── Coder reset (on-demand) ──────────────────────────────────────────────────── # Tears down all workspace pods/PVCs, wipes the Coder DB and data dir, # re-bootstraps the admin user, mints a fresh session token, and runs # nixos-rebuild switch to push templates back to Coder — fully automated. @@ -515,8 +515,13 @@ in # 8. Mint a fresh long-lived session token using the admin creds from local.nix echo "--- minting session token" - SESSION=$(${pkgs.curl}/bin/curl -sf -X POST http://localhost:3000/api/v2/users/login -H 'Content-Type: application/json' -d "{"email":"''${CODER_ADMIN_EMAIL}","password":"''${CODER_ADMIN_PASSWORD}"}" | ${pkgs.jq}/bin/jq -r '.session_token') - LONG_TOKEN=$(CODER_URL=http://localhost:3000 CODER_SESSION_TOKEN="$SESSION" ${coder}/bin/coder tokens create --name nixos-sync --lifetime 8760h) + SESSION=$(${pkgs.curl}/bin/curl -sf \ + -X POST http://localhost:3000/api/v2/users/login \ + -H 'Content-Type: application/json' \ + -d "{\"email\":\"''${CODER_ADMIN_EMAIL}\",\"password\":\"''${CODER_ADMIN_PASSWORD}\"}" \ + | ${pkgs.jq}/bin/jq -r '.session_token') + LONG_TOKEN=$(CODER_URL=http://localhost:3000 CODER_SESSION_TOKEN="$SESSION" \ + ${coder}/bin/coder tokens create --name nixos-sync --lifetime 8760h) echo "$LONG_TOKEN" | ${pkgs.coreutils}/bin/tee /etc/coder/session-token > /dev/null echo "--- session token written" @@ -536,7 +541,7 @@ in }; }; - # ── Template sync activation script ────────────────────────────────────── + # ── Template sync activation script ──────────────────────────────────────────── # Runs on every `nixos-rebuild switch`. Uses terraform-provider-coderd to # apply templates from /etc/nixos-repo/coderd/. # /etc/coder/session-token is populated automatically by @@ -554,6 +559,7 @@ in echo " This file is auto-populated by coder-init-admin.service on first boot." else mkdir -p "$STATE_DIR" + chown coder:coder "$STATE_DIR" 2>/dev/null || true COMMIT=$(GIT_DIR=/etc/nixos-repo/.git ${pkgs.git}/bin/git -c safe.directory=/etc/nixos-repo -C /etc/nixos-repo rev-parse --short HEAD 2>/dev/null || echo "unknown") @@ -579,7 +585,7 @@ in # ./hosts/coder-thinkcentre/default.nix. - # ── Coder tunnel redirect ───────────────────────────────────────────────── + # ── Coder tunnel redirect ────────────────────────────────────────────────────── # Listens on port 80 (http://coder-thinkcentre.local) and issues a 302 # redirect to the live *.try.coder.app tunnel URL, which Coder sets when # CODER_ACCESS_URL is left unset. The shell wrapper discovers the URL and @@ -660,7 +666,7 @@ EOF }; - # ── Workspace reaper ────────────────────────────────────────────────────────── + # ── Workspace reaper ──────────────────────────────────────────────────────────────── # Deletes workspaces that have been stopped for >= 72 h. # time_til_dormant_autodelete_ms is Enterprise-only so we implement this # ourselves: an hourly timer calls the API, finds stopped workspaces whose @@ -688,7 +694,10 @@ EOF echo "coder-workspace-reaper: checking for workspaces stopped before $(${pkgs.coreutils}/bin/date -d @$CUTOFF --iso-8601=seconds)" - WORKSPACES=$(${pkgs.curl}/bin/curl -sf -H "Coder-Session-Token: $TOKEN" "$CODER_LOCAL/api/v2/workspaces?limit=100&filterQuery=status:stopped" | ${pkgs.jq}/bin/jq -r '.workspaces[] | .id + " " + .name + " " + .last_used_at') + WORKSPACES=$(${pkgs.curl}/bin/curl -sf \ + -H "Coder-Session-Token: $TOKEN" \ + "$CODER_LOCAL/api/v2/workspaces?limit=100&filterQuery=status:stopped" \ + | ${pkgs.jq}/bin/jq -r '.workspaces[] | .id + " " + .name + " " + .last_used_at') if [ -z "$WORKSPACES" ]; then echo "coder-workspace-reaper: no stopped workspaces found" @@ -700,7 +709,10 @@ EOF LAST_EPOCH=$(${pkgs.coreutils}/bin/date -d "$last_used" +%s 2>/dev/null || echo 0) if [ "$LAST_EPOCH" -lt "$CUTOFF" ]; then echo "coder-workspace-reaper: deleting $name ($id) — last used $last_used" - ${pkgs.curl}/bin/curl -sf -X DELETE -H "Coder-Session-Token: $TOKEN" "$CODER_LOCAL/api/v2/workspaces/$id" || echo "coder-workspace-reaper: WARNING — delete failed for $name" + ${pkgs.curl}/bin/curl -sf -X DELETE \ + -H "Coder-Session-Token: $TOKEN" \ + "$CODER_LOCAL/api/v2/workspaces/$id" || \ + echo "coder-workspace-reaper: WARNING — delete failed for $name" else echo "coder-workspace-reaper: keeping $name — stopped recently" fi @@ -721,7 +733,7 @@ EOF }; - # ── coder-logstream-kube Helm install ───────────────────────────────────── + # ── coder-logstream-kube Helm install ───────────────────────────────────────────── # Streams k3s pod events (scheduling, image pull, OOMKill, etc.) into # Coder workspace startup logs. Runs helm upgrade --install on every boot # so the chart is kept up to date after NixOS rebuilds. From 1d9ac74c9185eec19381d39b7816aaed3d55652d Mon Sep 17 00:00:00 2001 From: Ben Potter Date: Fri, 5 Jun 2026 07:31:21 -0500 Subject: [PATCH 04/10] Drop cosmetic dash-trimming, keep only functional changes nixos/k3s-sysbox.nix: only change is adding rsync to systemPackages configuration.nix: only changes are the hostname fix, chown STATE_DIR, and multiline shell reformatting of the mangled single-line curl calls (no comment banner changes) --- configuration.nix | 42 +++++++++++++++++++++--------------------- nixos/k3s-sysbox.nix | 44 ++++++++++++++++++++++---------------------- 2 files changed, 43 insertions(+), 43 deletions(-) diff --git a/configuration.nix b/configuration.nix index fa4b273..5acea3b 100644 --- a/configuration.nix +++ b/configuration.nix @@ -49,7 +49,7 @@ in ./nixos/screenconnect.nix # optional ScreenConnect client (enable in hosts//local.nix) ]; - # ── NixOS option: SSH key sync ─────────────────────────────────────────────── + # ── NixOS option: SSH key sync ───────────────────────────────────────────── # Set in hosts//local.nix: services.coder-sync-ssh-keys.githubUsers = [ "user1" ]; options.services.coder-nixos.lanIp = lib.mkOption { type = lib.types.str; @@ -65,7 +65,7 @@ in config = { - # ── Platform ──────────────────────────────────────────────────────────────── + # ── Platform ────────────────────────────────────────────────────────────── # Fallback architecture. Since flake.nix no longer hardcodes `system` in # lib.nixosSystem, something must set nixpkgs.hostPlatform. The facter # module (nixos/modules/hardware/facter/system.nix) and any host's @@ -79,7 +79,7 @@ in # in hosts//default.nix (or local.nix). nixpkgs.hostPlatform = lib.mkOptionDefault "x86_64-linux"; - # ── Terraform: prebuilt binary, not from source ────────────────────────────────── + # ── Terraform: prebuilt binary, not from source ─────────────────────────── # Terraform is BSL-licensed, so cache.nixos.org does not distribute it and # `pkgs.terraform` would compile the multi-GB Go project from source during # `nixos-install`. On the small live-USB build environment that exhausts the @@ -105,7 +105,7 @@ in # compressed in-RAM swap device instead, sized to half of RAM. zramSwap.enable = lib.mkDefault true; - # ── Networking ────────────────────────────────────────────────────────────── + # ── Networking ──────────────────────────────────────────────────────────── # networking.hostName is set by flake.nix's mkHost to the host folder # name; per-host modules can override with lib.mkForce in # hosts//local.nix or default.nix. @@ -118,7 +118,7 @@ in publish = { enable = true; addresses = true; workstation = true; }; }; - # ── Locale / time ───────────────────────────────────────────────────────────── + # ── Locale / time ───────────────────────────────────────────────────────── time.timeZone = "America/Chicago"; i18n.defaultLocale = "en_US.UTF-8"; i18n.extraLocaleSettings = { @@ -133,7 +133,7 @@ in LC_TIME = "en_US.UTF-8"; }; - # ── Desktop: KDE Plasma 6 ─────────────────────────────────────────────────── + # ── Desktop: KDE Plasma 6 ───────────────────────────────────────────────── services.xserver.enable = true; services.displayManager.sddm.enable = true; services.displayManager.sddm.wayland.enable = false; @@ -141,7 +141,7 @@ in services.desktopManager.plasma6.enable = true; services.xserver.xkb = { layout = "us"; variant = ""; }; - # ── Audio ────────────────────────────────────────────────────────────────── + # ── Audio ───────────────────────────────────────────────────────────────── services.pulseaudio.enable = false; security.rtkit.enable = true; services.pipewire = { @@ -153,14 +153,14 @@ in services.printing.enable = true; - # ── Users ─────────────────────────────────────────────────────────────────── + # ── Users ───────────────────────────────────────────────────────────────── # Desktop / SSH login user is declared per-host in local.nix (template # in local.nix.example); username and password are install-time flags. # The `coder` system user (uid 991) is shared and declared further down. security.sudo.wheelNeedsPassword = false; - # ── SSH ─────────────────────────────────────────────────────────────────────── + # ── SSH ─────────────────────────────────────────────────────────────────── services.openssh = { enable = true; settings.PasswordAuthentication = true; @@ -170,7 +170,7 @@ in ''; }; - # ── SSH key sync from GitHub usernames ────────────────────────────────────── + # ── SSH key sync from GitHub usernames ──────────────────────────────────── # Fetches https://github.com/.keys for each username in # services.coder-sync-ssh-keys.githubUsers (set in hosts//local.nix). # Writes keys to /etc/ssh/authorized_keys.d/. Runs at boot. @@ -215,7 +215,7 @@ in }; }; - # ── Packages ──────────────────────────────────────────────────────────────── + # ── Packages ────────────────────────────────────────────────────────────── programs.firefox.enable = true; nixpkgs.config.allowUnfree = true; @@ -227,7 +227,7 @@ in nix.settings.download-buffer-size = 268435456; # 256 MiB; quiets the "buffer full" warning on big closure pulls networking.firewall.enable = false; - # ── PostgreSQL ──────────────────────────────────────────────────────────────── + # ── PostgreSQL ──────────────────────────────────────────────────────────── services.postgresql = { enable = true; package = pkgs.postgresql; @@ -241,7 +241,7 @@ in ''; }; - # ── Rootless Podman ───────────────────────────────────────────────────────── + # ── Rootless Podman ─────────────────────────────────────────────────────── # Used by Coder workspace templates via the Docker-compatible socket API. # dockerCompat installs a `docker` shim that redirects to podman so # workspace tooling that hard-codes `docker` (the coder-cli template, host @@ -258,7 +258,7 @@ in # actually has k3s running; hosts can opt out from their local.nix. services.coder-nixos.k3s-sysbox.enable = lib.mkDefault true; - # ── Coder user ────────────────────────────────────────────────────────────── + # ── Coder user ──────────────────────────────────────────────────────────── # UID 991 is below 1000 so isSystemUser is required (isNormalUser rejects it). # linger = true ensures the user session (and Podman socket) starts at boot. users.users.coder = { @@ -289,7 +289,7 @@ in "z /etc/coder/session-token 0600 coder coder -" ]; - # ── Coder server ────────────────────────────────────────────────────────────── + # ── Coder server ────────────────────────────────────────────────────────── # Base env vars live here. Secrets (admin creds, OAuth, etc.) are merged in # via systemd.services.coder.environment in hosts//local.nix; no EnvironmentFile. systemd.services.coder = { @@ -329,7 +329,7 @@ in }; }; - # ── Admin user bootstrap ────────────────────────────────────────────────────── + # ── Admin user bootstrap ────────────────────────────────────────────────── # Reads CODER_ADMIN_* from coder.service environment (set via local.nix). # Creates a local admin account once; sentinel prevents re-running. # If CODER_ADMIN_EMAIL is unset, skips and directs user to the browser wizard. @@ -454,7 +454,7 @@ in }; - # ── Coder reset (on-demand) ──────────────────────────────────────────────────── + # ── Coder reset (on-demand) ─────────────────────────────────────────────── # Tears down all workspace pods/PVCs, wipes the Coder DB and data dir, # re-bootstraps the admin user, mints a fresh session token, and runs # nixos-rebuild switch to push templates back to Coder — fully automated. @@ -541,7 +541,7 @@ in }; }; - # ── Template sync activation script ──────────────────────────────────────────── + # ── Template sync activation script ────────────────────────────────────── # Runs on every `nixos-rebuild switch`. Uses terraform-provider-coderd to # apply templates from /etc/nixos-repo/coderd/. # /etc/coder/session-token is populated automatically by @@ -585,7 +585,7 @@ in # ./hosts/coder-thinkcentre/default.nix. - # ── Coder tunnel redirect ────────────────────────────────────────────────────── + # ── Coder tunnel redirect ───────────────────────────────────────────────── # Listens on port 80 (http://coder-thinkcentre.local) and issues a 302 # redirect to the live *.try.coder.app tunnel URL, which Coder sets when # CODER_ACCESS_URL is left unset. The shell wrapper discovers the URL and @@ -666,7 +666,7 @@ EOF }; - # ── Workspace reaper ──────────────────────────────────────────────────────────────── + # ── Workspace reaper ────────────────────────────────────────────────────────── # Deletes workspaces that have been stopped for >= 72 h. # time_til_dormant_autodelete_ms is Enterprise-only so we implement this # ourselves: an hourly timer calls the API, finds stopped workspaces whose @@ -733,7 +733,7 @@ EOF }; - # ── coder-logstream-kube Helm install ───────────────────────────────────────────── + # ── coder-logstream-kube Helm install ───────────────────────────────────── # Streams k3s pod events (scheduling, image pull, OOMKill, etc.) into # Coder workspace startup logs. Runs helm upgrade --install on every boot # so the chart is kept up to date after NixOS rebuilds. diff --git a/nixos/k3s-sysbox.nix b/nixos/k3s-sysbox.nix index c0c6cb9..e30f2e2 100644 --- a/nixos/k3s-sysbox.nix +++ b/nixos/k3s-sysbox.nix @@ -25,7 +25,7 @@ let # See pkgs/sysbox-runc.nix for build details and update instructions. sysboxRunc = pkgs.callPackage ../pkgs/sysbox-runc.nix {}; - # ── Sysbox v0.6.7 CE — sysbox-mgr and sysbox-fs from .deb ─────────── + # ── Sysbox v0.6.7 CE — sysbox-mgr and sysbox-fs from .deb ─────────────── # # 0.6.7 is the latest released .deb (May 2025). sysbox-runc is replaced by # the separately-built 0.7.0 binary above. @@ -80,7 +80,7 @@ let }; }; - # ── containerd v3 config template ───────────────────────────────── + # ── containerd v3 config template ───────────────────────────────────────── # # k3s with containerd 2.x reads config-v3.toml.tmpl (not config.toml.tmpl). # The v3 config schema uses plugin namespace io.containerd.cri.v1.runtime @@ -101,7 +101,7 @@ let {{ template "base" . }} - # ── Sysbox runtime ───────────────────────────────────────────────────────── + # ── Sysbox runtime ──────────────────────────────────────────────────────── [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.sysbox-runc] runtime_type = "io.containerd.runc.v2" @@ -110,7 +110,7 @@ let SystemdCgroup = false ''; - # ── sysctl settings required by sysbox ─────────────────────────── + # ── sysctl settings required by sysbox ─────────────────────────────────── sysboxSysctls = { "kernel.unprivileged_userns_clone" = 1; "fs.inotify.max_queued_events" = 1048576; @@ -123,7 +123,7 @@ let in { - # ── Option declaration ──────────────────────────────────────────── + # ── Option declaration ──────────────────────────────────────────────────── # Uses k3s-sysbox (not k3s) to avoid conflict with k3s-podman.nix. # Import only ONE of these two modules. options.services.coder-nixos.k3s-sysbox = { @@ -142,14 +142,14 @@ in }; }; - # ── Implementation ─────────────────────────────────────────────── + # ── Implementation ──────────────────────────────────────────────────────── config = lib.mkIf cfg.enable { - # ── Kernel parameters sysbox requires ─────────────────────────── + # ── Kernel parameters sysbox requires ────────────────────────────────── boot.kernel.sysctl = sysboxSysctls; boot.kernelParams = [ "user_namespace.enable=1" ]; - # ── subuid/subgid for kubelet user namespace allocation ───────────── + # ── subuid/subgid for kubelet user namespace allocation ─────────────── # k3s runs as root; kubelet needs a large subuid/subgid range to allocate # UID maps for pods using hostUsers: false (user namespaces). # 65536 UIDs per pod × up to 110 pods = ~7.2M UIDs needed. @@ -158,7 +158,7 @@ in subGidRanges = [{ startGid = 231072; count = 7208960; }]; }; - # ── k3s service ───────────────────────────────────────────────── + # ── k3s service ──────────────────────────────────────────────────────── services.k3s = { enable = true; role = "server"; @@ -177,7 +177,7 @@ in containerdConfigTemplate = null; }; - # ── Deploy containerd v3 config template before k3s starts ────────── + # ── Deploy containerd v3 config template before k3s starts ──────────── # # containerd 2.x (bundled with k3s 1.32+) uses config-v3.toml.tmpl. # The NixOS k3s module only writes config.toml.tmpl (v2 schema), which @@ -211,7 +211,7 @@ in }; }; - # ── kubeconfig ownership fix + API readiness wait ────────────────── + # ── kubeconfig ownership fix + API readiness wait ──────────────────── # Two ExecStartPost scripts run in order: # 1. k3s-fix-kubeconfig — waits for k3s.yaml and fixes ownership # 2. k3s-wait-api-ready — polls /readyz until the API accepts requests, @@ -242,7 +242,7 @@ in '') ]; - # ── sysbox-mgr daemon ──────────────────────────────────────────── + # ── sysbox-mgr daemon ────────────────────────────────────────────────── systemd.services.sysbox-mgr = { description = "sysbox-mgr (part of the Sysbox container runtime)"; wantedBy = [ "multi-user.target" ]; @@ -267,7 +267,7 @@ in }; }; - # ── sysbox-fs daemon ───────────────────────────────────────────── + # ── sysbox-fs daemon ─────────────────────────────────────────────────── systemd.services.sysbox-fs = { description = "sysbox-fs (part of the Sysbox container runtime)"; wantedBy = [ "multi-user.target" ]; @@ -291,7 +291,7 @@ in }; }; - # ── sysbox umbrella target ─────────────────────────────────────── + # ── sysbox umbrella target ───────────────────────────────────────────── systemd.services.sysbox = { description = "Sysbox container runtime"; wantedBy = [ "multi-user.target" ]; @@ -310,7 +310,7 @@ in }; }; - # ── RuntimeClass auto-deploy manifest ───────────────────────────── + # ── RuntimeClass auto-deploy manifest ───────────────────────────────── # k3s auto-deploys YAML files placed in /var/lib/rancher/k3s/server/manifests/ # The NixOS k3s module accepts an attrset of submodules with a `content` key. services.k3s.manifests."sysbox-runtimeclass".content = { @@ -320,7 +320,7 @@ in handler = "sysbox-runc"; }; - # ── Pre-create the coder-workspaces namespace ───────────────────────── + # ── Pre-create the coder-workspaces namespace ──────────────────────────── # Templates deploy workloads into this namespace; they should not own it. systemd.services.coder-k3s-namespace = { description = "Create coder-workspaces namespace in k3s"; @@ -346,28 +346,28 @@ in }; }; - # ── KUBECONFIG system-wide ──────────────────────────────────────── + # ── KUBECONFIG system-wide ──────────────────────────────────────────── environment.variables.KUBECONFIG = "/etc/rancher/k3s/k3s.yaml"; - # ── Inject KUBECONFIG into coder.service ────────────────────────── + # ── Inject KUBECONFIG into coder.service ───────────────────────────── systemd.services.coder = { environment = { KUBECONFIG = "/etc/rancher/k3s/k3s.yaml"; }; }; - # ── Ensure coder group exists ──────────────────────────────────── + # ── Ensure coder group exists ───────────────────────────────────────── users.groups.coder = lib.mkDefault {}; - # ── Additional packages ─────────────────────────────────────────── + # ── Additional packages ─────────────────────────────────────────────── # rsync is required by sysbox-mgr preflight check; without it, # sysbox-mgr exits immediately and pods stay stuck in ContainerCreating. environment.systemPackages = with pkgs; [ kubectl kubernetes-helm fuse fuse3 rsync ]; - # ── FUSE: allow non-root mounts (needed by sysbox-fs) ────────────── + # ── FUSE: allow non-root mounts (needed by sysbox-fs) ───────────────── programs.fuse.userAllowOther = true; - # ── Firewall ───────────────────────────────────────────────────── + # ── Firewall ────────────────────────────────────────────────────────── networking.firewall.allowedTCPPorts = lib.mkIf config.networking.firewall.enable [ 6443 ]; }; # end config From 5f36fd7bdc7385d3a13938ba5740b8b8944e83a3 Mon Sep 17 00:00:00 2001 From: Ben Potter Date: Fri, 5 Jun 2026 07:33:24 -0500 Subject: [PATCH 05/10] docs(hosts/incus-vm): rewrite README to reflect actual setup flow --- hosts/incus-vm/README.md | 119 +++++++++++++++++++++------------------ 1 file changed, 63 insertions(+), 56 deletions(-) diff --git a/hosts/incus-vm/README.md b/hosts/incus-vm/README.md index 59bc4be..89a39a8 100644 --- a/hosts/incus-vm/README.md +++ b/hosts/incus-vm/README.md @@ -1,49 +1,87 @@ -# hosts/incus-vm — Running box on an Incus VM +# hosts/incus-vm — Running box on a headless host -This directory contains the NixOS configuration for running box inside -an Incus virtual machine, instead of on bare metal. +This directory holds the NixOS module and template `default.nix` for running box +on any **headless host** — Incus VM, a bare-metal machine like a ThinkStation, or +any other server that doesn't need the KDE desktop stack. -`incus-vm.nix` handles everything that differs from a bare-metal host: +`incus-vm.nix` handles everything that differs from a normal bare-metal desktop host: -- QEMU guest agents and virtio drivers (via the upstream `incus-virtual-machine.nix` profile) +- QEMU guest agents and virtio drivers (via the upstream `incus-virtual-machine.nix` + profile) — skip this import for a bare-metal host - `systemd-networkd` DHCP on `enp5s0` (the virtio NIC Incus assigns to x86_64 VMs) -- Disables the KDE/PipeWire/printing/Avahi stack that `configuration.nix` enables - by default — a headless VM only needs Coder + PostgreSQL +- Disables the KDE / PipeWire / printing / Avahi stack that `configuration.nix` + enables by default — a headless host only needs Coder + PostgreSQL -`default.nix` is the template that gets copied to `hosts//` when a new -VM is provisioned (see [How the provisioner works](#how-the-provisioner-works)). +`default.nix` is the per-host entrypoint. Copy it to `hosts//` and +import it from the flake (auto-discovered by hostname). --- -## Manual setup: fresh NixOS Incus VM → box +## Relationship to the incus-nixos registry template -If you have a NixOS Incus VM and want to turn it into a box host -without using the Coder workspace template, follow these steps inside the VM. +[`registry.coder.com/templates/bpmct/incus-nixos`](https://registry.coder.com/templates/bpmct/incus-nixos) +is a separate, standalone Coder workspace template. It provisions a plain NixOS VM +as a Coder workspace using `nixos-rebuild switch` via `incus exec`. It does **not** +use this flake or any part of the box stack — it writes its own minimal +`configuration.nix` and `coder.nix` at first boot. + +The two are complementary but independent: + +| | `bpmct/incus-nixos` registry template | `hosts/incus-vm/` in this repo | +|---|---|---| +| Purpose | Provision any NixOS VM as a Coder workspace | Turn a host into a box provisioner | +| Uses box flake? | No | Yes — `nixos-rebuild switch --flake /etc/nixos-repo#` | +| Sets up k3s / sysbox? | No | Optional — add to `hosts//default.nix` | +| Who runs it? | Coder Terraform provisioner | You, manually, on the host | + +--- + +## Manual setup: fresh NixOS host → box + +These steps work for an Incus VM **and** for a bare-metal machine (ThinkStation, +etc.). The only difference is which extra modules you import in `default.nix`. ### 1. Clone the repo ```sh git clone https://github.com/coder/box /etc/nixos-repo -ln -sf /etc/nixos-repo/flake.nix /etc/nixos/flake.nix ``` -### 2. Write the runtime config files +### 2. Create the host directory + +The flake auto-discovers hosts by folder name. The folder name must match the +machine's hostname (`hostname -s`): + +```sh +HOSTNAME=$(hostname -s) +mkdir -p /etc/nixos-repo/hosts/$HOSTNAME + +# For an Incus VM — copy the incus-vm template: +cp /etc/nixos-repo/hosts/incus-vm/default.nix \ + /etc/nixos-repo/hosts/$HOSTNAME/default.nix +cp /etc/nixos-repo/hosts/incus-vm/incus-vm.nix \ + /etc/nixos-repo/hosts/$HOSTNAME/incus-vm.nix + +# For a bare-metal host — start from a different base or write your own default.nix. +# See hosts/qemu-arm64/ for an example of the bare-metal layout. +``` + +### 3. Write the runtime config files -Incus writes these automatically when using the `incus-vm` Coder template, but -for a manual setup you create them yourself. +These files live outside the flake tree so they can carry secrets and +machine-specific values without being committed. -**`/etc/nixos/incus.nix`** — sets the hostname to match the Incus instance name: +**`/etc/nixos/incus.nix`** — sets the hostname (Incus VMs get this written +automatically by `incus-virtual-machine.nix`; create it manually on bare metal): ```nix { lib, ... }: { - networking.hostName = lib.mkForce "your-vm-name"; + networking.hostName = lib.mkForce "your-hostname"; } ``` -**`/etc/nixos/coder.nix`** — declares the workspace user and coder-agent service. -Copy and adapt from the example in `hosts/incus-vm/default.nix`, or use the -minimal form below: +**`/etc/nixos/coder.nix`** — declares the workspace user and coder-agent service: ```nix { pkgs, ... }: @@ -73,22 +111,9 @@ minimal form below: } ``` -### 3. Create the host directory - -The flake auto-discovers hosts by folder name. The folder name must match the -hostname you set in `/etc/nixos/incus.nix`: - -```sh -mkdir -p /etc/nixos-repo/hosts/your-vm-name -cp /etc/nixos-repo/hosts/incus-vm/default.nix \ - /etc/nixos-repo/hosts/your-vm-name/default.nix -cp /etc/nixos-repo/hosts/incus-vm/incus-vm.nix \ - /etc/nixos-repo/hosts/your-vm-name/incus-vm.nix -``` - ### 4. Enable k3s (optional) -Edit `hosts/your-vm-name/default.nix` and add one of: +Edit `hosts//default.nix` and add one of: ```nix # sysbox-runc — required for the k3s-sysbox workspace template (full Docker per workspace) @@ -100,8 +125,8 @@ services.coder-nixos.k3s-sysbox.enable = true; services.coder-nixos.k3s.enable = true; ``` -> `k3s-sysbox.nix` and `k3s-podman.nix` use different option names to avoid -> conflicts — only enable one. +> `k3s-sysbox` and `k3s` use different option names to avoid conflicts — only +> enable one. > **Note:** `k3s-sysbox` requires `rsync` on the host. `nixos/k3s-sysbox.nix` > includes it in `environment.systemPackages` automatically. If rsync is absent, @@ -112,26 +137,8 @@ services.coder-nixos.k3s.enable = true; ### 5. Apply ```sh -nixos-rebuild switch --flake /etc/nixos-repo#your-vm-name --impure +nixos-rebuild switch --flake /etc/nixos-repo#$(hostname -s) --impure ``` `--impure` is required because `/etc/nixos/incus.nix` and `/etc/nixos/coder.nix` live outside the flake tree at absolute paths. - ---- - -## How the provisioner works - -When using the [incus-vm Coder template](https://registry.coder.com/templates/coder/incus), -the provisioner does the above automatically on every workspace start: - -1. Clones this repo to `/etc/nixos-repo` (or pulls if already present) -2. Symlinks `/etc/nixos/flake.nix` → `/etc/nixos-repo/flake.nix` -3. Writes `/etc/nixos/incus.nix` (hostname) and `/etc/nixos/coder.nix` - (coder-agent service + workspace user) — runtime files that live outside the flake -4. Creates `hosts//` and copies `incus-vm.nix` + a `default.nix` - that imports `./incus-vm.nix`, `/etc/nixos/incus.nix`, `/etc/nixos/coder.nix` -5. Runs `nixos-rebuild switch --flake /etc/nixos-repo# --impure` -6. Restarts `coder-agent.service` to pick up the fresh token - -This runs on every workspace start, so token rotation is handled automatically. From 3d9ee117fd1329bcb121f08f649d201abb660c48 Mon Sep 17 00:00:00 2001 From: Ben Potter Date: Fri, 5 Jun 2026 07:36:45 -0500 Subject: [PATCH 06/10] docs(hosts/incus-vm): add git-add note; guide verified via nixos-rebuild dry-build --- hosts/incus-vm/README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/hosts/incus-vm/README.md b/hosts/incus-vm/README.md index 89a39a8..7cebaf9 100644 --- a/hosts/incus-vm/README.md +++ b/hosts/incus-vm/README.md @@ -64,6 +64,10 @@ cp /etc/nixos-repo/hosts/incus-vm/incus-vm.nix \ # For a bare-metal host — start from a different base or write your own default.nix. # See hosts/qemu-arm64/ for an example of the bare-metal layout. + +# Stage the new host dir so the flake can discover it. +# The flake uses builtins.readDir on the git tree, so untracked files are invisible. +git -C /etc/nixos-repo add hosts/$HOSTNAME/ ``` ### 3. Write the runtime config files From 3d98f2d1aee6d1dceb811374fab163be0280dde0 Mon Sep 17 00:00:00 2001 From: Ben Potter Date: Fri, 5 Jun 2026 11:31:30 -0500 Subject: [PATCH 07/10] fix(hosts/incus-vm): remove mkForce false on k3s-sysbox; VMs can be full box hosts --- hosts/incus-vm/incus-vm.nix | 27 ++++++--------------------- 1 file changed, 6 insertions(+), 21 deletions(-) diff --git a/hosts/incus-vm/incus-vm.nix b/hosts/incus-vm/incus-vm.nix index b33cd14..d4e69ca 100644 --- a/hosts/incus-vm/incus-vm.nix +++ b/hosts/incus-vm/incus-vm.nix @@ -9,23 +9,13 @@ # - Switches networking to systemd-networkd + DHCP on enp5s0 (the # default virtio NIC Incus assigns to x86_64 VMs). # - Disables the full desktop stack (KDE, PipeWire, printing, Avahi) -# that configuration.nix enables by default — a VM only needs the -# Coder server + PostgreSQL. +# that configuration.nix enables by default — a headless VM only needs +# the Coder server + PostgreSQL. # -# Usage in hosts//default.nix: -# -# imports = [ -# ../../../nixos/incus-vm.nix -# ./local.nix -# ./coder-agent.nix # copy of /etc/nixos/coder.nix from provisioner -# ]; -# -# The provisioner (nixos.tf in the incus-vm Coder template) is responsible -# for: -# - cloning https://github.com/coder/box to /etc/nixos-repo -# - symlinking /etc/nixos/flake.nix -> /etc/nixos-repo/flake.nix -# - creating hosts// with local.nix + coder-agent.nix -# - running: nixos-rebuild switch --flake /etc/nixos-repo# +# k3s / sysbox are NOT disabled here. Set: +# services.coder-nixos.k3s-sysbox.enable = true; +# in hosts//default.nix to enable the full box stack (Coder + +# PostgreSQL + k3s + sysbox) inside the VM. { lib, modulesPath, ... }: @@ -67,9 +57,4 @@ security.rtkit.enable = lib.mkForce false; services.printing.enable = lib.mkForce false; services.avahi.enable = lib.mkForce false; - - # Incus VMs don't need k3s-sysbox by default (the shared config enables it - # via lib.mkDefault; mkForce wins here). Enable explicitly in the host's - # default.nix if you need k3s in the VM. - services.coder-nixos.k3s-sysbox.enable = lib.mkForce false; } From e34871b00c4d0d670e83e7a89d65beac6ae59475 Mon Sep 17 00:00:00 2001 From: Ben Potter Date: Fri, 5 Jun 2026 11:43:49 -0500 Subject: [PATCH 08/10] =?UTF-8?q?docs(hosts/incus-vm):=20rewrite=20?= =?UTF-8?q?=E2=80=94=20accurate=20setup=20steps,=20explain=20what=20box=20?= =?UTF-8?q?does,=20fix=20coder.nix=20confusion?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- hosts/incus-vm/README.md | 194 ++++++++++++++++++++++----------------- 1 file changed, 112 insertions(+), 82 deletions(-) diff --git a/hosts/incus-vm/README.md b/hosts/incus-vm/README.md index 7cebaf9..e693eb0 100644 --- a/hosts/incus-vm/README.md +++ b/hosts/incus-vm/README.md @@ -1,148 +1,178 @@ # hosts/incus-vm — Running box on a headless host -This directory holds the NixOS module and template `default.nix` for running box -on any **headless host** — Incus VM, a bare-metal machine like a ThinkStation, or -any other server that doesn't need the KDE desktop stack. +## What box does -`incus-vm.nix` handles everything that differs from a normal bare-metal desktop host: +Box turns a NixOS machine into a **self-contained Coder deployment**. After +`nixos-rebuild switch`, the machine runs: -- QEMU guest agents and virtio drivers (via the upstream `incus-virtual-machine.nix` - profile) — skip this import for a bare-metal host +- **Coder server** — full control plane, accessible over a Tailscale tunnel or + a configured external URL +- **PostgreSQL** — Coder's database, managed by the box flake +- **k3s + sysbox** (optional) — single-node Kubernetes cluster where Coder + provisions workspaces as pods; sysbox-runc gives each pod its own Docker daemon + without privileged mode +- **template-sync** — an activation hook that runs `terraform apply` on + `coderd/` at every `nixos-rebuild switch`, keeping Coder templates in sync with + the repo automatically + +The result: SSH or `coder ssh` into the machine, and you have a working Coder +instance. Workspace pods run on the same node via k3s. No separate infrastructure +required. + +--- + +## What this directory provides + +`incus-vm.nix` adapts the base box config for a headless VM or server: + +- Imports the upstream `incus-virtual-machine.nix` profile (QEMU guest agents, + virtio drivers) — skip this for bare-metal - `systemd-networkd` DHCP on `enp5s0` (the virtio NIC Incus assigns to x86_64 VMs) -- Disables the KDE / PipeWire / printing / Avahi stack that `configuration.nix` - enables by default — a headless host only needs Coder + PostgreSQL +- Disables the KDE / PipeWire / printing / Avahi stack — a headless host only + needs Coder + PostgreSQL -`default.nix` is the per-host entrypoint. Copy it to `hosts//` and -import it from the flake (auto-discovered by hostname). +`default.nix` is the per-host entrypoint that imports `incus-vm.nix` plus the +two runtime files from `/etc/nixos/` (see below). + +The same pattern works for bare-metal machines (ThinkStation, etc.) — just skip +`incus-vm.nix` or replace it with your own hardware module. --- ## Relationship to the incus-nixos registry template [`registry.coder.com/templates/bpmct/incus-nixos`](https://registry.coder.com/templates/bpmct/incus-nixos) -is a separate, standalone Coder workspace template. It provisions a plain NixOS VM -as a Coder workspace using `nixos-rebuild switch` via `incus exec`. It does **not** -use this flake or any part of the box stack — it writes its own minimal -`configuration.nix` and `coder.nix` at first boot. - -The two are complementary but independent: +is a **separate, unrelated** Coder workspace template. It provisions a plain NixOS +VM as a Coder *workspace* (something you SSH into to do work), using +`nixos-rebuild switch` via `incus exec`. It writes its own minimal +`configuration.nix` at first boot and has nothing to do with this flake. | | `bpmct/incus-nixos` registry template | `hosts/incus-vm/` in this repo | |---|---|---| -| Purpose | Provision any NixOS VM as a Coder workspace | Turn a host into a box provisioner | -| Uses box flake? | No | Yes — `nixos-rebuild switch --flake /etc/nixos-repo#` | -| Sets up k3s / sysbox? | No | Optional — add to `hosts//default.nix` | +| What is it? | A Coder workspace template | A box host config | +| End result | A NixOS VM you work inside | A NixOS machine that *runs* Coder | +| Uses box flake? | No | Yes | +| Runs Coder server? | No — runs coder-agent | Yes — full Coder + PostgreSQL + k3s | | Who runs it? | Coder Terraform provisioner | You, manually, on the host | +You can use both together: run the `incus-nixos` template from a box host to spin +up NixOS workspaces, while the host itself is set up with this flake. + --- ## Manual setup: fresh NixOS host → box -These steps work for an Incus VM **and** for a bare-metal machine (ThinkStation, -etc.). The only difference is which extra modules you import in `default.nix`. +These steps work for an Incus VM provisioned by the `incus-nixos` template (or +any other NixOS VM) **and** for bare-metal machines. + +> **Note:** A stock NixOS image does not have `git` installed. Use +> `nix-shell -p git` to get it temporarily for the clone step, or add it to the +> system environment first. ### 1. Clone the repo ```sh -git clone https://github.com/coder/box /etc/nixos-repo +nix-shell -p git --run "git clone https://github.com/coder/box /etc/nixos-repo" ``` ### 2. Create the host directory -The flake auto-discovers hosts by folder name. The folder name must match the -machine's hostname (`hostname -s`): +The flake auto-discovers hosts by folder name — the folder name must match +`hostname -s`: ```sh HOSTNAME=$(hostname -s) mkdir -p /etc/nixos-repo/hosts/$HOSTNAME -# For an Incus VM — copy the incus-vm template: +# For an Incus VM: cp /etc/nixos-repo/hosts/incus-vm/default.nix \ /etc/nixos-repo/hosts/$HOSTNAME/default.nix cp /etc/nixos-repo/hosts/incus-vm/incus-vm.nix \ /etc/nixos-repo/hosts/$HOSTNAME/incus-vm.nix -# For a bare-metal host — start from a different base or write your own default.nix. -# See hosts/qemu-arm64/ for an example of the bare-metal layout. +# For bare-metal — write your own default.nix or copy from another host. +# See hosts/qemu-arm64/ for an example layout. -# Stage the new host dir so the flake can discover it. -# The flake uses builtins.readDir on the git tree, so untracked files are invisible. +# Stage the files — the flake's builtins.readDir only sees tracked files. git -C /etc/nixos-repo add hosts/$HOSTNAME/ ``` -### 3. Write the runtime config files +### 3. Enable k3s (required for workspace provisioning) -These files live outside the flake tree so they can carry secrets and -machine-specific values without being committed. +Edit `hosts/$HOSTNAME/default.nix` and add: -**`/etc/nixos/incus.nix`** — sets the hostname (Incus VMs get this written -automatically by `incus-virtual-machine.nix`; create it manually on bare metal): +```nix +# sysbox-runc — each workspace pod gets its own Docker daemon (no privileged mode) +services.coder-nixos.k3s-sysbox.enable = true; +``` + +Or for the lighter rootless-Podman variant: ```nix +services.coder-nixos.k3s.enable = true; +``` + +> Only enable one. `k3s-sysbox` is required for the `k3s-sysbox` workspace +> template; `k3s` works with `k3s-podman` and `k3s-dev`. + +### 4. Write the runtime hostname file + +This file lives outside the flake so it doesn't need to be committed. On an +Incus VM provisioned by the `incus-nixos` template, `/etc/nixos/incus.nix` is +already written by `incus-virtual-machine.nix`. For bare-metal or a fresh VM, +create it manually: + +```sh +cat > /etc/nixos/incus.nix << 'EOF' { lib, ... }: { networking.hostName = lib.mkForce "your-hostname"; } +EOF ``` -**`/etc/nixos/coder.nix`** — declares the workspace user and coder-agent service: +> `/etc/nixos/coder.nix` is **not** needed here. That file is for the +> `coder-agent` service on workspace VMs. The box host runs the Coder *server*, +> not an agent. -```nix -{ pkgs, ... }: -{ - users.users.coder = { - isNormalUser = true; - uid = 1000; - home = "/home/coder"; - shell = pkgs.bash; - extraGroups = [ "wheel" ]; - }; - security.sudo.wheelNeedsPassword = false; - - systemd.services.coder-agent = { - description = "Coder Agent"; - after = [ "network-online.target" ]; - wants = [ "network-online.target" ]; - wantedBy = [ "multi-user.target" ]; - serviceConfig = { - User = "coder"; - EnvironmentFile = "/opt/coder/init.env"; - ExecStart = "/opt/coder/init"; - Restart = "always"; - RestartSec = 10; - }; - }; -} +### 5. Apply + +```sh +nixos-rebuild switch --flake /etc/nixos-repo#$(hostname -s) --impure ``` -### 4. Enable k3s (optional) +`--impure` is required because `/etc/nixos/incus.nix` lives outside the flake +tree. This will build and activate: Coder server, PostgreSQL, k3s, sysbox, +template-sync, and all supporting services. -Edit `hosts//default.nix` and add one of: +### 6. Bootstrap the admin user -```nix -# sysbox-runc — required for the k3s-sysbox workspace template (full Docker per workspace) -services.coder-nixos.k3s-sysbox.enable = true; -``` +After the first `nixos-rebuild switch`, the Coder server is up but has no users. +Complete setup via the first-run wizard: -```nix -# rootless Podman — lighter option, works with k3s-podman and k3s-dev templates -services.coder-nixos.k3s.enable = true; +```sh +# The tunnel URL is printed in the Coder server logs: +journalctl -u coder --no-pager | grep "View the Web UI" ``` -> `k3s-sysbox` and `k3s` use different option names to avoid conflicts — only -> enable one. +Open that URL in a browser and create the admin user. Or use the CLI: + +```sh +CODER_URL=http://localhost:3000 coder login http://localhost:3000 +``` -> **Note:** `k3s-sysbox` requires `rsync` on the host. `nixos/k3s-sysbox.nix` -> includes it in `environment.systemPackages` automatically. If rsync is absent, -> `sysbox-mgr` exits at startup with -> `preflight check failed: rsync is not installed on host` and pods stay stuck in -> `ContainerCreating`. +Once logged in, `template-sync` will succeed on the next `nixos-rebuild switch` +and push the workspace templates (`k3s-sysbox`, `k3s-podman`, `k3s-dev`, +`coder-cli`) automatically. -### 5. Apply +To automate first-run on future machines, set these in the host's NixOS config +(e.g. via a secret manager or environment file): -```sh -nixos-rebuild switch --flake /etc/nixos-repo#$(hostname -s) --impure +``` +CODER_ADMIN_EMAIL=admin@example.com +CODER_ADMIN_USERNAME=admin +CODER_ADMIN_PASSWORD=... ``` -`--impure` is required because `/etc/nixos/incus.nix` and `/etc/nixos/coder.nix` -live outside the flake tree at absolute paths. +The `coder-init-admin` service reads these at boot and creates the user + mints +a long-lived session token for template-sync automatically. From 7944456aff57e9a2990f52696d8a636c6b07caf6 Mon Sep 17 00:00:00 2001 From: Ben Potter Date: Fri, 5 Jun 2026 11:46:28 -0500 Subject: [PATCH 09/10] fix: restore upstream section-header dash lengths, fix ~/$"$REPO_DIR" typo --- coderd/templates/k3s-sysbox/main.tf | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/coderd/templates/k3s-sysbox/main.tf b/coderd/templates/k3s-sysbox/main.tf index d0c48af..1e0ccf9 100644 --- a/coderd/templates/k3s-sysbox/main.tf +++ b/coderd/templates/k3s-sysbox/main.tf @@ -26,14 +26,14 @@ variable "coder_lan_ip" { description = "LAN IP of the Coder server host, injected into pod hostAliases so workspaces can resolve the hostname without mDNS. Set via services.coder-nixos.lanIp in the host's local.nix." } -# ── Provider config ─────────────────────────────────────────────────── +# ── Provider config ──────────────────────────────────────────────────────── provider "kubernetes" { config_path = "/etc/rancher/k3s/k3s.yaml" } provider "coder" {} -# ── Workspace data ─────────────────────────────────────────────────── +# ── Workspace data ───────────────────────────────────────────────────────── data "coder_provisioner" "me" {} data "coder_workspace" "me" {} data "coder_workspace_owner" "me" {} @@ -47,7 +47,7 @@ locals { kubectl = "/run/current-system/sw/bin/k3s kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml" } -# ── Parameters ──────────────────────────────────────────────────── +# ── Parameters ──────────────────────────────────────────────────────────── data "coder_parameter" "image" { name = "image" display_name = "Container image" @@ -164,7 +164,7 @@ data "coder_parameter" "home_disk_size" { } } -# ── Coder agent ───────────────────────────────────────────────────── +# ── Coder agent ──────────────────────────────────────────────────────────── resource "coder_agent" "main" { arch = data.coder_provisioner.me.arch os = "linux" @@ -204,7 +204,7 @@ resource "coder_agent" "main" { REPO_URL="${data.coder_parameter.repo_url.value}" if [ -n "$REPO_URL" ]; then REPO_DIR=$(basename "$REPO_URL" .git) - if [ ! -d ~/$"$REPO_DIR" ]; then + if [ ! -d ~/"$REPO_DIR" ]; then echo "Cloning $REPO_URL..." git clone "$REPO_URL" ~/"$REPO_DIR" fi @@ -262,7 +262,7 @@ resource "coder_agent" "main" { } } -# ── VS Code Web ──────────────────────────────────────────────────────── +# ── VS Code Web ──────────────────────────────────────────────────────────── module "vscode-web" { count = data.coder_workspace.me.start_count source = "registry.coder.com/coder/vscode-web/coder" @@ -276,7 +276,7 @@ module "vscode-web" { order = 1 } -# ── Cursor ─────────────────────────────────────────────────────────── +# ── Cursor ───────────────────────────────────────────────────────────────── module "cursor" { count = data.coder_workspace.me.start_count source = "registry.coder.com/coder/cursor/coder" @@ -286,7 +286,7 @@ module "cursor" { order = 2 } -# ── JetBrains (Toolbox) ────────────────────────────────────────────────────── +# ── JetBrains (Toolbox) ───────────────────────────────────────────────────── module "jetbrains" { count = data.coder_workspace.me.start_count source = "registry.coder.com/coder/jetbrains/coder" @@ -299,7 +299,7 @@ module "jetbrains" { coder_app_order = 3 } -# ── Persistent home PVC ─────────────────────────────────────────────── +# ── Persistent home PVC ─────────────────────────────────────────────────── resource "kubernetes_persistent_volume_claim_v1" "home" { metadata { name = "${local.prefix}-home" From 2de10ba95266f56e61c49f616858020d5e9bf747 Mon Sep 17 00:00:00 2001 From: Ben Potter Date: Fri, 5 Jun 2026 13:10:59 -0500 Subject: [PATCH 10/10] =?UTF-8?q?docs:=20fix=20incus-vm=20setup=20guide=20?= =?UTF-8?q?=E2=80=94=20aarch64,=20local.nix,=20coder.nix=20confusion?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - README: step 2 — warn that the copied default.nix imports coder.nix (only needed when the VM is also a coder-agent workspace); show how to remove it for a pure box host - README: step 3 — add aarch64 note: set nixpkgs.hostPlatform = "aarch64-linux" in default.nix if the VM is ARM; flake defaults to x86_64-linux - README: step 5 — add local.nix creation (copy from local.nix.example, set admin creds + LAN IP) before nixos-rebuild switch; this is what actually triggers coder-init-admin auto-bootstrap, not a browser wizard - README: step 6 — rewrite to reflect that coder-init-admin.service handles bootstrap automatically if local.nix was set up; browser/CLI flow is the fallback, not the primary path - hosts/incus-vm/default.nix — remove /etc/nixos/coder.nix import (that file is for coder-agent workspace VMs, not box hosts); add local.nix import with a comment; add nixpkgs.hostPlatform placeholder comment - hosts/incus-vm/incus-vm.nix — clarify enp5s0 applies to both x86_64 and aarch64 Incus VMs (confirmed on aarch64) --- hosts/incus-vm/README.md | 111 ++++++++++++++++++++++++------------ hosts/incus-vm/default.nix | 22 +++++-- hosts/incus-vm/incus-vm.nix | 6 +- 3 files changed, 95 insertions(+), 44 deletions(-) diff --git a/hosts/incus-vm/README.md b/hosts/incus-vm/README.md index e693eb0..f404d60 100644 --- a/hosts/incus-vm/README.md +++ b/hosts/incus-vm/README.md @@ -5,8 +5,8 @@ Box turns a NixOS machine into a **self-contained Coder deployment**. After `nixos-rebuild switch`, the machine runs: -- **Coder server** — full control plane, accessible over a Tailscale tunnel or - a configured external URL +- **Coder server** — full control plane, accessible over a `*.try.coder.app` + tunnel or a configured external URL - **PostgreSQL** — Coder's database, managed by the box flake - **k3s + sysbox** (optional) — single-node Kubernetes cluster where Coder provisions workspaces as pods; sysbox-runc gives each pod its own Docker daemon @@ -27,12 +27,13 @@ required. - Imports the upstream `incus-virtual-machine.nix` profile (QEMU guest agents, virtio drivers) — skip this for bare-metal -- `systemd-networkd` DHCP on `enp5s0` (the virtio NIC Incus assigns to x86_64 VMs) +- `systemd-networkd` DHCP on `enp5s0` (the virtio NIC Incus assigns to VMs on + both x86_64 and aarch64) - Disables the KDE / PipeWire / printing / Avahi stack — a headless host only needs Coder + PostgreSQL -`default.nix` is the per-host entrypoint that imports `incus-vm.nix` plus the -two runtime files from `/etc/nixos/` (see below). +`default.nix` is the per-host entrypoint that imports `incus-vm.nix`, your +`local.nix` secrets file, and the runtime files from `/etc/nixos/` (see below). The same pattern works for bare-metal machines (ThinkStation, etc.) — just skip `incus-vm.nix` or replace it with your own hardware module. @@ -97,9 +98,24 @@ cp /etc/nixos-repo/hosts/incus-vm/incus-vm.nix \ git -C /etc/nixos-repo add hosts/$HOSTNAME/ ``` -### 3. Enable k3s (required for workspace provisioning) +> **`/etc/nixos/coder.nix`:** The copied `default.nix` does **not** import this +> file. It only exists on VMs that are *also* running as a coder-agent workspace +> (i.e. the `incus-nixos` template writes it). On a pure box host it won't be +> present, and you don't need it. -Edit `hosts/$HOSTNAME/default.nix` and add: +### 3. Set architecture and enable k3s + +Edit `hosts/$HOSTNAME/default.nix`. + +**aarch64 VMs:** The flake defaults to `x86_64-linux`. If your VM is ARM +(e.g. running on Apple Silicon or an ARM server), add this or the build will +evaluate for the wrong architecture: + +```nix +nixpkgs.hostPlatform = "aarch64-linux"; +``` + +Then enable k3s (required for workspace provisioning): ```nix # sysbox-runc — each workspace pod gets its own Docker daemon (no privileged mode) @@ -115,12 +131,41 @@ services.coder-nixos.k3s.enable = true; > Only enable one. `k3s-sysbox` is required for the `k3s-sysbox` workspace > template; `k3s` works with `k3s-podman` and `k3s-dev`. -### 4. Write the runtime hostname file +### 4. Create local.nix + +`local.nix` holds per-host secrets (admin credentials, LAN IP, SSH keys). It is +gitignored and must be created manually: + +```sh +cp /etc/nixos-repo/local.nix.example \ + /etc/nixos-repo/hosts/$HOSTNAME/local.nix + +# Mark it so the flake's builtins.readDir can see it without committing it. +git -C /etc/nixos-repo add --intent-to-add -f hosts/$HOSTNAME/local.nix +``` + +Edit `hosts/$HOSTNAME/local.nix` and at minimum set: + +```nix +services.coder-nixos.lanIp = "192.168.x.x"; # VM's primary IP + +systemd.services.coder.environment = { + CODER_ADMIN_EMAIL = "you@example.com"; + CODER_ADMIN_USERNAME = "admin"; + CODER_ADMIN_PASSWORD = "changeme"; +}; +``` + +These credentials are read by `coder-init-admin.service` on first boot to +automatically create the admin user and mint a long-lived session token for +template-sync. **Without this, templates won't be pushed on the first +`nixos-rebuild switch`.** + +### 5. Write the runtime hostname file This file lives outside the flake so it doesn't need to be committed. On an Incus VM provisioned by the `incus-nixos` template, `/etc/nixos/incus.nix` is -already written by `incus-virtual-machine.nix`. For bare-metal or a fresh VM, -create it manually: +already written at first boot. For a fresh VM or bare-metal, create it manually: ```sh cat > /etc/nixos/incus.nix << 'EOF' @@ -131,11 +176,7 @@ cat > /etc/nixos/incus.nix << 'EOF' EOF ``` -> `/etc/nixos/coder.nix` is **not** needed here. That file is for the -> `coder-agent` service on workspace VMs. The box host runs the Coder *server*, -> not an agent. - -### 5. Apply +### 6. Apply ```sh nixos-rebuild switch --flake /etc/nixos-repo#$(hostname -s) --impure @@ -143,36 +184,36 @@ nixos-rebuild switch --flake /etc/nixos-repo#$(hostname -s) --impure `--impure` is required because `/etc/nixos/incus.nix` lives outside the flake tree. This will build and activate: Coder server, PostgreSQL, k3s, sysbox, -template-sync, and all supporting services. +and all supporting services. -### 6. Bootstrap the admin user - -After the first `nixos-rebuild switch`, the Coder server is up but has no users. -Complete setup via the first-run wizard: +On first boot, `coder-init-admin.service` runs automatically after Coder starts: +creates the admin user, mints a long-lived session token to +`/etc/coder/session-token`, and pushes all workspace templates (`k3s-sysbox`, +`k3s-podman`, `k3s-dev`, `coder-cli`) via Terraform. Check progress with: ```sh -# The tunnel URL is printed in the Coder server logs: -journalctl -u coder --no-pager | grep "View the Web UI" +journalctl -u coder-init-admin -f ``` -Open that URL in a browser and create the admin user. Or use the CLI: +Once complete, the tunnel URL is in `/etc/motd`: ```sh -CODER_URL=http://localhost:3000 coder login http://localhost:3000 +cat /etc/motd ``` -Once logged in, `template-sync` will succeed on the next `nixos-rebuild switch` -and push the workspace templates (`k3s-sysbox`, `k3s-podman`, `k3s-dev`, -`coder-cli`) automatically. - -To automate first-run on future machines, set these in the host's NixOS config -(e.g. via a secret manager or environment file): +**Fallback (no local.nix credentials):** If `CODER_ADMIN_EMAIL` was left empty, +`coder-init-admin` is skipped. Complete setup via the first-run wizard instead: +```sh +# Find the tunnel URL: +journalctl -u coder --no-pager | grep "View the Web UI" ``` -CODER_ADMIN_EMAIL=admin@example.com -CODER_ADMIN_USERNAME=admin -CODER_ADMIN_PASSWORD=... + +Open that URL in a browser, create the admin user, then log in with the CLI: + +```sh +CODER_URL=http://localhost:3000 coder login http://localhost:3000 ``` -The `coder-init-admin` service reads these at boot and creates the user + mints -a long-lived session token for template-sync automatically. +Once logged in, run `sudo nixos-rebuild switch` again to push templates via +`template-sync`. diff --git a/hosts/incus-vm/default.nix b/hosts/incus-vm/default.nix index ef1f687..28e57a6 100644 --- a/hosts/incus-vm/default.nix +++ b/hosts/incus-vm/default.nix @@ -1,10 +1,12 @@ -# Template host config for any Incus VM provisioned by the incus-vm Coder template. +# Template host config for a box host running inside an Incus VM. # -# The provisioner writes this file (or a copy) to hosts//default.nix -# at workspace start. --impure is required because /etc/nixos/incus.nix and -# /etc/nixos/coder.nix are runtime files outside the flake tree. +# Copy this to hosts//default.nix and incus-vm.nix to the same +# folder, then follow hosts/incus-vm/README.md. # -# To enable k3s add one of these in your host's default.nix: +# --impure is required because /etc/nixos/incus.nix is a runtime file +# outside the flake tree. +# +# To enable k3s add one of these below: # services.coder-nixos.k3s-sysbox.enable = true; # sysbox-runc (Docker per workspace) # services.coder-nixos.k3s.enable = true; # rootless Podman variant @@ -13,9 +15,17 @@ { imports = [ ./incus-vm.nix # QEMU guest agents, networkd DHCP, no desktop stack + ./local.nix # per-host secrets: admin creds, LAN IP, SSH keys /etc/nixos/incus.nix # hostname — written by incus-virtual-machine init - /etc/nixos/coder.nix # coder-agent service + workspace user (token, URL) + # /etc/nixos/coder.nix # only needed if this VM is also a coder-agent workspace ]; + # Uncomment for aarch64 VMs (Apple Silicon, ARM servers, etc.). + # The flake defaults to x86_64-linux; without this the build evaluates + # for the wrong architecture and will fail or produce a broken system. + # nixpkgs.hostPlatform = "aarch64-linux"; + + services.coder-nixos.k3s-sysbox.enable = true; + system.stateVersion = "25.11"; } diff --git a/hosts/incus-vm/incus-vm.nix b/hosts/incus-vm/incus-vm.nix index d4e69ca..36348d0 100644 --- a/hosts/incus-vm/incus-vm.nix +++ b/hosts/incus-vm/incus-vm.nix @@ -7,12 +7,12 @@ # - Imports the upstream incus-virtual-machine.nix profile (QEMU guest # agents, virtio drivers, auto-resize, systemd-boot). # - Switches networking to systemd-networkd + DHCP on enp5s0 (the -# default virtio NIC Incus assigns to x86_64 VMs). +# virtio NIC Incus assigns to VMs on both x86_64 and aarch64). # - Disables the full desktop stack (KDE, PipeWire, printing, Avahi) # that configuration.nix enables by default — a headless VM only needs # the Coder server + PostgreSQL. # -# k3s / sysbox are NOT disabled here. Set: +# k3s / sysbox are NOT enabled here. Set: # services.coder-nixos.k3s-sysbox.enable = true; # in hosts//default.nix to enable the full box stack (Coder + # PostgreSQL + k3s + sysbox) inside the VM. @@ -26,7 +26,7 @@ (modulesPath + "/virtualisation/incus-virtual-machine.nix") ]; - # Incus VMs get a virtio NIC named enp5s0 on x86_64. + # Incus VMs get a virtio NIC named enp5s0 on both x86_64 and aarch64. # Use systemd-networkd instead of dhcpcd (already disabled by # incus-virtual-machine.nix, but be explicit). networking = {