Skip to content

DESIGN.md

Latest commit

 

History

History
347 lines (268 loc) · 16 KB

DESIGN.md

File metadata and controls

347 lines (268 loc) · 16 KB

wisp design

This document captures the architecture, constraints, and technical decisions behind wisp. It serves as context for development.

Problem

Running a single network service on a single-board computer should not require a general-purpose operating system. Every package, daemon, shell, and login mechanism that ships with a traditional Linux distribution is attack surface you don't need and cognitive load you don't want.

Unikernels solve this for cloud/VM environments but require a hypervisor on SBCs, which defeats the purpose. Buildroot solves this but is a heavyweight build system designed for complex embedded products. gokrazy is close but Go-only and opinionated about updates and monitoring.

wisp is the minimal layer between hardware and a static binary. Nothing more.

Architecture

Boot sequence

Pi firmware (GPU, closed source)
  → reads config.txt from FAT32 boot partition
  → loads kernel Image and initrd into RAM
  → loads device tree blob (.dtb)
  → jumps to kernel

Kernel
  → unpacks initrd as rootfs (tmpfs, lives in RAM)
  → executes /init

/init (compiled Go binary)
  → mounts /dev (devtmpfs), /proc, /sys
  → loads kernel modules (if any)
  → configures network interface via netlink
  → drops privileges (setuid/setgid)
  → execs target binary as PID 1

After the final exec, the init binary is no longer in memory as a running process. The target binary IS the system.

Image layout

SD card:
└── boot (FAT32, ~60MB)
    ├── config.txt          # Pi firmware config
    ├── cmdline.txt         # Kernel command line
    ├── start4.elf          # Pi firmware (or start_cd.elf for cut-down)
    ├── fixup4.dat          # Pi firmware fixup
    ├── bcm2712-rpi-5-b.dtb # Device tree (board-specific)
    ├── overlays/           # Device tree overlays (if needed)
    ├── Image               # Linux kernel (arm64)
    └── initrd.img          # Everything else

There is no second partition. The entire userspace is the initrd, which lives in RAM after boot. The SD card can theoretically be removed after boot completes (though the firmware may need it present).

Initrd contents

initrd.img (gzipped cpio archive):
/
├── init                    # Compiled Go binary (the wisp init program)
├── etc/
│   ├── wisp.conf           # Network config, generated by wisp at build time
│   ├── resolv.conf         # DNS config, generated by wisp at build time
│   └── modules             # List of kernel modules to load (one per line)
├── lib/
│   └── modules/            # Kernel modules (.ko files)
├── dev/                    # Mount point for devtmpfs
├── proc/                   # Mount point for procfs
├── sys/                    # Mount point for sysfs
└── service/
    └── run                 # The target static binary (always named "run")

There is no shell in the initrd. /init is a small compiled Go program that is part of the wisp codebase. It is cross-compiled for the target architecture and embedded in the wisp binary via go:embed.

Go init binary

/init is a small Go program (cmd/init in the wisp repository) that runs as PID 1 until it execs the service binary. It is intentionally minimal — readable in under a minute. The init binary is architecture-portable: it is build-tagged linux (not linux && arm64) and uses golang.org/x/sys/unix for all syscalls, so it can be cross-compiled for any Linux architecture.

What it does, in order:

  1. Mount devtmpfs, proc, sysfs via unix.Mount
  2. Read /etc/wisp.conf for network configuration
  3. Load kernel modules listed in /etc/modules via unix.FinitModule
  4. Bring up the loopback interface
  5. Configure the primary interface (address, route) via netlink
  6. Drop privileges: unix.Setgid(1000), unix.Setuid(1000)
  7. unix.Exec("/service/run", ...) — from this point, the service binary IS the system

/etc/wisp.conf is a simple key=value file generated by wisp build at image build time:

IFACE=eth0
ADDR=192.168.1.100/24
GW=192.168.1.1

The init binary itself is identical across image builds. Only the config file changes. This allows the init binary to be pre-compiled for each target architecture and embedded in the wisp host binary via go:embed, avoiding the need to cross-compile during wisp build.

DHCP is not supported in v1. Static IP is required. DHCP support is a future consideration (see below).

Privilege model

The target binary runs as a non-root user (service, uid 1000). The init binary runs as root only long enough to mount filesystems and configure networking, then permanently drops privileges via unix.Setgid + unix.Setuid before calling unix.Exec.

Since the binary is PID 1, if it exits, the kernel panics. This is intentional. A crashed service should not silently restart — it should be observable as a hardware-level failure (no network response, no heartbeat). If restart behavior is desired, the binary should implement it internally.

PID 1 signal semantics: Linux does not deliver SIGTERM or SIGINT to PID 1 unless the process explicitly handles them. If your binary needs to respond to shutdown signals (e.g., for graceful drain on reboot), it must install signal handlers. Binaries that ignore signals will not respond to SIGTERM. This is a Linux kernel behavior, not a wisp constraint.

Design decisions

Pure Go implementation

wisp has no external tool dependencies beyond the Go toolchain. FAT32 image creation and cpio archive assembly are implemented in pure Go. The init binary uses golang.org/x/sys/unix for portable Linux syscalls (mount, netlink, module loading, exec). wisp runs natively on Linux and macOS without Docker or system tools (mkdosfs, cpio, etc.).

Off-the-shelf kernels

wisp does not compile kernels. Rationale:

  • Kernel compilation is slow, fragile, and configuration-heavy.
  • Pi hardware support requires specific patches that the Raspberry Pi kernel team maintains.
  • The kernel is not where attack surface reduction matters — the userspace is.
  • Pre-built kernels from Alpine or Raspberry Pi OS are tested, signed, and maintained by people whose job it is to do so.

wisp extracts kernel images and modules from distribution packages. The kernel source is tracked (distro, version, sha256) for reproducibility.

No kernel modules on disk

Kernel modules needed at boot are bundled into the initrd. This is standard practice for diskless/netboot systems. Modules not needed are excluded. The module set is determined by the target board profile.

Static binaries only

wisp does not support dynamically linked binaries. Rationale:

  • A static binary has zero runtime dependencies on the host system.
  • No libc version compatibility concerns.
  • No need to include shared libraries in the initrd.
  • Go, Rust, and Zig all produce static binaries easily.
  • If a binary requires dynamic linking, it is not a good fit for wisp.

No shell after boot

There is no shell anywhere in the wisp initrd — not during boot, not after. /init is a compiled binary, not a script. There is no way to get a shell on a running wisp system.

This means:

  • No SSH, no serial console login, no debug shell.
  • Debugging requires either: (a) adding logging/metrics to the binary, (b) serial console output from the kernel and init binary during boot, or (c) reflashing with a debug image that includes a shell.
  • This is the correct tradeoff for production. Debug images can be built separately.

No persistent storage

The rootfs is tmpfs (RAM-backed). Nothing is writable on the SD card after boot. The binary has no filesystem to write to unless it creates tmpfs directories itself.

Persistent state should be managed externally — over the network, in a database, in object storage. The wisp image is stateless and disposable.

Static IP only (v1)

v1 requires a static IP. The address, gateway, and DNS server are specified at build time and baked into /etc/wisp.conf and /etc/resolv.conf in the initrd.

Static IP is the right default for production: deterministic, no dependency on a DHCP server, no race condition between network acquisition and service startup.

DHCP support is a future consideration. Implementing it means adding a DHCP exchange to the init binary — either via a library or directly with UDP syscalls. Straightforward but adds scope to v1 unnecessarily.

Target board profiles

Each supported board has a JSON profile in internal/board/boards/ that defines everything board-specific. Profiles are embedded into the wisp binary via go:embed and parsed with board.Parse(). Adding a new board means adding a new JSON file — no code changes required (ideally).

Fields in a board profile:

  • name — Board identifier, used as the --target value.
  • arch — Target architecture (aarch64).
  • page_size — Required binary alignment in bytes (4096 or 16384).
  • network_interface — Primary network interface name (e.g., eth0).
  • kernel — Object with source, package, version, and sha256.
  • firmware — List of firmware files: url, dest filename, sha256.
  • dtb — Device tree blob filename.
  • boot_config — Content of config.txt written to the boot partition.
  • cmdline — Kernel command line written to cmdline.txt.
  • modules — List of kernel modules with path (in APK) and name (output filename).
  • qemu — QEMU emulation config: binary, machine, cpu, memory, accel, net_dev, extra. Present only for QEMU-bootable targets.

Example (internal/board/boards/qemu.json):

{
  "name": "qemu",
  "arch": "aarch64",
  "page_size": 4096,
  "network_interface": "eth0",
  "kernel": {
    "source": "alpine",
    "package": "linux-virt",
    "version": "6.18.13-r0",
    "sha256": ""
  },
  "cmdline": "rdinit=/init console=ttyAMA0 net.ifnames=0 quiet",
  "modules": [
    { "path": "kernel/net/core/failover.ko.gz", "name": "failover.ko" },
    { "path": "kernel/drivers/net/net_failover.ko.gz", "name": "net_failover.ko" },
    { "path": "kernel/drivers/net/virtio_net.ko.gz", "name": "virtio_net.ko" }
  ],
  "qemu": {
    "binary": "qemu-system-aarch64",
    "machine": "virt",
    "cpu": "host",
    "memory": "512M",
    "accel": "hvf",
    "net_dev": "virtio-net-pci"
  }
}

Raspberry Pi 5

  • SoC: BCM2712 (Cortex-A76, ARMv8.2)
  • Firmware: start4.elf, fixup4.dat from raspberrypi/firmware repo
  • DTB: bcm2712-rpi-5-b.dtb
  • Kernel: linux-rpi package from Alpine aarch64
  • Network: eth0 (Gigabit Ethernet via RP1 southbridge)
  • Architecture: aarch64
  • Page size: 16KB (kernel and all binaries must be 16KB-aligned)

QEMU (aarch64/virt)

  • Machine: virt (-machine virt -cpu host)
  • Firmware: none — QEMU loads kernel and initrd directly via -kernel/-initrd flags
  • DTB: generated by QEMU at runtime
  • Kernel: Alpine linux-virt package (lighter than linux-rpi, designed for VMs)
  • Network: virtio-net, interface eth0 (via net.ifnames=0 on kernel command line)
  • Modules: failover.ko, net_failover.ko, virtio_net.ko (for virtio networking)
  • Architecture: aarch64

wisp run --target qemu builds the kernel and initrd then launches QEMU with port forwarding configured. No SD card image is produced. Useful for development, integration testing, and verifying behavior before flashing hardware.

The QEMU board profile omits firmware, dtb, and boot_config fields. The cmdline uses console=ttyAMA0 instead of the serial console entries used on hardware.

Build pipeline

wisp is a CLI tool. The build process:

  1. Validate binary — Confirm it is a static ELF binary for the correct architecture.
  2. Fetch board assets — Download or cache kernel, firmware, DTBs for the target board. QEMU target: kernel only, no firmware or DTB.
  3. Build initrd — Assemble the cpio archive: the pre-compiled init binary (embedded in wisp), the generated wisp.conf, resolv.conf, target binary, kernel modules.
  4. Package for target:
    • Hardware targets (pi5): assemble a FAT32 disk image containing all boot files.
    • QEMU target: emit kernel + initrd.img and construct the QEMU invocation.
  5. Output — Write artifacts to the output directory (default: wisp-output/).

The core outputs are kernel + initrd.img. The SD card image is a packaging step that wraps these for hardware boot. QEMU uses the artifacts directly without packaging.

Caching

Board assets (kernels, firmware) are downloaded once and cached in ~/.cache/wisp/<board>/<version>/. Kernel packages are downloaded as Alpine APKs, and the kernel image and modules are extracted and decompressed into the cache directory. Firmware files are cached under ~/.cache/wisp/<board>/firmware/. A cached entry is reused as long as the version in the board profile matches. If a SHA256 checksum is specified, the download is verified against it.

Split initrd (future)

The kernel initrd spec allows pointing at multiple cpio archives that get merged. This enables:

base.cpio.gz      — init binary, modules (stable, rarely changes)
service.cpio.gz   — just the target binary (changes every deploy)

This optimization is not required for v1 but should be kept in mind during design. It enables faster iteration because you only rebuild the service archive when deploying a new binary.

Non-goals

  • Multiple binaries per image. wisp runs one binary. Use separate boards or a different tool.
  • Runtime package management. There is no package manager. The image is the deployment artifact.
  • Kernel compilation. wisp uses pre-built kernels.
  • GUI or display output. wisp is for headless network services.
  • Container support. No Docker, no containerd, no OCI. The binary runs directly.
  • Orchestration. wisp builds images. Fleet management is a separate concern.
  • Unified Kernel Images (UKI). UKIs are optimized for EFI boot, which Pi boards don't use natively. Avoid this path.

Future considerations

  • Raspberry Pi Zero 2 W: WiFi-only support requires firmware blobs for the wireless chipset, a wpa_supplicant binary in the initrd, WiFi credentials baked in, and a different network bring-up sequence in the init binary (associate before exec). This is a meaningful increase in complexity and a second code path through network configuration. Deferred to v2.
  • DHCP: Implement a minimal DHCP client in the init binary. The init binary already owns the network bring-up sequence, so DHCP fits naturally there. Useful for home lab and development where a static IP is inconvenient.
  • Network boot (PXE/TFTP): Pi 5 supports network boot. wisp could produce TFTP-servable artifacts instead of SD card images. The initrd-based architecture makes this natural — the QEMU target already separates kernel + initrd from packaging.
  • A/B partition updates: An alternative to reflashing where two boot partitions exist and a flag selects which to boot. Adds complexity but enables remote updates.
  • Watchdog: The Pi has a hardware watchdog. The init binary could enable it so the board reboots if the service binary hangs. Simple, useful, low complexity.
  • Metrics endpoint: A tiny sidecar that exposes basic health metrics (uptime, memory, temperature) without requiring changes to the service binary. Must be careful not to violate the "single binary" principle.

References