Local macOS VM E2E testing environment with Tart

[kryota-dev]

Summary

Set up a local E2E testing environment using macOS VMs to validate dotfiles configurations that cannot be tested in the current GitHub Actions CI (GUI apps, macOS defaults, fonts, etc.).

Background

The current CI (setup-validation.yml) explicitly excludes 7 scripts/features due to CI environment limitations:

• run_once_before_00 — Prerequisites (Xcode CLI, Homebrew)
• run_onchange_before_10 — brew bundle (cask/mas excluded)
• run_once_after_11 — 1Password validation
• run_onchange_after_20 — macOS defaults
• run_once_after_30 — Font installation
• run_once_after_90 — Interactive app downloads
• 46 cask apps in Brewfile

A local macOS VM environment enables testing these in a clean, reproducible macOS instance.

Solution

Use Tart (cirruslabs/tart) — a CLI-first macOS VM tool built on Apple's Virtualization.framework — with Packer for IaC-managed VM images.

Key capabilities

• Free for personal use (Fair Source License)
• CLI-driven lifecycle: tart clone → run → ip → stop → delete
• Packer plugin for reproducible base image builds
• Pre-built vanilla macOS images on ghcr.io (SSH + admin user ready)
• Host directory sharing via --dir flag or rsync over SSH
• GUI mode for visual verification (tart run shows macOS desktop)

Acceptance Criteria

• make e2e runs a full dotfiles E2E test in a clean macOS VM
• macOS defaults, cask apps, fonts, and file deployment are verified automatically

[kryota-dev]

📋 Research Report: Solution Comparison

VM Solutions Evaluated

Tool	ARM macOS	CLI/Automation	IaC	Shared Folders	GUI	Cost	Verdict
Tart	✅	✅ Full CLI	✅ Packer + OCI	✅ VirtioFS	✅ VNC	Free (personal)	★★★★★
Anka	✅	✅ Docker-like	✅ Packer + REST	✅ anka cp	✅	Paid (contact sales)	★★★★☆
Parallels	✅	✅ prlctl	✅ Packer/Vagrant	✅	✅ GPU	$120+/yr	★★★☆☆
UTM	✅	△ utmctl	✗	△ GUI-only config	✅	Free	★★☆☆☆
Lume	✅	✅ CLI + HTTP API	✅ Packer	✅	△	Free (MIT)	★★★☆☆
VirtualBuddy	✅	✗ GUI-only	✗	✅	✅	Free	★☆☆☆☆
VMware Fusion	✗ No macOS guest	—	—	—	—	—	Eliminated

Why Tart

1. Free — Fair Source License, unlimited personal use
2. CLI-first — Scriptable with tart clone/run/ip/exec/stop/delete
3. Packer plugin — VM image config as code (packer-plugin-tart v1.20.0)
4. OCI registry — Push/pull VM images like containers (ghcr.io)
5. Pre-built images — ghcr.io/cirruslabs/macos-sequoia-vanilla:latest (SSH + admin/admin ready)
6. Active maintenance — 5,000+ GitHub stars, 175 releases, v2.31.0
7. Production proven — Used by Atlassian, Figma, Snowflake, Expo

References

• Tart official site
• Tart GitHub
• Packer Plugin Tart
• cirruslabs/macos-image-templates
• The Sad State of macOS Virtualization in 2025

[kryota-dev]

📋 Research Report: Repository Analysis — What E2E Should Test

Current CI Coverage Gap

The setup-validation.yml workflow explicitly excludes these files:

home/private_dot_aws/config.tmpl          → 1Password required
home/dot_agents/skills/daily-planning/    → 1Password required
home/run_once_before_00-*                 → Prerequisites already in CI
home/run_onchange_before_10-*             → cask/mas filtered out
home/run_once_after_11-*                  → 1Password CLI required
home/run_once_after_30-*                  → Font download + GUI
home/run_once_after_90-*                  → Interactive (tty required)

Additionally, the CI filters dot_Brewfile to formula-only: grep -E '^(tap |brew )'

New E2E Test Coverage Matrix

Category	What to verify	CI	VM E2E
macOS defaults (16 settings)	defaults read for each domain/key	✗	✅
Cask apps (46 apps)	/Applications/*.app existence	✗	✅
Fonts (Moralerspace v2.0.0)	~/Library/Fonts/Moralerspace*	✗	✅
Homebrew from scratch	Full run_once_before_00 flow	✗	✅
brew bundle (full)	All formula + cask entries	✗	✅
VS Code settings	~/Library/Application Support/Code/User/settings.json	✗	✅
Ghostty config	~/.config/ghostty/config	✅	✅
Shell startup	zsh -i -c exit	✅	✅
mise tools (36 tools)	node --version, python3 --version, etc.	✅	✅
mas apps (12 apps)	App Store sign-in required	✗	✗ (VM limitation)
1Password integration	Requires manual desktop app setup	✗	△

macOS Defaults to Verify (all 16)

com.apple.HIToolbox      AppleFnUsageType              = 2
NSGlobalDomain           AppleKeyboardUIMode           = 3
NSGlobalDomain           AppleShowAllExtensions        = 1 (true)
NSGlobalDomain           AppleShowScrollBars           = "Always"
NSGlobalDomain           com.apple.keyboard.fnState    = 1 (true)
NSGlobalDomain           KeyRepeat                     = 2
NSGlobalDomain           NSAutomaticCapitalizationEnabled = 0 (false)
NSGlobalDomain           InitialKeyRepeat              = 25
com.apple.desktopservices DSDontWriteNetworkStores     = 1 (true)
com.apple.desktopservices DSDontWriteUSBStores         = 1 (true)
com.apple.dock           autohide-delay                = 0
com.apple.finder         AppleShowAllFiles             = 1 (true)
com.apple.finder         ShowStatusBar                 = 1 (true)
com.apple.finder         ShowPathbar                   = 1 (true)
com.apple.menuextra.clock DateFormat                   = "EEE d MMM HH:mm"
com.apple.terminal       StringEncodings               = (4)

[kryota-dev]

📋 Research Report: Tart Technical Deep Dive

Pre-built Image Tiers

Layer	Image	Contents
Vanilla	ghcr.io/cirruslabs/macos-sequoia-vanilla:latest	Clean macOS + auto-login + SSH + passwordless sudo + Gatekeeper disabled + Xcode CLI tools
Base	ghcr.io/cirruslabs/macos-sequoia-base:latest	Vanilla + Homebrew + CLI tools (gh, jq, mise, etc.) + Guest Agent
Xcode	ghcr.io/cirruslabs/macos-sequoia-xcode:N	Base + Xcode N + Flutter + Android SDK

Credentials: admin / admin for all images.

Base Image Choice for This Project

Scenario	Image	Pros	Cons
Full regression	vanilla	Tests Homebrew install from scratch, closest to "fresh Mac"	~38 min total
Quick validation	base	Skips Homebrew install, faster iteration	~24 min, doesn't test run_once_before_00

Recommendation: Use vanilla for scheduled/full tests, base for quick iteration during development.

Directory Sharing (--dir)

# Mount dotfiles repo into VM
tart run --dir=dotfiles:~/dotfiles vm-name
# Guest sees: /Volumes/My Shared Files/dotfiles

# Read-only mount
tart run --dir=dotfiles:~/dotfiles:ro vm-name

Known issues:

• Path contains spaces: /Volumes/My Shared Files/ (#517)
  • Workaround: symlink in VM — ln -s "/Volumes/My Shared Files/dotfiles" ~/dotfiles
• macOS Sonoma 14.2 broke auto-mount (#687) — fixed in 14.3
• Guest Agent triggers "network volume access" popup in CI (Discussion #1144)
  • Workaround: use base image (SIP disabled) or apply disable-sip.pkr.hcl

Recommendation: Use rsync over SSH instead of --dir for stability.

Networking

• Default (NAT): 192.168.64.0/24, tart ip to get VM address
• Softnet (--net-softnet): Better isolation, requires brew install cirruslabs/cli/softnet
• IP resolution: tart ip <vm> (ARP-based) or tart ip --resolver=agent <vm> (Guest Agent)
• Static IPs not natively supported (Discussion #468)

Guest Agent (tart-guest-agent)

Provides tart exec (run commands without SSH) and clipboard sharing.

• Pre-installed in base images, not in vanilla
• Install: brew install cirruslabs/cli/tart-guest-agent
• For this project: SSH is sufficient, guest agent is optional

Packer Integration

packer {
  required_plugins {
    tart = {
      version = ">= 1.14.0"
      source  = "github.com/cirruslabs/tart"
    }
  }
}

source "tart-cli" "dotfiles-base" {
  vm_base_name     = "ghcr.io/cirruslabs/macos-sequoia-vanilla:latest"
  vm_name          = "dotfiles-base"
  cpu_count        = 4
  memory_gb        = 8
  disk_size_gb     = 60
  ssh_username     = "admin"
  ssh_password     = "admin"
  ssh_timeout      = "120s"
  recovery_partition = "keep"  # Important: don't break softwareupdate
}

Key learnings from official templates:

• Each provisioner "shell" block runs in a fresh shell — must source ~/.zprofile every time
• Default shebang is /bin/sh -e (not bash)
• recovery_partition = "keep" is recommended (deleting breaks softwareupdate)
• run_extra_args = ["--dir=dotfiles:/path"] mounts host dirs during Packer build

[kryota-dev]

📋 Research Report: Known Issues & Constraints

Apple Platform Constraints (Cannot Be Worked Around)

Constraint	Detail	Impact
Max 2 concurrent macOS VMs	Kernel-level enforcement + EULA	No parallel testing
Apple Silicon only	Virtualization.framework limitation	Intel Macs not supported
No App Store in VM	Apple ID sign-in restrictions	mas entries (12 apps) cannot be tested
No disk resize after creation	Recovery partition blocks resize	Must set disk_size_gb correctly at creation
No snapshots	Virtualization.framework limitation	Must clone → test → delete each time
Guest OS ≤ Host OS	Cannot run newer macOS than host	Keep host OS updated

Tart-Specific Issues on macOS Sequoia (15)

1. Keychain unlock required (Sequoia FAQ)

# Must run before starting VMs on Sequoia
security create-keychain -p '' login.keychain
security unlock-keychain -p '' login.keychain
security login-keychain -s login.keychain

2. "Local Network" permission popup

# One-time fix on host
sudo defaults write com.apple.network.local-network \
  AllowedEthernetLocalNetworkAddresses -array "10.0.0.0/8" "172.16.0.0/12" "192.168.0.0/16"

3. Packet filter interference (#1015)

• Tart activates macOS PF, can conflict with third-party firewalls
• Workaround: disable conflicting firewall software during VM use

4. UI window may not appear (#928)

• Fixed in Tart v2.30.3

5. VMs not actually stopping (#967)

• tart list may show VM as stopped while it's still running
• Workaround: verify with tart list before creating new VMs

Auto-Pruning Behavior

tart pull/tart clone automatically delete least-recently-accessed VMs when disk space is low (default: 100 GB threshold).

• Disable with: TART_NO_AUTO_PRUNE=
• Consider setting this in Makefile to prevent unexpected deletions

Licensing

• Personal use: Free, unlimited
• Organization: Free up to 100 CPU cores or 4 hosts
• Beyond that: Gold ($12,000/yr, 500 cores), Platinum ($36,000/yr, 2000 cores)
• For this project (personal dotfiles): no cost

[kryota-dev]

📋 Research Report: Proposed Architecture & Implementation Plan

Directory Structure

dotfiles/
├── e2e/
│   ├── Makefile                         # VM lifecycle & test execution
│   ├── packer/
│   │   └── dotfiles-base.pkr.hcl       # (Optional) Custom base image
│   ├── scripts/
│   │   ├── provision.sh                 # chezmoi apply inside VM
│   │   ├── verify-defaults.sh           # macOS defaults verification
│   │   ├── verify-apps.sh              # Cask app verification
│   │   ├── verify-files.sh             # File deployment verification
│   │   ├── verify-fonts.sh             # Font installation verification
│   │   └── verify-shell.sh             # zsh startup + benchmark
│   └── README.md
├── home/                                # (Existing) chezmoi source
├── tests/                               # (Existing) CI bats tests
└── Makefile                             # (Existing) + e2e target

Test Flow

┌─────────────┐    ┌──────────────────┐    ┌─────────────────┐
│  1. Clone   │───▶│  2. Run + Wait   │───▶│  3. Provision   │
│  Base Image │    │  for SSH ready   │    │  (chezmoi apply)│
└─────────────┘    └──────────────────┘    └─────────────────┘
                                                    │
┌─────────────┐    ┌──────────────────┐             ▼
│  5. Cleanup │◀───│  4. Verify       │◀────────────┘
│  (destroy)  │    │  defaults, apps  │
└─────────────┘    └──────────────────┘

Test Levels

Level 1: CI (GitHub Actions)        ← Existing, no changes
  └── lint + bats + chezmoi apply (basic) + zsh startup

Level 2: VM E2E — "Clean Install"   ← New (automated)
  └── vanilla image → full setup → verify all
      ├── Homebrew install → brew bundle (formula + cask)
      ├── chezmoi apply → file deployment
      ├── mise tools → version checks
      ├── macOS defaults → defaults read verification
      ├── Fonts → ~/Library/Fonts/ check
      └── zsh startup + benchmark

Level 3: VM GUI — "Visual Check"    ← New (manual/semi-auto)
  └── tart run (GUI mode) for visual inspection
      ├── Dock / Finder appearance
      ├── VS Code launch + settings
      ├── Ghostty terminal + font rendering
      └── GUI app launch verification

File Transfer Strategy

Recommended: rsync over SSH (more stable than --dir shared folders)

# Copy dotfiles to VM
rsync -avz --exclude='.git' -e "sshpass -p admin ssh -o StrictHostKeyChecking=no" \
  ./ admin@$(tart ip dotfiles-e2e):~/dotfiles/

Rationale:

• Avoids shared folder space-in-path issue (#517)
• Avoids Sonoma 14.2 auto-mount regression (#687)
• Avoids Guest Agent security prompt (#1144)
• Works consistently across macOS versions

Implementation Phases

Phase 1: Foundation (MVP)

• Install Tart + sshpass
• Create e2e/Makefile with basic lifecycle targets
• Implement e2e-create, e2e-start, e2e-provision, e2e-cleanup
• Single verify script checking file deployment

Phase 2: Verification Scripts

• verify-defaults.sh — All 16 macOS defaults
• verify-files.sh — Extended from CI (add VS Code, Ghostty configs)
• verify-apps.sh — Key cask apps in /Applications/
• verify-fonts.sh — Moralerspace font files
• verify-shell.sh — zsh startup + benchmark

Phase 3: Optimization (Optional)

• Packer custom base image (Homebrew + formula pre-installed)
• Root Makefile integration (make e2e)
• README documentation for local E2E testing

[kryota-dev]

📋 Research Report: Prerequisites & Setup Guide

Host Requirements

• macOS 13.0 (Ventura) or later on Apple Silicon
• Sufficient disk space (~30 GB per VM image)
• No conflicting firewall software (or temporarily disabled)

Installation

# Tart (VM management)
brew install cirruslabs/cli/tart

# sshpass (non-interactive SSH password auth for automation)
brew install hudochenkov/sshpass/sshpass

# Packer (optional, for custom base image builds)
brew install packer

Sequoia (macOS 15) One-Time Setup

# Keychain setup (required for Virtualization.framework)
security create-keychain -p '' login.keychain
security unlock-keychain -p '' login.keychain
security login-keychain -s login.keychain

# Local network permission (prevents popup on each VM start)
sudo defaults write com.apple.network.local-network \
  AllowedEthernetLocalNetworkAddresses -array "10.0.0.0/8" "172.16.0.0/12" "192.168.0.0/16"
sudo defaults write com.apple.network.local-network \
  AllowedWiFiLocalNetworkAddresses -array "10.0.0.0/8" "172.16.0.0/12" "192.168.0.0/16"

Quick Validation

# Pull a vanilla macOS image (~20-30 GB, first time only)
tart clone ghcr.io/cirruslabs/macos-sequoia-vanilla:latest test-vm

# Start with GUI to verify it works
tart run test-vm

# In another terminal: verify SSH access
ssh admin@$(tart ip test-vm)  # password: admin

# Cleanup
tart stop test-vm
tart delete test-vm

Key Environment Variables

Variable	Purpose	Default
TART_NO_AUTO_PRUNE	Disable auto-pruning of old VMs	(unset = enabled)
GITHUB_TOKEN	Avoid mise GitHub API rate limits	(from gh auth token)

References

• Tart Quick Start
• Tart FAQ — Sequoia issues
• Available macOS images