docs(cpu-software): add Flipper OS storage architecture RFC

[xbizzybone]

What

Adds a design RFC for Flipper OS storage as a new page under
🐧 Linux (CPU Software).

Why

The Flipper OS notes state the architecture isn't pinned down yet.
This RFC proposes a concrete answer to the core open question: how do
profile overlays stay valid when the A/B base is atomically replaced
underneath them?

Summary of the proposal

• Three state classes: immutable A/B base (squashfs + dm-verity),
  per-profile Btrfs overlay (reflink clone/reset), and a separate
  /data partition that survives both profile resets and base updates.
• RAUC for atomic updates, with the MCU watchdog as a hardware
  rollback backstop.
• Stateless base + drop-in-only config + a build-time lint that
  forbids profiles from shadowing base files (eliminates /etc drift).
• MCU-owned boot-time profile selection handed to U-Boot over the
  Interconnect via a small CRC-checked message.

Includes RFC-2119 wording, prior art (OpenWrt, Android A/B, RAUC,
ostree/Silverblue), drawbacks, unresolved questions, and a phased
implementation plan.

This is a proposal for discussion — feedback and pushback welcome.
The nav entry is registered in archbee.json.

[xbizzybone]

I built a working Phase-0 reference implementation of this RFC and put it up as a public lab, so the storage model isn't just on paper:

→ https://github.com/xbizzybone/flipper-os-lab

It reproduces the whole Phase-0 storage stack on loop-backed disk images on any Linux host (no RK3576 board, no QEMU) — every mechanism is the genuine kernel feature, just on files instead of eMMC:

• Immutable base = squashfs + dm-verity (read-only lower); a tamper test flips a byte and watches verity reject it.
• Profiles = Btrfs subvolumes used as the OverlayFS upperdir.
• clone = btrfs subvolume snapshot (copy-on-write, O(metadata)); reset = restore from a read-only pristine snapshot — no re-flash.
• /data = separate partition, bind-mounted, survives both profile reset and base swap.
• base_min_version guard refuses to mount a profile against an incompatible base.
• Anti-drift lint implements the build-time MUST: it fails the profile if its upper shadows any base file outside a whitelisted drop-in dir. The demo plants an illegal shadow of etc/flipper/base.conf and the lint rejects it.

initramfs/flipper-overlay.sh is the reference local-bottom hook showing how the lab's boot <id> maps onto a real device — the only thing that changes is where the profile id comes from (lab CLI → kernel cmdline flipper.profile= → MCU BOOT_SELECTION over Interconnect, exactly as the RFC describes).

CI proof: GitHub Actions runs the full 8-step demo on ubuntu-22.04 with real dm-verity on every push — root hash, tamper rejection, lint pass+reject, and /data survival, exit 0: https://github.com/xbizzybone/flipper-os-lab/actions/workflows/demo.yml

Scope is honest about phase boundaries: A/B + RAUC + MCU watchdog (Phases 1–2) are referenced, not implemented yet — that's the roadmap. Two minor deltas vs. the RFC: the lab's /data is ext4 (RFC says Btrfs±LUKS — LUKS is Phase 3) and verity hashes live in a sidecar file rather than appended (cosmetic for the lab).

Happy to evolve this toward Phase 1 (QEMU aarch64 + U-Boot A/B slot selection + RAUC bootcount rollback) if useful for the discussion.

[alchark]

We’ve been thinking along similar lines, but without squashfs, which is rather limiting in our context, as it forces a single base image and precludes storage-level data deduplication if more than one are used.

Right now we are experimenting with a stack fully based around Btrfs snapshots, which allows multiple base profiles to coexist space-efficiently, with instantaneous cloning/branching via native CoW functionality. We haven’t yet considered the introduction of verity into the chain though (or discussed if it’s vital for our envisaged usage scenarios and target audience)

[Yury-MonZon]

Really nice work!

A couple of things I keep wondering about:

• Base swap vs profile: when A/B replaces the base and base_min_version no longer matches, does the profile get migrated, refused, or reset?
• Backup: any restore/export path for /data or for a profile itself (send/receive), or is that for later?
• Packages per profile: full userland in each overlay, or shared base + deltas? And does anything dedup across them?

Mainly trying to get my head around the day-2 side: updates, rollback, recovery, not just the layout.

[xbizzybone]

@alchark you're right on the substance, and I want to concede it cleanly: a squashfs base is an opaque compressed blob, so it forces a single active base per slot and can't extent-share with a second base — there's no storage-level dedup across bases. The PoC shows exactly that shape: one base.squashfs, with CoW sharing happening only among profiles on the Btrfs pool, never touching the base. So this isn't hypothetical.

I'll also own a gap in the RFC's framing. We listed "three options" and rejected full-image-per-profile on storage-scaling grounds ("N × image × 2"), but we never evaluated your design — multiple coexisting Btrfs base subvolumes that reflink-share. That's a distinct fourth point in the space, it answers the same scaling objection without giving up apt or waiting on deb-ostree, and it deserved an explicit weighing rather than silence. So I've gone ahead and added it: the latest revision on this PR now has Alternative D — All-Btrfs base (multiple reflink-sharing base subvolumes), credited to this review.

Worth being clear about what we already share: profiles are Btrfs subvolumes today, and the PoC demonstrates the CoW snapshot clone/branch/reset you describe (clone = btrfs subvolume snapshot, reset = restore from a read-only pristine snapshot — O(metadata) by construction, though I haven't benchmarked wall-clock time). So the profile layer is already the Btrfs-CoW half of what you're describing; where we diverge is putting the base itself on Btrfs.

The crux is verity, and it's genuinely a fork rather than a disagreement. Classic whole-device dm-verity needs a static read-only hash-tree device and is mutually exclusive with a writable Btrfs-snapshot base — that's exactly why the RFC pushed the base onto squashfs (it also keeps RAUC's A/B slots and the MCU/U-Boot early-boot path simple). What the PoC actually proves here is narrow but concrete: a byte-flipped base image is rejected against its sealed dm-verity root hash in CI. I don't want to overstate it — that's offline image verification, and RAUC signature verification / fs-verity / Btrfs-checksum integrity are not built yet.

You're asking the right question — is verity vital for your audience? For a tool whose owner deliberately runs untrusted code with physical access, base verity buys a narrow thing: continuous at-rest tamper-evidence (rootkit persistence, bad flash, bit-rot) plus a hash-pinned rollback target. It's an anti-persistence story, not load-bearing for the profile/clone/reset UX. So in the same revision I also reframed base integrity as an explicit pluggable axis — dm-verity / fs-verity / Btrfs-checksums / none-plus-signed-update — selected by threat model, rather than baking squashfs+dm-verity into the partition layout. dm-verity stays the default; choosing fs-verity or signed-update-only is precisely what unlocks Alternative D.

Net: if verity is a hard requirement, squashfs buys it cheaply at the cost of multi-base dedup. If it isn't, your all-Btrfs multi-base stack wins on storage and coexisting-bases UX essentially by construction. I'd love to chase a hybrid — sealed read-only Btrfs base snapshots + fs-verity — so we keep native CoW/branching and an integrity story. To write Alternative D up fairly I'd rather use your real numbers than assume: if you can share the multi-base layout you're experimenting with, and how you're handling atomic A/B update + rollback on a single Btrfs pool (RAUC's discrete-slot model doesn't map cleanly onto one shared pool — I'd expect snapshot-swap orchestration instead), I'll flesh it out with actual tradeoffs.

[xbizzybone]

@Yury-MonZon thanks — these are exactly the day-2 issues the layout has to earn, so let me be precise about what's designed vs. what actually runs today.

base_min_version mismatch after a base swap → refused, not migrated or reset. The manifest's base_min_version is an anti-corruption guard: on an incompatible base the profile MUST refuse to mount and offer a rebuild, rather than silently overlaying onto a base it wasn't built against. The Phase-0 PoC enforces the refusal in the boot path. Two honest caveats: the "rebuild" path is named in the RFC but not yet specified (what it regenerates, and from what), and the PoC currently does an exact base-version match rather than a real "≥ minimum" comparison — so the "min version" semantics are aspirational at the moment, not implemented.

Packages per profile & dedup → shared base + deltas, never a full userland per profile. Userland is served read-only from the single immutable base (squashfs + dm-verity) as the OverlayFS lower; each profile's Btrfs subvolume is just the upper, holding only its delta — drop-in config, packages baked at profile build time (the profile.toml [packages] manifest), and runtime writes. One scope caveat: the PoC's profiles currently carry only drop-in config — build-time package baking is Phase 3 and isn't implemented yet, so I'm describing that path from the design, not a running demo.

On dedup, to be precise rather than overclaim: there's dedup in exactly one place today — profiles live in a Btrfs pool, so clone is a reflink snapshot sharing extents with its parent via CoW (only the diff costs space, O(metadata)); the PoC proves that. What does not dedup: (1) two independently-built profiles baking the same package — no content-addressed store, so they get separate copies; and (2) anything across base images — the base is deliberately squashfs outside Btrfs, so multiple bases share no blocks. That second point is exactly what @alchark raised above, and it's now an open tradeoff in the RFC (Alternative D / unresolved question #4). The content-addressed answer that would dedup all of it (ostree, Option C) is acknowledged but deferred.

Backup / export → honest split. Profile export is on the roadmap but undesigned — btrfs send/receive is a clean fit since profiles are already CoW subvolumes, but there's no committed artifact format, no signing/verification story yet (the overlay isn't verity-protected, so an imported profile is user-trusted — a real trust gap for a "profile store"), and no cross-base-version compatibility on import. And /data backup/restore is a genuine gap — honestly "for later"; the RFC neither designs nor mentions it and the PoC doesn't touch it. Happy to spec the export artifact + trust model if there's appetite.

Day-2 overall (updates / rollback / recovery) → designed, not yet demonstrated. Rollback is a U-Boot boot-counter plus a flipper-health.service that only commits a slot after it proves itself, with the MCU watchdog as a hardware backstop that can force a rollback even if Linux wedges. But to be straight about scope: the PoC proves Phase 0 (the storage model) only — A/B + RAUC + watchdog are Phases 1–2 and aren't built yet. The natural next step is a Phase-1 demo (QEMU aarch64 + U-Boot A/B slot selection + RAUC boot-count rollback) to make the rollback story real rather than asserted.