diff --git a/README.md b/README.md index 7c14bc7f..46062580 100644 --- a/README.md +++ b/README.md @@ -507,8 +507,8 @@ inspect-tab Edit Mode. | exFAT | Yes | Yes | Yes (in-place + defragmenting clone) | — | Modern removable media (e.g. MiSTer SD cards). In-place resize trims trailing free space; the defragmenting clone (Compact Space toggle / shrink-to-minimum) repacks allocated clusters into a fresh, smaller volume, so a fragmented card backs up to ~its real data size. | | NTFS | Yes | Yes | Yes (in-place + defragmenting clone) | — | Windows NT / 2000 / XP. In-place resize trims trailing free space; the defragmenting clone (Compact Space toggle / shrink-to-minimum) repacks into a fresh, smaller NTFS volume (from-scratch clean-room formatter, validated to mount under ntfs-3g). Create blank volumes with `rb-cli new --fs ntfs` (selectable `--cluster-size` / `--sector-size`, 512 B–2 MiB clusters); the defragmenting clone inherits the source volume's cluster and sector size. | | ext2 / ext3 / ext4 | Yes | Yes | Yes | — | Early Linux installs onward | -| HFS (Mac OS Standard) | Yes | Yes | Yes | Yes (check + repair: replica copy, bitmap fixup, lost+found for orphans) | Classic Mac OS 68k / early PowerPC. Includes block-size expansion via clone (`Expand HFS Volume…`). | -| HFS+ / HFSX | Yes | Yes | Yes (defrag clone) | Yes (check + repair) | Mac OS Extended; hardlink resolution. | +| HFS (Mac OS Standard) | Yes | Yes | Yes | Yes (check + repair: replica copy, bitmap fixup, lost+found for orphans) | Classic Mac OS 68k / early PowerPC. Includes block-size expansion via clone (`Expand HFS Volume…`). Volumes written by rusty-backup are `fsck_hfs`-clean and mountable on real Mac OS (verified on Mac OS X 10.4). | +| HFS+ / HFSX | Yes | Yes | Yes (defrag clone) | Yes (check + repair) | Mac OS Extended; hardlink resolution. Create blank volumes with `rb-cli new --fs hfsplus` (`--case-sensitive` for HFSX). The catalog / extents-overflow / attributes B-trees grow their backing fork on demand, so an under-sized or foreign catalog fills in place without a spurious "disk full"; all written volumes are `fsck_hfs`-clean and mountable on macOS. | | btrfs | Yes | No | No | — | Modern Linux; read-only browse | | ProDOS | Yes | Yes | Yes | — | Apple II / IIgs | | CBM DOS (1541 / 1571 / 1581 / 8050 / 8250) | Yes | Yes | — (floppy, fixed geometry) | — | Commodore C64 / C128 / C16 / VIC-20 / PET. PETSCII names, bit-set-is-free BAM, linked-sector files. `.d64` / `.d71` / `.d81` / `.d80` / `.d82`; `.g64` GCR decoded to sectors. | diff --git a/docs/cli-reference.md b/docs/cli-reference.md index c452369c..6017e00d 100644 --- a/docs/cli-reference.md +++ b/docs/cli-reference.md @@ -19,6 +19,28 @@ Usage: rb-cli [OPTIONS] - `--log-file` — Mirror full trace-level log output to PATH regardless of `--log-level`. Useful on Windows cmd where redirection is awkward - `--config` — Path to a config file. Overrides the platform default location. See `rb-cli config path` for what that location is +## Path grammar (in-image paths) + +Verbs that take a path *inside* an image (`ls`, `get`, `get-binhex`, `put`, +`put-binhex`, `mkdir`, `rm`, `cp`, `locate`) address it with one of two +grammars: + +- **Slash** (default, every filesystem): `/` is the separator. A literal `/` + inside a single name — legal on classic-Mac HFS / HFS+ volumes, e.g. + `Oxyd b/w` — is written `\/`; a literal backslash is written `\\`. So + `rb-cli get-binhex IMG "/Games/Oxyd 3.6/Oxyd b\/w" out.hqx` extracts the + single file `Oxyd b/w` from the folder `Oxyd 3.6`. +- **Colon** (HFS / HFS+ only): because classic Mac OS reserves `:` as its path + separator, `:` can never appear in a name, so you may instead write the path + with `:` separators — the native Mac convention — and then `/` is ordinary + data needing no escape: `rb-cli get-binhex IMG ":Games:Oxyd 3.6:Oxyd b/w" + out.hqx`. A colon-grammar path is always literal (it never globs). + +On every other filesystem `:` is an ordinary filename byte and only the slash +grammar applies. Glob patterns (`*`, `?`, `[`, `{`) use the slash grammar; pass +`--literal` (or use the colon grammar) to address a name containing those +characters verbatim. + ## Verbs ### `api` @@ -643,7 +665,7 @@ Usage: get [OPTIONS] **Arguments** - `` — Image reference (`path` or `path@N` for the 1-based partition index) -- `` — Source path or glob inside the filesystem. Patterns containing `*`, `?`, `[`, or `{` walk the volume and extract every match. Pass `--literal` to extract a single path verbatim when its name contains those characters +- `` — Source path or glob inside the filesystem. Patterns containing `*`, `?`, `[`, or `{` walk the volume and extract every match. Pass `--literal` to extract a single path verbatim when its name contains those characters. A literal `/` in a name is written `\/` (or use a `:`-separated path on HFS / HFS+, which also forces literal) - `` — Destination path on the host. Single-match: the literal target file. Multi-match or directory source: a directory under which matched entries are laid out (created if it doesn't exist) **Options** @@ -677,7 +699,7 @@ Usage: get-binhex [OPTIONS] **Options** - `--password` — Password for encrypted containers (currently: WinImage IMZ) -- `-L` / `--literal` — Accepted for consistency with `ls`/`get`/`rm`; `get-binhex` always treats the source as an exact literal path (it never globs), so glob metacharacters in a name are addressed verbatim with or without it +- `-L` / `--literal` — Accepted for consistency with `ls`/`get`/`rm`; `get-binhex` always treats the source as an exact literal path (it never globs), so glob metacharacters in a name are addressed verbatim with or without it. A literal `/` in a name is written `\/` (or use a `:`-separated path on HFS / HFS+) ### `grow` @@ -739,7 +761,7 @@ Usage: locate [OPTIONS] **Arguments** - `` — Image reference (`path` or `path@N` for the 1-based partition index) -- `` — Path inside the filesystem (Mac path conventions; `/` is the separator — `:` is rejected for the same reason as the other verbs) +- `` — Path inside the filesystem. `/` is the separator; a literal `/` in a name is written `\/`. You may instead use `:` as the separator (the native classic-Mac convention), in which case `/` is plain data — e.g. `:System Folder:Oxyd b/w` **Options** @@ -827,7 +849,7 @@ Usage: mkdir [OPTIONS] **Arguments** - `` — Image reference (`path` or `path@N` for the 1-based partition index) -- `` — Directory path to create. The parent must exist (no `-p`-style auto-creation in Phase B) +- `` — Directory path to create. The parent must exist (no `-p`-style auto-creation in Phase B). A literal `/` in the new name is written `\/`; on HFS / HFS+ a `:`-separated path also works **Options** @@ -847,12 +869,14 @@ Usage: new [OPTIONS] --fs **Options** -- `--fs` — Filesystem to format. One of: hfs, hfv, fat, efs, affs, ntfs +- `--fs` — Filesystem to format. One of: hfs, hfsplus, hfv, fat, efs, affs, ntfs - `--size` — Volume size, accepting plain bytes or `K`/`KiB`/`M`/`MiB`/`G`/`GiB` suffixes (e.g. `800K`, `5M`). Defaults to 800K (an 800 KiB floppy) - `--name` — Volume label/name. Defaults to `rusty-backup`. HFS: up to 27 Mac Roman bytes. FAT: up to 11 chars (uppercased; non-ASCII → `_`). EFS: 6-byte fname/fpack. AFFS: up to 30 bytes - `--block-size` — HFS allocation block size in bytes. Must be a non-zero multiple of 512. When unset, the smallest size that keeps `total_blocks <= 65535` is chosen automatically. Ignored for other filesystems - `--catalog-size` — HFS Catalog B-tree initial size in bytes (rounded up to a whole allocation block). When unset, scales with volume size like hformat (~0.5%, clump-aligned, 24-block floor). Ignored for other filesystems - `--extents-size` — HFS Extents-overflow B-tree initial size in bytes (rounded up to a whole allocation block). When unset, ~half the catalog size. Ignored for other filesystems +- `--case-sensitive` — HFS+ only: format a case-sensitive (HFSX) volume instead of the default case-insensitive HFS+. Ignored for other filesystems +- `--min-catalog` — HFS+ only: minimum catalog B-tree size in bytes (a floor, rounded up to whole 4096-byte nodes). Set this *small* to make the catalog easy to outgrow and exercise the fork grow-on-full path. Ignored for other filesystems - `--affs-variant` — AFFS variant byte (0=OFS, 1=FFS, 2=OFS+intl, 3=FFS+intl, 4=OFS+dircache, 5=FFS+dircache). Defaults to 1 (FFS) - `--inodes` — EFS only: approximate total inode count. The formatter scales its cylinder groups to hit roughly this many inodes. Mutually exclusive with `--bytes-per-inode`; default density is ~1 inode/4 KiB - `--bytes-per-inode` — EFS only: inode density in bytes per inode (smaller = more inodes), floored at one inode per 512-byte block. Mutually exclusive with `--inodes` @@ -924,7 +948,7 @@ Usage: new-x68k-hdd [OPTIONS] ### `optical` -Optical-media verbs (rip / convert / browse / extract) +Optical-media verbs (drives / rip / convert / browse / extract) ``` Usage: optical @@ -959,6 +983,18 @@ Usage: convert --format - `--format` — Output format +### `optical drives` + +List connected physical optical drives and their device paths + +``` +Usage: drives [OPTIONS] +``` + +**Options** + +- `--remote` — Also query these daemons for their optical drives (repeatable), e.g. `--remote mister.local:7341`. Remote rows print an `rb://...` device arg you can pass straight to `optical rip --device` + ### `optical extract` Extract files from an optical disc image into a host folder @@ -986,7 +1022,7 @@ Usage: rip [OPTIONS] --device --output **Options** -- `--device` — Source drive (e.g. `/dev/sr0`, `disk6`, `\\.\E:`). See `rb-cli show devices` +- `--device` — Source drive: a local path (e.g. `/dev/sr0`, `disk6`, `\\.\E:`) or a remote daemon's drive as `rb://host:port/dev/sr0` (the daemon issues the SCSI reads; this side does the encoding). `rb-cli optical drives` lists local drives - `--output` — Output path: `.iso` for `--format iso`, `.cue` for `--format bincue` - `--format` — - `--eject` — Eject the disc after a successful rip @@ -1130,7 +1166,7 @@ Usage: put [OPTIONS] [HOST_FILE] [DST] - `` — Image reference (`path` or `path@N` for the 1-based partition index) - `` — Host file to copy. Required when not using `--zero` or `--boot` -- `` — Destination path inside the filesystem (cp-like positional) +- `` — Destination path inside the filesystem (cp-like positional). A literal `/` in the name is written `\/`; on HFS / HFS+ a `:`-separated path also works (so `/` is plain data) **Options** @@ -1161,7 +1197,7 @@ Usage: put-binhex [OPTIONS] **Options** -- `--dst-dir` — Destination directory inside the filesystem (`/` for root). The filename comes from the BinHex header. Defaults to `/` +- `--dst-dir` — Destination directory inside the filesystem (`/` for root). The filename comes from the BinHex header. Defaults to `/`. A literal `/` in a directory name is written `\/`; on HFS / HFS+ a `:`-separated path also works (so `/` is plain data). Defaults to `/` - `--rename` — Override the filename from the BinHex header - `--force` — Overwrite an existing entry at the destination path - `--clear-inited` — Clear the `hasBeenInited` Finder flag (0x0100) on the written file. Use when injecting an app onto a fresh disk so the Finder re-reads its `BNDL` and registers real icons (a file copied with `hasBeenInited` already set is treated as already-catalogued, so it shows a generic icon until a desktop rebuild). Mirrors what a MacBinary install does to byte 73 @@ -1271,7 +1307,7 @@ Usage: rm [OPTIONS] **Arguments** - `` — Image reference (`path` or `path@N` for the 1-based partition index) -- `` — Path or glob pattern inside the filesystem. Patterns containing `*`, `?`, `[`, or `{` walk the volume and delete every match. Pass `--literal` to delete a single path verbatim when its name contains those characters +- `` — Path or glob pattern inside the filesystem. Patterns containing `*`, `?`, `[`, or `{` walk the volume and delete every match. Pass `--literal` to delete a single path verbatim when its name contains those characters. A literal `/` in a name is written `\/` (or use a `:`-separated path on HFS / HFS+, which also forces literal) **Options** diff --git a/docs/hfsplus_btree_growth_plan.md b/docs/hfsplus_btree_growth_plan.md index b4250c94..53eadabe 100644 --- a/docs/hfsplus_btree_growth_plan.md +++ b/docs/hfsplus_btree_growth_plan.md @@ -1,6 +1,8 @@ # HFS+ catalog B-tree growth & variable-length keys — implementation plan -**Status:** complete (P1–P5 landed; §4b grow-on-full intentionally deferred). +**Status:** complete (P1–P5 landed; §4b grow-on-full now also landed — see +[`docs/todo_hfsplus_fork_growth.md`](todo_hfsplus_fork_growth.md), Phases A/B/C +shipped and `fsck_hfs`-validated on macOS). - **P1 (key-format descriptor / variable-length index keys, §4a) — landed.** The shared B-tree helpers in `hfs_common.rs` are now key-format-aware (`BTreeKeyFormat`), the HFS+ catalog/attributes inserts and the defrag builders @@ -171,7 +173,14 @@ Back-compat: classic HFS passes a `BTreeKeyFormat { big_keys:false, variable_index_keys:false, max_key_len:37 }` and the behaviour is byte-identical to today (lock this with a golden test). -### 4b. B-tree file growth on full — DEFERRED +### 4b. B-tree file growth on full — DONE (2026-06-29) + +> **Landed (2026-06-29).** Implemented as `grow_btree_fork` (Phases A/B/C) and +> validated `fsck_hfs`-clean on macOS — see +> [`docs/todo_hfsplus_fork_growth.md`](todo_hfsplus_fork_growth.md) for the full +> writeup, including the two foundational blank-builder bugs (Invalid BTH length, +> empty-tree representation) that a real Mac surfaced. The historical deferral +> note below is retained for context. > **Deferred (2026-06-28).** Implemented sizing (4c) instead and mirrored classic > HFS, which has **no** grow-on-full path at all — it relies entirely on diff --git a/docs/todo_hfsplus_fork_growth.md b/docs/todo_hfsplus_fork_growth.md index ed724513..9940b0d8 100644 --- a/docs/todo_hfsplus_fork_growth.md +++ b/docs/todo_hfsplus_fork_growth.md @@ -1,6 +1,38 @@ # HFS+ B-tree fork grow-on-full (§4b) — implementation + macOS validation plan -**Status:** TODO / not started. This is the one deferred step of +**Status: DONE — Phases A, B, and C all shipped and `fsck_hfs`-validated on +macOS.** The catalog / extents-overflow / attributes B-trees now grow their +backing fork when they run out of nodes via `HfsPlusFilesystem::grow_btree_fork` +(`src/fs/hfsplus.rs`), with a clean-`DiskFull` + retry-once at each of the three +insert methods. Phase A is contiguous tail growth; Phase B spills the 9th+ extent +into the extents-overflow B-tree (overflow-aware `write_fork_data_with_overflow`); +Phase C appends map nodes past the `(node_size-256)*8` ≈ 30,720-node header cap. + +**Two foundational bugs surfaced and fixed along the way** (every HFS+ image +rusty-backup *built from scratch* tripped them, but only a real Mac caught them — +which is exactly why this work was deferred until a Mac was available): + +1. **Invalid BTH length.** `write_blank_btree_header_node` laid the BTHeaderRec + (record 0) out as 128 bytes (record 1 at offset 142); Apple requires it to be + exactly 106 bytes (record 1 at 120, bitmap at 248). `fsck_hfs` rejected every + blank with "Invalid BTH length". Fixing it to the canonical 120/248 layout + also resolved the §5 header-vs-map-node capacity discrepancy — the bitmap now + addresses exactly `(node_size-256)*8` nodes, matching `map_nodes_required`. +2. **Empty-tree representation + trailing block.** An empty B-tree was written as + depth-1 with an empty root leaf (Apple uses depth-0 / no leaf), and the + allocation bitmap didn't reserve the trailing block holding the alternate + volume header. Both made `fsck_hfs` report "Invalid node structure" / + "Invalid volume free block count". `btree_insert_full` now bootstraps the root + leaf on the first insert into a depth-0 tree. + +Validated on macOS via `rb-cli new --fs hfsplus` (Phase 0) + `rb-cli batch` +(real `create_file` puts past the 8-extent limit) and the in-repo grow tests' +emitted images (`RB_EMIT_GROW_IMAGE` / `_SPILL_` / `_MAPC_` / `_DEFRAG_`), each +`fsck_hfs -f -n` "appears to be OK" and mountable. + +--- + +This is the one deferred step of [`docs/hfsplus_btree_growth_plan.md`](hfsplus_btree_growth_plan.md) (§4b). P1–P5 (variable-length index keys, blank catalog sizing, extents/attributes splitting, defrag re-verification, and density rotation) all shipped; this doc is the diff --git a/example/generate_cli_reference.rs b/example/generate_cli_reference.rs index d935ad73..622e88c4 100644 --- a/example/generate_cli_reference.rs +++ b/example/generate_cli_reference.rs @@ -37,6 +37,8 @@ fn main() -> std::io::Result<()> { writeln!(md, "## Global options\n").unwrap(); render_args(&mut md, &cmd, &[]); + md.push_str(PATH_GRAMMAR); + writeln!(md, "## Verbs\n").unwrap(); let mut sub_path = Vec::new(); render_subcommands(&mut md, &cmd, &mut sub_path); @@ -46,6 +48,35 @@ fn main() -> std::io::Result<()> { Ok(()) } +/// Static section documenting how in-image paths are tokenised. Lives in the +/// generator (not a per-arg help string) because it applies uniformly to every +/// path-taking verb (`ls`, `get`, `get-binhex`, `put`, `put-binhex`, `mkdir`, +/// `rm`, `cp`, `locate`). +const PATH_GRAMMAR: &str = "\ +## Path grammar (in-image paths) + +Verbs that take a path *inside* an image (`ls`, `get`, `get-binhex`, `put`, +`put-binhex`, `mkdir`, `rm`, `cp`, `locate`) address it with one of two +grammars: + +- **Slash** (default, every filesystem): `/` is the separator. A literal `/` + inside a single name — legal on classic-Mac HFS / HFS+ volumes, e.g. + `Oxyd b/w` — is written `\\/`; a literal backslash is written `\\\\`. So + `rb-cli get-binhex IMG \"/Games/Oxyd 3.6/Oxyd b\\/w\" out.hqx` extracts the + single file `Oxyd b/w` from the folder `Oxyd 3.6`. +- **Colon** (HFS / HFS+ only): because classic Mac OS reserves `:` as its path + separator, `:` can never appear in a name, so you may instead write the path + with `:` separators — the native Mac convention — and then `/` is ordinary + data needing no escape: `rb-cli get-binhex IMG \":Games:Oxyd 3.6:Oxyd b/w\" + out.hqx`. A colon-grammar path is always literal (it never globs). + +On every other filesystem `:` is an ordinary filename byte and only the slash +grammar applies. Glob patterns (`*`, `?`, `[`, `{`) use the slash grammar; pass +`--literal` (or use the colon grammar) to address a name containing those +characters verbatim. + +"; + fn render_subcommands(out: &mut String, cmd: &Command, path: &mut Vec) { let mut subs: Vec<&Command> = cmd.get_subcommands().collect(); subs.sort_by_key(|c| c.get_name()); diff --git a/src/cli/api/hfs.rs b/src/cli/api/hfs.rs index 17b427a6..7352dc4a 100644 --- a/src/cli/api/hfs.rs +++ b/src/cli/api/hfs.rs @@ -322,6 +322,73 @@ pub(crate) fn cmd_new_sized( Ok(()) } +/// Create a fresh blank **HFS+ / HFSX** volume (a bare single volume, no +/// partition map) at `image`. `case_sensitive` selects HFSX; `min_catalog` +/// is a *floor* on the catalog B-tree size in bytes — set it small to make +/// the catalog easy to outgrow (exercises the §4b fork-grow path with a few +/// thousand `put`s instead of hundreds of thousands). +pub(crate) fn cmd_new_hfsplus_sized( + image: PathBuf, + size_arg: &str, + name: &str, + block_size: Option, + case_sensitive: bool, + min_catalog: Option, +) -> Result<()> { + let size = parse_size(size_arg)?; + // HFS+ pins its node size at 4096, so the allocation block size must be a + // power of two in [512, 4096]. Default to 4096 (Apple's default for any + // non-trivial volume); the volume must hold at least 64 blocks. + let block_size = match block_size { + Some(bs) => { + if !bs.is_power_of_two() || !(512..=4096).contains(&bs) { + bail!("HFS+ block-size must be a power of two in [512, 4096] (got {bs})"); + } + bs + } + None => 4096, + }; + if size < block_size as u64 * 64 { + bail!( + "size {size} is too small for an HFS+ volume (need at least {} bytes \ + for 64 allocation blocks of {block_size})", + block_size as u64 * 64 + ); + } + + let bytes = crate::fs::hfsplus::create_blank_hfsplus_sized( + size, + block_size, + name, + case_sensitive, + min_catalog.unwrap_or(0), + 0, + ); + + let mut file = OpenOptions::new() + .create(true) + .write(true) + .truncate(true) + .open(&image) + .with_context(|| format!("opening {} for write", image.display()))?; + file.write_all(&bytes) + .with_context(|| format!("writing {}", image.display()))?; + file.flush().ok(); + + println!( + "Created {} volume {} ({} bytes, block_size={}{})", + if case_sensitive { "HFSX" } else { "HFS+" }, + image.display(), + bytes.len(), + block_size, + match min_catalog { + Some(c) => format!(", min-catalog={c}"), + None => String::new(), + } + ); + Ok(()) +} + // --------------------------------------------------------------------------- // info / ls // --------------------------------------------------------------------------- @@ -533,21 +600,18 @@ pub fn resolve_path( fs: &mut HfsFilesystem, path: &str, ) -> Result { + // HFS reserves `:`, so a `:` in the path is always a separator (colon + // grammar) and `/` is plain data; a plain `/`-path uses `\/` for a literal + // slash. See `crate::cli::parse::split_image_path`. + let components = crate::cli::parse::split_image_path(path, path.contains(':')); let mut current = fs.root().map_err(|e| anyhow!("root: {e}"))?; - let trimmed = path.trim_start_matches('/').trim_end_matches('/'); - if trimmed.is_empty() { - return Ok(current); - } - for component in trimmed.split('/') { - if component.is_empty() { - continue; - } + for component in &components { let children = fs .list_directory(¤t) .map_err(|e| anyhow!("list_directory: {e}"))?; let next = children .into_iter() - .find(|c| c.name == component) + .find(|c| &c.name == component) .ok_or_else(|| anyhow!("path component not found: {component}"))?; current = next; } diff --git a/src/cli/parse.rs b/src/cli/parse.rs index fe875fc8..f0ec9c41 100644 --- a/src/cli/parse.rs +++ b/src/cli/parse.rs @@ -44,8 +44,95 @@ pub fn pick_block_size(volume_bytes: u64) -> u32 { bs } +/// Split an in-image path into its decoded components. +/// +/// Two grammars, picked by the caller via `colon`: +/// +/// - **slash** (`colon == false`, every filesystem): `/`-delimited. A literal +/// `/` inside a component is written `\/`; a literal `\` is written `\\`. +/// Any other `\x` is preserved verbatim (the backslash is kept) so a stray +/// backslash in a real name survives a round trip. +/// - **colon** (`colon == true`, only filesystems where +/// [`crate::fs::filesystem::Filesystem::uses_colon_paths`] is true): `:`-delimited, +/// and `/` is an ordinary filename byte needing no escape. `:` can never +/// appear in an HFS/HFS+ name, so this grammar is unambiguous on those +/// volumes. +/// +/// Leading / trailing separators are ignored and empty components dropped, so +/// `/a/b`, `a/b`, and `/a/b/` all yield `["a", "b"]`. +pub fn split_image_path(path: &str, colon: bool) -> Vec { + if colon { + return path + .split(':') + .filter(|c| !c.is_empty()) + .map(|c| c.to_string()) + .collect(); + } + let mut components = Vec::new(); + let mut cur = String::new(); + let mut escaped = false; + for c in path.chars() { + if escaped { + match c { + '/' | '\\' => cur.push(c), + other => { + // Not an escape we recognise — keep the backslash literal. + cur.push('\\'); + cur.push(other); + } + } + escaped = false; + } else if c == '\\' { + escaped = true; + } else if c == '/' { + components.push(std::mem::take(&mut cur)); + } else { + cur.push(c); + } + } + if escaped { + // Trailing lone backslash: keep it as data. + cur.push('\\'); + } + components.push(cur); + components.into_iter().filter(|c| !c.is_empty()).collect() +} + +/// Split an in-image path into `(parent_components, basename)`, decoding with +/// the same escape / colon rules as [`split_image_path`]. The root path yields +/// `([], "")`. +pub fn split_image_parent(path: &str, colon: bool) -> (Vec, String) { + let mut components = split_image_path(path, colon); + match components.pop() { + Some(name) => (components, name), + None => (Vec::new(), String::new()), + } +} + +/// Encode a single path component for the slash grammar: a literal `\` becomes +/// `\\` and a literal `/` becomes `\/`. The inverse of one component of +/// [`split_image_path`] in slash mode. Callers that build a slash path from +/// known components (or external tooling mirroring this convention) use it so +/// the result round-trips back through [`split_image_path`]. +pub fn escape_path_component(component: &str) -> String { + let mut out = String::with_capacity(component.len()); + for c in component.chars() { + if c == '\\' || c == '/' { + out.push('\\'); + } + out.push(c); + } + out +} + /// Split a `/`-delimited Mac path into `(parent_dir, basename)`. Trailing /// slashes are stripped; the root `/` returns `("/", "")`. +/// +/// This is the plain `/`-splitter still used on the remote (`rb://`) path, +/// where addressing is slash-only and the daemon does its own resolution. +/// Local verbs go through [`split_image_parent`] / +/// [`crate::cli::verbs::ls::resolve_parent`] instead, which understand the `\/` +/// escape and colon grammar. pub fn split_mac_path(p: &str) -> Result<(String, String)> { let normalized = p.trim_end_matches('/'); if normalized.is_empty() || normalized == "/" { @@ -108,6 +195,74 @@ mod tests { assert_eq!(split_mac_path("/").unwrap(), ("/".into(), String::new())); } + #[test] + fn split_image_path_slash_basic() { + assert_eq!(split_image_path("/a/b/c", false), vec!["a", "b", "c"]); + assert_eq!(split_image_path("a/b/c", false), vec!["a", "b", "c"]); + assert_eq!(split_image_path("/a/b/c/", false), vec!["a", "b", "c"]); + assert!(split_image_path("/", false).is_empty()); + assert!(split_image_path("", false).is_empty()); + } + + #[test] + fn split_image_path_escaped_slash_is_one_component() { + // "Oxyd 3.6" / "Oxyd b/w" addressed with a \/ escape. + assert_eq!( + split_image_path(r"/Games/Oxyd 3.6/Oxyd b\/w", false), + vec!["Games", "Oxyd 3.6", "Oxyd b/w"] + ); + // A component that is literally just "/". + assert_eq!(split_image_path(r"/a/\//b", false), vec!["a", "/", "b"]); + } + + #[test] + fn split_image_path_double_backslash_is_literal_backslash() { + assert_eq!( + split_image_path(r"/a/b\\c/d", false), + vec!["a", r"b\c", "d"] + ); + // An unrecognised escape keeps the backslash. + assert_eq!(split_image_path(r"/a/b\xc", false), vec!["a", r"b\xc"]); + // Trailing lone backslash survives as data. + assert_eq!(split_image_path(r"/a/b\", false), vec!["a", r"b\"]); + } + + #[test] + fn split_image_path_colon_mode_slash_is_data() { + assert_eq!( + split_image_path(":Games:Oxyd 3.6:Oxyd b/w", true), + vec!["Games", "Oxyd 3.6", "Oxyd b/w"] + ); + // No leading colon also works; trailing colon dropped. + assert_eq!( + split_image_path("Games:Oxyd 3.6:", true), + vec!["Games", "Oxyd 3.6"] + ); + assert!(split_image_path(":", true).is_empty()); + } + + #[test] + fn split_image_parent_separates_basename() { + assert_eq!( + split_image_parent(r"/Apps/Oxyd 3.6/Oxyd b\/w", false), + (vec!["Apps".into(), "Oxyd 3.6".into()], "Oxyd b/w".into()) + ); + assert_eq!(split_image_parent("/", false), (Vec::new(), String::new())); + assert_eq!( + split_image_parent(":Apps:Oxyd b/w", true), + (vec!["Apps".into()], "Oxyd b/w".into()) + ); + } + + #[test] + fn escape_path_component_round_trips() { + for name in ["plain", "Oxyd b/w", r"a\b", r"weird/\name"] { + let escaped = escape_path_component(name); + // A single escaped component re-parses to exactly that name. + assert_eq!(split_image_path(&escaped, false), vec![name.to_string()]); + } + } + #[test] fn pick_block_size_floppy_and_large() { assert_eq!(pick_block_size(819_200), 512); diff --git a/src/cli/verbs/binhex.rs b/src/cli/verbs/binhex.rs index 69088436..afa90fb0 100644 --- a/src/cli/verbs/binhex.rs +++ b/src/cli/verbs/binhex.rs @@ -17,7 +17,6 @@ use std::path::PathBuf; use crate::cli::img_at::ImageRef; use crate::cli::logging::log_stderr; -use crate::cli::parse::split_mac_path; use crate::cli::resolve::{resolve_partition_rw, resolve_partition_streaming_with_password}; use crate::fs::binhex::{self, BinHexFile}; use crate::fs::filesystem::{CreateFileOptions, ResourceForkSource}; @@ -31,7 +30,9 @@ pub struct PutBinHexArgs { pub host_file: PathBuf, /// Destination directory inside the filesystem (`/` for root). The - /// filename comes from the BinHex header. Defaults to `/`. + /// filename comes from the BinHex header. Defaults to `/`. A literal `/` + /// in a directory name is written `\/`; on HFS / HFS+ a `:`-separated path + /// also works (so `/` is plain data). Defaults to `/`. #[arg(long = "dst-dir", default_value = "/")] pub dst_dir: String, @@ -70,6 +71,8 @@ pub struct GetBinHexArgs { /// Accepted for consistency with `ls`/`get`/`rm`; `get-binhex` always /// treats the source as an exact literal path (it never globs), so glob /// metacharacters in a name are addressed verbatim with or without it. + /// A literal `/` in a name is written `\/` (or use a `:`-separated path on + /// HFS / HFS+). #[arg(short = 'L', long = "literal", alias = "no-glob")] pub literal: bool, } @@ -95,18 +98,12 @@ pub fn run_put(args: PutBinHexArgs) -> Result<()> { ) .map_err(|e| anyhow!("opening filesystem for write: {e}"))?; - let (parent_path, _) = split_mac_path(&format!( - "{}/x", - args.dst_dir.trim_end_matches('/').trim_start_matches('/'), - ))?; - let parent_path = if parent_path == "/" { - "/".to_string() - } else { - parent_path - }; - let parent = super::ls::resolve_path(&mut *fs, &parent_path)?; + // `--dst-dir` is the destination *directory* (the filename comes from the + // BinHex header), resolved with the shared escape / colon grammar so a + // donor folder named with a literal `/` is addressable. + let parent = super::ls::resolve_path(&mut *fs, &args.dst_dir)?; if !parent.is_directory() { - bail!("parent is not a directory: {parent_path}"); + bail!("destination is not a directory: {}", args.dst_dir); } let existing = fs @@ -118,7 +115,7 @@ pub fn run_put(args: PutBinHexArgs) -> Result<()> { if !args.force { bail!( "{}/{} already exists (pass --force to overwrite)", - parent_path.trim_end_matches('/'), + args.dst_dir.trim_end_matches('/'), target_name ); } diff --git a/src/cli/verbs/get.rs b/src/cli/verbs/get.rs index 7e04bd90..922b0a45 100644 --- a/src/cli/verbs/get.rs +++ b/src/cli/verbs/get.rs @@ -31,7 +31,7 @@ use anyhow::{anyhow, bail, Context, Result}; use clap::Args; use std::path::{Path, PathBuf}; -use crate::cli::copy_paths::{base_name_of, compute_glob_root, has_glob_chars, strip_root_prefix}; +use crate::cli::copy_paths::{compute_glob_root, has_glob_chars, strip_root_prefix}; use crate::cli::glob::{collect_matches, compile_patterns}; use crate::cli::img_at::ImageRef; use crate::cli::logging::{log_stderr, out_stdout}; @@ -47,7 +47,8 @@ pub struct GetArgs { /// Source path or glob inside the filesystem. Patterns containing /// `*`, `?`, `[`, or `{` walk the volume and extract every match. /// Pass `--literal` to extract a single path verbatim when its name - /// contains those characters. + /// contains those characters. A literal `/` in a name is written `\/` + /// (or use a `:`-separated path on HFS / HFS+, which also forces literal). pub src: String, /// Destination path on the host. Single-match: the literal target @@ -170,9 +171,10 @@ pub fn run(args: GetArgs) -> Result<()> { // Decide the dispatch shape. Globs and exclude lists always go // through the glob walker; a literal source goes through resolve_path // and is treated as a single file or a recursive directory dump. - // `--literal` forces the resolve_path branch even for names that contain - // glob metacharacters. - if !args.literal && (has_glob_chars(&args.src) || !args.exclude.is_empty()) { + // `--literal` (or a colon-grammar path on HFS / HFS+) forces the + // resolve_path branch even for names that contain glob metacharacters. + let colon = super::ls::colon_mode(&*fs, &args.src); + if !args.literal && !colon && (has_glob_chars(&args.src) || !args.exclude.is_empty()) { return run_glob( &mut *fs, &args.src, @@ -197,8 +199,10 @@ pub fn run(args: GetArgs) -> Result<()> { // Directory dump: lay the tree under DST//..., // matching cp/rsync convention. Implemented as a recursive walk // over the live FS — no need to spin up the glob matcher for a - // single-rooted recursion. - let base_name = base_name_of(&args.src); + // single-rooted recursion. Decode the basename with the same escape / + // colon rules used to resolve it, so a `\/`-bearing source folder lands + // under its real name on the host. + let base_name = crate::cli::parse::split_image_parent(&args.src, colon).1; let target_root = if base_name.is_empty() { args.dst.clone() } else { diff --git a/src/cli/verbs/locate.rs b/src/cli/verbs/locate.rs index bbf61abc..7794c60a 100644 --- a/src/cli/verbs/locate.rs +++ b/src/cli/verbs/locate.rs @@ -37,8 +37,10 @@ pub struct LocateArgs { /// Image reference (`path` or `path@N` for the 1-based partition index). pub image: ImageRef, - /// Path inside the filesystem (Mac path conventions; `/` is the - /// separator — `:` is rejected for the same reason as the other verbs). + /// Path inside the filesystem. `/` is the separator; a literal `/` in a + /// name is written `\/`. You may instead use `:` as the separator (the + /// native classic-Mac convention), in which case `/` is plain data — + /// e.g. `:System Folder:Oxyd b/w`. pub path: String, /// Output format. `json` is the default because the load-bearing @@ -167,21 +169,18 @@ fn resolve_hfs_path( path: &str, ) -> Result { use crate::fs::filesystem::Filesystem; + // HFS reserves `:`, so a `:` in the path is always a separator (colon + // grammar) and `/` is plain data; a plain `/`-path uses `\/` for a literal + // slash. See `crate::cli::parse::split_image_path`. + let components = crate::cli::parse::split_image_path(path, path.contains(':')); let mut current = fs.root().map_err(|e| anyhow!("root: {e}"))?; - let trimmed = path.trim_start_matches('/').trim_end_matches('/'); - if trimmed.is_empty() { - return Ok(current); - } - for component in trimmed.split('/') { - if component.is_empty() { - continue; - } + for component in &components { let children = fs .list_directory(¤t) .map_err(|e| anyhow!("list_directory: {e}"))?; let next = children .into_iter() - .find(|c| c.name == component) + .find(|c| &c.name == component) .ok_or_else(|| anyhow!("path component not found: {component}"))?; current = next; } diff --git a/src/cli/verbs/ls.rs b/src/cli/verbs/ls.rs index 81c4c7d2..5aaffff0 100644 --- a/src/cli/verbs/ls.rs +++ b/src/cli/verbs/ls.rs @@ -24,6 +24,11 @@ pub struct LsArgs { /// patterns containing `*`, `?`, `[`, or `{` walk the volume and /// emit one line per match. Pass `--literal` to address a path /// verbatim when its name contains those characters. + /// + /// A literal `/` inside a name (classic-Mac volumes allow it) is written + /// `\/` (and a literal `\` as `\\`). On HFS / HFS+ you may instead use `:` + /// as the separator — the native Mac convention — so `/` is just data: + /// `:System Folder:Oxyd b/w`. #[arg(default_value = "/")] pub path: String, @@ -114,8 +119,11 @@ pub fn run(args: LsArgs) -> Result<()> { }; // `--literal` forces the exact-path branch even when the name contains - // glob metacharacters. - let use_glob = !args.literal && (has_glob_chars(&args.path) || !args.exclude.is_empty()); + // glob metacharacters. A colon-grammar path (HFS / HFS+ with `:`) is always + // an exact path — it addresses a literal `/`-bearing name — so it never globs. + let use_glob = !args.literal + && !colon_mode(&*fs, &args.path) + && (has_glob_chars(&args.path) || !args.exclude.is_empty()); if use_glob { // Glob path — walk the volume. @@ -212,25 +220,119 @@ fn has_glob_chars(s: &str) -> bool { s.chars().any(|c| matches!(c, '*' | '?' | '[' | '{')) } +/// True when `path` should be parsed with the colon grammar: only on a +/// filesystem that reserves `:` (HFS / HFS+) and only when the path actually +/// uses a `:` (so a plain `/`-path on HFS keeps working, with `\/` available +/// for a slash inside a name). +pub fn colon_mode(fs: &dyn Filesystem, path: &str) -> bool { + fs.uses_colon_paths() && path.contains(':') +} + /// Walk `path` inside a generic filesystem, one component at a time. +/// +/// The path is tokenised with [`crate::cli::parse::split_image_path`], so a +/// literal `/` inside a name can be addressed as `\/` (slash grammar) or by +/// using `:` separators on HFS / HFS+ (colon grammar). See that function. pub fn resolve_path(fs: &mut dyn Filesystem, path: &str) -> Result { + let colon = colon_mode(fs, path); + let components = crate::cli::parse::split_image_path(path, colon); + resolve_components(fs, &components) +} + +/// Walk a path given as already-decoded components (no further splitting). +pub fn resolve_components( + fs: &mut dyn Filesystem, + components: &[String], +) -> Result { let mut current = fs.root().map_err(|e| anyhow!("root: {e}"))?; - let trimmed = path.trim_start_matches('/').trim_end_matches('/'); - if trimmed.is_empty() { - return Ok(current); - } - for component in trimmed.split('/') { - if component.is_empty() { - continue; - } + for component in components { let children = fs .list_directory(¤t) .map_err(|e| anyhow!("list_directory: {e}"))?; let next = children .into_iter() - .find(|c| c.name == component) + .find(|c| &c.name == component) .ok_or_else(|| anyhow!("path component not found: {component}"))?; current = next; } Ok(current) } + +/// Resolve the parent directory of `path` and return `(parent_entry, basename)`, +/// decoding with the same escape / colon rules as [`resolve_path`]. Used by the +/// write verbs (`put`, `mkdir`, `rm`) so the leaf name they create / delete is +/// the decoded component (e.g. a literal `Oxyd b/w`). +pub fn resolve_parent( + fs: &mut dyn Filesystem, + path: &str, +) -> Result<(crate::fs::entry::FileEntry, String)> { + let colon = colon_mode(fs, path); + let (parent_components, name) = crate::cli::parse::split_image_parent(path, colon); + let parent = resolve_components(fs, &parent_components)?; + Ok((parent, name)) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::fs::filesystem::{CreateDirectoryOptions, CreateFileOptions, EditableFilesystem}; + use crate::fs::hfs::{create_blank_hfs, HfsFilesystem}; + use std::io::Cursor; + + /// Build a blank HFS volume holding `/Oxyd 3.6/Oxyd b/w` — a folder with a + /// child file whose name contains a literal `/` (legal on classic-Mac + /// volumes). Returns the opened filesystem. + fn hfs_with_slash_named_file() -> HfsFilesystem>> { + let img = create_blank_hfs(8 * 1024 * 1024, 4096, "Test").unwrap(); + let mut fs = HfsFilesystem::open(Cursor::new(img), 0).unwrap(); + let root = fs.root().unwrap(); + let folder = fs + .create_directory(&root, "Oxyd 3.6", &CreateDirectoryOptions::default()) + .unwrap(); + let data = b"app"; + let mut reader: &[u8] = data; + fs.create_file( + &folder, + "Oxyd b/w", + &mut reader, + data.len() as u64, + &CreateFileOptions::default(), + ) + .unwrap(); + fs.sync_metadata().unwrap(); + fs + } + + #[test] + fn resolve_slash_name_via_backslash_escape() { + let mut fs = hfs_with_slash_named_file(); + let entry = resolve_path(&mut fs, r"/Oxyd 3.6/Oxyd b\/w").unwrap(); + assert_eq!(entry.name, "Oxyd b/w"); + assert!(!entry.is_directory()); + } + + #[test] + fn resolve_slash_name_via_colon_grammar() { + let mut fs = hfs_with_slash_named_file(); + let entry = resolve_path(&mut fs, ":Oxyd 3.6:Oxyd b/w").unwrap(); + assert_eq!(entry.name, "Oxyd b/w"); + assert!(!entry.is_directory()); + } + + #[test] + fn unescaped_slash_name_mis_splits_and_is_not_found() { + // Without an escape or colon grammar, the `/` is a separator: the path + // resolves through a (nonexistent) folder `Oxyd b`, so it isn't found. + let mut fs = hfs_with_slash_named_file(); + assert!(resolve_path(&mut fs, "/Oxyd 3.6/Oxyd b/w").is_err()); + } + + #[test] + fn resolve_parent_returns_decoded_slash_basename() { + let mut fs = hfs_with_slash_named_file(); + let (parent, name) = resolve_parent(&mut fs, r"/Oxyd 3.6/Oxyd b\/w").unwrap(); + assert!(parent.is_directory()); + assert_eq!(parent.name, "Oxyd 3.6"); + assert_eq!(name, "Oxyd b/w"); + } +} diff --git a/src/cli/verbs/mkdir.rs b/src/cli/verbs/mkdir.rs index 77583082..6556e35a 100644 --- a/src/cli/verbs/mkdir.rs +++ b/src/cli/verbs/mkdir.rs @@ -6,6 +6,7 @@ use clap::Args; use crate::cli::img_at::ImageRef; use crate::cli::logging::log_stderr; +#[cfg(feature = "remote")] use crate::cli::parse::split_mac_path; use crate::cli::resolve::resolve_partition_rw; use crate::fs::filesystem::CreateDirectoryOptions; @@ -16,7 +17,8 @@ pub struct MkdirArgs { pub image: ImageRef, /// Directory path to create. The parent must exist (no `-p`-style - /// auto-creation in Phase B). + /// auto-creation in Phase B). A literal `/` in the new name is written + /// `\/`; on HFS / HFS+ a `:`-separated path also works. pub path: String, /// Accepted for consistency with `ls`/`get`/`rm`; `mkdir` always treats @@ -27,14 +29,14 @@ pub struct MkdirArgs { } pub fn run(args: MkdirArgs) -> Result<()> { - let (parent_path, name) = split_mac_path(&args.path)?; - if name.is_empty() { - bail!("directory path has no basename"); - } - // Remote: `rb-cli mkdir rb://host:port/img@N /NEWDIR` — stage + apply. + // Remote addressing is slash-only (the daemon resolves the path itself). #[cfg(feature = "remote")] if let Some(rref) = crate::remote::RemoteRef::parse(&args.image.path.to_string_lossy()) { + let (parent_path, name) = split_mac_path(&args.path)?; + if name.is_empty() { + bail!("directory path has no basename"); + } let mut session = crate::remote::RemoteSession::connect(&rref.addr())?; let sid = session.open_session(&rref.path, args.image.partition)?; session.stage_mkdir(sid, &parent_path, &name)?; @@ -53,9 +55,14 @@ pub fn run(args: MkdirArgs) -> Result<()> { ) .map_err(|e| anyhow!("opening filesystem for write: {e}"))?; - let parent = super::ls::resolve_path(&mut *fs, &parent_path)?; + // Resolve parent + leaf with the shared escape / colon grammar so a new + // directory whose name contains a literal `/` can be created. + let (parent, name) = super::ls::resolve_parent(&mut *fs, &args.path)?; + if name.is_empty() { + bail!("directory path has no basename"); + } if !parent.is_directory() { - bail!("parent is not a directory: {parent_path}"); + bail!("parent is not a directory: {}", args.path); } // Early duplicate check so the user-facing message names the path. diff --git a/src/cli/verbs/new.rs b/src/cli/verbs/new.rs index e3528d6f..49c7e987 100644 --- a/src/cli/verbs/new.rs +++ b/src/cli/verbs/new.rs @@ -15,6 +15,10 @@ use crate::fs::ntfs_format::{create_ntfs, NtfsFormatParams, NtfsGeometry}; pub enum FsKind { /// Classic HFS (Mac OS Standard). Hfs, + /// HFS+ / HFSX (Mac OS Extended). A bare single volume, no partition + /// map. Use --case-sensitive for HFSX and --min-catalog to floor the + /// catalog B-tree size. + Hfsplus, /// BasiliskII HFV — a flat classic-HFS volume, no partition table. /// Same on-disk bytes as `hfs`, but capped at 2047 MB and with the /// allocation block size auto-floored so the result is mountable by @@ -36,7 +40,7 @@ pub struct NewArgs { /// Image file to create. Overwritten if it already exists. pub image: PathBuf, - /// Filesystem to format. One of: hfs, hfv, fat, efs, affs, ntfs. + /// Filesystem to format. One of: hfs, hfsplus, hfv, fat, efs, affs, ntfs. #[arg(long, value_enum)] pub fs: FsKind, @@ -70,6 +74,18 @@ pub struct NewArgs { #[arg(long = "extents-size")] pub extents_size: Option, + /// HFS+ only: format a case-sensitive (HFSX) volume instead of the + /// default case-insensitive HFS+. Ignored for other filesystems. + #[arg(long = "case-sensitive")] + pub case_sensitive: bool, + + /// HFS+ only: minimum catalog B-tree size in bytes (a floor, rounded up + /// to whole 4096-byte nodes). Set this *small* to make the catalog easy + /// to outgrow and exercise the fork grow-on-full path. Ignored for other + /// filesystems. + #[arg(long = "min-catalog")] + pub min_catalog: Option, + /// AFFS variant byte (0=OFS, 1=FFS, 2=OFS+intl, 3=FFS+intl, /// 4=OFS+dircache, 5=FFS+dircache). Defaults to 1 (FFS). #[arg(long = "affs-variant", default_value = "1")] @@ -113,6 +129,9 @@ pub fn run(args: NewArgs) -> Result<()> { if (cluster_bytes.is_some() || args.sector_size.is_some()) && args.fs != FsKind::Ntfs { anyhow::bail!("--cluster-size / --sector-size are only valid with --fs ntfs"); } + if (args.case_sensitive || args.min_catalog.is_some()) && args.fs != FsKind::Hfsplus { + anyhow::bail!("--case-sensitive / --min-catalog are only valid with --fs hfsplus"); + } match args.fs { FsKind::Hfs => { let catalog_bytes = args @@ -136,6 +155,22 @@ pub fn run(args: NewArgs) -> Result<()> { extents_bytes, ) } + FsKind::Hfsplus => { + let min_catalog = args + .min_catalog + .as_deref() + .map(|s| parse_size(s).context("parsing --min-catalog")) + .transpose()? + .map(|v| v.min(u32::MAX as u64) as u32); + crate::cli::api::hfs::cmd_new_hfsplus_sized( + args.image, + &args.size, + &args.name, + args.block_size, + args.case_sensitive, + min_catalog, + ) + } FsKind::Hfv => { // A flat HFV is just a blank classic-HFS volume with the BasiliskII // limits enforced: <= 2047 MB and a block size that keeps diff --git a/src/cli/verbs/put.rs b/src/cli/verbs/put.rs index 5429a17f..ff4df5c0 100644 --- a/src/cli/verbs/put.rs +++ b/src/cli/verbs/put.rs @@ -22,7 +22,9 @@ use std::path::PathBuf; use crate::cli::img_at::ImageRef; use crate::cli::logging::log_stderr; -use crate::cli::parse::{split_mac_path, ZeroReader}; +#[cfg(feature = "remote")] +use crate::cli::parse::split_mac_path; +use crate::cli::parse::ZeroReader; use crate::cli::resolve::{resolve_partition_rw, resolve_partition_rw_forced, FsDispatchOverride}; use crate::fs::filesystem::CreateFileOptions; @@ -34,7 +36,9 @@ pub struct PutArgs { /// Host file to copy. Required when not using `--zero` or `--boot`. pub host_file: Option, - /// Destination path inside the filesystem (cp-like positional). + /// Destination path inside the filesystem (cp-like positional). A literal + /// `/` in the name is written `\/`; on HFS / HFS+ a `:`-separated path also + /// works (so `/` is plain data). pub dst: Option, /// Accepted for consistency with `ls`/`get`/`rm`; `put` always treats the @@ -121,15 +125,15 @@ pub fn run(args: PutArgs) -> Result<()> { (Some(_), Some(_)) => unreachable!("clap conflicts_with prevents both"), }; - let (parent_path, name) = split_mac_path(&dst)?; - if name.is_empty() { - bail!("destination path has no filename"); - } - // Remote destination: `rb-cli put rb://host:port/img@N HOST /DEST`. Upload - // the host file into the daemon's staging area and apply it. + // the host file into the daemon's staging area and apply it. Remote + // addressing is slash-only (the daemon resolves the path itself). #[cfg(feature = "remote")] if let Some(rref) = crate::remote::RemoteRef::parse(&args.image.path.to_string_lossy()) { + let (parent_path, name) = split_mac_path(&dst)?; + if name.is_empty() { + bail!("destination path has no filename"); + } return remote_put( &rref, args.image.partition, @@ -158,9 +162,14 @@ pub fn run(args: PutArgs) -> Result<()> { ) .map_err(|e| anyhow!("opening filesystem for write: {e}"))?; - let parent = super::ls::resolve_path(&mut *fs, &parent_path)?; + // Resolve parent + leaf with the shared escape / colon grammar so a file + // whose name contains a literal `/` can be written. + let (parent, name) = super::ls::resolve_parent(&mut *fs, &dst)?; + if name.is_empty() { + bail!("destination path has no filename"); + } if !parent.is_directory() { - bail!("parent is not a directory: {parent_path}"); + bail!("parent is not a directory: {dst}"); } // Duplicate check so we can honor --force consistently. diff --git a/src/cli/verbs/put_macbinary.rs b/src/cli/verbs/put_macbinary.rs index 762065dd..73dafc59 100644 --- a/src/cli/verbs/put_macbinary.rs +++ b/src/cli/verbs/put_macbinary.rs @@ -18,7 +18,6 @@ use std::path::PathBuf; use crate::cli::img_at::ImageRef; use crate::cli::logging::log_stderr; -use crate::cli::parse::split_mac_path; use crate::cli::resolve::resolve_partition_rw; use crate::fs::filesystem::{CreateFileOptions, ResourceForkSource}; @@ -73,18 +72,11 @@ pub fn run(args: PutMacBinaryArgs) -> Result<()> { ) .map_err(|e| anyhow!("opening filesystem for write: {e}"))?; - let (parent_path, _) = split_mac_path(&format!( - "{}/x", - args.dst_dir.trim_end_matches('/').trim_start_matches('/'), - ))?; - let parent_path = if parent_path == "/" { - "/".to_string() - } else { - parent_path - }; - let parent = super::ls::resolve_path(&mut *fs, &parent_path)?; + // `--dst-dir` is the destination *directory* (the filename comes from the + // MacBinary header), resolved with the shared escape / colon grammar. + let parent = super::ls::resolve_path(&mut *fs, &args.dst_dir)?; if !parent.is_directory() { - bail!("parent is not a directory: {parent_path}"); + bail!("destination is not a directory: {}", args.dst_dir); } let existing = fs @@ -96,7 +88,7 @@ pub fn run(args: PutMacBinaryArgs) -> Result<()> { if !args.force { bail!( "{}/{} already exists (pass --force to overwrite)", - parent_path.trim_end_matches('/'), + args.dst_dir.trim_end_matches('/'), target_name ); } diff --git a/src/cli/verbs/rm.rs b/src/cli/verbs/rm.rs index 910a58b6..73110932 100644 --- a/src/cli/verbs/rm.rs +++ b/src/cli/verbs/rm.rs @@ -7,7 +7,6 @@ use clap::Args; use crate::cli::glob::{collect_matches, compile_patterns}; use crate::cli::img_at::ImageRef; use crate::cli::logging::{log_stderr, out_stdout}; -use crate::cli::parse::split_mac_path; use crate::cli::resolve::{resolve_partition_rw_forced, FsDispatchOverride}; #[derive(Debug, Args)] @@ -18,7 +17,8 @@ pub struct RmArgs { /// Path or glob pattern inside the filesystem. Patterns containing /// `*`, `?`, `[`, or `{` walk the volume and delete every match. /// Pass `--literal` to delete a single path verbatim when its name - /// contains those characters. + /// contains those characters. A literal `/` in a name is written `\/` + /// (or use a `:`-separated path on HFS / HFS+, which also forces literal). pub path: String, /// Recursively delete directories (matches will include directories @@ -76,9 +76,10 @@ pub fn run(args: RmArgs) -> Result<()> { _ => true, }; - // `--literal` forces the exact single-path delete even for names that - // contain glob metacharacters. - if !args.literal && (has_glob_chars(&args.path) || !args.exclude.is_empty()) { + // `--literal` (or a colon-grammar path on HFS / HFS+) forces the exact + // single-path delete even for names that contain glob metacharacters. + let colon = super::ls::colon_mode(&*fs, &args.path); + if !args.literal && !colon && (has_glob_chars(&args.path) || !args.exclude.is_empty()) { // Glob path — collect everything, sort deepest-first so we delete // children before parents, then apply. let includes = compile_patterns(&args.path, case_insensitive)?; @@ -119,12 +120,12 @@ pub fn run(args: RmArgs) -> Result<()> { return Ok(()); } - // Literal-path single delete. - let (parent_path, name) = split_mac_path(&args.path)?; + // Literal-path single delete. Resolve parent + leaf with the shared escape + // / colon grammar so a name containing a literal `/` is addressable. + let (parent, name) = super::ls::resolve_parent(&mut *fs, &args.path)?; if name.is_empty() { bail!("path has no basename"); } - let parent = super::ls::resolve_path(&mut *fs, &parent_path)?; let children = fs .list_directory(&parent) .map_err(|e| anyhow!("list_directory: {e}"))?; diff --git a/src/fs/filesystem.rs b/src/fs/filesystem.rs index b7c79a6c..29e0f406 100644 --- a/src/fs/filesystem.rs +++ b/src/fs/filesystem.rs @@ -154,6 +154,20 @@ pub trait Filesystem: Send { } Ok(()) } + + /// True when this filesystem reserves `:` as its native path separator + /// (classic Mac HFS / HFS+), so `:` can never appear in a catalog name. + /// + /// When this is `true`, rb-cli accepts an in-image path written with `:` + /// separators (e.g. `:System Folder:Apple Menu Items`), which lets a literal + /// `/` be an ordinary filename byte — the only way to address a name like + /// `Oxyd b/w` unambiguously, since classic-Mac volumes allow `/` in names. + /// + /// Default `false`: `/` is the only separator, and a literal `/` inside a + /// component must be written `\/` (with `\\` for a literal backslash). + fn uses_colon_paths(&self) -> bool { + false + } } /// Aggregate fragmentation counts for a volume's user data forks. diff --git a/src/fs/hfs.rs b/src/fs/hfs.rs index de0e77f0..0ff0286f 100644 --- a/src/fs/hfs.rs +++ b/src/fs/hfs.rs @@ -1832,14 +1832,17 @@ impl HfsFilesystem { rec[8..12].copy_from_slice(creator_code); // filFlNum at offset 20 BigEndian::write_u32(&mut rec[20..24], file_id); - // filStBlk at offset 24 (first alloc block of data fork) - BigEndian::write_u16(&mut rec[24..26], data_start); + // filStBlk at offset 24 stays 0: it's the File Manager's in-memory + // "first allocation block" cache, NOT persisted on disk — the real + // fork location lives in filExtRec (offset 74). `fsck_hfs` flags a + // non-zero on-disk filStBlk as "Reserved fields in the catalog record + // have incorrect data" (E_CatalogFlagsNotZero). See `data_start` used + // for the extent record below. // filLgLen at offset 26 (data fork logical size) BigEndian::write_u32(&mut rec[26..30], data_size); // filPyLen at offset 30 (data fork physical size) BigEndian::write_u32(&mut rec[30..34], data_blocks as u32 * block_size); - // filRStBlk at offset 34 - BigEndian::write_u16(&mut rec[34..36], rsrc_start); + // filRStBlk at offset 34 stays 0 for the same reason as filStBlk. // filRLgLen at offset 36 (rsrc fork logical size) BigEndian::write_u32(&mut rec[36..40], rsrc_size); // filRPyLen at offset 40 (rsrc fork physical size) @@ -2744,6 +2747,10 @@ impl Filesystem for HfsFilesystem { validate_hfs_create_name(name).map(|_| ()) } + fn uses_colon_paths(&self) -> bool { + true + } + fn total_size(&self) -> u64 { self.mdb.total_blocks as u64 * self.mdb.block_size as u64 } @@ -3239,12 +3246,12 @@ impl EditableFilesystem for HfsFilesystem { )); } - // Update rsrc fork fields in file record: - // filRStBlk at offset 34 - BigEndian::write_u16( - &mut self.catalog_data[frec_start + 34..frec_start + 36], - rsrc_start, - ); + // Update rsrc fork fields in file record. filRStBlk (offset 34) + // stays 0 — it's the File Manager's in-memory cache, not persisted; + // a non-zero on-disk value makes fsck_hfs report "Reserved fields + // in the catalog record have incorrect data". The real location is + // the rsrc extent record at offset 86 (written below). + self.catalog_data[frec_start + 34..frec_start + 36].fill(0); // filRLgLen at offset 36 BigEndian::write_u32( &mut self.catalog_data[frec_start + 36..frec_start + 40], @@ -4380,6 +4387,70 @@ mod tests { assert_eq!(&read_back, test_data); } + /// `filStBlk` (offset 24) and `filRStBlk` (offset 34) of an HFS file record + /// are the File Manager's in-memory "first allocation block" cache and MUST + /// be zero on disk — the real fork location lives in the extent record at + /// offset 74 / 86. A non-zero on-disk value makes Apple's `fsck_hfs` (incl. + /// Mac OS X 10.4 Tiger, which fully supports HFS Standard) report + /// "Reserved fields in the catalog record have incorrect data" + /// (E_CatalogFlagsNotZero). Validated fsck_hfs-clean + mountable on Tiger. + #[test] + fn test_hfs_file_record_first_alloc_block_fields_are_zero() { + use super::super::filesystem::ResourceForkSource; + use super::hfs_common::walk_leaf_records; + + let img = make_editable_hfs_image(); + let mut fs = HfsFilesystem::open(Cursor::new(img), 0).unwrap(); + let root = fs.root().unwrap(); + + // A file with BOTH a data fork and a resource fork, so both filStBlk + // and filRStBlk are exercised. + let data = b"data fork contents that span a block"; + let options = CreateFileOptions { + resource_fork: Some(ResourceForkSource::Data(b"resource fork bytes".to_vec())), + ..Default::default() + }; + fs.create_file( + &root, + "forked.bin", + &mut Cursor::new(data.as_slice()), + data.len() as u64, + &options, + ) + .unwrap(); + fs.sync_metadata().unwrap(); + + let node_size = BigEndian::read_u16(&fs.catalog_data[32..34]) as usize; + let first_leaf = BigEndian::read_u32(&fs.catalog_data[24..28]); + let found = walk_leaf_records::( + &fs.catalog_data, + first_leaf, + node_size, + |_n, _i, _off, rec| { + let klen = rec[0] as usize; + let mut ds = 1 + klen; + if !ds.is_multiple_of(2) { + ds += 1; + } + let body = &rec[ds..]; + if body.first() != Some(&2) { + return None; // not a file record + } + let fil_st_blk = BigEndian::read_u16(&body[24..26]); + let fil_rst_blk = BigEndian::read_u16(&body[34..36]); + let data_ext_start = BigEndian::read_u16(&body[74..76]); + let rsrc_ext_start = BigEndian::read_u16(&body[86..88]); + assert_eq!(fil_st_blk, 0, "filStBlk (offset 24) must be 0 on disk"); + assert_eq!(fil_rst_blk, 0, "filRStBlk (offset 34) must be 0 on disk"); + // The real fork locations are still recorded in the extents. + assert_ne!(data_ext_start, 0, "data fork extent should be allocated"); + assert_ne!(rsrc_ext_start, 0, "rsrc fork extent should be allocated"); + Some(true) + }, + ); + assert_eq!(found, Some(true), "file record not found in catalog"); + } + /// Catalog dates surface on `FileEntry::mac_dates` (and the modify value on /// `modified`), and an in-place `set_dates` round-trips through sync + /// reopen. Exercises the M4 read path (catalog decode) plus the write path diff --git a/src/fs/hfs_common.rs b/src/fs/hfs_common.rs index daa5d7cb..856cf804 100644 --- a/src/fs/hfs_common.rs +++ b/src/fs/hfs_common.rs @@ -2128,13 +2128,36 @@ pub fn btree_insert_full( where F: Fn(&[u8], &[u8]) -> Ordering, { - let header = BTreeHeader::read(data); + let mut header = BTreeHeader::read(data); let node_size = header.node_size as usize; if node_size == 0 { return Err(super::filesystem::FilesystemError::InvalidData( "B-tree has zero node_size".into(), )); } + + // Bootstrap an empty (depth-0) tree: a fresh extents-overflow / attributes + // B-tree carries only its header node until the first record arrives + // (matching Apple's newfs_hfs, which lays out empty trees as + // depth=0/root=0 with no leaf). Allocate and initialise the root leaf so + // the normal insert path below has somewhere to land. + if header.depth == 0 || header.root_node == 0 { + let leaf = btree_alloc_node(data, node_size, header.total_nodes)?; + let off = leaf as usize * node_size; + let node = &mut data[off..off + node_size]; + node.fill(0); + node[8] = 0xFF; // kind = -1 (leaf) + node[9] = 1; // height = 1 + BigEndian::write_u16(&mut node[10..12], 0); // numRecords = 0 + BigEndian::write_u16(&mut node[node_size - 2..node_size], 14); // free-space offset + header.depth = 1; + header.root_node = leaf; + header.first_leaf_node = leaf; + header.last_leaf_node = leaf; + header.free_nodes = header.free_nodes.saturating_sub(1); + header.write(data); + } + let (leaf_idx, parent_chain) = btree_find_insert_leaf(data, &header, key_record, cmp); let off = leaf_idx as usize * node_size; @@ -2168,6 +2191,24 @@ where } } } + // A single-record insert into a previously-valid leaf splits at + // most 2-way (one new leaf), and the separator can cascade up, + // splitting one node per level plus a possible new root — at most + // `depth + 1` new nodes total. If the tree can't cover that exact + // worst case, bail *before* touching anything (the + // `btree_insert_record` above failed without mutating the full leaf, + // and a failed rotation leaves the tree unchanged), so a caller that + // grows the fork and retries can't end up with a half-applied split + // — which would duplicate the record. This makes DiskFull from here + // a clean no-op (see the grow-on-full retry in + // `HfsPlusFilesystem::insert_catalog_record` et al.). The bound is + // exact, not padded, so classic HFS (which has no grow path) still + // fills its pre-sized catalog right up to genuine capacity. + if h.free_nodes < h.depth as u32 + 1 { + return Err(super::filesystem::FilesystemError::DiskFull( + "no free B-tree nodes".into(), + )); + } let splits = btree_split_leaf_with_insert( data, node_size, leaf_idx, &mut h, key_record, kf, cmp, )?; diff --git a/src/fs/hfsplus.rs b/src/fs/hfsplus.rs index 56bb39e0..739a35a4 100644 --- a/src/fs/hfsplus.rs +++ b/src/fs/hfsplus.rs @@ -8,8 +8,8 @@ use super::filesystem::{ CreateDirectoryOptions, CreateFileOptions, EditableFilesystem, Filesystem, FilesystemError, }; use super::hfs_common::{ - self, bitmap_clear_bit_be, bitmap_collect_clear_runs_be, bitmap_set_bit_be, btree_free_node, - btree_remove_record, BTreeHeader, BTreeKeyFormat, + self, bitmap_clear_bit_be, bitmap_collect_clear_runs_be, bitmap_set_bit_be, bitmap_test_bit_be, + btree_free_node, btree_remove_record, BTreeHeader, BTreeKeyFormat, }; use super::CompactResult; @@ -17,12 +17,43 @@ const HFS_PLUS_SIGNATURE: u16 = 0x482B; const HFSX_SIGNATURE: u16 = 0x4858; /// HFS+ reserved CNIDs (Inside Macintosh: Files / TN1150). -#[allow(dead_code)] const HFSPLUS_EXTENTS_FILE_ID: u32 = 3; const HFSPLUS_CATALOG_FILE_ID: u32 = 4; const HFSPLUS_ALLOCATION_FILE_ID: u32 = 6; const HFSPLUS_ATTRIBUTES_FILE_ID: u32 = 8; +/// Which special-file B-tree a grow-on-full operation targets. Each is +/// backed by a `ForkData` in the volume header and an in-memory tree buffer +/// (`catalog_data` / `extents_overflow_data` / `attributes_data`). +/// See `grow_btree_fork` and `docs/todo_hfsplus_fork_growth.md` §4b. +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +enum BTreeFork { + Catalog, + Extents, + Attributes, +} + +impl BTreeFork { + /// Reserved CNID of the special file backing this B-tree — the key under + /// which its own overflow extents are recorded in the extents-overflow + /// B-tree (Phase B). + fn file_id(self) -> u32 { + match self { + BTreeFork::Catalog => HFSPLUS_CATALOG_FILE_ID, + BTreeFork::Extents => HFSPLUS_EXTENTS_FILE_ID, + BTreeFork::Attributes => HFSPLUS_ATTRIBUTES_FILE_ID, + } + } + + fn label(self) -> &'static str { + match self { + BTreeFork::Catalog => "catalog", + BTreeFork::Extents => "extents-overflow", + BTreeFork::Attributes => "attributes", + } + } +} + /// Volume attribute bit (`vh.attributes`) set when the volume carries a /// journal. We refuse to enter edit mode on these volumes until journal /// replay is implemented — see Step 4 of `docs/hfsplus_enhancements.md`. @@ -1812,15 +1843,33 @@ impl HfsPlusFilesystem { /// index keys. This is the same path classic HFS's `insert_catalog_record` /// uses; the previous hand-rolled split dance here packed less densely because /// it never rotated. - fn insert_catalog_record(&mut self, key_record: &[u8]) -> Result<(), FilesystemError> { + fn insert_catalog_record(&mut self, key_record: &[u8]) -> Result<(), FilesystemError> + where + R: std::io::Write, + { let cs = self.case_sensitive(); - let cmp = |a: &[u8], b: &[u8]| Self::catalog_compare(a, b, cs); - hfs_common::btree_insert_full( - &mut self.catalog_data, - key_record, - &BTreeKeyFormat::HFSPLUS_CATALOG, - &cmp, - )?; + // Insert; if the catalog runs out of free B-tree nodes, grow the fork + // and retry once (docs/todo_hfsplus_fork_growth.md §4b). `btree_insert_full` + // returns a *clean* DiskFull (no partial split) when it can't cover the + // worst-case cascade, so the retry can't duplicate the record. A second + // DiskFull means the volume itself is full — propagate it. + let mut grown = false; + loop { + let cmp = |a: &[u8], b: &[u8]| Self::catalog_compare(a, b, cs); + match hfs_common::btree_insert_full( + &mut self.catalog_data, + key_record, + &BTreeKeyFormat::HFSPLUS_CATALOG, + &cmp, + ) { + Ok(()) => break, + Err(FilesystemError::DiskFull(_)) if !grown => { + self.grow_btree_fork(BTreeFork::Catalog)?; + grown = true; + } + Err(e) => return Err(e), + } + } self.catalog_header = BTreeHeaderRecord::parse(&self.catalog_data[14..14 + 106]); Ok(()) } @@ -1929,22 +1978,38 @@ impl HfsPlusFilesystem { /// forced the classic 1-byte/0x25 shape, which is why this path was pinned to /// a single leaf level (docs/hfsplus_btree_growth_plan.md §4a/P3). fn insert_extents_overflow_record(&mut self, key_record: &[u8]) -> Result<(), FilesystemError> { - let data = self.extents_overflow_data.as_mut().ok_or_else(|| { - FilesystemError::Unsupported( - "volume has no extents-overflow B-tree — fragmented files cannot be created".into(), - ) - })?; - if BTreeHeader::read(data).node_size == 0 { - return Err(FilesystemError::InvalidData( - "extents-overflow B-tree has zero node_size".into(), - )); + { + let data = self.extents_overflow_data.as_ref().ok_or_else(|| { + FilesystemError::Unsupported( + "volume has no extents-overflow B-tree — fragmented files cannot be created" + .into(), + ) + })?; + if BTreeHeader::read(data).node_size == 0 { + return Err(FilesystemError::InvalidData( + "extents-overflow B-tree has zero node_size".into(), + )); + } + } + // Insert; grow the extents-overflow fork and retry once on a clean + // node-exhaustion DiskFull (see `insert_catalog_record`). + let mut grown = false; + loop { + let data = self.extents_overflow_data.as_mut().unwrap(); + match hfs_common::btree_insert_full( + data, + key_record, + &BTreeKeyFormat::HFSPLUS_EXTENTS, + &Self::extents_compare, + ) { + Ok(()) => return Ok(()), + Err(FilesystemError::DiskFull(_)) if !grown => { + self.grow_btree_fork(BTreeFork::Extents)?; + grown = true; + } + Err(e) => return Err(e), + } } - hfs_common::btree_insert_full( - data, - key_record, - &BTreeKeyFormat::HFSPLUS_EXTENTS, - &Self::extents_compare, - ) } /// Remove every overflow record belonging to `(file_id, fork_type)`. @@ -2101,12 +2166,28 @@ impl HfsPlusFilesystem { // The attributes B-tree, like the catalog, uses 2-byte big keys and // variable-length index separators, so it threads HFSPLUS_ATTRIBUTES // through the shared insert (which also rotates before splitting). - hfs_common::btree_insert_full( - data, - key_record, - &BTreeKeyFormat::HFSPLUS_ATTRIBUTES, - &Self::attr_compare, - ) + // Insert; grow the attributes fork and retry once on a clean + // node-exhaustion DiskFull (see `insert_catalog_record`). + let mut grown = false; + loop { + let data = self + .attributes_data + .as_mut() + .expect("ensure_attributes_loaded populated the buffer"); + match hfs_common::btree_insert_full( + data, + key_record, + &BTreeKeyFormat::HFSPLUS_ATTRIBUTES, + &Self::attr_compare, + ) { + Ok(()) => return Ok(()), + Err(FilesystemError::DiskFull(_)) if !grown => { + self.grow_btree_fork(BTreeFork::Attributes)?; + grown = true; + } + Err(e) => return Err(e), + } + } } /// Remove every attribute record matching `(cnid, name)`. Returns the @@ -2242,25 +2323,33 @@ impl HfsPlusFilesystem { if self.vh.attributes_file.logical_size == 0 { return Ok(()); } - let Some(ref data) = self.attributes_data else { + if self.attributes_data.is_none() { return Ok(()); - }; - write_fork_data( + } + write_fork_data_with_overflow( &mut self.reader, self.partition_offset, self.vh.block_size, &self.vh.attributes_file, - data, + HFSPLUS_ATTRIBUTES_FILE_ID, + 0, + self.extents_overflow_data.as_deref(), + self.attributes_data.as_deref().unwrap(), ) } - /// Write the catalog B-tree data back to disk through the catalog_file fork extents. + /// Write the catalog B-tree data back to disk through the catalog_file + /// fork extents — including any overflow extents past the 8 inline slots + /// that §4b growth may have added. fn write_catalog(&mut self) -> Result<(), FilesystemError> { - write_fork_data( + write_fork_data_with_overflow( &mut self.reader, self.partition_offset, self.vh.block_size, &self.vh.catalog_file, + HFSPLUS_CATALOG_FILE_ID, + 0, + self.extents_overflow_data.as_deref(), &self.catalog_data, ) } @@ -2393,6 +2482,290 @@ impl HfsPlusFilesystem { } } + // --- B-tree fork grow-on-full (docs/todo_hfsplus_fork_growth.md §4b) --- + + /// Mutable handle to the in-memory tree buffer for `kind`. Errors if the + /// volume has no such B-tree (e.g. growing the extents-overflow tree on a + /// volume that never built one). + fn tree_buf_mut(&mut self, kind: BTreeFork) -> Result<&mut Vec, FilesystemError> { + match kind { + BTreeFork::Catalog => Ok(&mut self.catalog_data), + BTreeFork::Extents => self.extents_overflow_data.as_mut().ok_or_else(|| { + FilesystemError::Unsupported("volume has no extents-overflow B-tree".into()) + }), + BTreeFork::Attributes => self.attributes_data.as_mut().ok_or_else(|| { + FilesystemError::Unsupported("volume has no attributes B-tree".into()) + }), + } + } + + /// The volume-header `ForkData` backing `kind`. + fn fork_mut(&mut self, kind: BTreeFork) -> &mut ForkData { + match kind { + BTreeFork::Catalog => &mut self.vh.catalog_file, + BTreeFork::Extents => &mut self.vh.extents_file, + BTreeFork::Attributes => &mut self.vh.attributes_file, + } + } + + /// First volume block immediately *after* the fork's last inline extent — + /// the candidate start for a contiguous tail growth. Returns 0 (never a + /// valid data block) when the fork has no extents. Overflow extents are + /// intentionally not consulted: if the true tail lies in an overflow + /// extent, `allocate_contiguous_after` simply finds the candidate block + /// busy and the caller falls back to the general allocator — correct, just + /// not optimally contiguous. + fn fork_last_inline_block(&self, kind: BTreeFork) -> u32 { + let fork = match kind { + BTreeFork::Catalog => &self.vh.catalog_file, + BTreeFork::Extents => &self.vh.extents_file, + BTreeFork::Attributes => &self.vh.attributes_file, + }; + let mut last = 0u32; + for e in &fork.extents { + if e.block_count == 0 { + break; + } + last = e.start_block + e.block_count; + } + last + } + + /// Try to claim `blocks` free volume blocks *immediately following* + /// `after_block`, so the fork's last extent simply gets longer (extent + /// count unchanged — the fragmentation-free common case). Returns the new + /// extent on success, or `None` if `after_block` is 0, the run would run + /// past the volume, or any block in it is already allocated. Updates the + /// bitmap, `free_blocks`, and `next_allocation` exactly like + /// `allocate_extents`. + fn allocate_contiguous_after( + &mut self, + after_block: u32, + blocks: u32, + ) -> Option { + if blocks == 0 || after_block == 0 { + return None; + } + self.ensure_bitmap().ok()?; + if after_block.checked_add(blocks)? > self.vh.total_blocks { + return None; + } + if self.vh.free_blocks < blocks { + return None; + } + let bitmap = self.bitmap.as_mut()?; + for i in 0..blocks { + if bitmap_test_bit_be(bitmap, after_block + i) { + return None; // tail is not free — caller falls back + } + } + for i in 0..blocks { + bitmap_set_bit_be(bitmap, after_block + i); + } + self.vh.free_blocks -= blocks; + self.vh.next_allocation = (after_block + blocks).max(self.vh.next_allocation); + Some(ExtentDescriptor { + start_block: after_block, + block_count: blocks, + }) + } + + /// Attach freshly-allocated `new_extents` to the fork backing `kind`, + /// merging the first one into the fork's last inline extent when they are + /// physically contiguous (so a contiguous tail grow keeps the extent count + /// unchanged), dropping the rest into free inline slots, and spilling any + /// remainder past the 8 inline slots into the extents-overflow B-tree + /// (Phase B). Bumps the fork's `total_blocks` / `logical_size`. + fn attach_extents_to_fork( + &mut self, + kind: BTreeFork, + new_extents: &[ExtentDescriptor], + ) -> Result<(), FilesystemError> { + let block_size = self.vh.block_size as u64; + let fork = self.fork_mut(kind); + let mut added_blocks = 0u32; + let mut overflow: Vec = Vec::new(); + for ext in new_extents { + if ext.block_count == 0 { + continue; + } + added_blocks += ext.block_count; + // Index of the last non-empty inline slot, if any. + let last_idx = (0..8) + .take_while(|&i| fork.extents[i].block_count != 0) + .last(); + let mut placed = false; + if let Some(i) = last_idx { + let le = fork.extents[i]; + if le.start_block + le.block_count == ext.start_block { + fork.extents[i].block_count += ext.block_count; + placed = true; + } + } + if !placed { + if let Some(i) = (0..8).find(|&i| fork.extents[i].block_count == 0) { + fork.extents[i] = *ext; + placed = true; + } + } + if !placed { + overflow.push(*ext); + } + } + fork.total_blocks += added_blocks; + fork.logical_size += added_blocks as u64 * block_size; + if !overflow.is_empty() { + self.spill_fork_extents_to_overflow(kind, &overflow)?; + } + Ok(()) + } + + /// Phase B: record fork `extents` that overflowed the 8 inline slots into + /// the extents-overflow B-tree, keyed by the special file's reserved CNID + /// (fork type 0). These append after the fork's existing inline + overflow + /// blocks, so their file-relative start is `inline_blocks + + /// existing_overflow_blocks`. The write side (`write_fork_data_with_overflow`) + /// and read side (`read_fork_with_overflow`) both consult these records. + fn spill_fork_extents_to_overflow( + &mut self, + kind: BTreeFork, + extents: &[ExtentDescriptor], + ) -> Result<(), FilesystemError> { + // The extents-overflow file cannot store its *own* overflow extents + // (it would have to contain a record describing where it lives) — HFS+ + // limits it to 8 inline extents. In practice it stays tiny, so this is + // a hard error if it ever happens, and the recursion guard that keeps + // a catalog/attributes spill (which grows the extents fork) from + // re-entering here for the extents fork itself. + if kind == BTreeFork::Extents { + return Err(FilesystemError::DiskFull( + "extents-overflow B-tree fork exceeded its 8 inline extents".into(), + )); + } + let file_id = kind.file_id(); + let fork = match kind { + BTreeFork::Catalog => &self.vh.catalog_file, + BTreeFork::Attributes => &self.vh.attributes_file, + BTreeFork::Extents => unreachable!(), + }; + let inline_blocks: u32 = fork.extents.iter().map(|e| e.block_count).sum(); + let existing_overflow: u32 = match self.extents_overflow_data.as_deref() { + Some(ed) => collect_hfsplus_overflow_extents(ed, file_id, 0, inline_blocks) + .iter() + .map(|e| e.block_count) + .sum(), + None => 0, + }; + let mut file_rel = inline_blocks + existing_overflow; + // HFS+ extents-overflow records hold up to 8 extent descriptors each. + for chunk in extents.chunks(8) { + let rec = Self::build_extents_overflow_record(file_id, 0, file_rel, chunk); + self.insert_extents_overflow_record(&rec)?; + file_rel += chunk.iter().map(|e| e.block_count).sum::(); + } + Ok(()) + } + + /// Grow the B-tree fork backing `kind` by at least one clump of nodes so a + /// subsequent `btree_insert_full` has free nodes to allocate. Allocates + /// volume blocks (preferring a contiguous tail extension), attaches them to + /// the fork, extends the in-memory tree buffer, and publishes the new nodes + /// to the tree header (`total_nodes` / `free_nodes`). The node-allocation + /// bitmap bits for the new nodes already read as 0 (free) inside the header + /// node's capacity; past that, a map node is appended to the bitmap chain + /// (§4b Phase C). + fn grow_btree_fork(&mut self, kind: BTreeFork) -> Result<(), FilesystemError> { + let block_size = self.vh.block_size; + + let (node_size, total_nodes) = { + let buf = self.tree_buf_mut(kind)?; + let h = BTreeHeader::read(buf); + (h.node_size as usize, h.total_nodes) + }; + if node_size == 0 || node_size < 256 { + return Err(FilesystemError::InvalidData(format!( + "{} B-tree has invalid node_size {node_size}", + kind.label() + ))); + } + let blocks_per_node = (node_size as u32 / block_size).max(1); + + // Grow by at least 8 nodes, rounded up to the fork's clump, to amortise + // the work and keep per-`put` workloads from re-entering grow on every + // insert (which would fragment the fork and defeat the contiguous-tail + // strategy). + let clump_nodes = (self.vh.data_clump_size as usize) + .div_ceil(node_size) + .max(1) as u32; + let grow_nodes = clump_nodes.max(8); + + // The header node's bitmap addresses (node_size - 256) * 8 node bits; + // each map node (BTNodeKind=2) chained off the header adds + // map_node_bitmap_bytes*8 more. When a grow would push the node count + // past the current addressable range, we must append one map node so + // the new node indices have bitmap bits (§4b Phase C). A single map + // node covers far more nodes than one grow adds, so at most one new map + // node is ever needed per grow. + let header_cap = ((node_size - 256) * 8) as u32; + let per_map = (hfs_common::map_node_bitmap_bytes(node_size) * 8) as u32; + let existing_map = { + let buf = self.tree_buf_mut(kind)?; + (hfs_common::btree_bitmap_segments(buf, node_size).len() as u32).saturating_sub(1) + }; + let current_capacity = header_cap + existing_map * per_map; + let need_map = total_nodes + grow_nodes > current_capacity; + if need_map && per_map == 0 { + return Err(FilesystemError::DiskFull(format!( + "{} B-tree node bitmap full and node size too small for map nodes", + kind.label() + ))); + } + let add_slots = grow_nodes + u32::from(need_map); + let grow_blocks = add_slots * blocks_per_node; + + // Allocate volume blocks, preferring a contiguous tail extension so the + // fork's last extent just gets longer. + let last_block = self.fork_last_inline_block(kind); + let new_extents = match self.allocate_contiguous_after(last_block, grow_blocks) { + Some(e) => vec![e], + None => self.allocate_extents(grow_blocks)?, + }; + + self.attach_extents_to_fork(kind, &new_extents)?; + + // Extend the in-memory tree buffer and publish the new nodes. The added + // bytes are zero, so the new node-bitmap bits read as free and + // `btree_alloc_node` will hand them out. + { + let buf = self.tree_buf_mut(kind)?; + buf.resize(buf.len() + add_slots as usize * node_size, 0); + + if need_map { + // The new map node takes the first new slot; the free data + // nodes follow it. Its own index is still inside the *existing* + // bitmap capacity (one grow adds far fewer nodes than the + // header/last-map-node segment can address), so we can mark it + // used after linking it into the chain. + let map_idx = total_nodes; + let tail = btree_map_chain_tail(buf, node_size); // 0 => link via header node 0 + hfs_common::init_map_node(buf, node_size, map_idx, tail, 0); + let linker_off = tail as usize * node_size; // node 0 is the header + BigEndian::write_u32(&mut buf[linker_off..linker_off + 4], map_idx); + hfs_common::btree_bitmap_set(buf, node_size, map_idx); + } + + let mut h = BTreeHeader::read(buf); + h.total_nodes += add_slots; + h.free_nodes += grow_nodes; // the map node is allocated, not free + h.write(buf); + } + self.vh.write_count = self.vh.write_count.wrapping_add(1); + if kind == BTreeFork::Catalog { + self.catalog_header = BTreeHeaderRecord::parse(&self.catalog_data[14..14 + 106]); + } + Ok(()) + } + /// Write file data to allocated blocks. Returns the ForkData describing /// the allocation. /// @@ -2797,6 +3170,10 @@ impl Filesystem for HfsPlusFilesystem { validate_hfsplus_create_name(name) } + fn uses_colon_paths(&self) -> bool { + true + } + fn total_size(&self) -> u64 { self.vh.total_blocks as u64 * self.vh.block_size as u64 } @@ -3369,6 +3746,90 @@ fn write_fork_data( Ok(()) } +/// Write a special-file B-tree fork that may have grown past its 8 inline +/// extents (§4b Phase B). Writes the inline extents first, then continues +/// into the fork's overflow extents pulled from the extents-overflow B-tree +/// — the write-side mirror of [`read_fork_with_overflow`]. Without this, a +/// grown catalog/attributes fork would be silently truncated on disk to its +/// first 8 extents even though it looks complete in memory. +#[allow(clippy::too_many_arguments)] +fn write_fork_data_with_overflow( + writer: &mut R, + partition_offset: u64, + block_size: u32, + fork: &ForkData, + file_id: u32, + fork_type: u8, + extents_overflow_data: Option<&[u8]>, + data: &[u8], +) -> Result<(), FilesystemError> { + let mut written = 0usize; + let write_extent = |writer: &mut R, ext: &ExtentDescriptor, written: &mut usize| { + if ext.block_count == 0 || *written >= data.len() { + return Ok(false); + } + let offset = partition_offset + ext.start_block as u64 * block_size as u64; + let extent_len = ext.block_count as u64 * block_size as u64; + let to_write = extent_len.min((data.len() - *written) as u64) as usize; + writer.seek(SeekFrom::Start(offset))?; + writer.write_all(&data[*written..*written + to_write])?; + *written += to_write; + Ok::(true) + }; + + for ext in &fork.extents { + if ext.is_empty() { + break; + } + if !write_extent(writer, ext, &mut written)? { + return Ok(()); + } + } + if written >= data.len() { + return Ok(()); + } + // Inline extents didn't cover the whole fork — continue into overflow. + let Some(ext_data) = extents_overflow_data else { + return Err(FilesystemError::InvalidData(format!( + "special file {file_id} fork {fork_type:#x}: {} bytes to write but only \ + inline extents and no extents-overflow B-tree", + data.len() + ))); + }; + let inline_blocks: u32 = fork.extents.iter().map(|e| e.block_count).sum(); + for ext in collect_hfsplus_overflow_extents(ext_data, file_id, fork_type, inline_blocks) { + if !write_extent(writer, &ext, &mut written)? { + break; + } + } + if written < data.len() { + return Err(FilesystemError::InvalidData(format!( + "special file {file_id} fork {fork_type:#x}: extents cover {written} of {} bytes", + data.len() + ))); + } + Ok(()) +} + +/// Index of the last node in a B-tree's map-node chain (`header.fLink` → +/// mapNode → … → 0), or 0 when the chain is empty (so the caller links the +/// first map node via the header node at index 0). Cycle-guarded. +fn btree_map_chain_tail(buf: &[u8], node_size: usize) -> u32 { + let mut next = BigEndian::read_u32(&buf[0..4]); // header.fLink + let mut last = 0u32; + let mut guard = 0; + while next != 0 && guard < 1_000_000 { + last = next; + let off = next as usize * node_size; + if off + 4 > buf.len() { + break; + } + next = BigEndian::read_u32(&buf[off..off + 4]); + guard += 1; + } + last +} + /// Decode a UTF-16BE byte slice to a String. fn decode_utf16be(data: &[u8]) -> String { let chars: Vec = data.chunks_exact(2).map(BigEndian::read_u16).collect(); @@ -5079,13 +5540,14 @@ fn build_blank_hfsplus_front( // // `to_nodes` rounds a byte budget up to whole nodes and clamps to // [4, header-bitmap capacity]. The blank's node-allocation bitmap lives - // entirely in the header node's record 2 (bytes 270..node_size-8 — see - // `write_blank_btree_header_node`), which addresses `(node_size - 278) * 8` + // entirely in the header node's record 2 (bytes 248..node_size-8 — see + // `write_blank_btree_header_node`), which addresses `(node_size - 256) * 8` // nodes without dedicated map nodes; cap there. Volumes whose 0.5% catalog - // would exceed this (~117 MiB at 4096, i.e. hundreds of GiB of volume) are + // would exceed this (~123 MiB at 4096, i.e. hundreds of GiB of volume) are // far larger than this tool targets, and the defrag/clone path — which does - // build map nodes — covers them. - let max_btree_nodes: u32 = ((node_size - 278) * 8) as u32; + // build map nodes — covers them. (The §4b live-grow path appends map nodes + // incrementally past this cap; see `grow_btree_fork`.) + let max_btree_nodes: u32 = ((node_size - 256) * 8) as u32; let to_nodes = |bytes: u64| -> u32 { bytes .div_ceil(node_size as u64) @@ -5119,19 +5581,36 @@ fn build_blank_hfsplus_front( let front_size = reserved_blocks as usize * block_size as usize; let mut front = vec![0u8; front_size]; + // The alternate volume header occupies the last 1024 bytes of the volume. + // Apple always marks the trailing allocation block(s) it touches as used + // (one block for block_size >= 1024, two for 512-byte blocks) and excludes + // them from the free-block count. Without this, fsck_hfs reports + // "Invalid volume free block count" / bitmap under-allocation. + let alt_vh_start_block: u32 = ((image_size as u64 - 1024) / bs) as u32; + let trailing_blocks: u32 = total_blocks - alt_vh_start_block; + // --- Allocation bitmap --- { let bitmap_off = bitmap_start as usize * block_size as usize; let bitmap = &mut front[bitmap_off..bitmap_off + (bitmap_bytes as usize).max(1)]; - for blk in 0..reserved_blocks { + let mut mark = |blk: u32| { // MSB-first big-endian: bit position (7 - blk%8) of byte (blk/8). let byte_idx = (blk / 8) as usize; let bit_pos = 7 - (blk % 8); bitmap[byte_idx] |= 1 << bit_pos; + }; + for blk in 0..reserved_blocks { + mark(blk); + } + for blk in alt_vh_start_block..total_blocks { + mark(blk); } } - // --- Extents-overflow B-tree (header + empty leaf) --- + // --- Extents-overflow B-tree (empty: header node only) --- + // An empty tree is depth-0 / root-0 with no leaf node, exactly as Apple's + // newfs_hfs lays it out (see `write_blank_btree_header_node`). The first + // fragmented-file insert bootstraps a root leaf via `btree_insert_full`. { let off = extents_start as usize * block_size as usize; write_blank_btree_header_node( @@ -5139,12 +5618,10 @@ fn build_blank_hfsplus_front( node_size, /* leaf_records= */ 0, /* total_nodes= */ extents_node_count, - /* free_nodes= */ extents_node_count - 2, // header + empty leaf used + /* free_nodes= */ extents_node_count - 1, // only the header node is used /* max_key_len= */ 10, /* key_compare_type= */ 0, // unused for extents ); - let leaf_off = off + node_size; - write_empty_leaf_node(&mut front[leaf_off..leaf_off + node_size]); } // --- Catalog B-tree (header + leaf with root + thread) --- @@ -5195,7 +5672,7 @@ fn build_blank_hfsplus_front( folder_count: 0, block_size, total_blocks, - free_blocks: total_blocks - reserved_blocks, + free_blocks: total_blocks - reserved_blocks - trailing_blocks, next_allocation: reserved_blocks, rsrc_clump_size: block_size, data_clump_size: block_size, @@ -5289,12 +5766,24 @@ pub(crate) fn write_blank_btree_header_node( BigEndian::write_u16(&mut node[10..12], 3); // 3 records // Record 0: BTHeaderRec at offset 14 (106 bytes). + // + // An *empty* tree (no leaf records, e.g. a fresh extents-overflow or + // attributes B-tree) is represented exactly as Apple's `newfs_hfs` does: + // treeDepth = 0, rootNode = 0, firstLeafNode = lastLeafNode = 0, and no + // leaf node allocated — only the header node is in use. A populated blank + // (the catalog, with its root folder + thread records) has a single + // root *leaf* at node 1. `fsck_hfs` rejects a depth-1 tree whose root is + // an empty leaf ("Invalid node structure"), which is why the empty case + // must collapse to depth 0. `btree_insert_full` bootstraps the first leaf + // on the initial insert into a depth-0 tree. + let empty = leaf_records == 0; + let (depth, root_first_last): (u16, u32) = if empty { (0, 0) } else { (1, 1) }; let hr = 14usize; - BigEndian::write_u16(&mut node[hr..hr + 2], 1); // depth = 1 - BigEndian::write_u32(&mut node[hr + 2..hr + 6], 1); // root = 1 + BigEndian::write_u16(&mut node[hr..hr + 2], depth); + BigEndian::write_u32(&mut node[hr + 2..hr + 6], root_first_last); // root BigEndian::write_u32(&mut node[hr + 6..hr + 10], leaf_records); - BigEndian::write_u32(&mut node[hr + 10..hr + 14], 1); // first leaf - BigEndian::write_u32(&mut node[hr + 14..hr + 18], 1); // last leaf + BigEndian::write_u32(&mut node[hr + 10..hr + 14], root_first_last); // first leaf + BigEndian::write_u32(&mut node[hr + 14..hr + 18], root_first_last); // last leaf BigEndian::write_u16(&mut node[hr + 18..hr + 20], node_size as u16); BigEndian::write_u16(&mut node[hr + 20..hr + 22], max_key_len); BigEndian::write_u32(&mut node[hr + 22..hr + 26], total_nodes); @@ -5303,11 +5792,18 @@ pub(crate) fn write_blank_btree_header_node( node[hr + 36] = 0; // kBTHFSTreeType node[hr + 37] = key_compare_type; - // Record 1: 128-byte user data at offset 142 (after 14+106=120, but - // 128-aligned for clarity → start at 142 to match make_editable_hfsplus_image). - let user_off: u16 = 142; - let bitmap_off: u16 = 270; - let free_off: u16 = (node_size as u16).saturating_sub(8); // some headroom + // Canonical TN1150 B-tree header node layout. The BTHeaderRec (record 0) + // is *exactly* 106 bytes — Apple's `fsck_hfs` rejects any other length + // ("Invalid BTH length"). So record 1 (the 128-byte reserved user-data + // record) starts at 14 + 106 = 120, and record 2 (the node-allocation + // bitmap) starts at 120 + 128 = 248, filling the node to the offset + // table. The bitmap therefore addresses (node_size - 256) * 8 nodes, + // which is exactly what `hfs_common::map_nodes_required` assumes — so the + // header-vs-map-node boundary is consistent (see the discrepancy noted in + // docs/todo_hfsplus_fork_growth.md §5, resolved by using 248 here). + let user_off: u16 = 120; + let bitmap_off: u16 = 248; + let free_off: u16 = (node_size as u16).saturating_sub(8); // fills to offset table let ot_end = node_size; BigEndian::write_u16(&mut node[ot_end - 2..ot_end], 14); // record 0 @@ -5315,11 +5811,16 @@ pub(crate) fn write_blank_btree_header_node( BigEndian::write_u16(&mut node[ot_end - 6..ot_end - 4], bitmap_off); BigEndian::write_u16(&mut node[ot_end - 8..ot_end - 6], free_off); - // Mark nodes 0 and 1 allocated in the bitmap. - node[bitmap_off as usize] = 0b11000000; + // Mark allocated nodes in the bitmap: node 0 (header) always; node 1 + // (the root leaf) only for a populated tree. An empty tree's node 1 is + // free and zero-filled, matching Apple. + node[bitmap_off as usize] = if empty { 0b10000000 } else { 0b11000000 }; } -/// Write an empty leaf node (kind=-1, height=1, 0 records). +/// Write an empty leaf node (kind=-1, height=1, 0 records). Only used by test +/// fixtures now — the production blank/defrag builders represent an empty tree +/// as depth 0 with no leaf node (the first insert bootstraps the root leaf). +#[cfg(test)] pub(crate) fn write_empty_leaf_node(node: &mut [u8]) { node.fill(0); node[8] = 0xFF; // kind = -1 @@ -6352,9 +6853,11 @@ mod tests { // 32 MiB / 4096 = 8192 blocks. The catalog is now volume-scaled // (~0.5%, docs/hfsplus_btree_growth_plan.md §4c): 32 MiB / 200 = // 167,772 bytes -> 41 nodes @ 4096. Reserved = 1 (boot/VH) + 1 (bitmap) - // + 4 (extents, default) + 41 (catalog) = 47. Free = 8145. + // + 4 (extents, default) + 41 (catalog) = 47 at the front, plus the + // trailing allocation block holding the alternate volume header (Apple + // reserves it; required for an fsck_hfs-clean free-block count). assert_eq!(fs.vh.total_blocks, 8192); - assert_eq!(fs.vh.free_blocks, 8192 - 47); + assert_eq!(fs.vh.free_blocks, 8192 - 47 - 1); // Edit-mode prep should succeed (unmounted bit set, no journal). fs.prepare_for_edit().expect("blank must accept edit prep"); } @@ -8108,6 +8611,291 @@ mod tests { ); } + #[test] + fn test_hfsplus_catalog_grows_map_nodes_past_cap_phase_c() { + // §4b Phase C (docs/todo_hfsplus_fork_growth.md): once a B-tree fork + // grows past the header node's bitmap capacity ((node_size-256)*8 = + // 30,720 nodes @ 4096), a map node (BTNodeKind=2) must be appended to + // the node-allocation bitmap chain so the new node indices are + // addressable. Pre-size the catalog just under the cap, then grow a few + // times to cross it — exercising the map-node append without needing + // the millions of records a from-scratch fill would take. + use super::hfs_common::{btree_bitmap_segments, BTreeHeader}; + type Fs = HfsPlusFilesystem>>; + let node_size = 4096usize; + let header_cap = ((node_size - 256) * 8) as u32; // 30,720 + + // Catalog pinned to ~16 nodes below the cap; 180 MiB volume holds it. + let min_catalog = (header_cap - 16) * node_size as u32; + let img = + create_blank_hfsplus_sized(180 * 1024 * 1024, 4096, "MapCap", false, min_catalog, 0); + let mut fs = Fs::open(std::io::Cursor::new(img), 0).unwrap(); + fs.prepare_for_edit().unwrap(); + + let start = BTreeHeader::read(fs.catalog_data()).total_nodes; + assert!( + start <= header_cap && start > header_cap - 64, + "blank catalog {start} not near cap" + ); + // No map node yet — the whole bitmap lives in the header node. + assert_eq!( + btree_bitmap_segments(fs.catalog_data(), node_size).len(), + 1, + "fresh near-cap catalog should have no map nodes" + ); + + // Grow until we cross the header bitmap cap. + let mut guard = 0; + while BTreeHeader::read(fs.catalog_data()).total_nodes <= header_cap { + fs.grow_btree_fork(BTreeFork::Catalog).unwrap(); + guard += 1; + assert!(guard < 64, "too many grows to cross the cap"); + } + let h = BTreeHeader::read(fs.catalog_data()); + assert!( + h.total_nodes > header_cap, + "did not cross the cap: {}", + h.total_nodes + ); + // A map-node segment must now exist in the bitmap chain. + let segs = btree_bitmap_segments(fs.catalog_data(), node_size); + assert!( + segs.len() >= 2, + "expected a map-node segment after crossing the cap, got {} segment(s)", + segs.len() + ); + + // fsck-clean with the map node in place. + fs.sync_metadata().unwrap(); + let result = fs.fsck().unwrap().unwrap(); + assert!( + result.errors.is_empty(), + "fsck after map-node grow: {:?}", + result + .errors + .iter() + .take(8) + .map(|e| &e.message) + .collect::>() + ); + + // Reopen re-reads the grown fork and the map chain. + let img = fs.reader.get_ref().clone(); + if let Ok(path) = std::env::var("RB_EMIT_MAPC_IMAGE") { + std::fs::write(&path, &img).unwrap(); + } + let fs2 = Fs::open(std::io::Cursor::new(img), 0).unwrap(); + assert_eq!( + BTreeHeader::read(fs2.catalog_data()).total_nodes, + h.total_nodes, + "reopen lost grown nodes" + ); + assert!( + btree_bitmap_segments(fs2.catalog_data(), node_size).len() >= 2, + "reopen lost the map node" + ); + } + + #[test] + fn test_hfsplus_catalog_grow_spills_to_overflow_phase_b() { + // §4b Phase B (docs/todo_hfsplus_fork_growth.md): real `create_file` + // calls interleave a data-block allocation with each catalog grow, so + // contiguous tail growth fails and the catalog fork fragments. Once it + // exhausts its 8 inline extents, further growth spills into the + // extents-overflow B-tree (keyed by the catalog's reserved CNID 4) and + // is persisted by the overflow-aware write path. The volume stays + // readable + fsck-clean, and a reopen re-reads the > 8-extent fork. + type Fs = HfsPlusFilesystem>>; + let img = create_blank_hfsplus_sized(16 * 1024 * 1024, 4096, "SpillVol", false, 0, 0); + let mut fs = Fs::open(std::io::Cursor::new(img), 0).unwrap(); + fs.prepare_for_edit().unwrap(); + let root = fs.root().unwrap(); + let mut dirs = Vec::new(); + for d in 0..20 { + dirs.push( + fs.create_directory(&root, &format!("d{d}"), &Default::default()) + .unwrap(), + ); + } + const N: u32 = 1200; + for i in 0..N { + let d = &dirs[(i % 20) as usize]; + let mut data = std::io::Cursor::new(vec![b'x']); + fs.create_file( + d, + &format!("f{i:05}.txt"), + &mut data, + 1, + &Default::default(), + ) + .unwrap_or_else(|e| panic!("create_file #{i} (grow+spill should cover it): {e}")); + } + + // The catalog fork saturated its 8 inline extents and spilled the rest + // into the extents-overflow B-tree. + let inline: u32 = fs + .vh + .catalog_file + .extents + .iter() + .map(|e| e.block_count) + .sum(); + let inline_count = fs + .vh + .catalog_file + .extents + .iter() + .filter(|e| e.block_count != 0) + .count(); + assert_eq!( + inline_count, 8, + "catalog should have filled all 8 inline extents" + ); + assert!( + fs.vh.catalog_file.total_blocks > inline, + "catalog total_blocks {} should exceed inline-extent blocks {inline} (overflow used)", + fs.vh.catalog_file.total_blocks + ); + + fs.sync_metadata().unwrap(); + let result = fs.fsck().unwrap().unwrap(); + assert!( + result.errors.is_empty(), + "fsck after overflow spill: {:?}", + result + .errors + .iter() + .take(8) + .map(|e| &e.message) + .collect::>() + ); + + // Reopen re-reads the > 8-extent catalog fork (inline + overflow) and + // every file is still enumerable. + let img = fs.reader.get_ref().clone(); + if let Ok(path) = std::env::var("RB_EMIT_SPILL_IMAGE") { + std::fs::write(&path, &img).unwrap(); + } + let mut fs2 = Fs::open(std::io::Cursor::new(img), 0).unwrap(); + let r2 = fs2.root().unwrap(); + let mut total = 0; + for de in fs2.list_directory(&r2).unwrap() { + if de.is_directory() { + total += fs2.list_directory(&de).unwrap().len(); + } + } + assert_eq!(total as u32, N, "reopen lost files: {total} != {N}"); + } + + #[test] + fn test_hfsplus_catalog_grows_fork_on_full_phase_a() { + // §4b Phase A (docs/todo_hfsplus_fork_growth.md): a volume whose catalog + // fork is *too small* for its eventual record count grows the fork in + // place through live inserts — no spurious DiskFull while the volume has + // free blocks — and stays fsck-clean. Contiguous tail growth keeps the + // catalog fork at a single extent. A reopen re-reads the grown fork. + use super::hfs_common::BTreeHeader; + type Fs = HfsPlusFilesystem>>; + + // 16 MiB volume, minimal (volume-scaled, ~21-node) catalog. Inserting + // far more records than that forces many grows. + let img = create_blank_hfsplus_sized(16 * 1024 * 1024, 4096, "GrowVol", false, 0, 0); + let mut fs = Fs::open(std::io::Cursor::new(img), 0).unwrap(); + fs.prepare_for_edit().unwrap(); + + let start_nodes = BTreeHeader::read(fs.catalog_data()).total_nodes; + + const DIRS: u32 = 20; + let root = fs.root().unwrap(); + let mut dir_ids = Vec::with_capacity(DIRS as usize); + for d in 0..DIRS { + let de = fs + .create_directory(&root, &format!("dir{d:03}"), &Default::default()) + .unwrap(); + dir_ids.push(de.location as u32); + } + let base_cnid = fs.vh.next_catalog_id; + + let n: u32 = 6_000; + let mut per_dir = vec![0i32; DIRS as usize]; + let empty_fork = ForkData::empty(); + for i in 0..n { + let hash = i.wrapping_mul(2_654_435_761); + let d = (hash % DIRS) as usize; + let name = format!("f{hash:08x}"); + let mut kr = Fs::build_catalog_key(dir_ids[d], &name); + if !kr.len().is_multiple_of(2) { + kr.push(0); + } + kr.extend_from_slice(&Fs::build_file_record( + base_cnid + i, + &empty_fork, + &empty_fork, + &[0u8; 4], + &[0u8; 4], + )); + fs.insert_catalog_record(&kr) + .unwrap_or_else(|e| panic!("insert #{i} into dir{d} (grow should cover it): {e}")); + per_dir[d] += 1; + } + fs.vh.file_count += n; + fs.vh.next_catalog_id = base_cnid + n; + for (d, &cnt) in per_dir.iter().enumerate() { + fs.update_parent_valence(dir_ids[d], cnt).unwrap(); + } + + // The fork grew (each grow adds >= 8 nodes; >= 2 grows expected here). + let end_nodes = BTreeHeader::read(fs.catalog_data()).total_nodes; + assert!( + end_nodes >= start_nodes + 16, + "catalog total_nodes only went {start_nodes} -> {end_nodes}; expected >= 2 grows" + ); + // Contiguous tail growth keeps the catalog fork at a single extent. + let cat_extents = fs + .vh + .catalog_file + .extents + .iter() + .filter(|e| e.block_count != 0) + .count(); + assert_eq!( + cat_extents, 1, + "contiguous tail growth should keep the catalog fork at 1 extent, got {cat_extents}" + ); + + // Persist the grown fork, bitmap, and VH, then fsck the result. + fs.sync_metadata().unwrap(); + + // fsck-clean after all the grows. + let result = fs.fsck().unwrap().unwrap(); + assert!( + result.errors.is_empty(), + "fsck found {} errors after grow: {:?}", + result.errors.len(), + result + .errors + .iter() + .take(8) + .map(|e| &e.message) + .collect::>() + ); + + // Reopen re-reads the grown fork (proves the fork extents + header + // persisted) and every record is still findable. + let img = fs.reader.get_ref().clone(); + // Optionally emit the grown image so a Mac can fsck_hfs it (the real + // §4b acceptance test): set RB_EMIT_GROW_IMAGE=/path. + if let Ok(path) = std::env::var("RB_EMIT_GROW_IMAGE") { + std::fs::write(&path, &img).unwrap(); + } + let mut fs2 = Fs::open(std::io::Cursor::new(img), 0).unwrap(); + let reopened_nodes = BTreeHeader::read(fs2.catalog_data()).total_nodes; + assert_eq!(reopened_nodes, end_nodes, "reopen lost grown nodes"); + let r2 = fs2.root().unwrap(); + let dirs = fs2.list_directory(&r2).unwrap(); + assert_eq!(dirs.len(), DIRS as usize, "reopen lost directories"); + } + #[test] fn test_hfsplus_extents_overflow_grows_multilevel() { // P3 (docs/hfsplus_btree_growth_plan.md): the extents-overflow B-tree — diff --git a/src/fs/hfsplus_defrag.rs b/src/fs/hfsplus_defrag.rs index c332c1b0..02ecc31e 100644 --- a/src/fs/hfsplus_defrag.rs +++ b/src/fs/hfsplus_defrag.rs @@ -421,8 +421,8 @@ use std::cmp::Ordering; use super::filesystem::FilesystemError; use super::hfs_common; use super::hfsplus::{ - write_blank_btree_header_node, write_empty_leaf_node, ExtentDescriptor, ForkData, - HfsPlusFilesystem, XattrKind, XattrRecord, HFSPLUS_VOLUME_UNMOUNTED_BIT, + write_blank_btree_header_node, ExtentDescriptor, ForkData, HfsPlusFilesystem, XattrKind, + XattrRecord, HFSPLUS_VOLUME_UNMOUNTED_BIT, }; thread_local! { @@ -1393,16 +1393,18 @@ fn init_btree_buffer(total_bytes: usize, max_key_len: u16, key_compare_type: u8) "btree size must be node-aligned" ); let total_nodes = (total_bytes / NODE_SIZE) as u32; - // Layout: 0 = header, 1 = empty leaf, 2..2+map_nodes = map nodes - // (BTNodeKind=2), then free nodes for B-tree growth. Map nodes are - // required whenever the catalog exceeds the header bitmap's capacity - // (~30,720 nodes at NODE_SIZE=4096, i.e. ~120 MiB catalog file). - // OS 9 volumes with 100k+ catalog records routinely need map nodes. + // Layout: 0 = header, 2..2+map_nodes = map nodes (BTNodeKind=2), then free + // nodes. Node 1 is left free; the first `btree_insert_full` bootstraps it + // into the root leaf (matching `write_blank_btree_header_node`'s canonical + // empty-tree shape: depth 0 / root 0, no leaf node until the first insert). + // Map nodes are required whenever the catalog exceeds the header bitmap's + // capacity (~30,720 nodes at NODE_SIZE=4096, i.e. ~120 MiB catalog file); + // OS 9 volumes with 100k+ catalog records routinely need them. let map_nodes = hfs_common::map_nodes_required(total_nodes, NODE_SIZE); - let used_nodes = 2 + map_nodes; + let used_nodes = 1 + map_nodes; // header + map nodes; node 1 is free assert!( total_nodes >= used_nodes + 2, - "need at least {} nodes (header + leaf + {} map nodes + 2 free), got {}", + "need at least {} nodes (header + {} map nodes + 2 free), got {}", used_nodes + 2, map_nodes, total_nodes, @@ -1417,17 +1419,16 @@ fn init_btree_buffer(total_bytes: usize, max_key_len: u16, key_compare_type: u8) max_key_len, key_compare_type, ); - write_empty_leaf_node(&mut buf[NODE_SIZE..2 * NODE_SIZE]); if map_nodes > 0 { // Header.fLink points at the first map node so `btree_bitmap_segments` // walks the chain when allocating from the global node space. BigEndian::write_u32(&mut buf[0..4], 2); // Mark each map node as allocated in the header's bitmap (record 2, - // at offset 270 — matches `write_blank_btree_header_node`). With - // NODE_SIZE=4096 the header bitmap covers ~30,720 bits; map nodes - // sit at indices 2..2+map_nodes, comfortably inside that range. - let bitmap_rec_offset = 270usize; + // at offset 248 — matches `write_blank_btree_header_node`'s canonical + // TN1150 layout). With NODE_SIZE=4096 the header bitmap covers ~30,720 + // bits; map nodes sit at indices 2..2+map_nodes, comfortably inside. + let bitmap_rec_offset = 248usize; let header_bitmap_len = NODE_SIZE - 256; // = 3840 bytes for i in 0..map_nodes { let bit = 2 + i; @@ -2848,6 +2849,13 @@ mod tests { ); assert_eq!(report.dirs_copied as usize, DIRS, "all dirs copied"); + // Emit the defrag-clone output so a Mac can fsck_hfs it — the canonical + // B-tree header layout fix makes this production path fsck_hfs-clean, + // not just clean per our own checker. + if let Ok(path) = std::env::var("RB_EMIT_DEFRAG_IMAGE") { + std::fs::write(&path, &out).unwrap(); + } + // The defrag-built target catalog must itself be multi-level... let cur = Cursor::new(out); let mut tgt = HfsPlusFilesystem::open(cur, 0).expect("target opens");