Evaluate and implement fuzzing tests

[unclesp1d3r]

Summary

Evaluate whether dedicated fuzzing tests (e.g., cargo-fuzz / libFuzzer) are appropriate for the project, and implement them if so. v0.3.0 introduces features with significantly more complex parsing and evaluation paths that would benefit from coverage-guided fuzzing.

Current State

• unsafe_code = "forbid" is enforced project-wide, so memory corruption is not a concern
• Property tests via proptest (tests/property_tests.rs) generate random inputs for the parser and evaluator, providing a lightweight form of fuzzing
• 94%+ line coverage with 1,000+ tests across unit, integration, property, and doc tests
• Attack surface is currently small: 4 types, 7 operators, absolute/from-end offsets

v0.3.0 Features That Would Benefit

These features introduce complex, data-dependent parsing and evaluation paths where edge cases are hard to enumerate manually:

• Regex type (regex/N): Pattern compilation from untrusted magic files, binary-safe matching against arbitrary file content. Malformed patterns, catastrophic backtracking, and binary edge cases are classic fuzzing targets.
• 64-bit integer types (quad, bequad, lequad): Wider value ranges increase the combinatorial space for comparison and bitwise operators.
• Floating-point types (float, double, befloat, lefloat): NaN, infinity, subnormals, and endianness handling create subtle edge cases.
• Search type: Variable-length pattern matching with offset ranges.

Suggested Fuzz Targets

1. Parser fuzzing: Feed random bytes as magic file content to the parser; verify it never panics
2. Evaluator fuzzing: Generate random (rule, buffer) pairs; verify evaluation never panics
3. End-to-end fuzzing: Random magic file + random target buffer through MagicDatabase::evaluate_buffer

Evaluation Criteria

• Does cargo-fuzz find bugs that proptest misses?
• Is the maintenance overhead (fuzz corpus, CI integration) justified?
• Does this contribute to OSSF Best Practices silver/gold badge requirements?

Acceptance Criteria

• Document evaluation decision (proceed or defer with rationale)
• If proceeding: fuzz/ directory with at least parser and evaluator targets
• If proceeding: CI job for scheduled fuzzing runs (not per-PR, too slow)
• If proceeding: Initial corpus seeded from existing test fixtures and src/builtin_rules.magic

[traycerai]

Plan

Observations

• The project pins stable Rust 1.91.0 in rust-toolchain.toml; cargo-fuzz (libFuzzer) requires nightly — the fuzz crate must carry its own rust-toolchain.toml override.
• unsafe_code = "forbid" is a workspace-level lint; the fuzz crate will be a separate, standalone workspace (standard cargo-fuzz layout), so it is unaffected.
• proptest already covers "never panics on arbitrary buffers" at the evaluator level (prop_evaluation_never_panics). Coverage-guided fuzzing adds structured exploration of the parser and the new v0.3.0 code paths (regex, quad, float, search).
• MagicDatabase::evaluate_buffer(&[u8]) is already public. A public load_from_str / load_from_reader entry point is needed for parser and end-to-end targets to avoid temp-file overhead.
• Parser modules: src/parser/{mod.rs, ast.rs, grammar.rs, hierarchy.rs, loader.rs, preprocessing.rs}. Evaluator modules: src/evaluator/{mod.rs, offset.rs, operators.rs, types.rs, strength.rs}.

Approach

Use cargo-fuzz (libFuzzer) in a standalone fuzz/ workspace with three targets mirroring the issue's suggested targets. The fuzz crate references the main crate as a path dependency. A scheduled GitHub Actions workflow (weekly, not per-PR) runs each target for a bounded time and uploads crash artifacts. Corpus is seeded from existing test fixtures and src/builtin_rules.magic.

---

Implementation Steps

1. Expose a load_from_str API on MagicDatabase

In src/lib.rs, add a public method MagicDatabase::load_from_str(content: &str) -> Result<Self> (or load_from_reader) that feeds raw magic-file text through the existing parser pipeline in src/parser/loader.rs. This is the entry point for both the parser fuzz target and the end-to-end fuzz target, avoiding temp-file I/O inside the harness.

Follow the existing pattern of with_builtin_rules() — construct a MagicDatabase, parse the content via the loader, and return Result<Self, LibmagicError>.

---

2. Create the fuzz/ directory structure

fuzz/
├── Cargo.toml
├── rust-toolchain.toml          ← nightly override (required by cargo-fuzz)
├── fuzz_targets/
│   ├── parser_fuzz.rs
│   ├── evaluator_fuzz.rs
│   └── end_to_end_fuzz.rs
└── corpus/
    ├── parser_fuzz/             ← seeded inputs (see step 4)
    ├── evaluator_fuzz/
    └── end_to_end_fuzz/

---

3. Author fuzz/Cargo.toml

• [package] with name = "libmagic-rs-fuzz", publish = false, edition = "2024".
• [package.metadata] cargo-fuzz = true (required marker).
• [dependencies]: libfuzzer-sys = "0.4" and libmagic-rs = { path = ".." }.
• Three [[bin]] entries — one per fuzz target file — each with test = false and doc = false.
• Do not add fuzz/ to the root Cargo.toml workspace members; it must remain a standalone workspace.

---

4. Author fuzz/rust-toolchain.toml

Specify channel = "nightly" with components = ["rustfmt", "llvm-tools"]. This overrides the root rust-toolchain.toml only for the fuzz crate, keeping the main crate on stable 1.91.0.

---

5. Implement the three fuzz targets

Each target follows the libfuzzer-sys harness pattern: fuzz_target!(|data: &[u8]| { … }). The body must never panic — all Results are silently discarded (the harness only cares about panics/aborts).

fuzz_targets/parser_fuzz.rs

• Convert data to a &str (via std::str::from_utf8; return early if invalid UTF-8 — or use a lossy conversion if the parser accepts arbitrary bytes).
• Call MagicDatabase::load_from_str(content).
• Discard the result; the goal is "no panic, no abort".

fuzz_targets/evaluator_fuzz.rs

• Call MagicDatabase::with_builtin_rules() (constructed once outside the harness using std::sync::OnceLock for efficiency).
• Call db.evaluate_buffer(data).
• Discard the result.

fuzz_targets/end_to_end_fuzz.rs

• Split data at the first 0x00 byte (or at a fixed offset like data.len() / 2) to produce magic_content and target_buffer.
• Attempt MagicDatabase::load_from_str(magic_content_str).
• If successful, call db.evaluate_buffer(target_buffer).
• Discard all results.

---

6. Seed the corpus

Populate fuzz/corpus/parser_fuzz/ and fuzz/corpus/end_to_end_fuzz/ by copying:

• src/builtin_rules.magic (split into individual rule lines as separate corpus entries).
• Existing magic fixture files from tests/ (any .magic files used by integration tests).

Populate fuzz/corpus/evaluator_fuzz/ with the known-pattern byte sequences already used in tests/property_tests.rs (ELF header \x7fELF, ZIP PK\x03\x04, empty buffer, etc.) as individual binary files.

---

7. Add fuzz recipes to justfile

Add a [group('fuzz')] section with recipes:

Recipe	Command
fuzz-parser	cd fuzz && cargo fuzz run parser_fuzz -- -max_total_time=60
fuzz-evaluator	cd fuzz && cargo fuzz run evaluator_fuzz -- -max_total_time=60
fuzz-e2e	cd fuzz && cargo fuzz run end_to_end_fuzz -- -max_total_time=60
fuzz-all	calls all three sequentially
fuzz-corpus-min	cd fuzz && cargo fuzz cmin <target> for each target

---

8. Create .github/workflows/fuzzing.yml

Trigger: schedule (weekly, e.g. Sunday 02:00 UTC) + workflow_dispatch

The workflow should:

1. Check out the repository.
2. Install nightly Rust (via dtolnay/rust-toolchain@nightly with llvm-tools component).
3. Install cargo-fuzz via cargo install cargo-fuzz.
4. For each target (parser_fuzz, evaluator_fuzz, end_to_end_fuzz):
   • Run cargo fuzz run <target> fuzz/corpus/<target> -- -max_total_time=300 (5 min per target).
5. On failure, upload the fuzz/artifacts/ directory as a GitHub Actions artifact (crash inputs).
6. Use continue-on-error: true per target step so all targets run even if one finds a crash.
7. Do not add this workflow to the lint-actions list in justfile (it uses nightly-specific syntax that actionlint may flag).

sequenceDiagram
    participant Scheduler as GitHub Scheduler (weekly)
    participant CI as fuzzing.yml workflow
    participant Fuzz as cargo-fuzz (nightly)
    participant Lib as libmagic-rs (path dep)

    Scheduler->>CI: trigger
    CI->>Fuzz: cargo fuzz run parser_fuzz (5 min)
    Fuzz->>Lib: MagicDatabase::load_from_str(arbitrary bytes)
    Lib-->>Fuzz: Ok / Err (no panic = pass)
    CI->>Fuzz: cargo fuzz run evaluator_fuzz (5 min)
    Fuzz->>Lib: db.evaluate_buffer(arbitrary bytes)
    Lib-->>Fuzz: Ok / Err (no panic = pass)
    CI->>Fuzz: cargo fuzz run end_to_end_fuzz (5 min)
    Fuzz->>Lib: load_from_str + evaluate_buffer
    Lib-->>Fuzz: Ok / Err (no panic = pass)
    CI-->>Scheduler: upload crash artifacts (if any)

Loading

---

9. Update AGENTS.md

In the Testing Strategy section, add a bullet for fuzzing:

• Fuzz Tests: cargo-fuzz targets in fuzz/ for parser, evaluator, and end-to-end paths. Run locally with just fuzz-all; scheduled weekly in CI via .github/workflows/fuzzing.yml.

In the Standard Commands section, add just fuzz-all and just fuzz-parser etc.

In the Current Limitations section (or a new Fuzzing section), document that the fuzz crate uses nightly Rust independently of the main crate's stable toolchain.

---

10. Update docs/src/security-assurance.md

Per the AGENTS.md requirement ("must be updated when new attack surface is introduced"), add a section documenting:

• The three fuzz targets and what attack surface they cover.
• That the fuzz crate is a separate workspace and does not affect the unsafe_code = "forbid" guarantee of the main crate.
• The scheduled CI cadence and crash artifact retention policy.

Import In IDE

🤖 Prompt for AI Agents

## Observations

- The project pins **stable Rust 1.91.0** in `rust-toolchain.toml`; `cargo-fuzz` (libFuzzer) requires **nightly** — the fuzz crate must carry its own `rust-toolchain.toml` override.
- `unsafe_code = "forbid"` is a **workspace-level** lint; the fuzz crate will be a **separate, standalone workspace** (standard `cargo-fuzz` layout), so it is unaffected.
- `proptest` already covers "never panics on arbitrary buffers" at the evaluator level (`prop_evaluation_never_panics`). Coverage-guided fuzzing adds structured exploration of the parser and the new v0.3.0 code paths (regex, quad, float, search).
- `MagicDatabase::evaluate_buffer(&[u8])` is already public. A public `load_from_str` / `load_from_reader` entry point is needed for parser and end-to-end targets to avoid temp-file overhead.
- Parser modules: `src/parser/{mod.rs, ast.rs, grammar.rs, hierarchy.rs, loader.rs, preprocessing.rs}`. Evaluator modules: `src/evaluator/{mod.rs, offset.rs, operators.rs, types.rs, strength.rs}`.

## Approach

Use `cargo-fuzz` (libFuzzer) in a standalone `fuzz/` workspace with three targets mirroring the issue's suggested targets. The fuzz crate references the main crate as a path dependency. A scheduled GitHub Actions workflow (weekly, not per-PR) runs each target for a bounded time and uploads crash artifacts. Corpus is seeded from existing test fixtures and `src/builtin_rules.magic`.

---

## Implementation Steps

### 1. Expose a `load_from_str` API on `MagicDatabase`

In `src/lib.rs`, add a public method `MagicDatabase::load_from_str(content: &str) -> Result<Self>` (or `load_from_reader`) that feeds raw magic-file text through the existing parser pipeline in `src/parser/loader.rs`. This is the entry point for both the parser fuzz target and the end-to-end fuzz target, avoiding temp-file I/O inside the harness.

> Follow the existing pattern of `with_builtin_rules()` — construct a `MagicDatabase`, parse the content via the loader, and return `Result<Self, LibmagicError>`.

---

### 2. Create the `fuzz/` directory structure

```
fuzz/
├── Cargo.toml
├── rust-toolchain.toml          ← nightly override (required by cargo-fuzz)
├── fuzz_targets/
│   ├── parser_fuzz.rs
│   ├── evaluator_fuzz.rs
│   └── end_to_end_fuzz.rs
└── corpus/
    ├── parser_fuzz/             ← seeded inputs (see step 4)
    ├── evaluator_fuzz/
    └── end_to_end_fuzz/
```

---

### 3. Author `fuzz/Cargo.toml`

- `[package]` with `name = "libmagic-rs-fuzz"`, `publish = false`, `edition = "2024"`.
- `[package.metadata] cargo-fuzz = true` (required marker).
- `[dependencies]`: `libfuzzer-sys = "0.4"` and `libmagic-rs = { path = ".." }`.
- Three `[[bin]]` entries — one per fuzz target file — each with `test = false` and `doc = false`.
- Do **not** add `fuzz/` to the root `Cargo.toml` workspace `members`; it must remain a standalone workspace.

---

### 4. Author `fuzz/rust-toolchain.toml`

Specify `channel = "nightly"` with `components = ["rustfmt", "llvm-tools"]`. This overrides the root `rust-toolchain.toml` only for the fuzz crate, keeping the main crate on stable 1.91.0.

---

### 5. Implement the three fuzz targets

Each target follows the `libfuzzer-sys` harness pattern: `fuzz_target!(|data: &[u8]| { … })`. The body must never panic — all `Result`s are silently discarded (the harness only cares about panics/aborts).

**`fuzz_targets/parser_fuzz.rs`**
- Convert `data` to a `&str` (via `std::str::from_utf8`; return early if invalid UTF-8 — or use a lossy conversion if the parser accepts arbitrary bytes).
- Call `MagicDatabase::load_from_str(content)`.
- Discard the result; the goal is "no panic, no abort".

**`fuzz_targets/evaluator_fuzz.rs`**
- Call `MagicDatabase::with_builtin_rules()` (constructed once outside the harness using `std::sync::OnceLock` for efficiency).
- Call `db.evaluate_buffer(data)`.
- Discard the result.

**`fuzz_targets/end_to_end_fuzz.rs`**
- Split `data` at the first `0x00` byte (or at a fixed offset like `data.len() / 2`) to produce `magic_content` and `target_buffer`.
- Attempt `MagicDatabase::load_from_str(magic_content_str)`.
- If successful, call `db.evaluate_buffer(target_buffer)`.
- Discard all results.

---

### 6. Seed the corpus

Populate `fuzz/corpus/parser_fuzz/` and `fuzz/corpus/end_to_end_fuzz/` by copying:
- `src/builtin_rules.magic` (split into individual rule lines as separate corpus entries).
- Existing magic fixture files from `tests/` (any `.magic` files used by integration tests).

Populate `fuzz/corpus/evaluator_fuzz/` with the known-pattern byte sequences already used in `tests/property_tests.rs` (ELF header `\x7fELF`, ZIP `PK\x03\x04`, empty buffer, etc.) as individual binary files.

---

### 7. Add `fuzz` recipes to `justfile`

Add a `[group('fuzz')]` section with recipes:

| Recipe | Command |
|---|---|
| `fuzz-parser` | `cd fuzz && cargo fuzz run parser_fuzz -- -max_total_time=60` |
| `fuzz-evaluator` | `cd fuzz && cargo fuzz run evaluator_fuzz -- -max_total_time=60` |
| `fuzz-e2e` | `cd fuzz && cargo fuzz run end_to_end_fuzz -- -max_total_time=60` |
| `fuzz-all` | calls all three sequentially |
| `fuzz-corpus-min` | `cd fuzz && cargo fuzz cmin <target>` for each target |

---

### 8. Create `.github/workflows/fuzzing.yml`

```
Trigger: schedule (weekly, e.g. Sunday 02:00 UTC) + workflow_dispatch
```

The workflow should:

1. Check out the repository.
2. Install nightly Rust (via `dtolnay/rust-toolchain@nightly` with `llvm-tools` component).
3. Install `cargo-fuzz` via `cargo install cargo-fuzz`.
4. For each target (`parser_fuzz`, `evaluator_fuzz`, `end_to_end_fuzz`):
   - Run `cargo fuzz run <target> fuzz/corpus/<target> -- -max_total_time=300` (5 min per target).
5. On failure, upload the `fuzz/artifacts/` directory as a GitHub Actions artifact (crash inputs).
6. Use `continue-on-error: true` per target step so all targets run even if one finds a crash.
7. Do **not** add this workflow to the `lint-actions` list in `justfile` (it uses nightly-specific syntax that `actionlint` may flag).

```mermaid
sequenceDiagram
    participant Scheduler as GitHub Scheduler (weekly)
    participant CI as fuzzing.yml workflow
    participant Fuzz as cargo-fuzz (nightly)
    participant Lib as libmagic-rs (path dep)

    Scheduler->>CI: trigger
    CI->>Fuzz: cargo fuzz run parser_fuzz (5 min)
    Fuzz->>Lib: MagicDatabase::load_from_str(arbitrary bytes)
    Lib-->>Fuzz: Ok / Err (no panic = pass)
    CI->>Fuzz: cargo fuzz run evaluator_fuzz (5 min)
    Fuzz->>Lib: db.evaluate_buffer(arbitrary bytes)
    Lib-->>Fuzz: Ok / Err (no panic = pass)
    CI->>Fuzz: cargo fuzz run end_to_end_fuzz (5 min)
    Fuzz->>Lib: load_from_str + evaluate_buffer
    Lib-->>Fuzz: Ok / Err (no panic = pass)
    CI-->>Scheduler: upload crash artifacts (if any)
```

---

### 9. Update `AGENTS.md`

In the **Testing Strategy** section, add a bullet for fuzzing:
- **Fuzz Tests**: `cargo-fuzz` targets in `fuzz/` for parser, evaluator, and end-to-end paths. Run locally with `just fuzz-all`; scheduled weekly in CI via `.github/workflows/fuzzing.yml`.

In the **Standard Commands** section, add `just fuzz-all` and `just fuzz-parser` etc.

In the **Current Limitations** section (or a new **Fuzzing** section), document that the fuzz crate uses nightly Rust independently of the main crate's stable toolchain.

---

### 10. Update `docs/src/security-assurance.md`

Per the AGENTS.md requirement ("must be updated when new attack surface is introduced"), add a section documenting:
- The three fuzz targets and what attack surface they cover.
- That the fuzz crate is a separate workspace and does not affect the `unsafe_code = "forbid"` guarantee of the main crate.
- The scheduled CI cadence and crash artifact retention policy.

---

Execution Information

Branch: main
Commit: f4da223

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.