Evaluator: implement indirect offset resolution

[unclesp1d3r]

Summary

Implement evaluation of indirect offsets. The AST representation (OffsetSpec::Indirect) already exists, but the evaluator returns a "not yet implemented" error at src/evaluator/offset.rs:159.

Context

Indirect offsets are a core feature of magic files that allow pointer-chasing through binary structures. The syntax (offset.type) reads a value at the given offset, interprets it as a pointer of the specified type, and uses the result as the actual offset for matching.

Examples:

• (0x3c.l) - Read a 32-bit long at offset 0x3c, use that value as the offset (PE executable header pointer)
• (4.s) - Read a 16-bit short at offset 4, use as offset

Current state: AST variant defined, evaluation stubbed with TODO.

Acceptance Criteria

• resolve_offset() handles OffsetSpec::Indirect by reading the pointer value from the buffer
• Supports all numeric types as the indirect type (byte, short, long, quad)
• Handles endianness correctly for indirect reads
• Supports optional adjustment arithmetic: (offset.type)+adjustment
• Bounds checking on both the pointer read and the resulting offset
• Unit tests for indirect offset resolution
• Integration tests with PE header detection (classic indirect offset use case)

Impact

HIGH - Indirect offsets are critical for complex binary format detection (PE executables, compound documents, archive formats with headers pointing to content).

Files to Modify

• src/evaluator/offset.rs - Implement indirect resolution (line ~159)
• src/evaluator/types.rs - May need to expose read functions for indirect use

References

• GNU file magic(5) man page: "indirect offsets"
• PE format detection uses (0x3c.l) extensively
• Epic Epic: libmagic-rs Phase 1 MVP #24

[traycerai]

Plan

Observations

The codebase has a well-structured evaluator module with existing offset resolution (file:src/evaluator/offset.rs) and type reading functions (file:src/evaluator/types.rs). The AST already defines OffsetSpec::Indirect with all necessary fields (base_offset, pointer_type, adjustment, endian). The TODO at line 159 in file:src/evaluator/offset.rs indicates where implementation is needed. Existing functions like read_byte, read_short, and read_long provide the foundation for reading pointer values. The error handling infrastructure is comprehensive with proper bounds checking patterns already established.

Approach

Implement indirect offset resolution by reading a pointer value at the base offset, converting it to a usize offset, applying the adjustment, and validating bounds. The implementation will leverage existing type reading functions (read_byte, read_short, read_long) to interpret the pointer value according to pointer_type and endian. A helper function will convert Value enum variants to usize offsets with proper error handling for negative values and overflow. The solution follows the established pattern of two-stage bounds checking: first for the pointer read, then for the resulting offset.

Implementation Steps

1. Implement Indirect Offset Resolution in file:src/evaluator/offset.rs

Replace the TODO block at lines 158-165 in the resolve_offset function with the following logic:

Step 1.1: Read the pointer value

• Call resolve_absolute_offset with base_offset to get the absolute position for reading the pointer
• Use the appropriate type reading function based on pointer_type:
  • For TypeKind::Byte: call types::read_byte(buffer, pointer_offset)
  • For TypeKind::Short: call types::read_short(buffer, pointer_offset, endian, signed)
  • For TypeKind::Long: call types::read_long(buffer, pointer_offset, endian, signed)
  • For TypeKind::String: return OffsetError::InvalidOffset (strings cannot be used as pointers)
• Handle the TypeReadError by converting it to LibmagicError::EvaluationError

Step 1.2: Convert Value to offset

• Create a helper function value_to_offset(value: &Value) -> Result<i64, OffsetError> that:
  • Matches on Value::Uint(u): converts to i64 with overflow check using i64::try_from(u)
  • Matches on Value::Int(i): returns the i64 directly
  • Matches on Value::Bytes(_) or Value::String(_): returns OffsetError::InvalidOffset
• Call this helper to convert the read pointer value to an i64

Step 1.3: Apply adjustment

• Add adjustment to the pointer value using checked arithmetic
• Use i64::checked_add(pointer_value, adjustment) to prevent overflow
• Return OffsetError::ArithmeticOverflow if overflow occurs

Step 1.4: Resolve final offset

• Call resolve_absolute_offset with the adjusted pointer value to get the final usize offset
• This provides automatic bounds checking for the resulting offset
• Convert any OffsetError to the appropriate LibmagicError::EvaluationError variant

2. Add Helper Function for Value Conversion

Add a private helper function in file:src/evaluator/offset.rs after the resolve_absolute_offset function:

/// Convert a Value to an i64 offset
/// 
/// Only Uint and Int values can be converted to offsets.
/// Returns InvalidOffset error for String or Bytes values.
fn value_to_offset(value: &Value) -> Result<i64, OffsetError>

This function should:

• Handle Value::Uint by attempting i64::try_from conversion
• Handle Value::Int by returning the value directly
• Return OffsetError::InvalidOffset for non-numeric values
• Return OffsetError::ArithmeticOverflow for values that don't fit in i64

3. Update Unit Tests in file:src/evaluator/offset.rs

Step 3.1: Update existing test

• Modify test_resolve_offset_indirect_not_implemented (line 328) to verify successful indirect offset resolution instead of expecting an error
• Rename it to test_resolve_offset_indirect_basic

Step 3.2: Add comprehensive unit tests

• test_resolve_offset_indirect_byte_pointer: Test with TypeKind::Byte as pointer type
• test_resolve_offset_indirect_short_pointer: Test with TypeKind::Short (both endiannesses)
• test_resolve_offset_indirect_long_pointer: Test with TypeKind::Long (both endiannesses)
• test_resolve_offset_indirect_with_adjustment: Test positive and negative adjustments
• test_resolve_offset_indirect_bounds_checking: Test pointer read out of bounds
• test_resolve_offset_indirect_result_out_of_bounds: Test when resulting offset exceeds buffer
• test_resolve_offset_indirect_overflow: Test arithmetic overflow in adjustment
• test_resolve_offset_indirect_negative_pointer: Test handling of signed pointer values
• test_value_to_offset_conversions: Test the helper function with all Value variants

4. Add Integration Tests

Create integration tests in file:tests/evaluator_tests.rs or a new test file:

Step 4.1: PE Header Detection Test

• Create a minimal PE executable header structure
• Use indirect offset (0x3c.l) to read PE signature location
• Verify the evaluator correctly follows the pointer chain
• Test pattern: DOS header at 0, PE offset pointer at 0x3c, PE signature at pointed location

Step 4.2: Multi-level Indirect Test

• Test nested structures where one indirect offset points to another structure
• Verify proper offset resolution through multiple levels

Step 4.3: Edge Cases

• Test with minimum buffer sizes
• Test with maximum valid offsets
• Test endianness handling (create buffers with different byte orders)

5. Documentation Updates

Step 5.1: Update function documentation

• Update the doc comment for resolve_offset in file:src/evaluator/offset.rs (line 111-141)
• Remove the note about indirect offsets not being implemented
• Add examples showing indirect offset usage

Step 5.2: Add examples

• Add a doc example showing PE header detection pattern
• Add an example showing adjustment usage

Implementation Flow

sequenceDiagram
    participant Caller
    participant resolve_offset
    participant resolve_absolute_offset
    participant read_typed_value
    participant value_to_offset
    
    Caller->>resolve_offset: OffsetSpec::Indirect
    resolve_offset->>resolve_absolute_offset: base_offset
    resolve_absolute_offset-->>resolve_offset: pointer_position
    resolve_offset->>read_typed_value: buffer, pointer_position, pointer_type
    read_typed_value-->>resolve_offset: Value (pointer)
    resolve_offset->>value_to_offset: Value
    value_to_offset-->>resolve_offset: i64 offset
    resolve_offset->>resolve_offset: checked_add(offset, adjustment)
    resolve_offset->>resolve_absolute_offset: adjusted_offset
    resolve_absolute_offset-->>resolve_offset: final_usize_offset
    resolve_offset-->>Caller: Result<usize>

Loading

Error Handling Strategy

Error Condition	Error Type	Handling
Base offset out of bounds	OffsetError::BufferOverrun	Convert to EvaluationError::BufferOverrun
Pointer read fails	TypeReadError::BufferOverrun	Convert to EvaluationError::BufferOverrun
Non-numeric pointer value	OffsetError::InvalidOffset	Convert to EvaluationError::InvalidOffset
Adjustment overflow	OffsetError::ArithmeticOverflow	Convert to EvaluationError::InvalidOffset
Result offset out of bounds	OffsetError::BufferOverrun	Convert to EvaluationError::BufferOverrun
String used as pointer type	OffsetError::InvalidOffset	Convert to EvaluationError::UnsupportedType

Testing Strategy

Test Category	Coverage
Unit Tests	All pointer types (byte, short, long), both endiannesses, signed/unsigned
Bounds Tests	Pointer read OOB, result offset OOB, empty buffers
Arithmetic Tests	Positive/negative adjustments, overflow conditions
Integration Tests	PE header detection, nested structures, real-world patterns
Edge Cases	Minimum buffers, maximum offsets, zero adjustments

Import In IDE

🤖 Prompt for AI Agents

## Observations

The codebase has a well-structured evaluator module with existing offset resolution (`file:src/evaluator/offset.rs`) and type reading functions (`file:src/evaluator/types.rs`). The AST already defines `OffsetSpec::Indirect` with all necessary fields (base_offset, pointer_type, adjustment, endian). The TODO at line 159 in `file:src/evaluator/offset.rs` indicates where implementation is needed. Existing functions like `read_byte`, `read_short`, and `read_long` provide the foundation for reading pointer values. The error handling infrastructure is comprehensive with proper bounds checking patterns already established.

## Approach

Implement indirect offset resolution by reading a pointer value at the base offset, converting it to a usize offset, applying the adjustment, and validating bounds. The implementation will leverage existing type reading functions (`read_byte`, `read_short`, `read_long`) to interpret the pointer value according to `pointer_type` and `endian`. A helper function will convert `Value` enum variants to usize offsets with proper error handling for negative values and overflow. The solution follows the established pattern of two-stage bounds checking: first for the pointer read, then for the resulting offset.

## Implementation Steps

### 1. Implement Indirect Offset Resolution in `file:src/evaluator/offset.rs`

Replace the TODO block at lines 158-165 in the `resolve_offset` function with the following logic:

**Step 1.1: Read the pointer value**
- Call `resolve_absolute_offset` with `base_offset` to get the absolute position for reading the pointer
- Use the appropriate type reading function based on `pointer_type`:
  - For `TypeKind::Byte`: call `types::read_byte(buffer, pointer_offset)`
  - For `TypeKind::Short`: call `types::read_short(buffer, pointer_offset, endian, signed)`
  - For `TypeKind::Long`: call `types::read_long(buffer, pointer_offset, endian, signed)`
  - For `TypeKind::String`: return `OffsetError::InvalidOffset` (strings cannot be used as pointers)
- Handle the `TypeReadError` by converting it to `LibmagicError::EvaluationError`

**Step 1.2: Convert Value to offset**
- Create a helper function `value_to_offset(value: &Value) -> Result<i64, OffsetError>` that:
  - Matches on `Value::Uint(u)`: converts to i64 with overflow check using `i64::try_from(u)`
  - Matches on `Value::Int(i)`: returns the i64 directly
  - Matches on `Value::Bytes(_)` or `Value::String(_)`: returns `OffsetError::InvalidOffset`
- Call this helper to convert the read pointer value to an i64

**Step 1.3: Apply adjustment**
- Add `adjustment` to the pointer value using checked arithmetic
- Use `i64::checked_add(pointer_value, adjustment)` to prevent overflow
- Return `OffsetError::ArithmeticOverflow` if overflow occurs

**Step 1.4: Resolve final offset**
- Call `resolve_absolute_offset` with the adjusted pointer value to get the final usize offset
- This provides automatic bounds checking for the resulting offset
- Convert any `OffsetError` to the appropriate `LibmagicError::EvaluationError` variant

### 2. Add Helper Function for Value Conversion

Add a private helper function in `file:src/evaluator/offset.rs` after the `resolve_absolute_offset` function:

```rust
/// Convert a Value to an i64 offset
/// 
/// Only Uint and Int values can be converted to offsets.
/// Returns InvalidOffset error for String or Bytes values.
fn value_to_offset(value: &Value) -> Result<i64, OffsetError>
```

This function should:
- Handle `Value::Uint` by attempting `i64::try_from` conversion
- Handle `Value::Int` by returning the value directly
- Return `OffsetError::InvalidOffset` for non-numeric values
- Return `OffsetError::ArithmeticOverflow` for values that don't fit in i64

### 3. Update Unit Tests in `file:src/evaluator/offset.rs`

**Step 3.1: Update existing test**
- Modify `test_resolve_offset_indirect_not_implemented` (line 328) to verify successful indirect offset resolution instead of expecting an error
- Rename it to `test_resolve_offset_indirect_basic`

**Step 3.2: Add comprehensive unit tests**
- `test_resolve_offset_indirect_byte_pointer`: Test with `TypeKind::Byte` as pointer type
- `test_resolve_offset_indirect_short_pointer`: Test with `TypeKind::Short` (both endiannesses)
- `test_resolve_offset_indirect_long_pointer`: Test with `TypeKind::Long` (both endiannesses)
- `test_resolve_offset_indirect_with_adjustment`: Test positive and negative adjustments
- `test_resolve_offset_indirect_bounds_checking`: Test pointer read out of bounds
- `test_resolve_offset_indirect_result_out_of_bounds`: Test when resulting offset exceeds buffer
- `test_resolve_offset_indirect_overflow`: Test arithmetic overflow in adjustment
- `test_resolve_offset_indirect_negative_pointer`: Test handling of signed pointer values
- `test_value_to_offset_conversions`: Test the helper function with all Value variants

### 4. Add Integration Tests

Create integration tests in `file:tests/evaluator_tests.rs` or a new test file:

**Step 4.1: PE Header Detection Test**
- Create a minimal PE executable header structure
- Use indirect offset `(0x3c.l)` to read PE signature location
- Verify the evaluator correctly follows the pointer chain
- Test pattern: DOS header at 0, PE offset pointer at 0x3c, PE signature at pointed location

**Step 4.2: Multi-level Indirect Test**
- Test nested structures where one indirect offset points to another structure
- Verify proper offset resolution through multiple levels

**Step 4.3: Edge Cases**
- Test with minimum buffer sizes
- Test with maximum valid offsets
- Test endianness handling (create buffers with different byte orders)

### 5. Documentation Updates

**Step 5.1: Update function documentation**
- Update the doc comment for `resolve_offset` in `file:src/evaluator/offset.rs` (line 111-141)
- Remove the note about indirect offsets not being implemented
- Add examples showing indirect offset usage

**Step 5.2: Add examples**
- Add a doc example showing PE header detection pattern
- Add an example showing adjustment usage

## Implementation Flow

```mermaid
sequenceDiagram
    participant Caller
    participant resolve_offset
    participant resolve_absolute_offset
    participant read_typed_value
    participant value_to_offset
    
    Caller->>resolve_offset: OffsetSpec::Indirect
    resolve_offset->>resolve_absolute_offset: base_offset
    resolve_absolute_offset-->>resolve_offset: pointer_position
    resolve_offset->>read_typed_value: buffer, pointer_position, pointer_type
    read_typed_value-->>resolve_offset: Value (pointer)
    resolve_offset->>value_to_offset: Value
    value_to_offset-->>resolve_offset: i64 offset
    resolve_offset->>resolve_offset: checked_add(offset, adjustment)
    resolve_offset->>resolve_absolute_offset: adjusted_offset
    resolve_absolute_offset-->>resolve_offset: final_usize_offset
    resolve_offset-->>Caller: Result<usize>
```

## Error Handling Strategy

| Error Condition | Error Type | Handling |
|----------------|------------|----------|
| Base offset out of bounds | `OffsetError::BufferOverrun` | Convert to `EvaluationError::BufferOverrun` |
| Pointer read fails | `TypeReadError::BufferOverrun` | Convert to `EvaluationError::BufferOverrun` |
| Non-numeric pointer value | `OffsetError::InvalidOffset` | Convert to `EvaluationError::InvalidOffset` |
| Adjustment overflow | `OffsetError::ArithmeticOverflow` | Convert to `EvaluationError::InvalidOffset` |
| Result offset out of bounds | `OffsetError::BufferOverrun` | Convert to `EvaluationError::BufferOverrun` |
| String used as pointer type | `OffsetError::InvalidOffset` | Convert to `EvaluationError::UnsupportedType` |

## Testing Strategy

| Test Category | Coverage |
|--------------|----------|
| Unit Tests | All pointer types (byte, short, long), both endiannesses, signed/unsigned |
| Bounds Tests | Pointer read OOB, result offset OOB, empty buffers |
| Arithmetic Tests | Positive/negative adjustments, overflow conditions |
| Integration Tests | PE header detection, nested structures, real-world patterns |
| Edge Cases | Minimum buffers, maximum offsets, zero adjustments |

---

Execution Information

Branch: main
Commit: 36adac5

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.