Implement Sigma correlation rule matching with sliding-window engine

[Copilot]

Adds full correlation rule evaluation to LogProcessor, covering all seven Sigma v2.1.0 correlation types with per-group sliding time windows, field alias resolution, and late-event discarding.

Breaking change

LogProcessor::start() now returns (Sender<LogEvent>, Receiver<DetectionResult>) instead of Receiver<Detection>. DetectionResult is an enum:

match detection_rx.recv().unwrap() {
    DetectionResult::Rule(d)        => println!("rule: {}", d.rule.title),
    DetectionResult::Correlation(c) => println!("correlation: {}", c.rule.title),
}

New types

• DetectionResult — unified channel output (Rule | Correlation)
• CorrelationDetection — triggered correlation rule + group_key + contributing_events

LogEvent timestamp

• Added timestamp: DateTime<Utc> field (defaults to Utc::now())
• get_time() accessor and with_timestamp() builder for explicit control

Correlation engine

• Internal worker thread receives raw Detections from the matcher pool and evaluates correlation state before forwarding to the output channel
• Per-(rule_index, group_key) VecDeque-backed sliding window; evicts events outside the timespan on every new arrival
• Supported types: event_count, value_count, temporal, temporal_ordered, value_sum, value_avg, value_percentile
• Field alias resolution maps group_by field names through the aliases map to the actual field name for the triggering rule
• ProcessorConfig::late_event_threshold_secs (default 3600 s) discards events arriving more than N seconds behind the most recent event in the window

New constructors

• LogProcessor::from_collection(&SigmaCollection) — loads detection rules and correlation rules in one call
• LogProcessor::with_correlation_rules(Vec<SigmaCorrelationRule>) — builder-style addition to an existing processor

Utilities

• parse_timespan(s: &str) -> Option<Duration> — parses Sigma timespan strings (1s, 5m, 1h, 2d, 1w)

Original prompt

Start implementation

The user has attached the following file paths as relevant context:

• .github\copilot-instructions.md

[Chronological Review: The conversation began with the user requesting the implementation of matching Sigma correlation rules based on a specified GitHub document. The user provided a detailed description of the requirements, including the need for timestamps in events, modifications to the existing `#sym:Event` trait, and the use of a message passing system for correlation matching. The user also emphasized the need for efficient data structures and handling of late-arriving events. The conversation progressed with the user actively working on the `c:\Users\thoma\OneDrive\Devel\sigma_engine\src\processor.rs` file, focusing on implementing these features.]

[Intent Mapping: The user explicitly requested the following:

1. "Implement matching of Sigma correlation rules as defined in the specification..."
2. "Events must now contain a timestamp."
3. "Add a get_time function to the #sym:Event trait."
4. "Use the already existing message passing system in #sym:impl LogProcessor for correlation matching."
5. "Create efficient data structures for maintenance of the correlation rule state."]

[Technical Inventory: The conversation involved the following technical concepts:

• #sym:SigmaCorrelationRule: A struct representing a fully-parsed Sigma correlation rule.
• #sym:Event trait: Required modification to include a timestamp.
• Message passing system: Existing system in #sym:impl LogProcessor to be utilized for correlation matching.
• UUIDs: To uniquely identify detections.
• Sliding time window: For correlation logic.
• Handling late-arriving events: With a configurable threshold.]

[Code Archaeology: The user is currently working on the processor.rs file, which is likely to contain the implementation logic for the correlation rules and event processing. The SigmaCorrelationRule struct was previously defined in types.rs, which includes various fields relevant to correlation rules.]

[Progress Assessment: The user has outlined the requirements for the correlation rule matching but has not yet implemented the changes. The focus is on defining the logic and data structures needed for the correlation matching process.]

[Context Validation: All critical information for continuation is captured, including the user's goals, the specific requirements for the implementation, and the current file being worked on.]

[Recent Commands Analysis: The last agent command was the user's request to implement the correlation rule matching, which included multiple specifications for functionality and structure. The immediate pre-summarization state involved the user actively working on the processor.rs file, detailing how to implement the correlation logic and event handling.]

1. Conversation Overview: - Primary Objectives: The user requested to "Implement matching of Sigma correlation rules as defined in the specification..." and specified various requirements for timestamps, event handling, and correlation logic. - Session Context: The conversation focused on implementing features in the Sigma engine, particularly in the `processor.rs` file, to support correlation rules. - User Intent Evolution: The user's needs have remained consistent, focusing on implementing a robust correlation matching system with specific requirements for event handling and data structures.

2. Technical Foundation:

• #sym:SigmaCorrelationRule: A struct for Sigma correlation rules, containing fields like title, id, description, and correlation details.
• #sym:Event trait: Needs modification to include a get_time function for timestamps.
• Message passing system: To be utilized for correlation matching in #sym:impl LogProcessor.
• UUIDs: To uniquely identify detections in the correlation process.

3. Codebase Status:

• processor.rs:
  • Purpose: Contains the implementation logic for processing events and correlation rules.
  • Current State: The user is actively working on implementing the correlation matching logic.
  • Key Code Segments: Not specified yet, as implementation is pending.
  • Dependencies: Relies on the #sym:SigmaCorrelationRule struct and the message passing system.

4. Problem Resolution:

• Issues Encountered: None explicitly mentioned, but challenges related to implementing the correlation logic and handling late events are anticipated.
• Solutions Implemented: No solutions have been implemented yet; the user is in the planning phase.
• Debugging Context: Ongoing development with no known issues reported.
• Lessons Learned: Insights will emerge as the implementation progresses.

5. Progress Tracking:

• Completed Tasks: None yet; the user has outlined requirements but not implemented them.
• Partially Complete Work: The user is in the process of defining the correlation matching logic.
• Validated Outcomes: None yet, as testing will follow implementatio...

Created from VS Code.

---

⚡ Quickly spin up Copilot coding agent tasks from anywhere on your macOS or Windows machine with Raycast.