SecOps decision engine
A lightweight SOC workflow currently with highlights around Alert-triage decision making, Detection-as-code and the utilisation of threat intelligence. As an ai engineer as well I will be looking to find reliable use-cases to integrate AI into existing practices to uncover specific pain points which can enhanced with the technology.
Python handles anything that touches AI APIs, data enrichment, or transformation logic. Go handles anything that needs to be fast, concurrent, and reliable at the boundary (ingestion, queueing, retries, the audit log writer). Async communication via message queues will decouple entities
Key Components- Threat Intel Enrichment Service (Python) - integrations of third-party threat intel APIs (VirusTotal, AbuseIPDB free tiers) to act as reusable dependency for downstream services. FastAPI, and redis for caching to respect rate limiting and allow for resource management. A confidence score is implemented across the sources
Alert Triage Pipeline Service (Go + Python) - go service owns ingestion and queue management, reads mock alert streams, handles retries, and routes to workers. The python worker calls your enrichment service and hits the LLM for classifcation and summarisation, applies confidence thresholds and writes decision with full explainability output. The audit log lives here. Key Compenent
Detection-as-Code Framework (Python) - YAML defined detections that run against log samples and feed alerts into the pipeline. Git-versioned, testable against sample datasets (potentially AI workflow again but with more strigent design), with false-positive tracking. The layer security teams can interact with. - accumulated history for detections e.g 'did it fire?', was the triage 'auto decision' or 'escalate', did it require a 'human override' this will be used to track the detection rule over it's lifecycle. - rule_id -> [outcomes] mapped, which allows for the derivation of false-positives per rule. - allows for isolated testing, without infrastructure dependencies.
The Mini SIEM is deliberately absent. System will get just enough log ingestion to feed the pipeline and nothing more. A thin Go service that tails files or reads from a socket. Won't be a full SIEM, so it will not be queryable, primarily just scaffolding.
I have opted to not focus too much on the SIEM platform aspect of this distributed build at least primarily to ensure I satisfy optimal time constraints and delve deeper into the less represented aspects of SOC workflows.
Top level docker compose.yml to bind system together as a system. Root compose will handle external dependencies (redis, rabbitMQ, db(lightweight for now))