InsightLog is an advanced, full‑featured log analysis framework inspired by SIEM capabilities. It provides end‑to‑end log ingestion, signature-based detection, burst/rate anomaly detection, correlation logic, PDF reporting, and a live dashboard — all designed to emulate the workflow of a real security operations environment.
InsightLog supports real‑world workflows such as:
- Investigating attacks using Apache or Syslog data
- Detecting suspicious patterns, brute-force attempts, 404 floods, or unusual behavior
- Analyzing attack timelines
- Generating professional executive PDF reports
- Using custom rules for detections
- Viewing security alert data through a clean dashboard
- Apache access logs (combined format)
- Syslog / auth logs
- Handles malformed, inconsistent, and real-world logs
- Rule Engine (signature-based)
- Burst Detection (rate-based anomalies)
- Correlation Engine (multi-alert relationships)
- Overview with charts
- Alerts (categorized by severity)
- Timeline analysis
- Correlation view
- Dark/Light mode
- Auto-refresh
-
Rich
report.json+report.csvoutputs -
Fully structured PDF report with:
- Cover page
- Summary
- Charts (Top IPs, Severity, URLs)
- Alerts table
- Correlation events
- Timeline snapshot
- Plugin system (
/plugins) - Rule file support (
rules.txt) - Burst stress-test samples
- Real examples in
/samples
flowchart TD
%% Nodes
A[Raw Log Files] -->|Input| B(Parser Modules)
B --> C{Rule Engine}
C --> D{Burst Detector}
D --> E{Correlation Engine}
E -->|Save Data| F[(report.json)]
%% Outputs
F -->|Fetch Data| G[Web Dashboard / Flask]
F -->|Export| H[PDF Generator]
%% Styling
style A fill:#f9f,stroke:#333,stroke-width:2px
style F fill:#ffe6cc,stroke:#d79b00,stroke-width:2px,stroke-dasharray: 5 5
style G fill:#e1f5fe,stroke:#01579b,stroke-width:2px
style H fill:#e1f5fe,stroke:#01579b,stroke-width:2px
InsightLog/
├── insightlog.py # CLI tool entrypoint
├── parser/
│ ├── apache_parser.py
│ ├── syslog_parser.py
│ ├── burst_detector.py
│ ├── correlation.py
│ ├── rules_engine.py
│ ├── pdf_export.py
│ ├── utils.py
│ ├── live_monitor.py
│ └── plugin_manager.py
│
├── dashboard/
│ ├── app.py # Flask dashboard backend
│ ├── __init__.py
│ ├── static/
│ │ ├── dashboard.js # Frontend logic
│ │ └── style.css # Dashboard styling
│ └── templates/
│ └── dashboard.html # UI layout
│
├── plugins/ # Custom user plugins
│ └── sample_geo_blocker.py
│
├── samples/ # Test log files
│ ├── apache_access.log
│ ├── apache_attack.log
│ ├── auth.log
│ ├── burst_test.log
│ └── live_test.log
│
├── output/ # Generated reports
│ ├── report.json
│ ├── report.csv
│ └── InsightLog_Report.pdf
│
├── rules.txt # User-customizable detection rules
├── requirements.txt
└── README.md
git clone https://github.com/yourusername/InsightLog.git
cd InsightLog
pip install -r requirements.txt
Any command containing --plot will generate popup visual charts:
- Top IPs bar chart
- Status code distribution
- Top URLs graph
- Severity breakdown
python insightlog.py --type <apache|syslog> --file <logfile> --plot
python insightlog.py --type apache --file samples/apache_access.log
python insightlog.py --type syslog --file samples/auth.log
python insightlog.py --type apache --file samples/apache_attack.log --rules rules.txt
python insightlog.py --type apache --file samples/apache_attack.log --rules rules.txt --debug-rules
python insightlog.py --type apache --file samples/burst_test.log
python insightlog.py --type syslog --file /var/log/auth.log --live
python insightlog.py --type apache --file samples/apache_access.log
python insightlog.py --type syslog --file samples/auth.log
python insightlog.py --type apache --file samples/apache_attack.log --rules rules.txt
python insightlog.py --type apache --file samples/apache_attack.log --rules rules.txt --debug-rules
python insightlog.py --type apache --file samples/burst_test.log
python insightlog.py --type syslog --file /var/log/auth.log --live
rules.txt is fully user-editable.
Format:
<event> [by_ip] >= <threshold> <SEVERITY>
Examples:
failed_login by_ip >= 5 ALERT
404 by_ip >= 10 WARNING
ip_requests >= 50 CRITICAL
Supports:
- Global matching
- Per-IP matching
- Debug tracing (
--debug-rules)
Burst detection identifies rapid-fire events across sliding windows:
- 5 seconds
- 30 seconds
- 300 seconds
Example output:
[CRITICAL] Burst detected for attacker_ip: 5s, 30s windows exceeded threshold
Burst logs can be tested using:
samples/burst_test.log
Identifies:
- Multi-rule trigger clusters
- High-impact attacker patterns
- Host/IP-based relationships
Example output:
[CORR] IP abc123 triggered 4 alerts from 3 rule categories
Run:
cd dashboard
python app.py
Open in browser:
http://127.0.0.1:5001/
Dashboard tabs:
- Overview – metrics + charts
- Alerts – rule detections
- Timeline – chronological event flow
- Correlations – grouped attacker behavior
Export full SIEM-style PDF:
http://127.0.0.1:5001/export/pdf
Includes:
- Cover page
- Executive summary
- Chart visuals
- Alerts table
- Correlation events
- Timeline snapshot
- Top IPs Bar Chart
- Status Code Distribution
- Top URLs Graph
Alexander P.B. Cybersecurity Researcher & Penetration Tester | Red Team & IoT Security Specialist
📧 Reach out via GitHub for research collaborations.