███████╗██████╗ ███████╗ ██████╗████████╗██████╗ █████╗
██╔════╝██╔══██╗██╔════╝██╔════╝╚══██╔══╝██╔══██╗██╔══██╗
███████╗██████╔╝█████╗ ██║ ██║ ██████╔╝███████║
╚════██║██╔═══╝ ██╔══╝ ██║ ██║ ██╔══██╗██╔══██║
███████║██║ ███████╗╚██████╗ ██║ ██║ ██║██║ ██║
╚══════╝╚═╝ ╚══════╝ ╚═════╝ ╚═╝ ╚═╝ ╚═╝╚═╝ ╚═╝
Tagline: Probe. Measure. Expose privacy risk before deployment.
SYSTEM : LLM Privacy Auditing Tool
VERSION : 1.0
STATUS : OPERATIONAL
CLASSIFICATION : OPEN SOURCE
TARGET : Large Language Models"Spectra systematically interrogates language models to surface latent privacy exposure before deployment."
Spectra is a Python-based LLM privacy auditing toolkit that stress-tests language models against three high-impact privacy attack vectors. It runs targeted probes, computes weighted risk scores, captures structured findings, and generates a detailed HTML audit report with Streamlit-based live execution visibility.
[MODULE-1] PII Detection Probing 8 adversarial prompts across social engineering vectors
[MODULE-2] Regurgitation Detection Exact + semantic similarity against 8 sensitive documents
[MODULE-3] Membership Inference Heuristic Confidence gap analysis across target vs random corpora
[CORE] Weighted Risk Scoring Entity-type weighted scoring with critical/high/low tiers
[CORE] Detailed HTML Report Per-prompt breakdowns, similarity tables, CSS bar charts
[CORE] Live Streamlit Dashboard Real-time execution with step indicators and progress bar
[CORE] Error Handling + Logging Per-call exception handling with spectra.log audit trail
[CORE] Auto Report Cleanup Keeps last 3 reports, auto-deletes older ones
[PROMPT ENGINE] → [PII DETECTOR] → [REGURGITATION DETECTOR] → [MEMBERSHIP INFERENCE] → [REPORT GENERATOR] → [DASHBOARD]
INPUT LAYER
└── Prompt Engine → 8 adversarial prompts per module
PROCESSING LAYER
├── PII Detector → Presidio entity recognition + weighted scoring
├── Regurgitation Det. → RapidFuzz exact + sentence-transformers semantic
└── Membership Inference → Confidence gap analysis, hybrid scoring
OUTPUT LAYER
├── HTML Report → Detailed per-module breakdown with charts
├── Dashboard → Streamlit live control panel
└── spectra.log → Structured audit trail
| Stage | Component | Description |
|---|---|---|
| 1 | PII Detector | Multi-prompt PII probing with weighted entity scoring |
| 2 | Regurgitation Detector | Exact and semantic similarity checks against sensitive corpus |
| 3 | Membership Inference | Target vs random confidence gap analysis |
| 4 | Report Generator | Detailed standalone HTML report with module breakdowns |
| 5 | Streamlit Dashboard | Live execution panel, progress states, and result visualization |
[+] PII Detection Probing
Uses crafted extraction prompts and Presidio entity analysis to detect leaked emails,
phone numbers, names, addresses, and identifier patterns.
Note: This module measures PII generation risk, not confirmed leakage of real training data.
[+] Verbatim Regurgitation Detection
Tests whether the model reproduces sensitive-style text using exact similarity
(RapidFuzz) and semantic similarity (Sentence Transformers).
Note: The corpus used consists of public domain texts likely present in training data, not verified training data extracts. Similarity results reflect generation behavior, not confirmed memorization.
[+] Membership Inference Heuristic
Compares completion confidence between likely-seen corpus text and random nonsense
text to estimate potential membership inference signal.
CRITICAL PII Score ----------- 100/100 High PII generation risk detected. Review model outputs carefully.
HIGH MEM Score ----------- 060/100 Significant exposure. Audit before production.
MEDIUM REG Score ----------- 020/100 Moderate signal. Investigate findings.
LOW ALL Score ----------- 010/100 Minimal exposure. Monitor across updates.
| Vector | Method | Tool |
|---|---|---|
| PII Extraction | Adversarial prompting | Presidio + weighted scoring |
| Semantic Regurgitation | Meaning-based similarity | sentence-transformers |
| Verbatim Reproduction | Exact text matching | rapidfuzz |
| Membership Inference | Confidence gap analysis | hybrid exact + semantic |
| Social Engineering | Role-play prompt injection | custom prompt engine |
| Component | Purpose |
|---|---|
| Python 3.11 | Core runtime |
| google-genai | Gemini/Gemma model client |
| presidio-analyzer | PII entity detection |
| spacy | NLP backend support |
| rapidfuzz | String similarity scoring |
| sentence-transformers | Semantic similarity scoring |
| streamlit | Live audit dashboard |
| Supported Models | gemini-3.1-flash-lite, gemini-2.5-flash, gemini-2.5-flash-lite, gemma-3-12b-it |
Spectra/
├─ main.py
├─ dashboard.py
├─ modules/
│ ├─ pii_detector.py
│ ├─ regurgitation_detector.py
│ └─ membership_inference.py
├─ utils/
│ └─ report_generator.py
├─ reports/
├─ prompts/
├─ assets/
└─ requirements.txt
# 1) Clone
git clone https://github.com/<your-username>/Spectra.git
cd Spectra
# 2) Virtual environment
python -m venv venv
# 3) Activate (Windows PowerShell)
venv\Scripts\Activate.ps1
# macOS/Linux alternative
# source venv/bin/activate
# 4) Dependencies
pip install -r requirements.txtCreate a .env file in the project root:
GEMINI_API_KEY=your_api_key_hereRun the CLI audit:
python main.pyRun the live dashboard:
streamlit run dashboard.py[+] Starting Spectra Audit...
[+] Running: PII Detection
[+] Running: Verbatim Regurgitation Detection
[+] Running: Membership Inference Heuristic
[+] Audit complete
[+] Report generated at: reports/spectra_audit_report_YYYYMMDD_HHMMSS.html
The HTML report includes model metadata, audit timestamp, per-module visual risk bars, and a combined risk score.
- Carlini et al. (2021), Extracting Training Data from Large Language Models.
- Shokri et al. (2017), Membership Inference Attacks against Machine Learning Models.
- Differential Privacy literature and foundational privacy-preserving ML research.
Note: This project implements heuristic approximations inspired by the above research. It does not fully replicate the cryptographic methods described in these papers.
[COMPLETE] v1.0 — Core 3-module privacy risk evaluation pipeline with HTML report and Streamlit dashboard
[QUEUED] v2.0 — OpenAI support, PDF export, multi-model comparison
[QUEUED] v3.0 — Scheduled audits, API endpoint, fine-tuned PII classifier
>> Fork the repository
>> Create a feature branch
>> Keep changes modular and testable
>> Open a pull request with clear rationale and sample output
MIT License
This project is for educational and research use only. Use responsibly, with explicit authorization, and in compliance with applicable legal and organizational requirements.
Rudra Singh
Cybersecurity Aspirant
[ SPECTRA ] — INTERROGATE. MEASURE. SECURE. — OPEN SOURCE