feat: Hyperscan-based secret redaction with SOM spans and overlap merge

- redact.py: use python-hyperscan with HS_FLAG_SOM_LEFTMOST for correct spans
- redact_patterns.py: curated patterns (OpenAI, AWS, GitHub, Stripe, etc.)
- tests/test_redact.py: unit tests for redaction (use fake test values for keys)
- docs/secrets.md: document implementation and pre-commit options

Co-authored-by: Cursor <cursoragent@cursor.com>

Author: pascalwhoop

README.md

@@ -124,3 +124,7 @@ uv run convx explore --output-path /path/to/your/repo
 uv run convx hooks install
 uv run convx hooks uninstall
 ```
+
+## Secrets
+
+Exports are redacted by default (API keys, tokens, passwords → `[REDACTED]`). Be mindful of secrets in your history repo. See [docs/secrets.md](docs/secrets.md) for details and pre-commit scanner options (Gitleaks, TruffleHog, detect-secrets, semgrep).

docs/secrets.md

@@ -1,6 +1,6 @@
 # Secret redaction
 
-convx redacts API keys, tokens, passwords, and similar secrets from exported files so they don’t end up in your history repo. 
+convx redacts API keys, tokens, passwords, and similar secrets from exported files so they don't end up in your history repo.
 
 > ⚠️ You are responsible for ensuring no secrets are committed; we provide no warranty or liability.
 
@@ -14,11 +14,80 @@ convx redacts API keys, tokens, passwords, and similar secrets from exported fil
   convx backup --output-path /path/to/repo --no-redact
   ```
 
-- **What gets redacted:** Patterns for credentials and secrets (e.g. OpenAI-style keys, `api_key=...`, `password=...`, AWS/GCP/Stripe/GitHub tokens). High-entropy but non-secret text (URLs, long IDs) is left as-is.
-- **What you see:** Matched secrets are replaced with markers like `[REDACTED:api_key]` or `[REDACTED:password]` in the written files.
+- **What gets redacted:** API keys (OpenAI, AWS, Stripe, GitHub, etc.), tokens, passwords, private keys, and similar credentials. Matched secrets are replaced with `[REDACTED]`.
+- **What you see:** All redacted values appear as `[REDACTED]` in the written files.
 
 ## How it works technically
 
-- **Library:** [plumbrc](https://pypi.org/project/plumbrc/) — pattern-based redaction (700+ built-in patterns). If plumbrc isn’t available at runtime, redaction is skipped and content is written unchanged.
+- **Library:** [Hyperscan](https://github.com/intel/hyperscan) via [python-hyperscan](https://github.com/darvid/python-hyperscan) — multi-pattern matching with `HS_FLAG_UTF8` and `HS_FLAG_SOM_LEFTMOST` so we get correct start/end spans. Text is scanned as UTF-8 bytes; match spans are merged when overlapping, then replaced from end to start so indices stay valid. All matches become `[REDACTED]`.
 - **When:** Redaction runs **after** rendering and **before** writing. Both Markdown and JSON outputs are passed through `redact_secrets()` in the engine; the index (`.convx/index.json`) is not redacted.
-- **Where in code:** `convx_ai.redact.redact_secrets(text, redact=True)` wraps `Plumbr(quiet=True).redact(text)`. The engine calls it for every session markdown and JSON blob (including child-session markdown) when `redact=True`; the CLI passes `redact=not no_redact` from `sync` and `backup`.
+- **Where in code:** `convx_ai.redact.redact_secrets(text, redact=True)`. The engine calls it for every session markdown and JSON blob when `redact=True`; the CLI passes `redact=not no_redact` from `sync` and `backup`.
+- **Patterns:** Curated regex set in `convx_ai.redact_patterns` (OpenAI, AWS, GitHub, Stripe, Slack, SendGrid, Discord, Google, Twilio, Telegram, Square, private keys, etc.). No entropy-based detection (avoids over-redacting URLs and long IDs in transcripts). Patterns must be Hyperscan-compatible (no backrefs, no lookahead/lookbehind).
+
+## Pattern sources
+
+Default patterns are derived from:
+
+- [mazen160/secrets-patterns-db](https://github.com/mazen160/secrets-patterns-db) — high-confidence and rules-stable datasets (1,600+ patterns, ReDoS-tested)
+- [gitleaks/gitleaks](https://github.com/gitleaks/gitleaks) — `config/gitleaks.toml` built-in rules
+- [Yelp/detect-secrets](https://github.com/Yelp/detect-secrets) — plugin regex patterns
+
+To add or change patterns, edit `src/convx_ai/redact_patterns.py`. Each entry is `(bytes_regex, unique_id)`; regex syntax is Hyperscan’s (PCRE-like, no backrefs/lookahead/lookbehind).
+
+## Pre-commit secret scanners (defense in depth)
+
+For additional protection, add a pre-commit hook to block secrets before they enter your repo. Popular options:
+
+| Tool | Language | Engine | Notes |
+|-----|----------|--------|-------|
+| **Gitleaks** | Go | RE2 | Single-pass scan, huge maintained pattern DB. Most popular pre-commit tool. |
+| **TruffleHog** | Go | RE2 | Similar to Gitleaks, strong verification and pattern coverage. |
+| **detect-secrets** | Python | regex | Yelp's tool, what most Python shops use in CI. Baseline-based. |
+| **semgrep** | OCaml | custom | AST-aware, more than regex. Heavily used in enterprise CI. |
+
+### Example configs
+
+**Gitleaks** (recommended):
+
+```yaml
+# .pre-commit-config.yaml
+repos:
+  - repo: https://github.com/gitleaks/gitleaks
+    rev: v8.18.0
+    hooks:
+      - id: gitleaks
+```
+
+**TruffleHog**:
+
+```yaml
+repos:
+  - repo: https://github.com/trufflesecurity/trufflehog
+    rev: v3.78.0
+    hooks:
+      - id: trufflehog
+```
+
+**detect-secrets** (Python shops):
+
+```yaml
+repos:
+  - repo: https://github.com/Yelp/detect-secrets
+    rev: v1.5.0
+    hooks:
+      - id: detect-secrets
+        args: ['--baseline', '.secrets.baseline']
+```
+
+Create a baseline with `detect-secrets scan > .secrets.baseline` before first use.
+
+**semgrep** (enterprise / AST-aware):
+
+```yaml
+repos:
+  - repo: https://github.com/semgrep/pre-commit
+    rev: v1.151.0
+    hooks:
+      - id: semgrep
+        args: ['--config', 'p/secrets', '--error']
+```

pyproject.toml

@@ -18,7 +18,7 @@ classifiers = [
 ]
 
 dependencies = [
-  "plumbrc>=1.0.0",
+  "hyperscan>=0.2.0",
   "tantivy>=0.22",
   "textual>=8.0",
   "typer>=0.12.0",

src/convx_ai/redact.py

@@ -1,16 +1,55 @@
 from __future__ import annotations
 
-try:
-    from plumbrc import Plumbr
-    _plumbr_available = True
-except Exception:
-    Plumbr = None
-    _plumbr_available = False
+import hyperscan
+
+from convx_ai.redact_patterns import PATTERNS
+
+REDACTED = "[REDACTED]"
+
+_db: hyperscan.Database | None = None
+
+
+def _get_db() -> hyperscan.Database:
+    global _db
+    if _db is None:
+        exprs, ids = zip(*PATTERNS)
+        _db = hyperscan.Database()
+        _db.compile(
+            expressions=list(exprs),
+            ids=list(ids),
+            elements=len(PATTERNS),
+            flags=[hyperscan.HS_FLAG_UTF8 | hyperscan.HS_FLAG_SOM_LEFTMOST] * len(PATTERNS),
+        )
+    return _db
+
+
+def _merge_overlaps(spans: list[tuple[int, int]]) -> list[tuple[int, int]]:
+    if not spans:
+        return []
+    sorted_spans = sorted(spans)
+    merged = [sorted_spans[0]]
+    for start, end in sorted_spans[1:]:
+        if start <= merged[-1][1]:
+            merged[-1] = (merged[-1][0], max(merged[-1][1], end))
+        else:
+            merged.append((start, end))
+    return merged
 
 
 def redact_secrets(text: str, *, redact: bool = True) -> str:
     if not redact:
         return text
-    if not _plumbr_available:
-        return text
-    return Plumbr(quiet=True).redact(text)
+    data = text.encode("utf-8")
+    matches: list[tuple[int, int]] = []
+
+    def on_match(id: int, from_: int, to: int, flags: int, context: list) -> None:
+        context.append((from_, to))
+
+    _get_db().scan(data, match_event_handler=on_match, context=matches)
+    spans = _merge_overlaps(matches)
+
+    for start, end in sorted(spans, key=lambda s: -s[1]):
+        char_start = len(data[:start].decode("utf-8"))
+        char_end = len(data[:end].decode("utf-8"))
+        text = text[:char_start] + REDACTED + text[char_end:]
+    return text

src/convx_ai/redact_patterns.py

@@ -0,0 +1,55 @@
+"""
+Secret detection patterns for redaction.
+
+Hyperscan-compatible: no backrefs, no lookahead/lookbehind.
+Patterns match the secret value only (no surrounding context).
+
+Sources:
+- mazen160/secrets-patterns-db (high-confidence, rules-stable)
+- gitleaks/gitleaks config
+- Yelp/detect-secrets plugins
+"""
+
+# (pattern, id) — ids must be unique
+PATTERNS: list[tuple[bytes, int]] = [
+    # OpenAI (detect-secrets, custom)
+    (br"sk-proj-[a-zA-Z0-9_-]{20,}", 0),
+    (br"sk-[a-zA-Z0-9_-]{20,}", 1),
+    (br"sk-[A-Za-z0-9-_]*[A-Za-z0-9]{20}T3BlbkFJ[A-Za-z0-9]{20}", 2),
+    # AWS (secrets-patterns-db, gitleaks)
+    (br"AKIA[0-9A-Z]{16}", 3),
+    (br"ASIA[0-9A-Z]{16}", 4),
+    (br"da2-[a-z0-9]{26}", 5),
+    # GitHub (detect-secrets)
+    (br"(?:ghp|gho|ghu|ghs|ghr)_[A-Za-z0-9_]{36}", 6),
+    # Stripe (secrets-patterns-db)
+    (br"sk_live_[0-9a-zA-Z]{24}", 7),
+    (br"rk_live_[0-9a-zA-Z]{24}", 8),
+    # Slack (detect-secrets — flexible format)
+    (br"xox(?:a|b|p|o|s|r)-(?:\d+-)+[a-zA-Z0-9]+", 9),
+    (br"https://hooks\.slack\.com/services/T[a-zA-Z0-9_]+/B[a-zA-Z0-9_]+/[a-zA-Z0-9_]+", 10),
+    # SendGrid (detect-secrets)
+    (br"SG\.[a-zA-Z0-9_-]{22}\.[a-zA-Z0-9_-]{43}", 11),
+    # Discord (detect-secrets)
+    (br"[MNO][a-zA-Z0-9_-]{23,25}\.[a-zA-Z0-9_-]{6}\.[a-zA-Z0-9_-]{27}", 12),
+    # Google (secrets-patterns-db)
+    (br"AIza[0-9A-Za-z_-]{35}", 13),
+    (br"ya29\.[0-9A-Za-z_-]+", 14),
+    # Twilio (secrets-patterns-db)
+    (br"SK[0-9a-fA-F]{32}", 15),
+    # Telegram (secrets-patterns-db)
+    (br"[0-9]+:AA[0-9A-Za-z_-]{33}", 16),
+    # Mailgun, Mailchimp (secrets-patterns-db)
+    (br"key-[0-9a-zA-Z]{32}", 17),
+    (br"[0-9a-f]{32}-us[0-9]{1,2}", 18),
+    # Square (secrets-patterns-db)
+    (br"sq0atp-[0-9A-Za-z_-]{22}", 19),
+    (br"sq0csp-[0-9A-Za-z_-]{43}", 20),
+    # Private keys (secrets-patterns-db, detect-secrets)
+    (br"-----BEGIN RSA PRIVATE KEY-----", 21),
+    (br"-----BEGIN DSA PRIVATE KEY-----", 22),
+    (br"-----BEGIN EC PRIVATE KEY-----", 23),
+    (br"-----BEGIN OPENSSH PRIVATE KEY-----", 24),
+    (br"-----BEGIN PGP PRIVATE KEY BLOCK-----", 25),
+    (br"-----BEGIN PRIVATE KEY-----", 26),
+]

tests/test_redact.py

@@ -0,0 +1,106 @@
+from __future__ import annotations
+
+import pytest
+
+from convx_ai.redact import REDACTED, redact_secrets
+
+
+def test_redact_disabled() -> None:
+    assert redact_secrets("sk-proj-abc123xyz", redact=False) == "sk-proj-abc123xyz"
+
+
+def test_openai_project_key() -> None:
+    text = "Use API key sk-proj-redact-test-abc123xyz for the demo."
+    assert REDACTED in redact_secrets(text)
+    assert "sk-proj-redact-test-abc123xyz" not in redact_secrets(text)
+
+
+def test_openai_legacy_key() -> None:
+    text = "sk-abcdefghijklmnopqrstuvwxyz123456"
+    assert redact_secrets(text) == REDACTED
+
+
+def test_aws_key() -> None:
+    text = "AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE"
+    out = redact_secrets(text)
+    assert REDACTED in out
+    assert "AKIAIOSFODNN7EXAMPLE" not in out
+
+
+def test_github_token() -> None:
+    text = "ghp_" + "x" * 36
+    assert redact_secrets(text) == REDACTED
+
+
+def test_stripe_key() -> None:
+    text = "sk_live_" + "0" * 24
+    assert redact_secrets(text) == REDACTED
+
+
+def test_slack_token() -> None:
+    text = "xoxb-1234-5678-abcdefghijkl"
+    assert redact_secrets(text) == REDACTED
+
+
+def test_slack_webhook() -> None:
+    text = "https://hooks.slack.com/services/T123/B456/abc123"
+    assert redact_secrets(text) == REDACTED
+
+
+def test_sendgrid_key() -> None:
+    text = "SG." + "a" * 22 + "." + "b" * 43
+    assert redact_secrets(text) == REDACTED
+
+
+def test_private_key_rsa() -> None:
+    text = "-----BEGIN RSA PRIVATE KEY-----"
+    assert redact_secrets(text) == REDACTED
+
+
+def test_private_key_ec() -> None:
+    text = "-----BEGIN EC PRIVATE KEY-----"
+    assert redact_secrets(text) == REDACTED
+
+
+def test_private_key_openssh() -> None:
+    text = "-----BEGIN OPENSSH PRIVATE KEY-----"
+    assert redact_secrets(text) == REDACTED
+
+
+def test_multiple_secrets() -> None:
+    secret1 = "sk-proj-" + "a" * 20
+    secret2 = "sk-proj-" + "b" * 20
+    text = f"key1={secret1} key2={secret2}"
+    out = redact_secrets(text)
+    assert secret1 not in out and secret2 not in out
+    assert out.count(REDACTED) == 2
+
+
+def test_no_false_positive_urls() -> None:
+    text = "https://example.com/path?param=value"
+    assert redact_secrets(text) == text
+
+
+def test_no_false_positive_short_sk() -> None:
+    text = "sk-abc"
+    assert redact_secrets(text) == text
+
+
+def test_no_false_positive_ghp_prefix() -> None:
+    text = "ghp_short"
+    assert redact_secrets(text) == text
+
+
+def test_google_api_key() -> None:
+    text = "AIza" + "a" * 35
+    assert redact_secrets(text) == REDACTED
+
+
+def test_twilio_key() -> None:
+    text = "SK" + "a" * 32
+    assert redact_secrets(text) == REDACTED
+
+
+def test_telegram_bot_token() -> None:
+    text = "12345:AA" + "a" * 33
+    assert redact_secrets(text) == REDACTED