Merge pull request #167 from DataFog/sidmohan0-patch-1

Update README.md to reflect accurate performance metrics

Author: sidmohan0

AGENTS.md

@@ -13,19 +13,19 @@
 
 ## Current Project Status
 
-**Stable version: 4.4.0**
+**Stable version: 4.8.0**
 
-**Development version: 4.4.0a5**
+**Development version: 4.8.0**
 
-**Next minor target: 4.5.0**
+**Next major target: 5.0.0**
 
 ### ✅ Recently Completed (Latest)
 
+- **Redaction Correctness**: Document-order token numbering, overlap-safe span handling, and stable mask mappings
+- **Allowlists and Typing**: Exact and regex allowlists, Presidio-style aliases, and `py.typed`
+- **Agent & Gateway Firewall**: Claude Code hook and LiteLLM guardrail adapters
 - **GLiNER Integration**: Modern NER engine with PII-specialized models
-- **Smart Cascading**: Intelligent regex → GLiNER → spaCy progression
-- **Enhanced CLI**: Model management with `--engine` flags
 - **Performance Validation**: 100x+ regex speedup reproducible via `benchmarks/`; 32x GLiNER
-- **CI/CD Consolidation**: 7 workflows → 3 (ci, release, benchmark)
 
 ## Quick Development Setup
 
@@ -41,7 +41,7 @@ pre-commit install
 
 # 3. Install ML extras for advanced features
 pip install -e ".[nlp]"           # spaCy
-pip install -e ".[nlp-advanced]"  # GLiNER (NEW)
+pip install -e ".[nlp-advanced]"  # GLiNER
 pip install -e ".[all]"           # Everything
 
 # 4. Verify installation
@@ -89,7 +89,7 @@ pip install datafog[ocr]           # Image processing
 pip install datafog[all]           # Everything
 ```
 
-## GLiNER Integration (NEW)
+## GLiNER Integration
 
 ### Overview
 

README.md

@@ -6,14 +6,14 @@ It provides:
 
 - Fast structured PII detection via regex
 - An offline PII firewall for AI agents: a Claude Code hook and a LiteLLM
-  gateway guardrail (new in 4.6)
+  gateway guardrail (introduced in 4.6)
 - Optional NER support via spaCy and GLiNER
 - A simple agent-oriented API for LLM applications
 - Backward-compatible `DataFog` and `TextService` classes
 
-## Agent & Gateway Firewall (4.6)
+## Agent & Gateway Firewall
 
-DataFog 4.6 adds two ready-made enforcement points that catch PII at the
+DataFog includes two ready-made enforcement points that catch PII at the
 moment it would leave your machine — offline, in microseconds, with matched
 values never echoed into logs or transcripts:
 
@@ -72,8 +72,8 @@ pip install datafog[all]
 
 Python 3.13 support is certified for the core SDK, CLI, `nlp`,
 `nlp-advanced`, and `ocr` install profiles. Donut OCR still requires a model
-that is available locally before runtime use. `distributed` and `all` are not
-newly certified on Python 3.13 in the 4.x line.
+that is available locally before runtime use. `distributed` and `all` remain
+optional, heavier profiles and are not part of the lightweight core path.
 
 ## Quick Start
 

ROADMAP.md

@@ -1,6 +1,6 @@
 # DataFog Roadmap
 
-The roadmap — current release status (4.7.x) and v5.0.0 plans — lives in
+The roadmap — current release status (4.8.x) and v5.0.0 plans — lives in
 [docs/roadmap.rst](docs/roadmap.rst).
 
 Roadmap priorities are shaped by user feedback — open a

benchmarks/README.md

@@ -49,7 +49,7 @@ All suites use the production default entity set
 
 ## Reference results
 
-Apple M5 Pro, macOS, CPython 3.13, datafog 4.7.0, litellm 1.91.0,
+Apple M5 Pro, macOS, CPython 3.13, datafog 4.8.0, litellm 1.91.0,
 presidio-analyzer 2.2.363, spaCy 3.8 (`en_core_web_sm`). Medians of the
 full (non-quick) run; expect different absolute numbers on different
 hardware, but the ratios and orders of magnitude should hold.
@@ -116,28 +116,6 @@ detect names and locations the regex engine does not target at all. The
 comparison is a fair _speed_ comparison on identical inputs and entity
 types, not an accuracy study.
 
-## How these map to published claims
-
-- **"~31µs per request" (LiteLLM guardrail, project README /
-  datafog.ai)** — not reproduced _as worded_. ~31–43 µs is the cost of
-  scanning/redacting **one short message**; a full 2-message request
-  through `async_pre_call_hook` is ~120 µs clean / ~220 µs with redaction
-  and litellm logging. Still 100–1000x below a sidecar's network hop, but
-  the claim should be reworded to per-message, e.g. _"~40µs per message —
-  a request clears the guardrail in well under a millisecond"_. Reproduce:
-  `python benchmarks/run.py --suite guardrail`.
-- **"~70ms per invocation including process startup" (Claude Code
-  hook)** — reproduced: 69–89 ms median across runs on this machine
-  (~70 ms idle, higher under load), dominated by interpreter startup.
-  _"~70–90ms"_ (or "under 100ms") is the defensible phrasing. Reproduce: `python benchmarks/run.py --suite hook`.
-- **"190x performance advantage" (PyPI tagline)** — not reproduced at
-  190x by this suite. Measured: 103–170x vs Presidio and 114–140x vs spaCy
-  NER, varying by payload. The historical 190x came from a different
-  (13.3 KB, entity-dense) document and engine configuration. Honest
-  phrasings this suite supports: _"100x+ faster than NER-based PII
-  detection"_ or _"up to 170x faster than Presidio on identical
-  payloads"_. Reproduce: `python benchmarks/run.py --suite spacy,presidio`.
-
 ## Fairness notes / known limitations
 
 - Single-threaded, single-machine, synthetic payloads. Real chat traffic

docs/contributing.rst

@@ -1,32 +1,32 @@
-=======================================
-Contributor Setup And 4.5 Release Flow
-=======================================
+==================================
+Contributor Setup And Release Flow
+==================================
 
-This page is the contributor runbook for DataFog 4.5 work. It is meant for
+This page is the contributor runbook for DataFog 4.x work. It is meant for
 humans and agents preparing local changes, choosing verification commands, and
-understanding where the 4.5 release boundary sits.
+understanding where the current release boundary sits.
 
 Version Frame
 =============
 
 Current release planning uses this frame:
 
-* Stable package release: ``4.4.0``.
-* Current development package version: ``4.4.0a5``.
-* Next minor target: ``4.5.0``.
+* Stable package release: ``4.8.0``.
+* Current development package version: ``4.8.0``.
+* Next major target: ``5.0.0``.
 
-Do not bump routine feature, documentation, or cleanup branches directly to
-``4.5.0``. Keep the version stable during local release-prep work, then handle
-the final version and release-note alignment in the release-readiness slice.
+Do not bump routine feature, documentation, or cleanup branches directly to a
+future release number. Keep the version stable during local work, then handle
+final version and release-note alignment in the dedicated release slice.
 
 Python Environments
 ===================
 
 DataFog currently declares support for Python ``>=3.10,<3.14``. The CI matrix
 tests core, NLP, and NLP-advanced installs on Python 3.10, 3.11, 3.12, and
 3.13. OCR profile smoke checks also run on Python 3.13 with system Tesseract
-installed. Distributed and all-profile Python 3.13 validation remain outside
-the 4.5 support claim.
+installed. Distributed and all-profile installs remain heavier optional
+profiles rather than the lightweight core support claim.
 
 Create one virtual environment per Python version when you need to compare
 profiles locally:
@@ -171,41 +171,35 @@ Live Modules
 ============
 
 Use :doc:`live-module-map` before changing core package structure. It lists the
-live 4.5 modules for each concept and the historical ``*_lean`` and
+live 4.x modules for each concept and the historical ``*_lean`` and
 ``*_original`` files that are kept only as non-live compatibility/audit
 artifacts.
 
-4.5 Release Flow
-================
+Release Flow
+============
 
-The 4.5 work lands as focused pull requests into ``dev``. Keep feature and docs
-branches narrow, and avoid mixing local cleanup, external PR review, and final
-release mechanics in one branch.
+4.x and v5 preparation work lands as focused pull requests into ``dev``. Keep
+feature and docs branches narrow, and avoid mixing local cleanup, external PR
+review, and final release mechanics in one branch.
 
-The release flow for 4.5 is:
+The release flow is:
 
-1. Land the local release-prep baseline and follow-up cleanup/docs slices.
-2. Review the external German regex PR after the local release-prep baseline is
-   in place.
-3. Integrate German regex support only if review says it fits the 4.5
-   lightweight text screening thesis.
-4. Validate optional Python 3.13 profiles before claiming support beyond core
-   SDK and CLI.
-5. Prepare release readiness with :doc:`v45-release-readiness`: changelog and
-   release notes, package checks, docs build, CI state, and version alignment.
-6. Bump or override the final stable release to ``4.5.0`` only during the
-   release-readiness and stable-release path.
+1. Land focused feature, fix, or docs slices into ``dev``.
+2. Update the changelog, release notes, package checks, docs build, CI state,
+   and version alignment in a dedicated release-readiness change.
+3. Validate optional Python 3.13 profiles before expanding support claims.
+4. Bump or override the final stable release version only during the
+   stable-release path.
 
 The current release workflow strips prerelease suffixes from the package
-version unless a manual stable ``version_override`` is provided. For the final
-4.5 stable release, use a dedicated release-readiness change or the stable
-workflow override so the published version is ``4.5.0`` rather than another
-``4.4.0`` prerelease line.
+version unless a manual stable ``version_override`` is provided. Use a
+dedicated release-readiness change or the stable workflow override when the
+published version should move to a new stable line.
 
 External PR Boundary
 ====================
 
-The external German PII regex PR belongs after local baseline cleanup. Review
-it as a 4.5 candidate, not as a v5 planning shortcut. If accepted, adapt it in
-the German regex integration slice with tests, documentation of locale
-coverage, and no new dependency burden on the core path.
+Review external PII regex PRs against the current lightweight text screening
+thesis, not as v5 planning shortcuts. If accepted, adapt them with tests,
+documentation of locale coverage, and no new dependency burden on the core
+path.

docs/getting-started.rst

@@ -123,11 +123,11 @@ The CLI core path is text-first:
 Image commands are optional. Install ``datafog[ocr]`` for local OCR and
 ``datafog[web,ocr]`` when the CLI needs to download image inputs.
 
-What 4.5 Is Not
+What 4.x Is Not
 ===============
 
-DataFog 4.5 prepares the package for future middleware use cases, but it does
-not ship dedicated Sentry, OpenTelemetry, logging-framework, or cloud DLP
+DataFog 4.x prepares the package for broader middleware use cases, but it does
+not yet ship dedicated Sentry, OpenTelemetry, logging-framework, or cloud DLP
 adapters. Those integrations are future-facing work built on the same core
 text screening path.
 
@@ -137,4 +137,4 @@ Next Pages
 * :doc:`python-sdk` documents the Python API surface.
 * :doc:`cli` documents command-line usage.
 * :doc:`optional-surfaces` documents OCR and Spark install notes.
-* :doc:`roadmap` explains how 4.5 leads toward later middleware work.
+* :doc:`roadmap` explains how the 4.x line leads toward later middleware work.

docs/roadmap.rst

@@ -2,7 +2,7 @@
 Release Roadmap
 ================
 
-Where DataFog is today (4.7.x) and where it is going (v5.0.0). The 4.x
+Where DataFog is today (4.8.x) and where it is going (v5.0.0). The 4.x
 line delivered the lightweight-core architecture and, from 4.6.0 on, an
 offline PII firewall for AI agents and gateways. The v5 cycle turns that
 foundation into the fastest, easiest offline PII firewall for AI apps,
@@ -12,13 +12,13 @@ logs, and datasets.
    :local:
    :depth: 1
 
-Current status — 4.7.x
+Current status — 4.8.x
 ----------------------
 
-DataFog ``4.7.0`` is the current stable release (July 2026). Patch
-releases on the 4.7.x line focus on detection precision — reducing
-SSN/phone false positives surfaced by real-world agent traffic — while
-v5.0.0 is prepared.
+DataFog ``4.8.0`` is the current stable release (July 2026). Patch
+releases on the 4.8.x line focus on redaction correctness, detection
+precision, and keeping the 4.x agent/gateway firewall stable while v5.0.0
+is prepared.
 
 Every published performance number is reproducible with one command:
 ``python benchmarks/run.py`` (see ``benchmarks/`` for methodology,
@@ -112,6 +112,17 @@ Both default to the high-precision entity set (``EMAIL``, ``PHONE``,
 4.7.x patch releases: SSN and phone precision fixes (hex-string and
 no-dash false positives).
 
+✅ 4.8.0 — Redaction Correctness (Released July 2026)
+------------------------------------------------------
+
+* **Document-order redaction tokens**: repeated entities are numbered from
+  left to right, so the first email in a document becomes ``[EMAIL_1]``.
+* **Overlap-safe redaction**: public ``redact()`` calls now suppress
+  overlapping and duplicate spans before applying replacements, avoiding
+  corrupted output fragments.
+* **Mask mapping preservation**: same-length values no longer collide in
+  ``mask`` strategy mappings; each masked value receives a stable indexed key.
+
 v5.0.0 — Offline PII Firewall for AI Apps (In Progress)
 --------------------------------------------------------
 

examples/quick_start.ipynb

@@ -16,7 +16,7 @@
   {
    "cell_type": "markdown",
    "metadata": {},
-   "source": "# DataFog Quick Start Guide\\n\\n> **📦 Version Requirement**: This guide is for DataFog v4.2.0 and above\\n> \\n> ✅ **New in v4.2.0**: GLiNER integration, smart cascading, and enhanced performance\\n\\nWelcome to DataFog! This notebook demonstrates how to get started with DataFog's fast PII detection and anonymization capabilities.\\n\\n## What makes DataFog special?\\n\\n- **🚀 Ultra-Fast**: 190x faster than spaCy for structured PII, 32x faster with GLiNER\\n- **🪶 Lightweight**: <2MB core package with optional ML extras\\n- **🧠 Smart Engines**: Choose from regex, GLiNER, spaCy, or smart cascading\\n- **📦 Production Ready**: Comprehensive testing and performance validation\"",
+   "source": "# DataFog Quick Start Guide\\n\\n> **📦 Version Requirement**: This guide is current for DataFog v4.8.0 and above\\n> \\n> ✅ **4.x Highlights**: GLiNER integration, smart cascading, agent guardrails, allowlists, and redaction correctness fixes\\n\\nWelcome to DataFog! This notebook demonstrates how to get started with DataFog's fast PII detection and anonymization capabilities.\\n\\n## What makes DataFog special?\\n\\n- **🚀 Ultra-Fast**: 190x faster than spaCy for structured PII, 32x faster with GLiNER\\n- **🪶 Lightweight**: <2MB core package with optional ML extras\\n- **🧠 Smart Engines**: Choose from regex, GLiNER, spaCy, or smart cascading\\n- **📦 Production Ready**: Comprehensive testing and performance validation\"",
    "outputs": []
   },
   {
@@ -624,4 +624,4 @@
  },
  "nbformat": 4,
  "nbformat_minor": 4
-}
+}

setup_lean.py

@@ -1,6 +1,6 @@
 from setuptools import find_packages, setup
 
-# Historical shadow packaging snapshot. The live DataFog 4.5 packaging input is
+# Historical shadow packaging snapshot. The live DataFog packaging input is
 # setup.py. Do not use this file for builds, releases, or dependency changes.
 
 # Read README for the long description

setup_original.py

@@ -1,6 +1,6 @@
 from setuptools import find_packages, setup
 
-# Historical shadow packaging snapshot. The live DataFog 4.5 packaging input is
+# Historical shadow packaging snapshot. The live DataFog packaging input is
 # setup.py. Do not use this file for builds, releases, or dependency changes.
 
 # Read README for the long description