Skip to content

Commit eef05e3

Browse files
authored
Update README.md to reflect accurate performance metrics
Removed outdated performance claims and rephrased metrics for clarity.
1 parent 8385682 commit eef05e3

1 file changed

Lines changed: 0 additions & 22 deletions

File tree

benchmarks/README.md

Lines changed: 0 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -116,28 +116,6 @@ detect names and locations the regex engine does not target at all. The
116116
comparison is a fair _speed_ comparison on identical inputs and entity
117117
types, not an accuracy study.
118118

119-
## How these map to published claims
120-
121-
- **"~31µs per request" (LiteLLM guardrail, project README /
122-
datafog.ai)** — not reproduced _as worded_. ~31–43 µs is the cost of
123-
scanning/redacting **one short message**; a full 2-message request
124-
through `async_pre_call_hook` is ~120 µs clean / ~220 µs with redaction
125-
and litellm logging. Still 100–1000x below a sidecar's network hop, but
126-
the claim should be reworded to per-message, e.g. _"~40µs per message —
127-
a request clears the guardrail in well under a millisecond"_. Reproduce:
128-
`python benchmarks/run.py --suite guardrail`.
129-
- **"~70ms per invocation including process startup" (Claude Code
130-
hook)** — reproduced: 69–89 ms median across runs on this machine
131-
(~70 ms idle, higher under load), dominated by interpreter startup.
132-
_"~70–90ms"_ (or "under 100ms") is the defensible phrasing. Reproduce: `python benchmarks/run.py --suite hook`.
133-
- **"190x performance advantage" (PyPI tagline)** — not reproduced at
134-
190x by this suite. Measured: 103–170x vs Presidio and 114–140x vs spaCy
135-
NER, varying by payload. The historical 190x came from a different
136-
(13.3 KB, entity-dense) document and engine configuration. Honest
137-
phrasings this suite supports: _"100x+ faster than NER-based PII
138-
detection"_ or _"up to 170x faster than Presidio on identical
139-
payloads"_. Reproduce: `python benchmarks/run.py --suite spacy,presidio`.
140-
141119
## Fairness notes / known limitations
142120

143121
- Single-threaded, single-machine, synthetic payloads. Real chat traffic

0 commit comments

Comments
 (0)