Skip to content

automata: fix leftmost-first miss in the reverse suffix optimization#1373

Open
RECHE23 wants to merge 1 commit into
rust-lang:masterfrom
RECHE23:reverse-suffix-leftmost
Open

automata: fix leftmost-first miss in the reverse suffix optimization#1373
RECHE23 wants to merge 1 commit into
rust-lang:masterfrom
RECHE23:reverse-suffix-leftmost

Conversation

@RECHE23

@RECHE23 RECHE23 commented Jul 5, 2026

Copy link
Copy Markdown

The reverse suffix optimization can skip a match that starts earlier than the matches it reports, violating leftmost-first semantics.

Repro (1.12.4, default config)

let re = regex::Regex::new("a|.aa").unwrap();
let spans: Vec<_> = re.find_iter("-#aa").map(|m| (m.start(), m.end())).collect();
assert_eq!(spans, vec![(1, 4)]);   // FAILS today: [(2, 3), (3, 4)]

[1,4) (".aa" matching #aa) is a match and starts leftmost. (Original fuzz find: \0|.\0\0 on "\0\0\0\0\n\x0b\0\0\0", missing (5,8); minimized by hand.)

Root cause

With RUST_LOG=trace: suffixes extract as Seq[E("a"), I("aa")], the LCS prefilter is memchr('a'), and ReverseSuffix is engaged. Candidates are processed in order of match end, and the first reverse+forward confirmed match is yielded. That is only correct when the leftmost-starting match also has the smallest candidate end at/after the resume point. Here branch 1's entire match (E("a")) is a proper suffix of branch 2's suffix ("aa"): a branch-1 match nests inside the tail of an overlapping branch-2 match that ends later but starts earlier. Candidate end 3 confirms (2,3) and is yielded; the true leftmost [1,4) (candidate end 4) is then behind the (correct in itself) anti-quadratic min_start clamp, so the miss is permanent.

The fix

Conservative: decline the optimization in ReverseSuffix::new when any exact suffix literal is a proper suffix of another suffix literal — the precise nesting that breaks the end-order ⇒ start-order assumption — falling back to the core engine. Common beneficiaries keep the optimization (single-branch suffixes, all-inexact seqs such as \w+@\w+, non-nested alternations like foo|bar).

A complete fix that keeps the optimization for these patterns would need candidate look-ahead across ends before yielding when overlap is possible; happy to explore that as a follow-up if you'd prefer.

Validation

  • New regression test in tests/regression_fuzz.rs (fails before, passes after).
  • cargo test --test integration (66) and cargo test -p regex-automata --lib (137): pass.

Found by a differential fuzzer (REAL-regex vs this crate).

The reverse suffix optimization prefilters on the longest common suffix
and processes candidates in order of match end, yielding the first
reverse+forward confirmed match. That is only correct when the leftmost
starting match also has the smallest candidate end. When one exact
suffix literal (an entire match of one alternation branch) is a proper
suffix of another suffix literal, a match can be nested inside the tail
of an overlapping match that ends later but starts earlier: the first
confirmed candidate is then not leftmost, and the (correct in itself)
anti-quadratic min_start clamp makes the missed match permanent.

For example, 'a|.aa' on '-#aa' reported [(2,3), (3,4)] instead of the
leftmost-first [(1,4)].

Fix conservatively: decline the optimization when any exact suffix
literal is a proper suffix of another suffix literal, falling back to
the core engine. Common beneficiaries (single branches, all-inexact
seqs such as '\w+@\w+', non-nested alternations like 'foo|bar') keep
the optimization.

Found by a differential fuzzer (REAL-regex vs this crate); minimized by
hand.
Copilot AI review requested due to automatic review settings July 5, 2026 16:29

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

RECHE23 added a commit to RECHE23/real-regex that referenced this pull request Jul 5, 2026
The upstream fix is up as a PR (rust-lang/regex#1373). The root cause is sharper than "prefilter resume": the
reverse-suffix optimization orders candidates by their END, but when one alternation branch's exact literal is
a proper suffix of another branch's match, ordering by end breaks the leftmost-by-START rule (the [E("a"),
I("aa")] shape). fuzz/known_rust_bugs/README and the crate README's divergence note now point at #1373 and
carry the corrected root cause — the "found by REAL's differential fuzzer" line now lives in their tracker.
RECHE23 added a commit to RECHE23/real-regex that referenced this pull request Jul 5, 2026
…s record — INNER-LITERAL IL.3

The theme closer. Re-running the duel with the route on gives the honest verdict, and one deferral turned the
headline line to parity: the reverse DFA is now built LAZILY, on the first candidate — a no-match haystack (the
literal never appears) pays only the memmem and never builds a reverse it would not use. The date
`\d{4}-\d{2}-\d{2}` no-match line went from ~201x rust to 0.020 vs 0.013 ns/B — 1.6x, the V0 target of <=2x met.

The duel gains the sparse rows the V0 debt named (date-sparse 4.2x, email-sparse 7.2x — the reverse+confirm
per match trails rust's tighter loop but sits far from 201x) and the key= flagship (1.9x). The class / digit /
field rows are unchanged (REAL still 5-6x ahead): the route only fires for a required inner literal, and the
exhaustive confirms it byte-identical to the core (serious=0, route on, 3.21M cases). BENCHMARKS.md §E.5
records the table.

design.dox §7.8 documents the full arc (extract -> boundary -> prefix compile -> reverse -> loop), the two
traps (the reverse DFA's dangling byte-program span; the dynamic VM's own state type leaving the route inert)
and the lesson verbatim — the reverse bound advances only on a yield, the linearity guard is the forward
backstop, not the reverse bound; it read as a missing heuristic but was a one-line wiring bug. It cites
rust-lang/regex#1373 (their sibling leftmost bug, found by REAL's fuzzer — the symmetry is instructive) and
names the alternation-sibling and multi-literal follow-ups. Full gate green.
RECHE23 added a commit to RECHE23/real-regex that referenced this pull request Jul 5, 2026
…3rd differential fuzzer, crate perf)

The first large train of the accumulation rhythm. Bumps the five version surfaces (pyproject, __init__,
version.hpp, Cargo.toml, Cargo.lock; CMake derives) to 2026.7.22 and adds the release notes.

Headline: the inner-literal prefilter decapitates the date line — `\d{4}-\d{2}-\d{2}` no-match went from 201x
of the regex crate to 1.9x (sparse haystacks 30-100x better, key= now beats the crate, dense email at parity,
class/digit/field unchanged; exhaustive serious=0 with the route on). Plus the third differential fuzzer
(real-regex vs the regex crate) and its four fixed divergence classes (empty-match iteration, $ end-only flag,
class-set-syntax declines, and the upstream rust-lang/regex#1373 it found), and find_iter now outruns the
crate on span throughput. Full notes in docs/release-notes/v2026.7.22.md. Full gate green.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants