Remove prefilter-bypassing search modes (regex / --any / -L)#33
Merged
Conversation
…toggles why: search --regex, search --any, and grep -L defeat the ripgrep prefilter and force a (near) full-corpus parse that materialises every matching record in memory. Until the engine can serve them within bounded memory, exposing the toggles advertises capability we do not back at scale. what: - Remove the search --regex / --any flags and the grep -L / --files-without-match flag, plus their SearchArgs / GrepArgs fields, builders, and the grep files-without-match render path. - Drop the regex / any_term parameters from the MCP search and validate_query tools and their request/echo models; stop advertising them in the server instructions. - Keep SearchQuery.regex / any_term: grep and find stay regex-by-default (ripgrep handles that), the query language and TUI still use them, and they remain the seam for re-adding the modes later. - Replace the obsolete flag tests with cases asserting the flags are now rejected; refresh the affected GrepArgs / SearchArgs test helpers.
why: The CLI search --regex / --any flags, the MCP search and validate_query regex / any_term parameters, and grep -L are no longer part of the surface, so the reference pages and the docs-only FastMCP shim must stop advertising them. what: - grep page: drop the "files without matches" section, remove -L from the eager-output list, and narrow the -v note to -c only. - FastMCP docs shim: mirror the trimmed search / validate_query signatures (literal AND terms, no regex / any_term). - Drop the contradictory "match any term" tutorial example and the regex mention in the validate_query "use when" note.
tony
changed the title
--any / -L)
tony
changed the title
--any / -L)
tony
changed the title
tony
force-pushed
the
drop-fullscan-search-modes
branch
from
a299af2 to
747f447
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three search modes that bypassed the ripgrep prefilter (
prefilter_sources_by_root)—triggering a full-corpus parse that materialized every record in memory (~1.3 GB RSS)—have been removed. Until these modes can be served within bounded memory, their public interface is suspended to ensure system stability:search --regexandsearch --anyregexandany_termparameters for thesearchandvalidate_querytools (and their associated models)grep -L/--files-without-matchagentgrep searchis now strictly literal AND-matching.grepandfindremain unaffected and regex-by-default, as they utilize ripgrep directly and avoid the memory bottleneck. The internalSearchQuery.regexandany_termfields are preserved for the query language, TUI, and future re-implementation.This change addresses a performance cliff inherent in the current in-memory matcher. Re-introducing these modes with bounded memory is tracked in #32.
Test plan
ruff check,ruff format,ty check,pytest --reruns 0(880 passed),just build-docs.search --regex/--any,grep -Lexit 2 with "unrecognized arguments") and that the MCPsearch/validate_querytools no longer acceptregex/any_term.agentgrep search foo --regex|--anyandagentgrep grep -L fooexit 2;agentgrep search fooandagentgrep grep foostill work.