feat: read any UTF-8 file and search HTML pages and scripts by aliasunder · Pull Request #746 · aliasunder/vault-cortex

aliasunder · 2026-11-07T06:35:32Z

What this changes

vault_read_file reads any UTF-8 text file, and vault_search finds HTML pages by their visible text and scripts by keyword. Files with an uppercase extension, such as a scanner's .PDF, are now searchable too.

Closes #738.

Reads

Any file outside the image, canvas, PDF and text-format types returns its source as text: HTML pages, .py/.sh/.ps1 scripts, files with an unlisted extension or none.
A file holding a NUL byte in its first 8,000 bytes (git's test for binary data, which also catches UTF-16 text) is refused with unsupported file type, and the message names the readable types. Bytes that are not UTF-8 get not valid UTF-8.
The existing text formats (.txt, .csv, .json, .log, …) keep their path with no NUL check, so a NUL-padded log still reads.
Both paths drop a leading byte-order mark, as main already did for the text formats; the tool text and docs now say so.

Search

Kind	Indexed text	Embedded
HTML (`.html`/`.htm`)	The first 100 KiB of the text a browser shows; a page over 16 MiB is never read and is found by name only	Yes
Scripts (`.py`/`.sh`/`.ps1`)	The first 100 KiB of the source; a script over 16 MiB is never read and is found by name only	No

HTML extraction uses htmlparser2's tokenizer, which keeps no element stack, so a page of unclosed tags costs no more than a closed one (1 MB of unclosed <div><span><b> in well under a second). Hidden elements end only at a closing tag with their own name, so a stray </noscript> never shows a template's text. An inline SVG's title and desc tooltips, which a browser never shows on the page, are left out too.
HTML pages and scripts are indexed only when they pass the same UTF-8 test reads apply. A watched file that stops passing loses its stored content.
Scripts stay out of embedding: the markdown chunker reads # comment lines as headings. The startup embedding pass now loads one file's text at a time instead of every file's at once.
Indexing reads no more of a file than search keeps. A text-format file (.txt, .csv, .log, …) is read only to its first 100 KiB, the part search indexes, so one very large log no longer loads whole into memory; its search results don't change. A script, like an HTML page, is read whole for the UTF-8 test, so a script over 16 MiB is never read and is found by name only.

Fixed: uppercase file extensions

One file-kind helper (obsidian-markdown/file-kinds.ts) maps the lowercased extension to a kind, and every extension check in reads, indexing, the file watcher, the note mover and the link tools uses it. Before this, Report.PDF, notes.TXT and Board.Canvas were never searchable, and a .Canvas's file cards were neither links nor rewritten when a note moved. Search results now report the extension lowercased, as vault_list_files does.

Fixed: rebuild time grew quadratically with notes and files

FTS5 cannot index the path column, so deleting a note's or file's full-text row by path scanned the whole table, once per note or file during each rebuild. Each notes_fts and file_content_fts row now shares its source row's rowid, and the upsert updates in place (ON CONFLICT (path) DO UPDATE … RETURNING rowid), so the FTS row is found by rowid. With 16,046 scripts indexed, the rebuild dropped from 1,171 s to 57 s, and 8,000 notes no longer spend about 48 s in upserts.

Tool text

vault_read_file describes any UTF-8 text file as readable, names the binary-data error, and lists the extensions vault_search indexes.
vault_list_files names the same extensions in its contrast with search, its readable-types line drops the closing list, and its first example drops a gloss that repeated the folder parameter's description.
vault_search names HTML pages and .py/.sh/.ps1 scripts among the file content it searches. Its Behavior line says each file's first 100 KiB of text is searched, a page or script over 16 MiB is found by name only and, while semantic matching runs, scripts match by keyword only. It also names search_mode "fts" by its value there, leaving the definition to Returns.
The extension lists are built from the file-kind map, and the 100 KiB and 16 MiB figures from the indexing constants, so they follow the code. The default tool list grows by 411 characters, within the size cap.

Docs

README's Files section, ARCHITECTURE's read dispatch and file-content index, SECURITY's read scope, the CLI's npm page, the SYNC_FILE_TYPES wording on every surface, and the DeepWiki module lists. DOCKERHUB.md now leaves out the README's Files section, as it does Memory and Tasks, to stay under Docker Hub's 25,000-byte cap.

Verification

Each phase's new tests fail on the code before it and pass after; mutation checks on the HTML extractor's name match, lowercasing and cap, the watcher's content removal, the script embedding gate and both rowid pairings each fail their own test.
Chunk count on Python's argparse documentation page: 31 chunks per 100 KiB of visible text.
Tool-definition review on default.json before each push that changed tool text (vault_read_file, vault_list_files, vault_search): every defect found was fixed (among them an undefined "text-format", a mechanism clause, an ambiguous keyword-only clause and limits listed among content types), and the final pass at b8bcb7ea, after the merge with main, found none and no unintended text changes.
Delta reviews (bug check, test audit, code quality) on the merge with main and the read-size change. Their fixes include matching readFile at a file's end, a test that reads the start of a sparse 3 GiB file readFile would refuse, and the 16 MiB boundary. A mutation check that reverts the read-size change fails its own tests.

Search ranking and cost A/B

npm run eval:search on a private copy of a real 2,888-note vault, with the Python standard library and 16,046 third-party .py files (231 MB) added, plus a planted script and HTML page for each precision query and one positive control per new kind. Main and this branch ran back to back on the same snapshot.

Controls: the planted script and page rank Pin Node 22 with strict engines, harden mermaid subgraph IDs #2 and feat: Phase 1 scaffold (vault-cortex remote MCP server) #1 on this branch and are missed on main.
Recall: every recall query keeps its result, and every query whose top result was a note keeps that note at feat: Phase 1 scaffold (vault-cortex remote MCP server) #1.
Precision: a planted page and script holding a query's exact words enter its top 5, but so does a .txt file with the same words on main, because text files are already indexed. Scripts rank below that text file and HTML one place above. No real script or page reached a precision query's top 5.
Cost: steady-state rebuild 39 s → 57 s, peak memory 762 → 861 MB, the first boot after upgrading +37 s once, index database 219 → about 820 MB.

HTML pages at scale

A second run added Python 3.15's HTML documentation to the same snapshot: 596 real pages, 12.3 MB of visible text in 4,471 chunks. Main at 2515ddd8 and this branch at a0c4113d each kept their own copy of the index and ran alternately, with the machine's load average between 3 and 24.

Ranking: compared with this branch's run before the pages were added, every query keeps its top result, and every expected note keeps its rank or moves up. A docs page reached a top 5 in 10 of 69 readings (23 queries at limits 20, 5 and 3), for 5 of the 23 queries. None was at feat: Phase 1 scaffold (vault-cortex remote MCP server) #1, and none appeared in a precision or sentinel query's top 5.
- In six readings the page sat at chore: add version 0.1.0 to package.json #5, in place of a PDF or another page.
- In the other four, a vague request for open tasks put Python's asyncio Task and queue pages at Pin Node 22 with strict engines, harden mermaid subgraph IDs #2 to fix(ci): add default bump value for mobile workflow_dispatch #4, below the right task board at feat: Phase 1 scaffold (vault-cortex remote MCP server) #1.
Search latency: unchanged. Warm queries at limit 20 took 704 and 715 ms median here against 725 and 764 ms on main. The reranker takes almost all of that: with it off, queries took 32 to 38 ms, with file results on or off.
Startup: the keyword index the server builds before it takes requests took 34, 49 and 51 s here against 14, 21 and 25 s on main. The full rebuild, embedding included, took 51 to 69 s against 25 to 36 s. The docs pages account for 1 to 2 s of that: runs with and without them took 52 and 47 s against 50 and 46 s. The rest is the 16,046 scripts, which are read whole and re-indexed on every boot.
First boot after upgrading: embedding the docs pages' 4,471 chunks took about 14 minutes (841 and 862 s), once. It runs in the background: the server answers requests throughout, and the pages are searchable by keyword from the start.
Memory and size: peak memory 829 to 932 MB here against 650 to 763 MB on main. The docs pages added about 90 MB to the index.

Live Test Results

Head: af54246

This round covers the merge with main, the change that limits how much of a file indexing reads, and dropping inline SVG tooltips from a page's indexed text. Every text-format, script and HTML file in the instance's index was hashed before and after each deploy, and the deployed tool list was compared with the committed snapshot.

Probe	Before (`:test` of this PR at `90be8c4a`)	After (`af542469`)
Indexed text of each of the instance's 16 text-format and script files, as SHA-256	recorded	identical for all 16, and identical to the `:test` of `7b480c3d`, which still read every file whole
The three of those files over 100 KiB (CSV files of 113 KB, 994 KB and 3.7 MB), which this build reads only to their first 100 KiB	102,400 bytes indexed each	the same, byte for byte
Indexed text of the instance's two real HTML pages (2.3 MB and 3.1 MB), neither holding an inline SVG tooltip	recorded	identical for both
Deployed `tools/list` against the committed `obsidian-sync.json` snapshot	all 33 tools match (the `:test` of `7b480c3d` before it differed in 26)	all 33 tools match

The previous round, at 7b480c3d (test-deploy run 42363319280), probed reads and search of synced throwaway .py, .sh, .ps1 and .html files. Reads returned each file's source; search found each script by its identifier and the page by its visible text, but not by a word only inside its <script>. The FTS rowid pairing also checked out on the instance's index. This round changes neither path for files under 16 MiB, except that an inline SVG's title and desc no longer count as page text.

On the instance: test-deploy run 42406045787 built af542469. The image's revision label is af542469692d9e655ca75993bef50c2699f5d889, and the container runs digest sha256:1b0ca98e9df686beea622f89d9f3243d7a15e00208b3e07013a99f7e5d255e0d, re-created at 19:27:28 UTC after the run built the image.

🤖 Generated with Claude Code

vault_read_file now returns HTML pages, scripts and files with no extension as text. A file outside the dedicated types is refused when a NUL byte sits in its first 8,000 bytes (binary data or UTF-16 text) or when its bytes are not UTF-8. The dedicated text formats keep their path with no NUL check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…xtension One file-kind helper (obsidian-markdown/file-kinds.ts) maps a lowercased extension to a kind, and every extension check in reading, indexing, the note mover and the link tools uses it. A scanner's .PDF, an .TXT or a .CSV is now searchable, a .Canvas's file cards count as links and are rewritten when a note moves, and search results report the extension lowercased, as vault_list_files does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

extractHtmlVisibleText runs htmlparser2's stackless tokenizer, so unclosed tags cost no more than closed ones. Hidden elements end only at a closing tag with their own name, block elements become line breaks, whitespace collapses, and reading stops once the text reaches the cap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

HTML pages are read as bytes and indexed only when they decode as UTF-8 with no early NUL byte, the test vault_read_file applies. Search holds a page's visible text, capped like other files; a page over 16 MiB is found by its name only. When a watched page stops reading as text, its stored content is removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…tcher Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

.py, .sh and .ps1 files are indexed for keyword search through the same UTF-8 test as HTML pages. Embedding skips them before their text loads, because the markdown chunker reads their comment lines as headings. The startup embedding pass now loads each file's text only when it reaches that file, instead of holding every file's text at once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

README's Files section, ARCHITECTURE's read dispatch and file-content index, SECURITY's read scope, the SYNC_FILE_TYPES wording on every surface, and the DeepWiki module lists now cover any-UTF-8 reads, HTML visible text and keyword-only scripts. DOCKERHUB.md leaves the README's Files section out, as it does Memory and Tasks, to stay under Docker Hub's 25,000-byte cap; its feature list still links there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

vault_read_file names any UTF-8 text file as readable, the binary-data error, and search coverage of HTML pages and scripts; vault_list_files' readable-types sentence drops its closing list; vault_search names HTML pages and scripts among the file content it searches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… tree Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…y keyword The file tools now name the extensions vault_search indexes, so 'text format' has a referent once reads cover any UTF-8 file; the binary-data error entry drops how the server detects binary data; vault_search says scripts match by keyword only while semantic matching runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The searched-extension lists in the file tools now come from the file-kind map, so they follow it when a type is added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

socket-security · 2026-11-07T06:35:56Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	htmlparser2@12.0.0

View full report

umm-actually · 2026-11-07T06:41:19Z

Update ARCHITECTURE's pass 2 file-content list and diagram for HTML and scripts
Low severity · correctness · high confidence

ARCHITECTURE.md:538 — beyond the diff's line ranges, in code the changes touch or depend on.

The Pass 2 step and the indexing-pipeline mermaid node still say file-content indexing covers only 'canvas, PDF, text', but this PR adds HTML pages and scripts to that pass. The same document's Files section and Query fusion step now list HTML and scripts, so ARCHITECTURE.md gives contradictory answers about which file kinds pass 2 indexes.

Failure scenario: A contributor or doc-derived reader (DeepWiki, agent) consulting ARCHITECTURE.md → Indexing to learn which file kinds rebuild pass 2 indexes sees only 'canvas, PDF, text' and concludes HTML pages and scripts are not FTS-indexed, contradicting the Files section's table in the same file and the search-index.ts implementation.

Suggested fix

Update the Pass 2 parenthetical to '(canvas, PDF, text, HTML, script → FTS5)' and the mermaid node to 'Index File Content\ncanvas + PDF + text + HTML + script → FTS5'.

umm-actually · z-ai/glm-5.3-flash

umm-actually · 2026-11-07T06:41:20Z

Add the end-to-end test the 'not valid UTF-8' Errors entry requires
Low severity · tests · medium confidence

src/vault-mcp/mcp-core/tools/asset-tools.ts:138 — beyond the diff's line ranges, in code the changes touch or depend on.

The tool's Errors section promises the exact text clients receive, and this error now fires for a much wider surface — every HTML page, script, and unlisted-extension read — yet no test in server-error-contracts.test.ts drives a schema-valid call that returns it, so the quoted wording can drift unobserved at the transport level.

Failure scenario: The message is reworded (e.g. to 'not UTF-8') in the data layer while the tool description keeps the old wording; clients matching on the documented text misread refusals and no test fails.

Suggested fix

Add a server-error-contracts test that plants a Latin-1 .html file and asserts the exact '[Error]: not valid UTF-8: "…" cannot be returned as text' result, mirroring the neighboring binary-file test.

umm-actually · z-ai/glm-5.3-flash

umm-actually · 2026-11-07T06:41:21Z

umm-actually re-reviewed at c6f5b56

1 new finding(s) posted (21 tracked finding(s) across all runs).

Context notes

Priority docs already in context: ARCHITECTURE.md, SECURITY.md, deploy/railway/README.md, deploy/render/README.md, deploy/remote/README.md
15 changed file(s) excluded from review: package-lock.json (built-in default list), src/vault-mcp/mcp-core/__tests__/__snapshots__/tool-surface/default.json (diff_exclude_paths input), src/vault-mcp/mcp-core/__tests__/__snapshots__/tool-surface/disabled-tools.json (diff_exclude_paths input), src/vault-mcp/mcp-core/__tests__/__snapshots__/tool-surface/embedding-off.json (diff_exclude_paths input), src/vault-mcp/mcp-core/__tests__/__snapshots__/tool-surface/file-tools-off.json (diff_exclude_paths input), src/vault-mcp/mcp-core/__tests__/__snapshots__/tool-surface/memory-off+embedding-off.json (diff_exclude_paths input), src/vault-mcp/mcp-core/__tests__/__snapshots__/tool-surface/memory-off+file-tools-off.json (diff_exclude_paths input), src/vault-mcp/mcp-core/__tests__/__snapshots__/tool-surface/memory-off.json (diff_exclude_paths input), src/vault-mcp/mcp-core/__tests__/__snapshots__/tool-surface/obsidian-sync.json (diff_exclude_paths input), src/vault-mcp/mcp-core/__tests__/__snapshots__/tool-surface/readonly+embedding-off.json (diff_exclude_paths input), src/vault-mcp/mcp-core/__tests__/__snapshots__/tool-surface/readonly+file-tools-off.json (diff_exclude_paths input), src/vault-mcp/mcp-core/__tests__/__snapshots__/tool-surface/readonly+memory-off+embedding-off.json (diff_exclude_paths input), src/vault-mcp/mcp-core/__tests__/__snapshots__/tool-surface/readonly+memory-off+file-tools-off.json (diff_exclude_paths input), src/vault-mcp/mcp-core/__tests__/__snapshots__/tool-surface/readonly+memory-off.json (diff_exclude_paths input), src/vault-mcp/mcp-core/__tests__/__snapshots__/tool-surface/readonly.json (diff_exclude_paths input)

umm-actually · z-ai/glm-5.3-flash

… the file-kind claim The rebuild decoded an HTML page or script outside the try that skips an unreadable file, so text past V8's string limit would fail the whole rebuild. ARCHITECTURE.md said every extension check goes through file-kinds.ts; the .md note checks do not. Ship-Check: pr-review · claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…med comments - html.ts: name the leading and trailing whitespace gaps, state why emit can pause a tokenizer declared below it, and correct the end() comment - asset-operations: one byte-cap check for whole and paged reads, one not-UTF-8 message, and the full read asset passed to the source reader - search-index: fileTextToIndexedText, html/script reader names, the HTML size cap's rationale without an invented 2-second rule, and the stale or misplaced comments around file indexing - file-watcher, hybrid-search, search-queries, note-mover: clearer constant, parameter and helper names, block bodies, and comment fixes - asset-tools: image extensions and the text cap read from their sources - generate-dockerhub-readme: drop the dead skip-level state and the no-op slash escape; name the size-warning margin - ARCHITECTURE.md: scope the file-kind claim to non-markdown files, say text formats index leniently, and list HTML and scripts in pass 2 Ship-Check: code-quality · claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…les stays linear FTS5 cannot index the path column, so deleting a file's FTS row by path scanned the whole table. A rebuild over 16,000 scripts took 1,171 s against 42 s without them. Each FTS row now shares its file_content row's rowid, and the upsert updates in place so the rowid holds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…HTML embedding - The rebuild skips an HTML page or script that fails to read or to decode, and keeps indexing the rest (decoding was moved inside the read's try). - removeFileContent leaves no keyword row behind for a file re-added at the same path. - An HTML page's visible text, not its source, is embedded; the rebuild's embedding pass never queues scripts. - vault_read_file dispatches a canvas and an image by kind when the path's extension is uppercase; the reranker labels an uppercase .PDF a PDF file. - The HTML extractor keeps a space that ends an inline tag's text and breaks no line for a block element inside a hidden one. - The backlink scan's plain and JSON-escaped stem spellings each have a case only they match. Ship-Check: test-audit · claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…t tests - The HTML extractor's speed test asserts the whole extracted text, not its first five characters. - The uppercase-canvas link tests assert each outgoing link and backlink entry whole, not one or two fields of it. - The script watcher test waits for the exact files embedded, not a count. Ship-Check: test-audit · claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…the rebuild's reader per file kind Deleting a note's FTS row by its unindexed path scanned the whole table, so a rebuild grew quadratically with the note count: 8,000 notes spent about 48 s in upserts. Notes now use the same rowid pairing as file content. The rebuild picks each file's reader with one switch over every file kind, so a new kind does not compile until it has one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The README's HTML and scripts bullet stated reading and search without the UTF-8 condition both apply: a page or script that holds a NUL byte early on or is not UTF-8, such as UTF-16 output, is refused by vault_read_file and left out of search. The sibling bullet already names UTF-8 for other text files. Ship-Check: bug-check · claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ages and scripts on the CLI page Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…, point the listing comment at getFileKind Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…, correct the UNINDEXED comment The rebuild and the watcher read and decoded a page over 16 MiB in full, then discarded its text. They now check its size first and skip the read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…page is never read Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

aliasunder · 2026-11-07T08:42:47Z

Findings from the umm-actually review at 8db118a that have no thread:

Issue comment 6703488105 (ARCHITECTURE pass 2 list and diagram omit HTML and scripts): already handled in dd3b423, which lists HTML and scripts in the pass 2 step and the indexing diagram.
Issue comment 6703488212 (no end-to-end test of the "not valid UTF-8" error): fixed in 344f763, which adds a server-error-contracts test reading a Latin-1 .html file and asserting the exact error text.

🔍 ship-check · pr-monitor · claude-opus-5-5

…he DeepWiki purpose; test: add a passing control page to the watcher's non-UTF-8 test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…read drops a leading byte-order mark SECURITY.md also names the image and PDF readers as the exception to reading any UTF-8 file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- vault_search says each file's first 100 KiB of text is searched, and that HTML visible text is searched for pages up to 16 MiB. - The reader comments, ARCHITECTURE.md and README say a larger page is never read and is found by its name; ARCHITECTURE.md item 3 and SECURITY.md now state the byte-order-mark and markdown exceptions. - fileTextToIndexedText checks the page size through isHtmlPageIndexedByNameOnly, and the embedding file filter reads a named field. - Tests: both 16 MiB tests refuse a two-argument read of the page; a script over 16 MiB is found by its text; a read drops a leading byte-order mark from a text format and an HTML page; a rebuild with file tools off runs no file embedding pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…html-and-scripts Brings in #744 (result extents, search bound, Behavior sections) and #745 (OAuth consent-request bounds). search-tools.ts kept both sides' imports; the tool-surface snapshots were regenerated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Indexing read every script and text-format file whole, though search keeps only a file's first 100 KiB, so one very large file could spike memory, with up to 16 such reads at once during a rebuild. - Text formats read only their first 100 KiB through a new readFileTextPrefix helper, which decodes the way readFile does. Search results do not change. - A script over 16 MiB is never read and is found by its name only, as an HTML page over that size already is. A script's UTF-8 check needs the whole file, so its read cannot stop at 100 KiB. - vault_search, README and ARCHITECTURE say a page or script over 16 MiB is found by its name only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…les example gloss vault_search glossed search_mode "fts" as "only keyword matching ran" in three places; the Behavior lines now name the mode by its value, and Returns keeps the definition. vault_list_files' first example repeated the folder parameter's own description. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…e wording Review fixes for the read-size change: - readFileTextPrefix turns an unfinished character at the end of a file into a replacement character, as readFile does, and returns "" for a maxBytes below one byte instead of throwing. - isIndexedByNameOnly is now isHtmlOrScriptTooLargeToRead, since a scanned PDF is also found by name only. Its comments state the memory reason for both kinds and the extraction-time reason for pages. - vault_search's opening sentence lists only what is searched; the 100 KiB, 16 MiB and keyword-only limits move to its Behavior line. - README and ARCHITECTURE wording; one comment unglued from its neighbour. - Tests: a sparse 3 GiB file read by its start, the 16 MiB boundary, an empty file, maxBytes edge cases, exact paths in the size tests, accurate comments on the readFile negatives, escapes for invisible characters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

readFileTextPrefix treated a file of exactly maxBytes as cut and dropped an unfinished character at its end. It now reads one byte past the limit to tell a cut from the file's end, so such a file ends in a replacement character, as readFile gives. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…exed text A tooltip inside an inline <svg> is never shown on the page, but the extractor indexed it as text, so a search could match a page on its icons' tooltips. Title and desc inside an open svg are now hidden like script; a self-closing <desc/> or <svg/> there holds nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…, not search Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

aliasunder and others added 12 commits November 6, 2026 23:33

test(search): cover HTML page indexing through the rebuild and the wa…

27510f4

…tcher Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test(search): cover script indexing through the rebuild and the watcher

5712fba

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs: list the file-kinds and html parsers in the AGENTS.md structure…

8507faa

… tree Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

fix(tools): name scripts as the subject of the keyword-only clause

8db118a

The searched-extension lists in the file tools now come from the file-kind map, so they follow it when a type is added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

umm-actually Bot reviewed Nov 7, 2026

View reviewed changes

Comment thread src/vault-mcp/mcp-core/tools/asset-tools.ts Resolved

Comment thread src/vault-mcp/obsidian-markdown/__tests__/html.test.ts Resolved

Comment thread src/vault-mcp/search/search-index.ts Resolved

aliasunder and others added 12 commits November 7, 2026 01:47

fix(files): name a binary file with no extension as type (none)

e937e21

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

fix(tools): name the script extensions vault_search indexes

c1cdf91

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs: scope the search-read agreement to the decode check, list web p…

8b96e15

…ages and scripts on the CLI page Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test: prove vault_read_file's not-UTF-8 error over the transport

344f763

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

refactor(search): list files to embed from the rebuild's own versions…

2962e60

…, point the listing comment at getFileKind Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

umm-actually Bot reviewed Nov 7, 2026

View reviewed changes

Comment thread src/vault-mcp/search/search-index.ts Resolved

aliasunder and others added 2 commits November 7, 2026 03:41

docs: say search holds no text from a refused page, and an oversized …

dca8acb

…page is never read Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

umm-actually Bot reviewed Nov 7, 2026

View reviewed changes

Comment thread .devin/wiki.json Resolved

Comment thread src/vault-mcp/search/__tests__/file-watcher.test.ts Resolved

docs: name pdf-engine as the parser layer's filesystem exception in t…

bd58e2e

…he DeepWiki purpose; test: add a passing control page to the watcher's non-UTF-8 test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

umm-actually Bot reviewed Nov 7, 2026

View reviewed changes

Comment thread src/vault-mcp/mcp-core/__tests__/tool-definitions.test.ts Resolved

Comment thread src/vault-mcp/mcp-core/tools/search-tools.ts Resolved

fix(tools): name the text formats vault_search indexes

1683ac2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

umm-actually Bot reviewed Nov 7, 2026

View reviewed changes

Comment thread SECURITY.md Resolved

Comment thread src/vault-mcp/mcp-core/tools/search-tools.ts Resolved

Comment thread src/vault-mcp/obsidian-markdown/html.ts Resolved

fix(tools): say search covers an HTML page's visible text and a text …

6a51239

…read drops a leading byte-order mark SECURITY.md also names the image and PDF readers as the exception to reading any UTF-8 file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

umm-actually Bot reviewed Nov 7, 2026

View reviewed changes

Comment thread src/vault-mcp/search/search-index.ts Resolved

umm-actually Bot reviewed Nov 7, 2026

View reviewed changes

Comment thread src/vault-mcp/mcp-core/tools/asset-tools.ts Resolved

aliasunder and others added 5 commits November 7, 2026 12:36

test(search): sort without mutating and brace a multi-line if

de29710

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

umm-actually Bot reviewed Nov 7, 2026

View reviewed changes

Comment thread src/utils/fs.ts Resolved

umm-actually Bot reviewed Nov 7, 2026

View reviewed changes

Comment thread src/vault-mcp/obsidian-markdown/html.ts Resolved

Comment thread src/vault-mcp/search/search-helpers.ts Resolved

umm-actually Bot reviewed Nov 7, 2026

View reviewed changes

Comment thread src/vault-mcp/mcp-core/__tests__/tool-definitions.test.ts Resolved

test(tools): drop a keyword-only expectation identical to the hybrid one

a0c4113

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

umm-actually Bot reviewed Nov 7, 2026

View reviewed changes

Comment thread src/vault-mcp/obsidian-markdown/html.ts Resolved

docs: say any other UTF-8 file reads as text but is found by browsing…

c6f5b56

…, not search Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

umm-actually Bot reviewed Nov 7, 2026

View reviewed changes

Comment thread src/vault-mcp/search/file-watcher.ts Resolved

aliasunder mentioned this pull request Nov 7, 2026

fix(oauth): forbid framing of the consent and setup pages and show where approval sends access #747

Open

4 tasks done

Conversation

aliasunder commented Nov 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this changes

Reads

Search

Fixed: uppercase file extensions

Fixed: rebuild time grew quadratically with notes and files

Tool text

Docs

Verification

Search ranking and cost A/B

HTML pages at scale

Live Test Results

Uh oh!

socket-security Bot commented Nov 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

umm-actually Bot commented Nov 7, 2026

Uh oh!

umm-actually Bot commented Nov 7, 2026

Uh oh!

umm-actually Bot commented Nov 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

aliasunder commented Nov 7, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

aliasunder commented Nov 7, 2026 •

edited

Loading

socket-security Bot commented Nov 7, 2026 •

edited

Loading

umm-actually Bot commented Nov 7, 2026 •

edited

Loading