Indexing 5–10× slower on HDD — main-thread I/O stalls trigger false parse timeouts (even on 0-byte files) and worker-recycle churn

[KnifeOfLife]

Environment

• @colbymchenry/codegraph v1.3.1, installed via npm (platform bundle, bundled Node)
• Windows, 12-core CPU, project on a mechanical HDD (high random-I/O latency)
• AV: Huorong — .codegraph/ AND the source tree both added to its trust zone (no effect; see below)
• Project: Java, 3,044 files scanned / 3,010 indexed / 34 parse errors, 88,277 nodes / 166,603 edges
• codegraph index → ~50 min (vs. a few min on SSD)

Summary

On an HDD, codegraph index collapses to ~1 file/sec with the process sitting at 0–2% CPU. The parse worker pool is active (8 workers) but starved. The root cause is not parsing and not the store CPU — it's that the main thread is I/O-bound on the HDD (synchronous node:sqlite store + fsp.readFile), and the parse timeout is measured from dispatch on the main-thread clock (parse-pool.ts:271-272), so whenever the main thread is blocked >10s it produces false TIMEOUTs on parses that actually finished instantly — including a 0-byte empty file. Each false timeout kills and respawns a worker, which re-reads the WASM grammars from the same slow HDD, amplifying the I/O load in a feedback loop that gets worse over time.

The smoking gun

codegraph index -v shows many [worker] TIMEOUT: <file> exceeded Xms lines. Among the timed-out files:

• one is 0 bytes (empty)
• smallest non-empty: 307 bytes
• largest: 77.2 KB

A 0-byte / 307-byte file cannot take 10s of tree-sitter CPU. The only way these hit PARSE_TIMEOUT_MS (index.ts:58, 10s base + 10s/100KB) is that the parse finished immediately but the main thread couldn't service the result within 10s — i.e. the main thread was blocked elsewhere (disk I/O). This rules out a parse-side or CPU cause and points squarely at main-thread I/O stall.

Measured data

Signal	Value	Meaning
CPU during "Parsing code"	0–2% on 12 cores	not computing — waiting on I/O
Pool	Parse worker pool: 8 worker(s)	parallelism exists, workers are idle
Disk throughput	1.7 MB/s	HDD random-IOPS ceiling, saturated
codegraph.db-wal	6.3 MB	WAL autocheckpoint healthy — not a WAL-bloat issue
codegraph.db	111 MB, growing	normal
Memory	315 MB	fine
Throughput	4% @12.5s, 9% @53.7s, decelerating	progressive slowdown
AV exclusion (.codegraph + source)	no change	not AV interception

Root-cause chain (with code refs)

1. Main thread is the I/O-bound serialization point. It serially does fsp.readFile (batched 10, index.ts:1717-1734) → dispatch to workers → synchronous node:sqlite store (storeExtractionResult, index.ts:2097-2222, via DatabaseSync in sqlite-adapter.ts). The store is heavy random-write: each insertNode maintains 7 indexes (schema.sql:89-95) + the nodes_ai FTS5 trigger (schema.sql:109-112) + name_segment_vocab; each edge maintains 5 indexes (schema.sql:151-175). On an HDD this is dominated by random seek latency.
2. Workers are starved by backpressure. When the main thread stalls on I/O, flushOrdered (index.ts:1661-1679) can't advance the commit cursor, the window (windowSize = max(4, pool.size*2) = 16, index.ts:1588) fills, and the backpressure loop (index.ts:1705-1708) blocks dispatch. All 8 workers go idle → 0% CPU.
3. False timeouts. The per-parse timer is armed at dispatch on the main-thread clock (parse-pool.ts:271-272). While the main thread is blocked in I/O, neither the worker's parse-result message nor the timer is serviced; when the blocked stretch exceeds 10s, onTimeout (parse-pool.ts:284-296) fires on an already-finished parse. This is why even a 0-byte file "times out".
4. I/O amplification via worker respawn. onTimeout terminates the worker and spawnOne()s a replacement (parse-pool.ts:294), which must re-load the grammar WASM from disk (parse-pool.ts:207-208 load-grammars). On the HDD this is another burst of random reads → more main-thread/OS I/O contention → more stalls → more false timeouts. This feedback is the "progressive slowdown".
5. False timeouts become permanent parse errors. The retry pass (index.ts:1819-1822) only retries errors matching 'Worker exited' / 'memory access out of bounds'. The timeout message 'Parse timed out after …' matches neither, so these files are dropped from the index (this accounts for most of the 34 "parse errors" — including the empty file). User-visible: good files silently missing from the graph on HDD.

Why this likely hasn't been seen internally

The dev/test target is macOS on NVMe (CLAUDE.md): random-I/O latency ~50µs, the synchronous store never stalls long enough to trip a 10s timeout, so the feedback never triggers. The architecture is only exposed on high-latency random-I/O storage (HDD, and likely synced/network folders).

User-side mitigations (no code change)

• Move the project to an SSD — resolves it entirely.
• Symlink .codegraph/ to an SSD (mklink /J) — removes the bulk of the random DB writes from the HDD.
• CODEGRAPH_PARSE_WORKERS=2 — fewer workers ⇒ fewer in-flight tasks per stall ⇒ fewer respawns ⇒ less grammar-reload churn. Modest, uncertain.

Suggested fixes (ranked)

1. Don't measure the parse timeout on the main-thread dispatch clock. Have the worker report its own parse start/end timestamps, or pause/suspend the timeout while the orchestrator knows the main thread is in a blocking store. This alone kills the false-timeout feedback. (parse-pool.ts:271-272, 284-296)
2. Make PARSE_TIMEOUT_MS env-overridable the way CODEGRAPH_PARSE_WORKERS already is (parse-pool.ts:63, index.ts:58). It's the most useful knob for slow-storage users, and currently hardcoded.
3. Throttle/debounce worker respawns on consecutive timeouts, or avoid respawning when the stall was main-thread-side — to stop the grammar-reload I/O amplification.
4. Retry timeout failures (or at least non-OOM timeouts) in the retry pass, so a blocked-main-thread spike doesn't permanently drop files (index.ts:1819-1822).
5. (Separate, lower priority) The store's write amplification (7 node indexes + FTS5 trigger + segment vocab + 5 edge indexes) is the reason the store is I/O-heavy in the first place; deferring FTS5 rebuild during bulk index (the same lesson as the v1.1.3 still indexes gitignored embedded repos; ignored benchmark corpus triggers resolving-refs watchdog #1065 delete-path fix) would lighten the random-write load that triggers the stalls.

Detailed code-level optimization plan

The ranked list above is the summary. Below is the concrete, source-grounded version (verified against v1.3.1, and corrected after an independent audit — see the ⚠️ notes). Each item lists: location, the change, what it breaks in the root-cause chain, risk, and HDD gain.

Invariants that must hold throughout: #1015 deterministic in-order commit (flushOrdered, index.ts:1661-1679), #850 store yield (commitYield, STORE_CHUNK=2000), node:sqlite DatabaseSync is not shareable across threads but a separate connection per thread is valid, and no retrieval latency/correctness regression.

Phase 1 — stop the bleeding (small, low-risk, shippable now)

1.1 Stop false timeouts — the trigger of the whole feedback loop.
Location: parse-pool.ts:271-272 (timer armed at dispatch), :284-296 (onTimeout), :219-232 (onMessage), parse-worker.ts.
Change: have the worker report its own parse duration; have the main thread decide validity from that, not from wall-clock.
⚠️ Critical race the naive version misses: after a long synchronous store, Node's event loop runs the timers phase before the poll phase, so onTimeout fires first, terminates the worker and settles the job — by the time onMessage runs, job.settled is already true and the result is discarded. So onTimeout must not kill/settle; it must defer:

// parse-worker.ts
const t0 = performance.now();
const result = extractFromSource(...);
postMessage({ type: 'parse-result', id, result, parseMs: performance.now() - t0 });

// parse-pool.ts onTimeout — do NOT terminate/settle, just mark + arm a hard safety kill
private onTimeout(w, job, ms) {
  if (job.settled || !this.workers.has(w)) return;
  job.timerExpired = true;                       // defer decision to onMessage
  if (!job.hardKill) job.hardKill = setTimeout(() => this.hardKill(w, job), ms * 3);
}

// parse-pool.ts onMessage (parse-result) — accept the result if the worker says it was fast
if (m.type === 'parse-result') {
  const job = this.inflight.get(w);
  if (job?.timerExpired && typeof m.parseMs === 'number' && m.parseMs < budgetFor(job.task)) {
    job.timerExpired = false;                    // main thread was stalled, not the parse — accept
  }
  // …existing settle / recycle path
}

The secondary hard-kill (ms * 3) is required for genuine WASM hangs where the worker never posts a result.
Breaks: root-cause link #3 (false timeouts). Risk: low. Gain: removes the feedback loop's trigger.

1.2 Make PARSE_TIMEOUT_MS env-overridable.
Location: index.ts:58 (const PARSE_TIMEOUT_MS = 10_000), mirroring CODEGRAPH_PARSE_WORKERS at parse-pool.ts:87 / index.ts:1551.

const PARSE_TIMEOUT_MS = Number(process.env.CODEGRAPH_PARSE_TIMEOUT_MS) || 10_000;

Breaks: gives slow-storage users an immediate knob to suppress false kills. Risk: very low. Gain: direct relief.

1.3 Retry 'timed out' errors so false timeouts don't permanently drop files.
Location: index.ts:1819-1822. Today only 'Worker exited' / 'memory access out of bounds' are retried; the timeout message 'Parse timed out after …' (parse-pool.ts:293) matches neither, so these files are silently lost from the index (most of the 34 "parse errors", including the empty file).

const retryableErrors = errors.filter(e => e.code === 'parse_error' && e.filePath &&
  (e.message.includes('Worker exited') ||
   e.message.includes('memory access out of bounds') ||
   e.message.includes('timed out')));   // ← add

The retry pass (index.ts:1824-1920) is serial and stores after the parse resolves, so it can't re-trigger the false-timeout window. Risk: low. Gain: correctness — no silently missing files on slow disks.

Phase 2 — kill the amplifier (medium)

2.1 Cache grammar WASM bytes so respawns don't re-read from disk.
Location: parse-pool.ts:207-208 (each new worker gets load-grammars → grammars.ts:316 WasmLanguage.load(wasmPath) reads the .wasm from dist/extraction/wasm/), plus parse-worker.ts, grammars.ts.
Change: pre-read each needed .wasm once on the main thread and pass the bytes via workerData; the worker loads from memory.
⚠️ Use a plain Buffer/Uint8Array, NOT SharedArrayBuffer. Node worker_threads have no COOP/COEP (that's browser-only); workerData structured-clones Buffer bytes efficiently. Confirmed web-tree-sitter Language.load() accepts Uint8Array (src/web-tree-sitter.d.ts:80).

// main thread, once
const grammarBuffers: Record<string, Buffer> = {};
for (const lang of neededLanguages) grammarBuffers[lang] = await fsp.readFile(resolveWasmPath(lang));
new Worker(parseWorkerPath, { workerData: { grammarBuffers, rootDir, … } });
// parse-worker.ts: Language.load(new Uint8Array(workerData.grammarBuffers[lang]))

Caveats: the wasm path-resolution logic (grammars.ts:313-315, vendor list vs __dirname/wasm) must be runnable on the main thread; ~40 grammars × ~300 KB ≈ 12 MB resident (acceptable).
Breaks: root-cause link #4 (grammar-reload I/O amplification) at the source — respawns become near-free. Risk: medium. Gain on HDD: large; this is the highest-ROI single change.

2.2 Throttle respawns/recycles on consecutive timeouts.
Location: parse-pool.ts:284-296 (onTimeout→spawnOne), :251-257 (recycle). Note timeouts deliberately don't charge CRASH_BUDGET (:289), and MAX_CONCURRENT_SPAWN=2 only bounds simultaneous cold-starts, not total respawns — so a false-timeout storm spawns unboundedly. Add exponential backoff (1s/2s/4s) on consecutive timeouts, reset on a successful parse.
Breaks: dampens link #4. Risk: medium (real crash storms are still bounded by CRASH_BUDGET). Gain: complements 2.1.

Phase 3 — structural fix (highest ROI, large refactor)

3.1 Move the store off the main thread onto a dedicated store worker (own connection + QueryBuilder).
This is the only change that converts the serialized main-thread pipeline (read → dispatch → synchronous store) into overlapping work (parse + read + store in parallel). It is the lever that takes HDD indexing from ~50 min to minutes.
Design:

• Spawn one store worker that opens its own node:sqlite connection (proven pattern: db/index.ts:237-263 runMaintenance opens new DatabaseSync(workerData.dbPath) in an inline worker) and its own QueryBuilder.
• Main thread posts { seq, filePath, content, contentHash, stats:{size,mtimeMs}, result } in file order; the worker runs the full storeExtractionResult logic (index.ts:2097-2222) on its own thread and ACKs.

⚠️ Complexity the headline understates:

• storeExtractionResult is densely coupled to this.queries (getFileByPath, deleteFile, getCrossFileIncomingEdgesWithTarget, insertNodes, insertEdges, insertUnresolvedRefsBatch, upsertFile) and this.rootDir — all must be re-homed to the worker.
• fs.Stats is not cleanly structured-cloneable — pass only { size, mtimeMs } (the only fields used, index.ts:2215-2216).
• Edge.metadata: Record<string, unknown> may hold non-clonable values → wrap postMessage in try/catch or constrain the type to JSON-serializable.
• Crash recovery: if the store worker dies mid-commit the ACK never arrives and flushOrdered hangs — need crash detection + re-queue of uncommitted work + an ACK-timeout on the main thread.
• Multi-connection WAL coherency: the main thread's reader connection won't see the store worker's writes until they're checkpointed. If MCP queries run during indexing, the store worker should PRAGMA wal_checkpoint(PASSIVE) after each file (or the reader checkpoints before reads).

Invariants:

• Missing parse-time concurrency #1015 preserved: the store worker applies results strictly in seq order (buffer out-of-order arrivals, commit in order) — same invariant as flushOrdered, relocated.
• codegraph cause 100% CPU #850 preserved and improved: the main thread now does only fsp.readFile (async, libuv) + dispatch, so it's more responsive; keep STORE_CHUNK=2000 chunking inside the store worker as belt-and-suspenders.
• Backpressure: bound completed by the existing nextSeq - nextToStore >= windowSize (index.ts:1705-1708); gate on store-worker ACK depth rather than the main-thread commit cursor.
  Risk: medium-high. Gain on HDD: the structural fix — parse/read/store overlap instead of serialize.

Phase 4 — lighten the write path (medium, schema-touching)

4.1 Defer the FTS5 rebuild during bulk index (apply the #1065 lesson to the insert path).
Location: schema.sql:109-124 (nodes_ai/ad/au triggers fire on every node INSERT/DELETE/UPDATE), index.ts indexAll start/end, db/index.ts.
Change: during indexAll, disable the three triggers, insert all nodes, then rebuild once: INSERT INTO nodes_fts(nodes_fts) VALUES('rebuild') (valid for external-content tables, content='nodes'); restore triggers for the sync path.
⚠️ Crash-consistency gap: if killed mid-index with triggers off, nodes_fts is permanently stale (old nodes + gaps + delete-ghosts), and indexAll only clears name_segment_vocab (index.ts:448), not FTS. Mitigation: clear nodes_fts at indexAll start, or write a fts_rebuild_pending marker that the next open() checks and rebuilds on. Keep triggers live on the sync/indexFile paths.
Breaks: lightens the random-write load that stalls the store. Risk: medium. Gain: meaningful, and stacks with 3.1.

4.2 Index pruning — only after query-frequency analysis.
Location: schema.sql:151 (idx_edges_kind on edges(kind)), :89-95.
⚠️ idx_edges_kind is NOT redundant — it's the only index with kind leftmost, used by getStats() (queries.ts:2013, GROUP BY kind, called by codegraph status). It is pure write overhead on the index path (one more random write per edge) but a covering index for that read. Pruneable only if that read is confirmed cold (it then falls back to a full scan, acceptable for a status command). Don't drop blind — measure first.

windowSize

windowSize = max(4, pool.size*2) = 16 (index.ts:1588). Bumping it alone gives little today (the main thread can't dispatch while it's blocked in the synchronous store). After 3.1, the main thread is free to dispatch continuously, so raising windowSize to ~64 then lets parse-ahead overlap saturate the freed main thread. Memory: 64 × up-to-1 MB file contents ≈ 64 MB peak, acceptable. So: pair windowSize with 3.1, not standalone.

New risks introduced (track during implementation)

• 3.1 multi-connection WAL visibility (store writer vs main-thread reader) — checkpoint cadence.
• 3.1 store-worker crash recovery — ACK timeout + re-queue.
• 4.1 FTS mid-index crash staleness — clear-on-start or rebuild marker.
• 3.1 Edge.metadata clone safety — type constraint or try/catch.
• Positive side-effect: once 3.1 + 2.1 land, the main thread is nearly never blocked, so timeout timers fire accurately — item 1.1's judgment becomes more reliable, not less.

Recommended sequencing

1. Phase 1 (1.1 with the race fix + 1.2 + 1.3): small, low-risk, ships immediate HDD relief (no more dropped files / false kills).
2. Phase 2 (2.1 + 2.2): remove the grammar-reload amplification.
3. Phase 3 (3.1): the structural fix that collapses ~50 min to minutes — treat as a real refactor, scope it to the coupling points above.
4. Phase 4 (4.1 with crash-consistency + 4.2 after analysis): write-path lightening.

Repro

On a Windows machine with the project on an HDD:

codegraph init && codegraph index -v

Watch: CPU stays 0–2%, TIMEOUT: lines appear for tiny/empty files, throughput decelerates, ~34 files end up as parse errors.