codegraph index parse phase is extremely slow on large projects — root cause analysis and optimization roadmap

[KnifeOfLife]

Problem

On large Java projects (10,000+ files), the parse phase of codegraph index is significantly slower than expected. This issue documents the root causes identified through code analysis and proposes a prioritized optimization roadmap.

Root Cause Analysis

The parse pipeline has several serialization bottlenecks that compound on large codebases.

1. Single-file, multi-transaction SQLite writes

Every call to storeExtractionResult() (src/extraction/index.ts:2065-2179) opens 4~5 separate SQLite transactions per file:

Transaction 1: deleteFile (DELETE nodes + DELETE file)
Transaction 2: insertNodes (N single-row INSERTs)
Transaction 3: insertEdges (M single-row INSERTs)
Transaction 4: insertUnresolvedRefs (R single-row INSERTs)
Auto-commit:    upsertFile

For a 10,000-file project, this generates 40,000~50,000 transaction commits. Each commit flushes the WAL and creates disk I/O, all running serially on the main thread (SQLite is not thread-safe, index.ts:1592). Meanwhile, parse workers may be idle waiting for the commit cursor to advance.

2. Single-row INSERT statements

Within each transaction, nodes and edges are inserted one row at a time (src/db/queries.ts:insertNode, insertEdge). A file with 100 nodes produces 100 separate INSERT OR REPLACE round-trips. SQLite performs significantly better with multi-row INSERT ... VALUES (...), (...) syntax, which would reduce the number of SQL compilation and B-tree insertion cycles.

3. Ordered commit backpressure (head-of-line blocking)

The rolling window design (windowSize = max(4, pool.size * 2), index.ts:1580) limits how far parsing can run ahead of the in-order commit cursor. The commit cursor advances in original file order to ensure deterministic graph output (resolution phase disambiguates same-named symbols by insertion order).

If a file at position N takes a long time to parse (e.g. a large generated Java file), all files at N+1 through N+windowSize that have already finished parsing are blocked from committing until position N completes:

File order:      [1] [2] [3] [4] [5] [6] [7] ...
Parse done:       ✓   ✓        ✓   ✓   ✓   ✓ ...
Commit cursor:    1   2   [blocked on 3]
                        ↑ files 4-8 finished but waiting

The backpressure (index.ts:1675) then prevents dispatching new parses, leaving workers idle.

4. WASM tree-sitter overhead

codegraph uses web-tree-sitter (WASM) for parsing, which introduces several performance penalties:

• WASM binary loading and compilation per language at startup
• WASM linear memory that only grows, never shrinks (WebAssembly specification limitation), forcing worker recycling every 250 files (WORKER_RECYCLE_INTERVAL, index.ts:66) to reclaim heap
• Worker termination and respawn cost (cold start includes grammar WASM reload)
• WASM ↔ JS marshalling overhead for every AST query

The 1 MB per-file size limit (MAX_FILE_SIZE, index.ts:125) exists partly because WASM heap pressure makes larger files dangerous.

5. Small I/O batch size

FILE_IO_BATCH_SIZE = 10 (index.ts:36) means files are read from disk in batches of 10. For 10,000 files, this requires 1,000 I/O round-trips, each gated by the backpressure window.

6. Small worker pool

Default pool size is clamp(CPU_COUNT - 1, 1, 8), hard-capped at 16 (MAX_PARSE_POOL_SIZE, parse-pool.ts:59). On machines with 32+ cores, 50~75% of CPU capacity is left idle during the CPU-bound parse phase.

Optimization Roadmap

P0 — Batch transaction commit (highest impact, smallest change)

Problem: 4~5 individual transactions per file, each committing to disk independently.

Solution: Wrap the entire storeExtractionResult() body in a single transaction. Better yet, accumulate results from N files in the flushOrdered callback and commit them together. For example, instead of committing per-file, batch-commit every 50~100 files:

pendingStore = []
flushOrdered:
  dequeue completed items into pendingStore
  if pendingStore.length >= BATCH_SIZE:
    transaction:
      for each item in pendingStore:
        run storeExtractionResult logic
    clear pendingStore

This reduces transaction count from O(files) to O(files / batchSize).

Files: src/extraction/index.ts (lines ~1589-1665, ~2065-2179)

Estimated speedup: 5~10x on the storage path.

P1 — Multi-row INSERTs

Problem: Individual INSERT OR REPLACE per node/edge row.

Solution: Replace the per-row loop in insertNodes() and insertEdges() with INSERT OR REPLACE INTO nodes (...) VALUES (...), (...), ... multi-row syntax. SQLite processes multi-row INSERTs as a single B-tree insertion operation, dramatically reducing overhead.

Files: src/db/queries.ts (insertNodes, insertEdges)

Estimated speedup: 2~5x on the SQL write path.

P2 — Remove ordered-commit dependency (medium change)

Problem: Insertion-order determinism forces head-of-line blocking, which keeps workers idle.

Solution: Pre-allocate stable IDs during the scan phase (before parsing begins) so that workers can write results to SQLite in any order without affecting resolution determinism. This eliminates the need for ordered commit entirely.

Approach: assign each file a sequential fileId during scanning. Use fileId as a prefix or component of node IDs. Since IDs are pre-determined, insertion order no longer matters for disambiguation, and flushOrdered can be replaced with free-order commit.

Trade-off: Requires auditing the resolution phase (ReferenceResolver) to ensure disambiguation no longer depends on insertion order.

Files: src/extraction/index.ts, src/db/queries.ts, possibly resolution phase

Estimated speedup: 2~3x (eliminates worker idle time from backpressure).

P3 — Native tree-sitter bindings (large change)

Problem: WASM overhead for CPU-bound parsing + forced worker recycling.

Solution: Replace web-tree-sitter with the native tree-sitter Node.js addon (C-based native binding via N-API). Benefits:

• Eliminates WASM loading/compilation time per grammar
• Removes the WASM heap growth problem — native heap is managed by the system allocator, so worker recycling at 250 files is no longer needed
• Removes WASM ↔ JS marshalling overhead
• Reduces per-worker memory footprint (no separate WASM linear memory region)

Files: src/extraction/ (parse-worker.ts, grammars.ts, tree-sitter.ts, parse-pool.ts)

Estimated speedup: 2~3x on parse CPU time.

P4 — Tune constants (trivial change)

Problem: Conservative batch/worker limits.

Solution:

• Increase FILE_IO_BATCH_SIZE from 10 to 50~100
• Raise MAX_PARSE_POOL_SIZE from 16 to 64 (or make it proportional to core count)
• Allow resolveParsePoolSize to scale more aggressively (e.g. clamp(cores - 1, 1, 32) instead of clamp(cores - 1, 1, 8))

Files: src/extraction/index.ts, src/extraction/parse-pool.ts

Estimated speedup: 1.5~2x.

Summary

Priority	Optimization	Est. Speedup	Risk	Scope
P0	Batch transaction commit	5~10x	Low	index.ts only
P1	Multi-row INSERTs	2~5x	Low	queries.ts only
P2	Remove ordered-commit dependency	2~3x	Medium	Multiple files
P3	Native tree-sitter bindings	2~3x	High	Extraction layer
P4	Tune constants	1.5~2x	None	Constants only

The highest ROI items are P0 and P1 — purely SQLite usage optimizations that require no architectural changes and no dependency swaps. Together they would reduce per-file write overhead by roughly an order of magnitude, making the parse phase on large Java projects significantly faster without altering codegraph's architecture.

[colbymchenry]

@KnifeOfLife Thanks for the thorough analysis — this is a genuinely useful map of the parse-phase bottlenecks, and the root causes you identified are real. Keeping this issue open as the tracking issue for parse throughput.

One heads-up first: v1.3.1 (released today, driven by #1212) reworked a lot of the code your line references point at, so it's worth re-baselining your measurements before building on them. What changed that touches your list:

• Your How do you use this across multiple projects #1 (per-file multi-transaction writes): partially addressed, but for responsiveness, not throughput — storeExtractionResult now commits large files in bounded chunks with cooperative yields behind a serialized flush chain, so a giant generated file no longer freezes the main loop mid-extraction. The ~4-5 transactions per file remain; your batch-multiple-files-per-transaction idea is still open and still the biggest single win available here, with the caveat that the crash-recovery story (a partially-stored file has no files record and simply re-indexes) needs to survive whatever batching shape we pick.
• Your Svelte 5 #3 (ordered-commit head-of-line blocking): the in-order commit is deliberate — resolution disambiguates same-named candidates by insertion order, so commit order is what keeps the graph deterministic (byte-identical across runs). The backpressure logic was reworked in 1.3.1 (it now also waits on the async commit chain), but the ordering invariant stays. Any proposal here has to preserve determinism; "commit out of order, then canonicalize" would be the shape to explore.
• Your git-hook potential issue when codegraph is not installed globally #2 (single-row INSERTs) and windows 11 didn't install, won't start... #4 (WASM overhead / worker recycling): untouched and both still valid. Multi-row INSERT ... VALUES inside the existing transactions is probably the lowest-risk item on your list.

Also relevant to your 10k+-file Java projects even though it's not the parse phase: 1.3.1 fixed the post-parse tail (the #1212 watchdog kills / OOMs and a Spring-heavy double file scan), so end-to-end codegraph index time on large Java repos should already look meaningfully different from your baseline.

If you re-run your measurements on 1.3.1, CODEGRAPH_SYNTH_TIMINGS=1 now prints per-phase timings to stderr (scan, framework detection, resolution passes, maintenance) which should make the parse-phase share of wall-clock easy to isolate. Happy to dig into the batching design here if you want to take a crack at it.

[KnifeOfLife]

@colbymchenry

Indexing extremely slow: 57 minutes for 3,044 files (parsing phase)

Environment

• OS: Windows
• Project: 3,044 files scanned, 3,010 indexed (34 parse errors)
• Graph outcome: 88,277 nodes, 166,603 edges
• Total time: 57m 27s
• Project: Java + Stringboot
• CodeGraph:1.3.1

---

Raw timing log

┌  Indexing project
|
|  . Scanning files...[phase-timing] scan: 437ms (3044 files)
[phase-timing] framework-detect: 11ms
|  * Scanning files - 3,044 found
|  * Parsing code - done
|  x Resolving refs  #########################  100%[synth-timing] closureCollEdges: 1043ms
|  o Resolving refs  #########################  100%[synth-timing] emitterEdges: 323ms
|  . Resolving refs  #########################  100%[synth-timing] renderEdges: 775ms
|  . Resolving refs  #########################  100%[synth-timing] jsxEdges: 344ms
|  + Resolving refs  #########################  100%[synth-timing] ifaceEdges: 876ms
|  o Resolving refs  #########################  100%[synth-timing] mybatisEdges: 466ms
|  O Resolving refs  #########################  100%[synth-timing] springEdges: 302ms
|  . Resolving refs  #########################  100%[synth-timing] insertMergedEdges: 19235ms
|  . Resolving refs  #########################  100%[synth-timing] chainedConformance: 123ms
[synth-timing] deferredThisMember: 0ms
|  * Resolving refs  #########################  100%[phase-timing] maintenance: 352ms
[phase-timing] count-recompute: 15ms
|  * Resolving refs - done
│
◆  Indexed 3,010 files (34 could not be parsed)
│
●  88,277 nodes, 166,603 edges in 57m 27s
│
◇  Error breakdown ──────────╮
│                            │
│  34 files failed to parse  │
│                            │
├────────────────────────────╯
│
●  See .codegraph/errors.log for details
│
●  The index is fully usable - only the failed files are missing.
│
└  Done

Scan + framework-detect + all resolving-refs phases sum to ~24 seconds. The remaining ~57 minutes is entirely in Parsing code (no [phase-timing] marker exists for it).

---

Root cause analysis

Detail see #1231