Daemon unresponsive: reindexMemoryArtifacts synchronous full-scan blocks event loop with thousands of memory artifacts

[Ostico]

Signet version: 0.99.3
Environment: Linux, Node v22.22.1, Bun 1.3.12
Related issues: N/A

---

Summary

The daemon becomes unresponsive after startup when thousands of memory artifact files exist. Instrumented profiling (resource-monitor, branch ostico/fd-diagnostic) confirmed the root cause: importExistingMemoryFiles() does synchronous I/O on ~7,900 files in a tight loop, blocking the event loop for ~1.9s and triggering chokidar to accumulate ~7,932 persistent read-only FDs that are never released — even on watcher.close().

---

Observed Behavior

• Daemon becomes unresponsive to HTTP requests including /health
• Event loop lag of 1,856ms confirmed by resource monitor
• ls -la /proc/<pid>/fd | wc -l shows ~8,000+ FDs, almost all pointing to .md artifact files
• watcher.close() releases zero memory FDs (chokidar v4 leak on Bun)

---

Expected Behavior

• Daemon should not watch immutable artifact files
• Memory import should be async and batched to avoid event loop blocking
• Daemon should remain responsive indefinitely under normal operation

---

Instrumented FD Progression (empirical, 2026-04-15)

Resource monitor snapshots at each daemon lifecycle point:

Stage	Total FDs	Memory .md FDs	Sockets	RSS
post-db-init	20	0	4	208 MB
post-watcher	20	0	4	215 MB
post-pipeline	284	240	8	244 MB
server-ready	376	320	9	249 MB
/health (~8s later)	8,025	7,932	13	263 MB
cleanup-start	8,021	7,932	9	249 MB
pre-cleanup-watcher	8,012	7,932	6	248 MB
post-cleanup-watcher	8,012	7,932	6	252 MB

Key observation: FDs jump from 376 → 8,025 in ~8 seconds after importExistingMemoryFiles() runs. watcher.close() releases 0 of 7,932 memory FDs.

---

Root Cause Analysis (corrected from instrumented data)

Corrected mechanism

The original issue described inotify watches as the FD source. Instrumented profiling disproved this — only 1 inotify instance exists. The actual mechanism:

• Chokidar v4 on Linux/Bun opens read-only FDs (lr-x) for every watched file, not inotify watches
• These FDs accumulate as importExistingMemoryFiles() does readFileSync on each file, which triggers chokidar's watcher to open persistent handles
• The user's FD limit is 1,048,576 (not the assumed 8,192), so FD exhaustion alone doesn't crash the daemon at current file counts
• The event loop blocking from ~7,900 synchronous reads is the primary responsiveness killer

Vector 1: importExistingMemoryFiles() (PRIMARY)

daemon.ts onListening callback → importExistingMemoryFiles() → iterates ALL ~7,916 .md files via readdirSync + readFileSync in a synchronous loop. Blocks the event loop for ~1.9s. Each file access triggers chokidar to open a persistent read-only FD.

Vector 2: Chokidar FD accumulation (SECONDARY)

Chokidar watches ~/.agents/memory/ including ~7,932 immutable artifact files. The ignored callback reduces this from 7,940 → ~880 FDs (87% reduction when tested), but doesn't fully prevent all FD creation. Additionally, watcher.close() leaks ~7,478 of ~7,940 FDs on Bun.

Vector 3: reindexMemoryArtifacts() (TERTIARY)

memory-lineage.ts:538-591 — full DELETE + synchronous re-read of all artifact files. Called from renderMemoryProjection() on synthesis. Now instrumented with logger.time() for ongoing monitoring.

---

Suggested Fix

Fix 1: Exclude artifact files from chokidar watcher

In watcher-ignore.ts, add the canonical artifact filename pattern:

/^\d{4}-\d{2}-\d{2}T.*--[a-z2-7]{16}--(summary|transcript|compaction|manifest)\.md$/

Also exclude MEMORY.backup-*.md files (309 files, not useful to watch). Tested reduction: 7,940 → ~880 FDs.

Fix 2: Make importExistingMemoryFiles() async with batching

Replace the synchronous loop with await-based batched processing (e.g., 50 files per tick) to keep the event loop responsive during startup.

Fix 3: Replace full-scan reindex with incremental approach

Replace the DELETE + full-read loop in reindexMemoryArtifacts() with an incremental strategy: query existing artifact hashes, compare against directory listing, only read/upsert new or changed files.

---

Impact

• Daemon becomes unresponsive, requiring manual restart
• All harness integrations lose memory injection
• No graceful degradation — process alive at 0% CPU but cannot accept connections
• Scales linearly with artifact count — gets worse over time

---

Diagnostic Tooling Added

Branch ostico/fd-diagnostic adds permanent resource monitoring:

• resource-monitor.ts — FD snapshot, event loop lag detection, periodic polling
• Lifecycle snapshots at: post-db-init, post-watcher, post-pipeline, server-ready, cleanup stages
• /health endpoint extended with additive resources key
• reindexMemoryArtifacts() instrumented with logger.time()

---

Environment Details

• Signet: 0.99.3
• OS: Linux
• Node: v22.22.1
• Bun: 1.3.12
• DB: SQLite (bun:sqlite)
• Date observed: 2026-04-15
• Profiled with: resource-monitor.ts on branch ostico/fd-diagnostic

Working on the fix.

[Ostico]

Need further analysis, even raising ulimits, the issue is still there.

[NicholaiVogel]

Looking into this

[Ostico]

Updated the issue description with instrumented profiling data from resource-monitor.ts. Root cause corrected — event loop blocking from sync I/O is the primary culprit, not FD exhaustion.

[Ostico]

Corrected Root Cause Analysis (2026-04-15)

Previous analysis attributed the event loop blocking to importExistingMemoryFiles(). After re-reading the code carefully, that attribution was incorrect. Here is the corrected analysis:

PRIMARY: reindexMemoryArtifacts() — synchronous full-scan (event loop blocker)

memory-lineage.ts:541 — This is a synchronous function (void return, no async) called from renderMemoryProjection() → handleSynthesisRequest() on every synthesis request. It:

1. Calls listCanonicalFiles() — readdirSync on ~7,600 canonical artifact files
2. Calls readFileSync on EVERY canonical file (line 582)
3. Calls parseFrontmatterDocument() on each file's content
4. Runs synchronous DB operations (DELETE all rows + SELECT tombstones + INSERT per file) in a tight loop
5. Zero yielding to the event loop — blocks for ~1.9 seconds

SECONDARY: syncManifestRefs() — hidden second O(n) scan

memory-lineage.ts:1269 — Called from renderMemoryProjection() after reindexMemoryArtifacts. Calls listCanonicalFiles() again, filters to manifest files, then loadManifest() (readFileSync) on ALL manifests. This is a second synchronous full-scan that was not previously identified.

TERTIARY: Chokidar FD accumulation

daemon.ts:1249-1267 watches ~/.agents/memory/ directory. Chokidar v4 on Bun/Linux opens persistent lr-x FDs for each file. watcher-ignore.ts does NOT exclude canonical artifact files or backup files, so ~7,900 persistent FDs accumulate. watcher.close() does not release these FDs (chokidar leak on Bun).

CORRECTION: importExistingMemoryFiles() is NOT the event loop blocker

importExistingMemoryFiles() is async and iterates ~7,953 files, BUT ingestMemoryMarkdown() filters out ALL artifact and backup files at line 992 (ARTIFACT_FILENAME_RE + MEMORY_BACKUP_FILENAME_RE match). The readFileSync at line 996 is never reached — zero qualifying files exist. Only cost is readdirSync (~7,953 entries) + basename() + regex test per file — fast, not the 1.9s block.

Empirical FD Progression (instrumented run)

Stage	Total FDs	Memory .md FDs	RSS
post-db-init	20	0	208 MB
post-watcher	20	0	215 MB
post-pipeline	284	240	244 MB
server-ready	376	320	249 MB
/health (~8s later)	8,025	7,932	263 MB

The FD jump from 376 → 8,025 happens after server-ready, aligning with the first client synthesis request which triggers renderMemoryProjection() → reindexMemoryArtifacts().

Fix Approach

1. Watcher ignore: Add canonical artifact + backup file patterns to watcher-ignore.ts (FD reduction from ~8,000 to ~20)
2. Incremental reindex: Make reindexMemoryArtifacts() track file mtime, only re-read changed files (event loop block from ~1.9s to <50ms)
3. Scope syncManifestRefs: Only process manifests whose underlying files changed
4. Short-circuit import: Pre-filter importExistingMemoryFiles() to avoid 7,953 unnecessary function calls

[Ostico]

Fixing.

[Ostico]

Fix Summary (2026-04-16) — branch ostico/fd-diagnostic

All three vectors identified in the root cause analysis have been addressed. The branch is pushed to Ostico/signetai for review.

---

Root Cause Recap

Three independent issues combined to make the daemon unresponsive:

1. Bun fs.watch() FD leak — Bun opens one O_RDONLY FD per file when watching a directory, and close() never releases them. Chokidar watches ~/.agents/memory/ (~8,000 artifact files) → ~8,000 leaked FDs. Confirmed via strace: 16,720 openat() syscalls for 8,360 files, zero released on watcher.close(). Node.js uses a single inotify FD for the same operation.

2. renderMemoryProjection() blocks the event loop — Called synchronously on every synthesis request. Runs reindexMemoryArtifacts() (full readdirSync + readFileSync on ~7,600 files) + syncManifestRefs() (second full scan). Blocks the event loop for ~1.5s, making /health and all other endpoints unresponsive during synthesis.

3. Chokidar watched immutable files — Canonical artifact files (*--summary.md, *--transcript.md, etc.) and MEMORY.backup-*.md files were not excluded from the watcher ignore list, causing unnecessary FD accumulation even without the Bun leak.

---

Fixes Applied (10 commits)

Vector 1 — FD Leak

Commit	What
ba94df1e	Remove memory/ from chokidar watch paths. The memory/ directory was passed directly to chokidar's watch list. All files inside are canonical artifacts or backups — already excluded by the ignored matcher, so no watcher events would fire anyway. Removing the directory watch eliminates ~8,000 leaked FDs at the source.
25a97c09	Exclude canonical artifacts from file watcher. Added ARTIFACT_FILENAME_RE and MEMORY_BACKUP_FILENAME_RE patterns to watcher-ignore.ts as defense-in-depth.

Vector 2 — Event Loop Blocking (synthesis worker thread)

Commit	What
d56e2cad	Offload renderMemoryProjection() to a node:worker_threads worker. The worker runs in its own V8 isolate with an independent SQLite + sqlite-vec connection (initDbAccessorLite). The daemon's event loop stays free during synthesis. Graceful fallback to synchronous rendering if the worker fails to initialize.

Worker thread architecture:

• synthesis-render-worker.ts — Worker script, opens its own DB connection
• synthesis-worker-protocol.ts — Shared typed message protocol with type guards (zero as any casts)
• hooks.ts — handleSynthesisRequest() delegates to worker via postMessage, falls back to sync if worker unavailable
• daemon.ts — Worker lifecycle (spawn on startup, terminate on shutdown), logged error handling
• db-accessor.ts — initDbAccessorLite() for lightweight DB access in worker isolate

Key design decisions:

• renderMemoryProjection() internals were not modified — the worker calls the same function, just in a separate thread
• Worker writes MEMORY.md to disk directly; the HTTP handler only returns the prompt (avoids double-write)
• 10s init timeout, 30s render timeout with graceful degradation
• Single worker instance — JS event loop guarantees no concurrent access to the synthesisWorker reference

Vector 3 — Incremental Reindex + Import Short-circuit

Commit	What
69fb86bc	Short-circuit importExistingMemoryFiles() when filtered file list is empty
b3096f26	Make reindexMemoryArtifacts() incremental (track mtime, skip unchanged files)
cee1f079	Reconcile cold-start cache with DB and clear stale manifest refs

Supporting

Commit	What
15e66031	Add LogCategory entries for "config" and "resources"
945fe1d5	Resource monitor unit tests
7af79a00	Content-parity regression test for incremental reindex
b8abf0ae	FD diagnostic stress test script

---

Code Cleanup (included in d56e2cad)

• Extracted checkCliAvailable() helper — deduplicated 4 identical CLI preflight spawn blocks in daemon.ts
• Removed dead runSynthesis() function from pipeline/synthesis-worker.ts (not exported, zero callers)
• Removed 3 unused imports from hooks-routes.ts

---

Stress Test Results

Stress test: 5 concurrent synthesis requests + 10 /health probes fired simultaneously.

Before (synchronous renderMemoryProjection):

Metric	Value
Synthesis p50	1.57s
Synthesis max	2.03s
Health during synthesis	~1.5s stalls (event loop blocked)
Mixed concurrent load (100 req)	10-25s response times

After (worker thread):

Metric	Value
Synthesis p50	128ms
Synthesis p95	135ms
Synthesis max	135ms
Health p50 during synthesis	9ms
Health max during synthesis	9ms
Health responses	10/10 healthy
FD delta	+2 (stable)

Improvement: 12x faster synthesis, 167x reduction in event loop blocking.

---

Verification

• 25/25 tests pass (13 existing + 12 new integration tests)
• TSC: 442 errors (2 fewer than baseline — dead code removal eliminated 2 pre-existing errors)
• Stress test: PASS (health < 100ms threshold, all synthesis completed, FD stable)
• 4-agent verification completed: plan compliance (PASS), code quality (PASS), manual QA (PASS), scope fidelity (PASS after triage)

---

Known Limitations (tracked as tech debt, non-blocking)

• Worker uses console.error instead of structured logger (V8 isolate has no access to daemon's logger instance)
• No automatic worker respawn — if the worker crashes, daemon falls back to synchronous rendering until restart
• configurePragmas duplicated in POC test (test-only, not production code)