feat: VS Code extension — Mnemonic for VS Code

[jkbennitt]

Overview

Build a VS Code extension that surfaces mnemonic's semantic memory system directly in the editor. The daemon and REST API already exist -- the extension is a smart client that connects to localhost:9999.

Goal: Developers should not have to leave the editor or open the dashboard. The right memory should appear at the right moment, automatically.

---

Architecture

VS Code Extension (TypeScript)
    | REST API + WebSocket (localhost:9999)
Mnemonic Daemon (already running)
    |
SQLite + LLM

Zero new infrastructure. Extension is a pure REST/WS client against the existing API.

---

Prerequisite: Daemon API Work

Before the extension MVP ships, the REST API needs parity with what MCP tools already expose.

Must-have for Phase 1 (~2.5 hours)

Endpoint	What	Why	Effort
Wire project, source, type, min_salience through POST /api/v1/query	Retrieval agent already supports these; REST handler does not expose them	Sidebar needs project filtering, CodeLens needs source filtering	~30 min
GET /api/v1/memories/by-file?path=...	Extract concepts from path server-side, return relevant memories	Core Related Memories sidebar feature -- cleaner than client-side concept extraction	~1 hour
POST /api/v1/query/batch	Array of queries in, array of results out	CodeLens performance -- one request per file, not per symbol	~1 hour

Must-have for Phase 2 (~3 hours)

Endpoint	What	Why
POST /api/v1/memories/{id}/amend	Update memory content in place	Memory editing in sidebar/WebView
POST /api/v1/memories/{id}/archive	Archive (forget) a memory	Memory management
GET /api/v1/sessions/{id}/summary	Session summary	Welcome back notification
REST equivalents for recall_project, get_context, session_summary	Expose MCP-only tools via REST	Sidebar project context, proactive suggestions

Existing API assets the extension can use today

• POST /api/v1/query -- semantic search (limited params, fixed in prereq)
• POST /api/v1/remember -- create memories
• POST /api/v1/feedback -- rate recall quality
• GET /api/v1/health -- daemon status + version + memory count
• GET /api/v1/memories/{id}/context -- rich memory detail
• GET /api/v1/activity -- daemon recent concept activity tracker
• GET /api/v1/system/update-check -- version check
• WS /ws -- real-time events (16 event types)
• SearchByEntity(name, entityType, limit) -- store method for symbol-level CodeLens

---

Features (prioritized)

Phase 1 -- MVP (ship first)

1. Memory Sidebar

TreeView panel in the Explorer sidebar:

• Related Memories -- updates based on the active file via GET /api/v1/memories/by-file?path=... (server-side concept extraction)
• Project Context -- pinned section showing active patterns, recent decisions/insights, current episodes
• Powered by project-filtered queries once prereq wiring is done

2. Quick Recall (Cmd+Shift+M)

QuickPick dialog for instant semantic search:

• Natural language input via POST /api/v1/query (with project filtering)
• Select a result to copy to clipboard or insert as comment

3. Status Bar

Minimal status bar item showing memory count and connection status.

• Click to open sidebar
• Daemon connection status via GET /api/v1/health
• Graceful degradation: empty/offline states, stale cache where possible

---

Phase 2 -- Real-time + Terminal

4. WebSocket Integration

Subscribe to /ws for real-time events:

• memory_encoded -> invalidate sidebar cache for affected file/concepts
• consolidation_completed -> update project context section
• pattern_discovered -> notify user of new patterns
• Foundation for session awareness and CodeLens cache invalidation

5. Session Awareness

Driven by WebSocket events (not polling). Uses consolidation_completed events + GET /api/v1/sessions/{id}/summary.

6. Terminal Error Memory

Pattern-match terminal errors against mnemonic error-type memories via POST /api/v1/query with type: error filtering. High value, straightforward once query params are wired.

7. Remember from Editor

Right-click context menu on selected code:

• Remember This Decision -> POST /api/v1/remember with type: decision
• Remember This Error -> type: error
• What Do I Know About This? -> recall scoped to selection
• Remember This Insight -> type: insight

---

Phase 3 -- CodeLens + Detail Views

8. Inline Memory Hints (CodeLens)

File-level batch query architecture, not per-symbol:

1. On file open, extract file path + top-level symbols via DocumentSymbolProvider
2. Single POST /api/v1/query/batch request with all symbols
3. Client-side distribute results to CodeLens positions
4. Cache invalidated via WebSocket memory_encoded events -- no polling

9. Memory Detail WebView

Click a memory in the sidebar to open a rich WebView panel showing full context via GET /api/v1/memories/{id}/context, associated memories, concepts, tags, salience score, and edit/forget controls.

10. Settings UI

• mnemonic.endpoint -- daemon URL (default: http://127.0.0.1:9999)
• mnemonic.token -- API auth token
• mnemonic.inlineHints.enabled -- toggle CodeLens
• mnemonic.inlineHints.minSalience -- threshold (0.0-1.0)
• mnemonic.sidebar.autoRefresh -- refresh interval (default: 30s)
• mnemonic.notifications.sessionStart -- toggle welcome-back notifications

---

What NOT to build

• Do not duplicate the dashboard -- web UI exists for deep exploration
• Do not add a chat interface -- Claude Code + MCP tools handles this
• Do not require auth for localhost
• Do not poll for real-time data -- use the WebSocket

Concept Extraction Strategy

Server-side via GET /api/v1/memories/by-file -- daemon has concepts.FromPath() in Go. Extension sends path, server extracts concepts and queries.

Error States / Graceful Degradation

• Daemon not running: status bar offline, sidebar/CodeLens empty with Start daemon action
• LLM unavailable: features still work (FTS search, cached embeddings)
• Slow responses: sidebar loading state, CodeLens uses stale cache

Repo Structure

TBD -- monorepo (vscode/ dir) or separate repo. Separate repo likely better for independent release cadence.

Tech Stack

• VS Code Extension API (TypeScript)
• REST client + WebSocket against localhost:9999
• TreeView for sidebar, CodeLens for inline hints, QuickPick for search
• WebView only for rich memory detail (Phase 3)

Publishing

• Publisher: AppSprout (appsprout or appsprout-dev)
• Marketplace: https://marketplace.visualstudio.com
• Requires Azure DevOps PAT with Marketplace Publish scope

Monetization Notes

The extension is free -- it drives adoption of the core product. Pro features (cloud sync, team memory) are what we charge for. The extension makes the free tier stickier by reducing friction to zero.

[CalebisGross]

Deep Review

@jkbennitt — the feature design is strong. Phased approach, "what not to build" discipline, focus on surfacing memory at the right moment. But there's a meaningful gap between what the issue assumes the REST API exposes and what it actually does today. Here's the full audit.

---

API Gaps That Block the Extension

POST /api/v1/query is weaker than the MCP recall tool. The REST handler only accepts 4 fields:

{ "query": string, "limit": int, "synthesize": bool, "include_reasoning": bool }

But the retrieval agent internally supports a much richer QueryRequest — Project, Source, State, Type, MinSalience, TimeFrom/To, ExcludeConcepts. The extension needs Project filtering at minimum (sidebar: "memories for this project"), and Source filtering for CodeLens (show only filesystem-origin memories). These fields exist in the retrieval agent — they're just not wired through the REST handler.

10 MCP tools have no REST equivalent:

Tool	Extension needs it for	Add difficulty
recall_project	Sidebar "Project Context" section	Medium
session_summary	Phase 2 "Welcome back" notification	Medium
get_context	Proactive suggestions from activity	Medium
batch_recall	Performance — sidebar + CodeLens need multiple queries	Low
amend	Phase 3 memory editing in WebView	Low
recall_session	Session awareness	Low
recall_timeline	Time-based browsing	Low
forget	Phase 3 memory management	Low
exclude_path / list_exclusions	Settings integration	Low

Phase 1 MVP can work with what exists today if we wire project/source/type filtering through /api/v1/query. Phase 2 and 3 require 5-6 new REST routes.

---

The "Related Memories for This File" Problem

This is the killer feature and the hardest to get right. How it actually has to work today:

1. Extract concepts from file path — e.g., /src/auth/token.go → ["auth", "token"]
2. Send as query string to POST /api/v1/query
3. Retrieval agent does FTS + embedding search + spread activation
4. Return ranked results

This works but it's indirect. Raw memories store file paths in metadata.path, but no REST route queries by path. A dedicated GET /api/v1/memories/by-file?path=/src/auth/token.go that does concept extraction server-side would be cleaner and more accurate.

Also worth noting: SearchByEntity(name, entityType, limit) exists in the store and searches structured concept sets for entities by name. If the extension extracts function names via VS Code's symbol provider, it could find memories referencing specific functions. Not mentioned in the issue but would make CodeLens dramatically better.

---

CodeLens Performance

"Query per top-level symbol" is the wrong architecture. A file with 20 exported functions = 20 HTTP requests on file open. Better approaches:

1. File-level batch query — one request with the file path + extracted symbols, return all relevant memories, client-side distribute to CodeLens positions
2. WebSocket subscription — subscribe to memory_encoded events, invalidate cache only when new memories match current file concepts. No polling.
3. batch_recall REST equivalent — send all symbol queries in one request

---

The WebSocket Is Underutilized

The issue doesn't mention the WebSocket at /ws. It broadcasts 16 event types including:

• memory_encoded — new memory ready
• consolidation_completed — includes pattern/abstraction counts
• pattern_discovered — new pattern detected
• watcher_event — real-time file activity

The "Session Awareness" feature (Phase 2) proposes polling for session summaries. The WebSocket already pushes consolidation_completed with exactly the data that notification needs.

---

Missing from the Issue

1. Auth token — config.yaml has an optional token field for API auth. Extension settings should include mnemonic.token.

2. Repo structure — monorepo (vscode/ dir) or separate repo? Affects CI, release cadence, version coupling.

3. GET /api/v1/memories/{id}/context — returns rich context (memory + resolution + concept set + attributes + episode). Perfect for Phase 3 "Memory Detail WebView" but not mentioned.

4. GET /api/v1/activity — returns daemon's recent concept activity tracker. Extension could highlight which files/concepts are "hot" right now.

5. Concept extraction — extension needs to extract concepts from file paths. Daemon has concepts.FromPath() in Go. Should the extension reimplement in TypeScript, or should the API accept a path and extract server-side?

6. Error states — what happens when daemon isn't running? Status bar shows "offline" but sidebar, CodeLens, and quick recall all need graceful degradation strategy. Stale cache? Empty state? "Start daemon" action?

---

Phase Ordering Suggestions

Phase 1 is right. Sidebar + Quick Recall + Status Bar. Ship it.

Phase 2 should lead with the WebSocket, not CodeLens. CodeLens is flashiest but has the hardest performance/relevance problems. Session awareness via WebSocket events is easier, higher confidence, and validates the real-time connection pattern CodeLens will need.

"Terminal Error Memory" (Phase 3) is undervalued. Consider moving to Phase 2. VS Code has terminal output access. Pattern-matching errors against mnemonic's error-type memories is high value — just a POST /api/v1/query with type filtering once we wire it through.

---

Prerequisite Work on Daemon Side

Before the extension MVP ships:

1. Wire project, source, type, min_salience through POST /api/v1/query — retrieval agent already supports these, handler just doesn't expose them. ~30 min.
2. Add GET /api/v1/memories/by-file?path=... — extract concepts from path server-side, return relevant memories. ~1 hour.
3. Add POST /api/v1/query/batch — array of queries in, array of results out. Needed for CodeLens. ~1 hour.

For Phase 2, add:
4. POST /api/v1/memories/{id}/amend
5. POST /api/v1/memories/{id}/archive (forget)
6. GET /api/v1/sessions/{id}/summary

[jkbennitt]

Updated the issue body to incorporate your feedback — added the prerequisite API work section (query param wiring, by-file endpoint, batch query), WebSocket-first architecture for Phase 2, reordered phases (Terminal Error Memory moved up, CodeLens moved to Phase 3 with batch architecture), added auth token to settings, concept extraction strategy, error states/graceful degradation, and documented all existing API assets the extension can use today.

[CalebisGross]

@jkbennitt updated spec looks great. A few things to tighten up:

1. Wrong endpoint name — "Remember from Editor" (Phase 2, item 7) references POST /api/v1/remember. That doesn't exist — it's POST /api/v1/memories. Someone will copy-paste from the spec.

2. by-file endpoint should accept symbols too. If the extension is already extracting top-level symbols via DocumentSymbolProvider for CodeLens, send them in the same request. One endpoint that takes path + optional symbols[] and returns memories grouped by match type (file-level vs symbol-level) serves both sidebar and CodeLens from a single call.

3. No debounce strategy for file switching. onDidChangeActiveTextEditor fires on every tab switch. Without a 300-500ms debounce, the sidebar hammers the API during rapid tab navigation.

4. WebSocket reconnection strategy. Laptop sleep, daemon restart, network hiccup — the WS will drop. Need exponential backoff with jitter, degrade to polling /api/v1/health until reconnected. Important since Phase 2 makes WS the backbone.

5. No testing strategy. How do you test an extension that depends on a running daemon? Mock API? Integration tests against a real daemon? Matters for CI — can't easily run the daemon in GitHub Actions.

6. Cache specifics are vague. "Stale cache where possible" — but what eviction? LRU per file? TTL? 50 open files = 50 cached results. Probably fine, worth a sentence on bounds.

None of these are blockers. Spec is solid.