feat(skill-memory): per-skill cross-session recall + historian auto-extraction

[iceteaSA]

Summary

Adds skill-memory — per-skill "motor memory" that gives a skill cross-session recall of its own hard-won lessons (gotchas, discoveries, fixes, workflow steps). When a skill's SKILL.md declares skill-memory: { enabled: true }, accumulated notes for that skill surface automatically in a <skill-memory> block appended to the skill tool's result on every load — and, with the historian extension, the historian writes those notes automatically during compaction (no agent action required).

It's fully opt-in per skill and cache-safe by construction: the block rides the tool-result tail (conversation), never the cached system/m[0] prefix, so it can't bust the prompt cache.

How it works

• Transparent recall — three-hook augmentation: tool.definition advertises an intent param; tool.execute.before stashes the intent (bounded TTL); the after-hook parses the skill's Base directory, reads its SKILL.md frontmatter, and formats the recall block. Lands in the tool RESULT (cache-safe).
• Write paths — ctx_skill_note (agent-authored) and the historian (auto-extracted). Both dedup on a normalized hash.
• Intent-scoped ranking — a recall cascade: model-matched embeddings → cosine blend over intent_embedding + delta_embedding (relevance/recency/hit weights tunable per skill via ranking_* frontmatter); FTS5 fallback over a content-linked skill_memory_fts vtable; flat recency×hit fallback.
• Historian auto-extraction — the historian sees TC: skill(<name>) markers in its chunk and emits a <skill_observations> block; both the OpenCode and Pi runners promote those post-commit as global notes (project_identity='*', source_type='historian'), recallable from any project.

Review units (4 commits)

The branch is organized into four coherent, reviewable phases:

1. P1 — transparent per-skill recall: skill_memory table migration, the three-hook augmentation, flat recall + storage, ctx_skill_note/ctx_skill_recall tools, opt-in distill-skill-memory dreamer task, TUI/ctx-status stats, docs.
2. P2 — embeddings + intent-scoped recall: delta_embedding + recall_count columns + skill_memory_fts FTS5 vtable; embed-on-write; the cosine/FTS recall cascade; a programmatic (no-LLM) reembed pre-step.
3. P3a — historian-extraction foundation: the TC: skill(<name>) marker keystone; origin_project + source_type columns; global-tier notes unified under project_identity='*' (collision-merge); partitionKey routing.
4. P3b — historian auto-extraction pipeline: prompt emits <skill_observations>, parser, validated-result threading, both runners promote via the shared promoteSkillObservations helper, plus an initializeDatabase self-heal net (re-creates skill_memory + ensureColumn so an upgraded DB recovers even if a migration row is lost).

Schema / migrations

Three migrations (numbered after the current master ceiling): skill_memory table, then delta_embedding/recall_count/FTS, then origin_project/source_type/* unification. LATEST_SUPPORTED_VERSION bumped in lockstep (the schema-version-fence test enforces it). Migration bodies are ensureColumn/IF NOT EXISTS idempotent.

Testing

Full plugin + Pi suites pass (one unrelated pre-existing full-suite ordering flake in tui-config.test.ts that passes in isolation), tsc clean both packages, lint clean, build produces all bundles. Dedicated coverage for the migrations (coexistence + fence), recall rungs, the tools, FTS triggers/backfill, the '*' collision-merge, and the historian promotion path on both runners. Verified working live: the historian auto-extraction writes genuine source_type='historian' notes, embeddings populate, and the read-side recall_count increments on surfacing.

Notes for reviewers

• Migration version numbers are placeholders relative to this fork's base — happy to renumber to whatever slots are free at merge time.
• The four commits are independently meaningful; if you'd prefer this as stacked PRs (P1 / P2 / historian), I can split it.

---

^{Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.}

---

Summary by cubic

Adds per-skill cross-session memory with embeddings + FTS ranking and historian auto-extraction. A <skill-memory> block is appended to each skill tool result; ctx_skill_note writes notes and ctx_skill_recall recalls them on demand.

• New Features

  • Transparent per-skill recall: the skill tool accepts optional intent; before-hook stashes it and after-hook injects recall; ctx_skill_note writes and ctx_skill_recall reads; Status/TUI show per-project totals and pinned counts.
  • Intent-scoped ranking: cosine blend over intent_embedding + delta_embedding with FTS5 fallback; read-side recall_count; re-embed pre-step backfills NULL/stale vectors before the distill run.
  • Historian auto-extraction: TC: skill(<name>) markers → <skill_observations>; both runners promote via promoteSkillObservations as global '*' notes (source_type='historian').
  • Distill task: opt-in distill-skill-memory dreamer task added to config/CLI/dashboard/schema; runs a programmatic re-embed pre-step and emits a read-only corpus health report; off by default.
  • Schema/self-heal: skill_memory table + FTS vtable/columns with migrations v50–v52 (LATEST_SUPPORTED_VERSION=52); initializeDatabase re-creates missing objects and rebuilds skill_memory_fts; supports opencode’s plain-path skill provenance.

• Bug Fixes

  • Cross-partition recall: union a skill’s partition with global '*' so project-local skills surface historian-written notes; writes/dedup remain exact-partition.
  • Frontmatter gating + parser: reject ctx_skill_note when skill-memory.enabled is false or missing; parser accepts inline { enabled: true }, is start-of-file anchored, tolerates BOM/whitespace, and strips inline # from unquoted scalars.
  • Provenance + tiering: accept plain filesystem paths and file://; correctly classify singular ~/.config/opencode/skill/** as global; derive tier via dirname(resolvedPath); normalize Windows paths.
  • Guidance + observability: drop ctx_memory cross-reference in skill guidance when memory is off; log recall errors; count per-note XML framing and clamp pinned budget.
  • Disabled-path safety: avoid init crash when the plugin is disabled; escape apostrophes in SQL.

^{Written for commit 3825d5f. Summary will update on new commits.}

Greptile Summary

Adds skill-memory — per-skill cross-session recall that appends a <skill-memory> block to each skill tool result when a skill opts in via skill-memory: { enabled: true } frontmatter. The feature is fully cache-safe (recall block lands in the tool result tail, never the cached prefix) and includes historian auto-extraction so accumulated lessons are written globally without any agent action.

• Schema + migrations: Three new migrations (v50–v52) add the skill_memory table, FTS5 vtable with content-linked triggers, delta_embedding/recall_count/origin_project/source_type columns, and a v52 global-tier collision-merge. initializeDatabase gains a self-heal FTS backfill and ensureColumn calls so an upgraded DB recovers even if a migration row is lost.
• Three-rung recall cascade: cosine blend over intent_embedding + delta_embedding (rung 1, model-matched) → FTS5 MATCH on (intent, delta) (rung 2) → flat recency×hit (rung 3); pinned notes prepended ahead of any ranked result, all subject to a token-budget fill with a clamped pinned sub-budget.
• Historian auto-extraction: TC: skill(<name>) markers in historian chunks trigger <skill_observations> output, parsed by compartment-parser and promoted as global '*'-partition notes by both the opencode and Pi runners via a shared promoteSkillObservations helper.

Confidence Score: 5/5

Safe to merge — all four issues identified in the previous review round have been addressed with regression tests, and no new defects were found in the core recall, storage, or historian-promotion paths.

The recall cascade, budget-fill logic, dedup paths, FTS triggers, migration idempotency, historian promotion gating, and intent-stash lifecycle were all reviewed in depth. The only findings are a duplicate call to recallPartitionPredicate inside searchSkillMemoryFts (which is pure and functionally harmless) and an API ambiguity in the write-path functions where callers pass a pre-computed partition key to functions that apply partitionKey internally (idempotent for all valid inputs). Both are style-level concerns with no current failure path.

storage.ts has the minor searchSkillMemoryFts double-call and write-function API ambiguity noted in comments; all other files are straightforward.

Important Files Changed

Filename	Overview
packages/plugin/src/features/magic-context/skill-memory/storage.ts	New file: core storage layer for skill_memory table — insert, dedup, recall (flat/FTS/cosine), bump counters, stats. Logic is correct; minor API ambiguity around whether callers should pass raw projectIdentity or pre-computed partitionKey result (both happen to be idempotent today). searchSkillMemoryFts calls recallPartitionPredicate twice instead of destructuring once.
packages/plugin/src/features/magic-context/skill-memory/recall.ts	New file: shared recall core used by both the transparent after-hook path and ctx_skill_recall tool. Recall cascade (cosine → FTS5 → flat recency×hit), budget fill with pinned-note sub-budget clamping, XML escaping, and error logging are all correctly implemented.
packages/plugin/src/features/magic-context/skill-memory/promote.ts	New file: historian-extraction promotion — writes global-tier '*'-partition notes with source_type='historian', hash-dedup with hit_count bump, best-effort per-item error handling. Correct.
packages/plugin/src/features/magic-context/skill-memory/provenance.ts	New file: parses 'Base directory for this skill:' from tool output (both legacy file:// URL and current plain-path forms), derives tier and skill_source. Session-scoped registry with composite sessionId:skillId keys prevents cross-session bleed. Correct.
packages/plugin/src/features/magic-context/skill-memory/frontmatter.ts	New file: custom YAML frontmatter parser for skill-memory: block. Handles inline flow-mapping, block form, inline comments, BOM, CRLF, quoted strings, and comment stripping for unquoted scalars. Start-anchored regex correctly prevents mid-document --- from being misparsed. Correct.
packages/plugin/src/hooks/magic-context/hook-handlers.ts	Adds createToolExecuteBeforeHook (intent stash) and extends createToolExecuteAfterHook with skill-load registry population and maybeInjectSkillMemory injection. Intent map keyed sessionID:callID with TTL+cap backstop; single hoisted await import shared by registry-populate and injection blocks. All previous review thread issues addressed.
packages/plugin/src/features/magic-context/migrations.ts	Adds migrations v50 (skill_memory table + indexes), v51 (delta_embedding + recall_count + FTS5 vtable/triggers + backfill rebuild), and v52 (origin_project + source_type + global-tier collision-merge). All use ensureColumn/IF NOT EXISTS idempotency. v52 merge logic correctly picks the earliest survivor and aggregates counters.
packages/plugin/src/features/magic-context/storage-db.ts	Adds skill_memory table, FTS vtable/triggers, and ensureColumn calls to initializeDatabase. Self-heal FTS backfill is guarded (ftsCount === 0 && rowCount > 0) so it fires only on the gap and degrades gracefully on failure.
packages/plugin/src/tools/ctx-skill-note/tools.ts	New write tool: validates kind, checks registry for skill load, guards on frontmatterConfig.enabled before any DB work, exact-hash dedup → semantic cosine dedup → insert, handles concurrent-insert race via NULL return. All previous review issues addressed.
packages/plugin/src/tools/ctx-skill-recall/tools.ts	New read tool: registry-first resolution falling back to ordered disk search matching opencode's discoverSkills() order. Guards on frontmatterConfig.enabled before recall. Delegates to shared recallSkillMemoryBlock. Correct.
packages/plugin/src/hooks/magic-context/compartment-parser.ts	Adds skill_observations block parsing via SKILL_OBS_ITEM_REGEX. unescapeXml applied to skillId and lesson; SKILL_OBS_KINDS set validates kind; items missing any required field are silently skipped. Correct.
packages/plugin/src/hooks/magic-context/compartment-runner-incremental.ts	Promotes historian-extracted skill observations via the same promotionActive && !discardedLast gate used for facts/primers, correctly preventing double-emit on provisional tails.
packages/pi-plugin/src/pi-historian-runner.ts	Pi-runner mirror of the opencode historian skill-observation promotion path. Same promotionActive && !discardedLast gate; skillObservations threaded through ValidationOutcome and validateHistorianResult. Correct.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Agent
    participant BeforeHook as tool.execute.before
    participant SkillTool as skill tool
    participant AfterHook as tool.execute.after
    participant Registry as SkillLoadRegistry
    participant Recall as recallSkillMemoryBlock
    participant DB as SQLite (skill_memory)
    participant Historian as Historian runner
    participant Promote as promoteSkillObservations

    Agent->>BeforeHook: "skill(name, intent=...)"
    BeforeHook->>BeforeHook: stashIntent(sessionID:callID, intent)
    BeforeHook->>SkillTool: forward call
    SkillTool-->>AfterHook: output with Base directory line
    AfterHook->>Registry: parseSkillProvenance → set(sessionID:skillId, entry)
    AfterHook->>AfterHook: getAndDeleteIntent(sessionID:callID)
    AfterHook->>Recall: recallSkillMemoryBlock(skill, intent, scope, projectIdentity)
    Recall->>DB: embed intent → cosine rank (rung 1)
    alt embeddings matched
        DB-->>Recall: ranked notes
    else FTS fallback (rung 2)
        DB-->>Recall: FTS MATCH notes
    else flat fallback (rung 3)
        DB-->>Recall: recency x hit notes
    end
    Recall->>DB: bumpRecallCountByIds
    Recall-->>AfterHook: skill-memory block
    AfterHook->>AfterHook: "output.output += block"
    AfterHook-->>Agent: tool result with appended skill-memory

    Agent->>DB: ctx_skill_note → insertSkillMemoryNote or bumpHitCount

    Historian->>Historian: parse skill_observations from chunk
    Historian->>Promote: promoteSkillObservations(db, originProject, obs)
    Promote->>DB: "insertSkillMemoryNote(tier=global, project_identity=*, source_type=historian)"

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Agent
    participant BeforeHook as tool.execute.before
    participant SkillTool as skill tool
    participant AfterHook as tool.execute.after
    participant Registry as SkillLoadRegistry
    participant Recall as recallSkillMemoryBlock
    participant DB as SQLite (skill_memory)
    participant Historian as Historian runner
    participant Promote as promoteSkillObservations

    Agent->>BeforeHook: "skill(name, intent=...)"
    BeforeHook->>BeforeHook: stashIntent(sessionID:callID, intent)
    BeforeHook->>SkillTool: forward call
    SkillTool-->>AfterHook: output with Base directory line
    AfterHook->>Registry: parseSkillProvenance → set(sessionID:skillId, entry)
    AfterHook->>AfterHook: getAndDeleteIntent(sessionID:callID)
    AfterHook->>Recall: recallSkillMemoryBlock(skill, intent, scope, projectIdentity)
    Recall->>DB: embed intent → cosine rank (rung 1)
    alt embeddings matched
        DB-->>Recall: ranked notes
    else FTS fallback (rung 2)
        DB-->>Recall: FTS MATCH notes
    else flat fallback (rung 3)
        DB-->>Recall: recency x hit notes
    end
    Recall->>DB: bumpRecallCountByIds
    Recall-->>AfterHook: skill-memory block
    AfterHook->>AfterHook: "output.output += block"
    AfterHook-->>Agent: tool result with appended skill-memory

    Agent->>DB: ctx_skill_note → insertSkillMemoryNote or bumpHitCount

    Historian->>Historian: parse skill_observations from chunk
    Historian->>Promote: promoteSkillObservations(db, originProject, obs)
    Promote->>DB: "insertSkillMemoryNote(tier=global, project_identity=*, source_type=historian)"

Loading

Comments Outside Diff (1)

1. packages/plugin/src/features/magic-context/skill-memory/recall.ts, line 481-484 (link)

   Bare catch {} makes recall errors invisible

   Any exception thrown inside recallSkillMemoryBlock — a SQLite error, embedding provider crash, or a coding mistake — is silently swallowed and returns an empty string. From the caller's perspective this is indistinguishable from "no notes exist" or "skill-memory disabled". At minimum a sessionLog or console.warn call here would let a developer diagnose why the recall block is absent without attaching a debugger.

_{Reviews (10): Last reviewed commit: "fix(skill-memory): accept plain filesyst..." | Re-trigger Greptile}

[cubic-dev-ai]

11 issues found across 78 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/plugin/src/features/magic-context/dreamer/task-executor.ts">

<violation number="1" location="packages/plugin/src/features/magic-context/dreamer/task-executor.ts:419">
P2: Re-embed pre-step errors are suppressed, allowing successful task completion reporting despite a failed prerequisite data maintenance step.</violation>
</file>

<sub>Note: This PR contains a large number of files. cubic only reviews up to 40 files per PR, so some files may not have been reviewed. cubic prioritizes the most important files to review.
On a pro plan you can use ultrareview for larger PRs.

Re-trigger cubic</sub>

[cubic-dev-ai]

4 issues found across 15 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/plugin/src/features/magic-context/dreamer/task-executor.ts">

<violation number="1" location="packages/plugin/src/features/magic-context/dreamer/task-executor.ts:419">
P2: Re-embed pre-step errors are suppressed, allowing successful task completion reporting despite a failed prerequisite data maintenance step.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

4 issues found across 78 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/plugin/src/features/magic-context/dreamer/task-executor.ts">

<violation number="1" location="packages/plugin/src/features/magic-context/dreamer/task-executor.ts:419">
P2: Re-embed pre-step errors are suppressed, allowing successful task completion reporting despite a failed prerequisite data maintenance step.</violation>
</file>

<sub>Note: This PR contains a large number of files. cubic only reviews up to 40 files per PR, so some files may not have been reviewed. cubic prioritizes the most important files to review.
On a pro plan you can use ultrareview for larger PRs.

Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 2 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/plugin/src/features/magic-context/skill-memory/provenance.test.ts">

<violation number="1" location="packages/plugin/src/features/magic-context/skill-memory/provenance.test.ts:107">
P2: The test "parses a PLAIN filesystem path for a PROJECT skill" verifies that the function returns the correct `tier` and `skillSource`, but it doesn't verify the `resolvedPath` output. Since this is the only test exercising plain-path resolution for a project-level skill, a regression in the path-joining logic — like a missing `/SKILL.md` suffix, incorrect directory concatenation, or broken normalization — would go undetected. The adjacent test for global plain paths (and six other tests covering file:// paths) all assert `resolvedPath`. Consider adding an assertion like `expect(result!.resolvedPath).toBe("${HOME}/projects/foo/.opencode/skills/my-skill/SKILL.md")` to match the coverage pattern of the rest of the test suite.</violation>
</file>

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: The test "parses a PLAIN filesystem path for a PROJECT skill" verifies that the function returns the correct tier and skillSource, but it doesn't verify the resolvedPath output. Since this is the only test exercising plain-path resolution for a project-level skill, a regression in the path-joining logic — like a missing /SKILL.md suffix, incorrect directory concatenation, or broken normalization — would go undetected. The adjacent test for global plain paths (and six other tests covering file:// paths) all assert resolvedPath. Consider adding an assertion like expect(result!.resolvedPath).toBe("${HOME}/projects/foo/.opencode/skills/my-skill/SKILL.md") to match the coverage pattern of the rest of the test suite.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/plugin/src/features/magic-context/skill-memory/provenance.test.ts, line 107:

<comment>The test "parses a PLAIN filesystem path for a PROJECT skill" verifies that the function returns the correct `tier` and `skillSource`, but it doesn't verify the `resolvedPath` output. Since this is the only test exercising plain-path resolution for a project-level skill, a regression in the path-joining logic — like a missing `/SKILL.md` suffix, incorrect directory concatenation, or broken normalization — would go undetected. The adjacent test for global plain paths (and six other tests covering file:// paths) all assert `resolvedPath`. Consider adding an assertion like `expect(result!.resolvedPath).toBe("${HOME}/projects/foo/.opencode/skills/my-skill/SKILL.md")` to match the coverage pattern of the rest of the test suite.</comment>

<file context>
@@ -91,4 +91,36 @@ describe("parseSkillProvenance", () => {
+    test("parses a PLAIN filesystem path for a PROJECT skill", () => {
+        const output = `Base directory for this skill: ${HOME}/projects/foo/.opencode/skills/my-skill`;
+        const result = parseSkillProvenance(output, "my-skill");
+        expect(result!.tier).toBe("project");
+        expect(result!.skillSource).toBe("opencode-project");
+    });
</file context>