Skip to content

feat(research): scaffold SDM variant module — subclass extension points + benchmark wrapper#1

Merged
salishforge merged 2 commits into
mainfrom
claude/evaluate-fork-strategy-s1pm2
May 8, 2026
Merged

feat(research): scaffold SDM variant module — subclass extension points + benchmark wrapper#1
salishforge merged 2 commits into
mainfrom
claude/evaluate-fork-strategy-s1pm2

Conversation

@salishforge
Copy link
Copy Markdown
Owner

Summary

This PR resolves the question driving the branch name (evaluate-fork-strategy) and lands the matching scaffolding so the SDM (Sparse Distributed Memory, Kanerva 1988) research variant can begin landing in follow-up commits.

Decision: keep developing the SDM variant in this fork (memforge-sdm). Upstream salishforge/memforge is a beta-quality public package with stable APIs, semver responsibilities, and external consumers — experimental retrieval and sleep-cycle algorithms don't fit that vehicle without first paying for a strategy/registry refactor (≈4–6 weeks of yak-shaving before any research can land). The fork has zero divergence from upstream main today, so rebase debt stays cheap as long as core edits are minimal and clearly marked.

Changes

Minimal core edits (all marked // VARIANT: for rebase visibility)

  • src/memory-manager.ts: privateprotected on queryHybrid, rerankWithLlm. New createSleepEngine protected factory so subclasses can swap engines without needing access to private fields. Sleep-cycle construction site dispatches through the factory instead of inline new SleepCycleEngine(...).
  • src/sleep-cycle.ts: privateprotected on phaseScoring, phaseTriage, reviseMemory.

Behavior is unchanged for upstream consumers.

New parallel research module (src/research/)

File Role
src/research/research-memory-manager.ts ResearchMemoryManager extends MemoryManager — passthrough overrides for queryHybrid / rerankWithLlm, plus createSleepEngine override that wires ResearchSleepCycleEngine.
src/research/research-sleep-cycle.ts ResearchSleepCycleEngine extends SleepCycleEngine — passthrough overrides for phaseScoring / phaseTriage / reviseMemory.
src/research/research-server.ts Mirrors src/server.ts on default port 3334 for head-to-head A/B benchmarks alongside the upstream server on 3333.
benchmarks/longmemeval/run-research.ts Thin wrapper around the existing benchmark runner; retargets MEMFORGE_URL:3334 and BENCHMARK_RESULTS_DIRbenchmarks/results-research so SDM runs don't overwrite the published 93.2% R@5 hybrid baseline.

All overrides currently delegate to super. SDM-specific algorithms (Hamming-radius readout, hard-location activation, address-space storage) land in follow-up commits scoped to these files only — no further changes to upstream-track files needed for the first variant.

Working principles for follow-up work

  • Subclass before editing core files. When core must be touched, prefix with // VARIANT: so rebases highlight conflict zones.
  • Keep public API stable — don't change MemoryManager.query() / sleep() signatures.
  • Pull upstream weekly via a sidecar branch first.
  • Reuse tests/integration.test.ts, tests/llm-paths.test.ts, and the LongMemEval harness unchanged — they hit MemoryManager's public API and are algorithm-agnostic.

Exit criteria

  • Upstream the variant if a single experiment shows ≥3 pp R@5 lift on the full LongMemEval 500-question harness and the implementation can be expressed as one or two clean strategy/registry interfaces. Then propose the strategy refactor to upstream and contribute as a strategy implementation.
  • Stay forked otherwise — fork is cheap to maintain while experiments iterate.

Test plan

  • npm run type-check — passes with 0 errors
  • Boot tsx src/research/research-server.ts, hit GET /memory/test-agent/health and a smoke query, confirm parity with upstream server on a smoke corpus
  • Run tsx benchmarks/longmemeval/run-research.ts against the same dataset/DB snapshot used for the published 93.2% R@5 baseline, confirm the passthrough scaffold reproduces baseline numbers (sanity check before introducing SDM logic)
  • Once SDM-specific overrides land in follow-up commits, record A/B numbers in benchmarks/research-runs.md

Plan file: /root/.claude/plans/continue-the-research-on-merry-rabbit.md

https://claude.ai/code/session_01CSK9SeCexUDxn37BEi2SoM


Generated by Claude Code

…ts + benchmark wrapper

Decision: keep developing the Sparse Distributed Memory (Kanerva) research
variant in this fork (memforge-sdm) rather than upstream memforge. Upstream
has stable APIs, semver, and external consumers; experimental retrieval and
sleep-cycle algorithms don't fit that vehicle without first paying for a
strategy/registry refactor (4–6 weeks). The fork is currently zero-divergence
from upstream main, so rebase debt stays cheap if we keep core edits minimal.

Minimal core edits (all marked `// VARIANT:` for rebase visibility):
- promote 5 private override targets to protected: queryHybrid, rerankWithLlm
  (memory-manager.ts), phaseScoring, phaseTriage, reviseMemory (sleep-cycle.ts)
- add `createSleepEngine` protected factory in MemoryManager so subclasses can
  swap engines without needing access to private fields

New parallel research module under src/research/:
- ResearchMemoryManager extends MemoryManager (passthrough overrides)
- ResearchSleepCycleEngine extends SleepCycleEngine (passthrough overrides)
- research-server.ts mirrors src/server.ts on port 3334 for head-to-head
  benchmarks against the upstream server
- benchmarks/longmemeval/run-research.ts retargets MEMFORGE_URL +
  BENCHMARK_RESULTS_DIR so SDM runs don't overwrite baseline numbers

Compiles clean (`npm run type-check` → 0 errors). All overrides currently
delegate to super; SDM-specific algorithms (Hamming-radius readout, hard-
location activation, etc.) land in follow-up commits to these files only.

See /root/.claude/plans/continue-the-research-on-merry-rabbit.md.

https://claude.ai/code/session_01CSK9SeCexUDxn37BEi2SoM
Copy link
Copy Markdown
Owner Author

CI status update: integration-tests (22) failure is not caused by this PR. Reproduced locally on plain main (without any of this PR's commits) — same error, same line.

Root cause: schema/schema.sql is out of sync with schema/migration-v2.4.sql (and likely v2.2/v2.3/v2.5). The CI workflow only applies schema.sql, but runtime code references columns added in migrations — e.g. src/memory-manager.ts:224 queries hot_tier.content_hash, which migration-v2.4.sql:28 adds via ALTER TABLE hot_tier ADD COLUMN ... content_hash TEXT; but schema.sql was never updated to match. Per project rules in CLAUDE.md, schema.sql is supposed to be the canonical from-scratch schema and stay in sync with migrations.

Fix coming in a separate PR (off main) so this PR stays scoped to the SDM scaffold. Once that schema-sync PR merges and main is rebased in here, this PR's CI will go green.

The 5 currently-passing tests in this run (from Input validation) are validation-only paths that don't touch the DB; everything that calls manager.add() fails on the missing column. Same on main.


Generated by Claude Code

@salishforge salishforge merged commit 67c1d62 into main May 8, 2026
2 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants