feat(eval): FamiliarAdapter + jp-realm-v0.1 corpus + first live readings

[jphein]

Second in a three-PR stack. Stacked on feat/mempalace-daemon-adapter — please merge that one first; this PR's diff against main will collapse to just the new commits afterward.

Summary

Three logically-grouped pieces ride together because they're tightly coupled — the corpus was authored to exercise both adapters, and the lessons doc is the writeup that came out of running them:

• FamiliarAdapter — read-side adapter for familiar.realm.watch, a deterministic retrieve→rerank→decay→compress pipeline that wraps a palace. Pairs cleanly with MemPalaceDaemonAdapter for A/B work.
• jp-realm-v0.1 corpus — 30-question hand-authored corpus (27 single-hop, 3 two-hop) covering JP's homelab + project knowledge. Authoring contract follows the spec's existing per-corpus convention; lives at sme/corpora/jp_realm_v0_1/questions.yaml.
• First live readings — head-to-head familiar vs daemon baselines, plus the v0.3 reflect-pipeline iterations that came out of running them. docs/ideas.md gains a "Lessons from the first live eval" section synthesizing what surfaced.

Design + plan docs in the PR

• docs/superpowers/specs/2026-04-26-familiar-adapter-design.md
• docs/superpowers/plans/2026-04-26-familiar-adapter.md

Notable refactor

refactor(adapters): extract /graph payload mapping into shared module (commit 4f213db in this PR) extracts the /graph snapshot mapping that both FamiliarAdapter and MemPalaceDaemonAdapter need, into sme/adapters/_graph_mapping.py. The daemon adapter is updated to use the shared module in this PR rather than the prior PR — keeps the prior PR scoped to "introduce the daemon adapter" and the refactor scoped to "share the part that's now duplicated."

Lessons doc highlights

The new section in docs/ideas.md captures the diagnostic patterns that fell out of the first run — including the system-gap-vs-pipeline-gap split that the q12/q13 case study illustrates (writing the missing drawers via palace-daemon flipped q12 from 0.0 to 1.0 — palace gap; left q13 at 0.0 — pipeline gap). This is the diagnostic the framework was built to provide; nice to see it land cleanly with the simplest possible scoring.

Testing

• tests/test_familiar_adapter.py — 26 tests covering query happy/error paths, graph snapshot, ingest stub, mock/no-mock modes, timeout behavior.
• tests/test_familiar_live.py — gated live smoke (requires FAMILIAR_LIVE=1), skipped in CI.
• All daemon tests from PR 1 still pass after the shared-module refactor.

Test plan

• python -m pytest tests/test_familiar_adapter.py tests/test_mempalace_daemon_adapter.py -v — 54 tests pass (26 familiar + 28 daemon)
• sme-eval retrieve --adapter familiar --questions sme/corpora/jp_realm_v0_1/questions.yaml --familiar-url $URL — CLI smoke against a running familiar
• cat baselines/jp_realm_v0_1_familiar_*.json | jq '.summary.mean_recall' — eyeball the recall progression across the v0.2/v0.3 baselines

The 19 commits split into design/plan, shared-module refactor, scaffold, query, graph snapshot, CLI wiring, then corpus authoring and the v0.2/v0.3 baseline iterations + lessons doc in roughly that order. Happy to walk through any specific commit if useful.

[Copilot]

Pull request overview

Adds two new read-side adapters (mempalace-daemon, familiar) to run SME retrieval + graph-snapshot evaluations against live HTTP services, plus a new JP-specific diagnostic corpus and first baseline artifacts / lessons writeup.

Changes:

• Introduce MemPalaceDaemonAdapter (palace-daemon HTTP) and FamiliarAdapter (familiar.realm.watch HTTP) with shared /graph payload projection.
• Wire new adapters + flags into the CLI and add extensive mocked + gated-live tests.
• Add the jp-realm-v0.1 question corpus plus baseline JSON outputs and documentation updates.

Reviewed changes

Copilot reviewed 26 out of 26 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/test_mempalace_daemon_integration.py	Gated live smoke tests for the daemon adapter.
tests/test_mempalace_daemon_adapter.py	Mocked unit tests for daemon adapter query + graph snapshot (fast-path + MCP fallback).
tests/test_familiar_live.py	Gated live smoke tests for FamiliarAdapter.
tests/test_familiar_adapter.py	Mocked unit tests for FamiliarAdapter behavior and CLI loading.
tests/fixtures/tiny_questions.yaml	Minimal smoke corpus fixture.
tests/conftest.py	Adds fake_urlopen_factory + _FakeResponse for HTTP-mocking urllib adapters.
sme/corpora/jp_realm_v0_1/questions.yaml	Adds the jp-realm-v0.1 diagnostic question corpus.
sme/cli.py	Adds adapter loading branches + new flags (--api-key, --kind, --mock/--no-mock, --familiar-timeout) and threads them into retrieve.
sme/adapters/mempalace_daemon.py	Implements palace-daemon HTTP adapter (/search, /graph, MCP fallback).
sme/adapters/familiar.py	Implements familiar HTTP adapter (/api/familiar/eval, /api/familiar/graph) and payload deserialization.
sme/adapters/_graph_mapping.py	Shared projection from /graph payload to (Entity, Edge) lists.
docs/superpowers/specs/2026-04-26-familiar-adapter-design.md	Design spec for FamiliarAdapter.
docs/superpowers/specs/2026-04-25-mempalace-daemon-adapter-design.md	Design spec for MemPalaceDaemonAdapter.
docs/ideas.md	Adds “Lessons from the first live eval” section and adapter forward-compat notes.
baselines/jp_realm_v0_1_familiar_v0.3_task1_bm25.json	Baseline output artifact (familiar).
baselines/jp_realm_v0_1_familiar_v0.3_reflect.json	Baseline output artifact (familiar).
baselines/jp_realm_v0_1_familiar_v0.3_candidate_window.json	Baseline output artifact (familiar).
baselines/jp_realm_v0_1_familiar_v0.3.9_reflect_cumulative.json	Baseline output artifact (familiar).
baselines/jp_realm_v0_1_familiar_20260426_v4_punct_fix.json	Baseline output artifact (familiar).
baselines/jp_realm_v0_1_familiar_20260426_v3.json	Baseline output artifact (familiar).
baselines/jp_realm_v0_1_familiar_20260426_v2.json	Baseline output artifact (familiar).
baselines/jp_realm_v0_1_familiar_20260426.json	Baseline output artifact (familiar).
README.md	Documents fork roadmap and usage for the new adapters/flags.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The skip condition only checks PALACE_DAEMON_URL, but the fixture constructs MemPalaceDaemonAdapter() which also requires PALACE_API_KEY (or an env_file). If a developer sets the URL but forgets the key, this will fail the test instead of skipping. Consider extending the skipif to require both PALACE_DAEMON_URL and PALACE_API_KEY (or otherwise skipping with a clearer reason).

Suggested change

	not os.environ.get("PALACE_DAEMON_URL"),
	reason="needs a running palace-daemon; set PALACE_DAEMON_URL to enable",
	not os.environ.get("PALACE_DAEMON_URL")
	or not os.environ.get("PALACE_API_KEY"),
	reason=(
	"needs a running palace-daemon; set PALACE_DAEMON_URL and "
	"PALACE_API_KEY to enable"
	),

[Copilot]

FamiliarAdapter.query() hard-codes "limit": 5 and does not accept an n_results kwarg, so the CLI's --n-results setting is silently ignored for this adapter (cmd_retrieve falls back after a TypeError). Consider adding an optional n_results parameter (and threading it through as the request's "limit") so familiar runs are comparable at different K values and the CLI flag behaves consistently across adapters.

[Copilot]

FamiliarAdapter.get_flat_retrieval() doesn't match the SMEAdapter optional-method contract: SMEAdapter.get_flat_retrieval returns a QueryResult, but this implementation returns list[Entity] and adds a nonstandard k parameter. This will break any generic Cat 7 / tooling that expects a QueryResult shape from get_flat_retrieval. Suggest returning a QueryResult (with context_string/answer empty if desired) and using the same n_results kwarg convention as other adapters.

Suggested change

	def get_flat_retrieval(self, question: str, k: int = 5) -> list[Entity]:
	"""Optional: same path as query() but returns only the entities,
	used by Cat 7 token-budget analysis where the answer text is
	irrelevant and only retrieval changes."""
	body = {
	"query": question[:250],
	"limit": k,
	"kind": "content",
	"mock": self.mock_inference,
	}
	status, payload = self._post_json("/api/familiar/eval", body)
	if status != 200 or not isinstance(payload, dict):
	return []
	raw_entities = payload.get("retrieved_entities") or []
	return [self._entity_from_payload(e) for e in raw_entities]
	def get_flat_retrieval(self, question: str, n_results: int = 5) -> QueryResult:
	"""Optional: same path as query() but returns the standard
	QueryResult shape so generic SME tooling can consume retrieval-only
	calls through the base adapter contract."""
	body = {
	"query": question[:250],
	"limit": n_results,
	"kind": "content",
	"mock": self.mock_inference,
	}
	status, payload = self._post_json("/api/familiar/eval", body)
	if status == -1:
	msg = payload.get("_network_error", "unknown network error") if isinstance(payload, dict) else "unknown error"
	return QueryResult(
	answer="",
	context_string="",
	retrieved_entities=[],
	retrieved_edges=[],
	error=f"familiar {msg}",
	)
	if status != 200:
	snippet = json.dumps(payload)[:200] if isinstance(payload, dict) else str(payload)[:200]
	return QueryResult(
	answer="",
	context_string="",
	retrieved_entities=[],
	retrieved_edges=[],
	error=f"familiar endpoint {status}: {snippet}",
	)
	return self._deserialize_query(payload)

[Copilot]

FamiliarAdapter.get_ontology_source() returns the string "declared", but SMEAdapter.get_ontology_source() is documented/used as a dict with keys like {type, schema, documentation} (e.g. cmd_analyze expects ontology.get(...)). Returning a string will raise at call sites if this adapter is used with ontology-reporting commands. Consider returning a dict like {"type": "declared", "schema": [...], "documentation": "..."} even if schema is empty for now.

Suggested change

	def get_ontology_source(self) -> str:
	"""Wings/rooms taxonomy is author-declared (mempalace_get_taxonomy
	MCP tool), not inferred from data. Same semantics as
	MemPalaceDaemonAdapter."""
	return "declared"
	def get_ontology_source(self) -> dict[str, Any]:
	"""Return a declared ontology descriptor compatible with SME
	ontology-reporting call sites.
	Familiar's wings/rooms taxonomy is author-declared
	(mempalace_get_taxonomy MCP tool), not inferred from graph data.
	The schema is currently surfaced as empty until this adapter
	exposes the declared taxonomy structure directly.
	"""
	return {
	"type": "declared",
	"schema": [],
	"documentation": (
	"Wings/rooms taxonomy is author-declared via the "
	"mempalace_get_taxonomy MCP tool and is not inferred from "
	"retrieved graph data."
	),
	}

[Copilot]

sme/adapters/mempalace_daemon.py imports defaultdict but doesn't use it (graph projection moved into sme.adapters._graph_mapping). With ruff enabled (pyproject.toml includes ruff in dev deps), this will fail lint due to an unused import. Remove the unused import to keep the module clean.

Suggested change

from collections import defaultdict

[M0nkeyFl0wer]

Approved. Stack continues to land cleanly.

Local verification on pr-6:

• pytest tests/test_familiar_adapter.py -v — 26/26 pass
• Full suite — 90 passed, 5 skipped
• ruff check — clean

What I liked:

• Refactor before composition. Extracting _project_graph into sme/adapters/_graph_mapping.py:project_graph rather than copy-pasting it into FamiliarAdapter is the right move — it pins the projection contract to a single function that both adapters import. The "Extracted verbatim from MemPalaceDaemonAdapter._project_graph 2026-04-26" comment makes the provenance auditable. Resolves the deferred unused-import lint from PR #5 too.
• mock_inference=True by default is the right call for SME runs. Substring scoring needs determinism; LLM-in-the-loop reads belong in a Cat 9 pass. The docstring says this explicitly, which is good — anyone who wonders "why are we mocking the inference?" gets the answer immediately.
• Familiar pipeline measurement strategy is well-framed. "POST /api/familiar/eval already returns SME-QueryResult shape natively" — designing familiar v0.2's chunk-C against the SME contract means the adapter is mostly deserialization, which is exactly the surface area we want at the eval boundary. Comparing familiar vs daemon adapter scores on the same corpus (the diff measuring rerank/decay/compress/grounding contribution) is a clean isolation.
• jp-realm-v0.1 corpus — 30 questions, expected_sources as substrings (matches the framework's existing matcher), Mode B diagnostic framing (no ingest, queries target known-existing content). The "seed authored 2026-04-26 by Claude with JP" note is honest about provenance.
• 8 baseline JSON files at consistent envelope shape (total, full_recall, partial_hit, mean_recall, mean_tokens) — the v0.2 → v0.3.x progression is captured, which means future regressions are diff-able rather than re-discoverable.

No requested changes.

Note for future spec work: the substring-on-context_string matcher inherited from the existing framework has the same precision/recall caveats I documented in mempalace#101 (filenames-containing-subjects scoring spuriously high). Worth flagging in a follow-up issue rather than blocking this PR; the corpus authoring contract you've outlined in tests/fixtures/tiny_questions.yaml and the v0.1 questions is independent of the matcher's known limitations.