[shiplog/plan] Reframe retrieval around map-first lexical search

[devallibus]

Context

shiplog currently describes itself as a GitHub-backed knowledge graph with optional memory-store integration, but its documented retrieval workflow is still too abstract. The current guidance says to search issues, PRs, commits, envelopes, and optional memory, but it does not define a clear retrieval order that gives the agent a cheap map before deeper reads.

The user supplied a design pattern from napkin: give the agent a terrain map first, run lexical search with visible match reasons, and treat native links as the graph instead of depending on embeddings or a vector store.

Design Summary

• Keep shiplog retrieval GitHub-native and explicit rather than relying on vector-memory rhetoric.
• Define a progressive-disclosure retrieval loop: map first, lexical search second, edge-following third, full-thread reads last.
• Use existing Shiplog structures as the graph: issue numbers, PR references, branch names, stacked markers, envelope fields, and timeline tags.
• Keep ork:remember / ork:memory optional and clearly secondary to the GitHub-native retrieval path.

Approach

Document a retrieval playbook that maps the napkin pattern onto Shiplog's existing primitives:

1. Build a terrain map from open issues, PRs, labels, titles, branch names, and latest state/history envelopes.
2. Run lexical search over issues, PRs, commits, and local repo docs before trying any optional memory layer.
3. Show why a result matched by surfacing headings or matching lines from the top hits before reading full threads.
4. Follow native edges such as #42, Closes #42, Addresses #42, [stack: #N], issue/<id>-..., and envelope references like issue, pr, supersedes, and amends.
5. Treat optional memory stores as an add-on for recall, not the default explanation of how Shiplog retrieval works.

Alternatives Considered

• Keep the current "knowledge graph" language unchanged: too vague about retrieval behavior, and it invites people to assume embeddings or hidden memory infra are central.
• Add a brand-new retrieval backend to Shiplog: out of scope for this repo, which is primarily a workflow/skill package.
• Drop optional memory integration entirely: unnecessary; the real fix is making the default retrieval path explicit and cheap.

Sources and Verification Status

• [verified] Napkin publicly describes a markdown-on-disk retrieval pattern centered on overview, lexical search, and wiki-link navigation.
  • Source: https://github.com/Michaelliv/napkin and https://www.npmjs.com/package/napkin-ai
  • Checked: The project describes local markdown retrieval, an overview/map concept, lexical search behavior, and native link traversal.
• [unverified] Benchmark figures supplied by the user for HotpotQA, including exact recall and F1 numbers.
  • Why unverified: I have not located a primary-source benchmark artifact in the upstream materials during this planning pass.
  • Next step: Verify the evaluation script or published benchmark notes before citing any numeric claims in Shiplog docs.

Tasks

• T1: Rewrite Phase 6 retrieval guidance around progressive disclosure [tier-3]

  • What: Update skills/shiplog/SKILL.md so Phase 6 explicitly defines a map-first, lexical, edge-following retrieval loop and positions optional memory as secondary.
  • Files: skills/shiplog/SKILL.md (modify)
  • Allowed to change: Phase 6 text, nearby retrieval wording that directly supports the new workflow
  • Must not change: Unrelated phases, branch/PR/review rules, or label semantics
  • Forbidden judgment calls: Do not introduce new product capabilities that are not documented elsewhere in the repo
  • Stop and ask if: The change would require adding new commands or non-doc code that is not already in scope
  • Verification: Read the updated phase and confirm the retrieval order is explicit, bounded, and consistent with artifact envelopes
  • Return artifact: Diff summary for skills/shiplog/SKILL.md
  • Decision budget: none
  • Accept when: A reader can tell what to do first, second, and third when looking up history
  • Context: This is the workflow definition that all other docs should align to

• T2: Update /shiplog lookup to show match reasons and edge-following [tier-3]

  • What: Revise commands/shiplog/lookup.md so it tells the agent to build a terrain map, inspect top hits for matching lines/headings, and follow native Shiplog edges before deep reading.
  • Files: commands/shiplog/lookup.md (modify)
  • Allowed to change: Task steps and output format for lookup
  • Must not change: Command purpose or Shiplog phase numbering
  • Forbidden judgment calls: Do not pretend the command already has BM25 or vector-search capabilities
  • Stop and ask if: The command needs tooling that cannot be expressed with gh, git, and read access
  • Verification: Read the command top-to-bottom and confirm it describes a coherent operator loop
  • Return artifact: Diff summary for commands/shiplog/lookup.md
  • Decision budget: none
  • Accept when: The command tells the operator not just what matched, but why and what to follow next
  • Context: This is where the map-first pattern becomes user-visible

• T3: Reposition README retrieval claims around explicit maps instead of hidden memory infrastructure [tier-3]

  • What: Update README.md so the retrieval story emphasizes progressive disclosure, GitHub-native links, and low-cost lookup before optional memory integrations.
  • Files: README.md (modify)
  • Allowed to change: Retrieval-related feature copy, command descriptions, and companion-skill wording tied to memory
  • Must not change: Installation instructions, unrelated feature descriptions, or the existing testimonial section unless needed for consistency
  • Forbidden judgment calls: Do not cite unverified benchmark numbers or claim Shiplog ships BM25/TF-IDF today
  • Stop and ask if: The README change requires a broader product-positioning rewrite outside retrieval
  • Verification: Read the edited sections and confirm they do not overclaim implementation details
  • Return artifact: Diff summary for README.md
  • Decision budget: none
  • Accept when: The README makes the retrieval philosophy legible without suggesting a vector backend
  • Context: The public-facing message should match the actual documented workflow

Open Questions

• Should Shiplog introduce a dedicated overview or map command later, or is hunt enough as the top-level terrain map?
• Should native edge-following be documented only in Phase 6, or also called out in resume/start flows?
• Is the phrase "knowledge graph" still worth keeping as a metaphor if the docs now emphasize lexical retrieval and explicit references?

---

Authored-by: openai/gpt-5.4 (codex)

[devallibus]

[shiplog/session-start] Reframing retrieval around map-first lexical search

Branch: issue/139-map-first-lexical-search
Worktree: C:/Users/Usuario/Documents/Developer/GitHub/issue/139-map-first-lexical-search
Approach: Update Phase 6, /shiplog lookup, and the README so retrieval is described as map first, lexical search second, native-edge traversal third, and optional memory last.

---

Authored-by: openai/gpt-5.4 (codex)
Captain's log — session start