Skip to content

Bug: vault_ask answers "no source" when retrieved chunks do contain the answer #40

@raphasouthall

Description

@raphasouthall

Summary

vault_ask occasionally returns a "the provided sources do not contain information about X" / [[no source]] answer even when the chunks it lists in sources[] clearly include the requested information (and vault_search confirms the same notes carry that information verbatim).

Distinct from #25 (confidently-wrong answers) — this is a confidently-empty answer: the synthesis LLM has the right chunks retrieved but reports them as absent.

Reproduction shape

  1. Edit a note so it contains an explicit, unambiguous statement of a fact F (e.g. an inline callout: "X is not deployed — verified ").
  2. Wait for re-index (confirmed by vault_search returning the new content and the regenerated note summary already mentioning F).
  3. Call vault_ask with a question directly targeting F, scoped to the workspace where the note lives.

Observed: sources[] includes the edited note(s); answer says the sources don't contain the information / cites [[no source]].

vault_ask on a slightly different phrasing of the same question (broader, less direct) does surface F correctly with citation — so the relevant chunk is reachable, just not selected for the sharper question.

Expected

If the retrieved chunks include the answer, synthesis should surface it (or at minimum quote the contradicting chunk). If the answer LLM judges the retrieved context insufficient, it should say "retrieved chunks were not specific enough" rather than asserting absence.

Possible angles

  • Top-k chunk selection may be retrieving the note but slicing a section that doesn't include the relevant paragraph. Worth logging which chunk offsets were actually passed to the synthesis prompt vs. just the source note paths.
  • Synthesis prompt may bias toward "I cannot answer" when chunks don't textually mirror the question's wording.
  • Could the answer LLM be receiving fewer chunks than sources[] implies? sources[] may be listing retrieved-but-not-passed notes.

Suggested fix

  • Log chunk-level content actually passed to the synthesis prompt (in addition to sources[] paths).
  • Add a synthesis check: if sources[] is non-empty but the answer asserts "no source", run a second verification pass that scans the chunk text for the entities the question used.
  • Consider returning chunks (not just paths) in sources[] so callers can spot the mismatch.

Workaround

Prefer vault_search with depth=full for accuracy-sensitive queries; treat vault_ask as exploratory.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions