Spike: Context-efficient documentation search & retrieval

[anand-testcompare]

Why This Exists

Current docs retrieval can consume too much context and encourage overly broad browsing before answering.

Desired Outcome

A retrieval approach that returns less noise and more relevance per token.

Scope

In scope (spike):

• Evaluate high-impact ways to reduce token cost and improve relevance.
• Compare approaches such as scope filtering, structural retrieval, and compact response formats.
• Propose a practical v1 direction with measurable impact.

Out of scope:

• Shipping a full re-architecture in this issue.

Exit Criteria

• Baseline token/cost profile documented for current behavior.
• Candidate approaches compared with pros/cons.
• Recommended v1 approach + follow-up implementation issues created.

Notes

Related to docs usability work in #19, but this spike is broader and focused on retrieval efficiency strategy.

[anand-testcompare]

If anyone has a good idea for this I'm open to the help: TLDR palantir docs are so spread out between different sections, sometimes only in announcements. It's fun when you're in exploration mode, but awful for LLMs who WILL hallucinate that something isn't possible cuz it 'checked the docs' but not the right one of 3-4 places. Don't believe me, try to find the docs for a custom vertex node/relationship logic. I've done the engineering work by ripping the docs into a raw parquet (using the same mechanism the UI uses behind the scenes

If nobody has a better idea... what i'm going to do is shove each page through an llm.. that llm will look for things like: palantir feature(s), programming language, maybe a few other tags.. , links/backlinks (to other docs), vector embedding. it will create something like this... (my obsidian notes)

that will all get crunched into a new parquet --> then we expose some opencode tools for 'quickSearch' which will do some high level vector embedding query the agent can use (with filters only for values that are in a fixed set like 'product') which will provide the underlying page references for the relevant chunks retrieve... and then there will be something like a 'deepRetrieval' where if the agent determines the most relevant docs... we assemble the larger relationship matrix and pull those docs even if they didn't come up in the initial vector embedding query by navigating links / tags. hybrid RAG/GraphRAG.

we need to keep it efficient tho. the pre-processing can be slower and run infrequently then exposed as a parquet or if needed libsql/sqllite others can just download. i dont want people dependent on a central server, but if opt in that's fine. maybe tursoDB is a good option here as there is they have some special vector type if we hit limitations with just doing parquet and processing in memory.

the part i don't love is this feels a bit elaborate. maybe thats just what's needed cuz if it was easy to unfuck someone else would have. will think on it for a couple days, if no input i will proceed w/ my plan