Feat/tool descriptions and confidence, MCP server

[bs258q]

feat: tool descriptions, confidence tracking, and no-match fallback

What

Three independent improvements to constrained decoding and inference:

1. Confidence threshold + no-match fallback

Always return best guess (default — no behavior change)

result = generate(model, params, tokenizer, query, tools)

Return no-match sentinel when model is uncertain

result = generate(..., threshold=0.5)

→ '{"match":false,"confidence":0.31}' when confidence < threshold

Get confidence score alongside result

result, conf = generate(..., return_confidence=True)

3. needle playground — launches a local web UI for testing and fine-tuning.

4. browser/workbench/workbench.html) — starts Needle as a local MCP tool router for agent workflows.

5. browser/workbench/workbench.html) — runs a single inference request with tool support.
   Add tokenizer training pipeline and extract src/tokenizer.py #6. Tool description support

Include a "description" field in each tool object for better semantic
matching. The encoder already tokenizes the full tools JSON string, so
descriptions flow through automatically — no structural change required.

{"name": "get_weather", "description": "Get current weather for a location", "parameters": {"location": "string"}}

Confidence = softmax max probability over valid name tokens at the first
constrained step. Useful for production: escalate uncertain calls to a
cloud LLM instead of silently returning a wrong tool.