Add SUGGESTION_MODE_MODEL env var to control suggestion mode LLM calls #42
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Problem: Each user message triggers three concurrent runAgentLoop calls
(main, suggestion, and others). When using large local models via Ollama,
each call takes 30-90 seconds of GPU time. Suggestion mode calls are low-value
and waste significant compute resources that could be serving the main request.
Changes implemented:
1. Suggestion mode detection (src/orchestrator/index.js)
- Added detectSuggestionMode() that scans the last user message for
the [SUGGESTION MODE: marker used by Claude Code's CLI
- Tags requests with _requestMode ("suggestion" vs "main") in sanitizePayload
2. Model override and skip logic (src/orchestrator/index.js, src/config/index.js)
- When SUGGESTION_MODE_MODEL=none, suggestion mode requests return an
empty response immediately without calling the LLM
- When set to a model name, suggestion mode uses that lighter model
instead of the default, freeing GPU for the main request
- When set to "default" (or unset), behavior is unchanged
3. Hallucination guard (src/orchestrator/index.js)
- Added guard to drop tool calls when no tools were sent to the model
- Some models (e.g. Llama 3.1) hallucinate tool_call blocks from
conversation history even when the request had zero tool definitions
4. Configuration (src/config/index.js, .env.example)
- Added SUGGESTION_MODE_MODEL env var with "default" as the default value
- Documented in .env.example with usage examples
5. Databricks client cleanup (src/clients/databricks.js)
- Minor adjustment to pass through suggestion mode model config
Testing:
- SUGGESTION_MODE_MODEL=none: suggestion mode calls return instantly (0ms)
- SUGGESTION_MODE_MODEL=llama3.2:1b: redirects to lighter model correctly
- SUGGESTION_MODE_MODEL=default (or unset): unchanged behavior
- Main agent loop unaffected in all cases
- Hallucination guard tested with Llama 3.1 no-tool requests
- npm run test:unit passes with no regressions
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
SUGGESTION_MODE_MODELenv var to skip or redirect suggestion mode LLM calls to a lighter modelProblem
Each user message triggers three concurrent
runAgentLoopcalls (main, suggestion, and others). Suggestion mode calls are helpful, but require LLM execution time (with local models potentially multiple seconds) and/or potentially creating payable token usage.Changes
detectSuggestionMode()to identify suggestion mode requests; early return whenSUGGESTION_MODE_MODEL=none; hallucination guard for models that hallucinate tool calls without toolsSUGGESTION_MODE_MODELconfig (default:"default")Configuration
Testing
SUGGESTION_MODE_MODEL=none: suggestion calls return instantly (0ms)SUGGESTION_MODE_MODEL=llama3.2:1b: redirects to lighter modelnpm run test:unitpasses with no regressions