Bug: LLM returns tool_use during synthesis phase (Phase 5)

[samkeen]

Description

During Phase 5 (Synthesis & Final Response), the LLM is unexpectedly returning tool_use content blocks instead of just text. This triggers a warning in the console but the workflow continues.

Error Message

[ClaudeClient] WARNING: Synthesis returned tool calls (unexpected)

Location

mcp-inspector-app/lib/llm/claude-client.ts:172-174

Expected Behavior

During synthesis (the second LLM inference), the LLM should only return text content blocks to synthesize the final natural language response using the tool results from Phase 4.

Actual Behavior

The LLM is returning both text and tool_use content blocks during synthesis. The code logs a warning and extracts the tool calls (which should be empty).

Server Logs

[ClaudeClient] Synthesis inference started
[ClaudeClient] Tool results: 1
[ClaudeClient] Synthesis inference completed in 3443ms
[ClaudeClient] Stop reason: tool_use
[ClaudeClient] Content blocks: 2
[POST /api/workflow/execute] Workflow complete: {
  success: true,
  toolsUsed: [ 'search_documentation' ],
  totalTime: 11207
}

Note: stop_reason: tool_use should be end_turn or similar during synthesis.

Test Case

Query: "Search AWS documentation for S3 bucket naming rules"

1. Phase 3 (Planning): LLM correctly selects search_documentation tool
2. Phase 4 (Execution): Tool executes successfully, returns results
3. Phase 5 (Synthesis): LLM returns tool_use blocks again (unexpected)

Root Cause Analysis

Possible causes:

1. Prompt issue: The synthesis prompt may not clearly indicate this is the final response phase
2. Tool definitions still present: We're passing tools parameter to the synthesis inference API call
3. Conversation history issue: The way we're building the conversation with tool results may confuse the model

Relevant Code

mcp-inspector-app/lib/llm/claude-client.ts:137-184

async synthesisInference(
  userMessage: string,
  planningContent: any[],
  toolResults: ClaudeToolResult[],
  tools: ClaudeTool[]
): Promise<InferenceResult> {
  // ...
  const response = await this.client.messages.create({
    model: this.model,
    max_tokens: this.maxTokens,
    messages,
    tools,  // <-- Should we remove tools here?
  });
  // ...
}

Potential Solutions

1. Remove tools parameter from synthesis call: Since we don't want the LLM to select more tools during synthesis
2. Add system prompt: Explicitly instruct the model this is the final response phase
3. Increase max_tokens: Current setting is 1024, which might be too low for complex responses
4. Validate conversation format: Ensure the buildConversationWithResults() function creates proper message structure

Impact

• Low severity: Workflow completes successfully
• Warning pollutes logs
• May cause confusion about the two-phase inference pattern
• Potential for infinite loops if we were to handle additional tool calls

Related

This is a key teaching point of the MCP Inspector app - showing that tool selection happens in Phase 3 (Planning) and synthesis happens in Phase 5 (Final Response). The LLM should NOT be making tool calls during synthesis.