Improve MCP tool descriptions for AI agent reliability

[janisz]

Summary

Enhanced tool descriptions with explicit usage patterns and STOP instructions to improve e2e test reliability with Claude models. Achieved
100% reliability with Opus/Sonnet and 90%+ with Haiku.

Problem

E2e tests with Claude CLI agents showed agents making excessive tool calls:

• General CVE questions ("Is CVE-X detected?") triggered 4-5 tool calls instead of expected 3
• Agents would call all three CVE tools, then verify results with additional calls
• This exceeded maxToolCalls=3-5 constraints and reduced reliability

Solution

1. CVE Tool Descriptions (clusters.go, deployments.go, nodes.go)

Added explicit STOP instructions for two usage patterns:

USAGE PATTERNS:

1. For general CVE questions ('Is CVE-X detected in my clusters?'):
   Call ALL THREE CVE tools exactly once each, then STOP and provide answer.
   Do NOT make verification calls or check twice.
2. For specific [deployment/node/orchestrator] questions: Use ONLY this tool once, then STOP.

Why: Prevents agents from making verification calls after getting initial results.

2. list_secured_clusters Description (tools.go)

Added WHEN TO USE guidance:

WHEN TO USE: Use this tool when the user asks to see or list all clusters.
IMPORTANT: Do NOT use this tool when checking for CVE vulnerabilities - use the CVE-specific tools instead.

Why: Prevents agents from calling list_secured_clusters when CVE-specific tools already filter by cluster.

3. Documentation (e2e-tests/README.md)

Added "Running Tests with Claude Code" section with:

• Prerequisites and setup instructions
• Agent configuration examples for Haiku, Sonnet, Opus
• Test results table showing model reliability

Test Results

Comprehensive testing with Claude CLI agents (20 runs each):

Model	Success Rate	Pattern
Opus	100% (20/20)	Perfect reliability
Sonnet	100% (20/20)	Perfect reliability
Haiku	90.0% (18/20)	Both failures in first 2 runs, then 18 consecutive passes

[github-actions]

E2E Test Results

Commit: a350bd0
Workflow Run: View Details
Artifacts: Download test results & logs

=== Evaluation Summary ===

  ✓ list-clusters (assertions: 3/3)
  ✓ cve-cluster-does-not-exist (assertions: 3/3)
  ✓ cve-detected-workloads (assertions: 3/3)
  ✓ cve-cluster-does-exist (assertions: 3/3)
  ✓ cve-clusters-general (assertions: 3/3)
  ✓ rhsa-not-supported (assertions: 2/2)
  ✓ cve-detected-clusters (assertions: 3/3)
  ✓ cve-multiple (assertions: 3/3)
  ✓ cve-log4shell (assertions: 3/3)
  ✓ cve-nonexistent (assertions: 3/3)
  ✓ cve-cluster-list (assertions: 3/3)

Tasks:      11/11 passed (100.00%)
Assertions: 32/32 passed (100.00%)
Tokens:     ~66874 (estimate - excludes system prompt & cache)
MCP schemas: ~13145 (included in token total)
Agent used tokens:
  Input:  19317 tokens
  Output: 26821 tokens
Judge used tokens:
  Input:  34472 tokens
  Output: 34006 tokens

[codecov-commenter]

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
361	2	359	12

View the full list of 2 ❄️ flaky test(s)

::policy 1

Flake rate in main: 100.00% (Passed 0 times, Failed 36 times)

Stack Traces | 0s run time

- test violation 1
- test violation 2
- test violation 3

::policy 4

Flake rate in main: 100.00% (Passed 0 times, Failed 36 times)

Stack Traces | 0s run time

- testing multiple alert violation messages 1
- testing multiple alert violation messages 2
- testing multiple alert violation messages 3

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}