Skip to content

TOOL_AUDIT.md

Latest commit

 

History

History
200 lines (162 loc) · 7.59 KB

TOOL_AUDIT.md

File metadata and controls

200 lines (162 loc) · 7.59 KB

Free Agent Tool Audit

Tool Mapping Analysis

Edge Function Tools (21 tools)

Tools that call Supabase edge functions via the free-agent orchestrator:

Tool ID Edge Function Status Notes
get_time time ✅ Mapped Utility tool
brave_search brave-search ✅ Mapped Web search - HIGH PRIORITY
google_search google-search ✅ Mapped Web search - HIGH PRIORITY
web_scrape web-scrape ✅ Mapped Content extraction - HIGH PRIORITY
read_github_repo github-fetch ✅ Mapped Code analysis - HIGH PRIORITY
read_github_file github-fetch ✅ Mapped Code analysis - HIGH PRIORITY
send_email send-email ✅ Mapped Communication
image_generation run-nano ✅ Mapped Generation
get_call_api api-call ✅ Mapped API
post_call_api api-call ✅ Mapped API
execute_sql external-db ✅ Mapped Database
read_database_schemas external-db ✅ Mapped Database
elevenlabs_tts elevenlabs-tts ✅ Mapped Generation
get_weather tool_weather ✅ Mapped Utility
read_zip_contents tool_zip-handler ✅ Mapped File
read_zip_file tool_zip-handler ✅ Mapped File
extract_zip_files tool_zip-handler ✅ Mapped File
pdf_info tool_pdf-handler ✅ Mapped Document
pdf_extract_text tool_pdf-handler ✅ Mapped Document
ocr_image tool_ocr-handler ✅ Mapped Document
pronghorn_post pronghorn-post ✅ Mapped Export

Frontend Tools (15 tools)

Tools handled directly in the React frontend:

Tool ID Handler Function Status Notes
read_blackboard executeReadBlackboard ✅ Implemented Memory - HIGH PRIORITY
write_blackboard executeWriteBlackboard ✅ Implemented Memory
read_file executeReadFile ✅ Implemented File
request_assistance executeRequestAssistance ✅ Implemented Interaction
export_word executeExportWord ✅ Implemented Export
export_pdf executeExportPdf ✅ Implemented Export
read_scratchpad executeReadScratchpad ✅ Implemented Memory - HIGH PRIORITY
write_scratchpad executeWriteScratchpad ✅ Implemented Memory - HIGH PRIORITY
read_prompt executeReadPrompt ✅ Implemented Memory
read_prompt_files executeReadPromptFiles ✅ Implemented Memory
read_attribute executeReadAttribute ✅ Implemented Memory - HIGH PRIORITY
read_artifact executeReadArtifact ✅ Implemented Memory
read_self executeReadSelf ✅ Implemented Advanced
write_self executeWriteSelf ✅ Implemented Advanced
spawn executeSpawn ✅ Implemented Advanced

Total: 36 Tools ✅

Known Issues & Improvements Needed

1. Context Passing Issues

Problem: previousToolResults only visible for ONE iteration (ephemeral)

Location: supabase/functions/free-agent/index.ts line 1075-1089

Fix: Enhanced error messages and warnings when tools are called repeatedly

2. Generic Error Messages

Problem: Errors like "LLM API error: 500" don't provide actionable guidance

Locations:

  • supabase/functions/free-agent/index.ts line 906-908 (LLM errors)
  • supabase/functions/free-agent/index.ts line 809-813 (Tool errors)

Fix: Add specific error messages with recovery suggestions

3. Tool Instance Support

Status: ✅ Already implemented (line 738-745)

  • Supports baseTool:instanceName format
  • Correctly looks up secrets from instance-specific overrides

4. SaveAs Parameter

Status: ✅ Documented but could be enhanced

  • All edge tools support saveAs parameter
  • Results automatically stored as named attributes
  • Could add better confirmation messages

5. Reference Resolution

Status: ✅ Fully implemented (lines 221-291)

  • Supports {{scratchpad}}, {{blackboard}}, {{attributes}}, {{attribute:name}}, {{artifacts}}, {{artifact:id}}
  • Works across all tool parameters

Priority Tool Testing Checklist

HIGH PRIORITY (Most used, highest failure risk)

  • brave_search - Test with various queries, validate numResults parameter
  • google_search - Test with various queries, validate numResults parameter
  • web_scrape - Test HTML, PDF, DOCX handling, validate maxCharacters
  • read_github_repo - Test public/private repos, branch parameter
  • read_github_file - Test single/multiple files, outputMode parameter
  • read_attribute - Test empty array (list all), specific names, non-existent attributes
  • write_scratchpad - Test append/replace modes, large content handling

MEDIUM PRIORITY

  • get_call_api - Test GET requests with headers, auth
  • post_call_api - Test POST requests with body, headers
  • send_email - Test plain text and HTML emails
  • image_generation - Test prompt generation
  • execute_sql - Test read and write queries
  • read_database_schemas - Test schema retrieval

LOWER PRIORITY

  • get_time - Simple utility, low risk
  • get_weather - Weather API call
  • read_blackboard - Frontend handler, well-tested
  • write_blackboard - Frontend handler, well-tested
  • ZIP tools (3 tools) - File handling
  • PDF tools (2 tools) - Document processing
  • ocr_image - OCR processing
  • pronghorn_post - Export functionality
  • elevenlabs_tts - TTS generation
  • Export tools (2 tools) - PDF/Word export

Recommendations

1. Enhanced Error Messages

Add actionable error messages to all edge functions:

// Before:
return { success: false, error: `${response.status}: ${errorText}` };

// After:
if (response.status === 429) {
  return {
    success: false,
    error: `Rate limit exceeded (429). Wait 60 seconds or try alternative: google_search instead of brave_search.`,
    retryable: true,
    fallbackTools: ['google_search']
  };
}

2. Tool Result Attributes Summary

Add to system prompt to show what's available:

## Available Attributes (Auto-Saved Data)

You have 3 saved attributes:
1. weather_data (from get_weather, iteration 3, 2.1 KB)
2. search_results (from brave_search, iteration 5, 15.3 KB)
3. repo_structure (from read_github_repo, iteration 7, 45.0 KB)

To access: read_attribute(["name"])

3. Scratchpad Validation

Prevent bloat by validating content size:

if (content.length > 50000) {
  return { success: false, error: "Content too large. Scratchpad is for SUMMARIES, not raw data." };
}

4. Loop Detection Improvements

Implement in useFreeAgentSession.ts:

  • Track last 3 tool calls with same signature
  • Check blackboard for repeated content
  • Inject intervention messages into previousToolResults

Edge Function Documentation Status

Function Documented Needs Doc Block
free-agent ❌ ✅
brave-search ❌ ✅
google-search ❌ ✅
web-scrape ❌ ✅
github-fetch ❌ ✅
send-email ❌ ✅
run-nano ❌ ✅
api-call ❌ ✅
external-db ❌ ✅
elevenlabs-tts ❌ ✅
tool_weather ❌ ✅
tool_zip-handler ❌ ✅
tool_pdf-handler ❌ ✅
tool_ocr-handler ❌ ✅
pronghorn-post ❌ ✅
time ❌ ✅

Next Steps

  1. ✅ Create comprehensive tool audit (THIS DOCUMENT)
  2. ⏭️ Add enhanced error messages to free-agent/index.ts
  3. ⏭️ Add formatAttributesSummary() function
  4. ⏭️ Add scratchpad validation to write_scratchpad handler
  5. ⏭️ Document all edge functions with comment blocks
  6. ⏭️ Create tool testing suite