Feature/triage#25
Open
manankapoor23 wants to merge 10 commits into
Open
Conversation
…sting - Created utils/local_store.py with LocalStore and Collection classes - Implements cosine similarity search using numpy - Maintains full compatibility with existing retriever interface - Updated loader.py to use local store instead of ChromaDB - Updated retriever.py to use local store for embedding queries - Kept BM25 fallback for low-confidence results - Updated config.py: CHROMA_DIR → LOCAL_STORE_DIR - Removed chromadb from requirements.txt - Updated main.py help text
## Pipeline Architecture - **Agents**: Router, Retriever, Grounding, Justification, LLM, Product Area - **Safety**: Pre-flight and post-flight validation checks - **Pipeline**: Orchestrates all agents in correct order ## Components Added ### 1. Router Agent (agents/router.py) - Keyword-based domain classification (claude, hackerrank, visa) - Confidence scoring without LLM cost - LLM fallback only when confidence < threshold - Handles company fuzzy matching ### 2. Retriever Agent (agents/retriever.py) - Semantic search via local embeddings store - BM25 fallback for low-confidence results - Returns top-5 relevant chunks with scores - Extracts metadata: domain, product_area, source_url, title ### 3. Product Area Agent (agents/product_area.py) - Extracts product_area from top retrieved chunk - Zero LLM cost (corpus-grounded) - Fallback to domain if no chunks available ### 4. Grounding Agent (agents/grounding.py) - Validates retrieved context is relevant - Returns confidence level and reasoning - Determines if enough context for safe response ### 5. LLM Agent (agents/llm-agent.py) - Generates response from context and ticket - Uses Claude Sonnet 4 with system prompts - Streaming support for large responses ### 6. Justification Agent (agents/justification.py) - Explains why response was generated - Cites sources and reasoning - Part of transparency pipeline ### 7. Safety Agents (agents/pre-safe.py, agents/post-safe.py) - Pre-flight: Injection detection, sensitive keyword checking - Post-flight: Response validation, safety filtering - Escalation triggers for high-risk issues ## Utilities - utils/chunker.py: Text chunking (512 tokens, 64 overlap) - utils/embeddings.py: sentence-transformers wrapper - utils/logger.py: Structured decision + runtime logging ## Pipeline Orchestration (pipeline.py) - Runs all agents in sequence for each ticket - Logs decisions and actions - Returns complete output dict for CSV writing - Handles null/empty inputs gracefully
## API Migration - ANTHROPIC_API_KEY → OPENAI_API_KEY in config.py - claude-sonnet-4-20250514 → gpt-4o for better performance/cost ratio ## Code Changes - agents/llm-agent.py: anthropic.Anthropic → OpenAI client - Updated message format to use OpenAI chat completions API - Added system message as first message in array - agents/router.py: anthropic.Anthropic → OpenAI client - Simplified message format for classification task - config.py: Updated API key env variable and model name - requirements.txt: Removed anthropic==0.97.0, added openai>=1.40.0 ## Environment Setup - Create/update code/.env with: OPENAI_API_KEY=your_key_here - Set environment variable before running: export OPENAI_API_KEY=your_key - .env file is gitignored for security ## API Compatibility - OpenAI chat.completions.create() API fully compatible with existing pipeline - No changes to pipeline.py orchestration required - All agent interfaces remain unchanged
## Changes - Added python-dotenv==1.0.1 to requirements.txt for .env file support - Updated config.py to load .env file using load_dotenv() - Environment variables now automatically loaded from code/.env ## Test Files Created ### test_openai_simple.py (Recommended) - Verifies .env file is loaded correctly - Tests OpenAI client initialization - Makes simple API call to verify credentials work - Tests router agent keyword matching (no API calls needed) - Provides helpful error messages and next steps ### test_openai.py (Comprehensive) - Full integration testing of all components - Tests basic API connectivity - Tests router and LLM agents - More detailed error reporting ## Usage 1. Update code/.env with your OpenAI API key 2. Run: python test_openai_simple.py 3. If all pass: python main.py --sample ## Test Results ✅ API Key loads from .env file ✅ OpenAI client initializes successfully ✅ API calls work (tested with simple prompt) ✅ Router agent keyword matching works (2/3 domains)
## Problem - Output CSV had duplicate columns (both capitalized and lowercase) - Extra columns like 'justification' that weren't needed ## Solution - Build output dataframe from scratch with only required columns - Input columns: Issue, Subject, Company - Output columns: Response, Product Area, Status, Request Type - Proper capitalization matching sample format ## Before Issue,Subject,Company,Response,Product Area,Status,Request Type,status,product_area,response,justification,request_type ## After Issue,Subject,Company,Response,Product Area,Status,Request Type Tested: ✅ Pipeline produces clean output CSV
- Hard grounding enforcement (score >= threshold required) - Forbidden pattern blocking (try, ensure, check, etc.) - Context-only generation (LLM must use only retrieved context) - Post-generation guardrails - Status alignment (escalated message -> escalated status) - Clean README and .env template
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.