Solution/support triage agent#30
Open
kauntiaakash2 wants to merge 5 commits into
Open
Conversation
…r.py-and-llm.py Robust Anthropic client handling, safer LLM response extraction and fallback, and TF-IDF fix for tiny corpora
…ion engine - Add 5-module pipeline: retriever, classifier, escalation, llm, agent - TF-IDF corpus index over hackerrank/claude/visa docs (deterministic, offline) - Rule-based domain detection + hard-escalation for fraud, billing disputes, account compromise, assessment integrity, lost/stolen card, legal signals - LLM response generation via Claude API grounded strictly in corpus context - Graceful degradation: full rule-based fallback when API key is absent - Fix ThinkingBlock AttributeError in API response parsing - 95% accuracy on sample_support_tickets.csv - Terminal CLI: python code/main.py [--sample] [--verbose] [--ticket] - Secrets via env vars only (ANTHROPIC_API_KEY); temperature=0 for determinism
There was a problem hiding this comment.
Pull request overview
Adds a terminal-based, multi-domain (HackerRank/Claude/Visa) support triage agent that routes tickets using an offline TF‑IDF retriever + rule-based classification/escalation, with optional Anthropic Claude assistance for classification/response generation.
Changes:
- Introduces the core triage pipeline (retrieval → classify → escalate → respond) and a terminal CLI for batch + single-ticket processing.
- Implements a deterministic local-corpus TF‑IDF retriever with a keyword fallback plus context formatting.
- Adds corpus management utilities (verification/stats + optional enrichment) and commits generated output CSVs for evaluation.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| support_tickets/sample_output.csv | Adds sample run output in the required 5-column schema. |
| support_tickets/output.csv | Updates/commits generated output for the main ticket set. |
| code/scraper.py | Adds corpus verification/stats plus optional enrichment/placeholder corpus tooling. |
| code/retriever.py | Implements corpus loading, chunking, TF‑IDF indexing, and retrieval utilities. |
| code/requirments.txt | Adds dependency list (but filename typo impacts install flow). |
| code/main.py | Adds terminal CLI, CSV I/O, progress display, and execution modes. |
| code/llm.py | Adds Anthropic client wrapper for classification/response/justification with fallbacks. |
| code/escalation.py | Adds rule-based escalation decision engine with reasons/risk levels. |
| code/env_utils.py | Adds .env loader helper. |
| code/classifier.py | Adds rule-based domain/request_type/product_area detection + risk signals. |
| code/agent.py | Adds orchestrator that ties retrieval, classification, escalation, and LLM together. |
| code/README.md | Documents architecture, setup, and usage of the agent. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+170
to
+179
| """Extract focused sub-queries from the ticket text.""" | ||
| import re | ||
| # Extract noun phrases and technical terms (simple heuristic) | ||
| phrases = [] | ||
| # Sentences as sub-queries | ||
| sentences = re.split(r"[.!?]\s+", issue) | ||
| for s in sentences[:3]: | ||
| s = s.strip() | ||
| if len(s.split()) >= 3: | ||
| phrases.append(s) |
Comment on lines
+134
to
+140
| if llm_result: | ||
| self._log(f"LLM classification: {llm_result}") | ||
| # Trust LLM for product_area and request_type refinement | ||
| if llm_result.get("product_area"): | ||
| product_area = llm_result["product_area"] | ||
| if llm_result.get("request_type") in ("product_issue", "feature_request", "bug", "invalid"): | ||
| request_type = llm_result["request_type"] |
Comment on lines
+1
to
+4
| # Core dependencies | ||
| anthropic>=0.25.0 # Claude API client | ||
| scikit-learn>=1.4.0 # TF-IDF vectorizer + cosine similarity | ||
| numpy>=1.26.0 # Numerical operations |
Comment on lines
+183
to
+191
| _INDEX: Optional[CorpusIndex] = None | ||
|
|
||
|
|
||
| def get_index(data_dir: Path = DATA_DIR) -> CorpusIndex: | ||
| global _INDEX | ||
| if _INDEX is None: | ||
| chunks = load_corpus(data_dir) | ||
| _INDEX = CorpusIndex(chunks) | ||
| return _INDEX |
Comment on lines
+50
to
+56
| "bug": re.compile( | ||
| r"\b(bug|broken|error|crash|not\s*working|doesn['\']t\s*work|" | ||
| r"glitch|issue\s*with|failing|failed|exception|500|404|" | ||
| r"incorrect\s*result|wrong\s*output|unexpected\s*behavior|" | ||
| r"regression|stopped\s*working|can['\']t\s*load|freezing|" | ||
| r"infinite\s*loop|timeout)\b", | ||
| re.IGNORECASE, |
|
|
||
| # ── Step 7: LLM refinement (if API available) ───────────────────────── | ||
| llm_result = None | ||
| if not escalation.should_escalate or escalation.risk_level in ("low", "medium"): |
Comment on lines
+214
to
+218
| args = parse_args() | ||
|
|
||
| # Check API key | ||
| load_env_file() | ||
| api_key = os.environ.get("ANTHROPIC_API_KEY", "") |
Comment on lines
+72
to
+75
| ```bash | ||
| cd code/ | ||
| pip install -r requirements.txt | ||
| ``` |
Comment on lines
+122
to
+128
| """ | ||
| for block in content_blocks: | ||
| # Handle both object-style and dict-style blocks | ||
| block_type = getattr(block, "type", None) or (block.get("type") if isinstance(block, dict) else None) | ||
| if block_type == "text": | ||
| return getattr(block, "text", None) or (block.get("text", "") if isinstance(block, dict) else "") | ||
| return "" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Solution: Multi-Domain Support Triage Agent
What it does
Terminal-based support triage agent across HackerRank, Claude, and Visa
using only the local corpus in data/. For each ticket: detects domain,
retrieves relevant docs, classifies risk, and either replies or escalates.
Architecture
Run
pip install -r code/requirements.txt
export ANTHROPIC_API_KEY=sk-ant-...
python code/main.py # → writes support_tickets/output.csv
python code/main.py --sample # → validate against expected outputs
Results
lost/stolen card, assessment integrity, legal compliance