diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md
index b42a64c..373bc9d 100644
--- a/.github/copilot-instructions.md
+++ b/.github/copilot-instructions.md
@@ -25,7 +25,8 @@ All three development phases are finished. The system is fully operational end-t
| Observability | Complete | `src/pharmagraphrag/observability.py` (Langfuse tracing) |
| Docker Compose | Complete | `docker-compose.yml` + `docker/` |
| CI/CD | Complete | `.github/workflows/ci.yml` + `deploy.yml` |
-| Tests | 221 passing | `tests/` |
+| Evaluation | Complete | `src/pharmagraphrag/evaluation/` (RAGAS metrics, agent eval, curated testset) |
+| Tests | 263 passing | `tests/` |
| Cloud Deployment | Live | Streamlit Cloud + Cloud Run + Neo4j Aura |
### Data at a Glance
@@ -102,7 +103,8 @@ FDA FAERS (CSV) + DailyMed (API)
- **UI**: Streamlit 1.54+ with streamlit-agraph, pyvis, plotly
- **Containers**: Docker Compose (Neo4j + API + UI + optional Ollama)
- **CI/CD**: GitHub Actions (ci.yml: lint+test on push; deploy.yml: CD on v* tags via Cloud Build)
-- **Testing**: pytest (221 tests passing)
+- **Evaluation**: RAGAS 0.4.3 (Faithfulness, Relevancy, Precision, Recall, Correctness) + custom agent tool accuracy
+- **Testing**: pytest (261 tests passing)
- **CI/CD**: GitHub Actions (ci.yml: lint + test matrix 3.11/3.13; deploy.yml: v* tags → Cloud Build → Cloud Run)
- **Cloud Build**: Google Cloud Build (cloudbuild.yaml) — downloads ChromaDB from GCS, builds Docker, deploys
- **Object Storage**: Google Cloud Storage (gs://pharmagraphrag-data for ChromaDB snapshots)
@@ -170,10 +172,19 @@ PharmaGraphRAG/
| | +-- __init__.py
| | +-- main.py # FastAPI app: POST /query, POST /agent/query, POST /agent/multi, GET /drug/{name}, GET /health
| | +-- models.py # Pydantic v2 request/response schemas (incl. AgentQueryRequest/Response)
+| +-- evaluation/
+| | +-- __init__.py
+| | +-- metrics.py # RAGAS metric wrappers (Faithfulness, Relevancy, Precision, Recall, Correctness)
+| | +-- dataset.py # Curated testset loader, EvalSample/EvalDataset
+| | +-- runner.py # Batch evaluation runner (calls API, computes RAGAS scores, exports CSV)
+| | +-- agent_eval.py # Agent tool selection accuracy (precision/recall/F1)
| +-- ui/
| +-- __init__.py
| +-- app.py # Streamlit chat: clickable follow-ups, confidence tooltips, pipeline steps (classic), nested sub-agent reasoning (multi)
| +-- components.py # Graph viz, sources panel, drug explorer
++-- data/
+| +-- evaluation/
+| +-- testset.json # 25 curated evaluation questions (8 types, ground truth, expected tools)
+-- tests/
| +-- __init__.py
| +-- test_download_faers.py # 2 tests
@@ -186,9 +197,11 @@ PharmaGraphRAG/
| +-- test_ui.py # 14 tests (Streamlit components + session state)
| +-- test_agent.py # 61 tests (9 tools, AgentResponse, StructuredResponse, multi-agent, endpoints)
| +-- test_observability.py # 13 tests (Langfuse init, callbacks, decorator, graceful degradation)
+| +-- test_evaluation.py # 40 tests (dataset, metrics, runner, agent eval, all mocked)
+-- scripts/
| +-- load_vectorstore.py # One-off: populate ChromaDB
| +-- validate_search.py # One-off: test semantic search queries
+| +-- run_evaluation.py # Batch eval: --mode classic|agent|multi|all, exports CSV reports
| +-- setup_demo.py # Demo setup: load graph + embeddings (~3 min)
| +-- migrate_neo4j.py # Migrate data between Neo4j instances
+-- docker/
@@ -287,7 +300,7 @@ PharmaGraphRAG/
- .gitignore: data/raw/, data/processed/, data/chroma/, .env, __pycache__, .pytest_cache
- **Deploy rule**: NEVER create version tags or trigger deployments without explicit user confirmation. Commits and pushes to main are fine; tags (v*) require user approval.
-### Testing (208 tests)
+### Testing (261 tests)
- pytest with fixtures for sample data and mocked services
- Mock Neo4j driver for graph tests
- Mock LLM API calls (never call real API in tests)
@@ -307,7 +320,17 @@ PharmaGraphRAG/
| test_ui.py | 14 | Streamlit components, session state |
| test_agent.py | 61 | 9 tools, AgentResponse, StructuredResponse, multi-agent supervisor, model selector, endpoints |
| test_observability.py | 13 | Langfuse init, callback handler, config builder, decorator, trace generation, flush |
-| **Total** | **221** | |
+| test_evaluation.py | 42 | RAGAS metrics, dataset loading, runner, agent tool eval, call_agent parsing, CSV export |
+| **Total** | **263** | |
+
+### Evaluation (RAGAS)
+- **Framework**: RAGAS 0.4.3 with Gemini via OpenAI-compatible endpoint
+- **Curated testset**: 25 questions across 8 types (drug_info, interaction, adverse_event, outcome, category, comparison, multi_drug, label_search)
+- **Reference-free metrics**: Faithfulness, Answer Relevancy
+- **Reference-based metrics**: Context Precision, Context Recall, Answer Correctness
+- **Agent evaluation**: Custom tool selection accuracy (precision/recall/F1), goal achievement tracking
+- **Batch runner**: Calls API endpoints (classic/agent/multi), computes metrics, exports CSV
+- **Script**: `scripts/run_evaluation.py --mode all --api-url http://localhost:8000`
## Key Design Decisions
diff --git a/README.md b/README.md
index 8ff73c5..7db9de1 100644
--- a/README.md
+++ b/README.md
@@ -3,7 +3,7 @@
[](https://github.com/jmponcebe/PharmaGraphRAG/actions/workflows/ci.yml)
[](https://github.com/jmponcebe/PharmaGraphRAG/actions/workflows/deploy.yml)
[](https://www.python.org/downloads/)
-[](#testing)
+[](#testing)
[](https://docs.astral.sh/ruff/)
[](LICENSE)
[](https://pharmagraphrag.streamlit.app)
@@ -66,7 +66,7 @@ A production-ready question-answering system that combines a **pharmaceutical kn
- **Agent Mode**: LangGraph ReAct agent that autonomously decides which tools to call (9 tools: drug info, adverse events, interactions, labels, drug search, event search, outcomes, comparison, categories) based on the question. Includes conversation memory, structured output (confidence + follow-ups), multi-agent supervisor with 3 specialized experts, per-query model selector (Flash for agents, Pro for supervisor), response caching and graceful fallback to classic pipeline
- **Transparent UI**: clickable follow-up suggestions, confidence level tooltips, pipeline steps expander (classic mode), nested sub-agent reasoning hierarchy (multi-agent mode)
- **Real FDA data**: 816K adverse event reports, 4,998 drugs, 365K causal relationships, 88 drug labels
-- **221 tests** with CI/CD on GitHub Actions (Python 3.11 + 3.13 matrix)
+- **263 tests** with CI/CD on GitHub Actions (Python 3.11 + 3.13 matrix)
- **Full stack**: data pipeline → knowledge graph → vector store → query engine → REST API → chat UI
- **One-click Codespaces**: try it instantly from your browser
@@ -75,7 +75,7 @@ A production-ready question-answering system that combines a **pharmaceutical kn
> *"What are the side effects of ibuprofen?"* · *"Does metformin interact with other drugs?"* · *"Compare the safety profiles of aspirin and clopidogrel"* · *"What drugs cause liver damage?"*
-Component Status — all modules complete, 221 tests passing
+Component Status — all modules complete, 263 tests passing
| Component | Status | Details |
| --- | --- | --- |
@@ -89,7 +89,8 @@ A production-ready question-answering system that combines a **pharmaceutical kn
| Docker Compose | ✅ Complete | Neo4j + API + UI + Ollama (optional profile) |
| CI/CD | ✅ Complete | GitHub Actions: lint, test matrix (3.11/3.13), Docker build |
| Agent Mode | ✅ Complete | LangGraph ReAct agent with 9 tools, conversation memory, structured output, multi-agent supervisor, nested reasoning |
-| Tests | ✅ 221 passing | Data (29) + vectors (35) + engine (37) + LLM (14) + API (18) + UI (14) + agent (61) + observability (13) |
+| Evaluation | ✅ Complete | RAGAS 0.4.3 (Faithfulness, Relevancy, Precision, Recall, Correctness) + agent tool accuracy (P/R/F1) |
+| Tests | ✅ 263 passing | Data (29) + vectors (35) + engine (37) + LLM (14) + API (18) + UI (14) + agent (61) + observability (13) + evaluation (42) |
@@ -231,7 +232,7 @@ In Agent Mode, the LLM autonomously decides which tools to call:
| UI | Streamlit + streamlit-agraph (graph visualization) |
| Containers | Docker Compose (multi-stage, non-root, healthchecks) |
| CI/CD | GitHub Actions (CI: lint + test matrix; CD: v* tags → Cloud Build → Cloud Run) |
-| Testing | pytest (221 tests, mocked services) |
+| Testing | pytest (263 tests, mocked services) |
| Linting | ruff (check + format) |
## Data Sources
@@ -398,11 +399,28 @@ gcloud run deploy pharmagraphrag-api --image gcr.io//pharmagraphrag-api
### Testing
```bash
-uv run pytest # Run all 221 tests
+uv run pytest # Run all 263 tests
uv run pytest -v # Verbose output
uv run pytest tests/test_engine.py # Specific module
```
+### Evaluation (RAGAS)
+
+Automated quality evaluation using [RAGAS](https://docs.ragas.io/) metrics against a curated testset of 25 questions (8 types: drug info, interactions, adverse events, outcomes, categories, comparisons, multi-drug, label search).
+
+```bash
+# Evaluate classic pipeline against local API
+python scripts/run_evaluation.py --mode classic --api-url http://localhost:8000
+
+# Evaluate all modes (classic + agent + multi)
+python scripts/run_evaluation.py --mode all --api-url http://localhost:8000
+
+# Against production
+python scripts/run_evaluation.py --mode all --api-url https://pharmagraphrag-api-893694384146.us-central1.run.app
+```
+
+**Metrics**: Faithfulness, Answer Relevancy (reference-free) + Context Precision, Context Recall, Answer Correctness (reference-based) + Agent tool selection accuracy (precision/recall/F1).
+
📋 Linting, formatting & type checking
@@ -445,6 +463,11 @@ src/pharmagraphrag/
├── api/ # REST API
│ ├── main.py # FastAPI app (POST /query, POST /agent/query, POST /agent/multi, GET /drug, GET /health)
│ └── models.py # Pydantic v2 request/response schemas
+├── evaluation/ # RAGAS evaluation framework
+│ ├── metrics.py # RAGAS metric wrappers (Faithfulness, Relevancy, Precision, Recall, Correctness)
+│ ├── dataset.py # Curated testset loader, EvalSample/EvalDataset
+│ ├── runner.py # Batch evaluation runner (calls API, computes RAGAS scores, exports CSV)
+│ └── agent_eval.py # Agent tool selection accuracy (precision/recall/F1)
└── ui/ # Chat interface
├── app.py # Streamlit app (chat, sidebar, settings)
└── components.py # Graph visualization, sources panel, drug explorer
diff --git a/data/evaluation/testset.json b/data/evaluation/testset.json
new file mode 100644
index 0000000..17a8426
--- /dev/null
+++ b/data/evaluation/testset.json
@@ -0,0 +1,186 @@
+{
+ "metadata": {
+ "version": "1.0.0",
+ "description": "Curated evaluation testset for PharmaGraphRAG. Questions span drug interactions, adverse events, outcomes, categories, and comparisons. Ground truth derived from FAERS data and DailyMed drug labels loaded in the knowledge graph.",
+ "created": "2025-05-17",
+ "author": "Jose Maria Ponce Bernabe",
+ "question_types": ["drug_info", "interaction", "adverse_event", "outcome", "category", "comparison", "multi_drug", "label_search"]
+ },
+ "samples": [
+ {
+ "id": "q01",
+ "question": "What are the most common adverse events reported for aspirin?",
+ "reference": "Common adverse events for aspirin include gastrointestinal haemorrhage, drug interaction, nausea, and platelet aggregation decreased, based on FAERS reports.",
+ "question_type": "drug_info",
+ "expected_tools": ["search_drug_info"]
+ },
+ {
+ "id": "q02",
+ "question": "Does warfarin interact with aspirin?",
+ "reference": "Yes, warfarin interacts with aspirin. Co-administration increases the risk of bleeding. This interaction is documented in DailyMed drug labels.",
+ "question_type": "interaction",
+ "expected_tools": ["search_drug_info", "list_drug_interactions"]
+ },
+ {
+ "id": "q03",
+ "question": "What drugs are known to cause hepatotoxicity?",
+ "reference": "Drugs reported to cause hepatotoxicity in FAERS include methotrexate, acetaminophen, and several others with varying report counts.",
+ "question_type": "adverse_event",
+ "expected_tools": ["find_drugs_for_adverse_event"]
+ },
+ {
+ "id": "q04",
+ "question": "What are the clinical outcomes associated with metformin use?",
+ "reference": "Clinical outcomes for metformin in FAERS include hospitalisation, other serious outcomes, and death, with hospitalisation being the most frequently reported.",
+ "question_type": "outcome",
+ "expected_tools": ["search_drug_info", "get_drug_outcomes"]
+ },
+ {
+ "id": "q05",
+ "question": "Which drug category does ibuprofen belong to?",
+ "reference": "Ibuprofen belongs to the Nonsteroidal Anti-inflammatory Drugs (NSAIDs) category.",
+ "question_type": "category",
+ "expected_tools": ["search_drug_info"]
+ },
+ {
+ "id": "q06",
+ "question": "Compare the adverse event profiles of aspirin and ibuprofen.",
+ "reference": "Both aspirin and ibuprofen are NSAIDs and share common adverse events like gastrointestinal haemorrhage and nausea. Aspirin has more reports of platelet aggregation decreased, while ibuprofen has more reports of renal impairment.",
+ "question_type": "comparison",
+ "expected_tools": ["compare_drugs"]
+ },
+ {
+ "id": "q07",
+ "question": "What does the drug label say about metformin interactions?",
+ "reference": "According to the DailyMed drug label, metformin may interact with cationic drugs that are eliminated by renal tubular secretion, carbonic anhydrase inhibitors, and alcohol. These interactions can affect metformin clearance or increase the risk of lactic acidosis.",
+ "question_type": "label_search",
+ "expected_tools": ["search_drug_labels"]
+ },
+ {
+ "id": "q08",
+ "question": "What are the side effects of lisinopril?",
+ "reference": "Adverse events reported for lisinopril in FAERS include cough, dizziness, hypotension, renal impairment, and hyperkalaemia.",
+ "question_type": "drug_info",
+ "expected_tools": ["search_drug_info"]
+ },
+ {
+ "id": "q09",
+ "question": "Which drugs interact with methotrexate?",
+ "reference": "Methotrexate interacts with NSAIDs, trimethoprim, and other drugs that can reduce its clearance, increasing toxicity risk. Interactions are documented in both FAERS co-occurrence data and DailyMed labels.",
+ "question_type": "interaction",
+ "expected_tools": ["list_drug_interactions", "search_drug_info"]
+ },
+ {
+ "id": "q10",
+ "question": "What drugs cause rhabdomyolysis?",
+ "reference": "Drugs reported to cause rhabdomyolysis in FAERS include statins (atorvastatin, simvastatin, rosuvastatin), and other medications. Statins are the most commonly reported drug class.",
+ "question_type": "adverse_event",
+ "expected_tools": ["find_drugs_for_adverse_event"]
+ },
+ {
+ "id": "q11",
+ "question": "Is atorvastatin associated with death outcomes in FAERS?",
+ "reference": "Yes, atorvastatin has death reported as one of its clinical outcomes in FAERS data, along with hospitalisation and other serious outcomes.",
+ "question_type": "outcome",
+ "expected_tools": ["search_drug_info", "get_drug_outcomes"]
+ },
+ {
+ "id": "q12",
+ "question": "What are the warnings for omeprazole according to its drug label?",
+ "reference": "Omeprazole warnings include risk of Clostridium difficile-associated diarrhea, bone fracture risk with long-term use, hypomagnesemia, and potential interactions with clopidogrel reducing its antiplatelet effect.",
+ "question_type": "label_search",
+ "expected_tools": ["search_drug_labels"]
+ },
+ {
+ "id": "q13",
+ "question": "Which drugs are in the same category as warfarin?",
+ "reference": "Warfarin belongs to the anticoagulant category. Other drugs in this category include heparin and related anticoagulants found in the knowledge graph.",
+ "question_type": "category",
+ "expected_tools": ["search_drug_info", "find_drugs_by_category"]
+ },
+ {
+ "id": "q14",
+ "question": "What are the most common adverse events across all drugs in the database?",
+ "reference": "The most commonly reported adverse events across FAERS include nausea, drug ineffective, headache, fatigue, diarrhoea, and dizziness.",
+ "question_type": "adverse_event",
+ "expected_tools": ["search_adverse_events"]
+ },
+ {
+ "id": "q15",
+ "question": "Compare warfarin and apixaban in terms of safety.",
+ "reference": "Both warfarin and apixaban are anticoagulants. Warfarin has more drug interactions documented and requires INR monitoring. Both can cause haemorrhage, but their adverse event profiles differ in frequency and severity according to FAERS data.",
+ "question_type": "comparison",
+ "expected_tools": ["compare_drugs"]
+ },
+ {
+ "id": "q16",
+ "question": "Can metformin and lisinopril be taken together?",
+ "reference": "Based on the knowledge graph data, metformin and lisinopril do not have a documented direct interaction. However, both can affect renal function, and monitoring is advisable.",
+ "question_type": "interaction",
+ "expected_tools": ["search_drug_info", "list_drug_interactions"]
+ },
+ {
+ "id": "q17",
+ "question": "What are the contraindications for aspirin?",
+ "reference": "According to the DailyMed label, aspirin is contraindicated in patients with known allergy to NSAIDs, patients with asthma, rhinitis, and nasal polyps syndrome, and in children and teenagers with viral infections due to Reye's syndrome risk.",
+ "question_type": "label_search",
+ "expected_tools": ["search_drug_labels"]
+ },
+ {
+ "id": "q18",
+ "question": "Search for drugs whose name contains 'statin'.",
+ "reference": "Drugs matching 'statin' include atorvastatin, simvastatin, rosuvastatin, pravastatin, and lovastatin, among others found in the knowledge graph.",
+ "question_type": "drug_info",
+ "expected_tools": ["search_drugs_by_name"]
+ },
+ {
+ "id": "q19",
+ "question": "What is the mechanism of action of omeprazole?",
+ "reference": "According to the drug label, omeprazole is a proton pump inhibitor (PPI) that suppresses gastric acid secretion by specific inhibition of the H+/K+-ATPase enzyme system at the secretory surface of the gastric parietal cell.",
+ "question_type": "label_search",
+ "expected_tools": ["search_drug_labels"]
+ },
+ {
+ "id": "q20",
+ "question": "Which adverse events are shared by both aspirin and warfarin?",
+ "reference": "Both aspirin and warfarin share adverse events related to bleeding, including gastrointestinal haemorrhage and haemorrhage. This is expected given their anticoagulant and antiplatelet mechanisms.",
+ "question_type": "comparison",
+ "expected_tools": ["compare_drugs", "search_drug_info"]
+ },
+ {
+ "id": "q21",
+ "question": "How many adverse event reports does ibuprofen have for nausea?",
+ "reference": "Ibuprofen has FAERS reports linking it to nausea as an adverse event, with a specific report count available in the knowledge graph's CAUSES relationship.",
+ "question_type": "drug_info",
+ "expected_tools": ["search_drug_info"]
+ },
+ {
+ "id": "q22",
+ "question": "What drugs should be avoided with warfarin?",
+ "reference": "Drugs that interact with warfarin include aspirin, NSAIDs (ibuprofen), certain antibiotics, and other anticoagulants. These interactions can increase bleeding risk.",
+ "question_type": "interaction",
+ "expected_tools": ["search_drug_info", "list_drug_interactions"]
+ },
+ {
+ "id": "q23",
+ "question": "Tell me about the adverse event profile of prednisone.",
+ "reference": "Prednisone adverse events in FAERS include weight increased, insomnia, immunosuppression, hyperglycaemia, and osteoporosis among others.",
+ "question_type": "drug_info",
+ "expected_tools": ["search_drug_info"]
+ },
+ {
+ "id": "q24",
+ "question": "What is the dosage information for metformin?",
+ "reference": "According to the DailyMed drug label, metformin is typically started at 500mg twice daily or 850mg once daily with meals, with gradual dose increases. Maximum recommended daily dose is 2550mg.",
+ "question_type": "label_search",
+ "expected_tools": ["search_drug_labels"]
+ },
+ {
+ "id": "q25",
+ "question": "Which drugs cause both nausea and headache as adverse events?",
+ "reference": "Multiple drugs in FAERS are associated with both nausea and headache, including common medications like ibuprofen, metformin, and lisinopril. These are among the most frequently reported adverse events across many drug classes.",
+ "question_type": "multi_drug",
+ "expected_tools": ["find_drugs_for_adverse_event"]
+ }
+ ]
+}
diff --git a/pyproject.toml b/pyproject.toml
index 4ba12e8..066ec98 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -50,6 +50,8 @@ dependencies = [
"langchain-google-genai>=4.2.1",
"langchain-core>=1.2.22",
"langfuse>=4.3.1",
+ "ragas>=0.4.3",
+ "datasets>=4.8.4",
]
[project.optional-dependencies]
diff --git a/scripts/run_evaluation.py b/scripts/run_evaluation.py
new file mode 100644
index 0000000..43a311f
--- /dev/null
+++ b/scripts/run_evaluation.py
@@ -0,0 +1,161 @@
+"""Batch evaluation script for PharmaGraphRAG.
+
+Runs the curated testset against one or more pipeline modes (classic, agent, multi)
+and generates RAGAS metric reports + agent tool evaluation results.
+
+Usage:
+ # Evaluate classic pipeline (requires local or Cloud Run API)
+ python scripts/run_evaluation.py --mode classic --api-url http://localhost:8000
+
+ # Evaluate agent mode
+ python scripts/run_evaluation.py --mode agent --api-url http://localhost:8000
+
+ # Evaluate all modes
+ python scripts/run_evaluation.py --mode all --api-url http://localhost:8000
+
+ # Evaluate against production
+ python scripts/run_evaluation.py --mode all --api-url https://pharmagraphrag-api-893694384146.us-central1.run.app
+
+ # Custom testset
+ python scripts/run_evaluation.py --testset data/evaluation/custom.json
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+import time
+from pathlib import Path
+
+# Ensure project root is on path
+project_root = Path(__file__).resolve().parent.parent
+sys.path.insert(0, str(project_root / "src"))
+
+from loguru import logger # noqa: E402
+
+from pharmagraphrag.evaluation.agent_eval import evaluate_agent_dataset # noqa: E402
+from pharmagraphrag.evaluation.dataset import load_testset # noqa: E402
+from pharmagraphrag.evaluation.runner import ( # noqa: E402
+ RunConfig,
+ compute_summary,
+ evaluate_dataset,
+ export_results,
+)
+
+
+def run_evaluation(
+ mode: str,
+ api_url: str,
+ testset_path: str | None,
+ output_dir: str,
+ model: str | None,
+) -> None:
+ """Run evaluation for a single pipeline mode."""
+ dataset = load_testset(testset_path)
+ logger.info("Loaded {} samples, evaluating in '{}' mode", len(dataset), mode)
+
+ config = RunConfig(api_url=api_url, mode=mode, model=model)
+
+ start = time.perf_counter()
+ results = evaluate_dataset(dataset, config)
+ elapsed = time.perf_counter() - start
+
+ # Export RAGAS metrics
+ output_path = Path(output_dir) / f"ragas_{mode}.csv"
+ export_results(results, output_path, config)
+
+ # Summary
+ summary = compute_summary(results)
+ logger.info("=== {} mode summary ({:.0f}s) ===", mode.upper(), elapsed)
+ for metric_name, avg_score in sorted(summary.items()):
+ logger.info(" {}: {:.3f}", metric_name, avg_score)
+
+ # Agent-specific evaluation (tool selection)
+ if mode in ("agent", "multi"):
+ agent_summary = evaluate_agent_dataset(dataset.samples)
+ logger.info("=== Agent tool selection ===")
+ logger.info(" Precision: {:.3f}", agent_summary.avg_tool_precision)
+ logger.info(" Recall: {:.3f}", agent_summary.avg_tool_recall)
+ logger.info(" F1: {:.3f}", agent_summary.avg_tool_f1)
+ logger.info(" Goal accuracy: {:.1%}", agent_summary.goal_accuracy)
+
+ # Export agent results
+ agent_path = Path(output_dir) / f"agent_tools_{mode}.csv"
+ _export_agent_results(agent_summary.results, agent_path)
+
+
+def _export_agent_results(results, output_path: Path) -> None:
+ """Export agent tool evaluation to CSV."""
+ import csv
+
+ output_path.parent.mkdir(parents=True, exist_ok=True)
+ with open(output_path, "w", newline="", encoding="utf-8") as f:
+ writer = csv.writer(f)
+ writer.writerow(
+ [
+ "sample_id",
+ "question",
+ "expected_tools",
+ "actual_tools",
+ "precision",
+ "recall",
+ "f1",
+ "goal_achieved",
+ ]
+ )
+ for r in results:
+ writer.writerow(
+ [
+ r.sample_id,
+ r.question[:100],
+ "|".join(r.expected_tools),
+ "|".join(r.actual_tools),
+ r.tool_precision,
+ r.tool_recall,
+ r.tool_f1,
+ r.goal_achieved,
+ ]
+ )
+ logger.info("Exported agent tool evaluation to {}", output_path)
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description="PharmaGraphRAG RAGAS Evaluation")
+ parser.add_argument(
+ "--mode",
+ choices=["classic", "agent", "multi", "all"],
+ default="classic",
+ help="Pipeline mode to evaluate (default: classic)",
+ )
+ parser.add_argument(
+ "--api-url",
+ default="http://localhost:8000",
+ help="Base URL of the PharmaGraphRAG API",
+ )
+ parser.add_argument(
+ "--testset",
+ default=None,
+ help="Path to testset JSON (default: data/evaluation/testset.json)",
+ )
+ parser.add_argument(
+ "--output-dir",
+ default="data/evaluation/results",
+ help="Output directory for CSV reports",
+ )
+ parser.add_argument(
+ "--model",
+ default=None,
+ help="LLM model to use for the pipeline (e.g. 'gemini-2.5-flash')",
+ )
+ args = parser.parse_args()
+
+ modes = ["classic", "agent", "multi"] if args.mode == "all" else [args.mode]
+
+ for mode in modes:
+ logger.info("Starting evaluation: mode={}, api={}", mode, args.api_url)
+ run_evaluation(mode, args.api_url, args.testset, args.output_dir, args.model)
+ logger.info("Completed evaluation for mode={}", mode)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/src/pharmagraphrag/agent/graph.py b/src/pharmagraphrag/agent/graph.py
index e03f560..256ae41 100644
--- a/src/pharmagraphrag/agent/graph.py
+++ b/src/pharmagraphrag/agent/graph.py
@@ -250,14 +250,16 @@ def run_agent(
agent = _get_agent(model)
+ effective_thread_id = thread_id or "default"
+
# Config with thread_id for checkpointer (conversation memory)
- config = {"configurable": {"thread_id": thread_id or "default"}}
+ config = {"configurable": {"thread_id": effective_thread_id}}
# Add Langfuse tracing if enabled
from pharmagraphrag.observability import build_callback_config
config = build_callback_config(
- session_id=thread_id,
+ session_id=effective_thread_id,
tags=["agent", "react"],
existing_config=config,
)
diff --git a/src/pharmagraphrag/agent/multi.py b/src/pharmagraphrag/agent/multi.py
index 89ab009..cfd3e8f 100644
--- a/src/pharmagraphrag/agent/multi.py
+++ b/src/pharmagraphrag/agent/multi.py
@@ -308,7 +308,7 @@ def run_multi_agent(
from pharmagraphrag.observability import build_callback_config
config = build_callback_config(
- session_id=thread_id,
+ session_id=thread_id or "default",
tags=["multi-agent", "supervisor"],
existing_config=config,
)
diff --git a/src/pharmagraphrag/engine/query_engine.py b/src/pharmagraphrag/engine/query_engine.py
index 486361d..1630779 100644
--- a/src/pharmagraphrag/engine/query_engine.py
+++ b/src/pharmagraphrag/engine/query_engine.py
@@ -165,6 +165,8 @@ def process_query(
logger.info("Processing query: '{}'", question[:100])
# Langfuse trace for classic pipeline
+ from contextlib import suppress
+
from pharmagraphrag.observability import is_enabled as _langfuse_on
_use_langfuse = _langfuse_on()
@@ -183,58 +185,60 @@ def process_query(
except Exception:
_langfuse_span = None
- # 1. Entity extraction
- entities = extract_entities(
- question,
- use_neo4j=use_neo4j_entities,
- fuzzy=fuzzy_match,
- )
- logger.info(
- "Entities — drugs: {}, adverse_events: {}",
- entities.drugs,
- entities.adverse_events,
- )
-
- # 2. Dual retrieval
- context = retrieve_context(
- drugs=entities.drugs,
- query=question,
- n_vector_results=n_vector_results,
- max_vector_chars=max_vector_chars,
- use_graph=use_graph,
- use_vector=use_vector,
- )
-
- # 3. Prompt assembly
- system_prompt = SYSTEM_PROMPT
- user_prompt = _build_user_prompt(question, context)
-
- result = QueryResult(
- question=question,
- entities=entities,
- context=context,
- system_prompt=system_prompt,
- user_prompt=user_prompt,
- )
-
- logger.info(
- "Query result — context: {} chars, drugs found: {}",
- len(result.user_prompt),
- context.drugs_found,
- )
-
- if _langfuse_span is not None:
- try:
- _langfuse_span.update(
- output={
- "drugs_extracted": entities.drugs,
- "drugs_found": context.drugs_found,
- "has_graph": context.has_graph,
- "has_vector": context.has_vector,
- },
- )
- _langfuse_span.__exit__(None, None, None)
- except Exception:
- pass
-
- return result
+ try:
+ # 1. Entity extraction
+ entities = extract_entities(
+ question,
+ use_neo4j=use_neo4j_entities,
+ fuzzy=fuzzy_match,
+ )
+ logger.info(
+ "Entities — drugs: {}, adverse_events: {}",
+ entities.drugs,
+ entities.adverse_events,
+ )
+
+ # 2. Dual retrieval
+ context = retrieve_context(
+ drugs=entities.drugs,
+ query=question,
+ n_vector_results=n_vector_results,
+ max_vector_chars=max_vector_chars,
+ use_graph=use_graph,
+ use_vector=use_vector,
+ )
+
+ # 3. Prompt assembly
+ system_prompt = SYSTEM_PROMPT
+ user_prompt = _build_user_prompt(question, context)
+
+ result = QueryResult(
+ question=question,
+ entities=entities,
+ context=context,
+ system_prompt=system_prompt,
+ user_prompt=user_prompt,
+ )
+
+ logger.info(
+ "Query result — context: {} chars, drugs found: {}",
+ len(result.user_prompt),
+ context.drugs_found,
+ )
+
+ if _langfuse_span is not None:
+ with suppress(Exception):
+ _langfuse_span.update(
+ output={
+ "drugs_extracted": entities.drugs,
+ "drugs_found": context.drugs_found,
+ "has_graph": context.has_graph,
+ "has_vector": context.has_vector,
+ },
+ )
+
+ return result
+ finally:
+ if _langfuse_span is not None:
+ with suppress(Exception):
+ _langfuse_span.__exit__(None, None, None)
diff --git a/src/pharmagraphrag/evaluation/__init__.py b/src/pharmagraphrag/evaluation/__init__.py
new file mode 100644
index 0000000..e8b7d84
--- /dev/null
+++ b/src/pharmagraphrag/evaluation/__init__.py
@@ -0,0 +1,5 @@
+"""Evaluation module for PharmaGraphRAG using RAGAS.
+
+Provides RAG quality metrics (faithfulness, relevancy, context precision/recall),
+agent evaluation metrics, and batch evaluation runners.
+"""
diff --git a/src/pharmagraphrag/evaluation/agent_eval.py b/src/pharmagraphrag/evaluation/agent_eval.py
new file mode 100644
index 0000000..3a04773
--- /dev/null
+++ b/src/pharmagraphrag/evaluation/agent_eval.py
@@ -0,0 +1,132 @@
+"""Agent-specific evaluation metrics.
+
+Evaluates agent tool selection accuracy by comparing expected tools
+from the curated testset against the tools actually called by the agent.
+Also provides agent goal accuracy (did the agent answer the question?).
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+
+from loguru import logger
+
+from pharmagraphrag.evaluation.dataset import EvalSample
+
+
+@dataclass
+class AgentEvalResult:
+ """Evaluation result for agent tool selection and goal accuracy."""
+
+ sample_id: str
+ question: str
+ expected_tools: list[str]
+ actual_tools: list[str]
+ tool_precision: float = 0.0
+ tool_recall: float = 0.0
+ tool_f1: float = 0.0
+ goal_achieved: bool = False
+
+ def to_dict(self) -> dict:
+ return {
+ "sample_id": self.sample_id,
+ "question": self.question,
+ "expected_tools": self.expected_tools,
+ "actual_tools": self.actual_tools,
+ "tool_precision": self.tool_precision,
+ "tool_recall": self.tool_recall,
+ "tool_f1": self.tool_f1,
+ "goal_achieved": self.goal_achieved,
+ }
+
+
+def evaluate_tool_selection(sample: EvalSample) -> AgentEvalResult:
+ """Evaluate whether the agent called the right tools.
+
+ Computes precision, recall, and F1 for tool selection by comparing
+ expected_tools (from testset) vs actual tool_calls (from agent response).
+
+ Args:
+ sample: EvalSample with expected_tools and tool_calls populated.
+
+ Returns:
+ AgentEvalResult with tool selection metrics.
+ """
+ expected = set(sample.expected_tools)
+ actual = set(sample.tool_calls)
+
+ if not expected and not actual:
+ return AgentEvalResult(
+ sample_id=sample.id,
+ question=sample.question,
+ expected_tools=sample.expected_tools,
+ actual_tools=sample.tool_calls,
+ tool_precision=1.0,
+ tool_recall=1.0,
+ tool_f1=1.0,
+ goal_achieved=bool(sample.answer and "ERROR" not in sample.answer),
+ )
+
+ true_positives = len(expected & actual)
+ precision = true_positives / len(actual) if actual else 0.0
+ recall = true_positives / len(expected) if expected else 0.0
+ f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
+
+ return AgentEvalResult(
+ sample_id=sample.id,
+ question=sample.question,
+ expected_tools=sample.expected_tools,
+ actual_tools=sample.tool_calls,
+ tool_precision=round(precision, 3),
+ tool_recall=round(recall, 3),
+ tool_f1=round(f1, 3),
+ goal_achieved=bool(sample.answer and "ERROR" not in sample.answer),
+ )
+
+
+@dataclass
+class AgentEvalSummary:
+ """Aggregated agent evaluation metrics."""
+
+ total_samples: int = 0
+ avg_tool_precision: float = 0.0
+ avg_tool_recall: float = 0.0
+ avg_tool_f1: float = 0.0
+ goal_accuracy: float = 0.0
+ results: list[AgentEvalResult] = field(default_factory=list)
+
+
+def evaluate_agent_dataset(samples: list[EvalSample]) -> AgentEvalSummary:
+ """Evaluate agent tool selection across all samples.
+
+ Args:
+ samples: List of EvalSample with tool_calls populated (after running agent pipeline).
+
+ Returns:
+ AgentEvalSummary with averaged metrics.
+ """
+ results = []
+ for sample in samples:
+ result = evaluate_tool_selection(sample)
+ results.append(result)
+ logger.debug(
+ "[{}] P={:.2f} R={:.2f} F1={:.2f} goal={}",
+ sample.id,
+ result.tool_precision,
+ result.tool_recall,
+ result.tool_f1,
+ result.goal_achieved,
+ )
+
+ n = len(results)
+ if n == 0:
+ return AgentEvalSummary()
+
+ return AgentEvalSummary(
+ total_samples=n,
+ avg_tool_precision=round(sum(r.tool_precision for r in results) / n, 3),
+ avg_tool_recall=round(sum(r.tool_recall for r in results) / n, 3),
+ avg_tool_f1=round(sum(r.tool_f1 for r in results) / n, 3),
+ goal_accuracy=round(sum(1 for r in results if r.goal_achieved) / n, 3),
+ results=results,
+ )
diff --git a/src/pharmagraphrag/evaluation/dataset.py b/src/pharmagraphrag/evaluation/dataset.py
new file mode 100644
index 0000000..78be82e
--- /dev/null
+++ b/src/pharmagraphrag/evaluation/dataset.py
@@ -0,0 +1,110 @@
+"""Evaluation dataset loader.
+
+Loads curated question-answer pairs from ``data/evaluation/testset.json``
+and converts them to RAGAS ``EvaluationDataset`` format.
+"""
+
+from __future__ import annotations
+
+import json
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any
+
+from loguru import logger
+
+DEFAULT_TESTSET_PATH = Path(__file__).resolve().parents[3] / "data" / "evaluation" / "testset.json"
+
+
+@dataclass
+class EvalSample:
+ """A single evaluation sample."""
+
+ id: str
+ question: str
+ reference: str = ""
+ question_type: str = ""
+ expected_tools: list[str] = field(default_factory=list)
+
+ # Populated after running the pipeline
+ answer: str = ""
+ contexts: list[str] = field(default_factory=list)
+ tool_calls: list[str] = field(default_factory=list)
+
+
+@dataclass
+class EvalDataset:
+ """Collection of evaluation samples."""
+
+ samples: list[EvalSample]
+ metadata: dict[str, Any] = field(default_factory=dict)
+
+ def __len__(self) -> int:
+ return len(self.samples)
+
+ def __iter__(self):
+ return iter(self.samples)
+
+ def by_type(self, question_type: str) -> list[EvalSample]:
+ return [s for s in self.samples if s.question_type == question_type]
+
+ @property
+ def question_types(self) -> list[str]:
+ return sorted({s.question_type for s in self.samples})
+
+
+def load_testset(path: Path | str | None = None) -> EvalDataset:
+ """Load evaluation testset from JSON file.
+
+ Args:
+ path: Path to the testset JSON file. Uses default if not provided.
+
+ Returns:
+ EvalDataset with loaded samples.
+ """
+ path = Path(path) if path else DEFAULT_TESTSET_PATH
+
+ if not path.exists():
+ raise FileNotFoundError(f"Testset not found: {path}")
+
+ with open(path) as f:
+ data = json.load(f)
+
+ samples = []
+ for item in data.get("samples", []):
+ samples.append(
+ EvalSample(
+ id=item["id"],
+ question=item["question"],
+ reference=item.get("reference", ""),
+ question_type=item.get("question_type", ""),
+ expected_tools=item.get("expected_tools", []),
+ )
+ )
+
+ logger.info("Loaded {} evaluation samples from {}", len(samples), path.name)
+ return EvalDataset(samples=samples, metadata=data.get("metadata", {}))
+
+
+def to_ragas_dataset(dataset: EvalDataset) -> list[dict[str, Any]]:
+ """Convert EvalDataset to RAGAS-compatible format.
+
+ Each sample becomes a dict with keys expected by RAGAS:
+ user_input, response, retrieved_contexts, reference.
+
+ Only includes samples where answer and contexts have been populated
+ (i.e., after running the pipeline).
+ """
+ ragas_samples = []
+ for sample in dataset.samples:
+ if not sample.answer or not sample.contexts:
+ continue
+ entry = {
+ "user_input": sample.question,
+ "response": sample.answer,
+ "retrieved_contexts": sample.contexts,
+ }
+ if sample.reference:
+ entry["reference"] = sample.reference
+ ragas_samples.append(entry)
+ return ragas_samples
diff --git a/src/pharmagraphrag/evaluation/metrics.py b/src/pharmagraphrag/evaluation/metrics.py
new file mode 100644
index 0000000..a2da29c
--- /dev/null
+++ b/src/pharmagraphrag/evaluation/metrics.py
@@ -0,0 +1,168 @@
+"""RAGAS metric wrappers for PharmaGraphRAG.
+
+Configures RAGAS metrics with Gemini as the evaluator LLM.
+Provides both reference-free metrics (faithfulness, answer relevancy)
+and reference-based metrics (context precision, context recall,
+answer correctness) for comprehensive RAG evaluation.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any
+
+from loguru import logger
+
+
+@dataclass
+class MetricResult:
+ """Result of a single metric evaluation."""
+
+ name: str
+ score: float
+ details: dict[str, Any] | None = None
+
+
+@dataclass
+class EvalResult:
+ """Aggregated evaluation result for a single sample."""
+
+ question: str
+ answer: str
+ metrics: list[MetricResult]
+ contexts: list[str] | None = None
+ reference: str | None = None
+
+ @property
+ def scores(self) -> dict[str, float]:
+ return {m.name: m.score for m in self.metrics}
+
+
+def _get_evaluator_llm(model: str = "gemini-2.0-flash"):
+ """Create a RAGAS-compatible LLM wrapper using Gemini via OpenAI compatibility.
+
+ Uses the Gemini OpenAI-compatible endpoint to avoid instructor/google-genai
+ SDK conflicts with safety settings.
+ """
+ import os
+
+ from openai import OpenAI
+ from ragas.llms import llm_factory
+
+ api_key = os.environ.get("GEMINI_API_KEY", "")
+ if not api_key:
+ raise ValueError("GEMINI_API_KEY env var is required for RAGAS evaluation")
+
+ client = OpenAI(
+ api_key=api_key,
+ base_url="https://generativelanguage.googleapis.com/v1beta/openai/",
+ )
+ return llm_factory(model, provider="openai", client=client)
+
+
+def _get_evaluator_embeddings(model: str = "text-embedding-004"):
+ """Create RAGAS-compatible embeddings using Gemini via OpenAI compatibility."""
+ import os
+
+ from openai import OpenAI
+ from ragas.embeddings import OpenAIEmbeddings
+
+ api_key = os.environ.get("GEMINI_API_KEY", "")
+ if not api_key:
+ raise ValueError("GEMINI_API_KEY env var is required for RAGAS evaluation")
+
+ client = OpenAI(
+ api_key=api_key,
+ base_url="https://generativelanguage.googleapis.com/v1beta/openai/",
+ )
+ return OpenAIEmbeddings(client=client, model=model)
+
+
+def get_reference_free_metrics(
+ llm=None,
+ embeddings=None,
+) -> list:
+ """Get metrics that don't require ground truth.
+
+ Returns faithfulness and answer relevancy metrics.
+ """
+ from ragas.metrics import AnswerRelevancy, Faithfulness
+
+ llm = llm or _get_evaluator_llm()
+ embeddings = embeddings or _get_evaluator_embeddings()
+
+ return [
+ Faithfulness(llm=llm),
+ AnswerRelevancy(llm=llm, embeddings=embeddings),
+ ]
+
+
+def get_reference_metrics(
+ llm=None,
+ embeddings=None,
+) -> list:
+ """Get metrics that require ground truth reference.
+
+ Returns context precision, context recall, and answer correctness.
+ """
+ from ragas.metrics import AnswerCorrectness, ContextPrecision, ContextRecall
+
+ llm = llm or _get_evaluator_llm()
+ embeddings = embeddings or _get_evaluator_embeddings()
+
+ return [
+ ContextPrecision(llm=llm),
+ ContextRecall(llm=llm),
+ AnswerCorrectness(llm=llm, embeddings=embeddings),
+ ]
+
+
+def get_all_metrics(llm=None, embeddings=None) -> list:
+ """Get all available metrics (reference-free + reference-based)."""
+ llm = llm or _get_evaluator_llm()
+ embeddings = embeddings or _get_evaluator_embeddings()
+
+ return get_reference_free_metrics(llm, embeddings) + get_reference_metrics(llm, embeddings)
+
+
+def score_sample(
+ question: str,
+ answer: str,
+ contexts: list[str],
+ reference: str | None = None,
+ llm=None,
+ embeddings=None,
+) -> EvalResult:
+ """Evaluate a single RAG response using RAGAS metrics.
+
+ Uses reference-free metrics always; adds reference-based metrics
+ if a ground truth reference is provided.
+ """
+ llm = llm or _get_evaluator_llm()
+ embeddings = embeddings or _get_evaluator_embeddings()
+
+ metrics = get_reference_free_metrics(llm, embeddings)
+ if reference:
+ metrics.extend(get_reference_metrics(llm, embeddings))
+
+ results = []
+ for metric in metrics:
+ try:
+ score = metric.single_score(
+ user_input=question,
+ response=answer,
+ retrieved_contexts=contexts,
+ reference=reference,
+ )
+ results.append(MetricResult(name=type(metric).__name__, score=score))
+ except Exception as exc:
+ logger.warning("Metric {} failed: {}", type(metric).__name__, exc)
+ results.append(MetricResult(name=type(metric).__name__, score=-1.0))
+
+ return EvalResult(
+ question=question,
+ answer=answer,
+ metrics=results,
+ contexts=contexts,
+ reference=reference,
+ )
diff --git a/src/pharmagraphrag/evaluation/runner.py b/src/pharmagraphrag/evaluation/runner.py
new file mode 100644
index 0000000..74b2c5e
--- /dev/null
+++ b/src/pharmagraphrag/evaluation/runner.py
@@ -0,0 +1,262 @@
+"""Evaluation runner for PharmaGraphRAG.
+
+Executes the curated testset against the classic pipeline, agent mode,
+and multi-agent mode, then computes RAGAS metrics for each response.
+Results are exported as CSV for analysis.
+"""
+
+from __future__ import annotations
+
+import csv
+import time
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any
+
+import httpx
+from loguru import logger
+
+from pharmagraphrag.evaluation.dataset import EvalDataset, EvalSample
+from pharmagraphrag.evaluation.metrics import EvalResult, MetricResult, score_sample
+
+
+@dataclass
+class RunConfig:
+ """Configuration for an evaluation run."""
+
+ api_url: str = "http://localhost:8000"
+ mode: str = "classic" # "classic", "agent", "multi"
+ model: str | None = None
+ timeout: float = 120.0
+ include_reference: bool = True
+
+
+@dataclass
+class PipelineResponse:
+ """Raw response from a PharmaGraphRAG pipeline call."""
+
+ answer: str = ""
+ contexts: list[str] = field(default_factory=list)
+ tool_calls: list[str] = field(default_factory=list)
+ latency_ms: float = 0.0
+ error: str | None = None
+ raw: dict[str, Any] = field(default_factory=dict)
+
+
+def _call_classic(question: str, config: RunConfig) -> PipelineResponse:
+ """Call the classic pipeline via POST /query."""
+ url = f"{config.api_url}/query"
+ body: dict[str, Any] = {"question": question}
+ if config.model:
+ body["model"] = config.model
+
+ start = time.perf_counter()
+ try:
+ resp = httpx.post(url, json=body, timeout=config.timeout)
+ latency = (time.perf_counter() - start) * 1000
+ resp.raise_for_status()
+ data = resp.json()
+
+ contexts = []
+ for src in data.get("sources", []):
+ snippet = src.get("snippet", "")
+ if snippet:
+ contexts.append(snippet)
+
+ return PipelineResponse(
+ answer=data.get("answer", ""),
+ contexts=contexts,
+ latency_ms=latency,
+ raw=data,
+ )
+ except Exception as exc:
+ latency = (time.perf_counter() - start) * 1000
+ return PipelineResponse(error=str(exc), latency_ms=latency)
+
+
+def _call_agent(
+ question: str, config: RunConfig, endpoint: str = "/agent/query"
+) -> PipelineResponse:
+ """Call the agent or multi-agent pipeline."""
+ url = f"{config.api_url}{endpoint}"
+ body: dict[str, Any] = {"question": question}
+ if config.model:
+ body["model"] = config.model
+
+ start = time.perf_counter()
+ try:
+ resp = httpx.post(url, json=body, timeout=config.timeout)
+ latency = (time.perf_counter() - start) * 1000
+ resp.raise_for_status()
+ data = resp.json()
+
+ # Extract contexts from tool results and graph/vector data
+ contexts = []
+ for tr in data.get("tool_results", []):
+ content = tr.get("content", "")
+ if content:
+ contexts.append(content[:2000])
+
+ # Also use vector data snippets
+ for vd in data.get("vector_data", []):
+ snippet = vd.get("text", vd.get("snippet", ""))
+ if snippet:
+ contexts.append(snippet)
+
+ tools = [tc.get("tool", "") for tc in data.get("tool_calls", [])]
+
+ return PipelineResponse(
+ answer=data.get("answer", ""),
+ contexts=contexts,
+ tool_calls=tools,
+ latency_ms=latency,
+ raw=data,
+ )
+ except Exception as exc:
+ latency = (time.perf_counter() - start) * 1000
+ return PipelineResponse(error=str(exc), latency_ms=latency)
+
+
+def run_pipeline(question: str, config: RunConfig) -> PipelineResponse:
+ """Route a question to the configured pipeline mode."""
+ if config.mode == "classic":
+ return _call_classic(question, config)
+ elif config.mode == "agent":
+ return _call_agent(question, config, endpoint="/agent/query")
+ elif config.mode == "multi":
+ return _call_agent(question, config, endpoint="/agent/multi")
+ else:
+ return PipelineResponse(error=f"Unknown mode: {config.mode}")
+
+
+def evaluate_sample(
+ sample: EvalSample,
+ config: RunConfig,
+ llm=None,
+ embeddings=None,
+) -> EvalResult:
+ """Run a single sample through the pipeline and evaluate with RAGAS."""
+ logger.info("[{}] Evaluating: {}", sample.id, sample.question[:60])
+
+ # Call pipeline
+ response = run_pipeline(sample.question, config)
+
+ if response.error:
+ logger.warning("[{}] Pipeline error: {}", sample.id, response.error)
+ return EvalResult(
+ question=sample.question,
+ answer=f"ERROR: {response.error}",
+ metrics=[MetricResult(name="error", score=-1.0)],
+ contexts=[],
+ reference=sample.reference if config.include_reference else None,
+ )
+
+ # Populate sample with response data
+ sample.answer = response.answer
+ sample.contexts = response.contexts
+ sample.tool_calls = response.tool_calls
+
+ # Compute RAGAS metrics
+ reference = sample.reference if config.include_reference else None
+ result = score_sample(
+ question=sample.question,
+ answer=response.answer,
+ contexts=response.contexts,
+ reference=reference,
+ llm=llm,
+ embeddings=embeddings,
+ )
+
+ # Add latency as a pseudo-metric
+ result.metrics.append(MetricResult(name="latency_ms", score=response.latency_ms))
+
+ return result
+
+
+def evaluate_dataset(
+ dataset: EvalDataset,
+ config: RunConfig,
+ llm=None,
+ embeddings=None,
+) -> list[EvalResult]:
+ """Evaluate all samples in the dataset.
+
+ Args:
+ dataset: Loaded evaluation dataset.
+ config: Pipeline configuration (mode, API URL, model).
+ llm: RAGAS evaluator LLM (created if not provided).
+ embeddings: RAGAS evaluator embeddings (created if not provided).
+
+ Returns:
+ List of EvalResult, one per sample.
+ """
+ from pharmagraphrag.evaluation.metrics import _get_evaluator_embeddings, _get_evaluator_llm
+
+ llm = llm or _get_evaluator_llm()
+ embeddings = embeddings or _get_evaluator_embeddings()
+
+ results = []
+ total = len(dataset)
+
+ for i, sample in enumerate(dataset, 1):
+ logger.info("[{}/{}] {}", i, total, sample.id)
+ result = evaluate_sample(sample, config, llm=llm, embeddings=embeddings)
+ results.append(result)
+
+ return results
+
+
+def export_results(
+ results: list[EvalResult],
+ output_path: Path | str,
+ config: RunConfig | None = None,
+) -> Path:
+ """Export evaluation results to CSV.
+
+ Args:
+ results: List of EvalResult from evaluate_dataset.
+ output_path: Path for the output CSV file.
+ config: Optional config for metadata in the output.
+
+ Returns:
+ Path to the written CSV file.
+ """
+ output_path = Path(output_path)
+ output_path.parent.mkdir(parents=True, exist_ok=True)
+
+ # Collect all unique metric names
+ all_metric_names = sorted({m.name for r in results for m in r.metrics})
+
+ with open(output_path, "w", newline="", encoding="utf-8") as f:
+ writer = csv.writer(f)
+
+ # Header
+ header = ["question", "answer_preview", "reference_preview", *all_metric_names]
+ writer.writerow(header)
+
+ # Rows
+ for result in results:
+ scores = result.scores
+ row = [
+ result.question,
+ (result.answer or "")[:200],
+ (result.reference or "")[:200],
+ *[scores.get(m, "") for m in all_metric_names],
+ ]
+ writer.writerow(row)
+
+ logger.info("Exported {} results to {}", len(results), output_path)
+ return output_path
+
+
+def compute_summary(results: list[EvalResult]) -> dict[str, float]:
+ """Compute average scores across all results for each metric."""
+ from collections import defaultdict
+
+ totals: dict[str, list[float]] = defaultdict(list)
+ for result in results:
+ for metric in result.metrics:
+ if metric.score >= 0: # Exclude errors (-1)
+ totals[metric.name].append(metric.score)
+
+ return {name: sum(scores) / len(scores) for name, scores in totals.items() if scores}
diff --git a/src/pharmagraphrag/observability.py b/src/pharmagraphrag/observability.py
index 32f7039..ca01427 100644
--- a/src/pharmagraphrag/observability.py
+++ b/src/pharmagraphrag/observability.py
@@ -28,6 +28,7 @@ def my_pipeline_step(...):
from pharmagraphrag.config import get_settings
_langfuse_initialized = False
+_langfuse_disabled = False
def _ensure_initialized() -> bool:
@@ -35,16 +36,20 @@ def _ensure_initialized() -> bool:
Returns True if Langfuse is available and configured.
"""
- global _langfuse_initialized
+ global _langfuse_initialized, _langfuse_disabled
if _langfuse_initialized:
return True
+ if _langfuse_disabled:
+ return False
settings = get_settings()
if not settings.langfuse_enabled:
+ _langfuse_disabled = True
return False
if not settings.langfuse_public_key or not settings.langfuse_secret_key:
logger.warning("Langfuse enabled but API keys not set — tracing disabled")
+ _langfuse_disabled = True
return False
try:
@@ -126,8 +131,8 @@ def build_callback_config(
callbacks = [*list(callbacks), handler]
config["callbacks"] = callbacks
- # Add langfuse metadata for trace attributes
- meta = config.get("metadata", {})
+ # Add langfuse metadata for trace attributes (copy to avoid mutation)
+ meta = dict(config.get("metadata", {}))
if session_id:
meta["langfuse_session_id"] = session_id
if user_id:
@@ -142,7 +147,6 @@ def build_callback_config(
def observe_fn(
*,
name: str | None = None,
- as_type: str = "span",
) -> Callable:
"""Decorator that wraps a function with Langfuse @observe tracing.
@@ -151,16 +155,23 @@ def observe_fn(
"""
def decorator(func: Callable) -> Callable:
+ traced_func: Callable | None = None
+
@wraps(func)
def wrapper(*args: Any, **kwargs: Any) -> Any:
+ nonlocal traced_func
+
+ if traced_func is not None:
+ return traced_func(*args, **kwargs)
+
if not _ensure_initialized():
return func(*args, **kwargs)
try:
from langfuse import observe
- traced = observe(name=name or func.__name__)(func)
- return traced(*args, **kwargs)
+ traced_func = observe(name=name or func.__name__)(func)
+ return traced_func(*args, **kwargs)
except Exception:
return func(*args, **kwargs)
diff --git a/test_api.py b/test_api.py
deleted file mode 100644
index f3de66d..0000000
--- a/test_api.py
+++ /dev/null
@@ -1,12 +0,0 @@
-from fastapi.testclient import TestClient
-from pharmagraphrag.api.main import app
-client = TestClient(app)
-resp = client.post('/query', json={'question': 'What are the side effects of aspirin?', 'use_llm': True})
-print(f'Status: {resp.status_code}')
-data = resp.json()
-print(f'Answer preview: {data.get("answer", "")[:200]}')
-print(f'Drugs found: {data.get("drugs_found_in_graph", [])}')
-print(f'Error: {data.get("error")}')
-from pharmagraphrag.observability import flush
-flush()
-print('Langfuse flushed')
diff --git a/tests/test_evaluation.py b/tests/test_evaluation.py
new file mode 100644
index 0000000..ebb8801
--- /dev/null
+++ b/tests/test_evaluation.py
@@ -0,0 +1,590 @@
+"""Tests for the evaluation module.
+
+Covers dataset loading, RAGAS metric wrappers, agent evaluation,
+and the evaluation runner. All external dependencies (LLM, API) are mocked.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from unittest.mock import MagicMock, patch
+
+import pytest
+
+from pharmagraphrag.evaluation.agent_eval import (
+ evaluate_agent_dataset,
+ evaluate_tool_selection,
+)
+from pharmagraphrag.evaluation.dataset import (
+ EvalDataset,
+ EvalSample,
+ load_testset,
+ to_ragas_dataset,
+)
+from pharmagraphrag.evaluation.metrics import EvalResult, MetricResult, score_sample
+from pharmagraphrag.evaluation.runner import (
+ PipelineResponse,
+ RunConfig,
+ _call_agent,
+ _call_classic,
+ compute_summary,
+ evaluate_sample,
+ export_results,
+ run_pipeline,
+)
+
+# ---------------------------------------------------------------------------
+# Fixtures
+# ---------------------------------------------------------------------------
+
+SAMPLE_TESTSET = {
+ "metadata": {"version": "1.0.0", "description": "test"},
+ "samples": [
+ {
+ "id": "q01",
+ "question": "What are the side effects of aspirin?",
+ "reference": "Aspirin causes GI bleeding and nausea.",
+ "question_type": "drug_info",
+ "expected_tools": ["search_drug_info"],
+ },
+ {
+ "id": "q02",
+ "question": "Does warfarin interact with aspirin?",
+ "reference": "Yes, warfarin interacts with aspirin.",
+ "question_type": "interaction",
+ "expected_tools": ["search_drug_info", "list_drug_interactions"],
+ },
+ ],
+}
+
+
+@pytest.fixture
+def testset_path(tmp_path: Path) -> Path:
+ path = tmp_path / "testset.json"
+ path.write_text(json.dumps(SAMPLE_TESTSET))
+ return path
+
+
+@pytest.fixture
+def sample_dataset() -> EvalDataset:
+ return EvalDataset(
+ samples=[
+ EvalSample(
+ id="q01",
+ question="What are the side effects of aspirin?",
+ reference="Aspirin causes GI bleeding.",
+ question_type="drug_info",
+ expected_tools=["search_drug_info"],
+ ),
+ EvalSample(
+ id="q02",
+ question="Does warfarin interact with aspirin?",
+ reference="Yes, they interact.",
+ question_type="interaction",
+ expected_tools=["search_drug_info", "list_drug_interactions"],
+ ),
+ ],
+ )
+
+
+@pytest.fixture
+def eval_sample_with_response() -> EvalSample:
+ sample = EvalSample(
+ id="q01",
+ question="What are the side effects of aspirin?",
+ reference="Aspirin causes GI bleeding.",
+ question_type="drug_info",
+ expected_tools=["search_drug_info"],
+ )
+ sample.answer = "Aspirin can cause gastrointestinal bleeding."
+ sample.contexts = ["ASPIRIN CAUSES GASTROINTESTINAL HAEMORRHAGE (report_count: 1523)"]
+ sample.tool_calls = ["search_drug_info"]
+ return sample
+
+
+# ===========================================================================
+# Dataset tests
+# ===========================================================================
+
+
+class TestLoadTestset:
+ def test_load_from_file(self, testset_path: Path):
+ dataset = load_testset(testset_path)
+ assert len(dataset) == 2
+ assert dataset.samples[0].id == "q01"
+ assert dataset.samples[0].question_type == "drug_info"
+
+ def test_load_preserves_expected_tools(self, testset_path: Path):
+ dataset = load_testset(testset_path)
+ assert dataset.samples[0].expected_tools == ["search_drug_info"]
+ assert "list_drug_interactions" in dataset.samples[1].expected_tools
+
+ def test_load_metadata(self, testset_path: Path):
+ dataset = load_testset(testset_path)
+ assert dataset.metadata["version"] == "1.0.0"
+
+ def test_load_missing_file(self, tmp_path: Path):
+ with pytest.raises(FileNotFoundError):
+ load_testset(tmp_path / "nonexistent.json")
+
+
+class TestEvalDataset:
+ def test_length(self, sample_dataset: EvalDataset):
+ assert len(sample_dataset) == 2
+
+ def test_iter(self, sample_dataset: EvalDataset):
+ items = list(sample_dataset)
+ assert len(items) == 2
+
+ def test_by_type(self, sample_dataset: EvalDataset):
+ drug_info = sample_dataset.by_type("drug_info")
+ assert len(drug_info) == 1
+ assert drug_info[0].id == "q01"
+
+ def test_question_types(self, sample_dataset: EvalDataset):
+ types = sample_dataset.question_types
+ assert "drug_info" in types
+ assert "interaction" in types
+
+ def test_by_type_empty(self, sample_dataset: EvalDataset):
+ assert sample_dataset.by_type("unknown") == []
+
+
+class TestToRagasDataset:
+ def test_converts_populated_samples(self, sample_dataset: EvalDataset):
+ sample_dataset.samples[0].answer = "GI bleeding"
+ sample_dataset.samples[0].contexts = ["some context"]
+ ragas = to_ragas_dataset(sample_dataset)
+ assert len(ragas) == 1 # only q01 has answer
+ assert ragas[0]["user_input"] == sample_dataset.samples[0].question
+ assert ragas[0]["response"] == "GI bleeding"
+
+ def test_skips_unanswered_samples(self, sample_dataset: EvalDataset):
+ ragas = to_ragas_dataset(sample_dataset)
+ assert len(ragas) == 0
+
+ def test_includes_reference(self, sample_dataset: EvalDataset):
+ sample_dataset.samples[0].answer = "test"
+ sample_dataset.samples[0].contexts = ["ctx"]
+ ragas = to_ragas_dataset(sample_dataset)
+ assert "reference" in ragas[0]
+ assert ragas[0]["reference"] == "Aspirin causes GI bleeding."
+
+
+# ===========================================================================
+# Metrics tests
+# ===========================================================================
+
+
+class TestMetricResult:
+ def test_creation(self):
+ mr = MetricResult(name="faithfulness", score=0.85)
+ assert mr.name == "faithfulness"
+ assert mr.score == 0.85
+
+ def test_with_details(self):
+ mr = MetricResult(name="test", score=0.5, details={"key": "val"})
+ assert mr.details["key"] == "val"
+
+
+class TestEvalResult:
+ def test_scores_property(self):
+ result = EvalResult(
+ question="test?",
+ answer="answer",
+ metrics=[
+ MetricResult(name="faithfulness", score=0.9),
+ MetricResult(name="relevancy", score=0.8),
+ ],
+ )
+ assert result.scores == {"faithfulness": 0.9, "relevancy": 0.8}
+
+
+class TestScoreSample:
+ @patch("pharmagraphrag.evaluation.metrics._get_evaluator_embeddings")
+ @patch("pharmagraphrag.evaluation.metrics._get_evaluator_llm")
+ @patch("pharmagraphrag.evaluation.metrics.get_reference_free_metrics")
+ def test_score_without_reference(self, mock_free, mock_llm, mock_emb):
+ mock_metric = MagicMock()
+ mock_metric.single_score.return_value = 0.85
+ type(mock_metric).__name__ = "Faithfulness"
+ mock_free.return_value = [mock_metric]
+
+ result = score_sample(
+ question="test?",
+ answer="answer",
+ contexts=["context"],
+ reference=None,
+ )
+ assert len(result.metrics) == 1
+ assert result.metrics[0].score == 0.85
+
+ @patch("pharmagraphrag.evaluation.metrics._get_evaluator_embeddings")
+ @patch("pharmagraphrag.evaluation.metrics._get_evaluator_llm")
+ @patch("pharmagraphrag.evaluation.metrics.get_reference_metrics")
+ @patch("pharmagraphrag.evaluation.metrics.get_reference_free_metrics")
+ def test_score_with_reference(self, mock_free, mock_ref, mock_llm, mock_emb):
+ mock_metric1 = MagicMock()
+ mock_metric1.single_score.return_value = 0.9
+ type(mock_metric1).__name__ = "Faithfulness"
+ mock_free.return_value = [mock_metric1]
+
+ mock_metric2 = MagicMock()
+ mock_metric2.single_score.return_value = 0.7
+ type(mock_metric2).__name__ = "ContextRecall"
+ mock_ref.return_value = [mock_metric2]
+
+ result = score_sample(
+ question="test?",
+ answer="answer",
+ contexts=["context"],
+ reference="ground truth",
+ )
+ assert len(result.metrics) == 2
+ assert result.scores["Faithfulness"] == 0.9
+ assert result.scores["ContextRecall"] == 0.7
+
+ @patch("pharmagraphrag.evaluation.metrics._get_evaluator_embeddings")
+ @patch("pharmagraphrag.evaluation.metrics._get_evaluator_llm")
+ @patch("pharmagraphrag.evaluation.metrics.get_reference_free_metrics")
+ def test_score_handles_metric_error(self, mock_free, mock_llm, mock_emb):
+ mock_metric = MagicMock()
+ mock_metric.single_score.side_effect = RuntimeError("API error")
+ type(mock_metric).__name__ = "Faithfulness"
+ mock_free.return_value = [mock_metric]
+
+ result = score_sample(
+ question="test?",
+ answer="answer",
+ contexts=["context"],
+ )
+ assert result.metrics[0].score == -1.0
+
+
+# ===========================================================================
+# Runner tests
+# ===========================================================================
+
+
+class TestRunConfig:
+ def test_defaults(self):
+ config = RunConfig()
+ assert config.mode == "classic"
+ assert config.api_url == "http://localhost:8000"
+ assert config.timeout == 120.0
+
+
+class TestPipelineResponse:
+ def test_defaults(self):
+ resp = PipelineResponse()
+ assert resp.answer == ""
+ assert resp.contexts == []
+ assert resp.error is None
+
+
+class TestCallClassic:
+ @patch("pharmagraphrag.evaluation.runner.httpx.post")
+ def test_successful_call(self, mock_post):
+ mock_response = MagicMock()
+ mock_response.json.return_value = {
+ "answer": "Aspirin causes GI bleeding.",
+ "sources": [
+ {"snippet": "ASPIRIN CAUSES GI_HAEMORRHAGE", "type": "graph"},
+ {"snippet": "Aspirin label text", "type": "vector"},
+ ],
+ }
+ mock_response.raise_for_status = MagicMock()
+ mock_post.return_value = mock_response
+
+ config = RunConfig(api_url="http://test:8000")
+ resp = _call_classic("test question?", config)
+
+ assert resp.answer == "Aspirin causes GI bleeding."
+ assert len(resp.contexts) == 2
+ assert resp.error is None
+
+ @patch("pharmagraphrag.evaluation.runner.httpx.post")
+ def test_handles_error(self, mock_post):
+ mock_post.side_effect = Exception("Connection refused")
+
+ config = RunConfig(api_url="http://test:8000")
+ resp = _call_classic("test?", config)
+
+ assert resp.error is not None
+ assert "Connection refused" in resp.error
+
+
+class TestCallAgent:
+ @patch("pharmagraphrag.evaluation.runner.httpx.post")
+ def test_parses_agent_response(self, mock_post):
+ """Verify _call_agent correctly parses tool_results, vector_data, and tool_calls."""
+ mock_response = MagicMock()
+ mock_response.json.return_value = {
+ "answer": "Aspirin interacts with Warfarin.",
+ "tool_calls": [
+ {"tool": "search_drug_info", "args": {}},
+ {"tool": "list_drug_interactions", "args": {}},
+ ],
+ "tool_results": [
+ {"tool": "search_drug_info", "content": "Drug info context"},
+ {"tool": "list_drug_interactions", "content": "Interaction context"},
+ ],
+ "vector_data": [
+ {"text": "Vector snippet about aspirin"},
+ {"snippet": "Another vector snippet"},
+ ],
+ }
+ mock_response.raise_for_status = MagicMock()
+ mock_post.return_value = mock_response
+
+ config = RunConfig(api_url="http://test:8000")
+ resp = _call_agent("test question?", config)
+
+ assert resp.answer == "Aspirin interacts with Warfarin."
+ assert resp.tool_calls == ["search_drug_info", "list_drug_interactions"]
+ assert len(resp.contexts) == 4 # 2 tool_results + 2 vector_data
+ assert "Drug info context" in resp.contexts
+ assert "Vector snippet about aspirin" in resp.contexts
+ assert resp.error is None
+
+ @patch("pharmagraphrag.evaluation.runner.httpx.post")
+ def test_handles_error(self, mock_post):
+ mock_post.side_effect = Exception("Timeout")
+
+ config = RunConfig(api_url="http://test:8000")
+ resp = _call_agent("test?", config)
+
+ assert resp.error is not None
+ assert "Timeout" in resp.error
+
+
+class TestRunPipeline:
+ @patch("pharmagraphrag.evaluation.runner._call_classic")
+ def test_routes_to_classic(self, mock_classic):
+ mock_classic.return_value = PipelineResponse(answer="test")
+ config = RunConfig(mode="classic")
+ resp = run_pipeline("question?", config)
+ mock_classic.assert_called_once()
+ assert resp.answer == "test"
+
+ @patch("pharmagraphrag.evaluation.runner._call_agent")
+ def test_routes_to_agent(self, mock_agent):
+ mock_agent.return_value = PipelineResponse(answer="agent answer")
+ config = RunConfig(mode="agent")
+ run_pipeline("question?", config)
+ mock_agent.assert_called_once_with("question?", config, endpoint="/agent/query")
+
+ @patch("pharmagraphrag.evaluation.runner._call_agent")
+ def test_routes_to_multi(self, mock_agent):
+ mock_agent.return_value = PipelineResponse(answer="multi answer")
+ config = RunConfig(mode="multi")
+ run_pipeline("question?", config)
+ mock_agent.assert_called_once_with("question?", config, endpoint="/agent/multi")
+
+ def test_unknown_mode(self):
+ config = RunConfig(mode="unknown")
+ resp = run_pipeline("question?", config)
+ assert resp.error is not None
+
+
+class TestEvaluateSample:
+ @patch("pharmagraphrag.evaluation.runner.score_sample")
+ @patch("pharmagraphrag.evaluation.runner.run_pipeline")
+ def test_successful_evaluation(self, mock_pipeline, mock_score):
+ mock_pipeline.return_value = PipelineResponse(
+ answer="GI bleeding",
+ contexts=["context"],
+ tool_calls=["search_drug_info"],
+ )
+ mock_score.return_value = EvalResult(
+ question="test?",
+ answer="GI bleeding",
+ metrics=[MetricResult(name="faithfulness", score=0.9)],
+ )
+
+ sample = EvalSample(id="q01", question="test?")
+ config = RunConfig()
+ result = evaluate_sample(sample, config)
+
+ assert result.metrics[0].score == 0.9
+ assert sample.answer == "GI bleeding"
+ assert sample.tool_calls == ["search_drug_info"]
+
+ @patch("pharmagraphrag.evaluation.runner.run_pipeline")
+ def test_pipeline_error(self, mock_pipeline):
+ mock_pipeline.return_value = PipelineResponse(error="timeout")
+
+ sample = EvalSample(id="q01", question="test?")
+ config = RunConfig()
+ result = evaluate_sample(sample, config)
+
+ assert result.metrics[0].name == "error"
+ assert result.metrics[0].score == -1.0
+
+
+class TestExportResults:
+ def test_exports_csv(self, tmp_path: Path):
+ results = [
+ EvalResult(
+ question="test?",
+ answer="answer",
+ metrics=[
+ MetricResult(name="faithfulness", score=0.9),
+ MetricResult(name="relevancy", score=0.8),
+ ],
+ reference="ground truth",
+ ),
+ ]
+ path = export_results(results, tmp_path / "results.csv")
+ assert path.exists()
+
+ content = path.read_text()
+ assert "faithfulness" in content
+ assert "0.9" in content
+
+
+class TestComputeSummary:
+ def test_averages(self):
+ results = [
+ EvalResult(
+ question="q1",
+ answer="a1",
+ metrics=[
+ MetricResult(name="faithfulness", score=0.8),
+ MetricResult(name="relevancy", score=0.6),
+ ],
+ ),
+ EvalResult(
+ question="q2",
+ answer="a2",
+ metrics=[
+ MetricResult(name="faithfulness", score=0.9),
+ MetricResult(name="relevancy", score=0.7),
+ ],
+ ),
+ ]
+ summary = compute_summary(results)
+ assert summary["faithfulness"] == pytest.approx(0.85)
+ assert summary["relevancy"] == pytest.approx(0.65)
+
+ def test_excludes_errors(self):
+ results = [
+ EvalResult(
+ question="q1",
+ answer="a1",
+ metrics=[
+ MetricResult(name="faithfulness", score=0.8),
+ MetricResult(name="error", score=-1.0),
+ ],
+ ),
+ ]
+ summary = compute_summary(results)
+ assert "error" not in summary
+ assert summary["faithfulness"] == 0.8
+
+
+# ===========================================================================
+# Agent evaluation tests
+# ===========================================================================
+
+
+class TestEvaluateToolSelection:
+ def test_perfect_match(self, eval_sample_with_response: EvalSample):
+ result = evaluate_tool_selection(eval_sample_with_response)
+ assert result.tool_precision == 1.0
+ assert result.tool_recall == 1.0
+ assert result.tool_f1 == 1.0
+ assert result.goal_achieved is True
+
+ def test_partial_match(self):
+ sample = EvalSample(
+ id="q02",
+ question="test?",
+ expected_tools=["search_drug_info", "list_drug_interactions"],
+ )
+ sample.answer = "Some answer"
+ sample.tool_calls = ["search_drug_info", "search_adverse_events"]
+
+ result = evaluate_tool_selection(sample)
+ assert result.tool_precision == 0.5 # 1/2
+ assert result.tool_recall == 0.5 # 1/2
+ assert result.tool_f1 == 0.5
+
+ def test_no_tools_expected_or_called(self):
+ sample = EvalSample(id="q03", question="test?", expected_tools=[])
+ sample.tool_calls = []
+
+ result = evaluate_tool_selection(sample)
+ assert result.tool_precision == 1.0
+ assert result.tool_recall == 1.0
+
+ def test_no_answer_means_goal_not_achieved(self):
+ sample = EvalSample(
+ id="q04",
+ question="test?",
+ expected_tools=["search_drug_info"],
+ )
+ sample.answer = ""
+ sample.tool_calls = ["search_drug_info"]
+
+ result = evaluate_tool_selection(sample)
+ assert result.goal_achieved is False
+
+ def test_error_answer_means_goal_not_achieved(self):
+ sample = EvalSample(
+ id="q05",
+ question="test?",
+ expected_tools=["search_drug_info"],
+ )
+ sample.answer = "ERROR: timeout"
+ sample.tool_calls = ["search_drug_info"]
+
+ result = evaluate_tool_selection(sample)
+ assert result.goal_achieved is False
+
+ def test_to_dict(self, eval_sample_with_response: EvalSample):
+ result = evaluate_tool_selection(eval_sample_with_response)
+ d = result.to_dict()
+ assert d["sample_id"] == "q01"
+ assert d["tool_f1"] == 1.0
+
+
+class TestEvaluateAgentDataset:
+ def test_summary_metrics(self):
+ samples = [
+ EvalSample(id="q01", question="q1", expected_tools=["tool_a"]),
+ EvalSample(id="q02", question="q2", expected_tools=["tool_a", "tool_b"]),
+ ]
+ samples[0].answer = "answer1"
+ samples[0].tool_calls = ["tool_a"]
+ samples[1].answer = "answer2"
+ samples[1].tool_calls = ["tool_a"]
+
+ summary = evaluate_agent_dataset(samples)
+ assert summary.total_samples == 2
+ assert summary.avg_tool_precision == 1.0 # both called tools were correct
+ assert summary.avg_tool_recall == 0.75 # q01: 1/1, q02: 1/2
+
+ def test_empty_dataset(self):
+ summary = evaluate_agent_dataset([])
+ assert summary.total_samples == 0
+
+
+# ===========================================================================
+# Integration: default testset loads
+# ===========================================================================
+
+
+class TestDefaultTestset:
+ def test_default_testset_exists(self):
+ """Verify the curated testset file exists and loads."""
+ from pharmagraphrag.evaluation.dataset import DEFAULT_TESTSET_PATH
+
+ assert DEFAULT_TESTSET_PATH.exists(), (
+ f"Expected curated default testset at {DEFAULT_TESTSET_PATH}"
+ )
+ dataset = load_testset()
+ assert len(dataset) >= 20
+ assert all(s.question for s in dataset.samples)
+ assert all(s.reference for s in dataset.samples)
diff --git a/tests/test_observability.py b/tests/test_observability.py
index 2cadad7..ac5089c 100644
--- a/tests/test_observability.py
+++ b/tests/test_observability.py
@@ -21,8 +21,10 @@ def _reset_state():
import pharmagraphrag.observability as obs
obs._langfuse_initialized = False
+ obs._langfuse_disabled = False
yield
obs._langfuse_initialized = False
+ obs._langfuse_disabled = False
def _settings_disabled(**overrides):
diff --git a/uv.lock b/uv.lock
index 12b78e0..10724c7 100644
--- a/uv.lock
+++ b/uv.lock
@@ -183,6 +183,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/38/0e/27be9fdef66e72d64c0cdc3cc2823101b80585f8119b5c112c2e8f5f7dab/anyio-4.12.1-py3-none-any.whl", hash = "sha256:d405828884fc140aa80a3c667b8beed277f1dfedec42ba031bd6ac3db606ab6c", size = 113592, upload-time = "2026-01-06T11:45:19.497Z" },
]
+[[package]]
+name = "appdirs"
+version = "1.4.4"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/d7/d8/05696357e0311f5b5c316d7b95f46c669dd9c15aaeecbb48c7d0aeb88c40/appdirs-1.4.4.tar.gz", hash = "sha256:7d5d0167b2b1ba821647616af46a749d1c653740dd0d2415100fe26e27afdf41", size = 13470, upload-time = "2020-05-11T07:59:51.037Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/3b/00/2344469e2084fb287c2e0b57b72910309874c3245463acd6cf5e3db69324/appdirs-1.4.4-py2.py3-none-any.whl", hash = "sha256:a841dacd6b99318a741b166adb07e19ee71a274450e68237b4650ca1055ab128", size = 9566, upload-time = "2020-05-11T07:59:49.499Z" },
+]
+
[[package]]
name = "asttokens"
version = "3.0.1"
@@ -723,6 +732,44 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/b8/5e/db279a3bfbd18d59d0598922a3b3c1454908d0969e8372260afec9736376/cuda_pathfinder-1.3.4-py3-none-any.whl", hash = "sha256:fb983f6e0d43af27ef486e14d5989b5f904ef45cedf40538bfdcbffa6bb01fb2", size = 30878, upload-time = "2026-02-11T18:50:31.008Z" },
]
+[[package]]
+name = "dataclasses-json"
+version = "0.6.7"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "marshmallow" },
+ { name = "typing-inspect" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/64/a4/f71d9cf3a5ac257c993b5ca3f93df5f7fb395c725e7f1e6479d2514173c3/dataclasses_json-0.6.7.tar.gz", hash = "sha256:b6b3e528266ea45b9535223bc53ca645f5208833c29229e847b3f26a1cc55fc0", size = 32227, upload-time = "2024-06-09T16:20:19.103Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c3/be/d0d44e092656fe7a06b55e6103cbce807cdbdee17884a5367c68c9860853/dataclasses_json-0.6.7-py3-none-any.whl", hash = "sha256:0dbf33f26c8d5305befd61b39d2b3414e8a407bedc2834dea9b8d642666fb40a", size = 28686, upload-time = "2024-06-09T16:20:16.715Z" },
+]
+
+[[package]]
+name = "datasets"
+version = "4.8.4"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "dill" },
+ { name = "filelock" },
+ { name = "fsspec", extra = ["http"] },
+ { name = "httpx" },
+ { name = "huggingface-hub" },
+ { name = "multiprocess" },
+ { name = "numpy" },
+ { name = "packaging" },
+ { name = "pandas" },
+ { name = "pyarrow" },
+ { name = "pyyaml" },
+ { name = "requests" },
+ { name = "tqdm" },
+ { name = "xxhash" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/22/22/73e46ac7a8c25e7ef0b3bd6f10da3465021d90219a32eb0b4d2afea4c56e/datasets-4.8.4.tar.gz", hash = "sha256:a1429ed853275ce7943a01c6d2e25475b4501eb758934362106a280470df3a52", size = 604382, upload-time = "2026-03-23T14:21:17.987Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b0/e5/247d094108e42ac26363ab8dc57f168840cf7c05774b40ffeb0d78868fcc/datasets-4.8.4-py3-none-any.whl", hash = "sha256:cdc8bee4698e549d78bf1fed6aea2eebc760b22b084f07e6fc020c6577a6ce6d", size = 526991, upload-time = "2026-03-23T14:21:15.89Z" },
+]
+
[[package]]
name = "decorator"
version = "5.2.1"
@@ -732,6 +779,24 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/4e/8c/f3147f5c4b73e7550fe5f9352eaa956ae838d5c51eb58e7a25b9f3e2643b/decorator-5.2.1-py3-none-any.whl", hash = "sha256:d316bb415a2d9e2d2b3abcc4084c6502fc09240e292cd76a76afc106a1c8e04a", size = 9190, upload-time = "2025-02-24T04:41:32.565Z" },
]
+[[package]]
+name = "dill"
+version = "0.4.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/81/e1/56027a71e31b02ddc53c7d65b01e68edf64dea2932122fe7746a516f75d5/dill-0.4.1.tar.gz", hash = "sha256:423092df4182177d4d8ba8290c8a5b640c66ab35ec7da59ccfa00f6fa3eea5fa", size = 187315, upload-time = "2026-01-19T02:36:56.85Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/1e/77/dc8c558f7593132cf8fefec57c4f60c83b16941c574ac5f619abb3ae7933/dill-0.4.1-py3-none-any.whl", hash = "sha256:1e1ce33e978ae97fcfcff5638477032b801c46c7c65cf717f95fbc2248f79a9d", size = 120019, upload-time = "2026-01-19T02:36:55.663Z" },
+]
+
+[[package]]
+name = "diskcache"
+version = "5.6.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/3f/21/1c1ffc1a039ddcc459db43cc108658f32c57d271d7289a2794e401d0fdb6/diskcache-5.6.3.tar.gz", hash = "sha256:2c3a3fa2743d8535d832ec61c2054a1641f41775aa7c556758a109941e33e4fc", size = 67916, upload-time = "2023-08-31T06:12:00.316Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/3f/27/4570e78fc0bf5ea0ca45eb1de3818a23787af9b390c0b0a0033a1b8236f9/diskcache-5.6.3-py3-none-any.whl", hash = "sha256:5e31b2d5fbad117cc363ebaf6b689474db18a1f6438bc82358b024abd4c2ca19", size = 45550, upload-time = "2023-08-31T06:11:58.822Z" },
+]
+
[[package]]
name = "distlib"
version = "0.4.0"
@@ -750,6 +815,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/12/b3/231ffd4ab1fc9d679809f356cebee130ac7daa00d6d6f3206dd4fd137e9e/distro-1.9.0-py3-none-any.whl", hash = "sha256:7bffd925d65168f85027d8da9af6bddab658135b840670a223589bc0c8ef02b2", size = 20277, upload-time = "2023-12-24T09:54:30.421Z" },
]
+[[package]]
+name = "docstring-parser"
+version = "0.18.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/e0/4d/f332313098c1de1b2d2ff91cf2674415cc7cddab2ca1b01ae29774bd5fdf/docstring_parser-0.18.0.tar.gz", hash = "sha256:292510982205c12b1248696f44959db3cdd1740237a968ea1e2e7a900eeb2015", size = 29341, upload-time = "2026-04-14T04:09:19.867Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a7/5f/ed01f9a3cdffbd5a008556fc7b2a08ddb1cc6ace7effa7340604b1d16699/docstring_parser-0.18.0-py3-none-any.whl", hash = "sha256:b3fcbed555c47d8479be0796ef7e19c2670d428d72e96da63f3a40122860374b", size = 22484, upload-time = "2026-04-14T04:09:18.638Z" },
+]
+
[[package]]
name = "durationpy"
version = "0.10"
@@ -924,6 +998,11 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/e6/ab/fb21f4c939bb440104cc2b396d3be1d9b7a9fd3c6c2a53d98c45b3d7c954/fsspec-2026.2.0-py3-none-any.whl", hash = "sha256:98de475b5cb3bd66bedd5c4679e87b4fdfe1a3bf4d707b151b3c07e58c9a2437", size = 202505, upload-time = "2026-02-05T21:50:51.819Z" },
]
+[package.optional-dependencies]
+http = [
+ { name = "aiohttp" },
+]
+
[[package]]
name = "gitdb"
version = "4.0.12"
@@ -1001,6 +1080,53 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/c4/ab/09169d5a4612a5f92490806649ac8d41e3ec9129c636754575b3553f4ea4/googleapis_common_protos-1.72.0-py3-none-any.whl", hash = "sha256:4299c5a82d5ae1a9702ada957347726b167f9f8d1fc352477702a1e851ff4038", size = 297515, upload-time = "2025-11-06T18:29:13.14Z" },
]
+[[package]]
+name = "greenlet"
+version = "3.4.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/86/94/a5935717b307d7c71fe877b52b884c6af707d2d2090db118a03fbd799369/greenlet-3.4.0.tar.gz", hash = "sha256:f50a96b64dafd6169e595a5c56c9146ef80333e67d4476a65a9c55f400fc22ff", size = 195913, upload-time = "2026-04-08T17:08:00.863Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/fb/c6/dba32cab7e3a625b011aa5647486e2d28423a48845a2998c126dd69c85e1/greenlet-3.4.0-cp311-cp311-macosx_11_0_universal2.whl", hash = "sha256:805bebb4945094acbab757d34d6e1098be6de8966009ab9ca54f06ff492def58", size = 285504, upload-time = "2026-04-08T15:52:14.071Z" },
+ { url = "https://files.pythonhosted.org/packages/54/f4/7cb5c2b1feb9a1f50e038be79980dfa969aa91979e5e3a18fdbcfad2c517/greenlet-3.4.0-cp311-cp311-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:439fc2f12b9b512d9dfa681c5afe5f6b3232c708d13e6f02c845e0d9f4c2d8c6", size = 605476, upload-time = "2026-04-08T16:24:37.064Z" },
+ { url = "https://files.pythonhosted.org/packages/d6/af/b66ab0b2f9a4c5a867c136bf66d9599f34f21a1bcca26a2884a29c450bd9/greenlet-3.4.0-cp311-cp311-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a70ed1cb0295bee1df57b63bf7f46b4e56a5c93709eea769c1fec1bb23a95875", size = 618336, upload-time = "2026-04-08T16:30:56.59Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/5c/8c5633ece6ba611d64bf2770219a98dd439921d6424e4e8cf16b0ac74ea5/greenlet-3.4.0-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c660bce1940a1acae5f51f0a064f1bc785d07ea16efcb4bc708090afc4d69e83", size = 613515, upload-time = "2026-04-08T15:56:32.478Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/df/950d15bca0d90a0e7395eb777903060504cdb509b7b705631e8fb69ff415/greenlet-3.4.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:ee407d4d1ca9dc632265aee1c8732c4a2d60adff848057cdebfe5fe94eb2c8a2", size = 1574623, upload-time = "2026-04-08T16:26:18.596Z" },
+ { url = "https://files.pythonhosted.org/packages/1a/e7/0839afab829fcb7333c9ff6d80c040949510055d2d4d63251f0d1c7c804e/greenlet-3.4.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:956215d5e355fffa7c021d168728321fd4d31fd730ac609b1653b450f6a4bc71", size = 1639579, upload-time = "2026-04-08T15:57:29.231Z" },
+ { url = "https://files.pythonhosted.org/packages/d9/2b/b4482401e9bcaf9f5c97f67ead38db89c19520ff6d0d6699979c6efcc200/greenlet-3.4.0-cp311-cp311-win_amd64.whl", hash = "sha256:5cb614ace7c27571270354e9c9f696554d073f8aa9319079dcba466bbdead711", size = 238233, upload-time = "2026-04-08T17:02:54.286Z" },
+ { url = "https://files.pythonhosted.org/packages/0c/4d/d8123a4e0bcd583d5cfc8ddae0bbe29c67aab96711be331a7cc935a35966/greenlet-3.4.0-cp311-cp311-win_arm64.whl", hash = "sha256:04403ac74fe295a361f650818de93be11b5038a78f49ccfb64d3b1be8fbf1267", size = 235045, upload-time = "2026-04-08T17:04:05.072Z" },
+ { url = "https://files.pythonhosted.org/packages/65/8b/3669ad3b3f247a791b2b4aceb3aa5a31f5f6817bf547e4e1ff712338145a/greenlet-3.4.0-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:1a54a921561dd9518d31d2d3db4d7f80e589083063ab4d3e2e950756ef809e1a", size = 286902, upload-time = "2026-04-08T15:52:12.138Z" },
+ { url = "https://files.pythonhosted.org/packages/38/3e/3c0e19b82900873e2d8469b590a6c4b3dfd2b316d0591f1c26b38a4879a5/greenlet-3.4.0-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:16dec271460a9a2b154e3b1c2fa1050ce6280878430320e85e08c166772e3f97", size = 606099, upload-time = "2026-04-08T16:24:38.408Z" },
+ { url = "https://files.pythonhosted.org/packages/b5/33/99fef65e7754fc76a4ed14794074c38c9ed3394a5bd129d7f61b705f3168/greenlet-3.4.0-cp312-cp312-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:90036ce224ed6fe75508c1907a77e4540176dcf0744473627785dd519c6f9996", size = 618837, upload-time = "2026-04-08T16:30:58.298Z" },
+ { url = "https://files.pythonhosted.org/packages/36/f7/229f3aed6948faa20e0616a0b8568da22e365ede6a54d7d369058b128afd/greenlet-3.4.0-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a1c4f6b453006efb8310affb2d132832e9bbb4fc01ce6df6b70d810d38f1f6dc", size = 615062, upload-time = "2026-04-08T15:56:33.766Z" },
+ { url = "https://files.pythonhosted.org/packages/08/97/d988180011aa40135c46cd0d0cf01dd97f7162bae14139b4a3ef54889ba5/greenlet-3.4.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:9b2d9a138ffa0e306d0e2b72976d2fb10b97e690d40ab36a472acaab0838e2de", size = 1573511, upload-time = "2026-04-08T16:26:20.058Z" },
+ { url = "https://files.pythonhosted.org/packages/d4/0f/a5a26fe152fb3d12e6a474181f6e9848283504d0afd095f353d85726374b/greenlet-3.4.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:8424683caf46eb0eb6f626cb95e008e8cc30d0cb675bdfa48200925c79b38a08", size = 1640396, upload-time = "2026-04-08T15:57:30.88Z" },
+ { url = "https://files.pythonhosted.org/packages/42/cf/bb2c32d9a100e36ee9f6e38fad6b1e082b8184010cb06259b49e1266ca01/greenlet-3.4.0-cp312-cp312-win_amd64.whl", hash = "sha256:a0a53fb071531d003b075c444014ff8f8b1a9898d36bb88abd9ac7b3524648a2", size = 238892, upload-time = "2026-04-08T17:03:10.094Z" },
+ { url = "https://files.pythonhosted.org/packages/b7/47/6c41314bac56e71436ce551c7fbe3cc830ed857e6aa9708dbb9c65142eb6/greenlet-3.4.0-cp312-cp312-win_arm64.whl", hash = "sha256:f38b81880ba28f232f1f675893a39cf7b6db25b31cc0a09bb50787ecf957e85e", size = 235599, upload-time = "2026-04-08T15:52:54.3Z" },
+ { url = "https://files.pythonhosted.org/packages/7a/75/7e9cd1126a1e1f0cd67b0eda02e5221b28488d352684704a78ed505bd719/greenlet-3.4.0-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:43748988b097f9c6f09364f260741aa73c80747f63389824435c7a50bfdfd5c1", size = 285856, upload-time = "2026-04-08T15:52:45.82Z" },
+ { url = "https://files.pythonhosted.org/packages/9d/c4/3e2df392e5cb199527c4d9dbcaa75c14edcc394b45040f0189f649631e3c/greenlet-3.4.0-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5566e4e2cd7a880e8c27618e3eab20f3494452d12fd5129edef7b2f7aa9a36d1", size = 610208, upload-time = "2026-04-08T16:24:39.674Z" },
+ { url = "https://files.pythonhosted.org/packages/da/af/750cdfda1d1bd30a6c28080245be8d0346e669a98fdbae7f4102aa95fff3/greenlet-3.4.0-cp313-cp313-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:1054c5a3c78e2ab599d452f23f7adafef55062a783a8e241d24f3b633ba6ff82", size = 621269, upload-time = "2026-04-08T16:30:59.767Z" },
+ { url = "https://files.pythonhosted.org/packages/54/78/0cbc693622cd54ebe25207efbb3a0eb07c2639cb8594f6e3aaaa0bb077a8/greenlet-3.4.0-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f82cb6cddc27dd81c96b1506f4aa7def15070c3b2a67d4e46fd19016aacce6cf", size = 617549, upload-time = "2026-04-08T15:56:34.893Z" },
+ { url = "https://files.pythonhosted.org/packages/ba/c0/8966767de01343c1ff47e8b855dc78e7d1a8ed2b7b9c83576a57e289f81d/greenlet-3.4.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:227a46251ecba4ff46ae742bc5ce95c91d5aceb4b02f885487aff269c127a729", size = 1575310, upload-time = "2026-04-08T16:26:21.671Z" },
+ { url = "https://files.pythonhosted.org/packages/b8/38/bcdc71ba05e9a5fda87f63ffc2abcd1f15693b659346df994a48c968003d/greenlet-3.4.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:5b99e87be7eba788dd5b75ba1cde5639edffdec5f91fe0d734a249535ec3408c", size = 1640435, upload-time = "2026-04-08T15:57:32.572Z" },
+ { url = "https://files.pythonhosted.org/packages/a1/c2/19b664b7173b9e4ef5f77e8cef9f14c20ec7fce7920dc1ccd7afd955d093/greenlet-3.4.0-cp313-cp313-win_amd64.whl", hash = "sha256:849f8bc17acd6295fcb5de8e46d55cc0e52381c56eaf50a2afd258e97bc65940", size = 238760, upload-time = "2026-04-08T17:04:03.878Z" },
+ { url = "https://files.pythonhosted.org/packages/9b/96/795619651d39c7fbd809a522f881aa6f0ead504cc8201c3a5b789dfaef99/greenlet-3.4.0-cp313-cp313-win_arm64.whl", hash = "sha256:9390ad88b652b1903814eaabd629ca184db15e0eeb6fe8a390bbf8b9106ae15a", size = 235498, upload-time = "2026-04-08T17:05:00.584Z" },
+ { url = "https://files.pythonhosted.org/packages/78/02/bde66806e8f169cf90b14d02c500c44cdbe02c8e224c9c67bafd1b8cadd1/greenlet-3.4.0-cp314-cp314-macosx_11_0_universal2.whl", hash = "sha256:10a07aca6babdd18c16a3f4f8880acfffc2b88dfe431ad6aa5f5740759d7d75e", size = 286291, upload-time = "2026-04-08T17:09:34.307Z" },
+ { url = "https://files.pythonhosted.org/packages/05/1f/39da1c336a87d47c58352fb8a78541ce63d63ae57c5b9dae1fe02801bbc2/greenlet-3.4.0-cp314-cp314-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:076e21040b3a917d3ce4ad68fb5c3c6b32f1405616c4a57aa83120979649bd3d", size = 656749, upload-time = "2026-04-08T16:24:41.721Z" },
+ { url = "https://files.pythonhosted.org/packages/d3/6c/90ee29a4ee27af7aa2e2ec408799eeb69ee3fcc5abcecac6ddd07a5cd0f2/greenlet-3.4.0-cp314-cp314-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:e82689eea4a237e530bb5cb41b180ef81fa2160e1f89422a67be7d90da67f615", size = 669084, upload-time = "2026-04-08T16:31:01.372Z" },
+ { url = "https://files.pythonhosted.org/packages/07/49/d4cad6e5381a50947bb973d2f6cf6592621451b09368b8c20d9b8af49c5b/greenlet-3.4.0-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4df3b0b2289ec686d3c821a5fee44259c05cfe824dd5e6e12c8e5f5df23085cf", size = 665621, upload-time = "2026-04-08T15:56:35.995Z" },
+ { url = "https://files.pythonhosted.org/packages/37/31/d1edd54f424761b5d47718822f506b435b6aab2f3f93b465441143ea5119/greenlet-3.4.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:8bff29d586ea415688f4cec96a591fcc3bf762d046a796cdadc1fdb6e7f2d5bf", size = 1622259, upload-time = "2026-04-08T16:26:23.201Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/c6/6d3f9cdcb21c4e12a79cb332579f1c6aa1af78eb68059c5a957c7812d95e/greenlet-3.4.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:8a569c2fb840c53c13a2b8967c63621fafbd1a0e015b9c82f408c33d626a2fda", size = 1686916, upload-time = "2026-04-08T15:57:34.282Z" },
+ { url = "https://files.pythonhosted.org/packages/63/45/c1ca4a1ad975de4727e52d3ffe641ae23e1d7a8ffaa8ff7a0477e1827b92/greenlet-3.4.0-cp314-cp314-win_amd64.whl", hash = "sha256:207ba5b97ea8b0b60eb43ffcacf26969dd83726095161d676aac03ff913ee50d", size = 239821, upload-time = "2026-04-08T17:03:48.423Z" },
+ { url = "https://files.pythonhosted.org/packages/71/c4/6f621023364d7e85a4769c014c8982f98053246d142420e0328980933ceb/greenlet-3.4.0-cp314-cp314-win_arm64.whl", hash = "sha256:f8296d4e2b92af34ebde81085a01690f26a51eb9ac09a0fcadb331eb36dbc802", size = 236932, upload-time = "2026-04-08T17:04:33.551Z" },
+ { url = "https://files.pythonhosted.org/packages/d4/8f/18d72b629783f5e8d045a76f5325c1e938e659a9e4da79c7dcd10169a48d/greenlet-3.4.0-cp314-cp314t-macosx_11_0_universal2.whl", hash = "sha256:d70012e51df2dbbccfaf63a40aaf9b40c8bed37c3e3a38751c926301ce538ece", size = 294681, upload-time = "2026-04-08T15:52:35.778Z" },
+ { url = "https://files.pythonhosted.org/packages/9e/ad/5fa86ec46769c4153820d58a04062285b3b9e10ba3d461ee257b68dcbf53/greenlet-3.4.0-cp314-cp314t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a58bec0751f43068cd40cff31bb3ca02ad6000b3a51ca81367af4eb5abc480c8", size = 658899, upload-time = "2026-04-08T16:24:43.32Z" },
+ { url = "https://files.pythonhosted.org/packages/43/f0/4e8174ca0e87ae748c409f055a1ba161038c43cc0a5a6f1433a26ac2e5bf/greenlet-3.4.0-cp314-cp314t-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:05fa0803561028f4b2e3b490ee41216a842eaee11aed004cc343a996d9523aa2", size = 665284, upload-time = "2026-04-08T16:31:02.833Z" },
+ { url = "https://files.pythonhosted.org/packages/19/da/991cf7cd33662e2df92a1274b7eb4d61769294d38a1bba8a45f31364845e/greenlet-3.4.0-cp314-cp314t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e60d38719cb80b3ab5e85f9f1aed4960acfde09868af6762ccb27b260d68f4ed", size = 661861, upload-time = "2026-04-08T15:56:37.269Z" },
+ { url = "https://files.pythonhosted.org/packages/36/c5/6c2c708e14db3d9caea4b459d8464f58c32047451142fe2cfd90e7458f41/greenlet-3.4.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:7f50c804733b43eded05ae694691c9aa68bca7d0a867d67d4a3f514742a2d53f", size = 1622182, upload-time = "2026-04-08T16:26:24.777Z" },
+ { url = "https://files.pythonhosted.org/packages/7a/4c/50c5fed19378e11a29fabab1f6be39ea95358f4a0a07e115a51ca93385d8/greenlet-3.4.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:2d4f0635dc4aa638cda4b2f5a07ae9a2cff9280327b581a3fcb6f317b4fbc38a", size = 1685050, upload-time = "2026-04-08T15:57:36.453Z" },
+ { url = "https://files.pythonhosted.org/packages/db/72/85ae954d734703ab48e622c59d4ce35d77ce840c265814af9c078cacc7aa/greenlet-3.4.0-cp314-cp314t-win_amd64.whl", hash = "sha256:1a4a48f24681300c640f143ba7c404270e1ebbbcf34331d7104a4ff40f8ea705", size = 245554, upload-time = "2026-04-08T17:03:50.044Z" },
+]
+
[[package]]
name = "grpcio"
version = "1.78.0"
@@ -1154,6 +1280,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/2a/39/e50c7c3a983047577ee07d2a9e53faf5a69493943ec3f6a384bdc792deb2/httpx-0.28.1-py3-none-any.whl", hash = "sha256:d909fcccc110f8c7faf814ca82a9a4d816bc5a6dbfea25d6591d6985b8ba59ad", size = 73517, upload-time = "2024-12-06T15:37:21.509Z" },
]
+[[package]]
+name = "httpx-sse"
+version = "0.4.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/0f/4c/751061ffa58615a32c31b2d82e8482be8dd4a89154f003147acee90f2be9/httpx_sse-0.4.3.tar.gz", hash = "sha256:9b1ed0127459a66014aec3c56bebd93da3c1bc8bb6618c8082039a44889a755d", size = 15943, upload-time = "2025-10-10T21:48:22.271Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d2/fd/6668e5aec43ab844de6fc74927e155a3b37bf40d7c3790e49fc0406b6578/httpx_sse-0.4.3-py3-none-any.whl", hash = "sha256:0ac1c9fe3c0afad2e0ebb25a934a59f4c7823b60792691f779fad2c5568830fc", size = 8960, upload-time = "2025-10-10T21:48:21.158Z" },
+]
+
[[package]]
name = "huggingface-hub"
version = "1.4.1"
@@ -1223,6 +1358,28 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12", size = 7484, upload-time = "2025-10-18T21:55:41.639Z" },
]
+[[package]]
+name = "instructor"
+version = "1.15.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "aiohttp" },
+ { name = "docstring-parser" },
+ { name = "jinja2" },
+ { name = "jiter" },
+ { name = "openai" },
+ { name = "pydantic" },
+ { name = "pydantic-core" },
+ { name = "requests" },
+ { name = "rich" },
+ { name = "tenacity" },
+ { name = "typer" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/dc/a4/832cfb15420360e26d2d85bd9d5fe1e4b839d52587574d389bc31284bf6f/instructor-1.15.1.tar.gz", hash = "sha256:c72406469d9025b742e83cf0c13e914b317db2089d08d889944e74fcd659ef94", size = 69948370, upload-time = "2026-04-03T01:51:30.107Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d8/c8/36c5d9b80aaf40ba9a7084a8fc18c967db6bf248a4cc8d0f0816b14284be/instructor-1.15.1-py3-none-any.whl", hash = "sha256:be81d17ba2b154a04ab4720808f24f9d6b598f80992f82eaf9cc79006099cf6c", size = 178156, upload-time = "2026-04-03T01:51:23.098Z" },
+]
+
[[package]]
name = "ipython"
version = "9.10.0"
@@ -1281,6 +1438,91 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/62/a1/3d680cbfd5f4b8f15abc1d571870c5fc3e594bb582bc3b64ea099db13e56/jinja2-3.1.6-py3-none-any.whl", hash = "sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67", size = 134899, upload-time = "2025-03-05T20:05:00.369Z" },
]
+[[package]]
+name = "jiter"
+version = "0.13.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/0d/5e/4ec91646aee381d01cdb9974e30882c9cd3b8c5d1079d6b5ff4af522439a/jiter-0.13.0.tar.gz", hash = "sha256:f2839f9c2c7e2dffc1bc5929a510e14ce0a946be9365fd1219e7ef342dae14f4", size = 164847, upload-time = "2026-02-02T12:37:56.441Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/71/29/499f8c9eaa8a16751b1c0e45e6f5f1761d180da873d417996cc7bddc8eef/jiter-0.13.0-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:ea026e70a9a28ebbdddcbcf0f1323128a8db66898a06eaad3a4e62d2f554d096", size = 311157, upload-time = "2026-02-02T12:35:37.758Z" },
+ { url = "https://files.pythonhosted.org/packages/50/f6/566364c777d2ab450b92100bea11333c64c38d32caf8dc378b48e5b20c46/jiter-0.13.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:66aa3e663840152d18cc8ff1e4faad3dd181373491b9cfdc6004b92198d67911", size = 319729, upload-time = "2026-02-02T12:35:39.246Z" },
+ { url = "https://files.pythonhosted.org/packages/73/dd/560f13ec5e4f116d8ad2658781646cca91b617ae3b8758d4a5076b278f70/jiter-0.13.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c3524798e70655ff19aec58c7d05adb1f074fecff62da857ea9be2b908b6d701", size = 354766, upload-time = "2026-02-02T12:35:40.662Z" },
+ { url = "https://files.pythonhosted.org/packages/7c/0d/061faffcfe94608cbc28a0d42a77a74222bdf5055ccdbe5fd2292b94f510/jiter-0.13.0-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:ec7e287d7fbd02cb6e22f9a00dd9c9cd504c40a61f2c61e7e1f9690a82726b4c", size = 362587, upload-time = "2026-02-02T12:35:42.025Z" },
+ { url = "https://files.pythonhosted.org/packages/92/c9/c66a7864982fd38a9773ec6e932e0398d1262677b8c60faecd02ffb67bf3/jiter-0.13.0-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:47455245307e4debf2ce6c6e65a717550a0244231240dcf3b8f7d64e4c2f22f4", size = 487537, upload-time = "2026-02-02T12:35:43.459Z" },
+ { url = "https://files.pythonhosted.org/packages/6c/86/84eb4352cd3668f16d1a88929b5888a3fe0418ea8c1dfc2ad4e7bf6e069a/jiter-0.13.0-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:ee9da221dca6e0429c2704c1b3655fe7b025204a71d4d9b73390c759d776d165", size = 373717, upload-time = "2026-02-02T12:35:44.928Z" },
+ { url = "https://files.pythonhosted.org/packages/6e/09/9fe4c159358176f82d4390407a03f506a8659ed13ca3ac93a843402acecf/jiter-0.13.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:24ab43126d5e05f3d53a36a8e11eb2f23304c6c1117844aaaf9a0aa5e40b5018", size = 362683, upload-time = "2026-02-02T12:35:46.636Z" },
+ { url = "https://files.pythonhosted.org/packages/c9/5e/85f3ab9caca0c1d0897937d378b4a515cae9e119730563572361ea0c48ae/jiter-0.13.0-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:9da38b4fedde4fb528c740c2564628fbab737166a0e73d6d46cb4bb5463ff411", size = 392345, upload-time = "2026-02-02T12:35:48.088Z" },
+ { url = "https://files.pythonhosted.org/packages/12/4c/05b8629ad546191939e6f0c2f17e29f542a398f4a52fb987bc70b6d1eb8b/jiter-0.13.0-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:0b34c519e17658ed88d5047999a93547f8889f3c1824120c26ad6be5f27b6cf5", size = 517775, upload-time = "2026-02-02T12:35:49.482Z" },
+ { url = "https://files.pythonhosted.org/packages/4d/88/367ea2eb6bc582c7052e4baf5ddf57ebe5ab924a88e0e09830dfb585c02d/jiter-0.13.0-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:d2a6394e6af690d462310a86b53c47ad75ac8c21dc79f120714ea449979cb1d3", size = 551325, upload-time = "2026-02-02T12:35:51.104Z" },
+ { url = "https://files.pythonhosted.org/packages/f3/12/fa377ffb94a2f28c41afaed093e0d70cfe512035d5ecb0cad0ae4792d35e/jiter-0.13.0-cp311-cp311-win32.whl", hash = "sha256:0f0c065695f616a27c920a56ad0d4fc46415ef8b806bf8fc1cacf25002bd24e1", size = 204709, upload-time = "2026-02-02T12:35:52.467Z" },
+ { url = "https://files.pythonhosted.org/packages/cb/16/8e8203ce92f844dfcd3d9d6a5a7322c77077248dbb12da52d23193a839cd/jiter-0.13.0-cp311-cp311-win_amd64.whl", hash = "sha256:0733312953b909688ae3c2d58d043aa040f9f1a6a75693defed7bc2cc4bf2654", size = 204560, upload-time = "2026-02-02T12:35:53.925Z" },
+ { url = "https://files.pythonhosted.org/packages/44/26/97cc40663deb17b9e13c3a5cf29251788c271b18ee4d262c8f94798b8336/jiter-0.13.0-cp311-cp311-win_arm64.whl", hash = "sha256:5d9b34ad56761b3bf0fbe8f7e55468704107608512350962d3317ffd7a4382d5", size = 189608, upload-time = "2026-02-02T12:35:55.304Z" },
+ { url = "https://files.pythonhosted.org/packages/2e/30/7687e4f87086829955013ca12a9233523349767f69653ebc27036313def9/jiter-0.13.0-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:0a2bd69fc1d902e89925fc34d1da51b2128019423d7b339a45d9e99c894e0663", size = 307958, upload-time = "2026-02-02T12:35:57.165Z" },
+ { url = "https://files.pythonhosted.org/packages/c3/27/e57f9a783246ed95481e6749cc5002a8a767a73177a83c63ea71f0528b90/jiter-0.13.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:f917a04240ef31898182f76a332f508f2cc4b57d2b4d7ad2dbfebbfe167eb505", size = 318597, upload-time = "2026-02-02T12:35:58.591Z" },
+ { url = "https://files.pythonhosted.org/packages/cf/52/e5719a60ac5d4d7c5995461a94ad5ef962a37c8bf5b088390e6fad59b2ff/jiter-0.13.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c1e2b199f446d3e82246b4fd9236d7cb502dc2222b18698ba0d986d2fecc6152", size = 348821, upload-time = "2026-02-02T12:36:00.093Z" },
+ { url = "https://files.pythonhosted.org/packages/61/db/c1efc32b8ba4c740ab3fc2d037d8753f67685f475e26b9d6536a4322bcdd/jiter-0.13.0-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:04670992b576fa65bd056dbac0c39fe8bd67681c380cb2b48efa885711d9d726", size = 364163, upload-time = "2026-02-02T12:36:01.937Z" },
+ { url = "https://files.pythonhosted.org/packages/55/8a/fb75556236047c8806995671a18e4a0ad646ed255276f51a20f32dceaeec/jiter-0.13.0-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:5a1aff1fbdb803a376d4d22a8f63f8e7ccbce0b4890c26cc7af9e501ab339ef0", size = 483709, upload-time = "2026-02-02T12:36:03.41Z" },
+ { url = "https://files.pythonhosted.org/packages/7e/16/43512e6ee863875693a8e6f6d532e19d650779d6ba9a81593ae40a9088ff/jiter-0.13.0-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:3b3fb8c2053acaef8580809ac1d1f7481a0a0bdc012fd7f5d8b18fb696a5a089", size = 370480, upload-time = "2026-02-02T12:36:04.791Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/4c/09b93e30e984a187bc8aaa3510e1ec8dcbdcd71ca05d2f56aac0492453aa/jiter-0.13.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:bdaba7d87e66f26a2c45d8cbadcbfc4bf7884182317907baf39cfe9775bb4d93", size = 360735, upload-time = "2026-02-02T12:36:06.994Z" },
+ { url = "https://files.pythonhosted.org/packages/1a/1b/46c5e349019874ec5dfa508c14c37e29864ea108d376ae26d90bee238cd7/jiter-0.13.0-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:7b88d649135aca526da172e48083da915ec086b54e8e73a425ba50999468cc08", size = 391814, upload-time = "2026-02-02T12:36:08.368Z" },
+ { url = "https://files.pythonhosted.org/packages/15/9e/26184760e85baee7162ad37b7912797d2077718476bf91517641c92b3639/jiter-0.13.0-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:e404ea551d35438013c64b4f357b0474c7abf9f781c06d44fcaf7a14c69ff9e2", size = 513990, upload-time = "2026-02-02T12:36:09.993Z" },
+ { url = "https://files.pythonhosted.org/packages/e9/34/2c9355247d6debad57a0a15e76ab1566ab799388042743656e566b3b7de1/jiter-0.13.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:1f4748aad1b4a93c8bdd70f604d0f748cdc0e8744c5547798acfa52f10e79228", size = 548021, upload-time = "2026-02-02T12:36:11.376Z" },
+ { url = "https://files.pythonhosted.org/packages/ac/4a/9f2c23255d04a834398b9c2e0e665382116911dc4d06b795710503cdad25/jiter-0.13.0-cp312-cp312-win32.whl", hash = "sha256:0bf670e3b1445fc4d31612199f1744f67f889ee1bbae703c4b54dc097e5dd394", size = 203024, upload-time = "2026-02-02T12:36:12.682Z" },
+ { url = "https://files.pythonhosted.org/packages/09/ee/f0ae675a957ae5a8f160be3e87acea6b11dc7b89f6b7ab057e77b2d2b13a/jiter-0.13.0-cp312-cp312-win_amd64.whl", hash = "sha256:15db60e121e11fe186c0b15236bd5d18381b9ddacdcf4e659feb96fc6c969c92", size = 205424, upload-time = "2026-02-02T12:36:13.93Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/02/ae611edf913d3cbf02c97cdb90374af2082c48d7190d74c1111dde08bcdd/jiter-0.13.0-cp312-cp312-win_arm64.whl", hash = "sha256:41f92313d17989102f3cb5dd533a02787cdb99454d494344b0361355da52fcb9", size = 186818, upload-time = "2026-02-02T12:36:15.308Z" },
+ { url = "https://files.pythonhosted.org/packages/91/9c/7ee5a6ff4b9991e1a45263bfc46731634c4a2bde27dfda6c8251df2d958c/jiter-0.13.0-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:1f8a55b848cbabf97d861495cd65f1e5c590246fabca8b48e1747c4dfc8f85bf", size = 306897, upload-time = "2026-02-02T12:36:16.748Z" },
+ { url = "https://files.pythonhosted.org/packages/7c/02/be5b870d1d2be5dd6a91bdfb90f248fbb7dcbd21338f092c6b89817c3dbf/jiter-0.13.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:f556aa591c00f2c45eb1b89f68f52441a016034d18b65da60e2d2875bbbf344a", size = 317507, upload-time = "2026-02-02T12:36:18.351Z" },
+ { url = "https://files.pythonhosted.org/packages/da/92/b25d2ec333615f5f284f3a4024f7ce68cfa0604c322c6808b2344c7f5d2b/jiter-0.13.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f7e1d61da332ec412350463891923f960c3073cf1aae93b538f0bb4c8cd46efb", size = 350560, upload-time = "2026-02-02T12:36:19.746Z" },
+ { url = "https://files.pythonhosted.org/packages/be/ec/74dcb99fef0aca9fbe56b303bf79f6bd839010cb18ad41000bf6cc71eec0/jiter-0.13.0-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:3097d665a27bc96fd9bbf7f86178037db139f319f785e4757ce7ccbf390db6c2", size = 363232, upload-time = "2026-02-02T12:36:21.243Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/37/f17375e0bb2f6a812d4dd92d7616e41917f740f3e71343627da9db2824ce/jiter-0.13.0-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:9d01ecc3a8cbdb6f25a37bd500510550b64ddf9f7d64a107d92f3ccb25035d0f", size = 483727, upload-time = "2026-02-02T12:36:22.688Z" },
+ { url = "https://files.pythonhosted.org/packages/77/d2/a71160a5ae1a1e66c1395b37ef77da67513b0adba73b993a27fbe47eb048/jiter-0.13.0-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:ed9bbc30f5d60a3bdf63ae76beb3f9db280d7f195dfcfa61af792d6ce912d159", size = 370799, upload-time = "2026-02-02T12:36:24.106Z" },
+ { url = "https://files.pythonhosted.org/packages/01/99/ed5e478ff0eb4e8aa5fd998f9d69603c9fd3f32de3bd16c2b1194f68361c/jiter-0.13.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:98fbafb6e88256f4454de33c1f40203d09fc33ed19162a68b3b257b29ca7f663", size = 359120, upload-time = "2026-02-02T12:36:25.519Z" },
+ { url = "https://files.pythonhosted.org/packages/16/be/7ffd08203277a813f732ba897352797fa9493faf8dc7995b31f3d9cb9488/jiter-0.13.0-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:5467696f6b827f1116556cb0db620440380434591e93ecee7fd14d1a491b6daa", size = 390664, upload-time = "2026-02-02T12:36:26.866Z" },
+ { url = "https://files.pythonhosted.org/packages/d1/84/e0787856196d6d346264d6dcccb01f741e5f0bd014c1d9a2ebe149caf4f3/jiter-0.13.0-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:2d08c9475d48b92892583df9da592a0e2ac49bcd41fae1fec4f39ba6cf107820", size = 513543, upload-time = "2026-02-02T12:36:28.217Z" },
+ { url = "https://files.pythonhosted.org/packages/65/50/ecbd258181c4313cf79bca6c88fb63207d04d5bf5e4f65174114d072aa55/jiter-0.13.0-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:aed40e099404721d7fcaf5b89bd3b4568a4666358bcac7b6b15c09fb6252ab68", size = 547262, upload-time = "2026-02-02T12:36:29.678Z" },
+ { url = "https://files.pythonhosted.org/packages/27/da/68f38d12e7111d2016cd198161b36e1f042bd115c169255bcb7ec823a3bf/jiter-0.13.0-cp313-cp313-win32.whl", hash = "sha256:36ebfbcffafb146d0e6ffb3e74d51e03d9c35ce7c625c8066cdbfc7b953bdc72", size = 200630, upload-time = "2026-02-02T12:36:31.808Z" },
+ { url = "https://files.pythonhosted.org/packages/25/65/3bd1a972c9a08ecd22eb3b08a95d1941ebe6938aea620c246cf426ae09c2/jiter-0.13.0-cp313-cp313-win_amd64.whl", hash = "sha256:8d76029f077379374cf0dbc78dbe45b38dec4a2eb78b08b5194ce836b2517afc", size = 202602, upload-time = "2026-02-02T12:36:33.679Z" },
+ { url = "https://files.pythonhosted.org/packages/15/fe/13bd3678a311aa67686bb303654792c48206a112068f8b0b21426eb6851e/jiter-0.13.0-cp313-cp313-win_arm64.whl", hash = "sha256:bb7613e1a427cfcb6ea4544f9ac566b93d5bf67e0d48c787eca673ff9c9dff2b", size = 185939, upload-time = "2026-02-02T12:36:35.065Z" },
+ { url = "https://files.pythonhosted.org/packages/49/19/a929ec002ad3228bc97ca01dbb14f7632fffdc84a95ec92ceaf4145688ae/jiter-0.13.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:fa476ab5dd49f3bf3a168e05f89358c75a17608dbabb080ef65f96b27c19ab10", size = 316616, upload-time = "2026-02-02T12:36:36.579Z" },
+ { url = "https://files.pythonhosted.org/packages/52/56/d19a9a194afa37c1728831e5fb81b7722c3de18a3109e8f282bfc23e587a/jiter-0.13.0-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ade8cb6ff5632a62b7dbd4757d8c5573f7a2e9ae285d6b5b841707d8363205ef", size = 346850, upload-time = "2026-02-02T12:36:38.058Z" },
+ { url = "https://files.pythonhosted.org/packages/36/4a/94e831c6bf287754a8a019cb966ed39ff8be6ab78cadecf08df3bb02d505/jiter-0.13.0-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9950290340acc1adaded363edd94baebcee7dabdfa8bee4790794cd5cfad2af6", size = 358551, upload-time = "2026-02-02T12:36:39.417Z" },
+ { url = "https://files.pythonhosted.org/packages/a2/ec/a4c72c822695fa80e55d2b4142b73f0012035d9fcf90eccc56bc060db37c/jiter-0.13.0-cp313-cp313t-win_amd64.whl", hash = "sha256:2b4972c6df33731aac0742b64fd0d18e0a69bc7d6e03108ce7d40c85fd9e3e6d", size = 201950, upload-time = "2026-02-02T12:36:40.791Z" },
+ { url = "https://files.pythonhosted.org/packages/b6/00/393553ec27b824fbc29047e9c7cd4a3951d7fbe4a76743f17e44034fa4e4/jiter-0.13.0-cp313-cp313t-win_arm64.whl", hash = "sha256:701a1e77d1e593c1b435315ff625fd071f0998c5f02792038a5ca98899261b7d", size = 185852, upload-time = "2026-02-02T12:36:42.077Z" },
+ { url = "https://files.pythonhosted.org/packages/6e/f5/f1997e987211f6f9bd71b8083047b316208b4aca0b529bb5f8c96c89ef3e/jiter-0.13.0-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:cc5223ab19fe25e2f0bf2643204ad7318896fe3729bf12fde41b77bfc4fafff0", size = 308804, upload-time = "2026-02-02T12:36:43.496Z" },
+ { url = "https://files.pythonhosted.org/packages/cd/8f/5482a7677731fd44881f0204981ce2d7175db271f82cba2085dd2212e095/jiter-0.13.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:9776ebe51713acf438fd9b4405fcd86893ae5d03487546dae7f34993217f8a91", size = 318787, upload-time = "2026-02-02T12:36:45.071Z" },
+ { url = "https://files.pythonhosted.org/packages/f3/b9/7257ac59778f1cd025b26a23c5520a36a424f7f1b068f2442a5b499b7464/jiter-0.13.0-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:879e768938e7b49b5e90b7e3fecc0dbec01b8cb89595861fb39a8967c5220d09", size = 353880, upload-time = "2026-02-02T12:36:47.365Z" },
+ { url = "https://files.pythonhosted.org/packages/c3/87/719eec4a3f0841dad99e3d3604ee4cba36af4419a76f3cb0b8e2e691ad67/jiter-0.13.0-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:682161a67adea11e3aae9038c06c8b4a9a71023228767477d683f69903ebc607", size = 366702, upload-time = "2026-02-02T12:36:48.871Z" },
+ { url = "https://files.pythonhosted.org/packages/d2/65/415f0a75cf6921e43365a1bc227c565cb949caca8b7532776e430cbaa530/jiter-0.13.0-cp314-cp314-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:a13b68cd1cd8cc9de8f244ebae18ccb3e4067ad205220ef324c39181e23bbf66", size = 486319, upload-time = "2026-02-02T12:36:53.006Z" },
+ { url = "https://files.pythonhosted.org/packages/54/a2/9e12b48e82c6bbc6081fd81abf915e1443add1b13d8fc586e1d90bb02bb8/jiter-0.13.0-cp314-cp314-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:87ce0f14c6c08892b610686ae8be350bf368467b6acd5085a5b65441e2bf36d2", size = 372289, upload-time = "2026-02-02T12:36:54.593Z" },
+ { url = "https://files.pythonhosted.org/packages/4e/c1/e4693f107a1789a239c759a432e9afc592366f04e901470c2af89cfd28e1/jiter-0.13.0-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0c365005b05505a90d1c47856420980d0237adf82f70c4aff7aebd3c1cc143ad", size = 360165, upload-time = "2026-02-02T12:36:56.112Z" },
+ { url = "https://files.pythonhosted.org/packages/17/08/91b9ea976c1c758240614bd88442681a87672eebc3d9a6dde476874e706b/jiter-0.13.0-cp314-cp314-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:1317fdffd16f5873e46ce27d0e0f7f4f90f0cdf1d86bf6abeaea9f63ca2c401d", size = 389634, upload-time = "2026-02-02T12:36:57.495Z" },
+ { url = "https://files.pythonhosted.org/packages/18/23/58325ef99390d6d40427ed6005bf1ad54f2577866594bcf13ce55675f87d/jiter-0.13.0-cp314-cp314-musllinux_1_1_aarch64.whl", hash = "sha256:c05b450d37ba0c9e21c77fef1f205f56bcee2330bddca68d344baebfc55ae0df", size = 514933, upload-time = "2026-02-02T12:36:58.909Z" },
+ { url = "https://files.pythonhosted.org/packages/5b/25/69f1120c7c395fd276c3996bb8adefa9c6b84c12bb7111e5c6ccdcd8526d/jiter-0.13.0-cp314-cp314-musllinux_1_1_x86_64.whl", hash = "sha256:775e10de3849d0631a97c603f996f518159272db00fdda0a780f81752255ee9d", size = 548842, upload-time = "2026-02-02T12:37:00.433Z" },
+ { url = "https://files.pythonhosted.org/packages/18/05/981c9669d86850c5fbb0d9e62bba144787f9fba84546ba43d624ee27ef29/jiter-0.13.0-cp314-cp314-win32.whl", hash = "sha256:632bf7c1d28421c00dd8bbb8a3bac5663e1f57d5cd5ed962bce3c73bf62608e6", size = 202108, upload-time = "2026-02-02T12:37:01.718Z" },
+ { url = "https://files.pythonhosted.org/packages/8d/96/cdcf54dd0b0341db7d25413229888a346c7130bd20820530905fdb65727b/jiter-0.13.0-cp314-cp314-win_amd64.whl", hash = "sha256:f22ef501c3f87ede88f23f9b11e608581c14f04db59b6a801f354397ae13739f", size = 204027, upload-time = "2026-02-02T12:37:03.075Z" },
+ { url = "https://files.pythonhosted.org/packages/fb/f9/724bcaaab7a3cd727031fe4f6995cb86c4bd344909177c186699c8dec51a/jiter-0.13.0-cp314-cp314-win_arm64.whl", hash = "sha256:07b75fe09a4ee8e0c606200622e571e44943f47254f95e2436c8bdcaceb36d7d", size = 187199, upload-time = "2026-02-02T12:37:04.414Z" },
+ { url = "https://files.pythonhosted.org/packages/62/92/1661d8b9fd6a3d7a2d89831db26fe3c1509a287d83ad7838831c7b7a5c7e/jiter-0.13.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:964538479359059a35fb400e769295d4b315ae61e4105396d355a12f7fef09f0", size = 318423, upload-time = "2026-02-02T12:37:05.806Z" },
+ { url = "https://files.pythonhosted.org/packages/4f/3b/f77d342a54d4ebcd128e520fc58ec2f5b30a423b0fd26acdfc0c6fef8e26/jiter-0.13.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e104da1db1c0991b3eaed391ccd650ae8d947eab1480c733e5a3fb28d4313e40", size = 351438, upload-time = "2026-02-02T12:37:07.189Z" },
+ { url = "https://files.pythonhosted.org/packages/76/b3/ba9a69f0e4209bd3331470c723c2f5509e6f0482e416b612431a5061ed71/jiter-0.13.0-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:0e3a5f0cde8ff433b8e88e41aa40131455420fb3649a3c7abdda6145f8cb7202", size = 364774, upload-time = "2026-02-02T12:37:08.579Z" },
+ { url = "https://files.pythonhosted.org/packages/b3/16/6cdb31fa342932602458dbb631bfbd47f601e03d2e4950740e0b2100b570/jiter-0.13.0-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:57aab48f40be1db920a582b30b116fe2435d184f77f0e4226f546794cedd9cf0", size = 487238, upload-time = "2026-02-02T12:37:10.066Z" },
+ { url = "https://files.pythonhosted.org/packages/ed/b1/956cc7abaca8d95c13aa8d6c9b3f3797241c246cd6e792934cc4c8b250d2/jiter-0.13.0-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:7772115877c53f62beeb8fd853cab692dbc04374ef623b30f997959a4c0e7e95", size = 372892, upload-time = "2026-02-02T12:37:11.656Z" },
+ { url = "https://files.pythonhosted.org/packages/26/c4/97ecde8b1e74f67b8598c57c6fccf6df86ea7861ed29da84629cdbba76c4/jiter-0.13.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:1211427574b17b633cfceba5040de8081e5abf114f7a7602f73d2e16f9fdaa59", size = 360309, upload-time = "2026-02-02T12:37:13.244Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/d7/eabe3cf46715854ccc80be2cd78dd4c36aedeb30751dbf85a1d08c14373c/jiter-0.13.0-cp314-cp314t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:7beae3a3d3b5212d3a55d2961db3c292e02e302feb43fce6a3f7a31b90ea6dfe", size = 389607, upload-time = "2026-02-02T12:37:14.881Z" },
+ { url = "https://files.pythonhosted.org/packages/df/2d/03963fc0804e6109b82decfb9974eb92df3797fe7222428cae12f8ccaa0c/jiter-0.13.0-cp314-cp314t-musllinux_1_1_aarch64.whl", hash = "sha256:e5562a0f0e90a6223b704163ea28e831bd3a9faa3512a711f031611e6b06c939", size = 514986, upload-time = "2026-02-02T12:37:16.326Z" },
+ { url = "https://files.pythonhosted.org/packages/f6/6c/8c83b45eb3eb1c1e18d841fe30b4b5bc5619d781267ca9bc03e005d8fd0a/jiter-0.13.0-cp314-cp314t-musllinux_1_1_x86_64.whl", hash = "sha256:6c26a424569a59140fb51160a56df13f438a2b0967365e987889186d5fc2f6f9", size = 548756, upload-time = "2026-02-02T12:37:17.736Z" },
+ { url = "https://files.pythonhosted.org/packages/47/66/eea81dfff765ed66c68fd2ed8c96245109e13c896c2a5015c7839c92367e/jiter-0.13.0-cp314-cp314t-win32.whl", hash = "sha256:24dc96eca9f84da4131cdf87a95e6ce36765c3b156fc9ae33280873b1c32d5f6", size = 201196, upload-time = "2026-02-02T12:37:19.101Z" },
+ { url = "https://files.pythonhosted.org/packages/ff/32/4ac9c7a76402f8f00d00842a7f6b83b284d0cf7c1e9d4227bc95aa6d17fa/jiter-0.13.0-cp314-cp314t-win_amd64.whl", hash = "sha256:0a8d76c7524087272c8ae913f5d9d608bd839154b62c4322ef65723d2e5bb0b8", size = 204215, upload-time = "2026-02-02T12:37:20.495Z" },
+ { url = "https://files.pythonhosted.org/packages/f9/8e/7def204fea9f9be8b3c21a6f2dd6c020cf56c7d5ff753e0e23ed7f9ea57e/jiter-0.13.0-cp314-cp314t-win_arm64.whl", hash = "sha256:2c26cf47e2cad140fa23b6d58d435a7c0161f5c514284802f25e87fddfe11024", size = 187152, upload-time = "2026-02-02T12:37:22.124Z" },
+ { url = "https://files.pythonhosted.org/packages/79/b3/3c29819a27178d0e461a8571fb63c6ae38be6dc36b78b3ec2876bbd6a910/jiter-0.13.0-graalpy311-graalpy242_311_native-macosx_10_12_x86_64.whl", hash = "sha256:b1cbfa133241d0e6bdab48dcdc2604e8ba81512f6bbd68ec3e8e1357dd3c316c", size = 307016, upload-time = "2026-02-02T12:37:42.755Z" },
+ { url = "https://files.pythonhosted.org/packages/eb/ae/60993e4b07b1ac5ebe46da7aa99fdbb802eb986c38d26e3883ac0125c4e0/jiter-0.13.0-graalpy311-graalpy242_311_native-macosx_11_0_arm64.whl", hash = "sha256:db367d8be9fad6e8ebbac4a7578b7af562e506211036cba2c06c3b998603c3d2", size = 305024, upload-time = "2026-02-02T12:37:44.774Z" },
+ { url = "https://files.pythonhosted.org/packages/77/fa/2227e590e9cf98803db2811f172b2d6460a21539ab73006f251c66f44b14/jiter-0.13.0-graalpy311-graalpy242_311_native-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:45f6f8efb2f3b0603092401dc2df79fa89ccbc027aaba4174d2d4133ed661434", size = 339337, upload-time = "2026-02-02T12:37:46.668Z" },
+ { url = "https://files.pythonhosted.org/packages/2d/92/015173281f7eb96c0ef580c997da8ef50870d4f7f4c9e03c845a1d62ae04/jiter-0.13.0-graalpy311-graalpy242_311_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:597245258e6ad085d064780abfb23a284d418d3e61c57362d9449c6c7317ee2d", size = 346395, upload-time = "2026-02-02T12:37:48.09Z" },
+ { url = "https://files.pythonhosted.org/packages/80/60/e50fa45dd7e2eae049f0ce964663849e897300433921198aef94b6ffa23a/jiter-0.13.0-graalpy312-graalpy250_312_native-macosx_10_12_x86_64.whl", hash = "sha256:3d744a6061afba08dd7ae375dcde870cffb14429b7477e10f67e9e6d68772a0a", size = 305169, upload-time = "2026-02-02T12:37:50.376Z" },
+ { url = "https://files.pythonhosted.org/packages/d2/73/a009f41c5eed71c49bec53036c4b33555afcdee70682a18c6f66e396c039/jiter-0.13.0-graalpy312-graalpy250_312_native-macosx_11_0_arm64.whl", hash = "sha256:ff732bd0a0e778f43d5009840f20b935e79087b4dc65bd36f1cd0f9b04b8ff7f", size = 303808, upload-time = "2026-02-02T12:37:52.092Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/10/528b439290763bff3d939268085d03382471b442f212dca4ff5f12802d43/jiter-0.13.0-graalpy312-graalpy250_312_native-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ab44b178f7981fcaea7e0a5df20e773c663d06ffda0198f1a524e91b2fde7e59", size = 337384, upload-time = "2026-02-02T12:37:53.582Z" },
+ { url = "https://files.pythonhosted.org/packages/67/8a/a342b2f0251f3dac4ca17618265d93bf244a2a4d089126e81e4c1056ac50/jiter-0.13.0-graalpy312-graalpy250_312_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7bb00b6d26db67a05fe3e12c76edc75f32077fb51deed13822dc648fa373bc19", size = 343768, upload-time = "2026-02-02T12:37:55.055Z" },
+]
+
[[package]]
name = "joblib"
version = "1.5.3"
@@ -1367,6 +1609,61 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/0c/70/05b685ea2dffcb2adbf3cdcea5d8865b7bc66f67249084cf845012a0ff13/kubernetes-35.0.0-py2.py3-none-any.whl", hash = "sha256:39e2b33b46e5834ef6c3985ebfe2047ab39135d41de51ce7641a7ca5b372a13d", size = 2017602, upload-time = "2026-01-16T01:05:25.991Z" },
]
+[[package]]
+name = "langchain"
+version = "1.2.14"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "langchain-core" },
+ { name = "langgraph" },
+ { name = "pydantic" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/af/2b/0ca77ee988a9f1c1f1d923115d7c91221ab434067bc36f2f637201aeee81/langchain-1.2.14.tar.gz", hash = "sha256:fc5511e8f8af7efee9e5a144da4392d700d627b301d240470db97272940ad317", size = 574190, upload-time = "2026-03-31T13:50:37.398Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/4c/87/324ae5fd9993f024339a452fc89e3fd808bccde87ef95c8dafab3de023c0/langchain-1.2.14-py3-none-any.whl", hash = "sha256:96da6d7338d5a6fc41eb4ec0db83f7ef5d03bb5efd17bb269f34ba4378ebdb4d", size = 112715, upload-time = "2026-03-31T13:50:35.997Z" },
+]
+
+[[package]]
+name = "langchain-classic"
+version = "1.0.3"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "langchain-core" },
+ { name = "langchain-text-splitters" },
+ { name = "langsmith" },
+ { name = "pydantic" },
+ { name = "pyyaml" },
+ { name = "requests" },
+ { name = "sqlalchemy" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/32/04/b01c09e37414bab9f209efa311502841a3c0de5bc6c35e729c8d8a9893c9/langchain_classic-1.0.3.tar.gz", hash = "sha256:168ef1dfbfb18cae5a9ff0accecc9413a5b5aa3464b53fa841561a3384b6324a", size = 10534933, upload-time = "2026-03-13T13:56:11.96Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ab/e6/cfdeedec0537ffbf5041773590d25beb7f2aa467cc6630e788c9c7c72c3e/langchain_classic-1.0.3-py3-none-any.whl", hash = "sha256:26df1ec9806b1fbff19d9085a747ea7d8d82d7e3fb1d25132859979de627ef79", size = 1041335, upload-time = "2026-03-13T13:56:09.677Z" },
+]
+
+[[package]]
+name = "langchain-community"
+version = "0.4.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "aiohttp" },
+ { name = "dataclasses-json" },
+ { name = "httpx-sse" },
+ { name = "langchain-classic" },
+ { name = "langchain-core" },
+ { name = "langsmith" },
+ { name = "numpy" },
+ { name = "pydantic-settings" },
+ { name = "pyyaml" },
+ { name = "requests" },
+ { name = "sqlalchemy" },
+ { name = "tenacity" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/53/97/a03585d42b9bdb6fbd935282d6e3348b10322a24e6ce12d0c99eb461d9af/langchain_community-0.4.1.tar.gz", hash = "sha256:f3b211832728ee89f169ddce8579b80a085222ddb4f4ed445a46e977d17b1e85", size = 33241144, upload-time = "2025-10-27T15:20:32.504Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f0/a4/c4fde67f193401512337456cabc2148f2c43316e445f5decd9f8806e2992/langchain_community-0.4.1-py3-none-any.whl", hash = "sha256:2135abb2c7748a35c84613108f7ebf30f8505b18c3c18305ffaecfc7651f6c6a", size = 2533285, upload-time = "2025-10-27T15:20:30.767Z" },
+]
+
[[package]]
name = "langchain-core"
version = "1.2.22"
@@ -1401,6 +1698,32 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/ec/7e/46c5973bd8b10a5c4c8a77136cf536e658796380a17c740246074901b038/langchain_google_genai-4.2.1-py3-none-any.whl", hash = "sha256:a7735289cf94ca3a684d830e09196aac8f6e75e647e3a0a1c3c9dc534ceb985e", size = 66500, upload-time = "2026-02-19T19:29:18.002Z" },
]
+[[package]]
+name = "langchain-openai"
+version = "1.1.12"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "langchain-core" },
+ { name = "openai" },
+ { name = "tiktoken" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/cc/fd/7dee16e882c4c1577d48db174d85aa3a0ee09ba61eb6a5d41650285ca80c/langchain_openai-1.1.12.tar.gz", hash = "sha256:ccf5ef02c896f6807b4d0e51aaf678a72ce81ae41201cae8d65e11eeff9ecb79", size = 1114119, upload-time = "2026-03-23T18:59:19.211Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/6e/a6/68fb22e3604015e6f546fa1d3677d24378b482855ae74710cbf4aec44132/langchain_openai-1.1.12-py3-none-any.whl", hash = "sha256:da71ca3f2d18c16f7a2443cc306aa195ad2a07054335ac9b0626dcae02b6a0c5", size = 88487, upload-time = "2026-03-23T18:59:17.978Z" },
+]
+
+[[package]]
+name = "langchain-text-splitters"
+version = "1.1.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "langchain-core" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/85/38/14121ead61e0e75f79c3a35e5148ac7c2fe754a55f76eab3eed573269524/langchain_text_splitters-1.1.1.tar.gz", hash = "sha256:34861abe7c07d9e49d4dc852d0129e26b32738b60a74486853ec9b6d6a8e01d2", size = 279352, upload-time = "2026-02-18T23:02:42.798Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/84/66/d9e0c3b83b0ad75ee746c51ba347cacecb8d656b96e1d513f3e334d1ccab/langchain_text_splitters-1.1.1-py3-none-any.whl", hash = "sha256:5ed0d7bf314ba925041e7d7d17cd8b10f688300d5415fb26c29442f061e329dc", size = 35734, upload-time = "2026-02-18T23:02:41.913Z" },
+]
+
[[package]]
name = "langfuse"
version = "4.3.1"
@@ -1668,6 +1991,18 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/70/bc/6f1c2f612465f5fa89b95bead1f44dcb607670fd42891d8fdcd5d039f4f4/markupsafe-3.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:32001d6a8fc98c8cb5c947787c5d08b0a50663d139f1305bac5885d98d9b40fa", size = 14146, upload-time = "2025-09-27T18:37:28.327Z" },
]
+[[package]]
+name = "marshmallow"
+version = "3.26.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "packaging" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/55/79/de6c16cc902f4fc372236926b0ce2ab7845268dcc30fb2fbb7f71b418631/marshmallow-3.26.2.tar.gz", hash = "sha256:bbe2adb5a03e6e3571b573f42527c6fe926e17467833660bebd11593ab8dfd57", size = 222095, upload-time = "2025-12-22T06:53:53.309Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/be/2f/5108cb3ee4ba6501748c4908b908e55f42a5b66245b4cfe0c99326e1ef6e/marshmallow-3.26.2-py3-none-any.whl", hash = "sha256:013fa8a3c4c276c24d26d84ce934dc964e2aa794345a0f8c7e5a7191482c8a73", size = 50964, upload-time = "2025-12-22T06:53:51.801Z" },
+]
+
[[package]]
name = "matplotlib-inline"
version = "0.2.1"
@@ -1911,6 +2246,26 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/81/08/7036c080d7117f28a4af526d794aab6a84463126db031b007717c1a6676e/multidict-6.7.1-py3-none-any.whl", hash = "sha256:55d97cc6dae627efa6a6e548885712d4864b81110ac76fa4e534c03819fa4a56", size = 12319, upload-time = "2026-01-26T02:46:44.004Z" },
]
+[[package]]
+name = "multiprocess"
+version = "0.70.19"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "dill" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/a2/f2/e783ac7f2aeeed14e9e12801f22529cc7e6b7ab80928d6dcce4e9f00922d/multiprocess-0.70.19.tar.gz", hash = "sha256:952021e0e6c55a4a9fe4cd787895b86e239a40e76802a789d6305398d3975897", size = 2079989, upload-time = "2026-01-19T06:47:39.744Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/7e/aa/714635c727dbfc251139226fa4eaf1b07f00dc12d9cd2eb25f931adaf873/multiprocess-0.70.19-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:1bbf1b69af1cf64cd05f65337d9215b88079ec819cd0ea7bac4dab84e162efe7", size = 144743, upload-time = "2026-01-19T06:47:24.562Z" },
+ { url = "https://files.pythonhosted.org/packages/0f/e1/155f6abf5e6b5d9cef29b6d0167c180846157a4aca9b9bee1a217f67c959/multiprocess-0.70.19-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:5be9ec7f0c1c49a4f4a6fd20d5dda4aeabc2d39a50f4ad53720f1cd02b3a7c2e", size = 144738, upload-time = "2026-01-19T06:47:26.636Z" },
+ { url = "https://files.pythonhosted.org/packages/af/cb/f421c2869d75750a4f32301cc20c4b63fab6376e9a75c8e5e655bdeb3d9b/multiprocess-0.70.19-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:1c3dce098845a0db43b32a0b76a228ca059a668071cfeaa0f40c36c0b1585d45", size = 144741, upload-time = "2026-01-19T06:47:27.985Z" },
+ { url = "https://files.pythonhosted.org/packages/e3/45/8004d1e6b9185c1a444d6b55ac5682acf9d98035e54386d967366035a03a/multiprocess-0.70.19-py310-none-any.whl", hash = "sha256:97404393419dcb2a8385910864eedf47a3cadf82c66345b44f036420eb0b5d87", size = 134948, upload-time = "2026-01-19T06:47:32.325Z" },
+ { url = "https://files.pythonhosted.org/packages/86/c2/dec9722dc3474c164a0b6bcd9a7ed7da542c98af8cabce05374abab35edd/multiprocess-0.70.19-py311-none-any.whl", hash = "sha256:928851ae7973aea4ce0eaf330bbdafb2e01398a91518d5c8818802845564f45c", size = 144457, upload-time = "2026-01-19T06:47:33.711Z" },
+ { url = "https://files.pythonhosted.org/packages/71/70/38998b950a97ea279e6bd657575d22d1a2047256caf707d9a10fbce4f065/multiprocess-0.70.19-py312-none-any.whl", hash = "sha256:3a56c0e85dd5025161bac5ce138dcac1e49174c7d8e74596537e729fd5c53c28", size = 150281, upload-time = "2026-01-19T06:47:35.037Z" },
+ { url = "https://files.pythonhosted.org/packages/7f/74/d2c27e03cb84251dfe7249b8e82923643c6d48fa4883b9476b025e7dc7eb/multiprocess-0.70.19-py313-none-any.whl", hash = "sha256:8d5eb4ec5017ba2fab4e34a747c6d2c2b6fecfe9e7236e77988db91580ada952", size = 156414, upload-time = "2026-01-19T06:47:35.915Z" },
+ { url = "https://files.pythonhosted.org/packages/a0/61/af9115673a5870fd885247e2f1b68c4f1197737da315b520a91c757a861a/multiprocess-0.70.19-py314-none-any.whl", hash = "sha256:e8cc7fbdff15c0613f0a1f1f8744bef961b0a164c0ca29bdff53e9d2d93c5e5f", size = 160318, upload-time = "2026-01-19T06:47:37.497Z" },
+ { url = "https://files.pythonhosted.org/packages/7e/82/69e539c4c2027f1e1697e09aaa2449243085a0edf81ae2c6341e84d769b6/multiprocess-0.70.19-py39-none-any.whl", hash = "sha256:0d4b4397ed669d371c81dcd1ef33fd384a44d6c3de1bd0ca7ac06d837720d3c5", size = 133477, upload-time = "2026-01-19T06:47:38.619Z" },
+]
+
[[package]]
name = "mypy"
version = "1.19.1"
@@ -1980,6 +2335,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/70/5c/ee71e2dd955045425ef44283f40ba1da67673cf06404916ca2950ac0cd39/neo4j-6.1.0-py3-none-any.whl", hash = "sha256:3bd93941f3a3559af197031157220af9fd71f4f93a311db687bd69ffa417b67d", size = 325326, upload-time = "2026-01-12T11:27:33.196Z" },
]
+[[package]]
+name = "nest-asyncio"
+version = "1.6.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/83/f8/51569ac65d696c8ecbee95938f89d4abf00f47d58d48f6fbabfe8f0baefe/nest_asyncio-1.6.0.tar.gz", hash = "sha256:6f172d5449aca15afd6c646851f4e31e02c598d553a667e38cafa997cfec55fe", size = 7418, upload-time = "2024-01-21T14:25:19.227Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a0/c4/c2971a3ba4c6103a3d10c4b0f24f461ddc027f0f09763220cf35ca1401b3/nest_asyncio-1.6.0-py3-none-any.whl", hash = "sha256:87af6efd6b5e897c81050477ef65c62e2b2f35d51703cae01aff2905b1852e1c", size = 5195, upload-time = "2024-01-21T14:25:17.223Z" },
+]
+
[[package]]
name = "networkx"
version = "3.6.1"
@@ -2271,6 +2635,25 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/3a/27/ecdd3ae7d49d9f54820ededce2d88ddc3333b9ac9bb5f1d0d6aa3148c686/onnxruntime-1.24.2-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:05a2792b5ef9278a89415a1f39d0a22192a872168257100503a5157165a38e7b", size = 17117770, upload-time = "2026-02-19T17:14:20.048Z" },
]
+[[package]]
+name = "openai"
+version = "2.32.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "anyio" },
+ { name = "distro" },
+ { name = "httpx" },
+ { name = "jiter" },
+ { name = "pydantic" },
+ { name = "sniffio" },
+ { name = "tqdm" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/ed/59/bdcc6b759b8c42dd73afaf5bf8f902c04b37987a5514dbc1c64dba390fef/openai-2.32.0.tar.gz", hash = "sha256:c54b27a9e4cb8d51f0dd94972ffd1a04437efeb259a9e60d8922b8bd26fe55e0", size = 693286, upload-time = "2026-04-15T22:28:19.434Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/1e/c1/d6e64ccd0536bf616556f0cad2b6d94a8125f508d25cfd814b1d2db4e2f1/openai-2.32.0-py3-none-any.whl", hash = "sha256:4dcc9badeb4bf54ad0d187453742f290226d30150890b7890711bda4f32f192f", size = 1162570, upload-time = "2026-04-15T22:28:17.714Z" },
+]
+
[[package]]
name = "opentelemetry-api"
version = "1.39.1"
@@ -2595,6 +2978,7 @@ version = "0.1.0"
source = { editable = "." }
dependencies = [
{ name = "chromadb" },
+ { name = "datasets" },
{ name = "fastapi" },
{ name = "google-genai" },
{ name = "httpx" },
@@ -2612,6 +2996,7 @@ dependencies = [
{ name = "pydantic-settings" },
{ name = "python-dotenv" },
{ name = "pyvis" },
+ { name = "ragas" },
{ name = "rapidfuzz" },
{ name = "requests" },
{ name = "sentence-transformers" },
@@ -2634,6 +3019,7 @@ dev = [
[package.metadata]
requires-dist = [
{ name = "chromadb", specifier = ">=0.6" },
+ { name = "datasets", specifier = ">=4.8.4" },
{ name = "fastapi", specifier = ">=0.115" },
{ name = "google-genai", specifier = ">=1.64.0" },
{ name = "httpx", specifier = ">=0.28" },
@@ -2656,6 +3042,7 @@ requires-dist = [
{ name = "pytest-cov", marker = "extra == 'dev'", specifier = ">=6.0" },
{ name = "python-dotenv", specifier = ">=1.0" },
{ name = "pyvis", specifier = ">=0.3" },
+ { name = "ragas", specifier = ">=0.4.3" },
{ name = "rapidfuzz", specifier = ">=3.14.3" },
{ name = "requests", specifier = ">=2.32" },
{ name = "ruff", marker = "extra == 'dev'", specifier = ">=0.9" },
@@ -3505,6 +3892,36 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/f1/12/de94a39c2ef588c7e6455cfbe7343d3b2dc9d6b6b2f40c4c6565744c873d/pyyaml-6.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:ebc55a14a21cb14062aa4162f906cd962b28e2e9ea38f9b4391244cd8de4ae0b", size = 149341, upload-time = "2025-09-25T21:32:56.828Z" },
]
+[[package]]
+name = "ragas"
+version = "0.4.3"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "appdirs" },
+ { name = "datasets" },
+ { name = "diskcache" },
+ { name = "instructor" },
+ { name = "langchain" },
+ { name = "langchain-community" },
+ { name = "langchain-core" },
+ { name = "langchain-openai" },
+ { name = "nest-asyncio" },
+ { name = "networkx" },
+ { name = "numpy" },
+ { name = "openai" },
+ { name = "pillow" },
+ { name = "pydantic" },
+ { name = "rich" },
+ { name = "scikit-network" },
+ { name = "tiktoken" },
+ { name = "tqdm" },
+ { name = "typer" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/d2/bc/3234517692ac0ffae1ec2ec940992e4057844c49ee6c51c07ce385bb98f1/ragas-0.4.3.tar.gz", hash = "sha256:1eb1f61dbc8613ad014fdb8d630cbe9a1caec1ea01664a106993cb756128c001", size = 44029626, upload-time = "2026-01-13T17:48:01.043Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/4d/e0/1fecd22c93d3ed66453cbbdefd05528331af4d33b2b76a370d751231912c/ragas-0.4.3-py3-none-any.whl", hash = "sha256:ef1d75f674c294e9a6e7d8e9ad261b6bf4697dad1c9cbd1a756ba7a6b4849a38", size = 466452, upload-time = "2026-01-13T17:47:59.2Z" },
+]
+
[[package]]
name = "rapidfuzz"
version = "3.14.3"
@@ -3984,6 +4401,33 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/60/22/d7b2ebe4704a5e50790ba089d5c2ae308ab6bb852719e6c3bd4f04c3a363/scikit_learn-1.8.0-cp314-cp314t-win_arm64.whl", hash = "sha256:f28dd15c6bb0b66ba09728cf09fd8736c304be29409bd8445a080c1280619e8c", size = 8002647, upload-time = "2025-12-10T07:08:51.601Z" },
]
+[[package]]
+name = "scikit-network"
+version = "0.33.5"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "numpy" },
+ { name = "scipy" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/46/6d/28b00fbef9ff7d8ba31861bf16705a1a74a1696fb65aab2a7c584f966bec/scikit_network-0.33.5.tar.gz", hash = "sha256:ae2149d9a280fdc4bbadd5f8a7b17c8af61c054bc3f834792bc61483e6783c12", size = 1784205, upload-time = "2025-11-19T09:45:14.402Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d4/ed/4a14f48ae7eceeb4bc2085c9467e7dda487b8728261a2556a3d2fdc99b0e/scikit_network-0.33.5-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:38a5a55f125b9ff574b085994e6374f783469faf1542d140c1630ad2c14127b9", size = 2854152, upload-time = "2025-11-19T09:44:43.049Z" },
+ { url = "https://files.pythonhosted.org/packages/30/f9/71e225866bedaf6256ba3cd720c5bdbbe4828df7574d3c9cb741789b0f85/scikit_network-0.33.5-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:6d796382dd914ccaa7d3fb990e148f5f095817690f07119d614f4d5bd63b347e", size = 2840146, upload-time = "2025-11-19T09:44:45.04Z" },
+ { url = "https://files.pythonhosted.org/packages/a0/ac/ed64fbc2a21074aa399d3e527695f4d4bb35be3346b5e09edf0637d58080/scikit_network-0.33.5-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:406af38a07ee1d631b62616496013fc5d941fffb5221fdb9e9091b87352a3f0c", size = 8005762, upload-time = "2025-11-19T09:44:47.186Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/8e/9631ea79d6144d746e9a73e79d392656ae0c1d2989b8183f4f13134ad253/scikit_network-0.33.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6d001988f07abe03ef4d8189da148968ad520f8cb8666d9ecee0d51c277bd466", size = 8061688, upload-time = "2025-11-19T09:44:49.034Z" },
+ { url = "https://files.pythonhosted.org/packages/05/d2/26d21c25245204ef0ac6dfd047e8b6181bdc40d486dd9daadaadfed1c0e6/scikit_network-0.33.5-cp311-cp311-win_amd64.whl", hash = "sha256:8b9338feb0b2bae0f32ddd77453125b4e6c2d365cfb1bb1334b016888466e42f", size = 2738782, upload-time = "2025-11-19T09:44:50.881Z" },
+ { url = "https://files.pythonhosted.org/packages/72/32/f092fca9a3ae256e0608ec6a4d8830023ea4ae478c2e27e94cf5802824f0/scikit_network-0.33.5-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:ced625228be1632595d11adaf22d61d1d4c788909cd4d8c364e720b2814aac7f", size = 2874698, upload-time = "2025-11-19T09:44:52.765Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/c7/5b2ce72f93b422af48a2b755fbcbab271bf980a6b46484f754d63978f1ff/scikit_network-0.33.5-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:808b625d28005c24b47cbdf65f780a3a355aa4080992e6b78f03434e873d06e6", size = 2854224, upload-time = "2025-11-19T09:44:55.574Z" },
+ { url = "https://files.pythonhosted.org/packages/86/02/974ae67f493ccf988108894e8a9dedfd00ca5113d2848e0b9fc2a4d18824/scikit_network-0.33.5-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f9f2d98059cd79bdb935ff6a638f1c3a12b0b1cb7ace9e1d3fb35476eeeaabaa", size = 7924650, upload-time = "2025-11-19T09:44:57.704Z" },
+ { url = "https://files.pythonhosted.org/packages/8d/34/b67e48e111916a6f09fe29a971a9716c14f78525e0ab7c46e6a6538cf2f6/scikit_network-0.33.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:aac2d3bc14214f02dac300624f5ec2650af9a98b52f304d300f0ec2813a0e544", size = 8012322, upload-time = "2025-11-19T09:45:00.079Z" },
+ { url = "https://files.pythonhosted.org/packages/c3/a2/49293b53b837b3248d19fffd41a9fd73dcb2eeabbab890cc4c8aa237b545/scikit_network-0.33.5-cp312-cp312-win_amd64.whl", hash = "sha256:2866b16aed9ef25ba42cb2f2e44ef2ad079337f336ce48d0604b55fa4af87688", size = 2746491, upload-time = "2025-11-19T09:45:01.997Z" },
+ { url = "https://files.pythonhosted.org/packages/d6/cd/0069244e970d27fa0ab0512394295a106605f00c271e85618182460d2c92/scikit_network-0.33.5-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:aa8b490a777e081dd6f69627a26b50cc4012f27fff683a1e4828819a88a5dcf2", size = 2862564, upload-time = "2025-11-19T09:45:03.93Z" },
+ { url = "https://files.pythonhosted.org/packages/51/37/85454864e50a65e528fe3b15eb3b41eb68b0c7f6d5c51c220b3198622ede/scikit_network-0.33.5-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:3615d073ba9ae1ae30dda2de747474cd23c86cededa82b317471ee9f9bebd1b2", size = 2842521, upload-time = "2025-11-19T09:45:06.31Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/b9/4023f35e430b51020f17b0f1d8933768cf1ed7cd1623cc089f5543048983/scikit_network-0.33.5-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:2408d3f4c81256a3193d536aad4a6ffcfbb05d096abe6a9cc0b6b5e275df876d", size = 7871695, upload-time = "2025-11-19T09:45:08.101Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/ec/e50755f7459130ba745c42c37665c5ae9a7c7e357f43400b5b8b966f902e/scikit_network-0.33.5-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:526490a1e0e8e49ad0f4cca193f581d60082a2fa8c9a825eb0b6936050b0d02b", size = 7968762, upload-time = "2025-11-19T09:45:10.347Z" },
+ { url = "https://files.pythonhosted.org/packages/45/2a/616974a0adb9d04a791570e9371caaef14c54f8806b04dd59c80a7b60289/scikit_network-0.33.5-cp313-cp313-win_amd64.whl", hash = "sha256:722c15fcede5e07ac008354bbd6ef375e0f5bf1fd52bd40271775997be2fb715", size = 2742482, upload-time = "2025-11-19T09:45:12.843Z" },
+]
+
[[package]]
name = "scipy"
version = "1.17.1"
@@ -4119,6 +4563,59 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/e9/44/75a9c9421471a6c4805dbf2356f7c181a29c1879239abab1ea2cc8f38b40/sniffio-1.3.1-py3-none-any.whl", hash = "sha256:2f6da418d1f1e0fddd844478f41680e794e6051915791a034ff65e5f100525a2", size = 10235, upload-time = "2024-02-25T23:20:01.196Z" },
]
+[[package]]
+name = "sqlalchemy"
+version = "2.0.49"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "greenlet", marker = "platform_machine == 'AMD64' or platform_machine == 'WIN32' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'ppc64le' or platform_machine == 'win32' or platform_machine == 'x86_64'" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/09/45/461788f35e0364a8da7bda51a1fe1b09762d0c32f12f63727998d85a873b/sqlalchemy-2.0.49.tar.gz", hash = "sha256:d15950a57a210e36dd4cec1aac22787e2a4d57ba9318233e2ef8b2daf9ff2d5f", size = 9898221, upload-time = "2026-04-03T16:38:11.704Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/60/b5/e3617cc67420f8f403efebd7b043128f94775e57e5b84e7255203390ceae/sqlalchemy-2.0.49-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:c5070135e1b7409c4161133aa525419b0062088ed77c92b1da95366ec5cbebbe", size = 2159126, upload-time = "2026-04-03T16:50:13.242Z" },
+ { url = "https://files.pythonhosted.org/packages/20/9b/91ca80403b17cd389622a642699e5f6564096b698e7cdcbcbb6409898bc4/sqlalchemy-2.0.49-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9ac7a3e245fd0310fd31495eb61af772e637bdf7d88ee81e7f10a3f271bff014", size = 3315509, upload-time = "2026-04-03T16:54:49.332Z" },
+ { url = "https://files.pythonhosted.org/packages/b1/61/0722511d98c54de95acb327824cb759e8653789af2b1944ab1cc69d32565/sqlalchemy-2.0.49-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4d4e5a0ceba319942fa6b585cf82539288a61e314ef006c1209f734551ab9536", size = 3315014, upload-time = "2026-04-03T16:56:56.376Z" },
+ { url = "https://files.pythonhosted.org/packages/46/55/d514a653ffeb4cebf4b54c47bec32ee28ad89d39fafba16eeed1d81dccd5/sqlalchemy-2.0.49-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:3ddcb27fb39171de36e207600116ac9dfd4ae46f86c82a9bf3934043e80ebb88", size = 3267388, upload-time = "2026-04-03T16:54:51.272Z" },
+ { url = "https://files.pythonhosted.org/packages/2f/16/0dcc56cb6d3335c1671a2258f5d2cb8267c9a2260e27fde53cbfb1b3540a/sqlalchemy-2.0.49-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:32fe6a41ad97302db2931f05bb91abbcc65b5ce4c675cd44b972428dd2947700", size = 3289602, upload-time = "2026-04-03T16:56:57.63Z" },
+ { url = "https://files.pythonhosted.org/packages/51/6c/f8ab6fb04470a133cd80608db40aa292e6bae5f162c3a3d4ab19544a67af/sqlalchemy-2.0.49-cp311-cp311-win32.whl", hash = "sha256:46d51518d53edfbe0563662c96954dc8fcace9832332b914375f45a99b77cc9a", size = 2119044, upload-time = "2026-04-03T17:00:53.455Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/59/55a6d627d04b6ebb290693681d7683c7da001eddf90b60cfcc41ee907978/sqlalchemy-2.0.49-cp311-cp311-win_amd64.whl", hash = "sha256:951d4a210744813be63019f3df343bf233b7432aadf0db54c75802247330d3af", size = 2143642, upload-time = "2026-04-03T17:00:54.769Z" },
+ { url = "https://files.pythonhosted.org/packages/49/b3/2de412451330756aaaa72d27131db6dde23995efe62c941184e15242a5fa/sqlalchemy-2.0.49-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:4bbccb45260e4ff1b7db0be80a9025bb1e6698bdb808b83fff0000f7a90b2c0b", size = 2157681, upload-time = "2026-04-03T16:53:07.132Z" },
+ { url = "https://files.pythonhosted.org/packages/50/84/b2a56e2105bd11ebf9f0b93abddd748e1a78d592819099359aa98134a8bf/sqlalchemy-2.0.49-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:fb37f15714ec2652d574f021d479e78cd4eb9d04396dca36568fdfffb3487982", size = 3338976, upload-time = "2026-04-03T17:07:40Z" },
+ { url = "https://files.pythonhosted.org/packages/2c/fa/65fcae2ed62f84ab72cf89536c7c3217a156e71a2c111b1305ab6f0690e2/sqlalchemy-2.0.49-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3bb9ec6436a820a4c006aad1ac351f12de2f2dbdaad171692ee457a02429b672", size = 3351937, upload-time = "2026-04-03T17:12:23.374Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/2f/6fd118563572a7fe475925742eb6b3443b2250e346a0cc27d8d408e73773/sqlalchemy-2.0.49-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:8d6efc136f44a7e8bc8088507eaabbb8c2b55b3dbb63fe102c690da0ddebe55e", size = 3281646, upload-time = "2026-04-03T17:07:41.949Z" },
+ { url = "https://files.pythonhosted.org/packages/c5/d7/410f4a007c65275b9cf82354adb4bb8ba587b176d0a6ee99caa16fe638f8/sqlalchemy-2.0.49-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:e06e617e3d4fd9e51d385dfe45b077a41e9d1b033a7702551e3278ac597dc750", size = 3316695, upload-time = "2026-04-03T17:12:25.642Z" },
+ { url = "https://files.pythonhosted.org/packages/d9/95/81f594aa60ded13273a844539041ccf1e66c5a7bed0a8e27810a3b52d522/sqlalchemy-2.0.49-cp312-cp312-win32.whl", hash = "sha256:83101a6930332b87653886c01d1ee7e294b1fe46a07dd9a2d2b4f91bcc88eec0", size = 2117483, upload-time = "2026-04-03T17:05:40.896Z" },
+ { url = "https://files.pythonhosted.org/packages/47/9e/fd90114059175cac64e4fafa9bf3ac20584384d66de40793ae2e2f26f3bb/sqlalchemy-2.0.49-cp312-cp312-win_amd64.whl", hash = "sha256:618a308215b6cececb6240b9abde545e3acdabac7ae3e1d4e666896bf5ba44b4", size = 2144494, upload-time = "2026-04-03T17:05:42.282Z" },
+ { url = "https://files.pythonhosted.org/packages/ae/81/81755f50eb2478eaf2049728491d4ea4f416c1eb013338682173259efa09/sqlalchemy-2.0.49-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:df2d441bacf97022e81ad047e1597552eb3f83ca8a8f1a1fdd43cd7fe3898120", size = 2154547, upload-time = "2026-04-03T16:53:08.64Z" },
+ { url = "https://files.pythonhosted.org/packages/a2/bc/3494270da80811d08bcfa247404292428c4fe16294932bce5593f215cad9/sqlalchemy-2.0.49-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8e20e511dc15265fb433571391ba313e10dd8ea7e509d51686a51313b4ac01a2", size = 3280782, upload-time = "2026-04-03T17:07:43.508Z" },
+ { url = "https://files.pythonhosted.org/packages/cd/f5/038741f5e747a5f6ea3e72487211579d8cbea5eb9827a9cbd61d0108c4bd/sqlalchemy-2.0.49-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:47604cb2159f8bbd5a1ab48a714557156320f20871ee64d550d8bf2683d980d3", size = 3297156, upload-time = "2026-04-03T17:12:27.697Z" },
+ { url = "https://files.pythonhosted.org/packages/88/50/a6af0ff9dc954b43a65ca9b5367334e45d99684c90a3d3413fc19a02d43c/sqlalchemy-2.0.49-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:22d8798819f86720bc646ab015baff5ea4c971d68121cb36e2ebc2ee43ead2b7", size = 3228832, upload-time = "2026-04-03T17:07:45.38Z" },
+ { url = "https://files.pythonhosted.org/packages/bc/d1/5f6bdad8de0bf546fc74370939621396515e0cdb9067402d6ba1b8afbe9a/sqlalchemy-2.0.49-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:9b1c058c171b739e7c330760044803099c7fff11511e3ab3573e5327116a9c33", size = 3267000, upload-time = "2026-04-03T17:12:29.657Z" },
+ { url = "https://files.pythonhosted.org/packages/f7/30/ad62227b4a9819a5e1c6abff77c0f614fa7c9326e5a3bdbee90f7139382b/sqlalchemy-2.0.49-cp313-cp313-win32.whl", hash = "sha256:a143af2ea6672f2af3f44ed8f9cd020e9cc34c56f0e8db12019d5d9ecf41cb3b", size = 2115641, upload-time = "2026-04-03T17:05:43.989Z" },
+ { url = "https://files.pythonhosted.org/packages/17/3a/7215b1b7d6d49dc9a87211be44562077f5f04f9bb5a59552c1c8e2d98173/sqlalchemy-2.0.49-cp313-cp313-win_amd64.whl", hash = "sha256:12b04d1db2663b421fe072d638a138460a51d5a862403295671c4f3987fb9148", size = 2141498, upload-time = "2026-04-03T17:05:45.7Z" },
+ { url = "https://files.pythonhosted.org/packages/28/4b/52a0cb2687a9cd1648252bb257be5a1ba2c2ded20ba695c65756a55a15a4/sqlalchemy-2.0.49-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:24bd94bb301ec672d8f0623eba9226cc90d775d25a0c92b5f8e4965d7f3a1518", size = 3560807, upload-time = "2026-04-03T16:58:31.666Z" },
+ { url = "https://files.pythonhosted.org/packages/8c/d8/fda95459204877eed0458550d6c7c64c98cc50c2d8d618026737de9ed41a/sqlalchemy-2.0.49-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a51d3db74ba489266ef55c7a4534eb0b8db9a326553df481c11e5d7660c8364d", size = 3527481, upload-time = "2026-04-03T17:06:00.155Z" },
+ { url = "https://files.pythonhosted.org/packages/ff/0a/2aac8b78ac6487240cf7afef8f203ca783e8796002dc0cf65c4ee99ff8bb/sqlalchemy-2.0.49-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:55250fe61d6ebfd6934a272ee16ef1244e0f16b7af6cd18ab5b1fc9f08631db0", size = 3468565, upload-time = "2026-04-03T16:58:33.414Z" },
+ { url = "https://files.pythonhosted.org/packages/a5/3d/ce71cfa82c50a373fd2148b3c870be05027155ce791dc9a5dcf439790b8b/sqlalchemy-2.0.49-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:46796877b47034b559a593d7e4b549aba151dae73f9e78212a3478161c12ab08", size = 3477769, upload-time = "2026-04-03T17:06:02.787Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/e8/0a9f5c1f7c6f9ca480319bf57c2d7423f08d31445974167a27d14483c948/sqlalchemy-2.0.49-cp313-cp313t-win32.whl", hash = "sha256:9c4969a86e41454f2858256c39bdfb966a20961e9b58bf8749b65abf447e9a8d", size = 2143319, upload-time = "2026-04-03T17:02:04.328Z" },
+ { url = "https://files.pythonhosted.org/packages/0e/51/fb5240729fbec73006e137c4f7a7918ffd583ab08921e6ff81a999d6517a/sqlalchemy-2.0.49-cp313-cp313t-win_amd64.whl", hash = "sha256:b9870d15ef00e4d0559ae10ee5bc71b654d1f20076dbe8bc7ed19b4c0625ceba", size = 2175104, upload-time = "2026-04-03T17:02:05.989Z" },
+ { url = "https://files.pythonhosted.org/packages/55/33/bf28f618c0a9597d14e0b9ee7d1e0622faff738d44fe986ee287cdf1b8d0/sqlalchemy-2.0.49-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:233088b4b99ebcbc5258c755a097aa52fbf90727a03a5a80781c4b9c54347a2e", size = 2156356, upload-time = "2026-04-03T16:53:09.914Z" },
+ { url = "https://files.pythonhosted.org/packages/d1/a7/5f476227576cb8644650eff68cc35fa837d3802b997465c96b8340ced1e2/sqlalchemy-2.0.49-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:57ca426a48eb2c682dae8204cd89ea8ab7031e2675120a47924fabc7caacbc2a", size = 3276486, upload-time = "2026-04-03T17:07:46.9Z" },
+ { url = "https://files.pythonhosted.org/packages/2e/84/efc7c0bf3a1c5eef81d397f6fddac855becdbb11cb38ff957888603014a7/sqlalchemy-2.0.49-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:685e93e9c8f399b0c96a624799820176312f5ceef958c0f88215af4013d29066", size = 3281479, upload-time = "2026-04-03T17:12:32.226Z" },
+ { url = "https://files.pythonhosted.org/packages/91/68/bb406fa4257099c67bd75f3f2261b129c63204b9155de0d450b37f004698/sqlalchemy-2.0.49-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:9e0400fa22f79acc334d9a6b185dc00a44a8e6578aa7e12d0ddcd8434152b187", size = 3226269, upload-time = "2026-04-03T17:07:48.678Z" },
+ { url = "https://files.pythonhosted.org/packages/67/84/acb56c00cca9f251f437cb49e718e14f7687505749ea9255d7bd8158a6df/sqlalchemy-2.0.49-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:a05977bffe9bffd2229f477fa75eabe3192b1b05f408961d1bebff8d1cd4d401", size = 3248260, upload-time = "2026-04-03T17:12:34.381Z" },
+ { url = "https://files.pythonhosted.org/packages/56/19/6a20ea25606d1efd7bd1862149bb2a22d1451c3f851d23d887969201633f/sqlalchemy-2.0.49-cp314-cp314-win32.whl", hash = "sha256:0f2fa354ba106eafff2c14b0cc51f22801d1e8b2e4149342023bd6f0955de5f5", size = 2118463, upload-time = "2026-04-03T17:05:47.093Z" },
+ { url = "https://files.pythonhosted.org/packages/cf/4f/8297e4ed88e80baa1f5aa3c484a0ee29ef3c69c7582f206c916973b75057/sqlalchemy-2.0.49-cp314-cp314-win_amd64.whl", hash = "sha256:77641d299179c37b89cf2343ca9972c88bb6eef0d5fc504a2f86afd15cd5adf5", size = 2144204, upload-time = "2026-04-03T17:05:48.694Z" },
+ { url = "https://files.pythonhosted.org/packages/1f/33/95e7216df810c706e0cd3655a778604bbd319ed4f43333127d465a46862d/sqlalchemy-2.0.49-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c1dc3368794d522f43914e03312202523cc89692f5389c32bea0233924f8d977", size = 3565474, upload-time = "2026-04-03T16:58:35.128Z" },
+ { url = "https://files.pythonhosted.org/packages/0c/a4/ed7b18d8ccf7f954a83af6bb73866f5bc6f5636f44c7731fbb741f72cc4f/sqlalchemy-2.0.49-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7c821c47ecfe05cc32140dcf8dc6fd5d21971c86dbd56eabfe5ba07a64910c01", size = 3530567, upload-time = "2026-04-03T17:06:04.587Z" },
+ { url = "https://files.pythonhosted.org/packages/73/a3/20faa869c7e21a827c4a2a42b41353a54b0f9f5e96df5087629c306df71e/sqlalchemy-2.0.49-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:9c04bff9a5335eb95c6ecf1c117576a0aa560def274876fd156cfe5510fccc61", size = 3474282, upload-time = "2026-04-03T16:58:37.131Z" },
+ { url = "https://files.pythonhosted.org/packages/b7/50/276b9a007aa0764304ad467eceb70b04822dc32092492ee5f322d559a4dc/sqlalchemy-2.0.49-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:7f605a456948c35260e7b2a39f8952a26f077fd25653c37740ed186b90aaa68a", size = 3480406, upload-time = "2026-04-03T17:06:07.176Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/c3/c80fcdb41905a2df650c2a3e0337198b6848876e63d66fe9188ef9003d24/sqlalchemy-2.0.49-cp314-cp314t-win32.whl", hash = "sha256:6270d717b11c5476b0cbb21eedc8d4dbb7d1a956fd6c15a23e96f197a6193158", size = 2149151, upload-time = "2026-04-03T17:02:07.281Z" },
+ { url = "https://files.pythonhosted.org/packages/05/52/9f1a62feab6ed368aff068524ff414f26a6daebc7361861035ae00b05530/sqlalchemy-2.0.49-cp314-cp314t-win_amd64.whl", hash = "sha256:275424295f4256fd301744b8f335cff367825d270f155d522b30c7bf49903ee7", size = 2184178, upload-time = "2026-04-03T17:02:08.623Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/30/8519fdde58a7bdf155b714359791ad1dc018b47d60269d5d160d311fdc36/sqlalchemy-2.0.49-py3-none-any.whl", hash = "sha256:ec44cfa7ef1a728e88ad41674de50f6db8cfdb3e2af84af86e0041aaf02d43d0", size = 1942158, upload-time = "2026-04-03T16:53:44.135Z" },
+]
+
[[package]]
name = "stack-data"
version = "0.6.3"
@@ -4219,6 +4716,60 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/32/d5/f9a850d79b0851d1d4ef6456097579a9005b31fea68726a4ae5f2d82ddd9/threadpoolctl-3.6.0-py3-none-any.whl", hash = "sha256:43a0b8fd5a2928500110039e43a5eed8480b918967083ea48dc3ab9f13c4a7fb", size = 18638, upload-time = "2025-03-13T13:49:21.846Z" },
]
+[[package]]
+name = "tiktoken"
+version = "0.12.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "regex" },
+ { name = "requests" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/7d/ab/4d017d0f76ec3171d469d80fc03dfbb4e48a4bcaddaa831b31d526f05edc/tiktoken-0.12.0.tar.gz", hash = "sha256:b18ba7ee2b093863978fcb14f74b3707cdc8d4d4d3836853ce7ec60772139931", size = 37806, upload-time = "2025-10-06T20:22:45.419Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/de/46/21ea696b21f1d6d1efec8639c204bdf20fde8bafb351e1355c72c5d7de52/tiktoken-0.12.0-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:6e227c7f96925003487c33b1b32265fad2fbcec2b7cf4817afb76d416f40f6bb", size = 1051565, upload-time = "2025-10-06T20:21:44.566Z" },
+ { url = "https://files.pythonhosted.org/packages/c9/d9/35c5d2d9e22bb2a5f74ba48266fb56c63d76ae6f66e02feb628671c0283e/tiktoken-0.12.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:c06cf0fcc24c2cb2adb5e185c7082a82cba29c17575e828518c2f11a01f445aa", size = 995284, upload-time = "2025-10-06T20:21:45.622Z" },
+ { url = "https://files.pythonhosted.org/packages/01/84/961106c37b8e49b9fdcf33fe007bb3a8fdcc380c528b20cc7fbba80578b8/tiktoken-0.12.0-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:f18f249b041851954217e9fd8e5c00b024ab2315ffda5ed77665a05fa91f42dc", size = 1129201, upload-time = "2025-10-06T20:21:47.074Z" },
+ { url = "https://files.pythonhosted.org/packages/6a/d0/3d9275198e067f8b65076a68894bb52fd253875f3644f0a321a720277b8a/tiktoken-0.12.0-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:47a5bc270b8c3db00bb46ece01ef34ad050e364b51d406b6f9730b64ac28eded", size = 1152444, upload-time = "2025-10-06T20:21:48.139Z" },
+ { url = "https://files.pythonhosted.org/packages/78/db/a58e09687c1698a7c592e1038e01c206569b86a0377828d51635561f8ebf/tiktoken-0.12.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:508fa71810c0efdcd1b898fda574889ee62852989f7c1667414736bcb2b9a4bd", size = 1195080, upload-time = "2025-10-06T20:21:49.246Z" },
+ { url = "https://files.pythonhosted.org/packages/9e/1b/a9e4d2bf91d515c0f74afc526fd773a812232dd6cda33ebea7f531202325/tiktoken-0.12.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:a1af81a6c44f008cba48494089dd98cccb8b313f55e961a52f5b222d1e507967", size = 1255240, upload-time = "2025-10-06T20:21:50.274Z" },
+ { url = "https://files.pythonhosted.org/packages/9d/15/963819345f1b1fb0809070a79e9dd96938d4ca41297367d471733e79c76c/tiktoken-0.12.0-cp311-cp311-win_amd64.whl", hash = "sha256:3e68e3e593637b53e56f7237be560f7a394451cb8c11079755e80ae64b9e6def", size = 879422, upload-time = "2025-10-06T20:21:51.734Z" },
+ { url = "https://files.pythonhosted.org/packages/a4/85/be65d39d6b647c79800fd9d29241d081d4eeb06271f383bb87200d74cf76/tiktoken-0.12.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:b97f74aca0d78a1ff21b8cd9e9925714c15a9236d6ceacf5c7327c117e6e21e8", size = 1050728, upload-time = "2025-10-06T20:21:52.756Z" },
+ { url = "https://files.pythonhosted.org/packages/4a/42/6573e9129bc55c9bf7300b3a35bef2c6b9117018acca0dc760ac2d93dffe/tiktoken-0.12.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:2b90f5ad190a4bb7c3eb30c5fa32e1e182ca1ca79f05e49b448438c3e225a49b", size = 994049, upload-time = "2025-10-06T20:21:53.782Z" },
+ { url = "https://files.pythonhosted.org/packages/66/c5/ed88504d2f4a5fd6856990b230b56d85a777feab84e6129af0822f5d0f70/tiktoken-0.12.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:65b26c7a780e2139e73acc193e5c63ac754021f160df919add909c1492c0fb37", size = 1129008, upload-time = "2025-10-06T20:21:54.832Z" },
+ { url = "https://files.pythonhosted.org/packages/f4/90/3dae6cc5436137ebd38944d396b5849e167896fc2073da643a49f372dc4f/tiktoken-0.12.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:edde1ec917dfd21c1f2f8046b86348b0f54a2c0547f68149d8600859598769ad", size = 1152665, upload-time = "2025-10-06T20:21:56.129Z" },
+ { url = "https://files.pythonhosted.org/packages/a3/fe/26df24ce53ffde419a42f5f53d755b995c9318908288c17ec3f3448313a3/tiktoken-0.12.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:35a2f8ddd3824608b3d650a000c1ef71f730d0c56486845705a8248da00f9fe5", size = 1194230, upload-time = "2025-10-06T20:21:57.546Z" },
+ { url = "https://files.pythonhosted.org/packages/20/cc/b064cae1a0e9fac84b0d2c46b89f4e57051a5f41324e385d10225a984c24/tiktoken-0.12.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:83d16643edb7fa2c99eff2ab7733508aae1eebb03d5dfc46f5565862810f24e3", size = 1254688, upload-time = "2025-10-06T20:21:58.619Z" },
+ { url = "https://files.pythonhosted.org/packages/81/10/b8523105c590c5b8349f2587e2fdfe51a69544bd5a76295fc20f2374f470/tiktoken-0.12.0-cp312-cp312-win_amd64.whl", hash = "sha256:ffc5288f34a8bc02e1ea7047b8d041104791d2ddbf42d1e5fa07822cbffe16bd", size = 878694, upload-time = "2025-10-06T20:21:59.876Z" },
+ { url = "https://files.pythonhosted.org/packages/00/61/441588ee21e6b5cdf59d6870f86beb9789e532ee9718c251b391b70c68d6/tiktoken-0.12.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:775c2c55de2310cc1bc9a3ad8826761cbdc87770e586fd7b6da7d4589e13dab3", size = 1050802, upload-time = "2025-10-06T20:22:00.96Z" },
+ { url = "https://files.pythonhosted.org/packages/1f/05/dcf94486d5c5c8d34496abe271ac76c5b785507c8eae71b3708f1ad9b45a/tiktoken-0.12.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:a01b12f69052fbe4b080a2cfb867c4de12c704b56178edf1d1d7b273561db160", size = 993995, upload-time = "2025-10-06T20:22:02.788Z" },
+ { url = "https://files.pythonhosted.org/packages/a0/70/5163fe5359b943f8db9946b62f19be2305de8c3d78a16f629d4165e2f40e/tiktoken-0.12.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:01d99484dc93b129cd0964f9d34eee953f2737301f18b3c7257bf368d7615baa", size = 1128948, upload-time = "2025-10-06T20:22:03.814Z" },
+ { url = "https://files.pythonhosted.org/packages/0c/da/c028aa0babf77315e1cef357d4d768800c5f8a6de04d0eac0f377cb619fa/tiktoken-0.12.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:4a1a4fcd021f022bfc81904a911d3df0f6543b9e7627b51411da75ff2fe7a1be", size = 1151986, upload-time = "2025-10-06T20:22:05.173Z" },
+ { url = "https://files.pythonhosted.org/packages/a0/5a/886b108b766aa53e295f7216b509be95eb7d60b166049ce2c58416b25f2a/tiktoken-0.12.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:981a81e39812d57031efdc9ec59fa32b2a5a5524d20d4776574c4b4bd2e9014a", size = 1194222, upload-time = "2025-10-06T20:22:06.265Z" },
+ { url = "https://files.pythonhosted.org/packages/f4/f8/4db272048397636ac7a078d22773dd2795b1becee7bc4922fe6207288d57/tiktoken-0.12.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:9baf52f84a3f42eef3ff4e754a0db79a13a27921b457ca9832cf944c6be4f8f3", size = 1255097, upload-time = "2025-10-06T20:22:07.403Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/32/45d02e2e0ea2be3a9ed22afc47d93741247e75018aac967b713b2941f8ea/tiktoken-0.12.0-cp313-cp313-win_amd64.whl", hash = "sha256:b8a0cd0c789a61f31bf44851defbd609e8dd1e2c8589c614cc1060940ef1f697", size = 879117, upload-time = "2025-10-06T20:22:08.418Z" },
+ { url = "https://files.pythonhosted.org/packages/ce/76/994fc868f88e016e6d05b0da5ac24582a14c47893f4474c3e9744283f1d5/tiktoken-0.12.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:d5f89ea5680066b68bcb797ae85219c72916c922ef0fcdd3480c7d2315ffff16", size = 1050309, upload-time = "2025-10-06T20:22:10.939Z" },
+ { url = "https://files.pythonhosted.org/packages/f6/b8/57ef1456504c43a849821920d582a738a461b76a047f352f18c0b26c6516/tiktoken-0.12.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:b4e7ed1c6a7a8a60a3230965bdedba8cc58f68926b835e519341413370e0399a", size = 993712, upload-time = "2025-10-06T20:22:12.115Z" },
+ { url = "https://files.pythonhosted.org/packages/72/90/13da56f664286ffbae9dbcfadcc625439142675845baa62715e49b87b68b/tiktoken-0.12.0-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:fc530a28591a2d74bce821d10b418b26a094bf33839e69042a6e86ddb7a7fb27", size = 1128725, upload-time = "2025-10-06T20:22:13.541Z" },
+ { url = "https://files.pythonhosted.org/packages/05/df/4f80030d44682235bdaecd7346c90f67ae87ec8f3df4a3442cb53834f7e4/tiktoken-0.12.0-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:06a9f4f49884139013b138920a4c393aa6556b2f8f536345f11819389c703ebb", size = 1151875, upload-time = "2025-10-06T20:22:14.559Z" },
+ { url = "https://files.pythonhosted.org/packages/22/1f/ae535223a8c4ef4c0c1192e3f9b82da660be9eb66b9279e95c99288e9dab/tiktoken-0.12.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:04f0e6a985d95913cabc96a741c5ffec525a2c72e9df086ff17ebe35985c800e", size = 1194451, upload-time = "2025-10-06T20:22:15.545Z" },
+ { url = "https://files.pythonhosted.org/packages/78/a7/f8ead382fce0243cb625c4f266e66c27f65ae65ee9e77f59ea1653b6d730/tiktoken-0.12.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:0ee8f9ae00c41770b5f9b0bb1235474768884ae157de3beb5439ca0fd70f3e25", size = 1253794, upload-time = "2025-10-06T20:22:16.624Z" },
+ { url = "https://files.pythonhosted.org/packages/93/e0/6cc82a562bc6365785a3ff0af27a2a092d57c47d7a81d9e2295d8c36f011/tiktoken-0.12.0-cp313-cp313t-win_amd64.whl", hash = "sha256:dc2dd125a62cb2b3d858484d6c614d136b5b848976794edfb63688d539b8b93f", size = 878777, upload-time = "2025-10-06T20:22:18.036Z" },
+ { url = "https://files.pythonhosted.org/packages/72/05/3abc1db5d2c9aadc4d2c76fa5640134e475e58d9fbb82b5c535dc0de9b01/tiktoken-0.12.0-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:a90388128df3b3abeb2bfd1895b0681412a8d7dc644142519e6f0a97c2111646", size = 1050188, upload-time = "2025-10-06T20:22:19.563Z" },
+ { url = "https://files.pythonhosted.org/packages/e3/7b/50c2f060412202d6c95f32b20755c7a6273543b125c0985d6fa9465105af/tiktoken-0.12.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:da900aa0ad52247d8794e307d6446bd3cdea8e192769b56276695d34d2c9aa88", size = 993978, upload-time = "2025-10-06T20:22:20.702Z" },
+ { url = "https://files.pythonhosted.org/packages/14/27/bf795595a2b897e271771cd31cb847d479073497344c637966bdf2853da1/tiktoken-0.12.0-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:285ba9d73ea0d6171e7f9407039a290ca77efcdb026be7769dccc01d2c8d7fff", size = 1129271, upload-time = "2025-10-06T20:22:22.06Z" },
+ { url = "https://files.pythonhosted.org/packages/f5/de/9341a6d7a8f1b448573bbf3425fa57669ac58258a667eb48a25dfe916d70/tiktoken-0.12.0-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:d186a5c60c6a0213f04a7a802264083dea1bbde92a2d4c7069e1a56630aef830", size = 1151216, upload-time = "2025-10-06T20:22:23.085Z" },
+ { url = "https://files.pythonhosted.org/packages/75/0d/881866647b8d1be4d67cb24e50d0c26f9f807f994aa1510cb9ba2fe5f612/tiktoken-0.12.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:604831189bd05480f2b885ecd2d1986dc7686f609de48208ebbbddeea071fc0b", size = 1194860, upload-time = "2025-10-06T20:22:24.602Z" },
+ { url = "https://files.pythonhosted.org/packages/b3/1e/b651ec3059474dab649b8d5b69f5c65cd8fcd8918568c1935bd4136c9392/tiktoken-0.12.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:8f317e8530bb3a222547b85a58583238c8f74fd7a7408305f9f63246d1a0958b", size = 1254567, upload-time = "2025-10-06T20:22:25.671Z" },
+ { url = "https://files.pythonhosted.org/packages/80/57/ce64fd16ac390fafde001268c364d559447ba09b509181b2808622420eec/tiktoken-0.12.0-cp314-cp314-win_amd64.whl", hash = "sha256:399c3dd672a6406719d84442299a490420b458c44d3ae65516302a99675888f3", size = 921067, upload-time = "2025-10-06T20:22:26.753Z" },
+ { url = "https://files.pythonhosted.org/packages/ac/a4/72eed53e8976a099539cdd5eb36f241987212c29629d0a52c305173e0a68/tiktoken-0.12.0-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:c2c714c72bc00a38ca969dae79e8266ddec999c7ceccd603cc4f0d04ccd76365", size = 1050473, upload-time = "2025-10-06T20:22:27.775Z" },
+ { url = "https://files.pythonhosted.org/packages/e6/d7/0110b8f54c008466b19672c615f2168896b83706a6611ba6e47313dbc6e9/tiktoken-0.12.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:cbb9a3ba275165a2cb0f9a83f5d7025afe6b9d0ab01a22b50f0e74fee2ad253e", size = 993855, upload-time = "2025-10-06T20:22:28.799Z" },
+ { url = "https://files.pythonhosted.org/packages/5f/77/4f268c41a3957c418b084dd576ea2fad2e95da0d8e1ab705372892c2ca22/tiktoken-0.12.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:dfdfaa5ffff8993a3af94d1125870b1d27aed7cb97aa7eb8c1cefdbc87dbee63", size = 1129022, upload-time = "2025-10-06T20:22:29.981Z" },
+ { url = "https://files.pythonhosted.org/packages/4e/2b/fc46c90fe5028bd094cd6ee25a7db321cb91d45dc87531e2bdbb26b4867a/tiktoken-0.12.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:584c3ad3d0c74f5269906eb8a659c8bfc6144a52895d9261cdaf90a0ae5f4de0", size = 1150736, upload-time = "2025-10-06T20:22:30.996Z" },
+ { url = "https://files.pythonhosted.org/packages/28/c0/3c7a39ff68022ddfd7d93f3337ad90389a342f761c4d71de99a3ccc57857/tiktoken-0.12.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:54c891b416a0e36b8e2045b12b33dd66fb34a4fe7965565f1b482da50da3e86a", size = 1194908, upload-time = "2025-10-06T20:22:32.073Z" },
+ { url = "https://files.pythonhosted.org/packages/ab/0d/c1ad6f4016a3968c048545f5d9b8ffebf577774b2ede3e2e352553b685fe/tiktoken-0.12.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:5edb8743b88d5be814b1a8a8854494719080c28faaa1ccbef02e87354fe71ef0", size = 1253706, upload-time = "2025-10-06T20:22:33.385Z" },
+ { url = "https://files.pythonhosted.org/packages/af/df/c7891ef9d2712ad774777271d39fdef63941ffba0a9d59b7ad1fd2765e57/tiktoken-0.12.0-cp314-cp314t-win_amd64.whl", hash = "sha256:f61c0aea5565ac82e2ec50a05e02a6c44734e91b51c10510b084ea1b8e633a71", size = 920667, upload-time = "2025-10-06T20:22:34.444Z" },
+]
+
[[package]]
name = "tokenizers"
version = "0.22.2"
@@ -4483,6 +5034,19 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/18/67/36e9267722cc04a6b9f15c7f3441c2363321a3ea07da7ae0c0707beb2a9c/typing_extensions-4.15.0-py3-none-any.whl", hash = "sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548", size = 44614, upload-time = "2025-08-25T13:49:24.86Z" },
]
+[[package]]
+name = "typing-inspect"
+version = "0.9.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "mypy-extensions" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/dc/74/1789779d91f1961fa9438e9a8710cdae6bd138c80d7303996933d117264a/typing_inspect-0.9.0.tar.gz", hash = "sha256:b23fc42ff6f6ef6954e4852c1fb512cdd18dbea03134f91f856a95ccc9461f78", size = 13825, upload-time = "2023-05-24T20:25:47.612Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/65/f3/107a22063bf27bdccf2024833d3445f4eea42b2e598abfbd46f6a63b6cb0/typing_inspect-0.9.0-py3-none-any.whl", hash = "sha256:9ee6fc59062311ef8547596ab6b955e1b8aa46242d854bfc78f4f6b0eff35f9f", size = 8827, upload-time = "2023-05-24T20:25:45.287Z" },
+]
+
[[package]]
name = "typing-inspection"
version = "0.4.2"