[EPIC] Agentic enhancements: intelligent search, orchestration, and evaluation

[bashandbone]

Overview

Enhance CodeWeaver with agentic capabilities for intelligent code search, orchestrated reasoning, and evaluation. This builds on the scaffolded agent and data provider infrastructure.

The Vision

Transform CodeWeaver from a semantic search system into an intelligent code understanding platform with:

• Agent-enhanced search: LLM agents that reason about search strategy
• Data source integration: External context (web search, documentation, etc.)
• Context agents: Internal orchestration for multi-step reasoning
• Graph-based pipelines: Structured orchestration with pydantic-graph
• Evaluation framework: Continuous quality improvement with pydantic-eval

Scaffolded Infrastructure

Already in place:

• codeweaver.providers.agent - Thin wrapper around pydantic-ai
• codeweaver.providers.data - Data provider scaffolding
• Registry integration in codeweaver.common.registry.provider and .models

Implementation Phases

Phase 1: Agent Integration in find_code Pipeline - See #124

• Integrate pydantic-ai agents into search pipeline
• Agent-driven query refinement and strategy selection
• Tool integration for code-aware reasoning

Phase 2: Data Provider Integration - See #125

• Integrate external data sources (Tavily, DuckDuckGo, etc.)
• Context enrichment from documentation and web
• Data source orchestration

Phase 3: Context Agent Tooling - See #126

• Internal "context agent" for orchestrated search/response
• Multi-step reasoning over code context
• Intelligent result synthesis

Phase 4: pydantic-graph Pipeline Orchestration - See #127

• Structured pipeline orchestration with pydantic-graph
• Strategy-based execution paths
• Complex workflow composition

Phase 5: Evaluation Framework with pydantic-eval - See #128

• Agent performance evaluation
• Pipeline quality metrics
• Continuous improvement infrastructure

Dependencies

• DI System ([EPIC] DI + Monorepo: Unified architecture transformation #116): Phase 3 ([DI Phase 4] pydantic-ai integration (agents + data sources) #119) provides DI for agents and data sources
• Provider System (Implement provider system for reranker models (mirror embedding system) #109): Reranker provider system for quality improvements
• May require additional provider capabilities (Expand vector provider support and fix capabilities definition #104)

Success Criteria

• Agents successfully enhance search quality
• External data sources integrated seamlessly
• Context agents provide intelligent multi-step reasoning
• Pipelines are composable and maintainable via pydantic-graph
• Evaluation framework enables continuous improvement
• Performance remains acceptable (< 2x baseline latency)

Constitutional Alignment

✅ Empirical Approach (Principle III): Evaluation framework enables evidence-based improvements
✅ Proven Patterns (Principle II): Leverages pydantic-ai, pydantic-graph, pydantic-eval
✅ AI-First Context (Principle I): Agents enhance code understanding
✅ Simplicity (Principle V): Graph-based orchestration clarifies complex workflows

Related Work

This work complements but is distinct from:

• [EPIC] DI + Monorepo: Unified architecture transformation #116 (DI System): Infrastructure that enables agent injection
• Implement provider system for reranker models (mirror embedding system) #109 (Reranker System): Quality improvements work together
• Add monitoring: track current file, provider health checks, latency tracking #112 (Monitoring): Telemetry for agent performance

Source

• Scaffolded code: src/codeweaver/providers/agent/, src/codeweaver/providers/data/
• Registry integration: src/codeweaver/common/registry/
• Branch: 003-our-aim-to

[github-actions]

Claude finished @bashandbone's task —— View job

---

Analyzing EPIC Issue #123

Working on analyzing this EPIC for agentic enhancements...

Tasks

• Review project constitution and CLAUDE.md
• Examine scaffolded agent and data provider code
• Review registry integration
• Analyze related issues and dependencies
• Evaluate each phase for feasibility and complexity
• Provide recommendations on labels and implementation approach
• Identify potential issues or concerns

Initial Findings

Scaffolded Infrastructure Status:

• ✅ Agent providers: Fully implemented at src/codeweaver/providers/agent/ - thin wrapper around pydantic-ai with 20+ agent model providers
• ❌ Data providers: NOT FOUND - the issue mentions codeweaver.providers.data/ but this directory does not exist in the codebase

--- | Branch: claude/issue-123-20251119-2122