feat: implement 30 auto-claude improvement ideas

Security Hardening:
- SSRF prevention in webhook URL validation (sec-001)
- Redact sensitive headers in webhook error messages (sec-002)
- Redis broker TLS/auth warnings for insecure connections (sec-003)
- Block dangerous module imports in dynamic agent loading (sec-004)
- Add pip-audit dependency scanning to CI pipeline (sec-005)

Performance:
- Batch INSERT eval_results with executemany() (perf-002)
- Replace Redis KEYS scan with SCAN in coordinator (perf-003)
- Pipeline Redis enqueue + TTL operations (perf-004)
- Cache grader instances by (name, config) within runs (perf-005)

UI/UX:
- Add Examples sections to CLI command help text (uiux-001)
- Show cumulative pass/fail counts in progress output (uiux-002)
- Standardize table output with fixed-width columns (uiux-003)
- Collapsible <details> blocks in GitHub PR comments (uiux-004)
- NO_COLOR env var support per no-color.org (uiux-005)

Code Quality:
- Split 935-line cli.py into 12 command submodules (cq-001)
- Centralize error handling with _fail() helper (cq-002)
- Split agentlens importer into client/mapper/repository (cq-003)
- Add stricter ruff rulesets and mypy config (cq-004)
- Add .pre-commit-config.yaml with ruff hooks (cq-005)

Code Improvements:
- Add --exclude-tag filtering for run command (ci-001)
- Persist runtime config metadata in EvalRun.config (ci-002)
- Add --retries/--retry-backoff-ms for transient failures (ci-003)
- DB-level LIMIT/OFFSET pagination in store queries (ci-004)
- Add Notifier protocol with webhook/github adapters (ci-005)

Documentation:
- Add distributed execution section to README (doc-001)
- Add adapters section with framework examples (doc-002)
- Expand AgentLens importer docs with modes/options (doc-003)
- Create docs/troubleshooting.md guide (doc-004)
- Update README badges and test count references (doc-005)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: amitpaz1

.github/workflows/ci.yml

@@ -18,4 +18,6 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
       - run: pip install -e ".[dev]"
+      - name: Security audit
+        run: pip install pip-audit && pip-audit --strict --desc
       - run: pytest

.pre-commit-config.yaml

@@ -0,0 +1,13 @@
+repos:
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.8.0
+    hooks:
+      - id: ruff
+        args: [--fix]
+      - id: ruff-format
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v5.0.0
+    hooks:
+      - id: trailing-whitespace
+      - id: end-of-file-fixer
+      - id: check-yaml

README.md

@@ -1,7 +1,7 @@
 # AgentEval 🧪
 
 [![PyPI](https://img.shields.io/pypi/v/agentevalkit)](https://pypi.org/project/agentevalkit/)
-[![Tests](https://img.shields.io/badge/tests-127%20passing-brightgreen)]()
+[![Tests](https://img.shields.io/badge/tests-passing-brightgreen)]()
 [![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)]()
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
 
@@ -134,13 +134,25 @@ Run the same suite multiple times and compare groups: `agenteval compare RUN_A1,
 
 ### 🔗 AgentLens Integration
 
-Import real agent sessions from [AgentLens](https://github.com/amitpaz/agentlens) as test suites:
+Import real agent sessions from [AgentLens](https://github.com/agentkitai/agentlens) as test suites:
 
 ```bash
+# From AgentLens SQLite database
 agenteval import --from agentlens --db sessions.db --output suite.yaml --grader contains
-# Imported 42 cases → suite.yaml
+
+# From AgentLens server API
+agenteval import-agentlens --url http://localhost:3000 --output suite.yaml --grader contains
+
+# With filtering and interactive review
+agenteval import --from agentlens --db sessions.db --output suite.yaml --filter-tag production --auto-assertions --interactive
 ```
 
+**Import modes:**
+- **SQLite mode** (`import --from agentlens --db path`) — reads directly from an AgentLens database file
+- **Server mode** (`import-agentlens --url URL`) — fetches sessions via the AgentLens HTTP API
+
+Sessions are converted to eval cases with input/output mapping and optional tool-call assertions. Use `--auto-assertions` to automatically generate expected fields from session data, and `--interactive` to review each case before saving.
+
 Turn production traffic into regression tests — no manual test writing needed.
 
 ### 💰 Cost & Latency Tracking
@@ -311,11 +323,102 @@ grader_config:
 
 ---
 
+## Adapters
+
+Adapters let you test agents built with popular frameworks without writing a custom callable.
+
+```bash
+pip install agentevalkit[langchain]   # LangChain
+pip install agentevalkit[crewai]      # CrewAI
+pip install agentevalkit[autogen]     # AutoGen
+```
+
+| Adapter | Framework Method | Install Extra |
+|---------|-----------------|---------------|
+| `langchain` | `agent.invoke(input)` | `[langchain]` |
+| `crewai` | `crew.kickoff(inputs={"input": ...})` | `[crewai]` |
+| `autogen` | `agent.run(input)` or `agent.initiate_chat(message=...)` | `[autogen]` |
+
+Usage with YAML suite defaults:
+
+```yaml
+# suite.yaml
+name: my-tests
+agent: my_module:my_chain
+defaults:
+  adapter: langchain
+```
+
+Or via CLI:
+
+```bash
+agenteval run --suite suite.yaml --adapter langchain
+```
+
+Each adapter extracts output, tool calls, and token usage from the framework's response format into a standard `AgentResult`.
+
+---
+
+## Distributed Execution
+
+Scale eval suites across multiple workers using Redis as a broker.
+
+### Setup
+
+```bash
+pip install agentevalkit[distributed]
+```
+
+### Start Workers
+
+```bash
+# Terminal 1: Start a worker
+agenteval worker --broker redis://localhost:6379 --agent my_module:my_agent
+
+# Terminal 2: Start another worker
+agenteval worker --broker redis://localhost:6379 --agent my_module:my_agent
+```
+
+### Run with Workers
+
+```bash
+agenteval run --suite suite.yaml --workers redis://localhost:6379 --worker-timeout 60
+```
+
+### How It Works
+
+1. The coordinator pushes eval cases to a Redis queue
+2. Workers pop cases, execute the agent, and push results back
+3. The coordinator collects results and builds the final `EvalRun`
+4. If no workers are detected, execution falls back to local mode automatically
+
+### Configuration
+
+- `--workers URL` — Redis broker URL (supports `redis://` and `rediss://` for TLS)
+- `--worker-timeout N` — Seconds to wait for worker results (default: 30)
+- Workers register heartbeats and are automatically detected by the coordinator
+
+> **Security:** Use `rediss://` URLs with authentication for production deployments. See [docs/troubleshooting.md](docs/troubleshooting.md) for Redis security guidance.
+
+---
+
+## Troubleshooting
+
+See [docs/troubleshooting.md](docs/troubleshooting.md) for solutions to common issues including:
+
+- Agent callable import errors (`module:function` format)
+- Missing dependency extras (`[distributed]`, `[langchain]`, etc.)
+- OpenAI API key setup for `llm-judge` grader
+- Compare command syntax
+- Redis connection issues for distributed execution
+
+---
+
 ## Contributing
 
 Contributions welcome! This project uses:
 
-- **pytest** for testing (127 tests passing)
+- **pytest** for testing
 - **ruff** for linting
 - **src layout** (`src/agenteval/`)
 

docs/troubleshooting.md

@@ -0,0 +1,230 @@
+# Troubleshooting
+
+Common issues and solutions for AgentEval.
+
+---
+
+## Agent Import Errors
+
+### `ValueError: agent_ref must use 'module:attr' format`
+
+The `--agent` flag expects `module:function` format:
+
+```bash
+# Wrong
+agenteval run --suite suite.yaml --agent my_agent
+
+# Correct
+agenteval run --suite suite.yaml --agent my_module:run_agent
+```
+
+### `ModuleNotFoundError: No module named 'my_module'`
+
+Ensure the module is importable from your current directory:
+
+```bash
+# Your agent file must be in the current directory or on PYTHONPATH
+ls my_module.py  # Should exist
+
+# Or install your package
+pip install -e .
+```
+
+### `AttributeError: module 'my_module' has no attribute 'run_agent'`
+
+Check that the function name after `:` matches an exported function in the module.
+
+---
+
+## Missing Dependencies
+
+### `ImportError: Redis is required for distributed execution`
+
+Install the distributed extra:
+
+```bash
+pip install agentevalkit[distributed]
+```
+
+### `ImportError: scipy is required for statistical comparison`
+
+Install the stats extra for Welch's t-test:
+
+```bash
+pip install agentevalkit[stats]
+# or: pip install scipy
+```
+
+AgentEval falls back to a pure-Python implementation if scipy is unavailable.
+
+### `ImportError` for adapter frameworks
+
+Install the appropriate extra:
+
+```bash
+pip install agentevalkit[langchain]   # LangChain adapter
+pip install agentevalkit[crewai]      # CrewAI adapter
+pip install agentevalkit[autogen]     # AutoGen adapter
+```
+
+---
+
+## LLM Judge Grader
+
+### `Error: OPENAI_API_KEY not set`
+
+The `llm-judge` grader requires an OpenAI API key (or compatible API):
+
+```bash
+export OPENAI_API_KEY=sk-...
+```
+
+You can also configure a custom API base in the grader config:
+
+```yaml
+grader: llm-judge
+grader_config:
+  model: gpt-4o-mini
+  api_base: https://your-api.com/v1
+```
+
+---
+
+## Compare Command
+
+### `Error: Could not parse compare arguments`
+
+The compare command accepts two formats:
+
+```bash
+# Two single runs
+agenteval compare RUN_ID_A RUN_ID_B
+
+# Two groups (comma-separated, with 'vs')
+agenteval compare RUN_A1,RUN_A2 vs RUN_B1,RUN_B2
+```
+
+Run IDs are the short hex IDs shown by `agenteval list`.
+
+### `Error: Run not found`
+
+Check available runs with:
+
+```bash
+agenteval list --limit 20
+```
+
+---
+
+## YAML Suite Errors
+
+### `Error: Suite file not found`
+
+Ensure the path is correct:
+
+```bash
+agenteval run --suite ./suites/my_suite.yaml
+```
+
+### `Error: Invalid suite format`
+
+Check your YAML syntax. Common issues:
+- Missing `name` field
+- Missing `cases` list
+- Incorrect indentation
+- Using tabs instead of spaces
+
+Minimal valid suite:
+
+```yaml
+name: my-tests
+agent: my_module:my_fn
+cases:
+  - name: test-1
+    input: "Hello"
+    expected:
+      output_contains: ["hello"]
+    grader: contains
+```
+
+---
+
+## Database Issues
+
+### `sqlite3.OperationalError: unable to open database file`
+
+Check that the directory exists and is writable:
+
+```bash
+# Default location
+ls -la agenteval.db
+
+# Custom location
+agenteval run --suite suite.yaml --db /path/to/results.db
+```
+
+### Corrupted database
+
+Delete and re-run evaluations:
+
+```bash
+rm agenteval.db
+agenteval run --suite suite.yaml
+```
+
+---
+
+## Redis / Distributed Execution
+
+### Workers not detected
+
+Ensure workers are running and connected to the same Redis instance:
+
+```bash
+# Check Redis connectivity
+redis-cli -u redis://localhost:6379 ping
+# Should return: PONG
+
+# Start a worker
+agenteval worker --broker redis://localhost:6379 --agent my_module:my_fn
+```
+
+### Redis authentication errors
+
+Use an authenticated URL:
+
+```bash
+agenteval run --suite suite.yaml --workers redis://:password@host:6379
+```
+
+### Security best practices
+
+For production, use TLS-encrypted connections:
+
+```bash
+# Use rediss:// scheme for TLS
+agenteval worker --broker rediss://:password@host:6380
+
+# With custom CA certificate
+export REDIS_CA_CERT=/path/to/ca.pem
+```
+
+---
+
+## CI Integration
+
+### Exit codes
+
+- `0` — All cases passed
+- `1` — One or more cases failed (or regressions detected with `--fail-on-regression`)
+
+### GitHub PR comments not posting
+
+Check your token permissions:
+
+```bash
+export GITHUB_TOKEN=ghp_...  # Needs 'pull_requests: write' permission
+agenteval github-comment --run-id RUN_ID --repo owner/repo --pr 123
+```
+
+See [docs/github-actions.md](github-actions.md) for full CI setup.