LogicHunter

Testing LLM agent frameworks with API-focused test generation and an agentic oracle.

LogicHunter is a testing framework for LLM agent libraries such as LangChain, LlamaIndex, and CrewAI. It generates API-focused tests, executes them, filters suspicious failures, and uses an agentic oracle to decide whether a failure is likely to be a framework bug, a documentation mismatch, a data-integrity issue, or expected user misuse.

What LogicHunter Does

• Builds target API profiles from installed Python packages.
• Generates seed tests with an LLM-based generator.
• Repairs invalid seeds and mutates valid seeds into executable test cases.
• Executes generated tests and preprocesses failures through classification, deduplication, and filtering.
• Runs an agentic oracle with documentation search, source-code search, test-case inspection, and sandboxed code execution.
• Saves verdicts, logs, and intermediate reasoning for later inspection.

Supported Targets

LogicHunter currently includes built-in support for:

• langchain / langchain_core
• llama_index
• crewai

Other Python libraries can be explored by extending the API list, source extraction logic, and framework-specific artifacts under src/.

Installation

LogicHunter is intended to run on Linux with Python 3.10. A conda environment is recommended because the target agent-framework dependencies can be large.

git clone <repo-url>
cd LogicHunter

conda create -n logichunter python=3.10 -y
conda activate logichunter
pip install -r requirements.txt

For sandboxed execution inside the oracle, install bubblewrap if it is available on your system:

sudo apt-get install bubblewrap

If bubblewrap is unavailable, LogicHunter can still run with an unsafe fallback for run_code; see src/README.md for details.

Configuration

LogicHunter uses an OpenAI-compatible API for test generation and oracle reasoning.

export LOGICHUNTER_API_KEY="your_api_key_here"
export LOGICHUNTER_BASE_URL="https://your-openai-compatible-endpoint/v1"

Optional tracing with Langfuse can be enabled by setting:

export LANGFUSE_PUBLIC_KEY="..."
export LANGFUSE_SECRET_KEY="..."
export LANGFUSE_HOST="https://cloud.langfuse.com"

More configuration options are documented in src/README.md.

Quick Start

After installing dependencies and configuring LOGICHUNTER_API_KEY and LOGICHUNTER_BASE_URL, you can quickly verify the end-to-end workflow from the repository root:

./demo_run.sh

The demo uses the first three LangChain APIs from examples/api_lists/langchain.txt, then runs test generation, preprocessing, and the Agentic Oracle.

For manual runs, most commands should be run from the src/ directory:

cd src

1. Prepare Target APIs

Create an api.txt file in src/, one fully qualified API name per line:

langchain_core.messages.ai.AIMessage
langchain_core.runnables.base.RunnableSequence

The examples/api_lists/ directory contains the API lists used in our paper:

• examples/api_lists/langchain.txt
• examples/api_lists/llama_index.txt
• examples/api_lists/crewai.txt

You can copy one of these lists directly:

cp ../examples/api_lists/langchain.txt api.txt

The repository also includes extracted usage snippets under src/artifacts/, which are used as few-shot examples during test generation.

2. Generate and Execute Tests

Run the full generation pipeline for a supported library:

python test_generation.py run langchain_core

To resume from a later pipeline step:

python test_generation.py run langchain_core 5

Generated seeds, fuzz tests, and raw execution results are written under src/data/.

3. Preprocess Failures

Prepare raw execution results for oracle analysis:

python preprocess.py data/test_results.json data/test_results_valid.json

This classifies outcomes, removes duplicate failures, and filters failures that do not appear to originate in the target framework.

4. Run the Agentic Oracle

Judge the preprocessed failures:

python oracle.py data/test_results_valid.json

Common options:

python oracle.py data/test_results_valid.json --max-cases 20
python oracle.py data/test_results_valid.json --model gpt-5-mini --temperature 0
python oracle.py data/test_results_valid.json --tools doc_search,code_search,run_code,get_test_case_information

Oracle outputs are saved under judge_analysis_results/ next to the input JSON file.

Verdicts

The agentic oracle uses six verdict categories:

• INTERNAL_ERROR: the framework likely fails internally.
• DOC_MISMATCH: observed behavior appears inconsistent with documentation or public API expectations.
• DATA_INTEGRITY: the failure suggests corruption, loss, or inconsistent handling of data/state.
• ROBUSTNESS: the framework rejects malformed or edge-case input in an acceptable way.
• USABILITY: the behavior is confusing or inconvenient but not clearly a correctness bug.
• MISUSE: the test appears to use the API incorrectly.

The positive bug categories are INTERNAL_ERROR, DOC_MISMATCH, and DATA_INTEGRITY.

Repository Layout

.
├── src/                    # LogicHunter implementation
│   ├── agent/              # Agentic oracle reasoning loop
│   ├── artifacts/          # Usage snippets and API-frequency data
│   ├── fuzz/               # Test generation, repair, mutation, execution
│   ├── prompts/            # Prompt templates
│   ├── tools/              # Oracle tools: docs, code search, run_code
│   ├── utils/              # Preprocessing, tracing, loading, tool parsing
│   ├── test_generation.py  # End-to-end generation pipeline
│   ├── preprocess.py       # Failure preprocessing pipeline
│   └── oracle.py           # Agentic oracle CLI
├── evaluation_script/      # Utility scripts, including coverage helpers
├── requirements.txt
├── LICENSE
└── README.md

Documentation

• src/README.md: source-tree guide, input/output formats, advanced configuration, and oracle options.

License

This project is licensed under the MIT License. See LICENSE for details.