TestAgent: A Multi-Agent LLM Framework for Repository-Level Unit Test Generation

TestAgent VS Code Extension

We provide TestAgent as a VS Code extension that integrates the full multi-agent workflow into everyday development. A demo video is available here:

TestAgent_Plugin_Demo.mp4

About TestAgent

TestAgent is a multi-agent LLM framework for repository-level unit test generation grounded in a Code Knowledge Graph (CKG). It organizes test development into three phases and assigns each phase to a specialized agent:

• Planner: analyzes the focal method and derives test points (behavioral objectives).
• Generator: synthesizes concrete tests, compiles and executes them, and iterates on failures.
• Reviewer: evaluates test adequacy across test points and decides whether to accept, revise, or extend.

TestAgent uses a three-layer architecture:

1. CKG layer: stores repository-level entities and relationships (classes, methods, dependencies, calls, tests).
2. Tool layer: provides deterministic operations (compile, execute, coverage, mutation) and retrieval/update APIs over the CKG.
3. Agent layer: coordinates planning, generation, and review using tool-mediated interactions.

Empirical results from the paper show strong effectiveness on real-world Java projects: 99.11% compilation success, 97.46% execution success, 92.34% line coverage, 90.24% branch coverage, and 83.69% mutation score. TestAgent also detects 154 real bugs with 92.22% precision. The framework remains effective across different LLM backends (e.g., GPT-4o, DeepSeek-V3, Qwen3-30B).

Repository Layout

• backend/ - FastAPI service implementing the multi-agent workflow and tool layer
• vscode-extension/ - VS Code extension for interactive usage

Running Steps

1. Clone the repository

2. Install prerequisites

• Java 11+ (required for CKG construction and test execution)
• Neo4j 5.x (CKG storage)
• Python 3.10+ (backend)
• Node.js 20+ (extension development)
• VS Code 1.99+

3. Start Neo4j

1. Install Neo4j and start a local instance.
2. Verify connectivity in Neo4j Browser: http://localhost:7474.
3. Note your credentials (default user is neo4j).

4. Configure the backend

Edit backend/Config.py and set the following fields:

• OPENAI_API_BASE
• OPENAI_API_KEY
• CKGConstruction_Jar_Path (path to CKGConstruction-1.0-SNAPSHOT.jar)
• Optional: RECURSION_LIMIT, Enable_Native_Function_Call, Limit_Retrieve_Test_Case

5. Install backend dependencies

pip install -r backend/requirements.txt

6. Start the backend service

python backend/start_server.py

The API defaults to http://localhost:8000 and exposes both task-based and streaming endpoints.

7. Configure the extension

Edit vscode-extension/config/userConfig.json (or use the Settings UI after launching the extension):

{
  "neo4j": {
    "serverUrl": "bolt://localhost:7687",
    "serverUser": "neo4j",
    "serverPassword": "your-neo4j-password"
  },
  "testAgent": {
    "recursionLimit": 50,
    "openaiApiBase": "https://api.openai.com/v1",
    "openaiApiKey": "your-openai-api-key",
    "enableNativeFunctionCall": false,
    "limitRetrieveTestCase": false,
    "ckgConstructionJarPath": "/path/to/CKGConstruction-1.0-SNAPSHOT.jar",
    "testGenerationApiUrl": "http://localhost:8000/generate_test_case_stream",
    "testGenerationLimit": 50
  }
}

We provide a prebuilt VSIX package in vscode-extension/. Steps 8 and 9 are optional and only required if you want to build or package the extension yourself.

8. Run the extension

cd vscode-extension
npm install
npm run compile

Launch the extension in VS Code (Run and Debug -> "Run Extension") or use code --extensionDevelopmentPath=....

9. Package the extension (VSIX)

cd vscode-extension
npx @vscode/vsce package

The VSIX package is generated in vscode-extension/ (for example: testagent-0.1.0.vsix).

Demo Workflow

This walkthrough mirrors the demonstration scenario described in the paper.

1. Open a target Java repository in VS Code.
2. Launch TestAgent from the Activity Bar.
3. Configure runtime settings (Neo4j connection, LLM API base/key, CKG JAR path, backend endpoint).
4. Build the Code Knowledge Graph (CKG):
   • Trigger "Generate Knowledge Graph".
   • The tool performs static analysis and stores entities and relations in Neo4j.
5. Inspect the CKG visualization:
   • Use "Show Knowledge Graph" to explore classes, methods, and dependencies.
6. Select focal methods:
   • Browse the testable method list in the sidebar.
   • Select one or multiple methods as focal targets.
7. Generate tests (Planner -> Generator -> Reviewer):
   • The Planner derives test points from method behavior and context retrieved from the CKG.
   • The Generator synthesizes tests and executes them, using tool feedback to refine failures.
   • The Reviewer evaluates adequacy and decides whether to accept or request refinements.
8. Monitor live progress:
   • Streamed logs and step updates show the agent workflow state, intermediate outputs, and diagnostics.
9. Review results:
   • Test cases are returned with execution summaries, coverage metrics, and mutation results.
10. Inject accepted tests:

• Use "Inject Test Case" (single) or "Inject All Test Cases" (batch) to write tests into src/test.

11. Iterate or extend:

• Refine test points, regenerate as needed, and re-run evaluation for improved coverage.

Additional Documentation

• Backend workflow: backend/README.md
• Extension workflow: vscode-extension/README.md

License

MIT License. See LICENSE.