MCP server for llamator: automate LLM red teaming workflows

Overview

This repository provides a production-oriented service wrapper around LLAMATOR for automated LLM red teaming. It exposes two integration surfaces:

• HTTP API (FastAPI) for job submission, job state retrieval, and artifacts access.
• MCP server (Streamable HTTP transport) for agent/tooling integrations, enabling LLAMATOR runs to be invoked as tools.

Execution is asynchronous and is orchestrated via ARQ + Redis. Artifacts are uploaded to MinIO and are retrieved through presigned URLs (returned as JSON; the API does not redirect).

Capabilities

• Asynchronous test runs with durable state persisted in Redis.
• Request persistence with secret redaction:
  • API keys are not stored in plaintext.
  • Stored payloads include only boolean markers (e.g. api_key_present).
• Artifacts lifecycle management:
  • Worker creates job-local artifacts under LLAMATOR_MCP_ARTIFACTS_ROOT/<job_id>/....
  • Artifacts are uploaded to MinIO as an archive named artifacts.zip.
  • HTTP API can list available objects under a job prefix and resolve presigned download links.
• Optional API-key protection for both HTTP and MCP interfaces via X-API-Key.
• OpenAPI schema (Swagger UI) with API-key authorization support.
• Prometheus metrics exposed at /metrics.

Deployment (Docker Compose)

Requirements:

• Docker
• Docker Compose

Start the full stack:

docker compose up --build

Default service endpoints:

• HTTP API: http://localhost:8000
• MinIO S3 endpoint: http://localhost:9000
• MinIO console: http://localhost:9001

Healthcheck:

curl -sS http://localhost:8000/v1/health

Configuration

All configuration is provided via environment variables prefixed with LLAMATOR_MCP_. A complete reference is available in DOCUMENTATION.md.

Typical local setup:

cp .env.example .env

Key configuration categories:

• Redis: connection DSN for job queue and state storage.
• MinIO: S3-compatible storage for artifacts.
• Attack/Judge models: OpenAI-compatible endpoints for LLAMATOR execution.
• API security: optional X-API-Key protection.
• Job execution: timeouts, TTLs, and retry behavior.

HTTP API usage

Create a run

curl -sS -X POST "http://localhost:8000/v1/tests/runs" \
  -H "Content-Type: application/json" \
  -H "X-API-Key: <optional>" \
  -d '{
    "tested_model": {
      "kind": "openai",
      "base_url": "http://host.docker.internal:1234/v1",
      "model": "llm",
      "api_key": "lm-studio"
    },
    "run_config": { "enable_reports": false },
    "plan": { "preset_name": "owasp:llm10", "num_threads": 1 }
  }'

The response contains:

• job_id (uuid4 hex, 32 characters)
• status (queued | running | succeeded | failed)
• created_at (UTC timestamp)

Retrieve job state

curl -sS "http://localhost:8000/v1/tests/runs/<job_id>" \
  -H "X-API-Key: <optional>"

Response includes:

• status: current job state
• result: aggregated metrics (when succeeded)
• error: error details (when failed)
• error_notice: compact user-facing error message (when failed)

Artifacts

List objects available for a job:

curl -sS "http://localhost:8000/v1/tests/runs/<job_id>/artifacts" \
  -H "X-API-Key: <optional>"

Resolve a presigned download URL for a specific object:

curl -sS "http://localhost:8000/v1/tests/runs/<job_id>/artifacts/<path>" \
  -H "X-API-Key: <optional>"

The download endpoint returns a JSON payload containing download_url and does not emit redirects.

MCP interface

The MCP server is mounted into the FastAPI application (default mount path: /mcp) and uses Streamable HTTP transport.

Exposed tools:

• create_llamator_run: submits a job, waits for completion, returns aggregated metrics and (if available) a presigned URL for artifacts.zip.
• get_llamator_run: returns aggregated metrics for a finished job and the optional artifacts archive URL.

Both tools return a consistent response schema:

{
  "job_id": "string",
  "aggregated": {
    "attack_name": {
      "metric": 0
    }
  },
  "artifacts_download_url": "string or null",
  "error_notice": "string or null"
}

Protocol notes, headers, and examples are documented in DOCUMENTATION.md.

Security model

• If LLAMATOR_MCP_API_KEY is empty, authentication is disabled.
• If configured, protected HTTP routes and the MCP app require X-API-Key: <value>.

Local development

Install dependencies:

poetry install

Run the API server:

uvicorn llamator_mcp_server.main:app --host 0.0.0.0 --port 8000

Run the worker:

arq llamator_mcp_server.worker_settings.WorkerSettings

Tutorial

A Jupyter notebook with step-by-step examples is available at notebooks/llamator_mcp_server_tutorial.ipynb. It demonstrates:

• HTTP API usage with curl
• MCP JSON-RPC protocol interaction
• Polling for job completion
• Artifacts retrieval

Tests

Integration tests are located in llamator-mcp-server/tests and rely on tests/.env.test.

Run:

pytest -q

License 📜

This project is licensed under the terms of the Creative Commons Attribution-ShareAlike 4.0 International license. See the LICENSE file for details.