PyQualify - AI-Powered QA & Security Analysis Tool

PyQualify is a command-line tool that performs automated quality assurance and security analysis across three modes: Web, Code, and API. It leverages LLM-based intelligence to classify findings, produce severity ratings, and generate actionable recommendations.

Features

Web Analysis - Security headers, form CSRF detection, SEO completeness, accessibility compliance, performance signals, broken link detection, CAPTCHA checks, HTTP request smuggling, case-sensitivity bypass detection, JSON hijacking, DOM-based XSS, open redirect, and server version disclosure
Code Analysis - Security vulnerabilities, bug risk detection, code quality metrics, test gap identification, dependency risk assessment, audit log checks, case-sensitivity bypass detection, known CVE detection, and password policy enforcement
API Analysis - Authentication enforcement, response integrity, schema conformance, injection vector testing, rate limiting verification, audit log manipulation, CAPTCHA bypass, HTTP request smuggling, case-sensitivity bypass, JSON hijacking, open redirect, server version disclosure, internal IP leakage, and application-level DoS
AI-Powered Classification - Findings are processed through an LLM for intelligent severity assignment, CWE/OWASP mapping, and contextual recommendations
Scoring & Grading - Numeric score (0-100), letter grade (A-F), and risk level for every analysis run
TUI Dashboard - Full-screen Textual dashboard with live metrics, scrollable issues table, real-time log feed, and issue detail panel (pyqualify dashboard)
PDF Reports - Professionally formatted PDF reports saved automatically to ~/Documents/PyQualify/
Tool Filtering - Enable or disable individual checks per run with --only and --disable
Color-Coded CLI Output - Severity-based coloring with graceful fallback for terminals without color support
Hierarchical Configuration - TOML config file, environment variables, and CLI arguments with clear precedence
Interactive Config Editor - Nano-like terminal editor for managing configuration
Docker Support - Official Docker image available on Docker Hub with automatic report saving to host Documents folder

Requirements

uv - used for Python version management, dependency locking, and running the project
An API key for one of the supported AI providers:
- OpenAI - GPT-4o, GPT-4-turbo, GPT-3.5-turbo, ...
- Anthropic - Claude 3.5 Sonnet, Claude 3 Opus, ... (uv add anthropic required)
- Google - Gemini 2.0 Flash, Gemini 1.5 Pro, ...
- Groq - Llama 3.3 70B, Mixtral 8x7B, ... (free API keys available at console.groq.com/keys)
For the TUI dashboard: a terminal of at least 80×24 characters

Installation

Via pip (recommended)

pip install pyqualify

Via uv

uv add pyqualify

From source

git clone <repo-url> pyqualify
cd pyqualify

# Install dependencies
uv sync

# Optional: add Anthropic SDK for Claude models
uv add anthropic

Via Docker

# Pull the latest image
docker pull enzox0/pyqualify:latest

# Or build locally
docker build -t enzox0/pyqualify:latest .

First Run - Setup

Before running any analysis, configure your AI provider:

uv run pyqualify setup

  Choose a provider:

  1  OpenAI      GPT-4o, GPT-4-turbo, GPT-3.5-turbo, ...
  2  Anthropic   Claude 3.5 Sonnet, Claude 3 Opus, ...
  3  Google      Gemini 2.0 Flash, Gemini 1.5 Pro, ...
  4  Groq        Llama 3.3 70B, Mixtral 8x7B, ...

  Provider [1/2/3/4]: 1

  OpenAI API key
  > sk-...

  Model (default: gpt-4o)
  >

  ✔ Configuration saved.
  provider   OpenAI  model   gpt-4o

Configuration is stored in ~/.pyqualify/config.toml. Sensitive values (API keys) are masked when listed. You can re-run setup at any time to switch providers or update your key.

Quick Start

Run setup first, then launch the interactive menu:

pyqualify setup   # one-time configuration
pyqualify         # interactive mode selector

  Select analysis mode:

  1  Web     Analyze a website for security, SEO, accessibility & performance
  2  Code    Analyze source code for vulnerabilities, quality & test gaps
  3  API     Analyze REST API endpoints for security & integrity

  Mode [1/2/3]: 1

  Target URL (e.g. https://example.com)
  > https://example.com

  Save PDF report to Documents/PyQualify/? [Y/n]: y

  Analyzing web page...

You can also pass arguments directly for scripting:

uv run pyqualify web https://example.com
uv run pyqualify code ./src --pdf
uv run pyqualify api https://api.example.com --json

TUI Dashboard

The dashboard command launches a full-screen Textual interface with live updates as analysis runs.

# Open the dashboard in idle state
pyqualify dashboard

# Auto-start analysis on launch
pyqualify dashboard web https://example.com
pyqualify dashboard code ./src
pyqualify dashboard api https://api.example.com

The dashboard requires a terminal of at least 80×24 characters. It exits with a non-zero code if the terminal is too small.

Panels

Panel	Description
Header	Tool name, version, and three status indicators (AI engine, analyzer, analysis state)
Metrics	Live score, grade, risk level, and per-severity issue counts
Issues	Scrollable table of discovered issues, sorted by severity
Issue Detail	Full issue details shown when pressing Enter on a row
Log	Real-time log feed from the analysis engine
Navigation	Context-sensitive keyboard shortcut hints

Keyboard Shortcuts

Key	Action
`q`	Quit
`Ctrl+C`	Force quit (exit code 130)
`?`	Show help overlay
`1`	Focus Metrics panel
`2`	Focus Issues panel
`3`	Focus Log panel
`Enter`	Show issue detail (when Issues panel is focused)
`Escape`	Close issue detail panel

Configuration

PyQualify uses a layered configuration system with the following precedence (highest wins):

CLI arguments
Environment variables (prefixed with PYQUALIFY_)
Configuration file (~/.pyqualify/config.toml)

The setup command writes to the config file. You can also manage values manually:

Setting Configuration

# Switch model after setup
uv run pyqualify config set model claude-3-5-sonnet-20241022

# List all configuration (sensitive values are masked)
uv run pyqualify config list

# Open interactive editor (not available on Windows - use config set instead)
uv run pyqualify config edit

Environment Variables

Any environment variable prefixed with PYQUALIFY_ is recognized:

export PYQUALIFY_API_KEY=sk-your-key
export PYQUALIFY_PROVIDER=openai        # openai | anthropic | google | groq
export PYQUALIFY_MODEL=gpt-4o
export PYQUALIFY_TIMEOUT=60
export PYQUALIFY_RATE_LIMIT_BURST=50
export PYQUALIFY_RATE_LIMIT_WINDOW=10

# Docker-specific
export PYQUALIFY_CONFIG_DIR=/app/config   # Path to config directory
export PYQUALIFY_REPORT_DIR=/app/reports  # Path to save reports

Docker Usage

First Run - Setup

Configure your AI provider (one-time setup):

# Windows (PowerShell)
docker run -v ${env:USERPROFILE}\.pyqualify:/app/config -it enzox0/pyqualify:latest setup

# Linux/macOS
docker run -v ~/.pyqualify:/app/config -it enzox0/pyqualify:latest setup

Run Analysis

Run an analysis with PDF report saved to your Documents folder:

# Windows (PowerShell)
docker run -v ${env:USERPROFILE}\.pyqualify:/app/config -v ${env:USERPROFILE}\Documents\PyQualify:/app/reports -it enzox0/pyqualify:latest web https://example.com --pdf

# Linux/macOS
docker run -v ~/.pyqualify:/app/config -v ~/Documents/PyQualify:/app/reports -it enzox0/pyqualify:latest web https://example.com --pdf

Using Docker Compose

# Build (if needed) and run setup
docker-compose build
docker-compose run --rm pyqualify setup

# Run analysis
docker-compose run --rm pyqualify web https://example.com --pdf

Configuration File

Located at ~/.pyqualify/config.toml:

provider    = "openai"
api_key     = "sk-your-openai-key"
model       = "gpt-4o"
timeout     = 60
max_retries = 3
retry_delay = 2.0

Commands

Command	Description
`pyqualify setup`	Run first - configure AI provider and API key
`pyqualify web <url>`	Analyze a web page
`pyqualify code <path>`	Analyze source code (file or directory)
`pyqualify api <base_url>`	Analyze API endpoints
`pyqualify dashboard [mode] [target]`	Launch the TUI dashboard
`pyqualify tools [category]`	List available tools for a category
`pyqualify config set <key> <value>`	Set a configuration value
`pyqualify config list`	List all configuration
`pyqualify config edit`	Open interactive config editor

Common Options

Option	Description
`--pdf`	Save a PDF report to `~/Documents/PyQualify/`
`--json`	Output raw JSON to stdout
`--only <tools>`	Run only the specified tools (comma-separated or repeated)
`--disable <tools>`	Skip the specified tools (comma-separated or repeated)
`--version`	Show version
`--help`	Show help

Tool Filtering

Each analysis mode has a set of named tools that can be individually enabled or disabled:

# List all available tools
uv run pyqualify tools

# List tools for a specific mode
uv run pyqualify tools web
uv run pyqualify tools code
uv run pyqualify tools api

# Run only specific tools
uv run pyqualify web https://example.com --only security-headers,seo
uv run pyqualify api https://api.example.com --only authentication,injection

# Skip specific tools
uv run pyqualify code ./src --disable test-gaps,quality

Web tools

Tool	Description
`security-headers`	Check for missing or misconfigured security headers
`forms`	Check forms for CSRF tokens and sensitive autocomplete
`seo`	Check for missing SEO elements (title, meta, OG tags)
`accessibility`	Check accessibility compliance (alt, headings, ARIA, labels)
`performance`	Check performance signals (inline scripts, lazy loading, load time)
`links`	Verify links for broken URLs and suspicious domains
`captcha`	Detect missing or weak CAPTCHA on sensitive forms
`smuggling-headers`	Check for Transfer-Encoding/Content-Length co-existence
`case-sensitivity`	Check if URL path casing changes bypass access controls
`json-hijacking`	Detect JSON hijacking vectors in HTML scripts
`open-redirect`	Detect open redirect parameters in forms and links
`server-version-disclosure`	Detect server version/technology in response headers
`dom-xss`	Detect DOM-based XSS sinks reading from URL fragments or query strings

Code tools

Tool	Description
`security`	Detect injection vulnerabilities, hardcoded secrets, insecure patterns
`bug-risks`	Detect null dereferences, uncaught exceptions, race conditions
`quality`	Detect dead code, duplicated logic, high complexity, magic numbers
`test-gaps`	Detect missing tests, weak assertions, untested branches
`dependencies`	Detect typosquatting, deprecated packages, wildcard imports
`audit-log`	Detect log injection, log suppression, audit log deletion
`case-sensitivity`	Detect missing case normalization in auth/routing comparisons
`known-vulnerabilities`	Detect imports of packages with known CVEs
`password-policy`	Detect weak or missing password policy enforcement

API tools

Tool	Description
`authentication`	Test authentication enforcement (no creds, expired/malformed tokens)
`response-integrity`	Test for information leakage and status code mismatches
`injection`	Test SQL, NoSQL, and command injection via payloads
`rate-limiting`	Test rate limiting by sending burst requests
`schema-conformance`	Validate response schema consistency across requests
`audit-log-manipulation`	Test for log injection via headers/params
`captcha-bypass`	Test if auth endpoints work without CAPTCHA
`http-request-smuggling`	Test for CL.TE / TE.CL request smuggling
`case-sensitivity`	Test for case-sensitive route/auth bypass
`json-hijacking`	Test for unprotected top-level JSON arrays
`open-redirect`	Test for open redirect via common redirect parameters
`server-version-disclosure`	Detect server version/technology in response headers
`internal-ip-leakage`	Detect private IP addresses and internal hostnames in responses
`application-dos`	Test for missing payload size and JSON depth limits

Output

CLI Output

Results are displayed with color-coded severity levels:

CRITICAL - Immediate action required
HIGH - Should be addressed soon
MEDIUM - Moderate risk
LOW - Minor improvement opportunity
INFO - Informational finding

A summary block shows the overall Score, Grade, and Risk Level before individual issues.

JSON Output

{
  "score": 72,
  "grade": "C",
  "risk_level": "high",
  "issues": [
    {
      "check": "missing-csp-header",
      "severity": "high",
      "title": "Missing Content-Security-Policy Header",
      "description": "The web server does not include a CSP header...",
      "evidence": "Response headers: {...}",
      "recommendation": "Add CSP header with restrictive directives...",
      "cwe": "CWE-693",
      "owasp": "A05:2021"
    }
  ],
  "summary": "Analysis found 12 issues across security and quality categories...",
  "metadata": {
    "timestamp": "2024-01-15T10:30:00Z",
    "target": "https://example.com",
    "mode": "web"
  }
}

PDF Report

The --pdf flag generates a professionally formatted PDF saved to ~/Documents/PyQualify/<target>/<timestamp>.pdf. The report includes:

Cover page with score, grade, and risk level
Executive summary from the AI engine
Severity breakdown
Full issues table with CWE/OWASP references
Top 5 prioritized recommendations

Scoring

The scoring algorithm starts at 100 and subtracts penalties per issue:

Severity	Penalty
CRITICAL	-20
HIGH	-10
MEDIUM	-5
LOW	-2
INFO	0

The score is clamped to 0-100. Letter grades map as:

Grade	Score Range
A	90-100
B	80-89
C	70-79
D	60-69
F	0-59

Architecture

PyQualify follows a layered pipeline architecture:

CLI Layer -> Analysis Engine -> AI Engine -> Scoring -> Report Generator
                                                     -> TUI Dashboard

Key design decisions:

Dependency Injection - Lightweight custom container for testability
Protocol-based interfaces - All major components implement Python protocols
Async I/O - httpx.AsyncClient for all network operations
Modular analyzers - Each analysis mode is an independent module with individually toggleable tools
Textual TUI - Full-screen dashboard built with Textual for live analysis monitoring

Development

# Install dev dependencies
uv sync --extra dev

# Run tests
uv run pytest

# Run tests with coverage
uv run pytest --cov=pyqualify --cov-report=html

# Run a specific test file
uv run pytest tests/test_scoring_engine.py

# Add a new dependency
uv add some-package

# Update the lock file
uv lock

Project Structure

pyqualify/
    pyproject.toml
    uv.lock
    README.md
    pyqualify/
        __init__.py
        __main__.py
        models.py
        container.py
        utils.py
        tool_registry.py
        cli/
        analyzers/
        ai/
        reporting/
        scoring/
        config/
        logging/
        tui/
    tests/
    docs/

Dependencies

click - CLI framework
httpx - Async HTTP client
beautifulsoup4 + lxml - HTML parsing
openai - LLM integration (also used for Google Gemini and Groq via compatible API)
jinja2 - HTML template rendering
reportlab - PDF report generation
textual - TUI dashboard framework

License

MIT

Name		Name	Last commit message	Last commit date
Latest commit History 73 Commits
.githooks		.githooks
.github		.github
docs		docs
pyqualify		pyqualify
tests		tests
.dockerignore		.dockerignore
.gitignore		.gitignore
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
docker-compose.yml		docker-compose.yml
publish.bat		publish.bat
pyproject.toml		pyproject.toml
setup-githooks.bat		setup-githooks.bat
uv.lock		uv.lock

Folders and files

Latest commit

History

Repository files navigation

PyQualify - AI-Powered QA & Security Analysis Tool

Features

Requirements

Installation

Via pip (recommended)

Via uv

From source

Via Docker

First Run - Setup

Quick Start

TUI Dashboard

Panels

Keyboard Shortcuts

Configuration

Setting Configuration

Environment Variables

Docker Usage

First Run - Setup

Run Analysis

Using Docker Compose

Configuration File

Commands

Common Options

Tool Filtering

Web tools

Code tools

API tools

Output

CLI Output

JSON Output

PDF Report

Scoring

Architecture

Development

Project Structure

Dependencies

License

About

Topics

Resources

License

Code of conduct

Contributing

Security policy

Uh oh!

Stars

Watchers

Forks

Releases 6

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages