Skip to content

Latest commit

 

History

History
218 lines (156 loc) · 6.05 KB

TESTING_GUIDE.md

File metadata and controls

218 lines (156 loc) · 6.05 KB

Testing Guide for Abstract Data Organization Profile

Overview

This repository now includes a comprehensive test suite that validates the configuration files and GitHub Actions workflow using property-based testing with Hypothesis.

What Was Changed

Based on the git diff between the current branch and main:

  1. .github/workflows/update-profile.yml - Modified to include --config .markdownlint.json flag
  2. .markdownlint.json - New configuration file with custom markdownlint rules

Test Suite Created

A comprehensive test suite has been created in the tests/ directory with 1,615 lines of test code across 4 test files.

Test Files

File Lines Purpose
test_markdownlint_config.py 334 Tests for .markdownlint.json configuration
test_workflow_config.py 446 Tests for GitHub Actions workflow
test_integration.py 500 Integration tests between files
test_markdown_validation.py 335 Markdown validation logic tests
Total 1,615 Complete test coverage

Test Coverage

The test suite includes:

  • ✅ 40+ test classes
  • ✅ 150+ test methods
  • ✅ 40+ property-based tests using Hypothesis
  • ✅ 3 state machines for stateful testing
  • ✅ 100% Hypothesis usage as required
  • ✅ Happy path scenarios
  • ✅ Edge cases and boundary conditions
  • ✅ Failure conditions
  • ✅ Integration testing
  • ✅ Real-world scenarios

Quick Start

1. Install Dependencies

pip install -r tests/requirements.txt

2. Run All Tests

pytest tests/ -v

3. Run Quick Test Script

./tests/run_tests.sh

Test Organization

test_markdownlint_config.py

Tests the .markdownlint.json configuration file:

  • Structure validation - JSON format, required fields, data types
  • Property tests - Line length, rule configurations, boolean values
  • Stateful tests - Configuration consistency via state machine
  • Edge cases - Parsing, boundaries, comparison operations
  • Integration - Workflow references, path correctness

Key highlights:

  • 7 test classes
  • State machine with rules and invariants
  • Property-based testing for line length thresholds
  • Validation of disabled rules (MD033, MD041, MD022, MD032)

test_workflow_config.py

Tests the .github/workflows/update-profile.yml workflow:

  • Workflow structure - YAML format, required keys, job configuration
  • Markdownlint step - Config flag usage, file references, command structure
  • Step ordering - Checkout first, Node.js setup, validation before commit
  • Property tests - Step counts, field validation, naming conventions
  • Stateful tests - Workflow consistency via state machine
  • Integration - Config file existence, path validation

Key highlights:

  • 8 test classes
  • State machine for workflow validation
  • Command structure verification
  • Secrets and credentials validation

test_integration.py

Integration tests between configuration and workflow:

  • Config-workflow integration - File references, path consistency
  • Configuration consistency - Token presence, rule validity
  • Path references - Relative paths, location correctness
  • Command-line integration - Component order, flag usage
  • Stateful tests - Integration state machine
  • Semantic consistency - Rule meanings, configuration appropriateness

Key highlights:

  • 11 test classes
  • Cross-file validation
  • Command-line argument testing
  • Semantic validation

test_markdown_validation.py

Tests markdown validation logic:

  • Validation logic - Line length thresholds, multi-line validation
  • Rule exemptions - Code blocks, tables excluded from line length
  • Disabled rules - HTML allowed, heading rules, spacing rules
  • Configuration effectiveness - Rule enforcement, default behavior
  • Markdown patterns - Headings, lists, links
  • Real-world scenarios - GitHub profiles, badges, SVG embeds

Key highlights:

  • 7 test classes
  • Real-world GitHub profile patterns
  • Badge and SVG handling
  • Exemption validation

Hypothesis Usage

All tests use Hypothesis for property-based and stateful testing as required.

Property-Based Testing

Tests validate properties across wide input ranges:

@given(st.integers(min_value=1, max_value=10000))
def test_line_length_property_positive_values(self, line_length):
    """Property: Line length must always be positive."""
    ...

Stateful Testing

State machines test complex behavioral properties:

class MarkdownlintConfigStateMachine(RuleBasedStateMachine):
    @rule()
    def verify_config_structure(self):
        """Rule: Configuration structure should always be valid."""
        ...
    
    @invariant()
    def config_is_valid_json_structure(self):
        """Invariant: Config should always be a dictionary."""
        ...

Hypothesis Strategies Used

  • st.integers() - Integer property testing
  • st.text() - Text generation with various alphabets
  • st.booleans() - Boolean property testing
  • st.sampled_from() - Testing with specific values
  • st.lists() - List generation and validation
  • st.dictionaries() - Dictionary structure testing
  • st.tuples() - Tuple testing
  • st.one_of() - Union type testing

Running Tests

Basic Commands

# Run all tests
pytest tests/ -v

# Run specific test file
pytest tests/test_markdownlint_config.py -v

# Run with Hypothesis statistics
pytest tests/ --hypothesis-show-statistics

# Run only property-based tests
pytest tests/ -k "property"

# Run only stateful tests
pytest tests/ -k "StateMachine"

# Run with verbose output
pytest tests/ -vv

# Run and stop on first failure
pytest tests/ -x

Advanced Commands

# Run with specific Hypothesis seed for reproducibility
pytest tests/ --hypothesis-seed=12345

# Increase Hypothesis examples for thorough testing
pytest tests/ --hypothesis-verbosity=verbose

# Run with coverage report
pytest tests/ --cov=. --cov-report=html

# Run in parallel (if pytest-xdist installed)
pytest tests/ -n auto

Test Files Structure