Create a type testing framework for Python (typemap-testing)

[AliiiBenn]

Title

Create a type testing framework for Python (typemap-testing)

Description

Build a comprehensive testing framework for Python type manipulation, similar to how TypeScript handles type testing. This package would provide tools to validate, test, and assert types at runtime and compile-time.

Motivation

The Problem

Python lacks proper type testing tools compared to TypeScript:

TypeScript	Python	Status
assertion<T, Expected>	N/A	Missing
@ts-expect-error	# type: ignore	Incomplete
Type equality tests	== doesn't work	Missing
dtslint	pytest-mypy-plugins	Limited

Why This Matters

1. Validation: Without proper testing, type utilities like Pick, Omit, KeyOf can't be validated
2. Regression prevention: Changes to type manipulation could break existing types
3. Documentation by tests: Tests serve as executable documentation
4. DX improvement: Developers need confidence their types work correctly

Proposed Solution

Package Structure

typemap-testing/
├── typemap_test/
│   ├── __init__.py
│   ├── assert_type.py      # assert_type[T, Expected]
│   ├── expect_error.py     # expect_error() function
│   ├── type_eq.py         # Type equality checker
│   ├── diff.py            # Type difference visualizer
│   ├── validators/        # Built-in type validators
│   │   ├── __init__.py
│   │   ├── pick_omit.py
│   │   └── keyof.py
│   └── plugins/
│       ├── __init__.py
│       └── mypy_plugin.py # Mypy integration
├── tests/
│   ├── test_assert_type.py
│   ├── test_pick_omit.py
│   ├── test_keyof.py
│   └── integration/
│       └── test_mypy.py
├── docs/
│   ├── getting-started.md
│   ├── assert-type.md
│   └── mypy-plugin.md
├── pyproject.toml
├── README.md
└── LICENSE

Core Features

1. Runtime Type Assertion

from typemap_test import assert_type

# Basic usage
assert_type[Pick[User, 'name'], {name: str}]

# With custom error message
assert_type[Pick[User, 'name'], {name: str}], "Pick should work"

# Tuple types
assert_type[Length[tuple[int, str, bool]], Literal[3]]

2. Expect Error (NOT a decorator - Python limitation)

Important: In Python, decorators cannot be applied to type aliases or variables (unlike TypeScript). Use functions instead:

from typemap_test import expect_error, assert_type_error

# Function style - wrap in lambda
expect_error("Cannot pick non-existent key", lambda: Pick[User, 'nonexistent_field'])

# Or using the dedicated function
Invalid = assert_type_error(
    "Cannot pick non-existent key",
    Pick[User, 'nonexistent']
)

# For runtime errors with eval_typing
expect_error("Invalid key", lambda: eval_typing(Invalid))

Alternative: Context manager style

from typemap_test import expect_error_ctx

# This block should raise an error
with expect_error_ctx("Cannot pick"):
    result = eval_typing(Pick[User, 'nonexistent'])

3. Type Equality

from typemap_test import type_eq

# Check if two types are equal
assert type_eq(Pick[User, 'name'], {name: str})
assert not type_eq(Pick[User, 'name'], {name: int})

# Works with complex types
assert type_eq(
    Partial[User],
    {name: str | None, email: str | None}
)

4. Mypy Integration

# pytest setup
# conftest.py
import pytest
from typemap_test.mypy_plugin import configure_mypy

@pytest.fixture(scope="session")
def mypy_config():
    return configure_mypy(
        strict=True,
        plugins=["typemap_test.mypy_plugin"]
    )

# pyproject.toml
[tool.mypy]
plugins = ["typemap_test.mypy_plugin"]

[tool.typemap_test]
strict = true
show_error_context = true

5. Property-Based Testing

from typemap_test import assert_type
from hypothesis import given, strategies as st

@given(keys=st.sets(st.sampled_from(['a', 'b', 'c'])))
def test_pick_commutative(keys):
    # Pick[Pick[T, K1], K2] == Pick[T, K1 & K2]
    inner = Pick[User, keys]
    result = Pick[inner, list(keys)[:1]]
    expected = Pick[User, list(keys)[:1]]
    assert_type[result, expected]

Implementation Plan

Phase 1: Core (Priority: High)

• Create package structure
• Implement assert_type (runtime)
• Implement type_eq
• Implement expect_error function
• Implement expect_error_ctx context manager
• Basic test suite

Phase 2: Mypy Integration (Priority: High)

• Create mypy plugin
• Implement assert_type checker for mypy
• Implement error checking for mypy
• Integration tests

Phase 3: Advanced Features (Priority: Medium)

• Type diff visualizer
• Property-based testing helpers
• TypeScript equivalence testing
• Documentation

Phase 4: Polish (Priority: Low)

• Ruff integration (investigate)
• Pyright integration
• CI/CD setup

Technical Considerations

Important: Python vs TypeScript Syntax

In Python, decorators cannot be applied to type aliases or variables:

# ❌ This does NOT work in Python
@tm_expect_error("Cannot pick")
type Invalid = Pick[User, 'nonexistent']

# ✅ Use functions instead
Invalid = assert_type_error("Cannot pick", Pick[User, 'nonexistent'])
expect_error("Cannot pick", lambda: eval_typing(Pick[User, 'nonexistent']))

Mypy Plugin Architecture

The mypy plugin should:

1. Intercept assert_type[T, Expected] calls
2. Use mypy's type comparison APIs
3. Generate detailed error messages with diff
4. For error checking, validate that expressions do/don't raise errors

Type Equality at Runtime

Using the existing type_eval from typemap:

• Evaluate both types
• Compare the results using structural equality
• Handle recursive types properly

References

• TypeScript assertion types: https://www.typescriptlang.org/docs/handbook/2/conditional-types.html#inferring-with-inference-within-conditional-types
• pyright strict mode: https://github.com/microsoft/pyright/blob/main/docs/type-evaluation.md
• pytest-mypy-plugins: https://github.com/typeddjango/pytest-mypy-plugins
• Django's static type testing: https://github.com/typeddjango/django-stubs

Acceptance Criteria

• assert_type works at runtime
• assert_type works with mypy (generates errors)
• expect_error function works
• expect_error_ctx context manager works
• type_eq correctly identifies equal/different types
• Tests for Pick, Omit, Partial, Required, KeyOf
• Documentation complete
• CI passes

Questions for Discussion

1. Should this be a separate package or part of typemap?
2. How to handle differences between mypy and pyright?
3. What level of strictness for the default configuration?