Skip to content

troubleshooting-guide.org

Latest commit

 

History

History
1261 lines (965 loc) · 30 KB

troubleshooting-guide.org

File metadata and controls

1261 lines (965 loc) · 30 KB

Python Package Development - Troubleshooting Guide

A practical troubleshooting companion to the Python Wheel Distributions Guide. This guide focuses on solving real problems you’ll encounter during Python package development.

Contents

Quick Diagnostic Reference

Symptom → Likely Cause

SymptomMost Common CauseQuick Fix
ModuleNotFoundError after installNot in virtual environmentsource .venv/bin/activate
Changes not taking effectNot in editable modepip install -e .
__version__ shows “unknown”Package not installedpip install -e .
Command not foundMissing [project.scripts]Check pyproject.toml
Tests can’t import packagePackage not installedpip install -e .
Import works in dev, fails in wheelMissing __init__.pyAdd __init__.py files
Old code still runningStale .pyc cachefind . -name "*.pyc" -delete
Build includes test filesWrong package discoveryCheck [tool.setuptools.packages.find]

Development Environment Problems

Virtual Environment Not Activated

Symptoms:

  • Package installs to system Python
  • ModuleNotFoundError even after install
  • Wrong Python version running

Diagnosis:

# Check which Python you're using
which python
python --version

# Check if in virtual environment
echo $VIRTUAL_ENV  # Should show path to your .venv

Solutions:

# Activate virtual environment
source .venv/bin/activate  # Linux/macOS
.venv\Scripts\activate     # Windows cmd
.venv\Scripts\Activate.ps1 # Windows PowerShell

# Verify activation
which python  # Should point to .venv/bin/python

Wrong Python Version

Problem: Using Python 2.x or incompatible Python 3.x version

Diagnosis:

python --version
python3 --version

# Check what your project requires
grep "requires-python" pyproject.toml

Solutions:

# Create venv with specific Python version
python3.9 -m venv .venv
python3.10 -m venv .venv

# On systems with multiple Python versions
/usr/bin/python3.11 -m venv .venv

pip Installing to Wrong Location

Symptoms:

  • Install seems to work but package not importable
  • Package installed but in different Python

Diagnosis:

# Check where pip installs
pip show pip | grep Location

# Check all Python paths
python -c "import sys; print('\n'.join(sys.path))"

# Find where your package is (if anywhere)
python -c "import your_package; print(your_package.__file__)"

Solutions:

# Always use python -m pip instead of bare pip
python -m pip install -e .

# Or explicitly use the venv pip
.venv/bin/pip install -e .

PATH Configuration Issues

Problem: Installed CLI commands not found

Diagnosis:

echo $PATH

# Find where command was installed
find ~/.local -name "your-command"
find .venv -name "your-command"

# For pipx
pipx list

Solutions:

# Add Python user scripts to PATH (in ~/.bashrc or ~/.zshrc)
export PATH="$HOME/.local/bin:$PATH"

# For pipx
pipx ensurepath

# Then reload shell or source config
source ~/.bashrc

Package Installation Issues

Module Not Found After Installation

Problem: ModuleNotFoundError: No module named 'your_package'

Root Causes & Solutions:

  1. Not in virtual environment (MOST COMMON) 
    # Verify you're in venv
echo $VIRTUAL_ENV

# Activate if not
source .venv/bin/activate
  2. Wrong directory during install 
    # Must be in directory containing pyproject.toml
ls pyproject.toml  # Should exist

# Install from correct location
pip install -e .
  3. Incorrect package configuration 
    # In pyproject.toml - must have both sections
[tool.setuptools]
package-dir = {"" = "src"}

[tool.setuptools.packages.find]
where = ["src"]
  4. Missing __init__.py files 
    # Every package directory needs __init__.py
src/
└── your_package/
    ├── __init__.py      # REQUIRED
    └── subpackage/
        ├── __init__.py  # REQUIRED
        └── module.py

Verification:

# Check if package is installed
pip list | grep your-package

# Check installation details
pip show your-package

# Verify Python can find it
python -c "import your_package; print(your_package.__file__)"

# Test import manually
python -c "import your_package; print(dir(your_package))"

Editable Install Not Working

Symptoms:

  • pip install -e . succeeds but package not importable
  • Changes not taking effect

Diagnosis:

# Check if actually installed in editable mode
pip show your-package | grep Location
# Should point to your source directory, not site-packages

# Check for .egg-link file
ls .venv/lib/python*/site-packages/*.egg-link

Solutions:

# Clean and reinstall
pip uninstall your-package
rm -rf *.egg-info
pip install -e .

# Force reinstall
pip install -e . --force-reinstall --no-deps

# Verify editable mode
pip list -v | grep your-package  # Should show "Editable project location"

Changes Not Taking Effect

Problem: Code changes don’t appear when running/testing

Diagnosis:

# Are you in editable mode?
pip show your-package | grep Location

# Check for cached bytecode
find . -name "*.pyc" | head
find . -type d -name "__pycache__" | head

Solutions:

  1. Ensure editable mode is active: 
    pip install -e .
  2. Clear Python cache: 
    # Remove all .pyc files and __pycache__ directories
find . -type f -name "*.pyc" -delete
find . -type d -name "__pycache__" -exec rm -rf {} +

# Or use pyclean (if available)
pyclean .
  3. Restart Python interpreter:
    • If using interactive Python, quit and restart
    • If using Jupyter, restart kernel
    • Modules are cached after first import
  4. Check you’re editing the right file: 
    # Find which file Python is actually importing
python -c "import your_package.module; print(your_package.module.__file__)"

Dependencies Not Installing Automatically

Problem: Package installs but required dependencies missing

Diagnosis:

# Check what dependencies are declared
grep -A 10 "dependencies" pyproject.toml

# Check what's actually installed
pip list

Solutions:

  1. Ensure dependencies listed in pyproject.toml: 
    [project]
dependencies = [
    "requests>=2.28.0",
    "click>=8.0.0",
]
  2. For development dependencies: 
    [project.optional-dependencies]
dev = [
    "pytest>=7.0",
    "pytest-cov>=3.0",
]
    
    # Install with dev dependencies
pip install -e ".[dev]"
  3. Reinstall to pick up new dependencies: 
    pip install -e . --force-reinstall

Package Structure and Configuration

Missing __init__.py Files

Problem: Package structure seems correct but imports fail

Diagnosis:

# Find all Python packages without __init__.py
find src -type d -name "*" | while read dir; do
    if [ -n "$(find "$dir" -maxdepth 1 -name "*.py" -not -name "__init__.py")" ]; then
        if [ ! -f "$dir/__init__.py" ]; then
            echo "Missing __init__.py: $dir"
        fi
    fi
done

Solution:

Every directory that should be a Python package MUST have __init__.py:

src/
└── your_package/
    ├── __init__.py          # REQUIRED - can be empty
    ├── module.py
    └── subpackage/
        ├── __init__.py      # REQUIRED - can be empty
        └── submodule.py

# Create missing __init__.py files
touch src/your_package/__init__.py
touch src/your_package/subpackage/__init__.py

Note: While Python 3.3+ supports namespace packages without __init__.py, it’s best practice to include them for clarity and compatibility.

Incorrect pyproject.toml Configuration

Problem: Package not discovered during build

Common Configuration Errors:

  1. Incorrect package-dir: 
    # WRONG - missing package-dir
[tool.setuptools.packages.find]
where = ["src"]

# CORRECT - both sections needed
[tool.setuptools]
package-dir = {"" = "src"}

[tool.setuptools.packages.find]
where = ["src"]
  2. Wrong package name in version(): 
    # In __init__.py - name must match [project] name exactly
from importlib.metadata import version

# WRONG - name mismatch
__version__ = version("my_package")  # When project name is "my-package"

# CORRECT - exact match with pyproject.toml [project] name
__version__ = version("my-package")
  3. Invalid TOML syntax: 
    # Validate your pyproject.toml
pip install validate-pyproject
validate-pyproject pyproject.toml

Verification:

# Check what packages setuptools will find
python -c "import setuptools; print(setuptools.find_packages('src'))"

# Should output: ['your_package', 'your_package.subpackage', ...]

Package Discovery Problems

Problem: Some modules missing from built wheel

Diagnosis:

# Check what's in your wheel
unzip -l dist/*.whl

# Compare with source
find src -name "*.py" | sort

Common Issues:

  1. Packages not auto-discovered: 
    # Explicit include if auto-discovery fails
[tool.setuptools.packages.find]
where = ["src"]
include = ["your_package*"]  # Explicitly include
exclude = ["tests*"]          # Explicitly exclude
  2. Missing data files: 
    # Include non-Python files
[tool.setuptools.package-data]
your_package = [
    "*.txt",
    "*.json",
    "data/**/*",
    "templates/**/*.html",
]

Version Showing as “unknown”

Problem: __version__ returns “unknown” or raises PackageNotFoundError

Root Causes & Solutions:

  1. Package not installed (MOST COMMON) 
    pip install -e .
  2. Package name mismatch: 
    # In __init__.py
from importlib.metadata import version, PackageNotFoundError

try:
    # This name MUST match [project] name in pyproject.toml EXACTLY
    __version__ = version("exact-package-name-here")
except PackageNotFoundError:
    __version__ = "unknown"
    
    # In pyproject.toml
[project]
name = "exact-package-name-here"  # Must match version() call
  3. Wrong Python environment: 
    # Verify environment
which python
echo $VIRTUAL_ENV

# Reinstall in correct environment
pip install -e .
  4. Using wrong metadata API: 
    # WRONG - old API (deprecated)
import pkg_resources
__version__ = pkg_resources.get_distribution("package").version

# CORRECT - modern API (Python 3.8+)
from importlib.metadata import version
__version__ = version("package")

Building and Distribution

Stale Build Artifacts

Problem: Old code persists in built wheels despite source changes

Symptoms:

  • Changes not appearing in wheel
  • Old files still in distribution
  • Build seems to use cached data

Solution:

# Complete cleanup of all build artifacts
rm -rf build/
rm -rf dist/
rm -rf *.egg-info
rm -rf src/*.egg-info
rm -rf .eggs/

# Clean Python cache
find . -type f -name "*.pyc" -delete
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
rm -rf .coverage htmlcov/

# Now build fresh
python -m build

Create a cleanup script:

# Save as clean.sh
#!/bin/bash
echo "Cleaning build artifacts..."
rm -rf build/ dist/ *.egg-info src/*.egg-info .eggs/
find . -type f -name "*.pyc" -delete
find . -type d -name "__pycache__" -exec rm -rf {} +
echo "Clean complete."

Code Works in Development but Not in Built Wheel

Problem: ModuleNotFoundError when using installed wheel, but editable install works fine

This is usually a packaging configuration issue, not a code issue.

Diagnosis:

# Compare what's in the wheel vs. source
echo "=== Source structure ==="
find src -name "*.py" | sort

echo "=== Wheel contents ==="
unzip -l dist/*.whl | grep "\.py$"

Common Causes & Solutions:

  1. Missing __init__.py files: 
    # Add __init__.py to all package directories
find src -type d -name "*" -exec sh -c '
    if [ -n "$(find "$1" -maxdepth 1 -name "*.py")" ]; then
        touch "$1/__init__.py"
    fi
' sh {} \;
  2. Package not discovered by setuptools: 
    # Verify package discovery
python -c "import setuptools; print(setuptools.find_packages('src'))"

# If your package is missing, check pyproject.toml
  3. Relative imports that work in editable but fail in wheel: 
    # WRONG - implicit relative import
from module import function

# CORRECT - explicit absolute import
from your_package.module import function
  4. Files outside package directory: 
    # WRONG - code.py won't be included
src/
├── your_package/
│   └── __init__.py
└── code.py  # Outside package!

# CORRECT
src/
└── your_package/
    ├── __init__.py
    └── code.py  # Inside package

Build Tool Errors

“No module named ‘setuptools’”

# Update build tools
python -m pip install --upgrade pip setuptools wheel build

# Try build again
python -m build

“error: invalid command ‘bdist_wheel’”

# Install wheel package
pip install wheel

# Use modern build tool instead
pip install build
python -m build

“Build backend requires ‘setuptools>=42’”

# Upgrade setuptools
pip install "setuptools>=42"

# Or update build-system requirements in pyproject.toml
[build-system]
requires = ["setuptools>=42", "wheel"]

Wrong Files Included in Distribution

Problem: Test files in wheel, or data files missing

Diagnosis:

# List wheel contents
unzip -l dist/*.whl

# Extract and inspect
unzip dist/*.whl -d /tmp/inspect
tree /tmp/inspect  # or ls -la /tmp/inspect

Solutions:

  1. Tests included in wheel (using src layout prevents this): 
    # With src layout, tests are automatically excluded
project/
├── src/your_package/  # Only this goes in wheel
└── tests/             # Automatically excluded
  2. Missing data files: 
    [tool.setuptools.package-data]
your_package = [
    "*.txt",
    "*.json",
    "data/**/*",
    "templates/**/*.html",
]
  3. Including unwanted files: 
    [tool.setuptools]
exclude-package-data = {
    "*" = ["*.c", "*.h", "*.so", "*.dylib"]
}

CLI Application Specific Issues

Command Not Found After Installation

Problem: bash: command not found: mycommand after successful install

Root Causes & Solutions:

  1. Missing [project.scripts] section: 
    # In pyproject.toml
[project.scripts]
mycommand = "my_package.cli:main"
#           └─ package.module:function format
  2. Entry point function not importable: 
    # Test if entry point works
python -c "from my_package.cli import main; print(main)"

# If this fails, the entry point won't work either
  3. Command installed but not in PATH: 
    # Find where command was installed
find .venv -name "mycommand"
find ~/.local -name "mycommand"

# Check PATH
echo $PATH | tr ':' '\n'

# Add to PATH if needed (in ~/.bashrc or ~/.zshrc)
export PATH="$HOME/.local/bin:$PATH"
  4. Reinstall needed after adding entry point: 
    pip uninstall my-package
pip install -e .

# Verify command exists
which mycommand

Entry Point Configuration Issues

Problem: Entry point syntax errors or incorrect references

Correct format:

[project.scripts]
command-name = "package.module:function"
#              └─ no .py extension, colon before function

Common mistakes:

# WRONG - .py extension
mycommand = "my_package.cli.py:main"

# WRONG - dot instead of colon
mycommand = "my_package.cli.main"

# WRONG - incorrect path
mycommand = "src.my_package.cli:main"

# CORRECT
mycommand = "my_package.cli:main"

Entry point function requirements:

# In my_package/cli.py

# Option 1: Function that returns exit code
def main():
    """CLI entry point."""
    try:
        # Your code here
        return 0  # Success
    except Exception as e:
        print(f"Error: {e}", file=sys.stderr)
        return 1  # Error

# Option 2: Function that calls sys.exit
def main():
    """CLI entry point."""
    try:
        # Your code here
        sys.exit(0)
    except Exception as e:
        print(f"Error: {e}", file=sys.stderr)
        sys.exit(1)

# Don't forget this for direct execution
if __name__ == "__main__":
    sys.exit(main())

pipx Installation Issues

Problem: Command works with pip but not with pipx

Diagnosis:

# Check pipx installation
pipx list

# Check pipx path
pipx environment

Solutions:

  1. PATH not configured: 
    # Ensure PATH is set
pipx ensurepath

# Restart shell or source config
source ~/.bashrc
  2. Conflicting installation: 
    # Remove pip installation first
pip uninstall my-package

# Install with pipx
pipx install my-package
  3. Installing from local wheel: 
    # Build first
python -m build

# Install with pipx
pipx install ./dist/my_package-*.whl

# Or install in editable mode
pipx install -e .

Testing Problems

Tests Can’t Import Package

Problem: ModuleNotFoundError when running pytest

Root Causes & Solutions:

  1. Package not installed (MOST COMMON): 
    pip install -e .
pytest
  2. Wrong import in tests: 
    # WRONG - importing from src
from src.my_package.module import function

# CORRECT - import from package name
from my_package.module import function
  3. Missing pytest configuration: 
    # In pyproject.toml
[tool.pytest.ini_options]
pythonpath = ["src"]  # Add src to Python path
testpaths = ["tests"]
  4. Running pytest from wrong directory: 
    # Must run from project root
cd /path/to/project  # Directory containing pyproject.toml
pytest

Tests Pass but Built Wheel Fails

Problem: All tests pass in editable mode, but wheel doesn’t work

This indicates a packaging issue, not a code issue.

Diagnosis:

# Test the actual wheel in isolation
python -m venv test_env
source test_env/bin/activate
pip install dist/*.whl

# Try importing
python -c "import my_package"

# Clean up
deactivate
rm -rf test_env

Common causes:

  • Missing __init__.py files → See “Code Works in Development but Not in Built Wheel”
  • Files outside package directory → See “Package Structure and Configuration”
  • Incorrect package discovery → Check pyproject.toml configuration

Pytest Configuration Issues

Problem: Pytest not finding tests or running incorrectly

Diagnosis:

# See what pytest is doing
pytest --collect-only

# Verbose output
pytest -v

# Show which tests pytest finds
pytest --co -v

Common configuration:

[tool.pytest.ini_options]
# Where to find tests
testpaths = ["tests"]

# Test file patterns
python_files = ["test_*.py", "*_test.py"]

# Test class patterns
python_classes = ["Test*"]

# Test function patterns
python_functions = ["test_*"]

# Add src to path (if package not installed)
pythonpath = ["src"]

# Default options
addopts = "-v --strict-markers"

Debugging Techniques and Tools

Package Inspection Commands

# Show package information
pip show package-name

# List all files in package
pip show -f package-name

# Check if package is editable
pip list -v | grep package-name

# Find package location
python -c "import package; print(package.__file__)"

# Show package contents
python -c "import package; print(dir(package))"

# Check version
python -c "import package; print(package.__version__)"

Python Path Debugging

# Show sys.path (where Python looks for imports)
python -c "import sys; print('\n'.join(sys.path))"

# Show sys.path with line numbers
python -c "import sys; import pprint; pprint.pprint(list(enumerate(sys.path)))"

# Check if module is findable
python -c "import importlib.util; print(importlib.util.find_spec('your_package'))"

# Show all installed packages locations
pip list -v

Verbose Build and Install

# Verbose pip install
pip install -v .
pip install -vv .   # Even more verbose
pip install -vvv .  # Maximum verbosity

# Verbose build
python -m build -v

# See what setuptools is doing
python -m setuptools.build_meta -v

Validation Tools

# Validate pyproject.toml
pip install validate-pyproject
validate-pyproject pyproject.toml

# Check Python files for syntax errors
python -m py_compile src/your_package/*.py
python -m compileall src/

# Lint pyproject.toml
pip install toml-sort
toml-sort --check pyproject.toml

# Check package metadata
pip install twine
twine check dist/*

Import Debugging

# Debug import issues interactively
python -v -c "import your_package"  # Shows all imports

# Or in Python:
import sys
sys.path.insert(0, '/path/to/debug')  # Temporarily add path

# See what gets imported
import importlib
import your_package
print(importlib.util.find_spec('your_package'))
print(your_package.__file__)
print(your_package.__path__)  # For packages

Wheel Inspection

# List wheel contents
unzip -l dist/*.whl

# Extract wheel
unzip dist/*.whl -d /tmp/wheel_inspect

# View wheel metadata
unzip -p dist/*.whl "*/METADATA" | less

# Check wheel tags
pip install wheel
wheel unpack dist/*.whl

Advanced Troubleshooting

Import System Edge Cases

Circular Imports

Problem: ImportError: cannot import name 'X' from partially initialized module

Solutions:

  1. Move import inside function: 
    # Instead of module-level import
def function():
    from .other_module import something  # Import here
    # Use something
  2. Restructure code to break cycle: 
    # Move shared code to separate module
module_a.py  → module_a.py
module_b.py  → module_b.py
             → shared.py  # New module

Relative Import Issues

Problem: ImportError: attempted relative import with no known parent package

Solutions:

# WRONG - relative imports in scripts
if __name__ == "__main__":
    from .module import function  # Fails!

# CORRECT - use absolute imports
if __name__ == "__main__":
    from package.module import function

# Or run as module
python -m package.script  # Instead of python package/script.py

Dependency Conflicts

Problem: Package requires conflicting versions of dependencies

Diagnosis:

# Check dependency tree
pip install pipdeptree
pipdeptree

# Show conflicts
pipdeptree --warn conflict

# Check what would be installed
pip install --dry-run your-package

Solutions:

  1. Use more flexible version constraints: 
    # Too strict
dependencies = ["requests==2.28.0"]

# Better - allows range
dependencies = ["requests>=2.28.0,<3.0.0"]
  2. Create isolated environment: 
    # Use pipx for CLI tools (isolation guaranteed)
pipx install your-package

# Or use dedicated venv
python -m venv isolated_env
source isolated_env/bin/activate
pip install your-package

Platform-Specific Issues

Windows Path Issues

# Windows uses backslashes
.venv\Scripts\activate

# In code, use pathlib for cross-platform paths
from pathlib import Path
config_file = Path(__file__).parent / "config.json"

macOS Permission Issues

# Don't use sudo with pip
# WRONG: sudo pip install package

# CORRECT: Use virtual environment
python3 -m venv .venv
source .venv/bin/activate
pip install package

Linux Distribution Differences

# Some distros separate python3-venv
# Debian/Ubuntu
sudo apt install python3-venv

# Fedora
sudo dnf install python3-venv

# Check if venv is available
python3 -m venv --help

Build Backend Problems

Problem: Build fails with cryptic setuptools errors

Diagnosis:

# Check build backend version
pip show setuptools build wheel

# Try different build backend
pip install hatchling  # Alternative to setuptools

Solutions:

  1. Update build tools: 
    pip install --upgrade setuptools build wheel pip
  2. Pin build requirements: 
    [build-system]
requires = [
    "setuptools>=65.0",
    "wheel>=0.38.0",
]
  3. Switch build backend (if needed): 
    # Use hatchling instead of setuptools
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

Emergency Quick Fixes

When everything fails, try these in order:

# 1. Nuclear option - start completely fresh
deactivate  # If in venv
rm -rf .venv
python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip setuptools wheel build

# 2. Clean everything
rm -rf build/ dist/ *.egg-info src/*.egg-info .eggs/
find . -type f -name "*.pyc" -delete
find . -type d -name "__pycache__" -exec rm -rf {} +

# 3. Reinstall package
pip install -e .

# 4. Verify
python -c "import your_package; print(your_package.__version__)"
pytest

Useful Resources