Skip to content

[quality] Python test files are never executed in CI #178

@kubestellar-hive

Description

@kubestellar-hive

Finding

The testing-lab repo contains Python test files across multiple suites:

  • tests/flatcar/test_flatcar_boot.py, test_flatcar_knuckle.py
  • tests/smoke/test_gnome_shell.py, test_notifications.py
  • tests/developer/test_micro_editor.py, test_ptyxis_term.py, test_podman_desktop.py, test_dakota_terminal.py
  • tests/homelab_access/test_access_probe.py, tests/homelab_backup/test_restore_drill.py
  • Shared helpers: tests/shared/wait_for_shell.py, tests/service_catalog/shared/kube.py

The only CI workflow is lint.yaml, which validates Argo Workflow YAML only. No workflow runs pytest or any Python test runner. Any syntax error, import failure, or broken assertion in the Python helpers will be invisible until a real VM test is triggered manually.

Recommendation

  1. Add a unit-tests.yml CI workflow that:
    • Runs pytest with --co (collect-only) to catch import errors and syntax failures on all test files
    • Optionally runs tests decorated with markers that do not require real VMs (e.g., @pytest.mark.no_vm)
  2. Add basic conftest.py fixtures that mock VM/SSH dependencies to enable unit-level testing of helper functions
  3. Run flake8 or ruff on the Python test sources

Priority

  • Impact: high — broken Python helpers silently survive in the repo
  • Effort: medium — needs mock fixtures for VM-dependent tests

Filed by quality agent (hold-gated mode)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions