Skip to content

ci-integration.md

Latest commit

 

History

History
191 lines (191 loc) · 7.3 KB

ci-integration.md

File metadata and controls

191 lines (191 loc) · 7.3 KB
Raw

CI Integration — ovoscope

This document explains how to wire ovoscope end-to-end tests into a repo's CI pipeline using gh-automations reusable workflows, and how to structure test files and fixtures.

Directory Layout

The workspace convention is:

my-skill-repo/
├── test/
│   └── end2end/
│       ├── test_intent_match.py       # TestCase classes using ovoscope
│       ├── test_session_state.py
│       └── fixtures/
│           ├── hello_world_adapt.json  # committed fixture files (optional)
│           └── hello_world_padatious.json
├── setup.py  (or pyproject.toml)
└── ...

Separate end2end/ from unittests/ so they can be run independently — end2end tests are slower (they spin up a MiniCroft) and may require extra dependencies.

pytest / unittest Configuration

Using pyproject.toml

[tool.pytest.ini_options]
testpaths = ["test"]
# Run only unit tests (fast):
# pytest test/unittests/
# Run only end2end tests (slow, requires skill installed):
# pytest test/end2end/

Using pytest.ini

[pytest]
testpaths = test

End2end tests are standard unittest.TestCase subclasses and work with both pytest and plain python -m unittest discover.

Install Dependencies in CI

End2end tests need ovoscope and all skills under test installed. Add an install step before running tests:

pip install ovoscope
pip install -e .          # install the skill from source (editable)

Or if testing multiple skills together:

pip install ovoscope \
    ovos-skill-hello-world \
    ovos-skill-weather

Verify the skills are discoverable before running:

python -c "
from ovos_plugin_manager.skills import find_skill_plugins
plugins = list(find_skill_plugins())
print('Found skills:', plugins)
assert 'ovos-skill-hello-world.openvoiceos' in plugins
"

Fixture JSON Files

Fixture files generated by End2EndTest.save() (see usage-guide.md Pattern 4) contain the expected message sequence serialised as JSON. When to commit fixtures:

  • Commit fixtures that test stable, deterministic interactions (e.g., a specific dialog line).
  • Do NOT commit fixtures where the speak utterance varies randomly — either omit the utterance key from expected data or use manual assertion instead.
  • Always generate fixtures with anonymize=True (the default) — this strips real location data. .gitignore pattern (if you generate fixtures locally but don't want to commit them):
test/end2end/fixtures/*.json

Or selectively ignore only generated/recording artifacts:

test/end2end/fixtures/recorded_*.json

GitHub Actions — End2End Job

Add an end2end job to your release_workflow.yml or a dedicated workflow. This example follows the gh-automations conventions used across all 203+ OVOS repos:

# .github/workflows/release_workflow.yml
name: Release workflow
on:
  pull_request:
    types: [closed]
    branches: [dev]
  workflow_dispatch:
jobs:
  build_tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - name: Install dependencies
        run: |
          pip install build pytest
          pip install ovoscope
          pip install -e .
      - name: Run unit tests
        run: pytest test/unittests/ -v
      - name: Run end2end tests
        run: pytest test/end2end/ -v --timeout=60
  publish_alpha:
    needs: build_tests
    if: github.event.pull_request.merged == true || github.event_name == 'workflow_dispatch'
    uses: OpenVoiceOS/gh-automations/.github/workflows/publish-alpha.yml@dev
    with:
      propose_release: true
    secrets: inherit

The build_tests job runs before publish_alpha — a failing end2end test blocks the release.

Standalone End2End Workflow

If your repo only needs end2end tests (no release automation), use a simpler workflow:

# .github/workflows/end2end.yml
name: End2End Tests
on:
  push:
    branches: [dev, master]
  pull_request:
    branches: [dev]
jobs:
  end2end:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - name: Install
        run: |
          pip install ovoscope pytest
          pip install -e .
      - name: Test
        run: pytest test/end2end/ -v --timeout=60

Known CI Gotchas

Skill plugin not found on PATH

Symptom: get_minicroft() hangs or find_skill_plugins() returns an empty list. Cause: The skill was not installed in editable mode (pip install -e .) or the entry point was not registered. Fix: Always install the skill package in the same environment as ovoscope:

pip install -e .          # registers entry points
pip install ovoscope

Missing .venv in CI

If you use uv locally, your .venv is not present in CI. Use pip directly in CI or add a uv pip install step. Do not rely on .venv being pre-activated.

MiniCroft hangs for >30 seconds

Padatious intent training can be slow on a cold CI runner. Set a generous --timeout in pytest and pass timeout=30 (or higher) to test.execute().

Flaky tests from session ID collisions

Each test that uses Session("same-id") shares session state with other tests using the same session ID. Use unique session IDs per test class, or generate them:

import uuid
session = Session(str(uuid.uuid4()))

GUI messages causing assertion failures

By default ignore_gui=True strips GUI namespace messages from the captured sequence. If you see unexpected messages related to gui.*, check whether a skill emits GUI messages unconditionally and whether your expected_messages list accounts for them.

ovoscope's Own CI Workflows

The ovoscope repository itself uses the standard OVOS workflow set:

Workflow File Trigger Purpose
Unit Tests unit_tests.yml PR/push to dev Runs pytest --cov=ovoscope on 58 tests, posts coverage comment
Build Tests build_tests.yml PR to dev, push to master Matrix build (Python 3.10, 3.11) with python -m build
License Check license_tests.yml PR to dev, push to master Calls gh-automations/license-check.yml reusable
Pip Audit pipaudit.yml Push to dev/master CVE scanning via pypa/gh-action-pip-audit
Release Alpha release_workflow.yml PR merge to dev Runs tests first, then calls publish-alpha.yml
Stable Release publish_stable.yml Push to master Calls publish-stable.yml with bot loop guard
Labels conventional-label.yaml PR open/edit Auto-labels PRs with conventional commit types
The release workflow gates alpha publishing on test success — a failing test blocks the release.

See Also