Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 100 additions & 0 deletions FAQ.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,33 @@ reply is emitted before the internal listener is registered. Use subscribe-emit-
`threading.Event` instead. `AudioServiceHarness.get_track_info()` and `list_backends()`
implement this pattern — `ovoscope/audio.py`.

## How do I test VAD (Voice Activity Detection) without a real microphone?

Use `MockVADEngine` from `ovoscope.listener`. It classifies all-zero bytes as silence and
any non-zero byte as speech. Inject it into `MiniListener(config, vad_instance=MockVADEngine())`
or use the declarative `VADTest` dataclass. No microphone, audio driver, or OPM plugin required.

```python
from ovoscope.listener import MockVADEngine, VADTest
VADTest(vad_instance=MockVADEngine(), audio_input=b"\\x01" * 512, expect_silence=False).execute()
```

## How do I test Wake Word detection without loading a real model?

Use `MockHotWordEngine(trigger_after=N)` from `ovoscope.listener`. It fires after exactly N
`update()` calls and auto-resets. Inject via `MiniListener(config, ww_instances={"hey_mycroft": engine})`
or use the declarative `WakeWordTest` dataclass.

```python
from ovoscope.listener import MockHotWordEngine, WakeWordTest
WakeWordTest(
ww_instances={"hey_mycroft": MockHotWordEngine(trigger_after=2)},
audio_chunks=[b"\\x00" * 512] * 4,
expect_detected=True,
expected_detection_frame=1,
).execute()
```

## What is `ovoscope`?
`ovoscope` is End-to-end test framework for OpenVoiceOS skills.
## How do I install it?
Expand Down Expand Up @@ -267,3 +294,76 @@ The override is patched into `Configuration()["intents"]` before `super().__init
```bash
python -c "from model2vec.inference import StaticModelPipeline; StaticModelPipeline.from_pretrained('Jarbas/ovos-model2vec-intents-distiluse-base-multilingual-cased-v2')"
```

---

## CLI

### How do I record a fixture from the command line?
```bash
ovoscope record --skill-id ovos-skill-hello-world.openvoiceos \
--utterance "hello" --output fixture.json
```

### How do I replay a fixture?
```bash
ovoscope run fixture.json --verbose
```

### How do I compare two fixture files?
```bash
ovoscope diff expected.json actual.json
```
Exit code 0 = identical, 1 = differences found.

### How do I scan my workspace for E2E coverage gaps?
```bash
ovoscope coverage "OpenVoiceOS Workspace/" --format table
```

---

## PHAL Testing

### Can I test PHAL plugins with ovoscope?
Yes — any PHAL plugin that communicates only via the MessageBus (no physical
hardware) is testable with `MiniPHAL` or `PHALTest` from `ovoscope.phal`.

### Which PHAL plugins require real hardware?
`ovos-PHAL-plugin-alsa`, `ovos-PHAL-plugin-mk1`, `ovos-PHAL-plugin-dotstar`.
These should use hardware-in-the-loop integration tests instead.

---

## OCP Testing

### How do I test an OCP skill without a real HTTP server?
Use `OCPTest` with `mock_responses` — keys are URL substrings matched
against actual requests, values are the JSON bodies returned.

### What message flow does OCP testing drive?
`recognizer_loop:utterance` → `ovos.common_play.query` → `ovos.common_play.query.response` → `ovos.common_play.start`

---

## GUI Assertions

### How do I assert that a skill showed a GUI page?
```python
from ovoscope import GUICaptureSession
with GUICaptureSession(mc.bus) as gui:
# ... trigger interaction ...
gui.assert_page_shown("my_skill", "main.qml")
```

---

## Coverage Scanner

### What entry-point groups does the scanner detect?
`opm.skill`, `opm.pipeline`, `opm.phal`, `opm.plugin.tts`, `opm.plugin.stt`,
`opm.plugin.audio`, `opm.common_play`, `opm.solver`.

### How is "covered" defined?
A repo is considered covered when `test/end2end/` (or `tests/end2end/`)
exists and contains at least one `.py` file (excluding `__init__.py`).
68 changes: 68 additions & 0 deletions MAINTENANCE_REPORT.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,31 @@
# Maintenance Report — `ovoscope`
## [2026-03-11] — Docs Gap Review and Fixes

- **AI Model**: Claude Sonnet 4.6
- **Actions Taken**:
- `docs/ocp.md`: Documented `execute()` return type (`List[Message]`), clarified `patch_targets` format (dotted Python path where symbol is used), added aiohttp example.
- `docs/pipeline.md`: Documented `assert_matches(intent_type=...)` as substring check with example; added `ovoscope/pipeline.py:LINE` citations to all methods.
- `docs/cli.md`: Corrected `--ignore-context` → `--include-context`, explained when/why to use it; clarified `validate` pydantic fallback trigger.
- `docs/end2end-test.md`, `docs/minicroft.md`, `docs/capture-session.md`: Added `ovoscope/__init__.py:LINE` source citations to class and key method definitions.
- `docs/capture-session.md`: Documented `finish()` idempotency.
- `docs/listener.md`: Added full VAD/WakeWord API section (`MockVADEngine`, `MockHotWordEngine`, `is_silence`, `extract_speech`, `detect_wakeword`, `scan_for_wakeword`, `VADTest`, `WakeWordTest`) with examples and `ovoscope/listener.py:LINE` citations. Updated constructor parameter table. Fixed stale line references.
- `docs/index.md`: Added `gui-testing.md` link; updated Public API section with `GUICaptureSession`, VAD/WW helpers; fixed "Does NOT Do" section for VAD/WW.
- `QUICK_FACTS.md`: Added entry-point groups table; updated test count (243) and coverage note.
- **Oversight**: No new code changes — docs only.

## [2026-03-11] — Add VAD and WakeWord Support to MiniListener

- **AI Model**: Claude Haiku 4.5
- **Actions Taken**:
- Extended `ovoscope/listener.py` with `MockVADEngine`, `MockHotWordEngine`, `VADTest`, `WakeWordTest`.
- Extended `MiniListener` with `vad_instance` / `ww_instances` constructor params.
- Added `is_silence()`, `extract_speech()`, `detect_wakeword()`, `scan_for_wakeword()` methods to `MiniListener` — `listener.py:466–600`.
- Extended `get_mini_listener()` factory with `vad_plugin`, `vad_instance`, `ww_plugin`, `ww_instances` params.
- Made `ovos_dinkum_listener` import lazy (graceful `ImportError`) so VAD/WW tests work without the full listener stack installed.
- Added 41 unit tests in `test/unittests/test_listener_vad_ww.py`.
- Updated `FAQ.md` with VAD and WakeWord testing Q&A.
- **Oversight**: 243 unit tests pass locally.

## [2026-03-11] — Enhance Audio Testing Robustness and CI

- **AI Model**: Claude Sonnet 4.6
Expand Down Expand Up @@ -301,7 +328,48 @@ level where every OVOS repo can adopt ovoscope end-to-end testing without readin
- **Actions Taken**: Read `ovoscope/__init__.py` (485 lines), `test/test_helloworld.py`,
`ovos-core/test/end2end/test_adapt.py`, and all existing docs; then generated enriched content.
- **Oversight**: Code examples are illustrative but not executed. Verify against live skill install before treating as runnable.

---

## 2026-03-11 — Phase 1–3 Feature Additions
Comment thread
coderabbitai[bot] marked this conversation as resolved.

**AI Model**: claude-sonnet-4-6
**Oversight**: Human review pending

### Actions Taken

Added CLI, PHAL harness, fixture differ, OCP harness, pipeline harness,
ecosystem coverage scanner, GUI capture session, and remote recorder.

**New modules:**
- `ovoscope/cli.py` — `ovoscope` CLI with `record`, `run`, `diff`, `validate`, `coverage`
- `ovoscope/diff.py` — `MessageDiff`, `FixtureDiffResult`, `diff_fixtures`
- `ovoscope/phal.py` — `MiniPHAL`, `PHALTest`
- `ovoscope/ocp.py` — `OCPTest`, `assert_ocp_query_response`
- `ovoscope/pipeline.py` — `PipelineHarness`
- `ovoscope/coverage.py` — `RepoCoverage`, `EcosystemCoverageReport`, `scan_workspace`
- `ovoscope/remote_recorder.py` — `RemoteRecorder`

**Extended modules:**
- `ovoscope/__init__.py` — added `GUICaptureSession`

**New docs:**
- `docs/cli.md`, `docs/phal.md`, `docs/ocp.md`, `docs/pipeline.md`
- `docs/usage-guide.md` — Patterns 9–12 appended

**New tests:**
- `test/unittests/test_diff.py` — 7 test methods
- `test/unittests/test_phal.py` — 8 test methods
- `test/unittests/test_coverage.py` — 11 test methods
- `test/unittests/test_cli.py` — 14 test methods

**pyproject.toml changes:**
- Added `[project.scripts] ovoscope = "ovoscope.cli:main"`

All 202 tests pass. No regressions introduced.

---

## [2026-03-08] — Initial compliance scaffold
### Changes
- Created `QUICK_FACTS.md` with machine-readable package metadata.
Expand Down
10 changes: 8 additions & 2 deletions QUICK_FACTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,17 @@ End-to-end test framework for OpenVoiceOS skills
| Repository | [https://github.com/TigreGotico/ovoscope](https://github.com/TigreGotico/ovoscope) |
| Python Support | >=3.10 |
| Status | Active development |
## Entry Points
| Group | Value | Description |
|-------|-------|-------------|
| `console_scripts` | `ovoscope = ovoscope.cli:main` | CLI entry point |
| `pytest11` | `ovoscope = ovoscope.pytest_plugin` | pytest plugin (auto-loaded by pytest) |

## Testing & CI
| Feature | Details |
|---------|---------|
| Unit Tests | 142 tests across `test/unittests/` (all passing) — +38 audio harness tests |
| Coverage | 89% overall (improved from 78%) |
| Unit Tests | 243 tests across `test/unittests/` (all passing) |
| Coverage | 53% overall (transformer/remote code excluded — requires optional deps) |
| Test Framework | pytest with custom fixtures |
| Coverage Reporter | py-cov-action/python-coverage-comment-action@v3 |
## CI Workflows
Expand Down
6 changes: 5 additions & 1 deletion docs/capture-session.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,14 @@
# CaptureSession
`CaptureSession` subscribes to all messages on the `FakeBus` and records them during a single test interaction. It handles synchronous responses (ordered, from the intent pipeline) and asynchronous responses (from external threads, unordered).
## Class: `CaptureSession`
## Class: `CaptureSession` — `ovoscope/__init__.py:488`
```python
from ovoscope import CaptureSession
```
A `dataclass` that wraps a `MiniCroft` and manages message collection for one test interaction.
`CaptureSession.finish` — `ovoscope/__init__.py:521`

> **Idempotency:** `finish()` may be called multiple times safely — subsequent calls
> return the same message list without re-subscribing or clearing state.
### Fields
| Field | Type | Default | Description |
|---|---|---|---|
Expand Down
134 changes: 134 additions & 0 deletions docs/cli.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
# ovoscope CLI

The `ovoscope` command-line tool provides five subcommands for recording,
replaying, diffing, validating, and scanning E2E test fixtures.

## Installation

After installing the package (``pip install ovoscope``), the ``ovoscope``
command is available on your ``$PATH``.

```bash
ovoscope --help
```

---

## Subcommands

### `ovoscope record` — Record a fixture

**In-process recording** (default): loads the skill(s) inside the current
process using `MiniCroft` — `cli.py:cmd_record`.

```bash
ovoscope record \
--skill-id ovos-skill-hello-world.openvoiceos \
--utterance "hello" \
--output fixture.json \
--lang en-US \
--timeout 20
```

**Live recording** from a running OVOS instance (`RemoteRecorder` —
`remote_recorder.py:RemoteRecorder.record`):

```bash
ovoscope record --live \
--bus-url ws://localhost:8181/core \
--skill-id ovos-skill-date-time.openvoiceos \
--utterance "what time is it" \
--output datetime_fixture.json
```

| Flag | Default | Description |
|------|---------|-------------|
| `--skill-id` | — | OPM skill IDs to load (repeatable). |
| `--utterance` | **required** | User utterance text. |
| `--output` | **required** | Output fixture JSON path. |
| `--lang` | `en-US` | Language tag. |
| `--pipeline` | None | Comma-separated pipeline stage IDs. |
| `--timeout` | `20.0` | Capture timeout in seconds. |
| `--live` | False | Use live OVOS instance via `RemoteRecorder`. |
| `--bus-url` | `ws://localhost:8181/core` | MessageBus URL (only for `--live`). |

---

### `ovoscope run` — Replay a fixture

Replays a saved fixture file and exits with code 1 on failure —
`cli.py:cmd_run`.

```bash
ovoscope run test/fixtures/hello.json
ovoscope run test/fixtures/hello.json --verbose --timeout 30
```

| Flag | Default | Description |
|------|---------|-------------|
| `fixture` | **required** | Path to fixture JSON file. |
| `--verbose` | False | Print failure details. |
| `--timeout` | `30.0` | Execution timeout in seconds. |

---

### `ovoscope diff` — Compare two fixtures

Compares two fixture files and prints a colored report —
`diff.py:diff_fixtures`, `cli.py:cmd_diff`.

```bash
ovoscope diff expected.json actual.json
ovoscope diff expected.json actual.json --no-color
```

Exits 0 if identical, 1 if differences are found.

| Flag | Default | Description |
|------|---------|-------------|
| `expected` | **required** | Reference fixture path. |
| `actual` | **required** | Fixture to compare against reference. |
| `--no-color` | False | Disable ANSI color codes. |
| `--include-context` | False | Include `context` fields in the comparison. By default context is ignored because it contains ephemeral routing metadata (`source`, `destination`, `session`) that varies between runs. Pass `--include-context` when you specifically want to assert routing behaviour. |

---

### `ovoscope validate` — Schema-validate fixtures

Validates one or more fixture files against the expected schema —
`cli.py:cmd_validate`.

```bash
ovoscope validate test/fixtures/*.json
```

Uses `pydantic_helpers.validate_fixture` when available (requires
`pip install ovoscope[pydantic]`); falls back to basic JSON structure
validation (checks required top-level keys and that `expected_messages`
is a list) when the `pydantic` extra is not installed.

---

### `ovoscope coverage` — Ecosystem coverage scan

Scans a workspace root for OVOS plugin repos and reports E2E test coverage —
`coverage.py:scan_workspace`, `cli.py:cmd_coverage`.

```bash
ovoscope coverage "OpenVoiceOS Workspace/" --format table
ovoscope coverage "OpenVoiceOS Workspace/" --format json
```

| Flag | Default | Description |
|------|---------|-------------|
| `workspace` | **required** | Workspace root directory. |
| `--format` | `table` | Output format: `table` or `json`. |

---

## Exit Codes

| Code | Meaning |
|------|---------|
| 0 | Success / no differences / all valid |
| 1 | Failure / differences found / validation error |
3 changes: 2 additions & 1 deletion docs/end2end-test.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
# End2EndTest
`End2EndTest` is the primary API. It wires together `MiniCroft`, `CaptureSession`, and all assertion logic into a single declarative test object.
## Class: `End2EndTest`
## Class: `End2EndTest` — `ovoscope/__init__.py:533`
```python
from ovoscope import End2EndTest
```
A `dataclass`. Configure once, call `.execute()` to run.
`End2EndTest.execute` — `ovoscope/__init__.py:602`
---
## Fields
### Core
Expand Down
Loading
Loading