diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index 19bc9a38..60f05d76 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -12,6 +12,6 @@ jobs: with: python_version: '3.11' coverage_source: 'ovos_utils' - test_path: 'test/' - install_extras: '' + test_path: 'test/unittests' + install_extras: '.[extras]' min_coverage: 0 diff --git a/CHANGELOG.md b/CHANGELOG.md index ed7c503e..91fec452 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,54 +1,100 @@ # Changelog -## [0.8.5a4](https://github.com/OpenVoiceOS/ovos-utils/tree/0.8.5a4) (2026-03-11) +## [0.13.3a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.13.3a1) (2026-06-29) -[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.8.6a2...0.8.5a4) +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.13.2a1...0.13.3a1) **Merged pull requests:** -- chore: Add comprehensive test suites and documentation [\#362](https://github.com/OpenVoiceOS/ovos-utils/pull/362) ([JarbasAl](https://github.com/JarbasAl)) -- fix: make the stopwatch test less strict [\#360](https://github.com/OpenVoiceOS/ovos-utils/pull/360) ([PureTryOut](https://github.com/PureTryOut)) +- fix: FakeBus folds the default session like any other \(drop owner-only\) [\#393](https://github.com/OpenVoiceOS/ovos-utils/pull/393) ([JarbasAl](https://github.com/JarbasAl)) -## [0.8.6a2](https://github.com/OpenVoiceOS/ovos-utils/tree/0.8.6a2) (2026-02-02) +## [0.13.2a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.13.2a1) (2026-06-29) -[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.8.6a1...0.8.6a2) +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.13.1a1...0.13.2a1) **Merged pull requests:** -- chore\(deps\): update actions/setup-python action to v6 [\#353](https://github.com/OpenVoiceOS/ovos-utils/pull/353) ([renovate[bot]](https://github.com/apps/renovate)) -- chore\(deps\): update actions/checkout action to v6 [\#352](https://github.com/OpenVoiceOS/ovos-utils/pull/352) ([renovate[bot]](https://github.com/apps/renovate)) +- fix: drop deprecated make\_default from FakeBus default-session sync [\#389](https://github.com/OpenVoiceOS/ovos-utils/pull/389) ([JarbasAl](https://github.com/JarbasAl)) -## [0.8.6a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.8.6a1) (2026-02-02) +## [0.13.1a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.13.1a1) (2026-06-29) -[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.8.5a3...0.8.6a1) +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.13.0a1...0.13.1a1) **Merged pull requests:** -- fix: prevent handler errors from aborting FakeBus emit [\#358](https://github.com/OpenVoiceOS/ovos-utils/pull/358) ([JarbasAl](https://github.com/JarbasAl)) +- fix: target a real shape-changing pair in namespace-migration tests [\#390](https://github.com/OpenVoiceOS/ovos-utils/pull/390) ([JarbasAl](https://github.com/JarbasAl)) -## [0.8.5a3](https://github.com/OpenVoiceOS/ovos-utils/tree/0.8.5a3) (2025-12-19) +## [0.13.0a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.13.0a1) (2026-06-27) -[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.8.5a2...0.8.5a3) +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.12.2a1...0.13.0a1) **Merged pull requests:** -- chore\(deps\): update dependency python to 3.14 [\#348](https://github.com/OpenVoiceOS/ovos-utils/pull/348) ([renovate[bot]](https://github.com/apps/renovate)) +- feat: AsyncFakeBus namespace migration + env/config flag parity [\#387](https://github.com/OpenVoiceOS/ovos-utils/pull/387) ([JarbasAl](https://github.com/JarbasAl)) -## [0.8.5a2](https://github.com/OpenVoiceOS/ovos-utils/tree/0.8.5a2) (2025-12-18) +## [0.12.2a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.12.2a1) (2026-06-27) -[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.8.5a1...0.8.5a2) +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.12.1a1...0.12.2a1) **Merged pull requests:** -- chore: Configure Renovate [\#347](https://github.com/OpenVoiceOS/ovos-utils/pull/347) ([renovate[bot]](https://github.com/apps/renovate)) +- fix: translate mirrored payload onto counterpart topic in FakeBus [\#385](https://github.com/OpenVoiceOS/ovos-utils/pull/385) ([JarbasAl](https://github.com/JarbasAl)) -## [0.8.5a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.8.5a1) (2025-11-07) +## [0.12.1a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.12.1a1) (2026-06-25) -[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.8.4...0.8.5a1) +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.12.0a1...0.12.1a1) **Merged pull requests:** -- fix: use timezone-aware datetime functions and update scheduler event names [\#343](https://github.com/OpenVoiceOS/ovos-utils/pull/343) ([JarbasAl](https://github.com/JarbasAl)) +- fix: raise ovos-spec-tools floor to 0.10.0a1 for NamespaceTranslator [\#383](https://github.com/OpenVoiceOS/ovos-utils/pull/383) ([JarbasAl](https://github.com/JarbasAl)) + +## [0.12.0a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.12.0a1) (2026-06-25) + +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.11.2a1...0.12.0a1) + +**Merged pull requests:** + +- feat: FakeBus mirrors the legacy\<-\>ovos.\* namespace migration [\#381](https://github.com/OpenVoiceOS/ovos-utils/pull/381) ([JarbasAl](https://github.com/JarbasAl)) + +## [0.11.2a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.11.2a1) (2026-06-20) + +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.11.1a1...0.11.2a1) + +**Merged pull requests:** + +- fix: allow json-database 1.x [\#379](https://github.com/OpenVoiceOS/ovos-utils/pull/379) ([JarbasAl](https://github.com/JarbasAl)) + +## [0.11.1a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.11.1a1) (2026-05-25) + +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.11.0a1...0.11.1a1) + +**Merged pull requests:** + +- fix: standardize\_lang\_tag macro=True preserves region \(restore langcodes semantics\) [\#377](https://github.com/OpenVoiceOS/ovos-utils/pull/377) ([JarbasAl](https://github.com/JarbasAl)) + +## [0.11.0a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.11.0a1) (2026-05-25) + +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.10.0a1...0.11.0a1) + +**Merged pull requests:** + +- feat: fakebus Message subclasses ovos\_spec\_tools.Message — no API break [\#375](https://github.com/OpenVoiceOS/ovos-utils/pull/375) ([JarbasAl](https://github.com/JarbasAl)) + +## [0.10.0a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.10.0a1) (2026-05-22) + +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.9.0a1...0.10.0a1) + +**Merged pull requests:** + +- feat: migrate ovos-utils onto ovos-spec-tools [\#373](https://github.com/OpenVoiceOS/ovos-utils/pull/373) ([JarbasAl](https://github.com/JarbasAl)) + +## [0.9.0a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.9.0a1) (2026-05-18) + +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.8.5...0.9.0a1) + +**Merged pull requests:** + +- feat: AsyncFakeBus alongside FakeBus [\#371](https://github.com/OpenVoiceOS/ovos-utils/pull/371) ([JarbasAl](https://github.com/JarbasAl)) diff --git a/docs/fakebus.md b/docs/fakebus.md index 49e863c9..037d770f 100644 --- a/docs/fakebus.md +++ b/docs/fakebus.md @@ -52,6 +52,65 @@ bus.emit(FakeMessage("recognizer_loop:utterance", {"utterances": ["hello"]})) | `close()` | Calls `on_close()` | | `create_client()` | Returns `self` | +For asyncio-native code, see [`AsyncFakeBus`](#asyncfakebus) below. + +--- + +## `AsyncFakeBus` + +`AsyncFakeBus` — `ovos_utils/fakebus.py:351` + +In-process stand-in for `AsyncMessageBusClient` (from `ovos-bus-client`). Use this when your code is asyncio-native and needs a drop-in fake bus without a WebSocket connection. The API surface mirrors the real async client: coroutine methods keep you inside the event loop, while handler registration stays synchronous to match `pyee` and the real client's contract. + +```python +import asyncio +from ovos_utils.fakebus import AsyncFakeBus, FakeMessage + +async def main(): + bus = AsyncFakeBus() + + received = [] + + def on_ping(message): + received.append(message) + + bus.on("test:ping", on_ping) + + await bus.emit(FakeMessage("test:ping", {"n": 1})) + print(received) # [FakeMessage("test:ping", ...)] + + await bus.close() + +asyncio.run(main()) +``` + +### Coroutine vs sync split + +| Sync (handler registration) | Async (I/O surface) | +|---|---| +| `on(msg_type, handler)` | `connect(*args, **kwargs)` | +| `once(msg_type, handler)` | `close()` | +| `remove(msg_type, handler)` | `emit(message)` | +| `remove_all_listeners(event_name)` | `wait_for_message(message_type, timeout)` | +| | `wait_for_response(message, reply_type, timeout)` | + +### Key Methods + +| Method | Description | Source | +|---|---|---| +| `connect()` | No-op; sets `connected_event` and `started_running = True` | `fakebus.py:409` | +| `close()` | Clears `connected_event`, calls `on_close()` | `fakebus.py:418` | +| `emit(message)` | Injects session, dispatches to `pyee` emitter | `fakebus.py:426` | +| `wait_for_message(message_type, timeout)` | Awaits a single message of that type | `fakebus.py:489` | +| `wait_for_response(message, reply_type, timeout)` | Emits a message and awaits the reply | `fakebus.py:513` | +| `create_client()` | Returns `self` (backwards-compat shim) | `fakebus.py:543` | +| `run_forever()` | Sets `started_running = True` (backwards-compat shim) | `fakebus.py:546` | +| `run_in_thread()` | Calls `run_forever()` (backwards-compat shim) | `fakebus.py:549` | + +### Session Handling + +Session injection side effects are identical to `FakeBus`: `emit()` populates `message.context["session"]` from `SessionManager`, and `on_message()` feeds incoming messages back through `Session.from_message()` / `SessionManager.update()`. Both imports are lazy so the class works without `ovos-bus-client` installed. + --- ## `FakeMessage` diff --git a/docs/index.md b/docs/index.md index 1df0d780..2358430f 100644 --- a/docs/index.md +++ b/docs/index.md @@ -11,7 +11,7 @@ Shared utility library used by all OVOS components. Provides logging, process li |---|---| | `ovos_utils.log` | `LOG` — OVOS-wide logging class with optional file rotation | | `ovos_utils.process_utils` | `ProcessStatus`, `RuntimeRequirements`, `PIDLock`, `MonotonicEvent` | -| `ovos_utils.fakebus` | `FakeBus`, `FakeMessage` — in-process bus for testing without a live WebSocket | +| `ovos_utils.fakebus` | `FakeBus`, `AsyncFakeBus`, `FakeMessage` — in-process bus for testing without a live WebSocket | | `ovos_utils.events` | `EventContainer`, `EventSchedulerInterface`, handler wrappers | | `ovos_utils.file_utils` | Resource resolution, vocab loading, `FileWatcher` | | `ovos_utils.network_utils` | `get_ip()`, `is_connected_dns()`, `is_connected_http()`, `check_captive_portal()` | @@ -60,6 +60,6 @@ pip install ovos-utils - [Logging](log.md) — `LOG`, `init_service_logger()`, `log_deprecation()`, `deprecated` decorator - [Process Utilities](process-utils.md) — `ProcessStatus`, `RuntimeRequirements`, `PIDLock`, `MonotonicEvent` -- [FakeBus](fakebus.md) — `FakeBus`, `FakeMessage` — in-process message bus for testing +- [FakeBus](fakebus.md) — `FakeBus`, `AsyncFakeBus`, `FakeMessage` — in-process message bus for testing - [Events](events.md) — `EventContainer`, `EventSchedulerInterface`, handler wrappers - [Utilities](utilities.md) — file, network, sound, threading, XDG helpers diff --git a/ovos_logs_console_script b/ovos_logs_console_script new file mode 100644 index 00000000..e69de29b diff --git a/ovos_utils/bracket_expansion.py b/ovos_utils/bracket_expansion.py index 06304e36..5170a0cd 100644 --- a/ovos_utils/bracket_expansion.py +++ b/ovos_utils/bracket_expansion.py @@ -2,43 +2,34 @@ import re from typing import List, Dict import warnings -from ovos_utils.log import deprecated +from ovos_spec_tools import expand as _spec_expand -def expand_template(template: str) -> List[str]: - def expand_optional(text): - """Replace [optional] with two options: one with and one without.""" - return re.sub(r"\[([^\[\]]+)\]", lambda m: f"({m.group(1)}|)", text) - - def expand_alternatives(text): - """Expand (alternative|choices) into a list of choices.""" - parts = [] - for segment in re.split(r"(\([^\(\)]+\))", text): - if segment.startswith("(") and segment.endswith(")"): - options = segment[1:-1].split("|") - parts.append(options) - else: - parts.append([segment]) - return itertools.product(*parts) - - def fully_expand(texts): - """Iteratively expand alternatives until all possibilities are covered.""" - result = set(texts) - while True: - expanded = set() - for text in result: - options = list(expand_alternatives(text)) - expanded.update(["".join(option).strip() for option in options]) - if expanded == result: # No new expansions found - break - result = expanded - return sorted(result) # Return a sorted list for consistency +from ovos_utils.log import deprecated +from ovos_utils.version import VERSION_MAJOR - # Expand optional items first - template = expand_optional(template) - # Fully expand all combinations of alternatives - return fully_expand([template]) +@deprecated("import 'expand' from 'ovos_spec_tools' instead", + f"{VERSION_MAJOR + 1}.0.0") +def expand_template(template: str) -> List[str]: + """Expand a sentence template to its sample set. + + Resolves ``(a|b)`` alternatives and ``[optional]`` segments; named + ``{slot}`` placeholders are carried through unchanged. The samples are + returned sorted. + + .. deprecated:: + Import :func:`expand` from ``ovos_spec_tools`` directly — it is the + single conformant OVOS-INTENT-1 expander, and what this now delegates + to. A template the specification rejects as malformed (a single-branch + group, an empty sample, a slot-only template, …) raises + :class:`ovos_spec_tools.MalformedTemplate`. + """ + # stacklevel=3: warn() -> expand_template body -> @deprecated wrapper -> caller + warnings.warn("expand_template is deprecated; import 'expand' from " + "'ovos_spec_tools' instead", + DeprecationWarning, stacklevel=3) + return sorted(_spec_expand(template)) def expand_slots(template: str, slots: Dict[str, List[str]]) -> List[str]: @@ -53,7 +44,7 @@ def expand_slots(template: str, slots: Dict[str, List[str]]) -> List[str]: list[str]: A list of all expanded combinations. """ # Expand alternatives and optional components - base_expansions = expand_template(template) + base_expansions = sorted(_spec_expand(template)) # Process slots all_sentences = [] diff --git a/ovos_utils/dialog.py b/ovos_utils/dialog.py index d44636da..086003aa 100644 --- a/ovos_utils/dialog.py +++ b/ovos_utils/dialog.py @@ -1,20 +1,31 @@ import os import random import re +import warnings from os.path import join from pathlib import Path from typing import Optional -from ovos_utils.bracket_expansion import expand_template +from ovos_spec_tools import expand + from ovos_utils.file_utils import resolve_resource_file from ovos_utils.lang import translate_word -from ovos_utils.log import LOG, log_deprecation +from ovos_utils.log import LOG, deprecated, log_deprecation +from ovos_utils.version import VERSION_MAJOR class MustacheDialogRenderer: """A dialog template renderer based on the mustache templating language.""" + @deprecated("use the OVOS-INTENT-2 §4.2 dialog renderer in " + "'ovos_spec_tools' ('render' / 'DialogRenderer')", + f"{VERSION_MAJOR + 1}.0.0") def __init__(self): + warnings.warn( + "MustacheDialogRenderer is deprecated; use the OVOS-INTENT-2 §4.2 " + "dialog renderer in 'ovos_spec_tools' ('render' / " + "'DialogRenderer')", + DeprecationWarning, stacklevel=3) self.templates = {} self.recent_phrases = [] @@ -92,7 +103,7 @@ def render(self, template_name, context=None, index=None): line = template_functions[index % len(template_functions)] # Replace {key} in line with matching values from context line = line.format(**context) - line = random.choice(expand_template(line)) + line = random.choice(sorted(expand(line))) # Here's where we keep track of what we've said recently. Remember, # this is by line in the .dialog file, not by exact phrase @@ -104,6 +115,8 @@ def render(self, template_name, context=None, index=None): return line +@deprecated("use 'ovos_spec_tools.LocaleResources' to load .dialog resources", + f"{VERSION_MAJOR + 1}.0.0") def load_dialogs(dialog_dir: str, renderer: Optional[MustacheDialogRenderer] = None) -> \ MustacheDialogRenderer: @@ -116,6 +129,10 @@ def load_dialogs(dialog_dir: str, Returns: a loaded instance of a dialog renderer """ + warnings.warn( + "load_dialogs is deprecated; use 'ovos_spec_tools.LocaleResources' " + "to load .dialog resources", + DeprecationWarning, stacklevel=3) if renderer is None: renderer = MustacheDialogRenderer() @@ -132,6 +149,8 @@ def load_dialogs(dialog_dir: str, return renderer +@deprecated("use the OVOS-INTENT-2 §4.2 dialog renderer in 'ovos_spec_tools' " + "('render' / 'DialogRenderer')", f"{VERSION_MAJOR + 1}.0.0") def get_dialog(phrase: str, lang: str = None, context: Optional[dict] = None) -> str: """ @@ -149,6 +168,10 @@ def get_dialog(phrase: str, lang: str = None, str: a randomized and/or translated version of the phrase """ + warnings.warn( + "get_dialog is deprecated; use the OVOS-INTENT-2 §4.2 dialog renderer " + "in 'ovos_spec_tools' ('render' / 'DialogRenderer')", + DeprecationWarning, stacklevel=3) if not lang: log_deprecation("Expected a string lang and got None.", "0.1.0") try: diff --git a/ovos_utils/fakebus.py b/ovos_utils/fakebus.py index c0287274..3e4d6dd2 100644 --- a/ovos_utils/fakebus.py +++ b/ovos_utils/fakebus.py @@ -1,8 +1,10 @@ -import json -from copy import deepcopy -from threading import Event +import asyncio import warnings +from os import environ +from threading import Event + from ovos_utils.log import LOG, log_deprecation +from ovos_spec_tools import NamespaceTranslator from pyee import EventEmitter @@ -15,12 +17,59 @@ def dig_for_message(): return None +# sentinel: lets us tell "kwarg not passed" apart from "kwarg passed True/False" +_UNSET = object() + + +def _bus_flag(env_var, config_key, default=True): + """Resolve a boolean bus flag the way ``MessageBusClient._bus_flag`` does. + + Precedence: env var (when set) > ``websocket.`` in ovos_config + > ``default``. The env var wins when set to a truthy/falsy string; ovos_config + is optional, so any failure to read it falls back to ``default``. + + Kept layering-clean: mirrors ``ovos_bus_client.client.client._bus_flag`` + without importing from bus-client (bus-client depends on utils, not vice-versa). + """ + val = environ.get(env_var) + if val is not None: + return val.strip().lower() in ("1", "true", "yes", "on") + try: + from ovos_config import Configuration + return bool(Configuration().get("websocket", {}).get(config_key, default)) + except Exception: + return default + + +def _resolve_bus_flags(kwargs): + """Build the namespace ``NamespaceTranslator`` for a fake bus instance. + + An explicitly-passed ``modernize``/``emit_legacy`` kwarg wins (back-compat for + callers passing ``emit_legacy=True/False``); otherwise the flag is resolved via + env var -> ``websocket.*`` config -> default ``True``, matching the real client. + """ + modernize = kwargs.get("modernize", _UNSET) + if modernize is _UNSET: + modernize = _bus_flag("OVOS_BUS_MODERNIZE", "modernize", default=True) + emit_legacy = kwargs.get("emit_legacy", _UNSET) + if emit_legacy is _UNSET: + emit_legacy = _bus_flag("OVOS_BUS_EMIT_LEGACY", "emit_legacy", default=True) + return NamespaceTranslator(modernize=modernize, emit_legacy=emit_legacy) + + class FakeBus: def __init__(self, *args, **kwargs): self.started_running = False self.session_id = "default" self.ee = kwargs.get("emitter") or EventEmitter() self.ee.on("error", self.on_error) + # mirror MessageBusClient's namespace migration so the test/satellite + # double bridges legacy<->ovos.* topics identically. Flags resolve the + # same way the real client does: explicit modernize=/emit_legacy= kwarg + # wins, else env var -> websocket.* config -> default on. + self._translator = _resolve_bus_flags(kwargs) + self._handler_guards = {} # handler -> shared mirror-guard + self._dedup_registrations = {} # handler -> [(msg_type, wrapped), ...] self.on_open() try: self.session_id = kwargs["session"].session_id @@ -31,6 +80,22 @@ def __init__(self, *args, **kwargs): self.on_default_session_update) def on(self, msg_type, handler): + # wrap handlers on migrated topics so a handler subscribed to both the + # legacy and ovos.* topic fires once (the mirror is dropped) + if self._translator.is_migrated(msg_type): + guard = self._handler_guards.get(handler) + if guard is None: + guard = self._translator.new_mirror_guard() + self._handler_guards[handler] = guard + + def wrapped(message=None): + if guard(message): + return + return handler(message) + + self.ee.on(msg_type, wrapped) + self._dedup_registrations.setdefault(handler, []).append((msg_type, wrapped)) + return self.ee.on(msg_type, handler) def once(self, msg_type, handler): @@ -50,6 +115,20 @@ def emit(self, message): self.ee.emit(message.msg_type, message) except Exception as e: LOG.exception(f"Error in event handler for '{message.msg_type}': {e}") + # namespace migration: also dispatch the counterpart topic(s) so a + # listener on either namespace receives the event (consumers dedupe). + # the mirrored payload is reshaped into the counterpart topic's shape + # (identity for payload-compatible renames, a per-topic transform for + # shape-changing ones) so a listener on it receives the payload in *its* + # shape -- matching MessageBusClient's bridge. + for topic in self._translator.counterpart_topics(message.msg_type): + try: + translated = self._translator.translate_payload( + from_topic=message.msg_type, to_topic=topic, + data=message.data) + self.ee.emit(topic, message.forward(topic, translated)) + except Exception as e: + LOG.exception(f"Error in counterpart dispatch for '{topic}': {e}") self.on_message(message.serialize()) def on_message(self, *args): @@ -66,9 +145,10 @@ def on_message(self, *args): try: # replicate side effects from ovos_bus_client.session import Session, SessionManager sess = Session.from_message(parsed_message) - if sess.session_id != "default": - # 'default' can only be updated by core - SessionManager.update(sess) + # every session — including the default id — folds onto the singleton + # (value-passing; nothing is owner-only, matching the spec-tools + # SessionManager and the real MessageBusClient) + SessionManager.update(sess) except ImportError: pass # don't care @@ -77,7 +157,10 @@ def on_default_session_update(self, message): from ovos_bus_client.session import Session, SessionManager new_session = message.data["session_data"] sess = Session.deserialize(new_session) - SessionManager.update(sess, make_default=True) + # payload is default_session.serialize() (id == "default"); the + # SessionManager singleton syncs default_session by id, so the + # deprecated make_default flag is not needed. + SessionManager.update(sess) LOG.debug("synced default_session") except ImportError: pass # don't care @@ -135,6 +218,18 @@ def rcv(m): return msg def remove(self, msg_type, handler): + regs = self._dedup_registrations.get(handler) + if regs: + for ev, wrapped in [r for r in regs if r[0] == msg_type]: + try: + self.ee.remove_listener(ev, wrapped) + except Exception: + pass + regs.remove((ev, wrapped)) + if not regs: + self._dedup_registrations.pop(handler, None) + self._handler_guards.pop(handler, None) + return try: self.ee.remove_listener(msg_type, handler) except Exception: @@ -165,182 +260,340 @@ def close(self): self.on_close() -class _MutableMessage(type): - """ To override isinstance checks we need to use a metaclass """ +# The reference Message envelope lives in ovos-spec-tools (OVOS-MSG-1). +# ovos-utils re-exports it under the historical ``FakeMessage`` name and +# attaches the one legacy convenience method downstream still uses — +# ``publish`` — to the class at import time. ``as_dict`` is now on the +# spec-tools class itself; the ``data['destination']`` promotion the +# old ``reply`` did was always a bug (data is the payload, context owns +# routing) and is gone. +# +# The old ``_MutableMessage`` metaclass / dynamic ``__new__`` indirection +# (which tried to return an ``ovos_bus_client.Message`` at runtime if +# bus-client was installed) is no longer needed: spec-tools is a hard +# dependency, the canonical class is always present, and +# ``ovos-bus-client.Message`` is the **same** class (bus-client attaches +# ``publish`` to it too — both attachments are idempotent). +from typing import Any, Dict, Optional + +from ovos_spec_tools.message import Message as FakeMessage +from ovos_utils.log import deprecated +from ovos_utils.version import VERSION_MAJOR + + +# OVOS-MSG-1 defines forward / reply / response as the three normative +# derivations (§5). ``publish`` is a bus-client tradition outside the +# spec; it survives as an attached method for one more major release so +# downstream consumers can migrate. +_PUBLISH_REMOVAL_VERSION = f"{VERSION_MAJOR + 1}.0.0" + + +@deprecated( + "Message.publish is deprecated; use Message.forward (relay under a " + "new topic, preserves context) or Message.reply (§5.2 swap) — both " + "are OVOS-MSG-1 normative", + _PUBLISH_REMOVAL_VERSION) +def _publish(self, msg_type: str, data: Dict[str, Any], + context: Optional[Dict[str, Any]] = None) -> FakeMessage: + """Relay under a new topic without the §5.2 swap; drop ``target``. + + .. deprecated:: + Not part of OVOS-MSG-1 (the spec defines ``forward`` / + ``reply`` / ``response`` as the only normative derivations). + Slated for removal in the next major; use :meth:`forward` + when you do not want the routing-key swap, or :meth:`reply` + when you do. + """ + import warnings + # stacklevel=3: warn() -> body -> @deprecated wrapper -> caller + warnings.warn( + "Message.publish is deprecated; use Message.forward (no §5.2 " + "swap) or Message.reply (with swap) instead — both are " + "OVOS-MSG-1 normative derivations. ``publish`` will be removed " + f"in ovos-utils {_PUBLISH_REMOVAL_VERSION}.", + DeprecationWarning, stacklevel=3) + context = context or {} + new_context = dict(self.context) + new_context.update(context) + new_context.pop("target", None) + return self.__class__(msg_type, data, new_context) + + +# Attach publish() to the spec-tools Message so the method appears on +# every Message instance regardless of which package the caller imported +# the class from. Idempotent with ovos-bus-client's identical attachment. +FakeMessage.publish = _publish - def __instancecheck__(self, instance): - try: - from ovos_bus_client.message import Message as _MycroftMessage - if isinstance(instance, _MycroftMessage): - return True - except ImportError: - pass - return super().__instancecheck__(instance) +class Message(FakeMessage): + """Deprecated alias for the OVOS-MSG-1 ``Message`` envelope. -# fake Message object to allow usage without ovos-bus-client installed -class FakeMessage(metaclass=_MutableMessage): - """ fake Message object to allow usage with FakeBus without ovos-bus-client installed""" + ``from ovos_utils.fakebus import Message`` is in the wild and stays + importable through one more release. New code should import the + envelope where it lives — :class:`ovos_spec_tools.Message` (or + :class:`ovos_bus_client.Message`, which is a subclass). + """ def __new__(cls, *args, **kwargs): - try: # most common case - from ovos_bus_client import Message as _M - return _M(*args, **kwargs) - except ImportError: - pass - return super().__new__(cls) + warnings.warn( + "ovos_utils.fakebus.Message is deprecated; import " + "ovos_spec_tools.Message (or ovos_bus_client.Message)", + DeprecationWarning, + stacklevel=2, + ) + log_deprecation( + "please import Message from ovos_spec_tools / " + "ovos_bus_client directly", "1.0.0") + return FakeMessage(*args, **kwargs) - def __init__(self, msg_type, data=None, context=None): - """Used to construct a message object - Message objects will be used to send information back and forth - between processes of mycroft service, voice, skill and cli - """ - self.msg_type = msg_type - self.data = data or {} - self.context = context or {} +class AsyncFakeBus: + """In-process stand-in for ``AsyncMessageBusClient``. + + Mirrors the same surface as the real async bus client: ``connect`` / + ``close`` / ``emit`` / ``wait_for_message`` / ``wait_for_response`` are + coroutines; ``on`` / ``once`` / ``remove`` stay synchronous. + + No WebSocket, no thread, no real I/O — every emit dispatches + synchronously through a ``pyee.EventEmitter`` to whatever handlers + are registered. - def __eq__(self, other): + Useful both in tests (drop-in for ``AsyncMessageBusClient``) and at + runtime (anywhere a sync component expects the legacy ``FakeBus`` but + the surrounding code is asyncio-native). + + The session-injection side effects match ``FakeBus`` so multi-turn + flows behave identically. + """ + + def __init__(self, *args, **kwargs): + self.started_running = False + self.session_id = "default" + self.ee = kwargs.get("emitter") or EventEmitter() + self.ee.on("error", self.on_error) + # mirror MessageBusClient's namespace migration (see FakeBus.__init__). + self._translator = _resolve_bus_flags(kwargs) + self._handler_guards = {} # handler -> shared mirror-guard + self._dedup_registrations = {} # handler -> [(msg_type, wrapped), ...] + self.connected_event = asyncio.Event() + self.connected_event.set() + self.on_open() try: - return other.msg_type == self.msg_type and \ - other.data == self.data and \ - other.context == self.context + self.session_id = kwargs["session"].session_id except Exception: - return False + pass # don't care - def serialize(self): - """This returns a string of the message info. + self.on("ovos.session.update_default", + self.on_default_session_update) - This makes it easy to send over a websocket. This uses - json dumps to generate the string with type, data and context + # ------------------------------------------------------------------ + # Handler registration (sync — matches AsyncMessageBusClient) + # ------------------------------------------------------------------ - Returns: - str: a json string representation of the message. - """ - return json.dumps({'type': self.msg_type, - 'data': self.data, - 'context': self.context}) + def on(self, msg_type, handler): + # wrap handlers on migrated topics so a handler subscribed to both the + # legacy and ovos.* topic fires once (the mirror is dropped) -- same as + # FakeBus.on / MessageBusClient.on. + if self._translator.is_migrated(msg_type): + guard = self._handler_guards.get(handler) + if guard is None: + guard = self._translator.new_mirror_guard() + self._handler_guards[handler] = guard + + def wrapped(message=None): + if guard(message): + return + return handler(message) + + self.ee.on(msg_type, wrapped) + self._dedup_registrations.setdefault(handler, []).append((msg_type, wrapped)) + return + self.ee.on(msg_type, handler) + + def once(self, msg_type, handler): + self.ee.once(msg_type, handler) - @staticmethod - def deserialize(value): - """This takes a string and constructs a message object. + def remove(self, msg_type, handler): + regs = self._dedup_registrations.get(handler) + if regs: + for ev, wrapped in [r for r in regs if r[0] == msg_type]: + try: + self.ee.remove_listener(ev, wrapped) + except Exception: + pass + regs.remove((ev, wrapped)) + if not regs: + self._dedup_registrations.pop(handler, None) + self._handler_guards.pop(handler, None) + return + try: + self.ee.remove_listener(msg_type, handler) + except Exception: + pass - This makes it easy to take strings from the websocket and create - a message object. This uses json loads to get the info and generate - the message object. + def remove_all_listeners(self, event_name): + self.ee.remove_all_listeners(event_name) - Args: - value(str): This is the json string received from the websocket + # ------------------------------------------------------------------ + # Lifecycle (async) + # ------------------------------------------------------------------ - Returns: - FakeMessage: message object constructed from the json string passed - int the function. - value(str): This is the string received from the websocket + async def connect(self, *args, **kwargs): + """No-op for the fake bus; matches the real client's lifecycle. + + Returns immediately with ``connected_event`` set. """ - obj = json.loads(value) - return FakeMessage(obj.get('type') or '', - obj.get('data') or {}, - obj.get('context') or {}) + self.started_running = True + self.connected_event.set() + return self + + async def close(self): + self.connected_event.clear() + self.on_close() - def forward(self, msg_type, data=None): - """ Keep context and forward message + # ------------------------------------------------------------------ + # emit (async) — same dispatch shape as FakeBus.emit + # ------------------------------------------------------------------ + + async def emit(self, message): + if "session" not in message.context: + try: # replicate side effects + from ovos_bus_client.session import Session, SessionManager + sess = SessionManager.sessions.get(self.session_id) or \ + Session(self.session_id) + message.context["session"] = sess.serialize() + except ImportError: # don't care + message.context["session"] = {"session_id": self.session_id} + self.ee.emit("message", message.serialize()) + try: + self.ee.emit(message.msg_type, message) + except Exception as e: + LOG.exception(f"Error in event handler for '{message.msg_type}': {e}") + # namespace migration: also dispatch the counterpart topic(s) with the + # payload reshaped into each counterpart's shape -- same as FakeBus.emit. + for topic in self._translator.counterpart_topics(message.msg_type): + try: + translated = self._translator.translate_payload( + from_topic=message.msg_type, to_topic=topic, + data=message.data) + self.ee.emit(topic, message.forward(topic, translated)) + except Exception as e: + LOG.exception(f"Error in counterpart dispatch for '{topic}': {e}") + self.on_message(message.serialize()) - This will take the same parameters as a message object but use - the current message object as a reference. It will copy the context - from the existing message object. + # ------------------------------------------------------------------ + # Sync helpers used internally — same as FakeBus + # ------------------------------------------------------------------ - Args: - msg_type (str): type of message - data (dict): data for message + def on_message(self, *args): + """Handle an incoming websocket message. - Returns: - FakeMessage: Message object to be used on the reply to the message + @param args: + message (str): serialized Message """ - data = data or {} - return FakeMessage(msg_type, data, context=self.context) - - def reply(self, msg_type, data=None, context=None): - """Construct a reply message for a given message - - This will take the same parameters as a message object but use - the current message object as a reference. It will copy the context - from the existing message object and add any context passed in to - the function. Check for a destination passed in to the function from - the data object and add that to the context as a destination. If the - context has a source then that will be swapped with the destination - in the context. The new message will then have data passed in plus the - new context generated. - - Args: - msg_type (str): type of message - data (dict): data for message - context: intended context for new message + if len(args) == 1: + message = args[0] + else: + message = args[1] + parsed_message = FakeMessage.deserialize(message) + try: # replicate side effects + from ovos_bus_client.session import Session, SessionManager + sess = Session.from_message(parsed_message) + # every session — including the default id — folds onto the singleton + # (value-passing; nothing is owner-only, matching the spec-tools + # SessionManager and the real MessageBusClient) + SessionManager.update(sess) + except ImportError: + pass # don't care + + def on_default_session_update(self, message): + try: # replicate side effects + from ovos_bus_client.session import Session, SessionManager + new_session = message.data["session_data"] + sess = Session.deserialize(new_session) + # payload is default_session.serialize() (id == "default"); the + # SessionManager singleton syncs default_session by id, so the + # deprecated make_default flag is not needed. + SessionManager.update(sess) + LOG.debug("synced default_session") + except ImportError: + pass # don't care + + def on_error(self, error): + LOG.error(error) + + def on_open(self): + pass + + def on_close(self): + pass + + # ------------------------------------------------------------------ + # Waiters (async) + # ------------------------------------------------------------------ + + async def wait_for_message(self, message_type, timeout=3.0): + """Wait for a message of a specific type. + + Arguments: + message_type (str): the message type of the expected message + timeout: seconds to wait before timeout, defaults to 3 Returns: - FakeMessage: Message object to be used on the reply to the message - """ - data = deepcopy(data) or {} - context = context or {} - - new_context = deepcopy(self.context) - for key in context: - new_context[key] = context[key] - if 'destination' in data: - new_context['destination'] = data['destination'] - if 'source' in new_context and 'destination' in new_context: - s = new_context['destination'] - new_context['destination'] = new_context['source'] - new_context['source'] = s - return FakeMessage(msg_type, data, context=new_context) - - def response(self, data=None, context=None): - """Construct a response message for the message - - Constructs a reply with the data and appends the expected - ".response" to the message - - Args: - data (dict): message data - context (dict): message context - Returns - (Message) message with the type modified to match default response + The received message or None if the response timed out """ - return self.reply(self.msg_type + '.response', data, context) + evt = asyncio.Event() + captured = {"msg": None} - def publish(self, msg_type, data, context=None): - """ - Copy the original context and add passed in context. Delete - any target in the new context. Return a new message object with - passed in data and new context. Type remains unchanged. + def _rcv(m): + captured["msg"] = m + evt.set() + + self.ee.once(message_type, _rcv) + try: + await asyncio.wait_for(evt.wait(), timeout=timeout) + except asyncio.TimeoutError: + pass + return captured["msg"] - Args: - msg_type (str): type of message - data (dict): date to send with message - context: context added to existing context + async def wait_for_response(self, message, reply_type=None, timeout=3.0): + """Send a message and wait for a response. + + Arguments: + message (Message): message to send + reply_type (str): the message type of the expected reply. + Defaults to ".response". + timeout: seconds to wait before timeout, defaults to 3 Returns: - FakeMessage: Message object to publish + The received message or None if the response timed out """ - context = context or {} - new_context = self.context.copy() - for key in context: - new_context[key] = context[key] + reply_type = reply_type or message.msg_type + ".response" + evt = asyncio.Event() + captured = {"msg": None} - if 'target' in new_context: - del new_context['target'] + def _rcv(m): + captured["msg"] = m + evt.set() - return FakeMessage(msg_type, data, context=new_context) + self.ee.once(reply_type, _rcv) + await self.emit(message) + try: + await asyncio.wait_for(evt.wait(), timeout=timeout) + except asyncio.TimeoutError: + pass + return captured["msg"] + # ------------------------------------------------------------------ + # Backwards-compat passthroughs so AsyncFakeBus is a drop-in even for + # code paths that still call the threading-era helpers. + # ------------------------------------------------------------------ -class Message(FakeMessage): - """just for compat, stuff in the wild importing from here even with deprecation warnings...""" + def create_client(self): + return self - def __new__(cls, *args, **kwargs): - warnings.warn( - "import from ovos-bus-client directly", - DeprecationWarning, - stacklevel=2, - ) - log_deprecation( - "please import from ovos-bus-client directly! this import has been deprecated since version 0.1.0", "1.0.0") - return FakeMessage(*args, **kwargs) + def run_forever(self): + self.started_running = True + + def run_in_thread(self): + self.run_forever() diff --git a/ovos_utils/file_utils.py b/ovos_utils/file_utils.py index 55bb8fa8..881578e0 100644 --- a/ovos_utils/file_utils.py +++ b/ovos_utils/file_utils.py @@ -9,7 +9,7 @@ from threading import RLock from typing import Optional, List -from ovos_utils.bracket_expansion import expand_template +from ovos_spec_tools import expand from ovos_utils.log import LOG, log_deprecation @@ -238,7 +238,7 @@ def read_vocab_file(path: str) -> List[List[str]]: for line in voc_file.readlines(): if line.startswith('#') or line.strip() == '': continue - vocab.append(expand_template(line.lower())) + vocab.append(sorted(expand(line.lower()))) return vocab diff --git a/ovos_utils/geolocation.py b/ovos_utils/geolocation.py index b9b20494..a586a9f6 100644 --- a/ovos_utils/geolocation.py +++ b/ovos_utils/geolocation.py @@ -3,8 +3,9 @@ import requests from requests.exceptions import RequestException, Timeout +from ovos_spec_tools import standardize_lang + from ovos_utils import timed_lru_cache -from ovos_utils.lang import standardize_lang_tag from ovos_utils.log import LOG from ovos_utils.network_utils import get_external_ip, is_valid_ip @@ -234,7 +235,7 @@ def get_ip_geolocation(ip: Optional[str] = None, raise ValueError(f"Invalid IP address: {ip}") # normalize language to expected values by ip-api.com - lang = standardize_lang_tag(lang).split("-")[0] + lang = standardize_lang(lang).split("-")[0] if lang not in ["en", "de", "es", "pt", "fr", "ja", "zh", "ru"]: LOG.warning(f"Language unsupported by ip-api.com ({lang}), defaulting to english") lang = "en" diff --git a/ovos_utils/lang/__init__.py b/ovos_utils/lang/__init__.py index 25c63471..3a5fd316 100644 --- a/ovos_utils/lang/__init__.py +++ b/ovos_utils/lang/__init__.py @@ -1,47 +1,71 @@ +import warnings from os import listdir from os.path import isdir, join from typing import Optional from ovos_utils.file_utils import resolve_resource_file +from ovos_utils.log import deprecated +from ovos_utils.version import VERSION_MAJOR +@deprecated("use 'standardize_lang' from 'ovos_spec_tools' instead", + f"{VERSION_MAJOR + 1}.0.0") def standardize_lang_tag(lang_code: str, macro=True) -> str: - """https://langcodes-hickford.readthedocs.io/en/sphinx/index.html""" + """Normalize a BCP-47 language tag. + + ``macro`` controls **macrolanguage substitution** per + :func:`langcodes.standardize_tag` — it swaps a sublanguage for its + macrolanguage (``cmn`` -> ``zh``, ``nb`` -> ``no``). It does **not** + strip the region: ``"en-US"`` round-trips through both ``macro=True`` + and ``macro=False`` unchanged. + + .. deprecated:: + Use :func:`ovos_spec_tools.standardize_lang` — the conformant OVOS + language-tag normalizer. ``standardize_lang`` always returns the + region-preserving form (it does not take a ``macro`` argument); + if you need macrolanguage substitution, call + :func:`langcodes.standardize_tag` directly. + """ + # stacklevel=3: warn() -> this body -> @deprecated wrapper -> caller + warnings.warn("standardize_lang_tag is deprecated; use 'standardize_lang' " + "from 'ovos_spec_tools' instead", + DeprecationWarning, stacklevel=3) try: - from langcodes import standardize_tag as std - return str(std(lang_code, macro=macro)) - except Exception: - if macro: - return lang_code.split("-")[0].lower() - if "-" in lang_code: - a, b = lang_code.split("-", 1) - return f"{a.lower()}-{b.upper()}" - return lang_code.lower() + from langcodes import standardize_tag + return str(standardize_tag(lang_code, macro=macro)) + except ImportError: + # langcodes is optional. Without it, fall back to the spec-tools + # normalizer (region-preserving). The ``macro`` argument is a + # no-op in this branch because macrolanguage tables live in + # langcodes itself. + from ovos_spec_tools import standardize_lang + return standardize_lang(lang_code) -def get_language_dir(base_path: str, lang: str ="en-US") -> Optional[str]: - """ checks for all language variations and returns best path """ - lang = standardize_lang_tag(lang) +@deprecated("use 'closest_lang' from 'ovos_spec_tools' " + "(or 'ovos_spec_tools.LocaleResources')", + f"{VERSION_MAJOR + 1}.0.0") +def get_language_dir(base_path: str, lang: str = "en-US") -> Optional[str]: + """Return the best-matching ``/`` directory under ``base_path``. - candidates = [] - for f in listdir(base_path): - if isdir(f"{base_path}/{f}"): - try: - from langcodes import tag_distance - score = tag_distance(lang, f) - except Exception: # not a valid language code - continue - # https://langcodes-hickford.readthedocs.io/en/sphinx/index.html#distance-values - # 0 -> These codes represent the same language, possibly after filling in values and normalizing. - # 1- 3 -> These codes indicate a minor regional difference. - # 4 - 10 -> These codes indicate a significant but unproblematic regional difference. - if score < 10: - candidates.append((f"{base_path}/{f}", score)) - if not candidates: + .. deprecated:: + Use :func:`ovos_spec_tools.closest_lang` to resolve a language tag + against the available ones, or :class:`ovos_spec_tools.LocaleResources` + which resolves locale directories itself. + """ + # stacklevel=3: warn() -> this body -> @deprecated wrapper -> caller + warnings.warn("get_language_dir is deprecated; use 'closest_lang' from " + "'ovos_spec_tools' (or 'ovos_spec_tools.LocaleResources')", + DeprecationWarning, stacklevel=3) + from ovos_spec_tools import closest_lang + try: + names = [f for f in listdir(base_path) if isdir(join(base_path, f))] + except (FileNotFoundError, NotADirectoryError): return None - # sort by distance to target lang code - candidates = sorted(candidates, key=lambda k: k[1]) - return candidates[0][0] + # closest_lang accepts a tag distance below 10 — the same threshold this + # used previously (OVOS-INTENT-2 §2.2). + match = closest_lang(lang, names) + return join(base_path, match) if match is not None else None def translate_word(name, lang='en-US'): diff --git a/ovos_utils/version.py b/ovos_utils/version.py index 3a22553d..ea59d7f0 100644 --- a/ovos_utils/version.py +++ b/ovos_utils/version.py @@ -1,8 +1,8 @@ # START_VERSION_BLOCK VERSION_MAJOR = 0 -VERSION_MINOR = 8 -VERSION_BUILD = 5 -VERSION_ALPHA = 4 +VERSION_MINOR = 13 +VERSION_BUILD = 3 +VERSION_ALPHA = 1 # END_VERSION_BLOCK __version__ = f"{VERSION_MAJOR}.{VERSION_MINOR}.{VERSION_BUILD}" + (f"a{VERSION_ALPHA}" if VERSION_ALPHA else "") diff --git a/pyproject.toml b/pyproject.toml index d4534048..597e101a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -13,7 +13,7 @@ requires-python = ">=3.9" dependencies = [ "pexpect~=4.9", "requests~=2.26", - "json_database~=0.10", + "json_database>=0.10,<2.0.0", "kthread~=0.2", "watchdog", "pyee>=8.0.0", @@ -21,6 +21,7 @@ dependencies = [ "rich-click~=1.7", "rich~=13.7", "python-dateutil", + "ovos-spec-tools>=0.16.1a2", ] [project.urls] @@ -33,7 +34,7 @@ extras = [ "ovos-plugin-manager>=0.0.25", "ovos-config>=0.0.12", "ovos-workshop>=0.0.13", - "ovos_bus_client>=0.0.8", + "ovos_bus_client>=2.6.2a2", "langcodes", "timezonefinder", "oauthlib~=3.2", diff --git a/requirements/extras.txt b/requirements/extras.txt deleted file mode 100644 index 2f77edff..00000000 --- a/requirements/extras.txt +++ /dev/null @@ -1,9 +0,0 @@ -rapidfuzz>=3.6,<4.0 -ovos-plugin-manager>=0.0.25 -ovos-config>=0.0.12 -ovos-workshop>=0.0.13 -ovos_bus_client>=0.0.8 -langcodes -timezonefinder -oauthlib~=3.2 -orjson \ No newline at end of file diff --git a/requirements/requirements.txt b/requirements/requirements.txt deleted file mode 100644 index 49a447aa..00000000 --- a/requirements/requirements.txt +++ /dev/null @@ -1,9 +0,0 @@ -pexpect~=4.9 -requests~=2.26 -json_database~=0.10 -kthread~=0.2 -watchdog -pyee>=8.0.0 -combo-lock~=0.2 -rich-click~=1.7 -rich~=13.7 \ No newline at end of file diff --git a/test/unittests/test_async_fakebus.py b/test/unittests/test_async_fakebus.py new file mode 100644 index 00000000..a3a1f79d --- /dev/null +++ b/test/unittests/test_async_fakebus.py @@ -0,0 +1,267 @@ +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +"""Tests for AsyncFakeBus. + +Mirrors test_fakebus.py shape but exercises the async surface +(connect / close / emit / wait_for_message / wait_for_response) plus +the sync handler-registration contract that matches +AsyncMessageBusClient. +""" +import asyncio +import unittest + +from ovos_utils.fakebus import AsyncFakeBus, FakeMessage + + +def _run(coro): + """Tiny helper so we can use plain unittest.TestCase.""" + return asyncio.run(coro) + + +class TestAsyncFakeBusLifecycle(unittest.TestCase): + def test_constructs_connected(self): + bus = AsyncFakeBus() + self.assertTrue(bus.connected_event.is_set()) + + def test_session_id_from_kwargs(self): + class _Sess: + session_id = "from-kwarg" + bus = AsyncFakeBus(session=_Sess()) + self.assertEqual(bus.session_id, "from-kwarg") + + def test_connect_is_noop_but_sets_event(self): + bus = AsyncFakeBus() + bus.connected_event.clear() + _run(bus.connect()) + self.assertTrue(bus.connected_event.is_set()) + self.assertTrue(bus.started_running) + + def test_close_clears_connected_event(self): + bus = AsyncFakeBus() + self.assertTrue(bus.connected_event.is_set()) + _run(bus.close()) + self.assertFalse(bus.connected_event.is_set()) + + +class TestAsyncFakeBusHandlerRegistration(unittest.TestCase): + def test_on_then_emit_dispatches(self): + bus = AsyncFakeBus() + seen = [] + bus.on("hello", lambda m: seen.append(m)) + _run(bus.emit(FakeMessage("hello", {"x": 1}))) + self.assertEqual(len(seen), 1) + self.assertEqual(seen[0].msg_type, "hello") + self.assertEqual(seen[0].data["x"], 1) + + def test_once_fires_only_once(self): + bus = AsyncFakeBus() + seen = [] + bus.once("evt", lambda m: seen.append(m)) + _run(bus.emit(FakeMessage("evt"))) + _run(bus.emit(FakeMessage("evt"))) + self.assertEqual(len(seen), 1) + + def test_remove_handler(self): + bus = AsyncFakeBus() + seen = [] + + def cb(m): + seen.append(m) + + bus.on("evt", cb) + bus.remove("evt", cb) + _run(bus.emit(FakeMessage("evt"))) + self.assertEqual(seen, []) + + def test_remove_all_listeners(self): + bus = AsyncFakeBus() + bus.on("evt", lambda m: None) + bus.on("evt", lambda m: None) + bus.remove_all_listeners("evt") + self.assertEqual(bus.ee.listeners("evt"), []) + + def test_remove_unknown_handler_does_not_raise(self): + bus = AsyncFakeBus() + # not registered → silent + bus.remove("evt", lambda m: None) + + +class TestAsyncFakeBusEmit(unittest.TestCase): + def test_emit_injects_session_context_when_missing(self): + bus = AsyncFakeBus() + msg = FakeMessage("hello", {}) + self.assertNotIn("session", msg.context) + _run(bus.emit(msg)) + self.assertIn("session", msg.context) + + def test_emit_dispatches_raw_message_event(self): + bus = AsyncFakeBus() + raws = [] + bus.on("message", lambda raw: raws.append(raw)) + _run(bus.emit(FakeMessage("hello"))) + self.assertEqual(len(raws), 1) + self.assertIn("hello", raws[0]) + + +class TestAsyncFakeBusWaitForMessage(unittest.TestCase): + def test_returns_matched_message_emitted_concurrently(self): + bus = AsyncFakeBus() + + async def scenario(): + async def feed(): + await asyncio.sleep(0.02) + await bus.emit(FakeMessage("ping", {"flood_id": "x"})) + asyncio.create_task(feed()) + got = await bus.wait_for_message("ping", timeout=1.0) + return got + + got = _run(scenario()) + self.assertIsNotNone(got) + self.assertEqual(got.msg_type, "ping") + + def test_returns_none_on_timeout(self): + bus = AsyncFakeBus() + + async def scenario(): + return await bus.wait_for_message("never", timeout=0.05) + + self.assertIsNone(_run(scenario())) + + +class TestAsyncFakeBusWaitForResponse(unittest.TestCase): + def test_default_reply_type_is_msg_type_response(self): + bus = AsyncFakeBus() + + async def scenario(): + # echo the request as .response when the request arrives + def echo(m): + # synchronous dispatch — fire the reply inline + # cannot await here; schedule on the loop instead + asyncio.create_task( + bus.emit(FakeMessage(m.msg_type + ".response", + {"echoed": m.data}))) + bus.on("ask", echo) + return await bus.wait_for_response( + FakeMessage("ask", {"q": 1}), timeout=1.0, + ) + + reply = _run(scenario()) + self.assertIsNotNone(reply) + self.assertEqual(reply.msg_type, "ask.response") + self.assertEqual(reply.data["echoed"], {"q": 1}) + + def test_explicit_reply_type(self): + bus = AsyncFakeBus() + + async def scenario(): + def respond(m): + asyncio.create_task(bus.emit(FakeMessage("pong"))) + bus.on("ping", respond) + return await bus.wait_for_response( + FakeMessage("ping"), reply_type="pong", timeout=1.0, + ) + + reply = _run(scenario()) + self.assertIsNotNone(reply) + self.assertEqual(reply.msg_type, "pong") + + def test_returns_none_on_timeout(self): + bus = AsyncFakeBus() + + async def scenario(): + return await bus.wait_for_response( + FakeMessage("never"), timeout=0.05, + ) + + self.assertIsNone(_run(scenario())) + + +class TestAsyncFakeBusCompatShims(unittest.TestCase): + def test_create_client_returns_self(self): + bus = AsyncFakeBus() + self.assertIs(bus.create_client(), bus) + + def test_run_forever_flips_started_running(self): + bus = AsyncFakeBus() + bus.started_running = False + bus.run_forever() + self.assertTrue(bus.started_running) + + def test_run_in_thread_alias(self): + bus = AsyncFakeBus() + bus.started_running = False + bus.run_in_thread() + self.assertTrue(bus.started_running) + + +class TestAsyncFakeBusNamespaceMigration(unittest.TestCase): + """AsyncFakeBus mirrors FakeBus / MessageBusClient namespace migration.""" + + def test_legacy_emit_reaches_spec_listener(self): + bus = AsyncFakeBus() # both flags default on + got = [] + bus.on("ovos.utterance.speak", lambda m: got.append(m.msg_type)) + _run(bus.emit(FakeMessage("speak", {"utterance": "hi"}))) + self.assertEqual(got, ["ovos.utterance.speak"]) # modernize bridged it + + def test_spec_emit_reaches_legacy_listener(self): + bus = AsyncFakeBus() + got = [] + bus.on("speak", lambda m: got.append(m.msg_type)) + _run(bus.emit(FakeMessage("ovos.utterance.speak", {"utterance": "hi"}))) + self.assertEqual(got, ["speak"]) # emit_legacy bridged it + + def test_counterpart_payload_is_translated(self): + # a spec listener on the counterpart of a SHAPE-CHANGING legacy topic + # receives the payload reshaped into ITS shape. detach_intent -> + # ovos.intent.deregister splits "skill:intent" into skill_id/intent_name. + bus = AsyncFakeBus() + got = [] + bus.on("ovos.intent.deregister", lambda m: got.append(dict(m.data))) + _run(bus.emit(FakeMessage("detach_intent", + {"intent_name": "skill.foo:HelloIntent"}))) + self.assertEqual(got, [{"skill_id": "skill.foo", "intent_name": "HelloIntent"}]) + + def test_dual_listener_fires_once(self): + bus = AsyncFakeBus() + calls = [] + handler = lambda m: calls.append(m.msg_type) + bus.on("speak", handler) + bus.on("ovos.utterance.speak", handler) + _run(bus.emit(FakeMessage("speak", {"utterance": "hi"}))) + self.assertEqual(len(calls), 1) # mirror deduped + + def test_distinct_listeners_each_fire_once(self): + bus = AsyncFakeBus() + legacy, spec = [], [] + bus.on("speak", lambda m: legacy.append(1)) + bus.on("ovos.utterance.speak", lambda m: spec.append(1)) + _run(bus.emit(FakeMessage("speak", {"utterance": "hi"}))) + self.assertEqual((len(legacy), len(spec)), (1, 1)) + + def test_flags_off_no_bridging(self): + bus = AsyncFakeBus(modernize=False, emit_legacy=False) + got = [] + bus.on("ovos.utterance.speak", lambda m: got.append(m.msg_type)) + _run(bus.emit(FakeMessage("speak", {"utterance": "hi"}))) + self.assertEqual(got, []) + + def test_remove_cleans_up(self): + bus = AsyncFakeBus() + calls = [] + handler = lambda m: calls.append(1) + bus.on("speak", handler) + bus.on("ovos.utterance.speak", handler) + bus.remove("speak", handler) + bus.remove("ovos.utterance.speak", handler) + self.assertNotIn(handler, bus._handler_guards) + _run(bus.emit(FakeMessage("speak", {"utterance": "hi"}))) + self.assertEqual(calls, []) + + +if __name__ == "__main__": + unittest.main() diff --git a/test/unittests/test_bracket_expansion.py b/test/unittests/test_bracket_expansion.py index 5d6bcbe1..f3b06bb2 100644 --- a/test/unittests/test_bracket_expansion.py +++ b/test/unittests/test_bracket_expansion.py @@ -35,20 +35,15 @@ def test_expand_slots(self): 'change the brightness to high and color to blue'] self.assertEqual(expanded_sentences, expected_sentences) + def test_malformed_template_raises(self): + # a template whose expansion would yield an empty string is malformed + # (OVOS-INTENT-1 §3.6) — it raises rather than producing '' + from ovos_spec_tools import MalformedTemplate + with self.assertRaises(MalformedTemplate): + expand_template("[(this|that) is optional]") + def test_expand_template(self): # Test for template expansion - templates = [ - "[hello,] (call me|my name is) {name}", - "Expand (alternative|choices) into a list of choices.", - "sentences have [optional] words ", - "alternative words can be (used|written)", - "sentence[s] can have (pre|suf)fixes mid word too", - "do( the | )thing(s|) (old|with) style and( no | )spaces", - "[(this|that) is optional]", - "tell me a [{joke_type}] joke", - "play {query} [in ({device_name}|{skill_name}|{zone_name})]" - ] - expected_outputs = { "[hello,] (call me|my name is) {name}": [ "call me {name}", @@ -60,9 +55,11 @@ def test_expand_template(self): "Expand alternative into a list of choices.", "Expand choices into a list of choices." ], + # an emptied [optional] no longer leaves a double space — + # OVOS-INTENT-1 §4.1 normalizes whitespace to single spaces "sentences have [optional] words ": [ - "sentences have words", - "sentences have optional words" + "sentences have optional words", + "sentences have words" ], "alternative words can be (used|written)": [ "alternative words can be used", @@ -92,12 +89,8 @@ def test_expand_template(self): "do things with style and no spaces", "do things with style and spaces" ], - "[(this|that) is optional]": [ - '', - 'that is optional', - 'this is optional'], "tell me a [{joke_type}] joke": [ - "tell me a joke", + "tell me a joke", "tell me a {joke_type} joke" ], "play {query} [in ({device_name}|{skill_name}|{zone_name})]": [ diff --git a/test/unittests/test_fakebus.py b/test/unittests/test_fakebus.py index 918fa4ff..a94d737d 100644 --- a/test/unittests/test_fakebus.py +++ b/test/unittests/test_fakebus.py @@ -57,9 +57,27 @@ def test_response(self) -> None: self.assertEqual(resp.msg_type, "my.request.response") self.assertEqual(resp.data["result"], "ok") + def test_publish_emits_deprecation_warning(self) -> None: + """``publish`` is not part of OVOS-MSG-1 and is scheduled for + removal — every call must fire a DeprecationWarning.""" + import warnings + msg = self._make_message("pub.type", {}, {"source": "x"}) + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + msg.publish("new.type", {"payload": 1}) + deps = [w for w in caught + if issubclass(w.category, DeprecationWarning) + and "publish" in str(w.message)] + self.assertTrue(deps, + "FakeMessage.publish() did not emit a " + "DeprecationWarning") + def test_publish(self) -> None: msg = self._make_message("pub.type", {}, {"target": "skill", "source": "x"}) - published = msg.publish("new.type", {"payload": 1}) + import warnings + with warnings.catch_warnings(): + warnings.simplefilter("ignore", DeprecationWarning) + published = msg.publish("new.type", {"payload": 1}) self.assertNotIn("target", published.context) self.assertEqual(published.data["payload"], 1) diff --git a/test/unittests/test_fakebus_namespace_migration.py b/test/unittests/test_fakebus_namespace_migration.py new file mode 100644 index 00000000..3eb4b79b --- /dev/null +++ b/test/unittests/test_fakebus_namespace_migration.py @@ -0,0 +1,150 @@ +"""FakeBus mirrors MessageBusClient's legacy<->ovos.* namespace migration, so +e2e/satellite tests exercise the real cross-namespace behaviour.""" +import asyncio +import unittest +from unittest.mock import patch + +from ovos_utils.fakebus import AsyncFakeBus, FakeBus, Message + + +def _run(coro): + return asyncio.run(coro) + + +class TestFakeBusNamespaceMigration(unittest.TestCase): + def test_legacy_emit_reaches_spec_listener(self): + bus = FakeBus() # both flags default on + got = [] + bus.on("ovos.utterance.speak", lambda m: got.append(m.msg_type)) + bus.emit(Message("speak", {"utterance": "hi"})) + self.assertEqual(got, ["ovos.utterance.speak"]) # modernize bridged it + + def test_spec_emit_reaches_legacy_listener(self): + bus = FakeBus() + got = [] + bus.on("speak", lambda m: got.append(m.msg_type)) + bus.emit(Message("ovos.utterance.speak", {"utterance": "hi"})) + self.assertEqual(got, ["speak"]) # emit_legacy bridged it + + def test_dual_listener_fires_once(self): + bus = FakeBus() + calls = [] + handler = lambda m: calls.append(m.msg_type) + bus.on("speak", handler) + bus.on("ovos.utterance.speak", handler) + bus.emit(Message("speak", {"utterance": "hi"})) + self.assertEqual(len(calls), 1) # mirror deduped + + def test_distinct_listeners_each_fire_once(self): + bus = FakeBus() + legacy, spec = [], [] + bus.on("speak", lambda m: legacy.append(1)) + bus.on("ovos.utterance.speak", lambda m: spec.append(1)) + bus.emit(Message("speak", {"utterance": "hi"})) + self.assertEqual((len(legacy), len(spec)), (1, 1)) + + def test_flags_off_no_bridging(self): + bus = FakeBus(modernize=False, emit_legacy=False) + got = [] + bus.on("ovos.utterance.speak", lambda m: got.append(m.msg_type)) + bus.emit(Message("speak", {"utterance": "hi"})) + self.assertEqual(got, []) # no translation -> spec listener not reached + + def test_unmapped_topic_untouched(self): + bus = FakeBus() + got = [] + bus.on("my.custom.topic", lambda m: got.append(m.msg_type)) + bus.emit(Message("my.custom.topic", {"x": 1})) + self.assertEqual(got, ["my.custom.topic"]) + + def test_shape_changing_payload_reshaped_for_spec_listener(self): + # a spec listener on the counterpart of a SHAPE-CHANGING legacy topic + # receives the payload in ITS shape, not a verbatim legacy copy. + # detach_intent -> ovos.intent.deregister splits the compound + # "skill:intent" name into skill_id + intent_name. + bus = FakeBus() + got = [] + bus.on("ovos.intent.deregister", lambda m: got.append(dict(m.data))) + bus.emit(Message("detach_intent", {"intent_name": "skill.foo:HelloIntent"})) + self.assertEqual(len(got), 1) + self.assertEqual(got[0], {"skill_id": "skill.foo", "intent_name": "HelloIntent"}) + + def test_shape_changing_payload_reshaped_for_legacy_listener(self): + bus = FakeBus() + got = [] + bus.on("detach_intent", lambda m: got.append(dict(m.data))) + bus.emit(Message("ovos.intent.deregister", + {"skill_id": "skill.foo", "intent_name": "HelloIntent"})) + self.assertEqual(len(got), 1) + # rejoined to the legacy compound shape + self.assertEqual(got[0].get("intent_name"), "skill.foo:HelloIntent") + + def test_payload_compatible_rename_delivered_equivalent(self): + bus = FakeBus() + got = [] + bus.on("ovos.utterance.speak", lambda m: got.append(dict(m.data))) + bus.emit(Message("speak", {"utterance": "hi", "lang": "en-us"})) + self.assertEqual(got, [{"utterance": "hi", "lang": "en-us"}]) # identity + + def test_remove_cleans_up(self): + bus = FakeBus() + calls = [] + handler = lambda m: calls.append(1) + bus.on("speak", handler) + bus.on("ovos.utterance.speak", handler) + bus.remove("speak", handler) + bus.remove("ovos.utterance.speak", handler) + self.assertNotIn(handler, bus._handler_guards) + bus.emit(Message("speak", {"utterance": "hi"})) + self.assertEqual(calls, []) + + +class TestFakeBusFlagResolution(unittest.TestCase): + """When the kwarg is omitted, flags resolve via env -> websocket.* config -> + default True, matching MessageBusClient._bus_flag. An explicit kwarg wins.""" + + def _legacy_mirrored(self, bus): + # emit a legacy topic; if emit_legacy bridging is on a spec listener fires + got = [] + bus.on("ovos.utterance.speak", lambda m: got.append(m.msg_type)) + if isinstance(bus, AsyncFakeBus): + _run(bus.emit(Message("speak", {"utterance": "hi"}))) + else: + bus.emit(Message("speak", {"utterance": "hi"})) + return got == ["ovos.utterance.speak"] + + def test_default_true_no_env_mirrors(self): + with patch.dict("os.environ", {}, clear=False): + import os + os.environ.pop("OVOS_BUS_MODERNIZE", None) + os.environ.pop("OVOS_BUS_EMIT_LEGACY", None) + self.assertTrue(self._legacy_mirrored(FakeBus())) + self.assertTrue(self._legacy_mirrored(AsyncFakeBus())) + + def test_env_false_disables_mirror(self): + with patch.dict("os.environ", + {"OVOS_BUS_MODERNIZE": "false", + "OVOS_BUS_EMIT_LEGACY": "false"}): + self.assertFalse(self._legacy_mirrored(FakeBus())) + self.assertFalse(self._legacy_mirrored(AsyncFakeBus())) + + def test_explicit_kwarg_beats_env(self): + # env says off, but an explicit modernize=True kwarg still mirrors + with patch.dict("os.environ", + {"OVOS_BUS_MODERNIZE": "false", + "OVOS_BUS_EMIT_LEGACY": "false"}): + self.assertTrue(self._legacy_mirrored(FakeBus(modernize=True))) + self.assertTrue(self._legacy_mirrored(AsyncFakeBus(modernize=True))) + + def test_explicit_false_kwarg_beats_unset_env(self): + import os + with patch.dict("os.environ", {}, clear=False): + os.environ.pop("OVOS_BUS_MODERNIZE", None) + os.environ.pop("OVOS_BUS_EMIT_LEGACY", None) + # default would mirror; explicit modernize=False suppresses it + self.assertFalse(self._legacy_mirrored(FakeBus(modernize=False))) + self.assertFalse(self._legacy_mirrored(AsyncFakeBus(modernize=False))) + + +if __name__ == "__main__": + unittest.main() diff --git a/test/unittests/test_lang.py b/test/unittests/test_lang.py index 19cbeb20..2f9cdb73 100644 --- a/test/unittests/test_lang.py +++ b/test/unittests/test_lang.py @@ -24,38 +24,41 @@ class TestStandardizeLangTag(unittest.TestCase): """Tests for standardize_lang_tag.""" - def test_macro_strips_region(self) -> None: - """standardize_lang_tag(macro=True) should return bare language code.""" + def test_macro_preserves_region(self) -> None: + """standardize_lang_tag(macro=True) preserves the region. + + ``macro`` is a langcodes concept — it controls *macrolanguage* + substitution (``cmn`` -> ``zh``, ``nb`` -> ``no``), not region + stripping. ``en-US`` round-trips unchanged.""" from ovos_utils.lang import standardize_lang_tag - # When langcodes not available, falls back to split on '-' - with patch.dict("sys.modules", {"langcodes": None}): - result = standardize_lang_tag("en-US", macro=True) - self.assertEqual(result, "en") + self.assertEqual(standardize_lang_tag("en-US", macro=True), "en-US") + self.assertEqual(standardize_lang_tag("en-us", macro=True), "en-US") def test_non_macro_preserves_region(self) -> None: - """standardize_lang_tag(macro=False) should keep the region part.""" + """standardize_lang_tag(macro=False) preserves the region too — + the difference between macro=True/False is macrolanguage + substitution, not region handling.""" from ovos_utils.lang import standardize_lang_tag - with patch.dict("sys.modules", {"langcodes": None}): - result = standardize_lang_tag("en-us", macro=False) - self.assertEqual(result, "en-US") + self.assertEqual(standardize_lang_tag("en-us", macro=False), "en-US") + + def test_macro_substitutes_macrolanguage(self) -> None: + """With ``macro=True``, langcodes maps a sublanguage onto its + macrolanguage. ``cmn`` (Mandarin) -> ``zh`` (Chinese); + ``macro=False`` keeps the original tag.""" + from ovos_utils.lang import standardize_lang_tag + self.assertEqual(standardize_lang_tag("cmn", macro=True), "zh") + self.assertEqual(standardize_lang_tag("cmn", macro=False), "cmn") - def test_no_region_tag(self) -> None: - """standardize_lang_tag with no '-' should return lowercased tag.""" + def test_fallback_without_langcodes(self) -> None: + """With langcodes unavailable, ``standardize_lang_tag`` falls + back to spec-tools (also region-preserving). ``macro`` is a + no-op in this branch.""" from ovos_utils.lang import standardize_lang_tag with patch.dict("sys.modules", {"langcodes": None}): - result = standardize_lang_tag("EN", macro=False) - self.assertEqual(result, "en") - - def test_with_langcodes_library(self) -> None: - """standardize_lang_tag should call langcodes.standardize_tag when available.""" - mock_langcodes = unittest.mock.MagicMock() - mock_langcodes.standardize_tag.return_value = "en" - - with patch.dict("sys.modules", {"langcodes": mock_langcodes}): - from ovos_utils.lang import standardize_lang_tag - result = standardize_lang_tag("en-US", macro=True) - # Result is whatever langcodes returns - self.assertIsInstance(result, str) + self.assertEqual( + standardize_lang_tag("en-us", macro=True), "en-US") + self.assertEqual( + standardize_lang_tag("EN", macro=False), "en") class TestGetLanguageDir(unittest.TestCase):