WIP: Test: notebooks (#47)

* Conditionally skip when not testing jupyter

* Make notebook testing a separate job

This should have the nice benefit of no testing examples across the
board when not running nbval and

* Fix ci build

* Glob notebooks

* pre-commit: Exclude examples from mypy check

* Just exclude examples/conftest.py

* See if -s shows the error

* TEMP: Skip comparing stderr on CI

Just checking if this passes the build

* WIP: Try to figure out why pygfx+jupyter has problems

Claude thinks this will work?

* Ignore all stderr on pygfx

Not sure we could do better with the current state of nbval

Author: gselzer

.github/workflows/ci.yml

@@ -81,12 +81,17 @@ jobs:
           sudo apt install -y libegl1-mesa-dev libgl1-mesa-dri libxcb-xfixes0-dev mesa-vulkan-drivers
 
       - name: Install dependencies
-        run: uv sync --no-dev --group test --extra ${{ matrix.gfx }} --extra ${{matrix.canvas}} ${{ matrix.python-version >= '3.12' && matrix.gfx == 'pygfx' && '--extra imgui' || '' }} ${{ matrix.canvas == 'pyqt6' && '--group testqt' || '' }}
+        run: uv sync --no-dev --group test --extra ${{ matrix.gfx }} --extra ${{matrix.canvas}} ${{ matrix.python-version >= '3.12' && matrix.gfx == 'pygfx' && '--extra imgui' || '' }} ${{ matrix.canvas == 'pyqt6' && '--group testqt' || '' }} ${{ matrix.canvas == 'jupyter' && '--group testjupyter' || '' }}
 
       - name: Test
         shell: bash
         run: uv run --no-sync coverage run -p -m pytest -v
 
+      - name: Test notebooks
+        if: matrix.canvas == 'jupyter'
+        shell: bash
+        run: uv run --no-sync coverage run -p -m pytest -v --nbval examples/*.ipynb
+
       - name: Upload coverage
         uses: actions/upload-artifact@v6
         with:

.pre-commit-config.yaml

@@ -41,6 +41,9 @@ repos:
         types_or: [python, pyi]
         require_serial: true
         entry: uv run mypy
+        # Prevents mypy from complaining about multiple conftest modules
+        # this one is the least necessary to check
+        exclude: "^examples/conftest\\.py$"
       - id: pyright
         name: pyright
         language: system

examples/conftest.py

@@ -0,0 +1,44 @@
+"""Pytest setup for notebook tests."""
+
+from typing import TYPE_CHECKING
+
+from nbval.plugin import IPyNbCell  # type: ignore
+
+if TYPE_CHECKING:
+    import pytest
+
+
+def pytest_collection_modifyitems(
+    session: "pytest.Session", config: "pytest.Config", items: "list[pytest.Item]"
+) -> None:
+    """Add sanitizers to Notebook tests."""
+    for item in items:
+        # For each notebook cell being tested...
+        if not isinstance(item, IPyNbCell):
+            continue
+        try:
+            # FIXME: On pygfx some lines are printed to stderr that we don't want to
+            # compare. Ideally we could ignore ONLY these lines, e.g.
+            # libEGL warning: DRI3 error: Could not get DRI3 device
+            # libEGL warning: Ensure your X server supports DRI3 to get accelerated...
+            # Unable to find extension: VK_EXT_physical_device_drm
+            import pygfx  # noqa: F401
+
+            item.parent.skip_compare += ("stderr",)  # pyright: ignore
+        except ImportError:
+            pass
+
+        # Add the following (regex) sanitations...
+        # (Note that both the expected and actual cell outputs will be sanitized)
+        item.parent.sanitize_patterns.update(  # pyright: ignore
+            {
+                # -- Convert all jupyter canvases to the same (pygfx) class name -- #
+                r"CanvasBackend": "JupyterRenderCanvas",
+                # -- Convert canvas sizes to integers (vispy uses floats) -- #
+                r"(\d+)\.0px": r"\1px",  # e.g. 600.0px -> 600px
+                # -- Normalize div snapshot ids (unique to each run) -- #
+                r"snapshot-[a-f0-9]+": "snapshot-XXXXX",
+                # -- Normalize canvas data (within the img tag) -- #
+                r"data:image/png;base64,[A-Za-z0-9+/=]+": "data:image/png;base64,XXXXX",
+            }
+        )

pyproject.toml

@@ -77,6 +77,7 @@ imgui = [
 [dependency-groups]
 test = ["pytest>=8", "pytest-cov>=6"]
 testqt = [{ include-group = "test" }, "pytest-qt >=4.4"]
+testjupyter = [{ include-group = "test" }, "nbval"]
 docs = [
     "mike>=2.1.3",
     "mkdocs>=1.6.1",