Test collection order broken when using page fixture

[joniemi]

Description

Given a folder contains multiple test modules each of which defining a test function that references the page fixture and one parametrized variable, when calling pytest --collect-only, then the list of test cases is not in module order but shuffled at seemingly random.

The issue occurs since pytest version 8.4.0. Tested with python 3.13.7, playwright 1.56.0 and pytest-playwright 0.7.1.

How to reproduce

pip install playwright pytest pytest-playwright

test_a.py

import pytest

@pytest.mark.parametrize("param", ["apple", "orange", "pear"])
def test_1(page, param):
    assert True

test_b.py

import pytest

@pytest.mark.parametrize("param", ["dog", "cat", "mouse"])
def test_2(page, param):
    assert True

Calling pytest --collect-only results in

<Package tests>
  <Module test_a.py>
    <Function test_1[chromium-apple]>
  <Module test_b.py>
    <Function test_2[chromium-dog]>
  <Module test_a.py>
    <Function test_1[chromium-orange]>
  <Module test_b.py>
    <Function test_2[chromium-cat]>
  <Module test_a.py>
    <Function test_1[chromium-pear]>
  <Module test_b.py>
    <Function test_2[chromium-mouse]>

Note: After removing the page fixture from one of the tests, the collection order is as expected:

<Package tests>
  <Module test_a.py>
    <Function test_1[apple]>
    <Function test_1[orange]>
    <Function test_1[pear]>
  <Module test_b.py>
    <Function test_2[chromium-dog]>
    <Function test_2[chromium-cat]>
    <Function test_2[chromium-mouse]>

[ltsuda]

Do you think is related to new feature from pytest 8.4.0 itself and not from playwright-pytest?

https://github.com/pytest-dev/pytest/releases/tag/8.4.0

See "New features" -> "#12749: pytest traditionally collects classes/functions in the test module namespace even if they are imported from another file."

[joniemi]

Yes, I think so too. I guess it's up to the playwright plugin to adapt to the breaking changes. I opened an issue in pytest some time ago and they adviced me to open an issue here.