Skip to content

[WIP] Do not rely on vitest for memory tests#1791

Open
peaBerberian wants to merge 1 commit intodevfrom
simple-test-lib
Open

[WIP] Do not rely on vitest for memory tests#1791
peaBerberian wants to merge 1 commit intodevfrom
simple-test-lib

Conversation

@peaBerberian
Copy link
Copy Markdown
Collaborator

@peaBerberian peaBerberian commented Jan 23, 2026

This is a low priority proposal.
Even if not merged, I plan to use it at least on my side to be able to run memory tests on my computer.


Until now, we relied on vitest + webdriverio to run our "memory tests", which check for memory leaks.

Those two are huge and complex libraries, a complexity that we don't need for those tests (or even for integration tests): vitest can readily patch native browser APIs, mock files, hoist logic blocks before tests - none of that is needed here.

More importantly: I always struggled to run them on my computer.
After some headaches, I had a container-based setup that worked (without a container it ended up picking the wrong network interface, or not finding the browser, or timeouting ...) but now it is broken again since very recently (the browser does not run, not sure what changed, but looking up why it broke each time is too time consuming).


Yet what we want to do in our memory tests is extremely simple by comparison to what those libs allow: we want to run a test page on a specific browser (with specific flags), then we want to log the results and finally exit either with success or failure depending on that result.

On that point we already have our own testing library for our "performance tests", because the paradigm of those tests is very different from the ones offered by the testing libs we know of (those tests compare the timings of some operations of two builds of the RxPlayer, by running them hundreds of time on each, and then check if there's a sensible diff with a null hypothesis test + some heuristics).

So vitest doesn't work for me, is very complex for reasons we don't need, and we already have a simpler setup which works for our need.

Adapting that setup for our memory tests was very straightforward. For now I copy-pasted the performance tests's run.js file (the test server + browser runner) and adapted it to memory tests just because I did not spend any time factorizing yet.

I also wrote here a small vitest/jest-compatible mininal testing library (providing describe / it / beforeEach etc. - and re-exporting vitest's assertions).
I use esbuild to bundle the test file and then run a browser on the output, exactly like for our performance tests.

@peaBerberian peaBerberian force-pushed the simple-test-lib branch 22 times, most recently from 70a05b4 to 5f71d29 Compare January 26, 2026 12:48
@canalplus canalplus deleted a comment from github-actions Bot Jan 26, 2026
@canalplus canalplus deleted a comment from github-actions Bot Jan 26, 2026
@peaBerberian peaBerberian force-pushed the dev branch 6 times, most recently from 374c14e to 7bf9f96 Compare January 26, 2026 17:50
@peaBerberian peaBerberian force-pushed the dev branch 3 times, most recently from 0142e34 to 1fd9df3 Compare January 27, 2026 11:59
@github-actions
Copy link
Copy Markdown

✅ Automated performance checks have passed on commit c1a3988d34fd95835bc04287bf2e754cd6d5fe12 with the base branch dev.

Details

Performance tests 1st run output

No significative change in performance for tests:

Name Mean Median
loading 25.12ms -> 25.34ms (-0.221ms, z: 0.26171) 35.70ms -> 35.70ms
seeking 363.53ms -> 359.44ms (4.086ms, z: 0.84014) 16.95ms -> 16.95ms
audio-track-reload 32.19ms -> 32.33ms (-0.134ms, z: 1.11138) 48.00ms -> 48.15ms
cold loading multithread 51.29ms -> 50.59ms (0.704ms, z: 10.76502) 76.05ms -> 74.85ms
seeking multithread 57.04ms -> 43.02ms (14.022ms, z: 0.06342) 19.95ms -> 20.10ms
audio-track-reload multithread 30.70ms -> 30.37ms (0.330ms, z: 3.61789) 45.00ms -> 44.70ms
hot loading multithread 20.31ms -> 20.28ms (0.031ms, z: 3.29128) 29.85ms -> 29.55ms

@canalplus canalplus deleted a comment from github-actions Bot Feb 27, 2026
@peaBerberian peaBerberian added the Priority: 4 (Very low) This issue or PR has a very low priority. label Apr 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Priority: 4 (Very low) This issue or PR has a very low priority.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant