Refactored tests to fix OS-related path handling

[psmyrek]

🚀 Summary

Refactored tests to fix OS-related path handling.

---

📌 Related issues

• Closes [dev] Some unit tests fail in ckeditor5-dev-tests package on Windows ckeditor5#19931.

---

💡 Additional information

I fixed path-related tests not only in ckeditor5-dev-tests package, but also in ckeditor5-dev-translations. Also, one flaky test in ckeditor5-dev-release-tools has been fixed too.

[przemyslaw-zan]

pnpm run test in ckeditor5-dev-tests fails 2 tests in
createserver.js:
should listen on given port and should use "readline" ... on Windows, both with
listen EACCES: permission denied:

Failed run

$ pnpm run test

> @ckeditor/ckeditor5-dev-tests@56.1.0 test C:\Repositories\ckeditor5-dev\packages\ckeditor5-dev-tests
> vitest run --config vitest.config.js


 RUN  v4.1.2 C:/Repositories/ckeditor5-dev/packages/ckeditor5-dev-tests

 ✓ tests/utils/getdefinitionsfromfile.js (5 tests) 10ms
 ✓ tests/utils/getrelativefilepath.js (14 tests) 12ms
 ✓ tests/utils/automated-tests/assertions/attribute.js (6 tests) 12ms
 ✓ tests/utils/automated-tests/assertions/equal-markup.js (8 tests) 23ms
 ✓ tests/utils/transformfileoptiontotestglob.js (35 tests) 71ms
 ✓ tests/utils/manual-tests/copyassets.js (1 test) 10ms
 ✓ tests/utils/getprotobufjsinquirewebpackrule.js (1 test) 5ms
 ✓ tests/utils/automated-tests/treatwarningsaserrorswebpackplugin.js (1 test) 721ms
     ✓ should reassign warnings to errors, emit the code and make the bundle fail  719ms
 ✓ tests/tasks/runautomatedtests.js (28 tests) 531ms
 ✓ tests/utils/parsearguments.js (46 tests) 56ms
 ✓ tests/utils/automated-tests/getkarmaconfig.js (11 tests) 38ms
 ✓ tests/utils/manual-tests/getwebpackconfig.js (6 tests) 9ms
 ✓ tests/utils/automated-tests/getwebpackconfig.js (14 tests) 128ms
 ✓ tests/utils/manual-tests/compilehtmlfiles.js (13 tests) 23ms
 ✓ tests/utils/manual-tests/removedir.js (2 tests) 5ms
 ✓ tests/utils/manual-tests/compilescripts.js (8 tests) 10ms
 ✓ tests/tasks/runmanualtests.js (16 tests) 9ms
 ✓ tests/index.js (3 tests) 3ms
(node:28576) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGINT listeners added to [process]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit
(Use `node --trace-warnings ...` to show where the warning was created)
 ❯ tests/utils/manual-tests/createserver.js (17 tests | 2 failed) 2887ms
     ✓ should start http server 69ms
     × should listen on given port 1013ms
     ✓ should listen on 8125 port if no specific port was given 67ms
     ✓ should call the specified callback when the server is running (e.g. to allow running web sockets) 61ms
     × should use "readline" to listen to the SIGINT event on Windows 1011ms
     ✓ should try next port when the requested port is in use 63ms
     ✓ should reject when a non-EADDRINUSE error occurs 3ms
       ✓ should handle a request for a favicon (`/favicon.ico`) 54ms
       ✓ should handle a root request (`/`) 68ms
       ✓ should handle a request for a static resource (`*.html`) 60ms
       ✓ should handle a request for a static resource (`*.js`) 65ms
       ✓ should handle a request for a static resource (`*.json`) 65ms
       ✓ should handle a request for a static resource (`*.js.map`) 52ms
       ✓ should handle a request for a static resource (`*.css`) 52ms
       ✓ should handle a request for a static resource (`*.png`) 65ms
       ✓ should handle a request for a static resource (`*.jpg`) 52ms
       ✓ should handle a request for a non-existing resource 66ms

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Failed Tests 2 ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯

 FAIL  tests/utils/manual-tests/createserver.js > createManualTestServer() > should listen on given port
AssertionError: expected "vi.fn()" to be called at least once
 ❯ tests/utils/manual-tests/createserver.js:66:25
     64|
     65|   await vi.waitFor( () => {
     66|    expect( loggerStub ).toHaveBeenCalled();
       |                         ^
     67|   } );
     68|

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/2]⎯

 FAIL  tests/utils/manual-tests/createserver.js > createManualTestServer() > should use "readline" to listen to the SIGINT event on Windows
AssertionError: expected "vi.fn()" to be called at least once
 ❯ tests/utils/manual-tests/createserver.js:118:52
    116|
    117|   await vi.waitFor( () => {
    118|    expect( vi.mocked( readline ).createInterface ).toHaveBeenCalled();
       |                                                    ^
    119|   } );
    120|

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[2/2]⎯

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Unhandled Errors ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯

Vitest caught 2 unhandled errors during the test run.
This might cause false positive tests. Resolve unhandled errors to make sure your tests are not affected.

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Unhandled Rejection ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Error: listen EACCES: permission denied 0.0.0.0:49888
 ❯ Server.setupListenHandle [as _listen2] node:net:1926:21
 ❯ listenInCluster node:net:2005:12
 ❯ Server.listen node:net:2110:7
 ❯ Server.Mock [as listen] ../../node_modules/.pnpm/@vitest+spy@4.1.2/node_modules/@vitest/spy/dist/index.js:332:34
 ❯ tryListenOnPort lib/utils/manual-tests/createserver.js:52:9
     50|  } );
     51|
     52|  server.listen( port, () => {
       |         ^
     53|   // SIGINT isn't caught on Windows in process. However, `CTRL+C` can be caught
     54|   // by `readline` module. After that we can emit SIGINT to the process manually.
 ❯ lib/utils/manual-tests/createserver.js:24:3
 ❯ Module.createManualTestServer lib/utils/manual-tests/createserver.js:23:9
 ❯ tests/utils/manual-tests/createserver.js:63:25
 ❯ ../../node_modules/.pnpm/@vitest+runner@4.1.2/node_modules/@vitest/runner/dist/chunk-artifact.js:302:11

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Serialized Error: { code: 'EACCES', errno: -4092, syscall: 'listen', address: '0.0.0.0', port: 49888 }
This error originated in "tests/utils/manual-tests/createserver.js" test file. It doesn't mean the error was thrown inside the file itself, but while it was running.
The latest test that might've caused the error is "should listen on given port". It might mean one of the following:
- The error was thrown, while Vitest was running this test.
- If the error occurred after the test had been completed, this was the last documented test before it was thrown.

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Unhandled Rejection ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Error: listen EACCES: permission denied 0.0.0.0:49900
 ❯ Server.setupListenHandle [as _listen2] node:net:1926:21
 ❯ listenInCluster node:net:2005:12
 ❯ Server.listen node:net:2110:7
 ❯ Server.Mock [as listen] ../../node_modules/.pnpm/@vitest+spy@4.1.2/node_modules/@vitest/spy/dist/index.js:332:34
 ❯ tryListenOnPort lib/utils/manual-tests/createserver.js:52:9
     50|  } );
     51|
     52|  server.listen( port, () => {
       |         ^
     53|   // SIGINT isn't caught on Windows in process. However, `CTRL+C` can be caught
     54|   // by `readline` module. After that we can emit SIGINT to the process manually.
 ❯ lib/utils/manual-tests/createserver.js:24:3
 ❯ Module.createManualTestServer lib/utils/manual-tests/createserver.js:23:9
 ❯ tests/utils/manual-tests/createserver.js:115:25
 ❯ ../../node_modules/.pnpm/@vitest+runner@4.1.2/node_modules/@vitest/runner/dist/chunk-artifact.js:302:11

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Serialized Error: { code: 'EACCES', errno: -4092, syscall: 'listen', address: '0.0.0.0', port: 49900 }
This error originated in "tests/utils/manual-tests/createserver.js" test file. It doesn't mean the error was thrown inside the file itself, but while it was running.
The latest test that might've caused the error is "should use "readline" to listen to the SIGINT event on Windows". It might mean one of the following:
- The error was thrown, while Vitest was running this test.
- If the error occurred after the test had been completed, this was the last documented test before it was thrown.
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯


 Test Files  1 failed | 18 passed (19)
      Tests  2 failed | 233 passed (235)
     Errors  2 errors
   Start at  17:45:25
   Duration  5.70s (transform 2.88s, setup 7.85s, import 20.10s, tests 4.56s, environment 2ms)

 ELIFECYCLE  Test failed. See above for more details.

Cause: these tests bind a real socket, and their ports fall inside a Windows
reserved TCP range (netsh interface ipv4 show excludedportrange protocol=tcp →
49832-49931 here). Same issue the PR already fixed for 49800 → 49700, but two
ports were missed.

Fix Option A - relocate the ports

Move them below the reserved range (e.g. 49236/49237, next to the existing 49234).

Fix Option B - mock server.listen

Stop binding real sockets in the unit tests. In the beforeEach createServer mock,
replace the bare spy with a mock that just fires the callback:

vi.spyOn( server, 'listen' ).mockImplementation( ( port, callback ) => {
	process.nextTick( callback );
	return server;
} );

Also rework should try next port when the requested port is in use to simulate
EADDRINUSE (emit it on the first port, succeed on the next) instead of binding a
real blocking server. No dependency on OS port availability - reliable on every machine.

[psmyrek]

Thanks for the review. Option B is now implemented to skip binding the real socket.