Optimize Slow CI Tests to Reduce Runtime

[bhimrazy]

Optimize Slow CI Tests to Reduce Runtime

Description:
Our CI tests are hitting timeout limits due to some slow tests and increased test count. Increasing timeouts helps temporarily, but we need to optimize tests for faster CI runs.

Follow up to #526

What to do:

• Identify and analyze slow tests (e.g., test_server_with_disabled_timeout, test_readme).
• Refactor or mock heavy dependencies to speed them up.
• Improve overall CI runtime for quicker feedback.

Slowest 100 durations

============================ slowest 100 durations =============================
15.61s call     tests/test_simple.py::test_server_with_disabled_timeout[False]
13.62s call     tests/test_simple.py::test_server_with_disabled_timeout[True]
10.02s call     tests/test_readme.py::test_readme
10.01s call     tests/test_transport.py::TestTransportIntegration::test_send_receive_cycle
9.27s call     tests/test_lit_server.py::test_inject_context
7.14s call     tests/e2e/test_e2e.py::test_e2e_openai_embedding_with_batching
7.09s call     tests/test_simple.py::test_workers_health_with_custom_health_method[True]
7.09s call     tests/test_simple.py::test_workers_health_with_custom_health_method[False]
6.56s call     tests/test_callbacks.py::test_request_tracker
6.39s call     tests/test_lit_server.py::test_concurrent_async_inference[100]
6.38s call     tests/test_lit_server.py::test_concurrent_async_inference[50]
6.30s call     tests/test_lit_server.py::test_concurrent_async_inference[10]
5.71s call     tests/e2e/test_e2e.py::test_e2e_openai_with_batching
5.58s call     tests/test_lit_server.py::test_http_exception
5.46s call     tests/e2e/test_e2e.py::test_openai_parity
5.33s call     tests/e2e/test_e2e.py::test_run
5.31s call     tests/e2e/test_e2e.py::test_run_with_port
5.16s call     tests/e2e/test_e2e.py::test_e2e_default_batching
5.13s call     tests/e2e/test_e2e.py::test_e2e_batched_streaming
5.09s call     tests/test_simple.py::test_workers_health_custom_path[False]
5.09s call     tests/test_simple.py::test_workers_health_custom_path[True]
5.08s call     tests/test_simple.py::test_workers_health[True]
5.08s call     tests/e2e/test_e2e.py::test_openai_embedding_parity
5.08s call     tests/test_simple.py::test_workers_health[False]
5.06s call     tests/e2e/test_e2e.py::test_openai_parity_with_tools
5.06s call     tests/e2e/test_e2e.py::test_openai_parity_with_audio_input
5.06s call     tests/e2e/test_e2e.py::test_openai_parity_with_image_input
5.06s call     tests/e2e/test_e2e.py::test_openai_parity_with_response_format
5.03s call     tests/e2e/test_e2e.py::test_e2e_default_spec
5.02s call     tests/e2e/test_e2e.py::test_e2e_default_async_streaming
5.02s call     tests/e2e/test_e2e.py::test_e2e_default_api
5.02s call     tests/e2e/test_e2e.py::test_e2e_single_streaming

Goal:
Faster, more reliable CI without needing longer timeouts.

[aniketmaurya]

I think we should consider separating e2e tests from the unit tests since they are different in nature from the rest.

e2es starts a real server whereas the rest of the tests are unit and function tests. This can save us a lot of CI time.

[bhimrazy]

Absolutely — and we could also run the unit tests in parallel to speed things up even more.

Do you think it makes sense to move all unit tests into a tests/unit directory to make this separation clearer?

[gspeter-max]

Test Organization Already Complete ✅

Hey @bhimrazy @aniketmaurya - I wanted to point out that the test directory structure you discussed has already been implemented!

Current state:

tests/
├── unit/           (32 test files) ✅
├── integration/    (1 test file) ✅
├── e2e/            (16 test files) ✅

pytest.ini already has the markers defined:

• unit - unit tests
• integration - integration tests
• e2e - end-to-end tests

---

The Real Issue: CI Doesn't Use This Organization ❌

The CI workflow (line 64 in .github/workflows/ci-testing.yml) still runs ALL tests together:

pytest --cov=litserve tests/ -v -s --durations=100

This means:

• Unit, integration, and E2E tests all run in every CI job
• No separation of fast vs slow tests
• E2E tests (which start real servers) run on every PR

---

Suggested Next Steps

The actual optimization needed is to update the CI configuration to:

1. Run unit tests separately - fast feedback on every PR
2. Run integration tests separately - medium feedback
3. Run E2E tests only on main/nightly - slow, real server tests

Example:

# Fast feedback - every PR
- pytest -m unit tests/unit/

# Medium feedback - every PR  
- pytest -m integration tests/integration/

# Slow - only main branch or nightly
- pytest -m e2e tests/e2e/

Should we update the CI workflow to take advantage of the existing test organization?