Skip to content

Conversation

@beinan
Copy link
Collaborator

@beinan beinan commented Feb 12, 2026

Summary

Adds comprehensive tests to verify that the Python Global Interpreter Lock (GIL) is properly released during graph operations, addressing issue #97.

Findings

The codebase already properly releases the GIL!

The existing BackgroundExecutor (introduced in python/src/executor.rs) correctly uses py.allow_threads() to release the GIL during all blocking operations:

  • block_on() method releases GIL before blocking on futures
  • spawn() method releases GIL when spawning background tasks
  • All Python binding methods use RT.block_on(Some(py), ...) which properly releases the GIL

Tests Added

This PR adds verification tests to document and ensure this behavior continues:

  1. test_gil_release_during_query_execution - Verifies GIL release during CypherQuery.execute()
  2. test_gil_release_during_to_sql - Verifies GIL release during CypherQuery.to_sql()
  3. test_gil_release_during_explain - Verifies GIL release during CypherQuery.explain()
  4. test_concurrent_query_execution - Tests parallel query execution across multiple threads
  5. test_gil_release_with_heavy_query (slow) - Tests GIL release with large datasets

Test Strategy

Each test:

  1. Starts a background thread that increments a heartbeat counter
  2. Executes a graph operation in the main thread
  3. Verifies the background thread made progress (heartbeats > threshold)

If GIL was not released, the background thread would be starved and heartbeats would be 0.

Test Results

============================= test session starts ==============================
python/tests/test_gil_release.py::test_gil_release_during_query_execution PASSED
python/tests/test_gil_release.py::test_gil_release_during_to_sql PASSED
python/tests/test_gil_release.py::test_concurrent_query_execution PASSED
python/tests/test_gil_release.py::test_gil_release_during_explain PASSED
python/tests/test_gil_release.py::test_gil_release_with_heavy_query PASSED

============================== 5 passed in 0.45s

✅ All tests pass

Closes

Closes #97

🤖 Generated with Claude Code

beinan and others added 2 commits February 12, 2026 09:38
Add comprehensive tests to verify that the Global Interpreter Lock (GIL)
is properly released during graph operations, allowing concurrent Python
thread execution.

Tests verify GIL release during:
- Query execution (execute)
- SQL translation (to_sql)
- Query explanation (explain)
- Concurrent query execution across multiple threads
- Heavy queries with large datasets

All tests pass, confirming that the BackgroundExecutor properly releases
the GIL via py.allow_threads() during blocking operations.

Addresses lance-format#97

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@beinan beinan closed this Feb 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Release GIL during remote storage operations to prevent blocking

1 participant